Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
acbbfdf9e2 | ||
|
|
06aca980d0 | ||
|
|
f3f1ceb833 | ||
|
|
b18ffa0b0a |
6
.gitignore
vendored
6
.gitignore
vendored
@@ -28,10 +28,8 @@ dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
!frontend/src/admin/lib/
|
||||
!frontend/src/admin/lib/**
|
||||
lib64/
|
||||
/lib/
|
||||
/lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
|
||||
15
TODO.md
15
TODO.md
@@ -11,10 +11,8 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
|
||||
- [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
|
||||
- [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
|
||||
- [ ] Earth high-resolution basemap tiles: implement the viewport-loaded imagery layer described in [Earth High Resolution Basemap Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-resolution-basemap-tiles-plan.md), using high-precision coastline as the alignment reference instead of replacing the globe with one huge texture.
|
||||
- [ ] Presentation controller ownership: replace the singleton card fallback in [presentation-controller.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/presentation-controller.js) with a presentation/card token check before BGP/News migrate onto the shared controller, so connectors only attach to their owning card.
|
||||
- [ ] BGP frontend maintainability: split [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by responsibility into data loading, marker rendering, overlays, and animation once the current interaction behavior is stable.
|
||||
- [ ] Optional BGP marker experiment: evaluate HTML markers for BGP incident/collector points if WebGL marker density or fixed screen-size clickability becomes a real blocker.
|
||||
- [ ] Earth news cruise: connect Earth news to the generic cruise queue via a news adapter rather than coupling news-specific sequencing into `main.js`.
|
||||
|
||||
## Compute Centers And Location
|
||||
|
||||
@@ -45,13 +43,6 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
|
||||
- [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches.
|
||||
- [ ] BGP geography fallback: evaluate `inetnum` / `inet6num` whois as a finer fallback layer after `prefix_geography`, `OpenGeoFeed`, and RIR delegated data.
|
||||
|
||||
## Platform
|
||||
|
||||
- [ ] Earth preferences scope: keep current device-local Earth preferences in `localStorage`; only design backend user preferences if account-level synchronization becomes a real product requirement.
|
||||
- [ ] System logs: finish a usable Planet log viewing flow that covers backend, frontend, AI Provider, and collector/task logs, with filtering and tailing.
|
||||
- [ ] Console UI modernization: gradually replace Ant Design with Planet-owned components and a consistent Tabler Icons based icon system.
|
||||
- [ ] Earth live sync: design a unified realtime invalidation path for summary/BGP/satellite updates if polling and current WebSocket channels become insufficient.
|
||||
|
||||
## Archive
|
||||
|
||||
Archived items stay here so old context is not lost. Completed items remain checked; obsolete, invalid, or superseded items stay unchecked and include the reason.
|
||||
@@ -75,6 +66,11 @@ Archived items stay here so old context is not lost. Completed items remain chec
|
||||
- [x] Added OpenGeoFeed as a high-quality prefix geography override source.
|
||||
- [x] Made RIR delegated data a prefix geography fallback rather than the primary source.
|
||||
- [x] Added route leak and path instability / flap detectors after the activity layer work.
|
||||
- [x] Console UI modernization. Admin is now the only console, legacy Ant Design / Admin Next code paths and dependencies have been removed, and current console UI uses Planet-owned components.
|
||||
- [x] Earth news cruise adapter. News cruise now uses `news-cruise-adapter.js` and is wired from `main.js` instead of keeping news-specific sequencing directly in the main Earth loop.
|
||||
- [x] Presentation controller ownership. `PresentationController` now guards async ownership through active request identity checks, and current callers pass per-request card targets so stale connector/card work cannot overwrite the active presentation.
|
||||
- [x] Earth live sync. Database writes now flow through `earth_data_change_events`, `earth_db_change_listener`, layer adapters, cache invalidation, and the `earth_updates` WebSocket channel; the Earth frontend debounces updates and refreshes BGP, cables, compute centers, satellites, vessels, news, and interactables by layer.
|
||||
- [x] System logs. Log sources now normalize into `LogEvent`, Admin supports snapshot filtering plus WebSocket tail/follow, task/detail views deep-link into prefiltered logs, and Admin runtime errors report through the `admin-client` log source.
|
||||
|
||||
### Obsolete Or Superseded
|
||||
|
||||
@@ -85,3 +81,4 @@ Archived items stay here so old context is not lost. Completed items remain chec
|
||||
- [ ] Earth surface material overlay for boundary calibration. Superseded by the high-precision boundary tile plan; future work must use source-faithful boundary/coastline data rather than overlay calibration against the coarse base map.
|
||||
- [ ] Hardcoded Earth news source extraction as a standalone task. Superseded by the broader Earth news source configuration and collector plans.
|
||||
- [ ] Country-level compute-center fallback placement as a standalone task. Superseded by the shared location pipeline and registry/manual-review backlog.
|
||||
- [ ] Earth preferences backend sync scope. Superseded by the current product decision to keep Earth preferences device-local in `localStorage` until account-level synchronization becomes a real requirement.
|
||||
|
||||
@@ -19,7 +19,6 @@ from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.user import User
|
||||
from app.models.vessel import AISRawObservation
|
||||
from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
|
||||
from app.services.scheduler import (
|
||||
sync_datasource_job,
|
||||
)
|
||||
@@ -165,6 +164,8 @@ async def _load_latest_tasks(
|
||||
async def _load_collected_record_counts(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
*,
|
||||
exact_vessel_counts: bool = False,
|
||||
) -> dict[str, int]:
|
||||
if not sources:
|
||||
return {}
|
||||
@@ -185,14 +186,46 @@ async def _load_collected_record_counts(
|
||||
or "ais" in source
|
||||
]
|
||||
if vessel_sources:
|
||||
raw_result = await db.execute(
|
||||
select(AISRawObservation.source, func.count(AISRawObservation.id))
|
||||
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
|
||||
.where(AISRawObservation.source.in_(vessel_sources))
|
||||
.group_by(AISRawObservation.source)
|
||||
if exact_vessel_counts:
|
||||
exact_result = await db.execute(
|
||||
select(AISRawObservation.source, func.count(AISRawObservation.id))
|
||||
.where(AISRawObservation.source.in_(vessel_sources))
|
||||
.group_by(AISRawObservation.source)
|
||||
)
|
||||
for source, count in exact_result.all():
|
||||
counts[source] = max(counts.get(source, 0), int(count or 0))
|
||||
return counts
|
||||
|
||||
# AIS raw observations can be tens of millions of rows. Use planner
|
||||
# statistics for the datasource list instead of blocking page load on
|
||||
# source-level count(*) scans.
|
||||
stats_result = await db.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
COALESCE(pg_class.reltuples, 0)::bigint AS total_rows,
|
||||
pg_stats.most_common_vals::text AS source_values,
|
||||
pg_stats.most_common_freqs::text AS source_freqs
|
||||
FROM pg_class
|
||||
LEFT JOIN pg_stats
|
||||
ON pg_stats.schemaname = 'public'
|
||||
AND pg_stats.tablename = 'ais_raw_observations'
|
||||
AND pg_stats.attname = 'source'
|
||||
WHERE pg_class.relname = 'ais_raw_observations'
|
||||
LIMIT 1
|
||||
"""
|
||||
)
|
||||
)
|
||||
for source, count in raw_result.all():
|
||||
counts[source] = max(counts.get(source, 0), int(count or 0))
|
||||
stats = stats_result.mappings().first()
|
||||
if stats:
|
||||
total_rows = int(stats["total_rows"] or 0)
|
||||
values = str(stats["source_values"] or "").strip("{}")
|
||||
freqs = str(stats["source_freqs"] or "").strip("{}")
|
||||
source_values = [value.strip('"') for value in values.split(",") if value]
|
||||
source_freqs = [float(value) for value in freqs.split(",") if value]
|
||||
for source, freq in zip(source_values, source_freqs):
|
||||
if source in vessel_sources:
|
||||
counts[source] = max(counts.get(source, 0), int(round(total_rows * freq)))
|
||||
|
||||
return counts
|
||||
|
||||
@@ -929,7 +962,7 @@ async def get_datasource_row(
|
||||
[datasource],
|
||||
include_endpoint=include_endpoint,
|
||||
)
|
||||
record_counts = await _load_collected_record_counts(db, [datasource.source])
|
||||
record_counts = await _load_collected_record_counts(db, [datasource.source], exact_vessel_counts=True)
|
||||
return {
|
||||
"data": serialize_datasource_row(
|
||||
datasource,
|
||||
|
||||
@@ -21,6 +21,12 @@ from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.tv_streams import get_tv_settings_payload
|
||||
from app.services.earth_news import (
|
||||
get_earth_news_sources_payload,
|
||||
reset_earth_news_sources_payload,
|
||||
save_earth_news_sources_payload,
|
||||
test_news_source_config,
|
||||
)
|
||||
from app.services.earth_boundaries import (
|
||||
EarthBoundaryBuildError,
|
||||
get_boundary_build_status,
|
||||
@@ -100,6 +106,19 @@ class EarthAboutPayload(BaseModel):
|
||||
meta: list[EarthAboutMetaItem] = Field(default_factory=list)
|
||||
|
||||
|
||||
class EarthNewsSourcesPayload(BaseModel):
|
||||
cache_version: int | None = None
|
||||
source_tags: list[dict[str, Any]] = Field(default_factory=list)
|
||||
categories: list[dict[str, Any]] = Field(default_factory=list)
|
||||
item_tag_rules: list[dict[str, Any]] = Field(default_factory=list)
|
||||
sources: list[dict[str, Any]] = Field(default_factory=list)
|
||||
health: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class EarthNewsSourceTestPayload(BaseModel):
|
||||
source: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
|
||||
merged = DEFAULT_EARTH_BRAND.copy()
|
||||
if payload:
|
||||
@@ -324,6 +343,38 @@ async def reset_earth_about(
|
||||
return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
|
||||
|
||||
|
||||
@router.get("/news-sources")
|
||||
async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
|
||||
return await get_earth_news_sources_payload(db)
|
||||
|
||||
|
||||
@router.put("/news-sources")
|
||||
async def update_earth_news_sources(
|
||||
payload: EarthNewsSourcesPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await save_earth_news_sources_payload(db, payload.model_dump())
|
||||
|
||||
|
||||
@router.delete("/news-sources")
|
||||
@router.post("/news-sources/reset")
|
||||
async def reset_earth_news_sources(
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await reset_earth_news_sources_payload(db)
|
||||
|
||||
|
||||
@router.post("/news-sources/test")
|
||||
async def test_earth_news_source(
|
||||
payload: EarthNewsSourceTestPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await test_news_source_config(payload.source, db=db)
|
||||
|
||||
|
||||
@router.get("/oobe-status")
|
||||
async def get_earth_oobe_status(
|
||||
current_user: User | None = Depends(_get_optional_current_user),
|
||||
|
||||
@@ -90,7 +90,7 @@ async def get_interactables_geojson(
|
||||
return interactables_to_geojson(items)
|
||||
|
||||
payload = await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("interactables", layer=layer or "all"),
|
||||
key=earth_layer_cache.key("interactables", interactable_layer=layer or "all"),
|
||||
policy=INTERACTABLE_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
|
||||
@@ -1,16 +1,92 @@
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
from app.services.earth_news import (
|
||||
ALLOWED_NEWS_CATEGORY_KEYS,
|
||||
SUPPORTED_NEWS_LOCALES,
|
||||
REGION_ANCHORS,
|
||||
get_earth_news_payload,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _parse_categories(raw: str | None) -> set[str] | None:
|
||||
if raw is None or not raw.strip():
|
||||
return None
|
||||
requested = {item.strip().lower() for item in raw.split(",") if item.strip()}
|
||||
invalid = sorted(requested - set(ALLOWED_NEWS_CATEGORY_KEYS))
|
||||
if invalid:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={
|
||||
"message": "Unsupported news categories.",
|
||||
"invalid_categories": invalid,
|
||||
"allowed_categories": list(ALLOWED_NEWS_CATEGORY_KEYS),
|
||||
},
|
||||
)
|
||||
return requested or None
|
||||
|
||||
|
||||
def _parse_source_ids(raw: str | None) -> set[str] | None:
|
||||
if raw is None or not raw.strip():
|
||||
return None
|
||||
return {item.strip() for item in raw.split(",") if item.strip()} or None
|
||||
|
||||
|
||||
def _parse_limit(raw: int | None) -> int:
|
||||
if raw is None:
|
||||
return 12
|
||||
if raw < 1:
|
||||
raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
|
||||
return min(raw, 100)
|
||||
|
||||
|
||||
def _parse_locale(raw: str | None) -> str:
|
||||
if raw is None or not raw.strip():
|
||||
return "zh-CN"
|
||||
requested = raw.strip()
|
||||
if requested not in SUPPORTED_NEWS_LOCALES:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={
|
||||
"message": "Unsupported news locale.",
|
||||
"invalid_locale": requested,
|
||||
"allowed_locales": sorted(SUPPORTED_NEWS_LOCALES),
|
||||
},
|
||||
)
|
||||
return requested
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
|
||||
categories: str | None = Query(None, description="Comma-separated news category keys"),
|
||||
sources: str | None = Query(None, description="Comma-separated news source ids"),
|
||||
limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
|
||||
locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon, db=db)
|
||||
normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
|
||||
if normalized_region is not None and normalized_region not in REGION_ANCHORS:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={
|
||||
"message": "Unsupported news region.",
|
||||
"invalid_region": normalized_region,
|
||||
"allowed_regions": list(REGION_ANCHORS.keys()),
|
||||
},
|
||||
)
|
||||
return await get_earth_news_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
region=normalized_region,
|
||||
categories=_parse_categories(categories),
|
||||
source_ids=_parse_source_ids(sources),
|
||||
limit=_parse_limit(limit),
|
||||
locale=_parse_locale(locale),
|
||||
db=db,
|
||||
)
|
||||
|
||||
@@ -1,21 +1,19 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import json
|
||||
import secrets
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, status
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.config import ROOT_DIR, settings
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
from app.models.user import User
|
||||
from app.services.persistent_logs import record_audit_log, record_system_log
|
||||
from app.services.system_control import (
|
||||
@@ -38,49 +36,17 @@ from app.services.system_logs import (
|
||||
append_buffer_log,
|
||||
list_log_sources,
|
||||
normalize_log_level,
|
||||
read_database_log_snapshot,
|
||||
read_log_snapshot,
|
||||
read_observability_group_events,
|
||||
read_observability_groups,
|
||||
read_observability_raw_events,
|
||||
)
|
||||
from app.services.earth_layer_cache import earth_layer_cache
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _compact_log_context(context: dict | None) -> str:
|
||||
if not context:
|
||||
return ""
|
||||
allowed = {
|
||||
key: value
|
||||
for key, value in (context or {}).items()
|
||||
if key
|
||||
in {
|
||||
"status",
|
||||
"duration_ms",
|
||||
"provider",
|
||||
"model",
|
||||
"result_provider",
|
||||
"result_model",
|
||||
"collector_name",
|
||||
"datasource_id",
|
||||
"task_id",
|
||||
"snapshot_id",
|
||||
"raw_count",
|
||||
"transformed_count",
|
||||
"saved_count",
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"deleted",
|
||||
"result_count",
|
||||
"status_code",
|
||||
"error_type",
|
||||
"error",
|
||||
}
|
||||
}
|
||||
if not allowed:
|
||||
return ""
|
||||
return json.dumps(allowed, ensure_ascii=False, sort_keys=True)
|
||||
|
||||
|
||||
class RestartTaskCreate(BaseModel):
|
||||
action: str
|
||||
|
||||
@@ -146,12 +112,104 @@ class EarthClientLogEventCreate(BaseModel):
|
||||
url: str | None = None
|
||||
module: str | None = None
|
||||
detail: str | None = None
|
||||
fingerprint: str | None = None
|
||||
occurrence_count: int = 1
|
||||
metadata: dict[str, object] | None = None
|
||||
|
||||
|
||||
class EarthClientLogEventResponse(BaseModel):
|
||||
accepted: bool
|
||||
source_id: str
|
||||
level: str
|
||||
fingerprint: str | None = None
|
||||
|
||||
|
||||
class ServiceLogEventCreate(BaseModel):
|
||||
source: str = "ai-provider"
|
||||
service: str = "ai-provider"
|
||||
module: str | None = None
|
||||
category: str | None = None
|
||||
event: str = "service.runtime_log"
|
||||
level: str = "error"
|
||||
message: str
|
||||
fingerprint: str | None = None
|
||||
occurrence_count: int = 1
|
||||
request_id: str | None = None
|
||||
trace_id: str | None = None
|
||||
task_id: str | None = None
|
||||
source_id: int | str | None = None
|
||||
provider: str | None = None
|
||||
context: dict[str, object] | None = None
|
||||
|
||||
|
||||
async def ingest_client_log_event(
|
||||
source_id: str,
|
||||
*,
|
||||
service: str,
|
||||
event: str,
|
||||
default_module: str,
|
||||
default_category: str,
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
) -> EarthClientLogEventResponse:
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
append_buffer_log(
|
||||
source_id,
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
context={
|
||||
"category": payload.category or "",
|
||||
"url": payload.url or "",
|
||||
"module": payload.module or "",
|
||||
"detail": payload.detail or "",
|
||||
"fingerprint": payload.fingerprint or "",
|
||||
"occurrence_count": max(1, int(payload.occurrence_count or 1)),
|
||||
"metadata": payload.metadata or {},
|
||||
},
|
||||
)
|
||||
await record_system_log(
|
||||
source=source_id,
|
||||
service=service,
|
||||
module=payload.module or default_module,
|
||||
event=event,
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or default_category,
|
||||
context={
|
||||
"url": payload.url or "",
|
||||
"detail": payload.detail or "",
|
||||
"module": payload.module or "",
|
||||
"client_ip": request.client.host if request.client else "",
|
||||
"metadata": payload.metadata or {},
|
||||
},
|
||||
fingerprint=payload.fingerprint,
|
||||
occurrence_count=max(1, int(payload.occurrence_count or 1)),
|
||||
)
|
||||
return EarthClientLogEventResponse(accepted=True, source_id=source_id, level=normalized_level, fingerprint=payload.fingerprint)
|
||||
|
||||
|
||||
def require_observability_ingest_token(
|
||||
authorization: str | None,
|
||||
ingest_token: str | None,
|
||||
) -> None:
|
||||
expected_token = settings.OBSERVABILITY_INGEST_TOKEN.strip()
|
||||
if not expected_token:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="Observability service ingestion is not configured",
|
||||
)
|
||||
provided = ""
|
||||
if ingest_token:
|
||||
provided = ingest_token.strip()
|
||||
elif authorization:
|
||||
scheme, _, token = authorization.partition(" ")
|
||||
if scheme.lower() == "bearer":
|
||||
provided = token.strip()
|
||||
if not provided or not secrets.compare_digest(provided, expected_token):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Invalid observability ingestion token",
|
||||
)
|
||||
|
||||
|
||||
class EarthLayerCacheStatusResponse(BaseModel):
|
||||
@@ -376,98 +434,116 @@ async def get_system_log_sources(
|
||||
}
|
||||
|
||||
|
||||
async def read_database_log_snapshot(
|
||||
source_id: str,
|
||||
*,
|
||||
limit: int,
|
||||
level: str,
|
||||
levels: str | None,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
search: str | None,
|
||||
db: AsyncSession,
|
||||
) -> dict | None:
|
||||
selected_levels = set(normalize_log_level(item) for item in (levels or level).split(",") if item.strip())
|
||||
selected_levels.discard("all")
|
||||
search_query = (search or "").strip().lower()
|
||||
lines: list[str] = []
|
||||
@router.get("/logs/observability/groups")
|
||||
async def get_observability_log_groups(
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
level: str = "all",
|
||||
levels: str | None = Query(None, description="Comma-separated log levels"),
|
||||
start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
|
||||
normalized_start_date = validate_log_date(start_date, "start_date")
|
||||
normalized_end_date = validate_log_date(end_date, "end_date")
|
||||
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
|
||||
return await read_observability_groups(
|
||||
limit=limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
db=db,
|
||||
)
|
||||
|
||||
if source_id == "system-db":
|
||||
query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(limit * 5)
|
||||
result = await db.execute(query)
|
||||
records = result.scalars().all()
|
||||
for record in records:
|
||||
record_level = normalize_log_level(record.level)
|
||||
if selected_levels and record_level not in selected_levels:
|
||||
continue
|
||||
occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
|
||||
if start_date and occurred_at and occurred_at < start_date:
|
||||
continue
|
||||
if end_date and occurred_at and occurred_at > end_date:
|
||||
continue
|
||||
line = " ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
record_level.upper(),
|
||||
record.source,
|
||||
record.category or "",
|
||||
record.event or "",
|
||||
f"request_id={record.request_id}" if record.request_id else "",
|
||||
record.message,
|
||||
_compact_log_context(record.context),
|
||||
]
|
||||
if part
|
||||
)
|
||||
if search_query and search_query not in line.lower():
|
||||
continue
|
||||
lines.append(line)
|
||||
elif source_id == "audit-db":
|
||||
query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(limit * 5)
|
||||
result = await db.execute(query)
|
||||
records = result.scalars().all()
|
||||
for record in records:
|
||||
occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
|
||||
if start_date and occurred_at and occurred_at < start_date:
|
||||
continue
|
||||
if end_date and occurred_at and occurred_at > end_date:
|
||||
continue
|
||||
line = " ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
"INFO",
|
||||
record.action,
|
||||
record.target_type or "",
|
||||
record.target_id or "",
|
||||
record.result or "",
|
||||
]
|
||||
if part
|
||||
)
|
||||
if search_query and search_query not in line.lower():
|
||||
continue
|
||||
lines.append(line)
|
||||
else:
|
||||
return None
|
||||
|
||||
lines = list(reversed(lines[:limit]))
|
||||
return {
|
||||
"source_id": source_id,
|
||||
"name": "系统事件" if source_id == "system-db" else "审计事件",
|
||||
"kind": "database",
|
||||
"location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
|
||||
"description": "数据库持久化日志",
|
||||
"category": "database" if source_id == "system-db" else "audit",
|
||||
"status": "ok" if lines else "empty",
|
||||
"level": level,
|
||||
"selected_levels": sorted(selected_levels),
|
||||
"search_query": search or "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": limit,
|
||||
"line_count": len(lines),
|
||||
"lines": lines,
|
||||
}
|
||||
@router.get("/logs/observability/groups/{fingerprint}/events")
|
||||
async def get_observability_group_events(
|
||||
fingerprint: str,
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
|
||||
payload = await read_observability_group_events(fingerprint, limit=limit, db=db)
|
||||
if payload is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Observability group not found")
|
||||
return payload
|
||||
|
||||
|
||||
@router.get("/logs/observability/raw")
|
||||
async def get_observability_raw_events(
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
level: str = "all",
|
||||
levels: str | None = Query(None, description="Comma-separated log levels"),
|
||||
start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
|
||||
normalized_start_date = validate_log_date(start_date, "start_date")
|
||||
normalized_end_date = validate_log_date(end_date, "end_date")
|
||||
return await read_observability_raw_events(
|
||||
limit=limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
db=db,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/logs/service", response_model=EarthClientLogEventResponse)
|
||||
async def ingest_service_log(
|
||||
payload: ServiceLogEventCreate,
|
||||
authorization: str | None = Header(default=None),
|
||||
ingest_token: str | None = Header(default=None, alias="X-Planet-Observability-Token"),
|
||||
):
|
||||
require_observability_ingest_token(authorization, ingest_token)
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
source = (payload.source or "ai-provider").strip() or "ai-provider"
|
||||
context = dict(payload.context or {})
|
||||
if payload.request_id:
|
||||
context["request_id"] = payload.request_id
|
||||
if payload.trace_id:
|
||||
context["trace_id"] = payload.trace_id
|
||||
if payload.task_id:
|
||||
context["task_id"] = payload.task_id
|
||||
if payload.source_id is not None:
|
||||
context["source_id"] = payload.source_id
|
||||
if payload.provider:
|
||||
context["provider"] = payload.provider
|
||||
await record_system_log(
|
||||
source=source,
|
||||
service=(payload.service or source).strip() or source,
|
||||
module=payload.module or source,
|
||||
event=(payload.event or "service.runtime_log").strip() or "service.runtime_log",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or "service-runtime",
|
||||
context=context,
|
||||
fingerprint=payload.fingerprint,
|
||||
occurrence_count=max(1, int(payload.occurrence_count or 1)),
|
||||
)
|
||||
return EarthClientLogEventResponse(
|
||||
accepted=True,
|
||||
source_id=source,
|
||||
level=normalized_level,
|
||||
fingerprint=payload.fingerprint,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
|
||||
@@ -533,31 +609,28 @@ async def ingest_earth_client_log(
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
):
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
append_buffer_log(
|
||||
return await ingest_client_log_event(
|
||||
"earth-client",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
context={
|
||||
"category": payload.category or "",
|
||||
"url": payload.url or "",
|
||||
"module": payload.module or "",
|
||||
"detail": payload.detail or "",
|
||||
},
|
||||
)
|
||||
await record_system_log(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module=payload.module or "earth-client",
|
||||
event="earth.client.runtime_log",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or "client-runtime",
|
||||
context={
|
||||
"url": payload.url or "",
|
||||
"detail": payload.detail or "",
|
||||
"module": payload.module or "",
|
||||
"client_ip": request.client.host if request.client else "",
|
||||
},
|
||||
default_module="earth-client",
|
||||
default_category="client-runtime",
|
||||
payload=payload,
|
||||
request=request,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/logs/admin-client", response_model=EarthClientLogEventResponse)
|
||||
async def ingest_admin_client_log(
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
):
|
||||
return await ingest_client_log_event(
|
||||
"admin-client",
|
||||
service="admin",
|
||||
event="admin.client.runtime_log",
|
||||
default_module="admin-client",
|
||||
default_category="client-runtime",
|
||||
payload=payload,
|
||||
request=request,
|
||||
)
|
||||
return {"accepted": True, "source_id": "earth-client", "level": normalized_level}
|
||||
|
||||
@@ -29,7 +29,8 @@ async def list_tasks(
|
||||
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
|
||||
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
|
||||
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
|
||||
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
|
||||
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress,
|
||||
ct.task_type, ct.source, ds.source as datasource_source
|
||||
FROM collection_tasks ct
|
||||
JOIN data_sources ds ON ct.datasource_id = ds.id
|
||||
WHERE 1=1
|
||||
@@ -39,12 +40,19 @@ async def list_tasks(
|
||||
|
||||
if datasource_id:
|
||||
query += " AND ct.datasource_id = :datasource_id"
|
||||
count_query += " WHERE ct.datasource_id = :datasource_id"
|
||||
count_query += " AND ct.datasource_id = :datasource_id"
|
||||
params["datasource_id"] = datasource_id
|
||||
if status:
|
||||
query += " AND ct.status = :status"
|
||||
count_query += " AND ct.status = :status"
|
||||
params["status"] = status
|
||||
statuses = [item.strip() for item in status.split(",") if item.strip()]
|
||||
if len(statuses) > 1:
|
||||
placeholders = ", ".join(f":status_{index}" for index, _item in enumerate(statuses))
|
||||
query += f" AND ct.status IN ({placeholders})"
|
||||
count_query += f" AND ct.status IN ({placeholders})"
|
||||
params.update({f"status_{index}": item for index, item in enumerate(statuses)})
|
||||
else:
|
||||
query += " AND ct.status = :status"
|
||||
count_query += " AND ct.status = :status"
|
||||
params["status"] = statuses[0] if statuses else status
|
||||
|
||||
query += f" ORDER BY ct.created_at DESC LIMIT {page_size} OFFSET {offset}"
|
||||
|
||||
@@ -76,6 +84,9 @@ async def list_tasks(
|
||||
"phase_unit": t[13],
|
||||
"total_records": t[14],
|
||||
"progress": t[15],
|
||||
"task_type": t[16],
|
||||
"source": t[17] or t[18],
|
||||
"datasource_source": t[18],
|
||||
}
|
||||
for t in tasks
|
||||
],
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import re
|
||||
from urllib.parse import quote, urljoin
|
||||
|
||||
import httpx
|
||||
@@ -10,6 +11,26 @@ from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_u
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_HLS_URI_ATTRIBUTE_RE = re.compile(r'URI="([^"]+)"')
|
||||
|
||||
|
||||
def _proxied_tv_url(url: str) -> str:
|
||||
return f"/api/v1/tv/proxy?url={quote(url, safe='')}"
|
||||
|
||||
|
||||
def _rewrite_hls_uri_attributes(line: str, *, base_url: str) -> str:
|
||||
def replace(match: re.Match[str]) -> str:
|
||||
uri = match.group(1)
|
||||
absolute_url = urljoin(base_url, uri)
|
||||
return f'URI="{_proxied_tv_url(absolute_url)}"'
|
||||
|
||||
return _HLS_URI_ATTRIBUTE_RE.sub(replace, line)
|
||||
|
||||
|
||||
def _should_strip_hls_metadata_line(line: str) -> bool:
|
||||
normalized = line.strip().upper()
|
||||
return normalized.startswith("#EXT-X-MEDIA:") and "TYPE=SUBTITLES" in normalized
|
||||
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
@@ -56,11 +77,16 @@ async def proxy_tv_stream(
|
||||
rewritten_lines: list[str] = []
|
||||
for line in manifest_text.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
if not stripped:
|
||||
rewritten_lines.append(line)
|
||||
continue
|
||||
if stripped.startswith("#"):
|
||||
if _should_strip_hls_metadata_line(line):
|
||||
continue
|
||||
rewritten_lines.append(_rewrite_hls_uri_attributes(line, base_url=response_url))
|
||||
continue
|
||||
absolute_url = urljoin(response_url, stripped)
|
||||
rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
|
||||
rewritten_lines.append(_proxied_tv_url(absolute_url))
|
||||
return Response(
|
||||
content="\n".join(rewritten_lines),
|
||||
media_type="application/vnd.apple.mpegurl",
|
||||
|
||||
@@ -6,11 +6,14 @@ from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
|
||||
from jose import jwt, JWTError
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.log_tail import LOG_TAIL_CHANNEL, log_tail_manager
|
||||
|
||||
logger = get_logger(__name__, service="api")
|
||||
router = APIRouter()
|
||||
@@ -37,6 +40,28 @@ async def authenticate_token(token: str) -> Optional[dict]:
|
||||
return None
|
||||
|
||||
|
||||
async def load_websocket_user_role(user_id: str | None) -> str | None:
|
||||
if not user_id:
|
||||
return None
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(
|
||||
text("SELECT role, is_active FROM users WHERE id = :id"),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"WebSocket user role lookup failed",
|
||||
event="auth.websocket.role_lookup_failed",
|
||||
context={"user_id": user_id, "error": str(exc)},
|
||||
)
|
||||
return None
|
||||
if row is None or not row[1]:
|
||||
return None
|
||||
return str(row[0] or "")
|
||||
|
||||
|
||||
@router.websocket("/ws")
|
||||
async def websocket_endpoint(
|
||||
websocket: WebSocket,
|
||||
@@ -59,6 +84,7 @@ async def websocket_endpoint(
|
||||
|
||||
is_anonymous = payload is None
|
||||
user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
|
||||
user_role = await load_websocket_user_role(user_id) if payload else None
|
||||
supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [
|
||||
"gpu_clusters",
|
||||
"submarine_cables",
|
||||
@@ -70,6 +96,8 @@ async def websocket_endpoint(
|
||||
"earth_news",
|
||||
EARTH_UPDATES_CHANNEL,
|
||||
]
|
||||
if user_role == "super_admin":
|
||||
supported_channels = [*supported_channels, LOG_TAIL_CHANNEL]
|
||||
await manager.connect(websocket, user_id)
|
||||
|
||||
try:
|
||||
@@ -100,6 +128,7 @@ async def websocket_endpoint(
|
||||
payload_data = data.get("data", {})
|
||||
if not isinstance(payload_data, dict):
|
||||
payload_data = {}
|
||||
log_tail_config = None
|
||||
channels = payload_data.get("channels", [])
|
||||
if isinstance(channels, str):
|
||||
channels = [channels]
|
||||
@@ -108,6 +137,26 @@ async def websocket_endpoint(
|
||||
channel = payload_data.get("channel")
|
||||
if channel and channel not in channels:
|
||||
channels = [*channels, channel]
|
||||
if LOG_TAIL_CHANNEL in channels:
|
||||
if user_role != "super_admin":
|
||||
await websocket.send_json(
|
||||
{
|
||||
"type": "subscription_error",
|
||||
"data": {"channel": LOG_TAIL_CHANNEL, "detail": "Only super_admin can subscribe logs"},
|
||||
}
|
||||
)
|
||||
channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
|
||||
else:
|
||||
try:
|
||||
log_tail_config = await log_tail_manager.subscribe(websocket, payload_data)
|
||||
except ValueError as exc:
|
||||
await websocket.send_json(
|
||||
{
|
||||
"type": "subscription_error",
|
||||
"data": {"channel": LOG_TAIL_CHANNEL, "detail": str(exc)},
|
||||
}
|
||||
)
|
||||
channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
|
||||
if is_anonymous:
|
||||
channels = [channel for channel in channels if channel in supported_channels]
|
||||
vessel_subscription = None
|
||||
@@ -131,14 +180,20 @@ async def websocket_endpoint(
|
||||
"action": "subscribe",
|
||||
"channels": [
|
||||
*channels,
|
||||
*([LOG_TAIL_CHANNEL] if log_tail_config else []),
|
||||
*(["vessels"] if vessel_subscription else []),
|
||||
],
|
||||
"vessels": vessel_subscription,
|
||||
"logs_tail": log_tail_config.__dict__ if log_tail_config else None,
|
||||
},
|
||||
}
|
||||
)
|
||||
elif data.get("type") == "unsubscribe":
|
||||
channels = data.get("data", {}).get("channels", [])
|
||||
if isinstance(channels, str):
|
||||
channels = [channels]
|
||||
if LOG_TAIL_CHANNEL in channels:
|
||||
await log_tail_manager.unsubscribe(websocket)
|
||||
manager.unsubscribe(websocket, channels)
|
||||
await websocket.send_json(
|
||||
{
|
||||
@@ -159,4 +214,5 @@ async def websocket_endpoint(
|
||||
except WebSocketDisconnect:
|
||||
pass
|
||||
finally:
|
||||
await log_tail_manager.disconnect(websocket)
|
||||
manager.disconnect(websocket, user_id)
|
||||
|
||||
@@ -41,6 +41,7 @@ class Settings(BaseSettings):
|
||||
AI_PROVIDER_SERVICE_TOKEN: str = ""
|
||||
AI_PROVIDER_TIMEOUT_SECONDS: int = 60
|
||||
AI_PROVIDER_RETRY_ATTEMPTS: int = 2
|
||||
OBSERVABILITY_INGEST_TOKEN: str = ""
|
||||
|
||||
@property
|
||||
def REDIS_URL(self) -> str:
|
||||
|
||||
@@ -13,7 +13,7 @@ from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.system_log import SystemLog, AuditLog
|
||||
from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
|
||||
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
@@ -37,6 +37,8 @@ __all__ = [
|
||||
"ComputeCenterLocationRecord",
|
||||
"SystemLog",
|
||||
"AuditLog",
|
||||
"ObservabilityEvent",
|
||||
"ObservabilityEventGroup",
|
||||
"PlaygroundSession",
|
||||
"PlaygroundMessage",
|
||||
"VesselPosition",
|
||||
|
||||
@@ -38,3 +38,46 @@ class AuditLog(Base):
|
||||
ip = Column(String(64), nullable=True)
|
||||
details = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class ObservabilityEvent(Base):
|
||||
__tablename__ = "observability_events"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
source = Column(String(50), nullable=False, index=True)
|
||||
service = Column(String(50), nullable=True, index=True)
|
||||
module = Column(String(120), nullable=True, index=True)
|
||||
category = Column(String(80), nullable=True, index=True)
|
||||
event = Column(String(160), nullable=True, index=True)
|
||||
level = Column(String(20), nullable=False, index=True)
|
||||
message = Column(Text, nullable=False)
|
||||
fingerprint = Column(String(80), nullable=False, index=True)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
trace_id = Column(String(64), nullable=True, index=True)
|
||||
task_id = Column(String(120), nullable=True, index=True)
|
||||
source_ref_id = Column(String(120), nullable=True, index=True)
|
||||
provider = Column(String(120), nullable=True, index=True)
|
||||
user_id = Column(Integer, nullable=True, index=True)
|
||||
context = Column(JSON, nullable=False, default=dict)
|
||||
occurrence_count = Column(Integer, nullable=False, default=1)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class ObservabilityEventGroup(Base):
|
||||
__tablename__ = "observability_event_groups"
|
||||
|
||||
fingerprint = Column(String(80), primary_key=True)
|
||||
source = Column(String(50), nullable=False, index=True)
|
||||
service = Column(String(50), nullable=True, index=True)
|
||||
module = Column(String(120), nullable=True, index=True)
|
||||
category = Column(String(80), nullable=True, index=True)
|
||||
event = Column(String(160), nullable=True, index=True)
|
||||
last_level = Column(String(20), nullable=False, index=True)
|
||||
sample_message = Column(Text, nullable=False)
|
||||
sample_detail = Column(Text, nullable=True)
|
||||
affected_sources = Column(JSON, nullable=False, default=list)
|
||||
count = Column(Integer, nullable=False, default=0)
|
||||
first_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
|
||||
last_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
|
||||
@@ -322,6 +322,21 @@ class AISStreamCollector(BaseCollector):
|
||||
last_success_at=now if data else None,
|
||||
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
|
||||
)
|
||||
if snapshot_id is not None:
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.record_count = records_added
|
||||
snapshot.status = "success"
|
||||
snapshot.completed_at = now
|
||||
snapshot.summary = {
|
||||
"created": records_added,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"deleted": 0,
|
||||
"storage": "ais_raw_observations",
|
||||
}
|
||||
await db.commit()
|
||||
await self.update_progress(records_added, force=True)
|
||||
return records_added
|
||||
|
||||
@@ -25,11 +25,17 @@ FALLBACK_GROUPS = (
|
||||
"starlink",
|
||||
"gps-ops",
|
||||
"galileo",
|
||||
"glonass",
|
||||
"glo-ops",
|
||||
"beidou",
|
||||
"leo",
|
||||
"geo",
|
||||
"iridium-next",
|
||||
"stations",
|
||||
"visual",
|
||||
"weather",
|
||||
"science",
|
||||
"cubesat",
|
||||
"amateur",
|
||||
"last-30-days",
|
||||
)
|
||||
FETCH_RETRY_ATTEMPTS = 3
|
||||
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
|
||||
@@ -220,27 +226,28 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
try:
|
||||
for group in FALLBACK_GROUPS:
|
||||
group_url = self._group_url(group)
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is None:
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is not None:
|
||||
body_path = cached_path
|
||||
else:
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
|
||||
) from exc
|
||||
body_path = cached_path
|
||||
|
||||
group_records = await self._load_downloaded_payload(
|
||||
body_path,
|
||||
|
||||
@@ -119,6 +119,21 @@ class VesselAISCollector(BaseCollector):
|
||||
last_success_at=now if data else None,
|
||||
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
|
||||
)
|
||||
if snapshot_id is not None:
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.record_count = records_added
|
||||
snapshot.status = "success"
|
||||
snapshot.completed_at = now
|
||||
snapshot.summary = {
|
||||
"created": records_added,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"deleted": 0,
|
||||
"storage": "ais_raw_observations",
|
||||
}
|
||||
await db.commit()
|
||||
await self._broadcast_vessel_snapshot(data)
|
||||
await self.update_progress(records_added, force=True)
|
||||
|
||||
@@ -54,6 +54,9 @@ DATA_WRITE_JOB_TYPES = (JOB_TYPE_COLLECT, JOB_TYPE_CLEAR_DATA, JOB_TYPE_CLEAR_CA
|
||||
SOURCE_LOCK_JOB_STATUSES = (JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
|
||||
QUEUE_POLL_SECONDS = 0.35
|
||||
JOB_STALE_LOCK_MINUTES = 90
|
||||
ORPHAN_CANCELLING_GRACE_SECONDS = 30
|
||||
JOB_RECOVERY_SWEEP_SECONDS = 15
|
||||
DATA_DELETE_BATCH_SIZE = 50_000
|
||||
DEFAULT_WORKER_CONCURRENCY = 2
|
||||
|
||||
RUNNING_DATA_JOB_TASKS: dict[int, asyncio.Task[Any]] = {}
|
||||
@@ -281,6 +284,7 @@ class DataJobWorker:
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
self._stop_event: asyncio.Event | None = None
|
||||
self._running: set[asyncio.Task[Any]] = set()
|
||||
self._last_recovery_sweep_at: datetime | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if self._task and not self._task.done():
|
||||
@@ -303,6 +307,11 @@ class DataJobWorker:
|
||||
await self._recover_stale_running_jobs()
|
||||
while not self._stop_event.is_set():
|
||||
self._running = {task for task in self._running if not task.done()}
|
||||
if (
|
||||
self._last_recovery_sweep_at is None
|
||||
or (_utcnow() - self._last_recovery_sweep_at).total_seconds() >= JOB_RECOVERY_SWEEP_SECONDS
|
||||
):
|
||||
await self._recover_stale_running_jobs()
|
||||
if len(self._running) >= self.concurrency:
|
||||
await asyncio.sleep(QUEUE_POLL_SECONDS)
|
||||
continue
|
||||
@@ -316,7 +325,9 @@ class DataJobWorker:
|
||||
self._running.add(runner)
|
||||
|
||||
async def _recover_stale_running_jobs(self) -> None:
|
||||
self._last_recovery_sweep_at = _utcnow()
|
||||
cutoff = _utcnow() - timedelta(minutes=JOB_STALE_LOCK_MINUTES)
|
||||
orphan_cancelling_cutoff = _utcnow() - timedelta(seconds=ORPHAN_CANCELLING_GRACE_SECONDS)
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
@@ -332,6 +343,21 @@ class DataJobWorker:
|
||||
job.error_message = "Marked failed after stale data job lock timeout"
|
||||
if stale_jobs:
|
||||
await db.commit()
|
||||
orphan_result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.status == JOB_STATUS_CANCELLING)
|
||||
.where(CollectionTask.locked_at.is_(None))
|
||||
.where(CollectionTask.requested_cancel_at.is_not(None))
|
||||
.where(CollectionTask.requested_cancel_at < orphan_cancelling_cutoff)
|
||||
)
|
||||
for job in orphan_result.scalars().all():
|
||||
if job.id in RUNNING_DATA_JOB_TASKS:
|
||||
continue
|
||||
await _cancel_task_without_runner(
|
||||
db,
|
||||
job,
|
||||
reason=job.cancel_reason or "cancelled_after_orphaned_runner",
|
||||
)
|
||||
|
||||
async def _claim_next_job(self) -> int | None:
|
||||
async with async_session_factory() as db:
|
||||
@@ -477,15 +503,22 @@ async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
|
||||
count_result = await db.execute(
|
||||
select(CollectedData.id).where(CollectedData.source == source)
|
||||
deleted_count = await _delete_table_rows_by_source(
|
||||
db,
|
||||
task,
|
||||
table_name="collected_data",
|
||||
source_column="source",
|
||||
source=source,
|
||||
)
|
||||
derived_deleted_counts = await _clear_derived_datasource_data_in_batches(
|
||||
db,
|
||||
task,
|
||||
source,
|
||||
progress_offset=deleted_count,
|
||||
)
|
||||
collected_ids = [row[0] for row in count_result.all()]
|
||||
derived_deleted_counts = await clear_derived_datasource_data(db, source)
|
||||
if collected_ids:
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.id.in_(collected_ids)))
|
||||
deleted_count = len(collected_ids)
|
||||
derived_deleted_count = sum(derived_deleted_counts.values())
|
||||
if any(key.startswith("ais_") for key in derived_deleted_counts):
|
||||
await db.execute(text("ANALYZE ais_raw_observations"))
|
||||
|
||||
task.records_processed = deleted_count + derived_deleted_count
|
||||
task.total_records = task.records_processed
|
||||
@@ -504,10 +537,99 @@ async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
task.phase = "completed"
|
||||
task.phase_message = "数据库数据已清理"
|
||||
task.completed_at = _utcnow()
|
||||
datasource = await db.get(DataSource, task.datasource_id)
|
||||
if datasource is not None:
|
||||
datasource.last_status = JOB_STATUS_SUCCESS
|
||||
datasource.last_run_at = task.completed_at
|
||||
await db.execute(
|
||||
DataSnapshot.__table__.update()
|
||||
.where(DataSnapshot.source == source)
|
||||
.values(is_current=False)
|
||||
)
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
|
||||
|
||||
async def _delete_table_rows_by_source(
|
||||
db: AsyncSession,
|
||||
task: CollectionTask,
|
||||
*,
|
||||
table_name: str,
|
||||
source_column: str,
|
||||
source: str,
|
||||
progress_offset: int = 0,
|
||||
) -> int:
|
||||
deleted = 0
|
||||
while True:
|
||||
result = await db.execute(
|
||||
text(
|
||||
f"""
|
||||
WITH doomed AS (
|
||||
SELECT ctid
|
||||
FROM {table_name}
|
||||
WHERE {source_column} = :source
|
||||
LIMIT :batch_size
|
||||
),
|
||||
deleted_rows AS (
|
||||
DELETE FROM {table_name}
|
||||
USING doomed
|
||||
WHERE {table_name}.ctid = doomed.ctid
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT COUNT(*) FROM deleted_rows
|
||||
"""
|
||||
),
|
||||
{"source": source, "batch_size": DATA_DELETE_BATCH_SIZE},
|
||||
)
|
||||
batch_deleted = max(int(result.scalar_one() or 0), 0)
|
||||
if batch_deleted <= 0:
|
||||
break
|
||||
deleted += batch_deleted
|
||||
task.records_processed = progress_offset + deleted
|
||||
task.phase_current = task.records_processed
|
||||
task.phase_unit = "records"
|
||||
task.phase_message = f"正在删除数据:{task.records_processed} 条"
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
return deleted
|
||||
|
||||
|
||||
async def _clear_derived_datasource_data_in_batches(
|
||||
db: AsyncSession,
|
||||
task: CollectionTask,
|
||||
source: str,
|
||||
progress_offset: int = 0,
|
||||
) -> dict[str, int]:
|
||||
deleted_counts: dict[str, int] = {}
|
||||
if source in {"barentswatch_vessels", "aisstream_vessels"}:
|
||||
deleted_counts["ais_conflict_records"] = await _delete_table_rows_by_source(
|
||||
db,
|
||||
task,
|
||||
table_name="ais_conflict_records",
|
||||
source_column="selected_source",
|
||||
source=source,
|
||||
progress_offset=progress_offset + sum(deleted_counts.values()),
|
||||
)
|
||||
deleted_counts["ais_source_health"] = await _delete_table_rows_by_source(
|
||||
db,
|
||||
task,
|
||||
table_name="ais_source_health",
|
||||
source_column="source",
|
||||
source=source,
|
||||
progress_offset=progress_offset + sum(deleted_counts.values()),
|
||||
)
|
||||
deleted_counts["ais_raw_observations"] = await _delete_table_rows_by_source(
|
||||
db,
|
||||
task,
|
||||
table_name="ais_raw_observations",
|
||||
source_column="source",
|
||||
source=source,
|
||||
progress_offset=progress_offset + sum(deleted_counts.values()),
|
||||
)
|
||||
return deleted_counts
|
||||
return await clear_derived_datasource_data(db, source)
|
||||
|
||||
|
||||
async def _run_clear_cache_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
source = str(task.source or (task.payload or {}).get("source") or "").strip()
|
||||
if not source:
|
||||
|
||||
@@ -62,11 +62,11 @@ def interactables_to_geojson(items: list[EarthInteractable]) -> dict[str, Any]:
|
||||
def invalidate_interactable_cache(layer: str | None = None) -> int:
|
||||
layer_key = str(layer or "*").strip() or "*"
|
||||
deleted = earth_layer_cache.delete_pattern(
|
||||
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:{layer_key}*"
|
||||
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:interactable_layer:{layer_key}*"
|
||||
)
|
||||
if layer_key != "all":
|
||||
deleted += earth_layer_cache.delete_pattern(
|
||||
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:all*"
|
||||
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:interactable_layer:all*"
|
||||
)
|
||||
return deleted
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
tables=frozenset({"vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
@@ -163,17 +164,24 @@ async def clear_derived_datasource_data(db: AsyncSession, source: str) -> dict[s
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
|
||||
|
||||
model_by_key: dict[str, Any] = {
|
||||
"bgp_observations": BGPObservation,
|
||||
"bgp_anomalies": BGPAnomaly,
|
||||
"bgp_incidents": BGPIncident,
|
||||
"ais_raw_observations": AISRawObservation,
|
||||
"ais_conflict_records": AISConflictRecord,
|
||||
"ais_source_health": AISSourceHealth,
|
||||
}
|
||||
deleted_counts: dict[str, int] = {}
|
||||
for key in adapter.derived_models:
|
||||
model = model_by_key.get(key)
|
||||
if model is None:
|
||||
continue
|
||||
result = await db.execute(model.__table__.delete().where(model.source == source))
|
||||
if key == "ais_conflict_records":
|
||||
result = await db.execute(model.__table__.delete().where(model.selected_source == source))
|
||||
else:
|
||||
result = await db.execute(model.__table__.delete().where(model.source == source))
|
||||
deleted_counts[key] = int(result.rowcount or 0)
|
||||
return deleted_counts
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -11,6 +11,7 @@ from app.services.earth_news import (
|
||||
ParsedNewsItem,
|
||||
apply_enrichment_patch_to_item,
|
||||
build_anchor_location_patch,
|
||||
_news_meta_patch,
|
||||
)
|
||||
|
||||
|
||||
@@ -34,6 +35,8 @@ def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
|
||||
|
||||
|
||||
def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
|
||||
location_meta = dict(record.location_meta or {})
|
||||
news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {}
|
||||
item = ParsedNewsItem(
|
||||
id=record.id,
|
||||
title=record.title,
|
||||
@@ -49,11 +52,29 @@ def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
|
||||
enrichment_status=record.enrichment_status or "pending",
|
||||
enrichment_error=record.enrichment_error,
|
||||
enriched_at=_coerce_datetime(record.enriched_at),
|
||||
source_tags=list(news_meta.get("source_tags") or []),
|
||||
feed_id=str(news_meta.get("feed_id") or ""),
|
||||
feed_type=str(news_meta.get("feed_type") or "rss"),
|
||||
feed_default_category=str(news_meta.get("feed_default_category") or "other"),
|
||||
category=str(news_meta.get("category") or "other"),
|
||||
item_tags=list(news_meta.get("item_tags") or []),
|
||||
tagging_source=str(news_meta.get("tagging_source") or "rules"),
|
||||
tagging_confidence=float(news_meta.get("tagging_confidence") or 0),
|
||||
importance_score=int(news_meta.get("importance_score") or 0),
|
||||
importance_level=str(news_meta.get("importance_level") or "low"),
|
||||
importance_reasons=list(news_meta.get("importance_reasons") or []),
|
||||
market_impact=str(news_meta.get("market_impact") or "none"),
|
||||
)
|
||||
return apply_enrichment_patch_to_item(item, _location_patch_from_record(record))
|
||||
|
||||
|
||||
def _query_sort_key(active_region: str):
|
||||
if active_region == "global":
|
||||
return (
|
||||
EarthNewsItem.published_at.is_(None),
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.feed_name.asc(),
|
||||
)
|
||||
return (
|
||||
EarthNewsItem.region != active_region,
|
||||
EarthNewsItem.published_at.is_(None),
|
||||
@@ -62,28 +83,93 @@ def _query_sort_key(active_region: str):
|
||||
)
|
||||
|
||||
|
||||
def _category_filter_clause(categories: set[str] | None):
|
||||
if not categories:
|
||||
return None
|
||||
return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category").in_(sorted(categories))
|
||||
|
||||
|
||||
def _source_filter_clause(source_ids: set[str] | None):
|
||||
if not source_ids:
|
||||
return None
|
||||
return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id").in_(sorted(source_ids))
|
||||
|
||||
|
||||
def _record_source_id(record: EarthNewsItem) -> str:
|
||||
location_meta = dict(record.location_meta or {})
|
||||
news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {}
|
||||
source_id = str(news_meta.get("source_id") or "").strip()
|
||||
if source_id:
|
||||
return source_id
|
||||
if isinstance(record.id, str) and ":" in record.id:
|
||||
return record.id.split(":", 1)[0]
|
||||
return record.feed_name or record.source or record.id
|
||||
|
||||
|
||||
def _diversify_records_by_source(records: list[EarthNewsItem], *, limit: int) -> list[EarthNewsItem]:
|
||||
if limit <= 0 or len(records) <= limit:
|
||||
return records[:limit]
|
||||
buckets: dict[str, list[EarthNewsItem]] = {}
|
||||
order: list[str] = []
|
||||
for record in records:
|
||||
source_id = _record_source_id(record)
|
||||
if source_id not in buckets:
|
||||
buckets[source_id] = []
|
||||
order.append(source_id)
|
||||
buckets[source_id].append(record)
|
||||
|
||||
diversified: list[EarthNewsItem] = []
|
||||
while len(diversified) < limit and order:
|
||||
next_order: list[str] = []
|
||||
for source_id in order:
|
||||
bucket = buckets.get(source_id) or []
|
||||
if bucket and len(diversified) < limit:
|
||||
diversified.append(bucket.pop(0))
|
||||
if bucket:
|
||||
next_order.append(source_id)
|
||||
order = next_order
|
||||
return diversified
|
||||
|
||||
|
||||
async def list_earth_news_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
active_region: str,
|
||||
limit: int,
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
regions = {"global", active_region}
|
||||
result = await db.execute(
|
||||
query_limit = limit if source_ids else min(max(limit * 4, limit), 100)
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.where(EarthNewsItem.region.in_(regions))
|
||||
.order_by(*_query_sort_key(active_region))
|
||||
.limit(limit)
|
||||
.limit(query_limit)
|
||||
)
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
if active_region != "global":
|
||||
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
|
||||
category_clause = _category_filter_clause(categories)
|
||||
if category_clause is not None:
|
||||
query = query.where(category_clause)
|
||||
source_clause = _source_filter_clause(source_ids)
|
||||
if source_clause is not None:
|
||||
query = query.where(source_clause)
|
||||
result = await db.execute(query)
|
||||
records = list(result.scalars().all())
|
||||
if not source_ids:
|
||||
records = _diversify_records_by_source(records, limit=limit)
|
||||
else:
|
||||
records = records[:limit]
|
||||
return [record_to_parsed_news_item(record) for record in records]
|
||||
|
||||
|
||||
async def list_earth_news_cruise_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
limit: int,
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
result = await db.execute(
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.order_by(
|
||||
EarthNewsItem.region.asc(),
|
||||
@@ -93,6 +179,13 @@ async def list_earth_news_cruise_items(
|
||||
)
|
||||
.limit(limit)
|
||||
)
|
||||
category_clause = _category_filter_clause(categories)
|
||||
if category_clause is not None:
|
||||
query = query.where(category_clause)
|
||||
source_clause = _source_filter_clause(source_ids)
|
||||
if source_clause is not None:
|
||||
query = query.where(source_clause)
|
||||
result = await db.execute(query)
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
|
||||
|
||||
@@ -101,13 +194,13 @@ async def get_earth_news_freshness(
|
||||
*,
|
||||
active_region: str,
|
||||
) -> tuple[int, datetime | None]:
|
||||
regions = {"global", active_region}
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.count(EarthNewsItem.id),
|
||||
func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)),
|
||||
).where(EarthNewsItem.region.in_(regions))
|
||||
query = select(
|
||||
func.count(EarthNewsItem.id),
|
||||
func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)),
|
||||
)
|
||||
if active_region != "global":
|
||||
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
|
||||
result = await db.execute(query)
|
||||
count, newest = result.one()
|
||||
item_count = int(count or 0)
|
||||
if item_count == 0:
|
||||
@@ -115,6 +208,33 @@ async def get_earth_news_freshness(
|
||||
return item_count, _coerce_datetime(newest)
|
||||
|
||||
|
||||
async def get_earth_news_feed_coverage(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
active_region: str,
|
||||
recent_after: datetime | None = None,
|
||||
) -> set[tuple[str, str]]:
|
||||
query = select(
|
||||
EarthNewsItem.id,
|
||||
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"),
|
||||
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"),
|
||||
)
|
||||
if active_region != "global":
|
||||
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
|
||||
if recent_after is not None:
|
||||
query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after)
|
||||
result = await db.execute(query)
|
||||
coverage: set[tuple[str, str]] = set()
|
||||
for item_id, source_id, feed_id in result.all():
|
||||
normalized_source_id = str(source_id or "").strip()
|
||||
normalized_feed_id = str(feed_id or "").strip()
|
||||
if not normalized_source_id and isinstance(item_id, str) and ":" in item_id:
|
||||
normalized_source_id = item_id.split(":", 1)[0]
|
||||
if normalized_source_id and normalized_feed_id:
|
||||
coverage.add((normalized_source_id, normalized_feed_id))
|
||||
return coverage
|
||||
|
||||
|
||||
async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem]) -> int:
|
||||
if not items:
|
||||
return 0
|
||||
@@ -165,12 +285,20 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
|
||||
record.homepage_url = item.homepage_url
|
||||
record.published_at = item.published_at
|
||||
record.last_seen_at = now
|
||||
location_meta = dict(record.location_meta or {})
|
||||
location_meta["news_meta"] = _news_meta_patch(item)
|
||||
record.location_meta = location_meta
|
||||
if item.localizations:
|
||||
merged_localizations = {
|
||||
**dict(record.localizations or {}),
|
||||
**dict(item.localizations or {}),
|
||||
}
|
||||
record.content_language = item.content_language
|
||||
record.localizations = dict(item.localizations or {})
|
||||
record.enrichment_status = item.enrichment_status
|
||||
record.enrichment_error = item.enrichment_error
|
||||
record.enriched_at = item.enriched_at
|
||||
record.localizations = merged_localizations
|
||||
if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at:
|
||||
record.enrichment_status = item.enrichment_status
|
||||
record.enrichment_error = item.enrichment_error
|
||||
record.enriched_at = item.enriched_at
|
||||
changed += 1
|
||||
await db.flush()
|
||||
return changed
|
||||
|
||||
161
backend/app/services/log_tail.py
Normal file
161
backend/app/services/log_tail.py
Normal file
@@ -0,0 +1,161 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import WebSocket
|
||||
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.system_logs import (
|
||||
DEFAULT_LOG_LINE_LIMIT,
|
||||
LOG_SOURCES,
|
||||
MAX_LOG_LINE_LIMIT,
|
||||
read_database_log_events,
|
||||
read_log_events,
|
||||
)
|
||||
|
||||
DATABASE_LOG_SOURCE_IDS = {"system-db", "audit-db"}
|
||||
LOG_TAIL_CHANNEL = "logs_tail"
|
||||
LOG_TAIL_INTERVAL_SECONDS = 1.5
|
||||
LOG_TAIL_SCAN_MULTIPLIER = 5
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogTailConfig:
|
||||
source_id: str
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT
|
||||
level: str = "all"
|
||||
levels: str | None = None
|
||||
start_date: str | None = None
|
||||
end_date: str | None = None
|
||||
search: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class LogTailSubscription:
|
||||
config: LogTailConfig
|
||||
emitted_cursors: set[str] = field(default_factory=set)
|
||||
task: asyncio.Task | None = None
|
||||
|
||||
|
||||
class LogTailManager:
|
||||
def __init__(self) -> None:
|
||||
self._subscriptions: dict[WebSocket, LogTailSubscription] = {}
|
||||
|
||||
def normalize_config(self, payload: dict[str, Any]) -> LogTailConfig:
|
||||
source_id = str(payload.get("source_id") or payload.get("source") or "").strip()
|
||||
if not source_id:
|
||||
raise ValueError("source_id is required")
|
||||
if source_id not in LOG_SOURCES and source_id not in DATABASE_LOG_SOURCE_IDS:
|
||||
raise ValueError("Log source not found")
|
||||
try:
|
||||
limit = int(payload.get("limit") or DEFAULT_LOG_LINE_LIMIT)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("limit must be a number") from exc
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise ValueError(f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
|
||||
return LogTailConfig(
|
||||
source_id=source_id,
|
||||
limit=limit,
|
||||
level=str(payload.get("level") or "all"),
|
||||
levels=str(payload.get("levels")).strip() if payload.get("levels") else None,
|
||||
start_date=str(payload.get("start_date")).strip() if payload.get("start_date") else None,
|
||||
end_date=str(payload.get("end_date")).strip() if payload.get("end_date") else None,
|
||||
search=str(payload.get("search")).strip() if payload.get("search") else None,
|
||||
)
|
||||
|
||||
async def subscribe(self, websocket: WebSocket, payload: dict[str, Any]) -> LogTailConfig:
|
||||
config = self.normalize_config(payload)
|
||||
await self.unsubscribe(websocket)
|
||||
subscription = LogTailSubscription(config=config)
|
||||
subscription.task = asyncio.create_task(self._run_tail(websocket, subscription))
|
||||
self._subscriptions[websocket] = subscription
|
||||
return config
|
||||
|
||||
async def unsubscribe(self, websocket: WebSocket) -> None:
|
||||
subscription = self._subscriptions.pop(websocket, None)
|
||||
if subscription and subscription.task:
|
||||
subscription.task.cancel()
|
||||
try:
|
||||
await subscription.task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
async def disconnect(self, websocket: WebSocket) -> None:
|
||||
await self.unsubscribe(websocket)
|
||||
|
||||
async def _run_tail(self, websocket: WebSocket, subscription: LogTailSubscription) -> None:
|
||||
first_frame = True
|
||||
while True:
|
||||
events = await self._read_events(subscription.config)
|
||||
if first_frame:
|
||||
visible_events = events[-subscription.config.limit :]
|
||||
subscription.emitted_cursors.update(event.cursor for event in visible_events)
|
||||
await self._send_frame(websocket, subscription.config, "snapshot", visible_events)
|
||||
first_frame = False
|
||||
else:
|
||||
new_events = [
|
||||
event
|
||||
for event in events
|
||||
if event.cursor not in subscription.emitted_cursors
|
||||
]
|
||||
if new_events:
|
||||
visible_events = new_events[-subscription.config.limit :]
|
||||
subscription.emitted_cursors.update(event.cursor for event in visible_events)
|
||||
await self._send_frame(websocket, subscription.config, "append", visible_events)
|
||||
await asyncio.sleep(LOG_TAIL_INTERVAL_SECONDS)
|
||||
|
||||
async def _read_events(self, config: LogTailConfig):
|
||||
scan_limit = max(config.limit * LOG_TAIL_SCAN_MULTIPLIER, config.limit)
|
||||
if config.source_id in DATABASE_LOG_SOURCE_IDS:
|
||||
async with async_session_factory() as db:
|
||||
events = await read_database_log_events(
|
||||
config.source_id,
|
||||
scan_limit=scan_limit,
|
||||
level=config.level,
|
||||
levels=config.levels,
|
||||
start_date=config.start_date,
|
||||
end_date=config.end_date,
|
||||
search=config.search,
|
||||
db=db,
|
||||
)
|
||||
return events or []
|
||||
events = read_log_events(
|
||||
config.source_id,
|
||||
scan_limit=scan_limit,
|
||||
level=config.level,
|
||||
levels=config.levels,
|
||||
start_date=config.start_date,
|
||||
end_date=config.end_date,
|
||||
search=config.search,
|
||||
)
|
||||
return events or []
|
||||
|
||||
async def _send_frame(self, websocket: WebSocket, config: LogTailConfig, mode: str, events) -> None:
|
||||
await websocket.send_json(
|
||||
{
|
||||
"type": "data_frame",
|
||||
"channel": LOG_TAIL_CHANNEL,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"payload": {
|
||||
"mode": mode,
|
||||
"source_id": config.source_id,
|
||||
"line_count": len(events),
|
||||
"lines": [event.line for event in events],
|
||||
"filters": {
|
||||
"limit": config.limit,
|
||||
"level": config.level,
|
||||
"levels": config.levels,
|
||||
"start_date": config.start_date,
|
||||
"end_date": config.end_date,
|
||||
"search": config.search,
|
||||
},
|
||||
"status": "ok",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
log_tail_manager = LogTailManager()
|
||||
@@ -1,14 +1,185 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger, sanitize_log_value
|
||||
from app.core.request_context import get_request_id
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
HLS_TRANSIENT_RE = re.compile(r"(index|chunk|segment)[_-]?\d+(?:_\d+)?\.(?:ts|m4s|vtt)", re.IGNORECASE)
|
||||
QUERY_RE = re.compile(r"([?&](?:m|t|token|expires|signature|X-Amz-[^=]+)=[^&\\s]+)", re.IGNORECASE)
|
||||
UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.IGNORECASE)
|
||||
CONNECTION_RE = re.compile(r"\bconn_[A-Za-z0-9:._-]+\b")
|
||||
NUMBER_RE = re.compile(r"\b\d{5,}\b")
|
||||
|
||||
|
||||
def normalize_observability_text(value: Any) -> str:
|
||||
text = str(sanitize_log_value(value or "")).strip()
|
||||
text = QUERY_RE.sub("", text)
|
||||
text = HLS_TRANSIENT_RE.sub("<hls-fragment>", text)
|
||||
text = UUID_RE.sub("<uuid>", text)
|
||||
text = CONNECTION_RE.sub("<connection>", text)
|
||||
text = NUMBER_RE.sub("<number>", text)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def build_observability_fingerprint(
|
||||
*,
|
||||
source: str,
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
category: str | None = None,
|
||||
event: str | None = None,
|
||||
message: str,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> str:
|
||||
context = context or {}
|
||||
stable_context = {
|
||||
key: context.get(key)
|
||||
for key in (
|
||||
"task_type",
|
||||
"source_id",
|
||||
"source",
|
||||
"provider",
|
||||
"status_code",
|
||||
"error_type",
|
||||
"details",
|
||||
)
|
||||
if context.get(key) not in (None, "")
|
||||
}
|
||||
raw = "|".join(
|
||||
[
|
||||
normalize_observability_text(source),
|
||||
normalize_observability_text(service),
|
||||
normalize_observability_text(module),
|
||||
normalize_observability_text(category),
|
||||
normalize_observability_text(event),
|
||||
normalize_observability_text(message),
|
||||
normalize_observability_text(stable_context),
|
||||
]
|
||||
)
|
||||
return hashlib.sha1(raw.encode("utf-8", errors="replace")).hexdigest()
|
||||
|
||||
|
||||
def _context_text(context: dict[str, Any] | None, key: str) -> str | None:
|
||||
value = (context or {}).get(key)
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return str(value)
|
||||
|
||||
|
||||
async def record_observability_event(
|
||||
*,
|
||||
source: str,
|
||||
level: str,
|
||||
message: str,
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
event: str | None = None,
|
||||
request_id: str | None = None,
|
||||
trace_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
category: str | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
fingerprint: str | None = None,
|
||||
occurred_at: datetime | None = None,
|
||||
occurrence_count: int = 1,
|
||||
) -> None:
|
||||
normalized_context = sanitize_log_value(context or {})
|
||||
if not isinstance(normalized_context, dict):
|
||||
normalized_context = {"value": normalized_context}
|
||||
safe_message = str(sanitize_log_value(message))
|
||||
normalized_level = str(level or "info").lower()
|
||||
count = max(1, int(occurrence_count or 1))
|
||||
event_time = occurred_at or datetime.now(UTC)
|
||||
event_fingerprint = fingerprint or build_observability_fingerprint(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
category=category,
|
||||
event=event,
|
||||
message=safe_message,
|
||||
context=normalized_context,
|
||||
)
|
||||
detail = _context_text(normalized_context, "detail") or _context_text(normalized_context, "error")
|
||||
affected_sources = sorted(
|
||||
{
|
||||
item
|
||||
for item in (
|
||||
source,
|
||||
service,
|
||||
module,
|
||||
_context_text(normalized_context, "source_id"),
|
||||
_context_text(normalized_context, "source"),
|
||||
)
|
||||
if item
|
||||
}
|
||||
)
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
ObservabilityEvent(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
category=category,
|
||||
event=event,
|
||||
level=normalized_level,
|
||||
message=safe_message,
|
||||
fingerprint=event_fingerprint,
|
||||
occurred_at=event_time,
|
||||
request_id=request_id or get_request_id(),
|
||||
trace_id=trace_id,
|
||||
user_id=user_id,
|
||||
task_id=_context_text(normalized_context, "task_id"),
|
||||
source_ref_id=_context_text(normalized_context, "source_id") or _context_text(normalized_context, "source"),
|
||||
provider=_context_text(normalized_context, "provider"),
|
||||
context=normalized_context,
|
||||
occurrence_count=count,
|
||||
)
|
||||
)
|
||||
group = await session.get(ObservabilityEventGroup, event_fingerprint)
|
||||
if group is None:
|
||||
session.add(
|
||||
ObservabilityEventGroup(
|
||||
fingerprint=event_fingerprint,
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
category=category,
|
||||
event=event,
|
||||
last_level=normalized_level,
|
||||
sample_message=safe_message,
|
||||
sample_detail=detail,
|
||||
affected_sources=affected_sources,
|
||||
count=count,
|
||||
first_seen_at=event_time,
|
||||
last_seen_at=event_time,
|
||||
)
|
||||
)
|
||||
else:
|
||||
group.count = int(group.count or 0) + count
|
||||
group.last_seen_at = event_time
|
||||
group.last_level = normalized_level
|
||||
group.sample_message = safe_message
|
||||
group.sample_detail = detail
|
||||
merged_sources = sorted(set(group.affected_sources or []) | set(affected_sources))
|
||||
group.affected_sources = merged_sources
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist observability event",
|
||||
event="observability_event.persist.failed",
|
||||
context={"event_name": event, "source": source},
|
||||
)
|
||||
|
||||
|
||||
async def record_system_log(
|
||||
*,
|
||||
@@ -23,6 +194,8 @@ async def record_system_log(
|
||||
user_id: int | None = None,
|
||||
category: str | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
fingerprint: str | None = None,
|
||||
occurrence_count: int = 1,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
@@ -48,6 +221,21 @@ async def record_system_log(
|
||||
event="system_log.persist.failed",
|
||||
context={"event_name": event, "source": source},
|
||||
)
|
||||
await record_observability_event(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
level=level,
|
||||
message=message,
|
||||
request_id=request_id,
|
||||
trace_id=trace_id,
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=context,
|
||||
fingerprint=fingerprint,
|
||||
occurrence_count=occurrence_count,
|
||||
)
|
||||
|
||||
|
||||
async def record_audit_log(
|
||||
|
||||
@@ -5,6 +5,7 @@ import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import hashlib
|
||||
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
@@ -13,6 +14,9 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.security import redis_client
|
||||
from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
DEFAULT_LOG_LINE_LIMIT = 200
|
||||
MAX_LOG_LINE_LIMIT = 1000
|
||||
@@ -99,6 +103,16 @@ class StructuredLogEntry:
|
||||
search_text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogEvent:
|
||||
source_id: str
|
||||
cursor: str
|
||||
timestamp: datetime | None
|
||||
level: str | None
|
||||
line: str
|
||||
search_text: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyLogMarker:
|
||||
date_token: str
|
||||
@@ -106,6 +120,10 @@ class DailyLogMarker:
|
||||
dominant_level: str
|
||||
|
||||
|
||||
def _normalize_search_query(search: str | None) -> str:
|
||||
return (search or "").strip().lower()
|
||||
|
||||
|
||||
def _planet_state_dir() -> Path:
|
||||
configured = os.getenv("PLANET_STATE_DIR")
|
||||
if configured:
|
||||
@@ -135,7 +153,7 @@ LOG_SOURCES: dict[str, LogSource] = {
|
||||
name="前端开发服务",
|
||||
kind="file",
|
||||
location=_state_log_path("frontend.log"),
|
||||
description="控制台与 Earth 前端开发服务输出。",
|
||||
description="控制台与智能星球前端开发服务输出。",
|
||||
category="service",
|
||||
fallback_locations=("/tmp/planet_frontend.log",),
|
||||
),
|
||||
@@ -150,13 +168,22 @@ LOG_SOURCES: dict[str, LogSource] = {
|
||||
),
|
||||
"earth-client": LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
name="智能星球浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报的运行时错误与关键业务日志。",
|
||||
description="智能星球浏览器端上报的运行时错误与关键业务日志。",
|
||||
category="client",
|
||||
buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
|
||||
),
|
||||
"admin-client": LogSource(
|
||||
source_id="admin-client",
|
||||
name="控制台浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:admin-client",
|
||||
description="控制台浏览器端上报的运行时错误。",
|
||||
category="client",
|
||||
buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:admin-client",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -365,6 +392,59 @@ def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
|
||||
)
|
||||
|
||||
|
||||
def compact_log_context(context: dict | None) -> str:
|
||||
if not context:
|
||||
return ""
|
||||
allowed = {
|
||||
key: value
|
||||
for key, value in (context or {}).items()
|
||||
if key
|
||||
in {
|
||||
"status",
|
||||
"duration_ms",
|
||||
"provider",
|
||||
"model",
|
||||
"result_provider",
|
||||
"result_model",
|
||||
"collector_name",
|
||||
"datasource_id",
|
||||
"task_id",
|
||||
"snapshot_id",
|
||||
"raw_count",
|
||||
"transformed_count",
|
||||
"saved_count",
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"deleted",
|
||||
"result_count",
|
||||
"status_code",
|
||||
"error_type",
|
||||
"error",
|
||||
"route",
|
||||
"module",
|
||||
}
|
||||
}
|
||||
if not allowed:
|
||||
return ""
|
||||
return json.dumps(allowed, ensure_ascii=False, sort_keys=True)
|
||||
|
||||
|
||||
def context_search_aliases(context: dict | None) -> str:
|
||||
if not context:
|
||||
return ""
|
||||
aliases: list[str] = []
|
||||
for key, value in sorted((context or {}).items()):
|
||||
if value is None or isinstance(value, (dict, list, tuple, set)):
|
||||
continue
|
||||
normalized_key = str(key).strip()
|
||||
normalized_value = str(value).strip()
|
||||
if not normalized_key or not normalized_value:
|
||||
continue
|
||||
aliases.append(f"{normalized_key}={normalized_value}")
|
||||
return " ".join(aliases)
|
||||
|
||||
|
||||
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
path = resolve_file_log_path(source)
|
||||
if not path.exists():
|
||||
@@ -437,6 +517,400 @@ def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLo
|
||||
return []
|
||||
|
||||
|
||||
def _database_event_from_system_record(record: SystemLog) -> LogEvent:
|
||||
record_level = normalize_log_level(record.level)
|
||||
line = " ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
record_level.upper(),
|
||||
record.source,
|
||||
record.category or "",
|
||||
record.event or "",
|
||||
f"request_id={record.request_id}" if record.request_id else "",
|
||||
record.message,
|
||||
compact_log_context(record.context),
|
||||
]
|
||||
if part
|
||||
)
|
||||
search_text = " ".join(
|
||||
[
|
||||
line,
|
||||
f"id={record.id}",
|
||||
f"user_id={record.user_id}" if record.user_id else "",
|
||||
context_search_aliases(record.context),
|
||||
json.dumps(record.context or {}, ensure_ascii=False, sort_keys=True),
|
||||
]
|
||||
).lower()
|
||||
return LogEvent(
|
||||
source_id="system-db",
|
||||
cursor=f"system-db:{record.id}",
|
||||
timestamp=record.occurred_at,
|
||||
level=None if record_level == LOG_LEVEL_ALL else record_level,
|
||||
line=line,
|
||||
search_text=search_text,
|
||||
)
|
||||
|
||||
|
||||
def _database_event_from_audit_record(record: AuditLog) -> LogEvent:
|
||||
line = " ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
"INFO",
|
||||
record.action,
|
||||
record.target_type or "",
|
||||
record.target_id or "",
|
||||
record.result or "",
|
||||
f"request_id={record.request_id}" if record.request_id else "",
|
||||
]
|
||||
if part
|
||||
)
|
||||
search_text = " ".join(
|
||||
[
|
||||
line,
|
||||
f"id={record.id}",
|
||||
f"actor_id={record.actor_id}" if record.actor_id else "",
|
||||
record.actor_name or "",
|
||||
context_search_aliases(record.details),
|
||||
json.dumps(record.details or {}, ensure_ascii=False, sort_keys=True),
|
||||
]
|
||||
).lower()
|
||||
return LogEvent(
|
||||
source_id="audit-db",
|
||||
cursor=f"audit-db:{record.id}",
|
||||
timestamp=record.occurred_at,
|
||||
level=LOG_LEVEL_INFO,
|
||||
line=line,
|
||||
search_text=search_text,
|
||||
)
|
||||
|
||||
|
||||
async def read_database_log_events(
|
||||
source_id: str,
|
||||
*,
|
||||
scan_limit: int,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
db: AsyncSession,
|
||||
) -> list[LogEvent] | None:
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
if source_id == "system-db":
|
||||
query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(scan_limit)
|
||||
result = await db.execute(query)
|
||||
events = [_database_event_from_system_record(record) for record in result.scalars().all()]
|
||||
elif source_id == "audit-db":
|
||||
query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(scan_limit)
|
||||
result = await db.execute(query)
|
||||
events = [_database_event_from_audit_record(record) for record in result.scalars().all()]
|
||||
else:
|
||||
return None
|
||||
|
||||
events = list(reversed(events))
|
||||
return [
|
||||
event
|
||||
for event in events
|
||||
if event_matches_levels(event, selected_levels)
|
||||
and event_matches_search(event, search_query)
|
||||
and event_matches_date_range(event, start_date, end_date)
|
||||
]
|
||||
|
||||
|
||||
async def read_database_log_snapshot(
|
||||
source_id: str,
|
||||
*,
|
||||
limit: int,
|
||||
level: str,
|
||||
levels: str | None,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
search: str | None,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any] | None:
|
||||
events = await read_database_log_events(
|
||||
source_id,
|
||||
scan_limit=limit * 5,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
search=search,
|
||||
db=db,
|
||||
)
|
||||
if events is None:
|
||||
return None
|
||||
visible_events = events[-limit:]
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
return {
|
||||
"source_id": source_id,
|
||||
"name": "系统事件" if source_id == "system-db" else "审计事件",
|
||||
"kind": "database",
|
||||
"location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
|
||||
"description": "数据库持久化日志",
|
||||
"category": "database" if source_id == "system-db" else "audit",
|
||||
"status": "ok" if visible_events else "empty",
|
||||
"level": level,
|
||||
"selected_levels": list(selected_levels),
|
||||
"search_query": search or "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": build_daily_log_markers_from_events(events),
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible_events),
|
||||
"lines": [event.line for event in visible_events],
|
||||
}
|
||||
|
||||
|
||||
def _observability_group_matches(
|
||||
group: ObservabilityEventGroup,
|
||||
*,
|
||||
selected_levels: tuple[str, ...],
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
search: str | None,
|
||||
) -> bool:
|
||||
if selected_levels and group.last_level not in selected_levels:
|
||||
return False
|
||||
if start_date or end_date:
|
||||
if group.last_seen_at is None:
|
||||
return False
|
||||
date_token = group.last_seen_at.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
return False
|
||||
if end_date and date_token > end_date:
|
||||
return False
|
||||
query = _normalize_search_query(search)
|
||||
if not query:
|
||||
return True
|
||||
haystack = " ".join(
|
||||
[
|
||||
group.fingerprint or "",
|
||||
group.source or "",
|
||||
group.service or "",
|
||||
group.module or "",
|
||||
group.category or "",
|
||||
group.event or "",
|
||||
group.last_level or "",
|
||||
group.sample_message or "",
|
||||
group.sample_detail or "",
|
||||
json.dumps(group.affected_sources or [], ensure_ascii=False, sort_keys=True),
|
||||
]
|
||||
).lower()
|
||||
return query in haystack
|
||||
|
||||
|
||||
def _serialize_observability_group(group: ObservabilityEventGroup) -> dict[str, Any]:
|
||||
return {
|
||||
"fingerprint": group.fingerprint,
|
||||
"source": group.source,
|
||||
"service": group.service,
|
||||
"module": group.module,
|
||||
"category": group.category,
|
||||
"event": group.event,
|
||||
"level": group.last_level,
|
||||
"message": group.sample_message,
|
||||
"detail": group.sample_detail,
|
||||
"affected_sources": group.affected_sources or [],
|
||||
"count": group.count or 0,
|
||||
"first_seen_at": group.first_seen_at.isoformat() if group.first_seen_at else None,
|
||||
"last_seen_at": group.last_seen_at.isoformat() if group.last_seen_at else None,
|
||||
}
|
||||
|
||||
|
||||
def _serialize_observability_event(record: ObservabilityEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"id": record.id,
|
||||
"source": record.source,
|
||||
"service": record.service,
|
||||
"module": record.module,
|
||||
"category": record.category,
|
||||
"event": record.event,
|
||||
"level": record.level,
|
||||
"message": record.message,
|
||||
"fingerprint": record.fingerprint,
|
||||
"occurred_at": record.occurred_at.isoformat() if record.occurred_at else None,
|
||||
"request_id": record.request_id,
|
||||
"trace_id": record.trace_id,
|
||||
"task_id": record.task_id,
|
||||
"source_id": record.source_ref_id,
|
||||
"provider": record.provider,
|
||||
"user_id": record.user_id,
|
||||
"context": record.context or {},
|
||||
"occurrence_count": record.occurrence_count or 1,
|
||||
}
|
||||
|
||||
|
||||
async def read_observability_groups(
|
||||
*,
|
||||
limit: int,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any]:
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
scan_limit = max(limit * 5, limit, DEFAULT_LOG_LINE_LIMIT)
|
||||
result = await db.execute(
|
||||
select(ObservabilityEventGroup)
|
||||
.order_by(ObservabilityEventGroup.last_seen_at.desc().nullslast())
|
||||
.limit(scan_limit)
|
||||
)
|
||||
groups = [
|
||||
group
|
||||
for group in result.scalars().all()
|
||||
if _observability_group_matches(
|
||||
group,
|
||||
selected_levels=selected_levels,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
search=search,
|
||||
)
|
||||
][:limit]
|
||||
return {
|
||||
"mode": "grouped",
|
||||
"line_limit": limit,
|
||||
"line_count": len(groups),
|
||||
"groups": [_serialize_observability_group(group) for group in groups],
|
||||
"filters": {
|
||||
"level": level,
|
||||
"levels": list(selected_levels),
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"search": search or "",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
async def read_observability_group_events(
|
||||
fingerprint: str,
|
||||
*,
|
||||
limit: int,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any] | None:
|
||||
group = await db.get(ObservabilityEventGroup, fingerprint)
|
||||
if group is None:
|
||||
return None
|
||||
result = await db.execute(
|
||||
select(ObservabilityEvent)
|
||||
.where(ObservabilityEvent.fingerprint == fingerprint)
|
||||
.order_by(ObservabilityEvent.occurred_at.desc().nullslast(), ObservabilityEvent.id.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
events = list(reversed(result.scalars().all()))
|
||||
return {
|
||||
"fingerprint": fingerprint,
|
||||
"group": _serialize_observability_group(group),
|
||||
"line_limit": limit,
|
||||
"line_count": len(events),
|
||||
"events": [_serialize_observability_event(record) for record in events],
|
||||
}
|
||||
|
||||
|
||||
async def read_observability_raw_events(
|
||||
*,
|
||||
limit: int,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any]:
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
query = select(ObservabilityEvent).order_by(ObservabilityEvent.occurred_at.desc().nullslast(), ObservabilityEvent.id.desc())
|
||||
if selected_levels:
|
||||
query = query.where(ObservabilityEvent.level.in_(selected_levels))
|
||||
result = await db.execute(query.limit(max(limit * 5, limit)))
|
||||
records = result.scalars().all()
|
||||
search_query = _normalize_search_query(search)
|
||||
visible: list[ObservabilityEvent] = []
|
||||
for record in records:
|
||||
if start_date or end_date:
|
||||
if record.occurred_at is None:
|
||||
continue
|
||||
date_token = record.occurred_at.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
continue
|
||||
if end_date and date_token > end_date:
|
||||
continue
|
||||
if search_query:
|
||||
haystack = " ".join(
|
||||
[
|
||||
record.source or "",
|
||||
record.service or "",
|
||||
record.module or "",
|
||||
record.category or "",
|
||||
record.event or "",
|
||||
record.message or "",
|
||||
record.fingerprint or "",
|
||||
record.request_id or "",
|
||||
record.trace_id or "",
|
||||
record.task_id or "",
|
||||
record.source_ref_id or "",
|
||||
record.provider or "",
|
||||
json.dumps(record.context or {}, ensure_ascii=False, sort_keys=True),
|
||||
]
|
||||
).lower()
|
||||
if search_query not in haystack:
|
||||
continue
|
||||
visible.append(record)
|
||||
if len(visible) >= limit:
|
||||
break
|
||||
visible = list(reversed(visible))
|
||||
return {
|
||||
"mode": "raw",
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible),
|
||||
"events": [_serialize_observability_event(record) for record in visible],
|
||||
"lines": [
|
||||
" ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
record.level.upper(),
|
||||
record.source,
|
||||
record.category or "",
|
||||
record.event or "",
|
||||
f"fingerprint={record.fingerprint}",
|
||||
record.message,
|
||||
]
|
||||
if part
|
||||
)
|
||||
for record in visible
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _stable_hash(value: str) -> str:
|
||||
return hashlib.sha1(value.encode("utf-8", errors="replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def build_log_events(source_id: str, entries: list[StructuredLogEntry]) -> list[LogEvent]:
|
||||
events: list[LogEvent] = []
|
||||
seen: dict[str, int] = {}
|
||||
for entry in entries:
|
||||
stable_value = entry.raw_line or entry.display_line
|
||||
digest = _stable_hash(stable_value)
|
||||
occurrence = seen.get(digest, 0) + 1
|
||||
seen[digest] = occurrence
|
||||
events.append(
|
||||
LogEvent(
|
||||
source_id=source_id,
|
||||
cursor=f"{source_id}:{digest}:{occurrence}",
|
||||
timestamp=entry.timestamp,
|
||||
level=entry.level,
|
||||
line=entry.display_line,
|
||||
search_text=entry.search_text,
|
||||
)
|
||||
)
|
||||
return events
|
||||
|
||||
|
||||
def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
|
||||
if not selected_levels:
|
||||
return True
|
||||
@@ -469,6 +943,34 @@ def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
|
||||
return query in entry.search_text
|
||||
|
||||
|
||||
def event_matches_levels(event: LogEvent, selected_levels: tuple[str, ...]) -> bool:
|
||||
if not selected_levels:
|
||||
return True
|
||||
return event.level in selected_levels
|
||||
|
||||
|
||||
def event_matches_date_range(event: LogEvent, start_date: str | None, end_date: str | None) -> bool:
|
||||
if not start_date and not end_date:
|
||||
return True
|
||||
if event.timestamp is None:
|
||||
return False
|
||||
date_token = event.timestamp.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
return False
|
||||
if end_date and date_token > end_date:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def event_matches_search(event: LogEvent, search: str | None) -> bool:
|
||||
if search is None:
|
||||
return True
|
||||
query = search.strip().lower()
|
||||
if not query:
|
||||
return True
|
||||
return query in event.search_text
|
||||
|
||||
|
||||
def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
|
||||
grouped: dict[str, list[StructuredLogEntry]] = {}
|
||||
for entry in entries:
|
||||
@@ -503,6 +1005,69 @@ def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str,
|
||||
return [marker.__dict__ for marker in markers]
|
||||
|
||||
|
||||
def build_daily_log_markers_from_events(events: list[LogEvent]) -> list[dict[str, Any]]:
|
||||
grouped: dict[str, list[LogEvent]] = {}
|
||||
for event in events:
|
||||
if event.timestamp is None:
|
||||
continue
|
||||
date_token = event.timestamp.astimezone(UTC).date().isoformat()
|
||||
grouped.setdefault(date_token, []).append(event)
|
||||
|
||||
markers: list[DailyLogMarker] = []
|
||||
for date_token, group in sorted(grouped.items()):
|
||||
level_counts = Counter(
|
||||
event.level
|
||||
for event in group
|
||||
if event.level in SUPPORTED_LOG_LEVELS and event.level != LOG_LEVEL_ALL
|
||||
)
|
||||
dominant_level = LOG_LEVEL_INFO
|
||||
if level_counts:
|
||||
dominant_level = sorted(
|
||||
level_counts.items(),
|
||||
key=lambda item: (
|
||||
-item[1],
|
||||
("error", "warning", "info", "debug").index(item[0]),
|
||||
),
|
||||
)[0][0]
|
||||
markers.append(
|
||||
DailyLogMarker(
|
||||
date_token=date_token,
|
||||
total=len(group),
|
||||
dominant_level=dominant_level,
|
||||
)
|
||||
)
|
||||
return [marker.__dict__ for marker in markers]
|
||||
|
||||
|
||||
def read_log_events(
|
||||
source_id: str,
|
||||
scan_limit: int,
|
||||
*,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
) -> list[LogEvent] | None:
|
||||
source = LOG_SOURCES.get(source_id)
|
||||
if source is None:
|
||||
return None
|
||||
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
events = build_log_events(source_id, read_source_entries(source, scan_limit))
|
||||
marker_events = [
|
||||
event
|
||||
for event in events
|
||||
if event_matches_levels(event, selected_levels) and event_matches_search(event, search_query)
|
||||
]
|
||||
return [
|
||||
event
|
||||
for event in marker_events
|
||||
if event_matches_date_range(event, start_date, end_date)
|
||||
]
|
||||
|
||||
|
||||
def read_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int,
|
||||
@@ -520,18 +1085,18 @@ def read_log_snapshot(
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
|
||||
all_entries = read_source_entries(source, scan_limit)
|
||||
marker_entries = [
|
||||
entry
|
||||
for entry in all_entries
|
||||
if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
|
||||
all_events = build_log_events(source_id, read_source_entries(source, scan_limit))
|
||||
marker_events = [
|
||||
event
|
||||
for event in all_events
|
||||
if event_matches_levels(event, selected_levels) and event_matches_search(event, search_query)
|
||||
]
|
||||
filtered_entries = [
|
||||
entry
|
||||
for entry in marker_entries
|
||||
if matches_date_range(entry, start_date, end_date)
|
||||
filtered_events = [
|
||||
event
|
||||
for event in marker_events
|
||||
if event_matches_date_range(event, start_date, end_date)
|
||||
]
|
||||
visible_entries = filtered_entries[-limit:]
|
||||
visible_events = filtered_events[-limit:]
|
||||
|
||||
compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
|
||||
return {
|
||||
@@ -552,8 +1117,8 @@ def read_log_snapshot(
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
],
|
||||
"daily_markers": build_daily_log_markers(marker_entries),
|
||||
"daily_markers": build_daily_log_markers_from_events(marker_events),
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible_entries),
|
||||
"lines": [entry.display_line for entry in visible_entries],
|
||||
"line_count": len(visible_events),
|
||||
"lines": [event.line for event in visible_events],
|
||||
}
|
||||
|
||||
@@ -655,6 +655,8 @@ async def test_ingest_earth_client_log_accepts_public_events():
|
||||
"message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
|
||||
"category": "startup-load",
|
||||
"module": "layer-startup",
|
||||
"fingerprint": "client-test",
|
||||
"occurrence_count": 3,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
@@ -668,10 +670,98 @@ async def test_ingest_earth_client_log_accepts_public_events():
|
||||
assert persisted_kwargs["event"] == "earth.client.runtime_log"
|
||||
assert persisted_kwargs["category"] == "startup-load"
|
||||
assert persisted_kwargs["level"] == "error"
|
||||
assert persisted_kwargs["fingerprint"] == "client-test"
|
||||
assert persisted_kwargs["occurrence_count"] == 3
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_admin_client_log_accepts_public_events():
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
|
||||
with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/admin-client",
|
||||
json={
|
||||
"level": "error",
|
||||
"message": "控制台发生未处理 Promise 错误",
|
||||
"category": "unhandledrejection",
|
||||
"module": "admin",
|
||||
"url": "http://test/logs",
|
||||
"detail": "stack preview",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["accepted"] is True
|
||||
assert data["source_id"] == "admin-client"
|
||||
mock_append_buffer_log.assert_called_once()
|
||||
mock_record_system_log.assert_awaited_once()
|
||||
persisted_kwargs = mock_record_system_log.await_args.kwargs
|
||||
assert persisted_kwargs["source"] == "admin-client"
|
||||
assert persisted_kwargs["event"] == "admin.client.runtime_log"
|
||||
assert persisted_kwargs["category"] == "unhandledrejection"
|
||||
assert persisted_kwargs["context"]["url"] == "http://test/logs"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_service_log_requires_configured_token(monkeypatch):
|
||||
monkeypatch.setattr(settings, "OBSERVABILITY_INGEST_TOKEN", "")
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/service",
|
||||
json={"message": "AI provider failed"},
|
||||
headers={"X-Planet-Observability-Token": "secret"},
|
||||
)
|
||||
|
||||
assert response.status_code == 503
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_service_log_accepts_internal_token(monkeypatch):
|
||||
monkeypatch.setattr(settings, "OBSERVABILITY_INGEST_TOKEN", "service-secret")
|
||||
transport = ASGITransport(app=app)
|
||||
with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/service",
|
||||
json={
|
||||
"source": "ai-provider",
|
||||
"service": "ai-provider",
|
||||
"module": "provider",
|
||||
"category": "connectivity",
|
||||
"event": "ai.provider.test.failed",
|
||||
"level": "error",
|
||||
"message": "Provider connectivity failed",
|
||||
"fingerprint": "ai-provider-test",
|
||||
"occurrence_count": 4,
|
||||
"provider": "minimax",
|
||||
"trace_id": "trace-123",
|
||||
"context": {"status_code": 502},
|
||||
},
|
||||
headers={"Authorization": "Bearer service-secret"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["accepted"] is True
|
||||
assert data["source_id"] == "ai-provider"
|
||||
mock_record_system_log.assert_awaited_once()
|
||||
persisted_kwargs = mock_record_system_log.await_args.kwargs
|
||||
assert persisted_kwargs["event"] == "ai.provider.test.failed"
|
||||
assert persisted_kwargs["fingerprint"] == "ai-provider-test"
|
||||
assert persisted_kwargs["occurrence_count"] == 4
|
||||
assert persisted_kwargs["context"]["provider"] == "minimax"
|
||||
assert persisted_kwargs["context"]["trace_id"] == "trace-123"
|
||||
assert persisted_kwargs["context"]["status_code"] == 502
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_layer_cache_status_requires_super_admin(auth_headers, monkeypatch):
|
||||
def override_get_current_user():
|
||||
|
||||
@@ -74,6 +74,6 @@ def test_interactable_cache_invalidation_clears_layer_and_all(monkeypatch):
|
||||
|
||||
assert deleted == 2
|
||||
assert patterns == [
|
||||
"earth:layer:v1:interactables:layer:places*",
|
||||
"earth:layer:v1:interactables:layer:all*",
|
||||
"earth:layer:v1:interactables:interactable_layer:places*",
|
||||
"earth:layer:v1:interactables:interactable_layer:all*",
|
||||
]
|
||||
|
||||
@@ -4,14 +4,20 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import (
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
NewsTargetLocation,
|
||||
ParsedNewsItem,
|
||||
apply_news_classification,
|
||||
default_earth_news_sources_payload,
|
||||
normalize_earth_news_sources_payload,
|
||||
_fetch_source,
|
||||
_enrich_items_with_target_locations,
|
||||
_extract_target_location_from_text,
|
||||
_parse_feed_entries,
|
||||
_serialize_item,
|
||||
get_earth_news_payload,
|
||||
test_news_source_config as run_news_source_config_test,
|
||||
)
|
||||
from app.services.earth_news_queue import NewsTargetLocationMessage
|
||||
from app.services.earth_news_worker import process_target_location_message
|
||||
@@ -164,6 +170,479 @@ def test_parse_aggregated_rss_splits_publisher_from_title():
|
||||
assert items[0].source == "Reuters"
|
||||
|
||||
|
||||
def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
|
||||
source = NewsFeedSource(
|
||||
id="36kr",
|
||||
name="36氪",
|
||||
region="asia-pacific",
|
||||
feed_url="https://36kr.com/feed",
|
||||
homepage_url="https://www.36kr.com/",
|
||||
source_tags=("china", "business_news"),
|
||||
default_category="business",
|
||||
)
|
||||
xml = """
|
||||
<rss>
|
||||
<channel>
|
||||
<item>
|
||||
<title>中国电商平台发布季度增长数据</title>
|
||||
<description>平台表示,跨境电商订单量同比增长。</description>
|
||||
<link>https://36kr.com/p/example</link>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
"""
|
||||
|
||||
items = _parse_feed_entries(xml, source)
|
||||
payload_zh = _serialize_item(items[0], active_region="global", locale="zh-CN")
|
||||
payload_en = _serialize_item(items[0], active_region="global", locale="en-US")
|
||||
|
||||
assert items[0].content_language == "zh-CN"
|
||||
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
|
||||
|
||||
|
||||
def test_default_news_sources_include_business_and_ecommerce_sources():
|
||||
payload = default_earth_news_sources_payload()
|
||||
sources_by_id = {source["id"]: source for source in payload["sources"]}
|
||||
source_ids = {source["id"] for source in payload["sources"]}
|
||||
category_keys = {category["key"] for category in payload["categories"]}
|
||||
tag_keys = {tag["key"] for tag in payload["source_tags"]}
|
||||
|
||||
assert "cnbc-business" in source_ids
|
||||
assert "36kr" in source_ids
|
||||
assert "techcrunch" in source_ids
|
||||
assert "retaildive" in source_ids
|
||||
assert "prnewswire-retail" in source_ids
|
||||
assert "google-news" in source_ids
|
||||
assert "global-scan" not in source_ids
|
||||
assert "google-americas" not in source_ids
|
||||
assert "google-europe" not in source_ids
|
||||
assert "google-mea" not in source_ids
|
||||
assert "google-apac" not in source_ids
|
||||
assert "businesswire-ecommerce" in source_ids
|
||||
assert "us-census-ecommerce" in source_ids
|
||||
assert "mofcom-data" in source_ids
|
||||
assert "stats-china-online-retail" in source_ids
|
||||
assert "ebrun" in source_ids
|
||||
assert sources_by_id["36kr"]["source_type"] == "rss"
|
||||
assert sources_by_id["36kr"]["homepage_url"] == "https://www.36kr.com/"
|
||||
assert sources_by_id["36kr"]["feed_directory_url"] == "https://www.36kr.com/rss-center"
|
||||
kr_feeds = {feed["id"]: feed for feed in sources_by_id["36kr"]["feeds"]}
|
||||
assert set(kr_feeds) == {"feed", "article", "newsflash", "moment"}
|
||||
assert kr_feeds["feed"]["url"] == "https://36kr.com/feed"
|
||||
assert kr_feeds["article"]["url"] == "https://36kr.com/feed-article"
|
||||
assert kr_feeds["newsflash"]["url"] == "https://36kr.com/feed-newsflash"
|
||||
assert kr_feeds["moment"]["url"] == "https://36kr.com/feed-moment"
|
||||
assert all(feed["enabled"] is True for feed in kr_feeds.values())
|
||||
assert all(feed["default_category"] == "business" for feed in kr_feeds.values())
|
||||
assert "https://36kr.com/feed-article" in sources_by_id["36kr"]["feed_urls"]
|
||||
assert "https://36kr.com/feed-newsflash" in sources_by_id["36kr"]["feed_urls"]
|
||||
assert "https://36kr.com/feed-moment" in sources_by_id["36kr"]["feed_urls"]
|
||||
assert sources_by_id["ebrun"]["source_type"] == "rss"
|
||||
assert sources_by_id["ebrun"]["homepage_url"] == "https://www.ebrun.com/"
|
||||
assert sources_by_id["ebrun"]["feed_directory_url"] == "https://www.ebrun.com/rss/"
|
||||
ebrun_feeds = {feed["id"]: feed for feed in sources_by_id["ebrun"]["feeds"]}
|
||||
assert {"b2c", "b2b", "retail", "o2o", "service", "data", "policy"}.issubset(ebrun_feeds)
|
||||
assert all(feed["enabled"] is True for feed in ebrun_feeds.values())
|
||||
assert all(feed["default_category"] == "ecommerce" for feed in ebrun_feeds.values())
|
||||
assert "https://www.ebrun.com/rss/news_b2c.xml" in sources_by_id["ebrun"]["feed_urls"]
|
||||
assert "https://www.ebrun.com/rss/news_retail.xml" in sources_by_id["ebrun"]["feed_urls"]
|
||||
assert sources_by_id["businesswire-ecommerce"]["source_type"] == "reference"
|
||||
assert sources_by_id["businesswire-ecommerce"]["enabled"] is False
|
||||
assert sources_by_id["google-news"]["source_type"] == "aggregated"
|
||||
assert sources_by_id["google-news"]["homepage_url"] == "https://news.google.com/"
|
||||
assert sources_by_id["google-news"]["feed_directory_url"] == "https://news.google.com/rss"
|
||||
google_feeds = {feed["id"]: feed for feed in sources_by_id["google-news"]["feeds"]}
|
||||
assert set(google_feeds) == {"world", "americas", "europe", "middle-east-africa", "asia-pacific"}
|
||||
assert all(feed["type"] == "aggregated" for feed in google_feeds.values())
|
||||
assert all(feed["enabled"] is True for feed in google_feeds.values())
|
||||
assert google_feeds["world"]["region"] == "global"
|
||||
assert google_feeds["europe"]["region"] == "europe"
|
||||
assert sources_by_id["stats-china-online-retail"]["source_type"] == "rss"
|
||||
assert sources_by_id["stats-china-online-retail"]["enabled"] is True
|
||||
assert "https://www.stats.gov.cn/sj/zxfb/rss.xml" in sources_by_id["stats-china-online-retail"]["feed_urls"]
|
||||
assert {"business", "ecommerce", "finance"}.issubset(category_keys)
|
||||
assert {"official_data", "business_news", "ecommerce", "press_release", "finance", "logistics"}.issubset(tag_keys)
|
||||
|
||||
|
||||
def test_default_enabled_fetchable_sources_have_explicit_types_and_urls():
|
||||
payload = default_earth_news_sources_payload()
|
||||
for source in payload["sources"]:
|
||||
source_type = source["source_type"]
|
||||
assert source_type in {"rss", "atom", "aggregated", "reference"}
|
||||
if source_type == "reference":
|
||||
assert source["enabled"] is False
|
||||
assert source["feeds"] == []
|
||||
continue
|
||||
if source["enabled"]:
|
||||
assert source["feed_url"]
|
||||
assert source["feed_urls"]
|
||||
assert source["feeds"]
|
||||
assert any(feed["enabled"] for feed in source["feeds"])
|
||||
for feed in source["feeds"]:
|
||||
assert feed["url"] != source["homepage_url"]
|
||||
assert feed["url"] != source.get("feed_directory_url", "")
|
||||
|
||||
|
||||
def test_legacy_news_source_urls_migrate_to_feed_children():
|
||||
payload = normalize_earth_news_sources_payload(
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "legacy-source",
|
||||
"name": "Legacy Source",
|
||||
"region": "global",
|
||||
"source_type": "rss",
|
||||
"feed_urls": ["https://example.com/a.xml", "https://example.com/b.xml"],
|
||||
"default_category": "business",
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
source = payload["sources"][0]
|
||||
|
||||
assert source["feed_urls"] == ["https://example.com/a.xml", "https://example.com/b.xml"]
|
||||
assert [feed["url"] for feed in source["feeds"]] == ["https://example.com/a.xml", "https://example.com/b.xml"]
|
||||
assert [feed["id"] for feed in source["feeds"]] == ["feed-1", "feed-2"]
|
||||
assert all(feed["default_category"] == "business" for feed in source["feeds"])
|
||||
|
||||
|
||||
def test_builtin_news_source_legacy_directory_url_is_repaired():
|
||||
payload = normalize_earth_news_sources_payload(
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "36kr",
|
||||
"name": "36氪",
|
||||
"region": "asia-pacific",
|
||||
"source_type": "rss",
|
||||
"homepage_url": "https://www.36kr.com/",
|
||||
"feed_url": "https://www.36kr.com/rss-center",
|
||||
"feed_urls": ["https://www.36kr.com/rss-center"],
|
||||
"feeds": [
|
||||
{
|
||||
"id": "feed-1",
|
||||
"name": "36氪",
|
||||
"url": "https://www.36kr.com/rss-center",
|
||||
"type": "rss",
|
||||
"enabled": True,
|
||||
"default_category": "business",
|
||||
}
|
||||
],
|
||||
"default_category": "business",
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
source = payload["sources"][0]
|
||||
feed_urls = {feed["url"] for feed in source["feeds"]}
|
||||
|
||||
assert source["homepage_url"] == "https://www.36kr.com/"
|
||||
assert source["feed_directory_url"] == "https://www.36kr.com/rss-center"
|
||||
assert "https://www.36kr.com/rss-center" not in feed_urls
|
||||
assert {
|
||||
"https://36kr.com/feed",
|
||||
"https://36kr.com/feed-article",
|
||||
"https://36kr.com/feed-newsflash",
|
||||
"https://36kr.com/feed-moment",
|
||||
}.issubset(feed_urls)
|
||||
|
||||
|
||||
def test_builtin_news_source_without_feed_children_gets_explicit_defaults():
|
||||
payload = normalize_earth_news_sources_payload(
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "ebrun",
|
||||
"name": "亿邦动力",
|
||||
"region": "asia-pacific",
|
||||
"source_type": "rss",
|
||||
"homepage_url": "https://www.ebrun.com/",
|
||||
"feed_url": "https://www.ebrun.com/rss/news_b2c.xml",
|
||||
"feed_urls": ["https://www.ebrun.com/rss/news_b2c.xml"],
|
||||
"default_category": "ecommerce",
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
source = payload["sources"][0]
|
||||
feed_urls = {feed["url"] for feed in source["feeds"]}
|
||||
|
||||
assert source["feed_directory_url"] == "https://www.ebrun.com/rss/"
|
||||
assert "https://www.ebrun.com/rss/" not in feed_urls
|
||||
assert {
|
||||
"https://www.ebrun.com/rss/news_b2c.xml",
|
||||
"https://www.ebrun.com/rss/news_b2b.xml",
|
||||
"https://www.ebrun.com/rss/news_retail.xml",
|
||||
"https://www.ebrun.com/rss/news_o2o.xml",
|
||||
"https://www.ebrun.com/rss/news_service.xml",
|
||||
"https://www.ebrun.com/rss/news_data.xml",
|
||||
"https://www.ebrun.com/rss/news_policy.xml",
|
||||
}.issubset(feed_urls)
|
||||
|
||||
|
||||
def test_builtin_fetchable_source_saved_as_reference_is_repaired():
|
||||
payload = normalize_earth_news_sources_payload(
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "stats-china-online-retail",
|
||||
"name": "国家统计局数据发布",
|
||||
"region": "asia-pacific",
|
||||
"source_type": "reference",
|
||||
"enabled": False,
|
||||
"homepage_url": "https://www.stats.gov.cn/sj/zxfb/",
|
||||
"feed_url": "https://www.stats.gov.cn/sj/zxfb/",
|
||||
"default_category": "ecommerce",
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
source = payload["sources"][0]
|
||||
|
||||
assert source["source_type"] == "rss"
|
||||
assert source["enabled"] is True
|
||||
assert source["priority"] == 19
|
||||
assert source["source_tags"] == ["official_data", "ecommerce", "retail", "china"]
|
||||
assert source["default_category"] == "ecommerce"
|
||||
assert source["importance_weight"] == 36
|
||||
assert source["feed_directory_url"] == ""
|
||||
assert source["feeds"] == [
|
||||
{
|
||||
"id": "release",
|
||||
"name": "数据发布",
|
||||
"url": "https://www.stats.gov.cn/sj/zxfb/rss.xml",
|
||||
"type": "rss",
|
||||
"region": "asia-pacific",
|
||||
"enabled": True,
|
||||
"default_category": "ecommerce",
|
||||
"tags": [],
|
||||
"priority": 1,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_legacy_google_sources_merge_into_google_news_source():
|
||||
payload = normalize_earth_news_sources_payload(
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "global-scan",
|
||||
"name": "Global Monitor / World",
|
||||
"region": "global",
|
||||
"source_type": "aggregated",
|
||||
"feed_url": "https://news.google.com/rss/search?q=world",
|
||||
"homepage_url": "https://news.google.com/",
|
||||
},
|
||||
{
|
||||
"id": "google-europe",
|
||||
"name": "Global Monitor / Europe",
|
||||
"region": "europe",
|
||||
"source_type": "aggregated",
|
||||
"feed_url": "https://news.google.com/rss/search?q=europe",
|
||||
"homepage_url": "https://news.google.com/",
|
||||
},
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
sources_by_id = {source["id"]: source for source in payload["sources"]}
|
||||
|
||||
assert "global-scan" not in sources_by_id
|
||||
assert "google-europe" not in sources_by_id
|
||||
assert "google-news" in sources_by_id
|
||||
assert {feed["id"] for feed in sources_by_id["google-news"]["feeds"]} == {
|
||||
"world",
|
||||
"americas",
|
||||
"europe",
|
||||
"middle-east-africa",
|
||||
"asia-pacific",
|
||||
}
|
||||
|
||||
|
||||
def test_feed_child_default_category_overrides_source_default():
|
||||
source = NewsFeedSource(
|
||||
id="multi-feed",
|
||||
name="Multi Feed",
|
||||
region="global",
|
||||
feed_url="https://example.com/source.xml",
|
||||
homepage_url="https://example.com",
|
||||
default_category="business",
|
||||
)
|
||||
feed = NewsFeedEndpoint(
|
||||
id="ecommerce-feed",
|
||||
name="Ecommerce Feed",
|
||||
url="https://example.com/ecommerce.xml",
|
||||
default_category="ecommerce",
|
||||
)
|
||||
xml = """
|
||||
<rss>
|
||||
<channel>
|
||||
<item>
|
||||
<title>Quarterly results released</title>
|
||||
<description>Company update.</description>
|
||||
<link>https://example.com/results</link>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
"""
|
||||
|
||||
items = _parse_feed_entries(xml, source, feed=feed)
|
||||
|
||||
assert items[0].feed_id == "ecommerce-feed"
|
||||
assert items[0].feed_name == "Ecommerce Feed"
|
||||
assert items[0].feed_default_category == "ecommerce"
|
||||
assert items[0].category == "ecommerce"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_source_only_requests_enabled_feed_children(monkeypatch):
|
||||
source = NewsFeedSource(
|
||||
id="multi-feed",
|
||||
name="Multi Feed",
|
||||
region="global",
|
||||
feed_url="https://example.com/source.xml",
|
||||
homepage_url="https://example.com",
|
||||
feeds=(
|
||||
NewsFeedEndpoint(id="enabled", name="Enabled", url="https://example.com/enabled.xml", enabled=True),
|
||||
NewsFeedEndpoint(id="disabled", name="Disabled", url="https://example.com/disabled.xml", enabled=False),
|
||||
),
|
||||
)
|
||||
calls = []
|
||||
|
||||
async def fake_fetch_single(_client, feed_source, feed, *, config_payload=None):
|
||||
calls.append(feed.id)
|
||||
item = ParsedNewsItem(
|
||||
id=f"{feed_source.id}:{feed.id}:1",
|
||||
title="Fetched story",
|
||||
summary="Fetched summary",
|
||||
url=f"https://example.com/{feed.id}",
|
||||
source="Example",
|
||||
feed_name=feed.name,
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=None,
|
||||
feed_id=feed.id,
|
||||
)
|
||||
return feed_source, [item], None, {"source_id": feed_source.id, "feed_id": feed.id, "ok": True, "status": "ok", "item_count": 1, "count": 1}
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_single_feed_url", fake_fetch_single)
|
||||
|
||||
source_result, items, error, health = await _fetch_source(object(), source)
|
||||
|
||||
assert source_result.id == "multi-feed"
|
||||
assert calls == ["enabled"]
|
||||
assert error is None
|
||||
assert [item.feed_id for item in items] == ["enabled"]
|
||||
assert health["ok"] is True
|
||||
assert [result["feed_id"] for result in health["feed_results"]] == ["enabled"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_source_filters_google_feed_children_by_active_region(monkeypatch):
|
||||
source = NewsFeedSource(
|
||||
id="google-news",
|
||||
name="Google News",
|
||||
region="global",
|
||||
feed_url="https://news.google.com/rss",
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
feeds=(
|
||||
NewsFeedEndpoint(id="world", name="全球", url="https://example.com/world.xml", type="aggregated", region="global"),
|
||||
NewsFeedEndpoint(id="europe", name="欧洲", url="https://example.com/europe.xml", type="aggregated", region="europe"),
|
||||
NewsFeedEndpoint(id="americas", name="美洲", url="https://example.com/americas.xml", type="aggregated", region="americas"),
|
||||
),
|
||||
)
|
||||
calls = []
|
||||
|
||||
async def fake_fetch_single(_client, feed_source, feed, *, config_payload=None):
|
||||
calls.append(feed.id)
|
||||
item = ParsedNewsItem(
|
||||
id=f"{feed_source.id}:{feed.id}:1",
|
||||
title=f"{feed.name} headline",
|
||||
summary="Fetched summary",
|
||||
url=f"https://example.com/{feed.id}",
|
||||
source="Example",
|
||||
feed_name=feed.name,
|
||||
feed_region=feed.region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=None,
|
||||
feed_id=feed.id,
|
||||
)
|
||||
return feed_source, [item], None, {"source_id": feed_source.id, "feed_id": feed.id, "ok": True, "status": "ok", "item_count": 1, "count": 1}
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_single_feed_url", fake_fetch_single)
|
||||
|
||||
_source_result, items, error, health = await _fetch_source(object(), source, active_region="europe")
|
||||
|
||||
assert error is None
|
||||
assert calls == ["world", "europe"]
|
||||
assert [item.feed_region for item in items] == ["global", "europe"]
|
||||
assert [result["feed_id"] for result in health["feed_results"]] == ["world", "europe"]
|
||||
|
||||
|
||||
def test_parse_rdf_rss_items_with_namespaces():
|
||||
source = NewsFeedSource(
|
||||
id="dw-top",
|
||||
name="DW Top Stories",
|
||||
region="europe",
|
||||
feed_url="https://rss.dw.com/rdf/rss-en-top",
|
||||
homepage_url="https://www.dw.com/en/top-stories/s-9097",
|
||||
)
|
||||
xml = """
|
||||
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns="http://purl.org/rss/1.0/">
|
||||
<item rdf:about="https://example.com/dw">
|
||||
<title>German retail sales rise</title>
|
||||
<link>https://example.com/dw</link>
|
||||
<description>Retail summary</description>
|
||||
</item>
|
||||
</rdf:RDF>
|
||||
"""
|
||||
|
||||
items = _parse_feed_entries(xml, source)
|
||||
|
||||
assert len(items) == 1
|
||||
assert items[0].title == "German retail sales rise"
|
||||
|
||||
|
||||
def test_news_classification_marks_ecommerce_and_importance():
|
||||
source = NewsFeedSource(
|
||||
id="ebrun",
|
||||
name="亿邦动力",
|
||||
region="asia-pacific",
|
||||
feed_url="https://www.ebrun.com/rss/",
|
||||
homepage_url="https://www.ebrun.com/",
|
||||
source_tags=("business_news", "ecommerce", "china"),
|
||||
default_category="ecommerce",
|
||||
importance_weight=14,
|
||||
)
|
||||
item = ParsedNewsItem(
|
||||
id="ebrun:test",
|
||||
title="跨境电商平台 GMV 同比增长,物流履约效率提升",
|
||||
summary="订单量和网上零售额继续增长。",
|
||||
url="https://example.com/ecommerce",
|
||||
source="亿邦动力",
|
||||
feed_name="亿邦动力",
|
||||
feed_region="asia-pacific",
|
||||
homepage_url="https://www.ebrun.com/",
|
||||
published_at=None,
|
||||
)
|
||||
|
||||
apply_news_classification(item, source)
|
||||
|
||||
assert item.category == "ecommerce"
|
||||
assert "cross_border_ecommerce" in item.item_tags
|
||||
assert "logistics_fulfillment" in item.item_tags
|
||||
assert item.importance_level in {"high", "critical"}
|
||||
assert "命中电商数据指标" in item.importance_reasons
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_enrich_items_with_target_locations_uses_ai_and_geocode(monkeypatch):
|
||||
item = ParsedNewsItem(
|
||||
@@ -338,8 +817,8 @@ async def test_earth_news_payload_returns_anchor_items_and_enqueues_location_job
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
|
||||
async def fake_fetch_source(_client, feed_source):
|
||||
return feed_source, [item], None
|
||||
async def fake_fetch_source(_client, feed_source, **_kwargs):
|
||||
return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1}
|
||||
|
||||
async def fake_get_cached_target_location_patch(_item_id):
|
||||
return None
|
||||
@@ -402,7 +881,7 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
assert limit == 12
|
||||
return [item]
|
||||
|
||||
@@ -452,10 +931,10 @@ async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monke
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
return [current_item]
|
||||
|
||||
async def fake_list_earth_news_cruise_items(_db, *, limit):
|
||||
async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None):
|
||||
return [current_item, cruise_item]
|
||||
|
||||
async def fake_enqueue_target_location_job(_payload, **_kwargs):
|
||||
@@ -477,6 +956,89 @@ async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monke
|
||||
assert payload["cruise_items"][1]["region"] == "asia-pacific"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_passes_region_and_category_filters_to_store(monkeypatch):
|
||||
class FakeDb:
|
||||
execute = object()
|
||||
|
||||
captured = {}
|
||||
item = ParsedNewsItem(
|
||||
id="db:business",
|
||||
title="Business story",
|
||||
summary="Business summary",
|
||||
url="https://example.com/business",
|
||||
source="Stored Source",
|
||||
feed_name="Stored Feed",
|
||||
feed_region="europe",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
category="business",
|
||||
)
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
captured["freshness_region"] = active_region
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
|
||||
captured["items_region"] = active_region
|
||||
captured["items_categories"] = categories
|
||||
captured["items_source_ids"] = source_ids
|
||||
return [item]
|
||||
|
||||
async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None, source_ids=None):
|
||||
captured["cruise_categories"] = categories
|
||||
captured["cruise_source_ids"] = source_ids
|
||||
return [item]
|
||||
|
||||
async def fake_enqueue_target_location_job(_payload, **_kwargs):
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items)
|
||||
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", lambda _sources: (_ for _ in ()).throw(AssertionError("fresh database items should not fetch RSS")))
|
||||
|
||||
payload = await get_earth_news_payload(
|
||||
lat=35.0,
|
||||
lon=-100.0,
|
||||
region="europe",
|
||||
categories={"business", "ecommerce"},
|
||||
db=FakeDb(),
|
||||
)
|
||||
|
||||
assert captured["freshness_region"] == "europe"
|
||||
assert captured["items_region"] == "europe"
|
||||
assert captured["items_categories"] == {"business", "ecommerce"}
|
||||
assert captured["items_source_ids"] is None
|
||||
assert captured["cruise_categories"] == {"business", "ecommerce"}
|
||||
assert captured["cruise_source_ids"] is None
|
||||
assert payload["filters"] == {
|
||||
"region": "europe",
|
||||
"categories": ["business", "ecommerce"],
|
||||
"sources": [],
|
||||
"limit": 12,
|
||||
"locale": "zh-CN",
|
||||
}
|
||||
assert payload["items"][0]["category"] == "business"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_news_source_test_treats_type_reference_as_non_fetching():
|
||||
result = await run_news_source_config_test(
|
||||
{
|
||||
"id": "reference-only",
|
||||
"name": "Reference Only",
|
||||
"type": "reference",
|
||||
"feed_url": "https://example.com",
|
||||
}
|
||||
)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert result["health"]["status"] == "reference"
|
||||
assert "不参与 RSS/Atom 抓取" in result["error"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
|
||||
db = object()
|
||||
@@ -512,14 +1074,14 @@ async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatc
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 0, None
|
||||
|
||||
async def fake_fetch_rss_items_for_sources(_sources):
|
||||
return [item], []
|
||||
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
|
||||
return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}}
|
||||
|
||||
async def fake_upsert_earth_news_items(_db, items):
|
||||
upserted.extend(items)
|
||||
return len(items)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
return [item]
|
||||
|
||||
async def fake_enqueue_target_location_job(payload, **_kwargs):
|
||||
@@ -568,14 +1130,14 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
|
||||
|
||||
async def fake_fetch_rss_items_for_sources(_sources):
|
||||
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
|
||||
fetched.append(True)
|
||||
return [old_item], []
|
||||
return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}}
|
||||
|
||||
async def fake_upsert_earth_news_items(_db, items):
|
||||
return len(items)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
return [old_item]
|
||||
|
||||
async def fake_enqueue_target_location_job(_payload, **_kwargs):
|
||||
@@ -630,8 +1192,8 @@ async def test_earth_news_payload_merges_cached_location_patch(monkeypatch):
|
||||
},
|
||||
}
|
||||
|
||||
async def fake_fetch_source(_client, feed_source):
|
||||
return feed_source, [item], None
|
||||
async def fake_fetch_source(_client, feed_source, **_kwargs):
|
||||
return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1}
|
||||
|
||||
async def fake_get_cached_target_location_patch(_item_id):
|
||||
return cached_patch
|
||||
@@ -697,8 +1259,8 @@ async def test_earth_news_payload_requeues_cached_failed_localization(monkeypatc
|
||||
}
|
||||
enqueued = []
|
||||
|
||||
async def fake_fetch_source(_client, feed_source):
|
||||
return feed_source, [item], None
|
||||
async def fake_fetch_source(_client, feed_source, **_kwargs):
|
||||
return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1}
|
||||
|
||||
async def fake_get_cached_target_location_patch(_item_id):
|
||||
return cached_patch
|
||||
|
||||
@@ -9,6 +9,8 @@ import pytest
|
||||
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
from app.services import business_logs
|
||||
from app.services import persistent_logs
|
||||
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
@@ -98,6 +100,86 @@ def test_business_context_redacts_nested_sensitive_values():
|
||||
assert context["nested"]["safe"] == "visible"
|
||||
|
||||
|
||||
def test_observability_fingerprint_normalizes_hls_fragments():
|
||||
first = persistent_logs.build_observability_fingerprint(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module="tv",
|
||||
category="hls-proxy",
|
||||
event="hls.fragment.failed",
|
||||
message="HLS 分片加载失败: index_5_9086220.ts?m=1725933270",
|
||||
context={"status_code": 502},
|
||||
)
|
||||
second = persistent_logs.build_observability_fingerprint(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module="tv",
|
||||
category="hls-proxy",
|
||||
event="hls.fragment.failed",
|
||||
message="HLS 分片加载失败: index_5_9086361.ts?m=1725934270",
|
||||
context={"status_code": 502},
|
||||
)
|
||||
|
||||
assert first == second
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_record_observability_event_updates_group_count(monkeypatch):
|
||||
events: list[ObservabilityEvent] = []
|
||||
groups: dict[str, ObservabilityEventGroup] = {}
|
||||
|
||||
class FakeSession:
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def add(self, item):
|
||||
if isinstance(item, ObservabilityEvent):
|
||||
events.append(item)
|
||||
elif isinstance(item, ObservabilityEventGroup):
|
||||
groups[item.fingerprint] = item
|
||||
|
||||
async def get(self, model, key):
|
||||
if model is ObservabilityEventGroup:
|
||||
return groups.get(key)
|
||||
return None
|
||||
|
||||
async def commit(self):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(persistent_logs, "async_session_factory", lambda: FakeSession())
|
||||
|
||||
await persistent_logs.record_observability_event(
|
||||
source="earth-client",
|
||||
level="error",
|
||||
service="earth",
|
||||
module="tv",
|
||||
category="hls-proxy",
|
||||
event="hls.fragment.failed",
|
||||
message="HLS 分片加载失败: index_5_9086220.ts?m=1725933270",
|
||||
context={"status_code": 502},
|
||||
occurrence_count=2,
|
||||
)
|
||||
await persistent_logs.record_observability_event(
|
||||
source="earth-client",
|
||||
level="error",
|
||||
service="earth",
|
||||
module="tv",
|
||||
category="hls-proxy",
|
||||
event="hls.fragment.failed",
|
||||
message="HLS 分片加载失败: index_5_9086361.ts?m=1725934270",
|
||||
context={"status_code": 502},
|
||||
occurrence_count=1,
|
||||
)
|
||||
|
||||
assert len(events) == 2
|
||||
assert len(groups) == 1
|
||||
group = next(iter(groups.values()))
|
||||
assert group.count == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_emit_business_log_persists_sanitized_system_event(monkeypatch):
|
||||
events = []
|
||||
|
||||
@@ -2,8 +2,10 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from app.models.system_log import SystemLog
|
||||
from app.services import system_logs
|
||||
|
||||
|
||||
@@ -40,10 +42,10 @@ def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(mon
|
||||
{
|
||||
"earth-client": system_logs.LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
name="智能星球浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报日志",
|
||||
description="智能星球浏览器端上报日志",
|
||||
category="client",
|
||||
buffer_key=system_logs.get_buffer_log_key("earth-client"),
|
||||
)
|
||||
@@ -156,6 +158,52 @@ def test_append_buffer_log_persists_normalized_level(monkeypatch):
|
||||
assert payload["message"] == "feed delayed"
|
||||
|
||||
|
||||
def test_list_log_sources_includes_admin_client(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
|
||||
sources = system_logs.list_log_sources()
|
||||
|
||||
admin_source = next(item for item in sources if item["source_id"] == "admin-client")
|
||||
assert admin_source["kind"] == "buffer"
|
||||
assert admin_source["category"] == "client"
|
||||
|
||||
|
||||
def test_read_log_events_returns_stable_cursors(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"2026-04-22 08:00:00 INFO service booted",
|
||||
"2026-04-22 08:01:00 ERROR service failed",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
events = system_logs.read_log_events("backend", 50, level="error")
|
||||
|
||||
assert events is not None
|
||||
assert len(events) == 1
|
||||
assert events[0].source_id == "backend"
|
||||
assert events[0].cursor.startswith("backend:")
|
||||
assert events[0].line.endswith("ERROR service failed")
|
||||
|
||||
|
||||
def test_infer_log_level_prefers_leading_prefix_over_query_string():
|
||||
line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK'
|
||||
|
||||
@@ -215,3 +263,22 @@ def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monk
|
||||
"ERROR: bind failed",
|
||||
"2026-04-23 23:41:32 INFO service=backend message=request served",
|
||||
]
|
||||
|
||||
|
||||
def test_database_system_log_search_matches_context_key_value_aliases():
|
||||
record = SystemLog(
|
||||
id=2218,
|
||||
occurred_at=datetime(2026, 5, 28, 9, 14, 50, tzinfo=UTC),
|
||||
source="backend",
|
||||
service="collector",
|
||||
module="app.services.collectors.base",
|
||||
event="collector.run.failed",
|
||||
level="error",
|
||||
message="Collector run failed",
|
||||
context={"collector_name": "celestrak_tle", "datasource_id": 20, "task_id": 26906},
|
||||
)
|
||||
|
||||
event = system_logs._database_event_from_system_record(record)
|
||||
|
||||
assert system_logs.event_matches_search(event, "task_id=26906")
|
||||
assert system_logs.event_matches_search(event, "datasource_id=20")
|
||||
|
||||
35
backend/tests/test_tv_proxy.py
Normal file
35
backend/tests/test_tv_proxy.py
Normal file
@@ -0,0 +1,35 @@
|
||||
from app.api.v1.tv import _rewrite_hls_uri_attributes, _should_strip_hls_metadata_line
|
||||
|
||||
|
||||
def test_rewrite_hls_uri_attributes_rewrites_subtitle_manifest_url():
|
||||
line = '#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",URI="index_3_0.m3u8"'
|
||||
|
||||
rewritten = _rewrite_hls_uri_attributes(
|
||||
line,
|
||||
base_url="https://example.com/live/master.m3u8",
|
||||
)
|
||||
|
||||
assert 'URI="/api/v1/tv/proxy?url=https%3A%2F%2Fexample.com%2Flive%2Findex_3_0.m3u8"' in rewritten
|
||||
|
||||
|
||||
def test_rewrite_hls_uri_attributes_rewrites_absolute_uri():
|
||||
line = '#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=1234,URI="https://cdn.example.com/live/iframe.m3u8"'
|
||||
|
||||
rewritten = _rewrite_hls_uri_attributes(
|
||||
line,
|
||||
base_url="https://example.com/live/master.m3u8",
|
||||
)
|
||||
|
||||
assert 'URI="/api/v1/tv/proxy?url=https%3A%2F%2Fcdn.example.com%2Flive%2Fiframe.m3u8"' in rewritten
|
||||
|
||||
|
||||
def test_strip_hls_subtitle_media_metadata():
|
||||
line = '#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",URI="index_3_0.m3u8"'
|
||||
|
||||
assert _should_strip_hls_metadata_line(line) is True
|
||||
|
||||
|
||||
def test_keep_hls_audio_media_metadata():
|
||||
line = '#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="English",URI="audio.m3u8"'
|
||||
|
||||
assert _should_strip_hls_metadata_line(line) is False
|
||||
@@ -8,6 +8,72 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.69.0] — 2026-06-03
|
||||
|
||||
Released: 2026-06-03
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 新闻源治理能力,支持多 Feed 子项、源属性标签、新闻类型过滤、重要度规则和健康测试。
|
||||
- 新增观测日志聚合视图,按 fingerprint 汇总 Earth、Admin 和服务端重复运行时事件,并保留原始发生明细。
|
||||
- 改进 TV/HLS 播放恢复和代理重写,降低字幕、分片和源站波动导致的直播不可用噪声。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- Earth 新闻面板和 UE 端统一通过 `/api/v1/news/earth-feed` 使用 `categories` 与 `locale` 服务端过滤,Web 端新闻类型偏好仅保存在当前浏览器。
|
||||
- 控制台日志页新增重复统计、原始日志和审计日志模式,前端上报器会合并短窗口内的重复错误并提交 `occurrence_count`。
|
||||
- AI Provider / 服务端运行时可通过受保护的 observability ingest 入口写入结构化事件。
|
||||
- 新闻源文档新增中英文配置说明,并补齐 Earth 前端、控制台日志和公开 Docs 索引。
|
||||
|
||||
---
|
||||
|
||||
## [0.68.1] — 2026-05-28
|
||||
|
||||
Released: 2026-05-28
|
||||
|
||||
### Highlights
|
||||
- 修复 CelesTrak active 未更新窗口下清库后无法恢复的问题,fallback 会优先复用本地有效 group 缓存。
|
||||
- 修复数据源任务队列“查看日志”无法按 `task_id=...` 命中数据库结构化日志的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- CelesTrak fallback group 列表改为公开可用分组,移除失效 group,并在没有 active 缓存时仍可从本地 group 缓存恢复采集。
|
||||
- 数据库日志搜索补充 JSON context 的 `key=value` 别名,支持 `task_id=26906`、`datasource_id=20` 这类控制台跳转查询。
|
||||
- 补充 CelesTrak 缓存边界、数据源任务日志跳转和运维恢复说明的中英文文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.68.0] — 2026-05-28
|
||||
|
||||
Released: 2026-05-28
|
||||
|
||||
### Highlights
|
||||
- 新增数据源任务队列的实时指标校准和批量删除进度,让大表清理、取消和完成状态在控制台中可感知。
|
||||
- 新增智能星球 interactable 可插拔聚类策略,支持稳定 3D 球面聚类、动态屏幕聚类和 250% 以上自动散开。
|
||||
- 改进新设备启动流程,`planet.sh` 会在启动前同步前端依赖,避免缺失依赖导致控制台动态导入失败。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 数据源列表改为中文记录数指标,并对 AIS 大表使用统计估算 + 单条详情精确校准,降低首次加载成本。
|
||||
- 数据删除任务改为分批删除并广播进度,清理 AIS 衍生表后自动 `ANALYZE`,同时修复取消中任务恢复和状态文案。
|
||||
- Earth BGP、算力中心和 interactable 图层默认使用 `stable-spherical` 聚类,船舶实时层保留 `dynamic-screen`。
|
||||
- 新增中英文 Earth interactable clustering 文档,并补充采集队列、后端删除语义和前端依赖同步说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.67.0] — 2026-05-27
|
||||
|
||||
Released: 2026-05-27
|
||||
|
||||
### Highlights
|
||||
- 新增控制台日志实时跟随与前端运行时错误上报,帮助在控制台内直接排查 Admin / Earth 客户端异常。
|
||||
- 重构智能星球 Interactable 聚合和 wheel 缩放输入,保持真实地理锚点稳定,同时让鼠标滚轮和触控板拥有各自合适的缩放手感。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `/ws` 日志 tail 通道、数据库/文件日志统一事件读取,以及 Admin `error` / `unhandledrejection` / React ErrorBoundary 上报链路。
|
||||
- 优化控制台日志页状态颜色、跟随体验和运行时错误展示,并将 Admin 本地工具模块从 `lib` 命名迁移为局部 `utils`。
|
||||
- 修复智能星球国界/高清材质壳半径对齐、Interactable cluster 圆点显示、缩放目标累积和触控板连续缩放问题。
|
||||
- 更新 `planet.sh` 与 `.gitignore`,避免新环境构建污染锁文件并移除前端 `lib` 目录特殊放行。
|
||||
- 补充智能星球前端、渲染层级、控制台状态与运行时日志文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.66.3] — 2026-05-26
|
||||
|
||||
Released: 2026-05-26
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## 背景
|
||||
|
||||
状态:Phase 1 已经开始落地,Phase 2 的 BGP 事件 / 观测站迁移和 Phase 3 的算力中心迁移也已完成。`frontend/public/earth/js/interactable.js` 已新增,AIS 船只、BGP 事件、BGP 观测站和算力中心图层已经改为通过 `createInteractableLayer()` 使用通用批量 `Points`、hover / locked overlay、默认 glow、状态更新、asset icon 预加载、屏幕空间 picking、固定 / 距离缩放和跨 Interactable 同坐标避让。登陆点因 `THREE.Points` 边缘深度裁切和贴地层级要求,已退回专用 `THREE.Sprite` 黄色球路径,并与海缆同高度同 renderOrder。后续阶段聚焦把可复用的扩圈 / 雷达扇形动画正式沉淀成 `animations` 扩展。
|
||||
状态:Phase 1 已经开始落地,Phase 2 的 BGP 事件 / 观测站迁移和 Phase 3 的算力中心迁移也已完成。`frontend/public/earth/js/interactable.js` 已新增,AIS 船只、BGP 事件、BGP 观测站和算力中心图层已经改为通过 `createInteractableLayer()` 使用通用批量 `Points`、hover / locked overlay、默认 glow、状态更新、asset icon 预加载、屏幕空间 picking、固定 / 距离缩放和跨 Interactable 同坐标关系元数据。登陆点因 `THREE.Points` 边缘深度裁切和贴地层级要求,已退回专用 `THREE.Sprite` 黄色球路径,并与海缆同高度同 renderOrder。后续阶段聚焦把可复用的扩圈 / 雷达扇形动画正式沉淀成 `animations` 扩展。
|
||||
|
||||
当前实现说明和接入示例见:
|
||||
|
||||
@@ -100,9 +100,10 @@ createInteractableLayer({
|
||||
| `picking.throttleMs` | `100` | `80` | hover picking 节流。 |
|
||||
| `picking.skipWhileDragging` | `true` | `true` | 拖动和惯性期间跳过 hover picking。 |
|
||||
| `zIndexPolicy` | `"surface-icon"` | `"surface-icon"` | 预设层级策略,避免每个业务图层手写高度和 renderOrder。 |
|
||||
| `avoidance.enabled` | `true / false` | `true` | 是否参与跨 Interactable 的同坐标避让。默认开启,同一经纬度下的图标会沿地表切平面小幅排开,方便辨认和选择。 |
|
||||
| `avoidance.radius` | `number` | `1.1` | 同坐标避让的第一圈半径,单位为地球本地坐标单位。 |
|
||||
| `avoidance.enabled` | `true / false` | `true` | 是否记录跨 Interactable 的同坐标关系。该配置只生成 overlap 元数据,不允许移动真实 marker 坐标。 |
|
||||
| `avoidance.precision` | `number` | `4` | 经纬度归并精度,默认约等于只处理几乎完全重叠的图标。 |
|
||||
| `cluster.enabled` | `true / false` | 跟随 avoidance | 是否参与跨 Interactable 的当前帧屏幕重叠合并。只影响显示,不改变业务坐标。 |
|
||||
| `cluster.maxMarkersPerDot` | `number` | `14` | 单个聚合圆点的对象上限;超过后按屏幕局部邻近关系拆成多个较小圆点,避免一个点过大或跨区域串联。 |
|
||||
| `legend` | `{ label, color, shape }[]` | `[]` | 可选图例声明,业务层也可以继续自己导出。 |
|
||||
| `metadata` | object | `{}` | 业务扩展数据,不参与渲染但参与 tooltip / info-card / search。 |
|
||||
|
||||
|
||||
@@ -21,7 +21,9 @@ This is the current Intelligent Planet documentation entry point. Docs are organ
|
||||
- [Earth Satellite Footprint Policy](/home/ray/dev/linkong/planet/docs/technical/en/earth-satellite-footprint-policy.md): satellite footprint display boundaries and strategy
|
||||
- [BGP Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-bgp-context.md): BGP rendering, aggregation, and collector implementation in Earth
|
||||
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): `Interactable` API, lifecycle, and integration examples
|
||||
- [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md): pluggable cluster strategies, stable spherical clustering, and dynamic screen clustering boundaries
|
||||
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): close matrix for toolbar buttons, search, settings, news, and layer overlays
|
||||
- [Intelligent Planet News Source Configuration](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-sources.md): default sources, feed children, source property tags, content categories, importance rules, and configuration APIs
|
||||
|
||||
## Frontend Implementation
|
||||
|
||||
|
||||
@@ -75,6 +75,10 @@ async def run(self, db):
|
||||
|
||||
Manual trigger, data clearing, and cache clearing now enter the PostgreSQL data job queue. `collection_tasks` remains the task ledger. Collectors only own `fetch -> transform -> save`; the `data_jobs.py` worker claims `collect` / `clear_data` / `clear_cache` / `earth_refresh` jobs and writes progress back. Earth layer refresh relationships live in `earth_layer_adapters.py`; do not hand-code cache invalidation or WebSocket broadcasts inside individual collectors or buttons.
|
||||
|
||||
Data deletion runs in batches so AIS-scale tables are not locked by one huge statement. A `clear_data` job clears `collected_data`, then source-specific AIS derived tables, and broadcasts `records_processed` as it goes; the console queue renders only user-facing text such as `Deleting data` and `Delete complete`, while internal table names remain in logs and raw task details. After AIS cleanup, the backend runs `ANALYZE ais_raw_observations` so datasource-list estimates converge quickly. The datasource directory uses PostgreSQL statistics for AIS record counts by default to avoid a cold-start `count(*)`; opening a single datasource detail row requests the exact count for that source.
|
||||
|
||||
The CelesTrak TLE collector prefers the complete `active` catalog. If CelesTrak returns the "GP data has not updated" HTTP 403, the backend first reuses the active raw download cache under `$PLANET_CACHE_DIR/downloads/celestrak`; if that cache is missing, it enters fallback group mode. Fallback group mode uses only currently valid public CelesTrak groups, including `starlink`, `gps-ops`, `galileo`, `glo-ops`, `beidou`, `geo`, `iridium-next`, `stations`, `visual`, `weather`, `science`, `cubesat`, `amateur`, and `last-30-days`. Disaster-recovery fallback prefers valid local group caches before touching the network, so small-group update windows or unreliable HEAD metadata do not incorrectly fail recovery. Console `Clear Data` and `Clear Cache` jobs only touch database rows, Earth layer cache, and dashboard cache; they do not remove this raw download cache.
|
||||
|
||||
## III. Collector List
|
||||
|
||||
| Collector | Data type | Content | Frequency |
|
||||
|
||||
@@ -75,6 +75,8 @@ This is currently the most critical UI control entry point for the Earth fronten
|
||||
|
||||
Earth settings are now grouped by `data-settings-tab` and `data-settings-tab-panel`. Desktop and mobile share the same category semantics: Runtime, Display, Panels, Motion, Shortcuts, and System. When adding a setting, first choose its category, then add the DOM, persistence field, and restore logic; do not keep growing one long undifferentiated panel.
|
||||
|
||||
The news category selector in Display reuses the same chip-selector pattern as Cruise Modules. It only filters news categories for the current browser on the Earth frontend. It does not toggle layers, basemap, boundaries, TV, data points, BGP, vessels, satellites, or compute centers; those remain owned by the layer panel, media panel, and admin configuration. `controls.js` persists only `shared.newsCategoryFilters` and broadcasts `earth:news-category-filters-change`; `news.js` sends the selected categories to `/api/v1/news/earth-feed?categories=...&locale=zh-CN`, so Web and UE clients share the same backend category filtering path.
|
||||
|
||||
Shortcut configuration is a device-local preference owned by `controls.js`: read, capture, enable/disable, and reset all stay in the Earth frontend. It should not be written to backend user settings and should not affect other browsers. New shortcuts must provide a default key, display label, disabled/enabled state, and reset path instead of being hard-coded only in a keydown handler.
|
||||
|
||||
### 4. UI and Status Messages
|
||||
@@ -130,7 +132,7 @@ Responsibilities:
|
||||
- Terrain tile fetch, decode, displacement, and shading
|
||||
- Whole-globe land/ocean and border base overlays
|
||||
|
||||
The Earth surface is a stack of near-concentric shells, not a single mesh. The base sphere and HD texture overlay in `earth.js`, plus the land/ocean base in `country-boundaries.js`, need explicit radius separation. At far zoom, GPU depth precision drops; neighboring shells that are too close can z-fight and show black flicker blocks or snow. The current stable spacing is `landAltitudeOffset = 0.32` and `textureOverlayAltitudeOffset = 0.48`. When adding or adjusting whole-globe surface overlays, update [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md) and verify at 50% zoom.
|
||||
The Earth surface is a stack of near-concentric shells, not a single mesh. The base sphere and HD texture overlay in `earth.js`, plus the land/ocean base in `country-boundaries.js`, need explicit radius separation. At far zoom, GPU depth precision drops; neighboring shells that are too close can z-fight and show black flicker blocks or snow. The current stable spacing is `landAltitudeOffset = 0.32` and `EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` for the HD texture shell. Country borders, coastlines, claim lines, and country hover lines must use that same HD texture shell radius so they do not drift relative to the texture while the globe rotates. When adding or adjusting whole-globe surface overlays, update [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md) and verify at 50% zoom.
|
||||
|
||||
### 7. Layer Modules
|
||||
|
||||
@@ -313,6 +315,18 @@ If future cable, satellite, or news cruise is added, do not copy a new set of `m
|
||||
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
|
||||
- The business adapter pattern from [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
|
||||
|
||||
## View Control Feedback
|
||||
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) owns the Earth zoom state, and every zoom entry point must ultimately call `setZoomLevel()` to write the camera distance. Do not write `camera.position.z` from other modules, or the zoom percentage, drag sensitivity, and Interactable clustering thresholds will diverge again.
|
||||
|
||||
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps the projection-based behavior for high-frequency realtime layers such as vessels, and `none` disables clustering. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
|
||||
|
||||
Wheel input has two paths. Traditional mouse wheels keep the 10% step and short animation, using `wheelZoomTarget` as the logical base for continuous wheel input. Trackpads and high-precision wheels use the pixel delta for continuous zoom and call `setZoomLevel()` directly instead of passing through the 10% stepped animation. The trackpad path also filters a short-window, old-direction residual delta after a real direction change so inertia tails do not pull a just-reversed zoom back in the previous direction.
|
||||
|
||||
The gesture capsule updates at most once every 90ms and fades after 760ms. It is view feedback, not data loading progress, and should not be written into layer loading state.
|
||||
|
||||
[main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) only calls `setZoomLevel()` and `showZoomStatusCapsule()` for pinch zoom and motion zoom. Mouse wheel and zoom-button capsule feedback should stay in `controls.js` so the same zoom feedback does not spread across modules.
|
||||
|
||||
## Recommended Change Approach
|
||||
|
||||
For future Earth changes:
|
||||
|
||||
84
docs/technical/en/earth-interactable-clustering.md
Normal file
84
docs/technical/en/earth-interactable-clustering.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Intelligent Planet Interactable Clustering
|
||||
|
||||
Earth interactable icons are managed by `createInteractableLayer`. Rendering still uses Three.js `Points`, and picking, hover, locked state, tooltips, and cruise focus still depend on marker `userData`; the clustering strategy only decides which markers are represented by a cluster dot.
|
||||
|
||||
## Strategies
|
||||
|
||||
`cluster.strategy` supports three modes:
|
||||
|
||||
- `stable-spherical`: stable spherical clustering. It clusters by local 3D positions on the globe and caches topology by zoom band. Rotation and small zoom changes within the same band only update projection and size. Use it for semi-static layers such as BGP, compute centers, and Earth interactables.
|
||||
- `dynamic-screen`: dynamic screen-space clustering. This keeps the previous projection-based behavior and evaluates visible relationships per frame. Use it for high-frequency realtime layers such as vessels, or as a fallback.
|
||||
- `none`: no clustering. Every marker is shown independently. Use it for low-count layers or precision-first views.
|
||||
|
||||
`cluster: false` is equivalent to `strategy: "none"`. If a layer enables clustering without declaring a strategy, it keeps the compatible `dynamic-screen` behavior.
|
||||
|
||||
## Stable Spherical
|
||||
|
||||
`stable-spherical` moves cluster identity from screen distance to globe distance:
|
||||
|
||||
- Each marker uses `icon_base_position` as its geographic anchor.
|
||||
- By default, zoom levels above `2.5` force clustering off and show every marker as its original icon.
|
||||
- The current zoom maps to a discrete band; rotation and small zoom changes inside the same band do not recompute topology.
|
||||
- Clusters are recomputed only when the band, data revision, visibility, or filter state changes.
|
||||
- A cluster centroid is computed from member 3D positions and normalized back to the globe shell, so the cluster dot stays rigidly aligned to its geographic center.
|
||||
- Cluster dots do not participate in 2D avoidance, so screen-space repulsion cannot push them away from their real geographic projection.
|
||||
- Band changes use a small hysteresis margin so zooming at a boundary does not repeatedly bounce between two bands.
|
||||
- Newly created marker and cluster dots run a short scale + opacity ease. This is only a rendering transition; it does not change marker coordinates, picking objects, or locked state.
|
||||
|
||||
The stable strategy uses spherical bucket/hash neighbor lookup and must not use an all-pairs loop. Most frames only pay projection and material-size cost; topology cost is paid only on band or data changes.
|
||||
|
||||
## Configuration
|
||||
|
||||
```js
|
||||
const computeCenterIconLayer = createInteractableLayer({
|
||||
id: "computeCenters",
|
||||
// ...
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
minCount: 2,
|
||||
maxMarkersPerDot: 14,
|
||||
transitionMs: 220,
|
||||
bandHysteresis: 0.08,
|
||||
disableAboveZoom: 2.5,
|
||||
bands: [
|
||||
{ key: "far", maxZoom: 1.7, distance: 15 },
|
||||
{ key: "mid", maxZoom: 2.6, distance: 8 },
|
||||
{ key: "near", maxZoom: 3.5, distance: 4 },
|
||||
{ key: "detail", maxZoom: Infinity, distance: 0 },
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Realtime layers can stay dynamic:
|
||||
|
||||
```js
|
||||
const vesselIconLayer = createInteractableLayer({
|
||||
id: "vessels",
|
||||
// ...
|
||||
cluster: {
|
||||
strategy: "dynamic-screen",
|
||||
enabled: true,
|
||||
maxMarkersPerDot: 10,
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Tuning
|
||||
|
||||
- `distance` is the 3D globe-distance threshold and uses the same unit as `CONFIG.earthRadius`. Larger values cluster more aggressively.
|
||||
- The farthest band usually uses a larger `distance`; the nearest detail band usually uses `0` to split clusters into original icons.
|
||||
- `bandHysteresis` controls band-boundary stickiness. Too little can flicker near thresholds; too much can make band changes feel late.
|
||||
- `transitionMs` controls cluster split/merge easing. Keep it around 160-260ms; longer durations can make realtime layers feel sluggish.
|
||||
- `disableAboveZoom` controls the precision-view threshold. It defaults to `2.5`; above that zoom, no cluster dots are generated. Set it to `false` to disable the hard threshold.
|
||||
- High-frequency realtime layers should prefer `dynamic-screen` so frequent data changes do not trigger stable topology recomputation.
|
||||
- If a layer behaves poorly, switch it back to `dynamic-screen` or use `cluster: false`.
|
||||
|
||||
## Acceptance Checks
|
||||
|
||||
- Rotating the globe inside one zoom band should not cause clusters to flicker or regroup.
|
||||
- Cluster dots should stay aligned with the globe-surface centroid and should not be pushed by avoidance.
|
||||
- Zooming into the detail band should restore the layer's original icon textures and click behavior.
|
||||
- Zoom levels above 250% should not show cluster dots.
|
||||
- Realtime layers such as vessels should reflect incoming updates immediately.
|
||||
161
docs/technical/en/earth-news-sources.md
Normal file
161
docs/technical/en/earth-news-sources.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Earth News Source Configuration
|
||||
|
||||
Earth situational news is served through `/api/v1/news/earth-feed`. News source configuration lives in `SystemSetting.category = "earth_news_sources"`; when no database configuration exists, the backend uses the built-in default sources as the fallback seed.
|
||||
|
||||
## Default Sources
|
||||
|
||||
The default set contains four groups:
|
||||
|
||||
- **News feeds**: BBC World, DW Top Stories, CNBC Business, BBC Business, Guardian Business, NPR Business, MarketWatch, TechCrunch, Retail Dive, PR Newswire Retail, 36Kr, Ebrun, and China NBS data releases.
|
||||
- **Industry insight sources**: McKinsey Retail and Deloitte Retail.
|
||||
- **Official data sources**: China NBS data releases, US Census Retail / E-Commerce, MOFCOM Data, MOFCOM e-commerce updates, and China e-commerce logistics index.
|
||||
- **Lead sources**: BusinessWire Electronic Commerce; Google News is one aggregated source with feed children for global, Americas, Europe, Middle East / Africa, and Asia Pacific.
|
||||
|
||||
Config data sources are visible in Admin by default. If a source is not a stable RSS/Atom feed, it is kept disabled for automatic fetching until an administrator replaces it with a fetchable URL and enables it.
|
||||
|
||||
| Source | Type | Default state | Default category | Main tags | Purpose |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| BBC World | RSS | Enabled | Politics | `official_media`, `global` | Global public-news baseline |
|
||||
| DW Top Stories | RSS | Enabled | Politics | `official_media`, `europe` | Europe and international baseline |
|
||||
| CNBC Business | RSS | Enabled | Business | `business_news`, `us`, `global` | International business news |
|
||||
| BBC Business / Guardian Business / NPR Business / MarketWatch | RSS | Enabled | Business / Finance | `business_news`, `finance` | UK / US business and finance baseline |
|
||||
| TechCrunch / Retail Dive / PR Newswire Retail | RSS | Enabled | Technology / Business | `business_news`, `ecommerce`, `retail`, `press_release` | Technology, e-commerce, retail, and company announcements |
|
||||
| 36Kr | RSS | Enabled | Business | `business_news`, `ecommerce`, `china` | China business, venture, and newsflash feeds; homepage is `https://www.36kr.com/`, the Feed Directory is `https://www.36kr.com/rss-center`, and feed children are the general, article, newsflash, and moment feeds |
|
||||
| Ebrun | RSS | Enabled | E-commerce | `ecommerce`, `business_news`, `china`, `retail` | China e-commerce industry news; homepage is `https://www.ebrun.com/`, the Feed Directory is `https://www.ebrun.com/rss/`, and feed children are B2C, B2B, retail, O2O, service, data, and policy XML feeds |
|
||||
| China NBS data releases | RSS | Enabled | E-commerce | `official_data`, `ecommerce`, `retail`, `china` | Official data release RSS; retail and online retail items are identified by category and importance rules |
|
||||
| Google News | Aggregated | Enabled | Politics | `aggregated`, `low_stability` | One aggregated source with global, Americas, Europe, Middle East / Africa, and Asia Pacific feed children; lower priority than real RSS |
|
||||
| BusinessWire Electronic Commerce | Reference | Disabled | E-commerce | `press_release`, `ecommerce`, `low_stability` | Corporate announcement leads |
|
||||
| McKinsey Retail Insights | Reference | Disabled | Business | `industry_insight`, `retail` | Retail industry insight |
|
||||
| Deloitte Retail | Reference | Disabled | Business | `industry_insight`, `retail` | Retail industry insight |
|
||||
| US Census Retail / E-Commerce | Reference | Disabled | E-commerce | `official_data`, `ecommerce`, `retail`, `us` | US retail and e-commerce official data |
|
||||
| MOFCOM Data | Reference | Disabled | Business | `official_data`, `china` | China commerce data |
|
||||
| MOFCOM e-commerce updates | Reference | Disabled | E-commerce | `official_data`, `ecommerce`, `china` | China e-commerce policy and updates |
|
||||
| China e-commerce logistics index | Reference | Disabled | E-commerce | `official_data`, `ecommerce`, `logistics`, `china` | Logistics fulfillment and e-commerce activity |
|
||||
|
||||
`Reference` means a reference link or future collector lead. It records a homepage, report page, or data page and does not participate in RSS/Atom fetching. This lets commercial and official sources enter Admin governance without letting non-feed pages break the live news feed.
|
||||
|
||||
The news source model has two levels:
|
||||
|
||||
- `source` is the brand or aggregator, such as 36Kr, Ebrun, Google News, or BBC.
|
||||
- `homepage_url` is the source homepage, section page, or report page.
|
||||
- `feed_directory_url` is the Feed Directory page, such as an RSS subscription center or feed index. It is for human inspection and is not fetched.
|
||||
- `feeds` are the actual RSS, Atom, or Aggregated child entries under that source. Each feed child has `id / name / url / type / enabled / default_category / tags / priority`.
|
||||
|
||||
The backend iterates over every enabled feed child under the same source, fetches them independently, merges and deduplicates items, and writes per-feed diagnostics into `health.feed_results`. This is not a backup URL model: all four 36Kr subscription feeds, multiple Ebrun category XML feeds, and the five Google News regional RSS feeds can be enabled at the same time, and each feed can have its own default category and enabled state. HTML subscription-center pages belong in `feed_directory_url`, not in feed URLs. Every default enabled fetchable feed is tested item by item: RSS/Atom/Aggregated feeds must parse at least one item, while Reference sources only retain a reference URL and future collector lead.
|
||||
|
||||
Items that still remain Reference are not treated as broken feeds; no stable directly consumable RSS/Atom feed was verified:
|
||||
|
||||
- BusinessWire documents customizable RSS/Atom support, but the public pages do not expose a stable industry feed URL; the e-commerce industry page is kept as an announcement lead.
|
||||
- McKinsey and Deloitte retail insight pages are report/article collections, not public RSS feeds.
|
||||
- The US Census press-release RSS is reachable, but its items currently have empty links; the Quarterly E-Commerce page remains an official data reference.
|
||||
- MOFCOM data and China e-commerce logistics index pages do not expose stable RSS feeds yet; they should become dedicated collectors or be replaced with administrator-provided fetchable feeds.
|
||||
|
||||
## Source Property Tags and News Categories
|
||||
|
||||
News sources have `source_tags`, shown in Admin as source property tags. They describe the source, not the media name and not the content category of an individual story:
|
||||
|
||||
- `official_data`
|
||||
- `business_news`
|
||||
- `ecommerce`
|
||||
- `finance`
|
||||
- `retail`
|
||||
- `logistics`
|
||||
- `industry_insight`
|
||||
- `press_release`
|
||||
- `china`, `global`, `us`
|
||||
- `aggregated`, `low_stability`
|
||||
|
||||
Each news item has exactly one primary `category`. Defaults are politics, business, e-commerce, finance, sports, technology, military, disaster, energy, society, culture, and other. `item_tags` are item-level secondary tags, such as cross-border e-commerce, live commerce, retail data, logistics fulfillment, platform governance, AI, semiconductor, election, oil price, football, and supply chain.
|
||||
|
||||
The primary category is generated by a rule-based scorer over title, summary, and source text. If the rules do not match, the feed child default category is used first, then the source default category. AI enrichment does not block news display.
|
||||
|
||||
## Importance
|
||||
|
||||
Each item includes:
|
||||
|
||||
- `importance_score`
|
||||
- `importance_level`
|
||||
- `importance_reasons`
|
||||
- `market_impact`
|
||||
|
||||
Official data, e-commerce metrics, major platforms, and numeric business signals increase importance. Press releases start with a lower baseline and rise only when they match stronger platform, amount, M&A, or regulatory signals.
|
||||
|
||||
## Configuration and Cache
|
||||
|
||||
`GET /api/v1/earth/news-sources` returns the default or saved configuration. `PUT /api/v1/earth/news-sources` saves it, increments `cache_version`, and clears the process region cache. `POST /api/v1/earth/news-sources/reset` restores defaults. `POST /api/v1/earth/news-sources/test` tests one RSS/Atom/Aggregated source without writing news items.
|
||||
|
||||
## Feed Query and Category Filtering
|
||||
|
||||
The Web Earth client and UE client both consume `GET /api/v1/news/earth-feed`. The endpoint supports server-side filtering, so clients do not need to fetch the full list and apply the primary category filter locally.
|
||||
|
||||
- `lat` / `lon`: infer the active region from the current view, used by the Web Earth client.
|
||||
- `region`: explicitly select a region for UE or service integrations. Supported values include `global`, `americas`, `europe`, `asia-pacific`, and `middle-east-africa`. `global` is an aggregate view and can include every region; non-global regions include only their own region plus `global` sources.
|
||||
- `categories`: comma-separated news category keys, for example `business,ecommerce`. Omit it when all categories are selected.
|
||||
- `locale`: display locale, currently `zh-CN` or `en-US`, defaulting to `zh-CN`. Chinese RSS items are stored as Chinese source content and enriched with `en-US`; English RSS items are enriched with `zh-CN`.
|
||||
|
||||
Examples:
|
||||
|
||||
```http
|
||||
GET /api/v1/news/earth-feed?region=europe&categories=business,ecommerce
|
||||
GET /api/v1/news/earth-feed?lat=48&lon=10&categories=technology
|
||||
GET /api/v1/news/earth-feed?region=global&categories=business,ecommerce&locale=zh-CN
|
||||
```
|
||||
|
||||
Unknown category or locale values return `422` with the allowed values. The response includes `filters`, which confirms the region, category, and locale filters applied by the backend. `items` and `cruise_items` use the same category filter set.
|
||||
|
||||
The Web Earth category chips only store the current browser preference; changing them triggers a new API request. UE should pass its selected categories through the `categories` query parameter and does not need to perform the primary filtering itself.
|
||||
|
||||
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next `earth-feed` request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.
|
||||
|
||||
## Connectivity Monitoring
|
||||
|
||||
`POST /api/v1/earth/news-sources/test` tests one source and writes the result to `earth_news_sources.health[source_id]`. Normal RSS/Atom fetches update the same health map.
|
||||
|
||||
Health results include:
|
||||
|
||||
- `status`: `ok`, `empty`, `format_error`, `http_error`, `timeout`, `network_error`, or `reference`.
|
||||
- `status_code`, `content_type`, `item_count`, `latency_ms`, `error`, and `fetched_at`.
|
||||
- `feed_results`: per-feed diagnostics for multi-feed sources, including `feed_id`, `feed_name`, `feed_type`, `feed_url`, status, item count, and error.
|
||||
|
||||
Common diagnostics:
|
||||
|
||||
- HTML response: the configured URL is not an RSS/Atom feed, for example a web page listing RSS options.
|
||||
- HTTP 403: the source or CDN rejected the crawler request.
|
||||
- Reference: the source is a reference link only and must be converted to RSS, Atom, or Aggregated before fetch testing.
|
||||
|
||||
The Admin entry is `Earth Content -> News Sources`. It is not a raw whole-payload JSON editor. The UI has two layers:
|
||||
|
||||
- **News sources**: a left-side source list with filters for enabled, disabled, reference links, RSS/Atom/Aggregated, region, and source property tags; the right side edits one selected source and its feed child list.
|
||||
- **Policy rules**: global source property tags, news categories, item tag rules, and default health policy. Advanced JSON is reserved for diagnostics, not the default edit path.
|
||||
|
||||
The single-source form is split into source information and feed children:
|
||||
|
||||
- Source information covers name, ID, region, homepage URL, Feed Directory URL, source type, enabled state, source property tags, importance weight, fetch interval, timeout, failure threshold, and circuit breaker.
|
||||
- Feed children cover feed ID, name, real feed URL, type, enabled switch, default news category, priority, and feed tags. The `+` button under the feed child list creates a frontend-only draft; saving the source persists it, while canceling destroys the draft.
|
||||
|
||||
The per-source “test source” action tests all enabled feed children under the current source. The feed-row test action tests only that feed child. Both send to `/api/v1/earth/news-sources/test`, but the feed-row action submits the current source with only the selected feed child.
|
||||
|
||||
Reference links show that they only record a homepage, report page, or future collector lead and do not participate in RSS/Atom fetching. They can remain as commercial or official-data leads, but must be converted to RSS, Atom, or Aggregated with fetchable feed URLs before they can be enabled for fetching.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Admin["Admin: Earth Content / News Sources"] --> Source["Source config"]
|
||||
Source --> Feed["Feed children"]
|
||||
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
|
||||
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
|
||||
|
||||
Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||||
NewsAPI --> Resolver["Source Resolver"]
|
||||
Resolver --> Config
|
||||
Resolver --> Cache["Region Feed Cache"]
|
||||
Resolver --> Fetcher["RSS / Atom Fetcher"]
|
||||
Fetcher --> Parser["Feed Parser"]
|
||||
Parser --> Classifier["Classifier: category + item_tags + importance"]
|
||||
Classifier --> Store["earth_news_items"]
|
||||
Fetcher --> Health["source health"]
|
||||
Health --> Config
|
||||
Store --> EnrichQueue["Location / Localization Queue"]
|
||||
EnrichQueue --> AI["AI Provider"]
|
||||
Store --> NewsAPI
|
||||
UE["UE Client"] --> NewsAPI
|
||||
```
|
||||
@@ -18,14 +18,14 @@ Note: the layer control panel order and the registration / startup load order ar
|
||||
| 0 | Earth base sphere | `earth.js` | `CONFIG.earthRadius` | Surface picking fallback target | Dark base; still visible when all optional map layers are off. |
|
||||
| 0.2 | Country dark tint | `country-boundaries.js` | `tintAltitudeOffset` | Raycast disabled | Used when HD texture is off. |
|
||||
| 0.86 | Land/ocean base fill | `country-boundaries.js` | `landAltitudeOffset = 0.32`; ocean `#010609`, land `#080f1b` | Raycast disabled | Base map remains usable even when country borders are off; radius is separated from the base sphere to avoid far-zoom z-fighting. |
|
||||
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
|
||||
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
|
||||
| 1 | Atmospheric glow and clouds | `earth.js` | Atmosphere / cloud spheres | Not in normal object selection path | Cloud layer controlled by the "Cloud Layer" toggle. |
|
||||
| 1 | Submarine cables | `cables.js` | `CABLE_CONFIG.line.renderOrder` | Cable picking path | Preserves existing cable layer level. |
|
||||
| 1.2 | Real terrain | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` plus terrain displacement | Raycast disabled | Terrain overlays HD texture; temporarily hidden when HD texture is off, restores to prior state when re-enabled. |
|
||||
| 2.05 | Grid lines | `earth.js` | `CONFIG.earthRadius + 0.14` | Raycast disabled | Low-opacity lines over HD texture. |
|
||||
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset` | Raycast disabled | Only needs to stay above HD texture. |
|
||||
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
|
||||
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
|
||||
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset` | `depthTest: false`, raycast disabled | Neon red-orange hover line; China and Taiwan share the same highlight group. |
|
||||
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: false`, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
|
||||
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
|
||||
| 3-5 | BGP markers and overlays | `bgp.js` | Each marker's own renderOrder | BGP picking path | Preserves existing BGP visual level. |
|
||||
| 4.5 | Compute centers | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | Compute center picking path | Surface facilities, below satellites. |
|
||||
@@ -52,7 +52,8 @@ The Earth surface is not a single mesh. It is a stack of near-concentric shells:
|
||||
Maintenance rules:
|
||||
|
||||
- Do not reach first for hiding layers at far zoom. Check neighboring shell `altitudeOffset`, `renderOrder`, `depthTest`, and `depthWrite` first.
|
||||
- Whole-globe overlays such as the land/ocean base and HD texture must keep explicit separation from `CONFIG.earthRadius`; the current stable values are `landAltitudeOffset = 0.32` and `textureOverlayAltitudeOffset = 0.48`.
|
||||
- Whole-globe overlays such as the land/ocean base and HD texture must keep explicit separation from `CONFIG.earthRadius`; the current stable values are `landAltitudeOffset = 0.32` and `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`.
|
||||
- Country borders, coastlines, claim lines, and country hover lines must use `EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET`, exactly matching the HD texture shell. Do not give border lines an independent lower or higher offset, because globe rotation will make the lines appear to drift relative to the surface texture.
|
||||
- Any new whole-globe or near-whole-globe surface overlay must be screenshot-verified at 50% zoom and at common close zooms, with no black blocks, snow, flicker, or obvious floating.
|
||||
- If these radii change, update this document and the intent around the constants in `frontend/public/earth/js/constants.js`.
|
||||
|
||||
|
||||
@@ -92,6 +92,35 @@ The Admin sidebar theme switcher still reuses shared [SegmentedControl.tsx](/hom
|
||||
- Dark mode uses the same dark base, `#202938` slider, and dark external shadow semantics as Docs.
|
||||
- Product code only hides the text label and keeps icon + tooltip behavior; it should not recreate the private slider DOM.
|
||||
|
||||
## Admin Status Colors
|
||||
|
||||
Admin status labels should use [StatusText](/home/ray/dev/linkong/planet/frontend/src/admin/patterns/patterns.tsx) or [Badge](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/badge.tsx). Color variables come from [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css). New states should map to an existing tone instead of adding page-local hex colors.
|
||||
|
||||
`StatusText` is an indicator-light pill: the pill background and border stay on the component base color, while only the dot and text use the status color. `Badge` does not carry the indicator-light meaning, so it may use a light same-tone background and border for stronger hierarchy.
|
||||
|
||||
| Tone | Color variable | Meaning | Examples |
|
||||
| --- | --- | --- | --- |
|
||||
| `success` | `--an-success` | available, successful, connected, enabled | log source `Available`, collection `Success` |
|
||||
| `warning` | `--an-warning` | needs attention but is not necessarily failed | missing log file, degraded or skipped state |
|
||||
| `danger` | `--an-danger` | failed, unavailable, permission/connection error | `Docker unavailable`, endpoint failure |
|
||||
| `info` / `running` | `--an-info` | in progress, syncing, informational | `Following`, `Syncing` |
|
||||
| `neutral` | `--an-muted` | empty, disabled, not reported yet, unknown | `No reports yet`, disabled |
|
||||
| `ai` | fixed purple | AI-specific emphasis | AI generation or model action |
|
||||
|
||||
The log source list follows the same rule: `ok` is success; `empty` means the source exists but has not reported yet and is neutral; `missing` means an expected log file is not present and is warning; `docker_unavailable` / `source_unavailable` are danger.
|
||||
|
||||
## Admin Runtime Logs
|
||||
|
||||
Admin runtime errors are reported through [runtimeLogs.ts](/home/ray/dev/linkong/planet/frontend/src/admin/runtimeLogs.ts) to `/api/v1/system/logs/admin-client`. The reporter only runs on Admin routes and skips `/earth`, `/docs`, login, and registration pages so public-page noise does not enter the Admin client log source.
|
||||
|
||||
[AdminErrorBoundary.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/AdminErrorBoundary.tsx) catches React render failures and reuses the same reporter; global `error` and `unhandledrejection` events use that channel as well. Reporting failures must stay silent, because the logging path must not create another frontend error.
|
||||
|
||||
The Logs page follows log increments through the `/ws` `logs_tail` channel. File logs and database logs are both normalized into line events by the backend. When adding a new log source, wire it through the backend source registry and tail manager instead of adding a page-local poller.
|
||||
|
||||
The Logs page now opens in the grouped view by default. It reads `/api/v1/system/logs/observability/groups`, groups Earth, Admin, and service runtime reports by `fingerprint`, and then reads `/api/v1/system/logs/observability/groups/{fingerprint}/events` when an operator opens one group. Raw logs and audit logs remain separate views; only the raw-log view can follow WebSocket updates. Frontend reporters coalesce repeated errors in a short window and submit `occurrence_count`, while the backend writes both `system_logs` and `observability_events` / `observability_event_groups`, so the page should not add another browser-side aggregation pass over identical messages.
|
||||
|
||||
The datasource task queue `View Logs` action opens `/logs?source=system-db&search=task_id=<id>`. Backend database-log search indexes must expand simple JSON context fields into `key=value` aliases such as `task_id=26906` and `datasource_id=20`, so historical task logs remain discoverable without rerunning the task.
|
||||
|
||||
## Current Shared Components
|
||||
|
||||
### 1. `Scrollbar`
|
||||
|
||||
@@ -172,6 +172,10 @@ After selecting a task, the page shows the effective prompt, whether it is custo
|
||||
|
||||
The legacy link `/settings?tab=ai` redirects to `/ai?tab=providers`.
|
||||
|
||||
## Datasources and Task Logs
|
||||
|
||||
`/datasources` is the datasource directory. Built-in sources can be filtered by product domain, level, enabled state, latest run state, collected-data state, and keyword. With no rows selected the main action triggers all matching sources; selecting rows changes it to `Trigger Selected N`. The queue button opens a grouped task panel for running, completed, failed, and skipped work. Failed rows can be retried, completed rows can jump back to their datasource detail, and each task can open `/logs` filtered by its task id.
|
||||
|
||||
## System Settings
|
||||
|
||||
`/settings` manages system-level configuration. Sub-tabs:
|
||||
@@ -296,6 +300,8 @@ Adopt All is for batch processing the compute-center unresolved queue. It starts
|
||||
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
|
||||
News categories use the same chip selector as Cruise Modules. They only filter the news panel and news cruise items in the current browser; they do not affect layers, TV, data points, basemap, boundaries, collector jobs, or admin news-source configuration.
|
||||
|
||||
"Real Satellite Altitude" is enabled by default: satellite positions use a compressed display height based on TLE/SGP4 orbital altitude. LEO satellites remain close to the globe, while high-orbit satellites render farther out without leaving the normal view. The high-orbit display height is capped at about one quarter of the globe radius, so GEO / MEO objects remain visually separated from LEO without spreading trails and selection targets too far apart. Turning it off restores the legacy same-sphere satellite display. "Track Display" controls satellite trail visibility; trails are unavailable while the satellite layer is hidden.
|
||||
|
||||
"Hover Tooltip" controls the tooltip shown when the pointer hovers over the globe surface: `Country` shows country details only when land matches a country, and stays silent over oceans such as the Pacific; `Position` shows latitude, longitude, and elevation over land and ocean; `Full` is the default and shows country + position over land and position over ocean.
|
||||
|
||||
@@ -75,7 +75,7 @@ Cleanup order and boundaries:
|
||||
- Docker cleanup targets resources whose Compose project is `planet`, plus the explicit volumes `planet_postgres_data`, `planet_redis_data`, `postgres_data`, and `redis_data`; do not delete unlabeled volumes by a broad `planet_*` pattern, because another local project could own them.
|
||||
- Local build state removes `.venv`, frontend `node_modules` / `dist`, Planet state, and scattered Python / Vite cache directories. `$PLANET_CACHE_DIR/downloads` is preserved so upstream raw downloads such as CelesTrak can survive database resets and local rebuild cleanup.
|
||||
|
||||
After the reset, run `./planet.sh init` again to recreate tables and default seed data. Old collected records are not restored, and Earth OOBE is evaluated from the backend's real collection state on the next visit. When CelesTrak later returns its "GP data has not updated" HTTP 403, the backend first reuses the preserved download cache to repopulate the database; if no cache exists, wait for the next CelesTrak update window or use Space-Track as a fallback.
|
||||
After the reset, run `./planet.sh init` again to recreate tables and default seed data. Old collected records are not restored, and Earth OOBE is evaluated from the backend's real collection state on the next visit. When CelesTrak later returns its "GP data has not updated" HTTP 403, the backend first reuses the preserved download cache to repopulate the database; if no active cache exists, it tries valid CelesTrak fallback group caches; if no download cache exists at all, wait for the next CelesTrak update window or use Space-Track. Datasource `Clear Data` and `Clear Cache` actions in the console do not delete `$PLANET_CACHE_DIR/downloads/celestrak`.
|
||||
|
||||
## Health Check
|
||||
|
||||
@@ -286,6 +286,8 @@ bun run build
|
||||
|
||||
Do not use `npm run ...`. In the WSL / Windows mixed environment Bun avoids Node/npm path inconsistencies.
|
||||
|
||||
`./planet.sh start` / `init` now runs `bun install` before startup instead of only checking whether the Vite entry file exists. This keeps new devices, cleaned `node_modules`, and lockfile changes synchronized before the console loads, avoiding dynamic-import 500s caused by missing frontend dependencies.
|
||||
|
||||
Validate the frontend build:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -21,7 +21,9 @@
|
||||
- [智能星球卫星覆盖策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-satellite-footprint-policy.md):卫星 footprint 的显示边界和策略
|
||||
- [BGP 态势上下文](/home/ray/dev/linkong/planet/docs/technical/zh/earth-bgp-context.md):BGP 在智能星球中的渲染、聚合和观测站实现
|
||||
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md):`Interactable` 的接口、生命周期和接入示例
|
||||
- [智能星球可交互图标聚类策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-clustering.md):可插拔 cluster strategy、稳定球面聚类和动态屏幕聚类的适用边界
|
||||
- [智能星球工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索、设置、新闻、图层浮层的关闭矩阵
|
||||
- [智能星球新闻源配置](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-sources.md):默认新闻源、Feed 子项、源属性标签、内容类型、重要度规则和配置接口
|
||||
|
||||
## 前端技术实现
|
||||
|
||||
|
||||
@@ -75,6 +75,10 @@ async def run(self, db):
|
||||
|
||||
手动触发、删除数据、清理缓存现在统一进入 PostgreSQL 数据作业队列,任务账本仍是 `collection_tasks`。采集器只负责 `fetch -> transform -> save`,由 `data_jobs.py` worker 领取 `collect` / `clear_data` / `clear_cache` / `earth_refresh` 任务并回写进度。Earth 图层刷新关系集中在 `earth_layer_adapters.py`,不要再在单个采集器或按钮里手写缓存失效和 WebSocket 广播。
|
||||
|
||||
删除数据任务按批次执行,避免 AIS 这类千万级表一次性锁表。`clear_data` 会先清 `collected_data`,再按来源清理 AIS 衍生表,并持续广播 `records_processed`;前端任务队列只展示“正在删除数据 / 删除完成”,内部表名只保留在日志和原始任务详情。删除结束后后端会 `ANALYZE ais_raw_observations`,让数据源列表的估算指标尽快收敛。数据源目录页默认使用 PostgreSQL 统计信息估算 AIS 大表记录数,避免冷启动做 `count(*)`;打开单条详情时再用精确计数校准当前数据源。
|
||||
|
||||
CelesTrak TLE 采集优先拉取完整 `active` 目录。如果 CelesTrak 返回“本轮 GP 数据未更新”的 403,后端先复用 `$PLANET_CACHE_DIR/downloads/celestrak` 下的 active 原始下载缓存;没有 active 缓存时进入 fallback group 模式。fallback group 只使用 CelesTrak 当前公开有效的分组,例如 `starlink`、`gps-ops`、`galileo`、`glo-ops`、`beidou`、`geo`、`iridium-next`、`stations`、`visual`、`weather`、`science`、`cubesat`、`amateur` 和 `last-30-days`。救灾 fallback 会优先使用本地有效 group 缓存,避免 CelesTrak 小分组在未更新窗口或 HEAD 元数据异常时被误判失败。控制台的“删除数据库”和“清理缓存”只处理数据库记录、Earth layer cache 和 dashboard cache,不删除该原始下载缓存。
|
||||
|
||||
## 三、采集器列表
|
||||
|
||||
| 采集器 | 数据类型 | 数据内容 | 采集频率 |
|
||||
|
||||
@@ -75,6 +75,8 @@ Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,真实
|
||||
|
||||
Earth 设置面板现在按 `data-settings-tab` 和 `data-settings-tab-panel` 分类组织。桌面端和移动端使用同一组分类语义:运行、显示、面板、动捕、快捷键、系统。新增设置项时应先判断它属于哪个分类,再补 DOM、持久化字段和恢复逻辑;不要把所有控件继续堆到一个长面板里。
|
||||
|
||||
`显示` 分类里的新闻类型选择复用巡航模块的 chip 选择器形态,只控制星球端当前浏览器的新闻分类显示。它不会打开或关闭图层、底图、边界、TV、数据点、BGP、船舶、卫星或算力中心;这些仍由图层面板、媒体面板和控制台配置各自负责。`controls.js` 只持久化 `shared.newsCategoryFilters` 并广播 `earth:news-category-filters-change`,`news.js` 会把选中的类型拼到 `/api/v1/news/earth-feed?categories=...&locale=zh-CN`,让 Web 和 UE 走同一套后端类型过滤。
|
||||
|
||||
快捷键配置属于设备本地偏好,由 `controls.js` 负责读取、捕获、启用/禁用和重置。它不应写入后端用户设置,也不应影响其它浏览器。后续新增快捷键时,必须同时提供默认键、显示标签、可禁用状态和重置路径,避免只在 keydown handler 中硬编码。
|
||||
|
||||
### 4. UI 与状态消息
|
||||
@@ -138,7 +140,7 @@ Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/h
|
||||
- terrain tile 拉取、解码、位移、着色
|
||||
- 海陆基座与国界底图的整球 overlay
|
||||
|
||||
智能星球地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座球、高清材质 overlay、云层/大气,以及 `country-boundaries.js` 的海陆基座都需要明确半径间距。远距视图下 GPU 深度精度会下降,相邻 shell 过近会 z-fighting,表现为黑色闪烁块或雪花。当前稳定策略是让海陆基座使用 `landAltitudeOffset = 0.32`,高清材质使用 `textureOverlayAltitudeOffset = 0.48`;后续新增或调整整球地表 overlay 时,必须同步检查 [智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md),并在 50% 缩放视图验证。
|
||||
智能星球地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座球、高清材质 overlay、云层/大气,以及 `country-boundaries.js` 的海陆基座都需要明确半径间距。远距视图下 GPU 深度精度会下降,相邻 shell 过近会 z-fighting,表现为黑色闪烁块或雪花。当前稳定策略是让海陆基座使用 `landAltitudeOffset = 0.32`,高清材质使用 `EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`;国界线、coastline、claim 线和国界 hover 线也必须使用同一个高清材质壳半径,避免转动地球时相对高清贴图产生视差漂浮感。后续新增或调整整球地表 overlay 时,必须同步检查 [智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md),并在 50% 缩放视图验证。
|
||||
|
||||
### 7. 图层模块
|
||||
|
||||
@@ -381,23 +383,24 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
|
||||
|
||||
`Interactable` 不再把图标本体额外抬离业务高度。`altitudeOffset` 就是 marker、hover glow、locked glow 和 picking 共同使用的地表高度;这样船只图标会继续贴着船只轨迹线,不会因为单独抬高显示位置而显得漂浮。后续如果要解决边缘 glow 裁切,应优先考虑 glow 纹理、overlay 尺寸或图层专属特效,而不是把通用图标层整体抬高。
|
||||
|
||||
跨 Interactable 的同坐标避让也在公共层处理。每个 marker 会保留 `icon_base_position` 作为业务原始位置;当多个 Interactable marker 归入同一个经纬度 key 时,公共层会把它们沿地表切平面排成小圈,并刷新已创建的 `THREE.Points` geometry。这样视觉位置和屏幕空间 picking 位置一致,不需要业务层再单独判断“算力中心和 BGP 事件重叠”这类场景。
|
||||
跨 Interactable 的同坐标关系也在公共层记录,但真实位置必须始终以 `icon_base_position` 为准。缩放、避让、聚合和后续 spiderfy 展开都只能改变屏幕表现,不能写回 `marker.position` 或 `THREE.Points` 里的业务锚点;巡航定位、详情卡、搜索定位和 picking 返回对象都必须落回真实经纬度。多个图标归入同一个经纬度 key 时,公共层只写 `icon_avoidance_*` 元数据,供业务层弱化 halo 或显示聚合提示;真正的低缩放聚合应通过独立 cluster glyph / screen layout 层实现,而不是把对象沿地表切平面挪开。
|
||||
|
||||
`Interactable` 的单点显示只由全局地图缩放决定:170% 及以下强制显示小圆点,超过 170% 显示原图标。cluster 现在由 `cluster.strategy` 决定:`stable-spherical` 使用离散 zoom band 和 3D 球面分桶,BGP、算力中心和 Earth interactable 在同一 band 内旋转或细微缩放时不会重新计算聚合拓扑;`dynamic-screen` 保留屏幕空间聚类,适合船只这类实时高频图层;`none` 关闭聚类。稳定球面聚类的 cluster 圆点刚性落在成员 3D 质心投影上,不参与 2D 避让,避免缩放时被推离真实地理位置。cluster 圆点大小随包含对象数量增长,数量过多时按稳定地理顺序拆成多个较小圆点;数量默认只在 hover tooltip 中显示。这个过程只设置 `icon_cluster_*` 展示元数据和重建渲染 Points,不改变每个 marker 的真实经纬度。
|
||||
|
||||
接口细节、生命周期和接入示例见:
|
||||
|
||||
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
|
||||
- [智能星球可交互图标聚类策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-clustering.md)
|
||||
|
||||
### 视角控制反馈
|
||||
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一维护 Earth 缩放状态。滚轮缩放、缩放按钮和触屏双指捏合最终都会更新 `zoomLevel`,并通过 `showZoomStatusCapsule()` 显示当前缩放比例:
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一维护智能星球缩放状态,所有入口最终都必须调用 `setZoomLevel()` 写相机距离。不要在其他模块直接写 `camera.position.z`,否则缩放百分比、拖拽灵敏度和 Interactable 聚合阈值会再次分叉。
|
||||
|
||||
```javascript
|
||||
showGestureStatusMessage(`缩放 ${Math.round(zoomLevel * 100)}%`, "info");
|
||||
```
|
||||
滚轮输入分两类处理:传统鼠标滚轮保留 10% 档位和短动画,并用 `wheelZoomTarget` 作为连续滚动的逻辑基准;触控板 / 高精度滚轮按 pixel delta 连续缩放,直接调用 `setZoomLevel()`,不走 10% 档位动画。触控板路径还会过滤反向后的短窗口旧方向小 residual delta,避免惯性尾巴把用户刚反向的缩放又拉回旧方向。
|
||||
|
||||
该提示每 90ms 最多更新一次,显示 760ms 后淡出。它是视角反馈,不是数据加载进度,也不应该写进图层 loading 状态。
|
||||
|
||||
[main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 只负责在双指捏合缩放时调用 `setZoomLevel()` 和 `showZoomStatusCapsule()`。鼠标滚轮与缩放按钮的胶囊提示应继续放在 `controls.js`,避免同一种缩放反馈散落在多个模块。
|
||||
[main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 只负责在双指捏合缩放和动捕缩放时调用 `setZoomLevel()` 和 `showZoomStatusCapsule()`。鼠标滚轮与缩放按钮的胶囊提示应继续放在 `controls.js`,避免同一种缩放反馈散落在多个模块。
|
||||
|
||||
拖拽地球的旋转灵敏度会根据当前缩放连续衰减,而不是按某个缩放阈值分段:
|
||||
|
||||
|
||||
84
docs/technical/zh/earth-interactable-clustering.md
Normal file
84
docs/technical/zh/earth-interactable-clustering.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# 智能星球可交互图标聚类策略
|
||||
|
||||
智能星球的可交互图标由 `createInteractableLayer` 统一管理。图标仍然使用 Three.js `Points` 渲染,拾取、hover、locked、tooltip 和巡航焦点继续依赖 marker 的 `userData`;聚类策略只决定“哪些 marker 被合成一个 cluster dot”。
|
||||
|
||||
## 策略
|
||||
|
||||
`cluster.strategy` 支持三种模式:
|
||||
|
||||
- `stable-spherical`:稳定球面聚类。按地球局部 3D 坐标聚合,并用 zoom band 缓存拓扑;旋转和同档缩放只更新投影和尺寸,不重算谁和谁聚在一起。适合 BGP、超算/GPU 中心和 Earth interactable 这类半静态图层。
|
||||
- `dynamic-screen`:动态屏幕聚类。保留原来的屏幕空间聚类逻辑,每帧按可见投影关系判断。适合船舶等高频实时图层,也可作为稳定策略的回退。
|
||||
- `none`:不聚类。所有 marker 独立显示,适合低数量或需要精确展示的图层。
|
||||
|
||||
`cluster: false` 等价于 `strategy: "none"`。未显式声明 `strategy` 时,保持兼容行为:开启聚类的旧图层继续走 `dynamic-screen`。
|
||||
|
||||
## Stable Spherical
|
||||
|
||||
`stable-spherical` 的核心是把聚类身份从屏幕距离迁到球面距离:
|
||||
|
||||
- 每个 marker 使用 `icon_base_position` 作为真实地理锚点。
|
||||
- 默认 zoom 超过 `2.5` 时强制关闭聚类,所有 marker 展开为原图标。
|
||||
- 当前 zoom 只映射到离散 band;同一 band 内旋转地球或细微缩放不会重算拓扑。
|
||||
- 跨 band、数据变更、图层显隐变化时才重新计算 cluster。
|
||||
- cluster 质心由成员 3D 坐标平均后 normalize 回球壳半径,因此 cluster dot 刚性贴在地理质心投影上。
|
||||
- cluster dot 不参与 2D 避让,避免被屏幕排斥推离真实地理位置。
|
||||
- band 切换带有少量 hysteresis,避免缩放停在临界点时在两个 band 之间来回跳。
|
||||
- 新生成的 marker / cluster dot 会执行短 scale + opacity ease;这只是渲染过渡,不改变 marker 的真实经纬度、拾取对象或 locked 状态。
|
||||
|
||||
稳定策略使用球面 bucket/hash 邻域查询,禁止用全量双循环。这样大多数帧只承担投影与材质尺寸更新,聚类成本只在 band 或数据版本变化时支付。
|
||||
|
||||
## 配置示例
|
||||
|
||||
```js
|
||||
const computeCenterIconLayer = createInteractableLayer({
|
||||
id: "computeCenters",
|
||||
// ...
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
minCount: 2,
|
||||
maxMarkersPerDot: 14,
|
||||
transitionMs: 220,
|
||||
bandHysteresis: 0.08,
|
||||
disableAboveZoom: 2.5,
|
||||
bands: [
|
||||
{ key: "far", maxZoom: 1.7, distance: 15 },
|
||||
{ key: "mid", maxZoom: 2.6, distance: 8 },
|
||||
{ key: "near", maxZoom: 3.5, distance: 4 },
|
||||
{ key: "detail", maxZoom: Infinity, distance: 0 },
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
实时层可以保留动态策略:
|
||||
|
||||
```js
|
||||
const vesselIconLayer = createInteractableLayer({
|
||||
id: "vessels",
|
||||
// ...
|
||||
cluster: {
|
||||
strategy: "dynamic-screen",
|
||||
enabled: true,
|
||||
maxMarkersPerDot: 10,
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## 调参原则
|
||||
|
||||
- `distance` 是球面 3D 距离阈值,单位与 `CONFIG.earthRadius` 一致。值越大,越容易聚合。
|
||||
- 最远 band 用较大的 `distance` 降低视觉密度;最近 band 通常设为 `0`,让图标完全解散。
|
||||
- `bandHysteresis` 控制 band 边界滞回。值太小容易临界闪烁,值太大会让切换略显迟钝。
|
||||
- `transitionMs` 控制聚散过渡时间。建议保持在 160-260ms,过长会让实时层显得拖泥带水。
|
||||
- `disableAboveZoom` 控制精细查看阈值。默认 `2.5`,超过后不再生成 cluster;设为 `false` 可关闭这个硬阈值。
|
||||
- 高实时性图层优先用 `dynamic-screen`,避免数据频繁变更时触发稳定策略的拓扑重算。
|
||||
- 若某图层出现异常,可临时切回 `dynamic-screen` 或 `cluster: false`。
|
||||
|
||||
## 验收重点
|
||||
|
||||
- 同一 zoom band 内旋转地球,cluster 不应闪烁或重新聚散。
|
||||
- cluster dot 应跟随地球表面质心,不被避让逻辑推开。
|
||||
- 放大到 detail band 后,应恢复该图层原本的图标纹理和点击行为。
|
||||
- zoom 超过 250% 后不应再显示 cluster dot。
|
||||
- vessel 等实时层更新后,图标和 cluster 应立即反映最新数据。
|
||||
161
docs/technical/zh/earth-news-sources.md
Normal file
161
docs/technical/zh/earth-news-sources.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Earth 新闻源配置
|
||||
|
||||
Earth 态势新闻使用 `/api/v1/news/earth-feed` 输出给前端。新闻源配置存放在 `SystemSetting.category = "earth_news_sources"`;没有数据库配置时,后端使用内置默认源作为 fallback seed。
|
||||
|
||||
## 默认源
|
||||
|
||||
默认源包含四类:
|
||||
|
||||
- **新闻源**:BBC World、DW Top Stories、CNBC Business、BBC Business、Guardian Business、NPR Business、MarketWatch、TechCrunch、Retail Dive、PR Newswire Retail、36氪、亿邦动力、国家统计局数据发布。
|
||||
- **行业洞察源**:McKinsey Retail、Deloitte Retail。
|
||||
- **官方数据源**:国家统计局数据发布、US Census Retail / E-Commerce、商务数据中心、商务部电商动态、电商物流指数。
|
||||
- **线索源**:BusinessWire Electronic Commerce;Google News 作为一个聚合 source,下面挂 global / americas / europe / middle-east-africa / asia-pacific 五个区域 Feed 子项。
|
||||
|
||||
配置型数据源默认保留在 Admin 配置中,但若不是稳定 RSS/Atom,则默认不参与自动抓取。管理员可以在 `Earth 内容 -> 新闻源` 中改成可抓取 RSS、启用或禁用。
|
||||
|
||||
| 来源 | 类型 | 默认状态 | 默认主类型 | 主要标签 | 用途 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| BBC World | RSS | 启用 | 政治 | `official_media`, `global` | 全球公共新闻基线 |
|
||||
| DW Top Stories | RSS | 启用 | 政治 | `official_media`, `europe` | 欧洲与国际新闻基线 |
|
||||
| CNBC Business | RSS | 启用 | 商业 | `business_news`, `us`, `global` | 国际商业新闻 |
|
||||
| BBC Business / Guardian Business / NPR Business / MarketWatch | RSS | 启用 | 商业 / 金融 | `business_news`, `finance` | 英美商业与金融基线 |
|
||||
| TechCrunch / Retail Dive / PR Newswire Retail | RSS | 启用 | 科技 / 商业 | `business_news`, `ecommerce`, `retail`, `press_release` | 科技、电商、零售和企业公告 |
|
||||
| 36氪 | RSS | 启用 | 商业 | `business_news`, `ecommerce`, `china` | 国内商业、创投和快讯;主页是 `https://www.36kr.com/`,Feed 信息页是 `https://www.36kr.com/rss-center`,Feed 子项是综合资讯、文章资讯、最新快讯、动态内容 |
|
||||
| 亿邦动力 | RSS | 启用 | 电商 | `ecommerce`, `business_news`, `china`, `retail` | 国内电商行业新闻;主页是 `https://www.ebrun.com/`,Feed 信息页是 `https://www.ebrun.com/rss/`,Feed 子项是 B2C、B2B、零售、O2O、服务、数据、政策 XML |
|
||||
| 国家统计局数据发布 | RSS | 启用 | 电商 | `official_data`, `ecommerce`, `retail`, `china` | 官方数据发布 RSS;社零和网上零售条目由分类/重要度规则识别 |
|
||||
| Google News | Aggregated | 启用 | 政治 | `aggregated`, `low_stability` | 一个聚合 source,Feed 子项为全球、美洲、欧洲、中东与非洲、亚太区域兜底 RSS;优先级低于真实 RSS |
|
||||
| BusinessWire Electronic Commerce | Reference | 禁用 | 电商 | `press_release`, `ecommerce`, `low_stability` | 企业公告线索 |
|
||||
| McKinsey Retail Insights | Reference | 禁用 | 商业 | `industry_insight`, `retail` | 零售行业洞察 |
|
||||
| Deloitte Retail | Reference | 禁用 | 商业 | `industry_insight`, `retail` | 零售行业洞察 |
|
||||
| US Census Retail / E-Commerce | Reference | 禁用 | 电商 | `official_data`, `ecommerce`, `retail`, `us` | 美国零售和电商官方数据 |
|
||||
| 商务数据中心 | Reference | 禁用 | 商业 | `official_data`, `china` | 国内商务数据 |
|
||||
| 商务部电商动态 | Reference | 禁用 | 电商 | `official_data`, `ecommerce`, `china` | 国内电商政策与动态 |
|
||||
| 电商物流指数 | Reference | 禁用 | 电商 | `official_data`, `ecommerce`, `logistics`, `china` | 物流履约与电商景气度 |
|
||||
|
||||
`Reference` 源表示参考链接/未来采集器线索,只记录官网、报告页或数据页,不参与 RSS/Atom 抓取。这样可以把商业与官方数据源先纳入后台治理,同时避免不可抓取页面拖垮新闻 feed。
|
||||
|
||||
新闻源模型是两层结构:
|
||||
|
||||
- `source` 表示来源品牌或聚合器,例如 36氪、亿邦动力、Google News、BBC。
|
||||
- `homepage_url` 表示来源官网、栏目页或报告页。
|
||||
- `feed_directory_url` 表示 Feed 信息页,也就是 RSS 订阅中心或 Feed 聚合页,只用于人工查看,不参与抓取。
|
||||
- `feeds` 表示该来源下真正抓取的 RSS、Atom 或 Aggregated 子项。每个 Feed 子项都有 `id / name / url / type / enabled / default_category / tags / priority`。
|
||||
|
||||
后端会遍历同一 source 下所有启用的 Feed 子项,逐个抓取、合并去重,并把单个子项的检测结果写入 `health.feed_results`。这不是“备用地址”逻辑;36氪的四个订阅地址、亿邦的多个分类 XML、Google News 的五个区域 RSS 都可以同时启用,并且每个 Feed 可以单独配置默认新闻类型和启用状态。HTML 订阅中心或聚合页只能放在 `feed_directory_url`,不能放进 Feed 地址。默认启用的可抓 Feed 已逐项连通性检测:RSS/Atom/Aggregated Feed 必须解析到条目,Reference 源只保留参考地址和后续采集器线索。
|
||||
|
||||
当前仍保留为 Reference 的项不是“坏源”,而是没有找到稳定、可直接消费的 RSS/Atom:
|
||||
|
||||
- BusinessWire 官方说明支持可定制 RSS/Atom,但公开页面未暴露稳定行业 feed URL;当前保留电子商务行业页作为公告线索。
|
||||
- McKinsey / Deloitte 的零售洞察页是报告和文章集合,不是公开 RSS。
|
||||
- US Census 的 press release RSS 可访问,但条目链接为空;Quarterly E-Commerce 页面保留为官方数据参考链接。
|
||||
- 商务部数据、电商物流指数目前未找到稳定 RSS,后续应做专用 collector 或人工配置可抓 feed。
|
||||
|
||||
## 源属性标签与新闻类型
|
||||
|
||||
新闻源有 `source_tags`,在 Admin 中显示为“源属性标签”。它用于描述 source 的属性,不是媒体来源名,也不是新闻条目的内容类型。例如:
|
||||
|
||||
- `official_data`:官方数据
|
||||
- `business_news`:商业新闻
|
||||
- `ecommerce`:电商
|
||||
- `finance`:金融
|
||||
- `retail`:零售
|
||||
- `logistics`:物流
|
||||
- `industry_insight`:行业洞察
|
||||
- `press_release`:企业公告
|
||||
- `china`、`global`、`us`
|
||||
- `aggregated`、`low_stability`
|
||||
|
||||
单条新闻有一个主类型 `category`,默认类型包括:政治、商业、电商、金融、体育、科技、军事、灾害、能源、社会、文化、其他。`item_tags` 是条目级补充标签,例如跨境电商、直播电商、零售数据、物流履约、平台治理、AI、半导体、选举、油价、足球、供应链。
|
||||
|
||||
主类型优先由规则引擎根据标题、摘要、来源名打分生成;规则未命中时优先使用 Feed 子项的默认类型,再回退 source 默认类型。AI enrichment 不阻塞新闻展示。
|
||||
|
||||
## 重要度
|
||||
|
||||
每条新闻输出:
|
||||
|
||||
- `importance_score`
|
||||
- `importance_level`
|
||||
- `importance_reasons`
|
||||
- `market_impact`
|
||||
|
||||
官方数据源、电商指标、平台型公司、量化指标会提高重要度;企业公告基础权重较低,只有命中大平台、金额、并购、监管等信号时提升。
|
||||
|
||||
## 配置与缓存
|
||||
|
||||
`GET /api/v1/earth/news-sources` 返回默认或已保存配置。`PUT /api/v1/earth/news-sources` 保存配置并递增 `cache_version`,同时清理进程内 region cache。`POST /api/v1/earth/news-sources/reset` 恢复默认源。`POST /api/v1/earth/news-sources/test` 只测试单个 RSS/Atom/Aggregated 源,不写入新闻表。
|
||||
|
||||
## Feed 查询与类型过滤
|
||||
|
||||
星球端和 UE 端统一使用 `GET /api/v1/news/earth-feed` 获取新闻。接口支持服务端过滤,不要求客户端拿全量列表后自行筛选。
|
||||
|
||||
- `lat` / `lon`:按当前视角推断区域,适合 Web 星球端。
|
||||
- `region`:显式指定区域,适合 UE 端或服务端集成;可选值包括 `global`、`americas`、`europe`、`asia-pacific`、`middle-east-africa`。`global` 是全局聚合视图,会展示所有区域来源;其它区域只展示该区域和 `global` 来源。
|
||||
- `categories`:逗号分隔的新闻类型 key,例如 `business,ecommerce`。全选时可以不传。
|
||||
- `locale`:展示语言,支持 `zh-CN` 和 `en-US`,默认 `zh-CN`。中文 RSS 会以中文原文入库,并由后台补 `en-US`;英文 RSS 则由后台补 `zh-CN`。
|
||||
|
||||
示例:
|
||||
|
||||
```http
|
||||
GET /api/v1/news/earth-feed?region=europe&categories=business,ecommerce
|
||||
GET /api/v1/news/earth-feed?lat=48&lon=10&categories=technology
|
||||
GET /api/v1/news/earth-feed?region=global&categories=business,ecommerce&locale=zh-CN
|
||||
```
|
||||
|
||||
非法新闻类型或语言会返回 `422`,响应中包含允许值。响应体会带 `filters`,用于确认后端实际应用的区域、类型和语言过滤。`items` 和 `cruise_items` 使用同一套类型过滤规则。
|
||||
|
||||
Web 星球端的新闻类型按钮只保存当前浏览器的显示偏好;偏好变化后会重新请求接口。UE 端应直接把类型选择拼到 `categories` 参数里,不需要再做主过滤。
|
||||
|
||||
源测试只证明当前 RSS/Atom/XML 能解析到条目,不等于这些条目已经入库展示。展示链路还会检查区域、类型过滤和数据库新鲜度。保存或重置新闻源会递增配置版本并清理缓存;如果当前启用的 Feed 子项在库里没有近期条目,下一次 `earth-feed` 请求会补抓,避免新启用的 36氪、亿邦被旧 Google News 缓存挡住。
|
||||
|
||||
## 连通性监测
|
||||
|
||||
`POST /api/v1/earth/news-sources/test` 会测试单个源并把结果写入 `earth_news_sources.health[source_id]`。实际 RSS/Atom 抓取也会更新同一份健康状态。
|
||||
|
||||
健康结果包含:
|
||||
|
||||
- `status`:`ok`、`empty`、`format_error`、`http_error`、`timeout`、`network_error`、`reference`。
|
||||
- `status_code`、`content_type`、`item_count`、`latency_ms`、`error`、`fetched_at`。
|
||||
- `feed_results`:多 Feed source 的逐 Feed 子项检测结果,包含 `feed_id`、`feed_name`、`feed_type`、`feed_url`、状态、条数和错误。
|
||||
|
||||
常见诊断:
|
||||
|
||||
- 返回 HTML 页面:说明配置 URL 不是 RSS/Atom feed,例如把网页中心页当成 feed。
|
||||
- HTTP 403:通常是 CDN、反爬或源站拒绝抓取。
|
||||
- Reference:参考链接,不参与抓取;需要改为 RSS、Atom 或 Aggregated 后才可测试抓取。
|
||||
|
||||
Admin 入口是 `Earth 内容 -> 新闻源`。界面不是整包 JSON 编辑,而是两层:
|
||||
|
||||
- **新闻源**:左侧逐个 source 列表,支持按启用、停用、参考链接、RSS/Atom/Aggregated、区域和源属性标签筛选;右侧编辑当前 source 字段和 Feed 子项列表。
|
||||
- **策略规则**:保留源属性标签、新闻类型、条目标签规则、默认健康策略等全局规则。高级 JSON 只用于排障,不作为默认编辑路径。
|
||||
|
||||
单源表单分为“来源信息”和“Feed 子项”:
|
||||
|
||||
- 来源信息包括名称、ID、区域、主页 URL、Feed 信息页、源类型、启用开关、源属性标签、重要度权重、抓取间隔、超时、失败阈值和熔断开关。
|
||||
- Feed 子项包括 Feed ID、名称、真实 Feed URL、类型、启用开关、默认新闻类型、优先级和 Feed 标签。Feed 子项底部的 `+` 只新增一个前端草稿;保存 source 后才写入配置,取消会销毁草稿。
|
||||
|
||||
单源“测试源”会测试当前 source 下全部启用 Feed;Feed 子项上的测试按钮只测试当前 Feed。测试请求仍发送到 `/api/v1/earth/news-sources/test`,但 payload 里只带当前 source 和选中的 Feed 子项。
|
||||
|
||||
参考链接会显示“只记录官网、报告页或未来采集器线索,不参与 RSS/Atom 抓取”。它可作为商业或官方数据线索保留在配置中,但启用抓取前必须改成 RSS、Atom 或 Aggregated,并提供可抓取的 Feed 地址。
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Admin["Admin: Earth 内容 / 新闻源"] --> Source["Source 配置"]
|
||||
Source --> Feed["Feed 子项"]
|
||||
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
|
||||
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
|
||||
|
||||
Earth["Earth 新闻面板"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||||
NewsAPI --> Resolver["Source Resolver"]
|
||||
Resolver --> Config
|
||||
Resolver --> Cache["Region Feed Cache"]
|
||||
Resolver --> Fetcher["RSS / Atom Fetcher"]
|
||||
Fetcher --> Parser["Feed Parser"]
|
||||
Parser --> Classifier["Classifier: category + item_tags + importance"]
|
||||
Classifier --> Store["earth_news_items"]
|
||||
Fetcher --> Health["source health"]
|
||||
Health --> Config
|
||||
Store --> EnrichQueue["Location / Localization Queue"]
|
||||
EnrichQueue --> AI["AI Provider"]
|
||||
Store --> NewsAPI
|
||||
UE["UE Client"] --> NewsAPI
|
||||
```
|
||||
@@ -19,14 +19,14 @@
|
||||
| 0 | 智能星球基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
|
||||
| 0.2 | 国界暗色 tint | `country-boundaries.js` | `tintAltitudeOffset` | 禁用 raycast | 高清材质关闭时使用。 |
|
||||
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset = 0.32`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用;半径与基座球拉开以避免远距 z-fighting。 |
|
||||
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset = 0.48` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充;半径必须高于海陆基座并与基座球保持足够间距。 |
|
||||
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充;半径必须高于海陆基座并与基座球保持足够间距。 |
|
||||
| 1 | 大气辉光和云图 | `earth.js` | 大气 / 云层球 | 不走普通对象选择路径 | 云图由“大气云图”图层开关控制。 |
|
||||
| 1 | 海缆 / 登陆点 | `cables.js` | 海缆线和登陆点都使用 `renderOrder = 1`;半径偏移都为 `0.2`;登陆点是专用 `THREE.Sprite` 黄色扁平球 | 海缆走海缆拾取路径;登陆点 `depthTest: false` 保持球体完整,并用相机到球心的球体遮挡判断避免背面穿透 | 登陆点和海缆同层贴地,避免地表设施层的凌空感。 |
|
||||
| 1.2 | 真实地形 | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` 加地形位移 | 禁用 raycast | 地形压过高清材质;高清材质关闭时临时隐藏,重新开启后恢复原状态。 |
|
||||
| 2.05 | 经纬线 | `earth.js` | `CONFIG.earthRadius + 0.14` | 禁用 raycast | 低透明度显示在高清材质上。 |
|
||||
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset = 0.115` | `depthTest: true`,禁用 raycast | 线层使用独立 line geometry 与 `renderOrder` 控制;地形 `depthWrite: false`,所以地形开启时仍可见。 |
|
||||
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`;claim 线不再额外抬高 | `depthTest: true`,禁用 raycast | 线层使用独立 line geometry 与 `renderOrder` 控制,但半径与高清材质壳完全一致,避免转动地球时与高清贴图出现视差。 |
|
||||
| 2.29 | 国界 hover 光晕 | `country-boundaries.js` | hover 半径加 glow 偏移 | `depthTest: false`,禁用 raycast | 用 additive 光晕增强交界边和地形开启时的 hover 可见性。 |
|
||||
| 2.3 | 国界 hover 实线 | `country-boundaries.js` | `hoverAltitudeOffset = 0.115` | `depthTest: false`,禁用 raycast | 霓虹红橘 hover 线;与普通国界线同源半径对齐,避免重影;中国和中国(台湾)共享高亮组。 |
|
||||
| 2.3 | 国界 hover 实线 | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: false`,禁用 raycast | 霓虹红橘 hover 线;与普通国界线和高清材质同源半径对齐,避免重影和漂浮感;中国和中国(台湾)共享高亮组。 |
|
||||
| 3 | 卫星 footprint 填充 / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested;Iridium adapter 的 fill / ring 也使用同一 renderOrder | Footprint 在 land / texture / terrain 和国界线之上,但在算力中心和卫星之下。 |
|
||||
| 3-4.5 | BGP 观测站、事件扩散圈和事件 marker | `bgp.js`, `interactable.js` | BGP 观测站和事件 marker 均使用 `Interactable` 批量 `THREE.Points`;事件 marker 使用 `BGP_EVENT_RENDER_ORDER = 4.5`;观测站主图标使用 `BGP_COLLECTOR_RENDER_ORDER = 4.4` 和 `BGP_CONFIG.collectorAltitudeOffset = 0.2`;事件 overlay 进入 `bgp-event-overlay-layer`;观测站 halo 和覆盖扇形进入 `bgp-collector-radar-layer` | BGP 事件和观测站都通过 `Interactable` 屏幕空间 picking,并参与同坐标避让 | BGP 观测站主图标与船只同层;BGP 事件与算力中心同层;向外扩散圈、观测站雷达/覆盖动画继续由 BGP 业务逻辑驱动。 |
|
||||
| 4.3 | AIS 船只轨迹线 | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`;`CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | 跟随船只显隐,不单独参与拾取 | 选中船只后显示最近轨迹,低于船只 marker。 |
|
||||
@@ -55,7 +55,8 @@ Earth 的地表不是单一 mesh,而是多层近似同心球:基座球、海
|
||||
维护规则:
|
||||
|
||||
- 不要用“远距隐藏图层”作为第一反应;先检查相邻 shell 的 `altitudeOffset`、`renderOrder`、`depthTest` 和 `depthWrite`。
|
||||
- 海陆基座和高清材质这类整球 overlay 必须与 `CONFIG.earthRadius` 保持明确间距;当前稳定值为 `landAltitudeOffset = 0.32`、`textureOverlayAltitudeOffset = 0.48`。
|
||||
- 海陆基座和高清材质这类整球 overlay 必须与 `CONFIG.earthRadius` 保持明确间距;当前稳定值为 `landAltitudeOffset = 0.32`、`textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`。
|
||||
- 国界线、coastline、claim 线和国界 hover 线必须使用 `EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET`,与高清材质壳完全同半径;不要再用低于或高于高清材质的独立线层 offset,否则转动地球时会产生相对地表的视差漂浮感。
|
||||
- 新增整球或近整球地表 overlay 时,必须在 50% 缩放和常用近距视图各截一次图,确认没有黑块、雪花、闪烁,也没有明显漂浮感。
|
||||
- 如果必须调整这些半径,需同步更新本文和 `frontend/public/earth/js/constants.js` 的注释/常量意图。
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ curl http://localhost:8010/health
|
||||
|
||||
如果 `ss -ltnp` 显示前端已经监听 `0.0.0.0:3000`,但 Windows PowerShell 中 `Test-NetConnection <Windows局域网IP> -Port 3000` 仍失败,问题通常不在 Vite 或 `.zshrc`,而是在 Windows 侧端口占用、旧 `portproxy` 或防火墙。
|
||||
|
||||
`./planet.sh start --allow-lan` 会直接开放 `3000` / `8000` / `8010`,并在启动前检测端口、旧 `portproxy` 和 Windows 防火墙规则。端口被 Windows 侧 listener 占用时,脚本会请求管理员 PowerShell 清理;缺少入站放行时,也会触发一次 UAC 管理员 PowerShell 请求来自动创建。若自动请求被取消,可以手动清理:
|
||||
`./planet.sh start --allow-lan` 会直接开放 `3000` / `8000` / `8010`,并在启动前检测端口、旧 `portproxy` 和 Windows 防火墙规则。端口被 Windows 侧 listener 占用时,脚本会请求管理员 PowerShell 清理;缺少入站允许规则时,也会触发一次 UAC 管理员 PowerShell 请求来自动创建。若自动请求被取消,可以手动清理:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
@@ -128,7 +128,7 @@ New-NetFirewallRule -DisplayName "WSL Planet 8010" -Direction Inbound -Action Al
|
||||
|
||||
局域网设备访问 Windows 对外端口,例如 `http://<Windows局域网IP>:3000/earth`。
|
||||
|
||||
如果 `wslinfo --networking-mode` 输出 `mirrored`,还需要检查 Hyper-V firewall。普通 Windows 防火墙规则存在时,Hyper-V firewall 仍可能拦截外部设备进入 WSL。管理员 PowerShell 中按端口放行:
|
||||
如果 `wslinfo --networking-mode` 输出 `mirrored`,还需要检查 Hyper-V firewall。普通 Windows 防火墙规则存在时,Hyper-V firewall 仍可能拦截外部设备进入 WSL。管理员 PowerShell 中按端口开放访问:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallHyperVRule -Name "Planet-Frontend-3000" -DisplayName "Planet Frontend 3000" -Direction Inbound -VMCreatorId "{40E0AC32-46A5-438A-A0B2-2B479E8F2E90}" -Protocol TCP -LocalPorts 3000 -Action Allow
|
||||
|
||||
@@ -92,6 +92,35 @@ Admin 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/home/ra
|
||||
- dark 下使用与 Docs 一致的深色底座、`#202938` slider 和深色外投影。
|
||||
- 业务侧只隐藏文字 label 并保留 icon + tooltip,不重写 slider DOM。
|
||||
|
||||
## 控制台状态颜色
|
||||
|
||||
Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/frontend/src/admin/patterns/patterns.tsx) 或 [Badge](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/badge.tsx),颜色变量来自 [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css)。新增状态时不要在页面里临时写 hex;先判断语义,再映射到现有 tone。
|
||||
|
||||
`StatusText` 是带圆点的指示灯:胶囊背景和边框保持组件原色,只让圆点和文字变成状态色。`Badge` 不带指示灯语义,可以使用同 tone 的浅色背景和边框强化信息层级。
|
||||
|
||||
| Tone | 颜色变量 | 语义 | 示例 |
|
||||
| --- | --- | --- | --- |
|
||||
| `success` | `--an-success` | 可用、成功、已连接、已启用 | 日志源 `可用`、采集 `成功` |
|
||||
| `warning` | `--an-warning` | 需要关注但不一定失败 | 日志文件 `暂无日志`、降级或跳过 |
|
||||
| `danger` | `--an-danger` | 失败、不可用、权限/连接错误 | `Docker 不可用`、接口失败 |
|
||||
| `info` / `running` | `--an-info` | 进行中、同步中、普通信息态 | `跟随中`、`同步中` |
|
||||
| `neutral` | `--an-muted` | 空态、停用、未上报、未知 | `暂无上报`、`停用` |
|
||||
| `ai` | 固定紫色 | AI 相关突出态 | AI 生成、模型动作 |
|
||||
|
||||
日志源列表遵循同一规则:`ok` 显示 success;`empty` 表示来源存在但还没有上报,显示 neutral;`missing` 表示期望的日志文件暂时不存在,显示 warning;`docker_unavailable` / `source_unavailable` 显示 danger。
|
||||
|
||||
## 控制台运行时日志
|
||||
|
||||
控制台运行时错误由 [runtimeLogs.ts](/home/ray/dev/linkong/planet/frontend/src/admin/runtimeLogs.ts) 统一上报到 `/api/v1/system/logs/admin-client`。它只在控制台路由内启用,跳过 `/earth`、`/docs`、登录和注册页面,避免公开页面噪声进入控制台日志源。
|
||||
|
||||
[AdminErrorBoundary.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/AdminErrorBoundary.tsx) 捕获 React 渲染错误并复用同一上报函数;全局 `error` 和 `unhandledrejection` 也进入该通道。上报失败必须静默处理,不能因为日志系统异常再制造新的前端错误。
|
||||
|
||||
日志页通过 `/ws` 的 `logs_tail` channel 跟随日志增量;文件日志和数据库日志都由后端统一转换成行事件。新增日志源时优先接入后端 source registry 和 tail manager,不要在日志页写独立轮询器。
|
||||
|
||||
日志页默认进入“重复统计”视图,读取 `/api/v1/system/logs/observability/groups`,按 `fingerprint` 聚合 Earth、Admin 和服务端运行时上报;点击聚合项再读取 `/api/v1/system/logs/observability/groups/{fingerprint}/events` 展示发生明细。原始日志和审计日志仍保留为独立视图;只有原始日志视图允许通过 WebSocket 跟随。前端上报器会在短时间窗口内合并同一错误并提交 `occurrence_count`,后端同时写 `system_logs` 和 `observability_events` / `observability_event_groups`,所以日志页不要再按相同消息在浏览器端二次聚合。
|
||||
|
||||
数据源任务队列的“查看日志”入口跳转到 `/logs?source=system-db&search=task_id=<id>`。后端数据库日志搜索索引必须把 JSON context 中的简单字段同时展开为 `key=value` 别名,例如 `task_id=26906`、`datasource_id=20`,这样历史任务日志不依赖重新执行任务也能被精确查到。
|
||||
|
||||
## 当前共享组件
|
||||
|
||||
### 1. `Scrollbar`
|
||||
|
||||
@@ -236,7 +236,7 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
|
||||
|
||||
## 数据探索
|
||||
|
||||
- `/datasources`:数据源目录。`内置源` 支持按产品域、层级、启用状态、最近执行状态、是否已有采集数据和关键词筛选;未勾选时主按钮显示“触发全部”,勾选多行后会变成“触发已选 N”,并只提交所选数据源。右上角队列按钮空态显示队列图标,有任务时显示纯圆环总进度;点击后打开队列浮层,按运行中、完成、失败和跳过分组,失败项可重试,完成项可跳到详情。`实时源` 面向 AISStream / WebSocket 长连接,展示连接健康、累计入库、时间窗统计和启动 / 停止 / 重连操作。接口、凭证、请求头的编辑统一在 `/collection-management` 的"采集器"。
|
||||
- `/datasources`:数据源目录。`内置源` 支持按产品域、层级、启用状态、最近执行状态、是否已有采集数据和关键词筛选;未勾选时主按钮显示“触发全部”,勾选多行后会变成“触发已选 N”,并只提交所选数据源。右上角队列按钮空态显示队列图标,有任务时显示纯圆环总进度;点击后打开队列浮层,按运行中、完成、失败和跳过分组,失败项可重试,完成项可跳到详情或按任务编号打开系统日志。`实时源` 面向 AISStream / WebSocket 长连接,展示连接健康、累计入库、时间窗统计和启动 / 停止 / 重连操作。接口、凭证、请求头的编辑统一在 `/collection-management` 的"采集器"。
|
||||
- `/data`:采集后数据表,适合排查"数据是否已经进入系统"、"更新时间是否符合预期"、"某个数据源是否产出有效记录"
|
||||
- `/bgp`:BGP 专题页面,列表 + 详情 + 研判,与智能星球的 BGP 图层互补
|
||||
- `/alerts/system`、`/alerts/bgp`、`/alerts/situational`:系统、BGP、态势告警
|
||||
@@ -299,6 +299,8 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、低缩放圆点、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
|
||||
|
||||
新闻类型使用与巡航模块一致的标签选择器,只筛选当前浏览器里的新闻面板和新闻巡航条目,不影响图层、TV、数据点、底图、边界、采集任务或后台新闻源配置。
|
||||
|
||||
“真实卫星高度”默认开启:卫星会按 TLE/SGP4 算出的真实轨道高度做压缩分层显示,低轨仍靠近地球,高轨会更远但不会脱离当前视图。高轨显示高度会被压到地球半径外约四分之一以内,这样 GEO / MEO 仍能和 LEO 分层,但不会把视线、轨迹和选择操作拉得过散;关闭后恢复旧版所有卫星位于同一显示球面的效果。“轨迹显示”控制卫星轨迹线显隐,卫星图层关闭时轨迹也不可见。
|
||||
|
||||
“悬停提示”控制鼠标悬停地表时的 tooltip 内容:`国家` 只在陆地命中国家时显示国家信息,太平洋等海洋区域不弹出地表提示;`位置` 在陆地和海洋都显示纬度、经度和海拔;`完整` 是默认模式,陆地显示国家 + 位置,海洋显示位置。
|
||||
|
||||
@@ -75,7 +75,7 @@
|
||||
- Docker 清理只针对 Compose project 为 `planet` 的资源,以及显式列出的 `planet_postgres_data`、`planet_redis_data`、`postgres_data`、`redis_data`;不要按 `planet_*` 模式删除没有 label 的 volume,避免误删同机其他项目。
|
||||
- 本地编译状态会删除 `.venv`、前端 `node_modules` / `dist`、Planet state,以及散落的 Python / Vite 缓存目录;`$PLANET_CACHE_DIR/downloads` 会保留,用于保存 CelesTrak 这类受上游下载窗口限制的原始文件缓存。
|
||||
|
||||
重置后重新执行 `./planet.sh init` 会重建表和默认数据,但不会恢复旧采集结果;首次进入 Earth 时 OOBE 会重新按后端真实采集状态判断。触发 CelesTrak 采集时,如果上游返回“本轮 GP 数据未更新”的 403,后端会优先用保留的下载缓存重新写入数据库;如果下载缓存也不存在,只能等待 CelesTrak 下一次更新窗口或使用 Space-Track 作为 fallback。
|
||||
重置后重新执行 `./planet.sh init` 会重建表和默认数据,但不会恢复旧采集结果;首次进入 Earth 时 OOBE 会重新按后端真实采集状态判断。触发 CelesTrak 采集时,如果上游返回“本轮 GP 数据未更新”的 403,后端会优先用保留的下载缓存重新写入数据库;如果 active 缓存不存在,会尝试有效 CelesTrak 分组缓存作为 fallback;如果下载缓存也不存在,只能等待 CelesTrak 下一次更新窗口或使用 Space-Track。控制台里的数据源“删除数据库”和“清理缓存”不会删除 `$PLANET_CACHE_DIR/downloads/celestrak`。
|
||||
|
||||
## 健康检查
|
||||
|
||||
@@ -150,7 +150,7 @@ Planet 开发环境建议使用 WSL2。WSL1 下网络、文件系统和进程模
|
||||
wsl --set-version Ubuntu 2
|
||||
```
|
||||
|
||||
`--allow-lan` 会让前端、后端和 AI Provider 直接对开发机开放:前端 `3000`、后端 `8000`、AI Provider `8010`。脚本启动前会检查这三个端口;如果 WSL/Linux 侧无法释放端口,并检测到 Windows 侧 listener 或旧 `portproxy`,会请求管理员 PowerShell 清理。WSL 中运行时,Windows 本机一般可以通过 `localhost` 访问,局域网其他机器访问 Windows 局域网 IP 时还需要 Windows 防火墙放行。
|
||||
`--allow-lan` 会让前端、后端和 AI Provider 直接对开发机开放:前端 `3000`、后端 `8000`、AI Provider `8010`。脚本启动前会检查这三个端口;如果 WSL/Linux 侧无法释放端口,并检测到 Windows 侧 listener 或旧 `portproxy`,会请求管理员 PowerShell 清理。WSL 中运行时,Windows 本机一般可以通过 `localhost` 访问,局域网其他机器访问 Windows 局域网 IP 时还需要 Windows 防火墙允许访问。
|
||||
|
||||
建议按顺序排查:
|
||||
|
||||
@@ -162,7 +162,7 @@ curl http://localhost:8010/health
|
||||
ss -ltnp | grep -E ':3000|:8000|:8010'
|
||||
```
|
||||
|
||||
如果服务已经启动但局域网 IP 仍访问失败,优先清理旧 `portproxy` 并确认 Windows 防火墙放行。脚本会自动检测并请求管理员 PowerShell 处理;自动请求被取消时,手动兜底命令如下:
|
||||
如果服务已经启动但局域网 IP 仍访问失败,优先清理旧 `portproxy` 并确认 Windows 防火墙允许访问。脚本会自动检测并请求管理员 PowerShell 处理;自动请求被取消时,手动兜底命令如下:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
@@ -286,6 +286,8 @@ bun run build
|
||||
|
||||
不要使用 `npm run ...`。项目在 WSL / Windows 混合环境优先依赖 Bun,避免 Node/npm 路径差异。
|
||||
|
||||
`./planet.sh start` / `init` 会在启动前执行一次 `bun install`,而不是只检查 Vite 入口文件是否存在。这样新设备、清过 `node_modules` 的环境或 lockfile 已变更的环境,都能在进入控制台前同步前端依赖,避免动态 import 因缺失依赖返回 500。
|
||||
|
||||
验证前端构建:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -16,12 +16,16 @@
|
||||
## Current Version
|
||||
|
||||
- `main` 当前主线历史推导到:`0.16.5`
|
||||
- `dev` 当前开发分支历史推导到:`0.66.3`
|
||||
- `dev` 当前开发分支历史推导到:`0.69.0`
|
||||
|
||||
## Timeline
|
||||
|
||||
| Version | Type | Branch | Commit | Summary |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `0.69.0` | feature | `dev` | `pending` | 新增 Earth 新闻源治理、新闻类型服务端过滤、观测日志 fingerprint 聚合和 TV/HLS 播放恢复改进 |
|
||||
| `0.68.1` | bugfix | `dev` | `pending` | 修复 CelesTrak fallback group/cache 恢复链路,并让数据源任务日志可按 task_id / datasource_id 搜索 |
|
||||
| `0.68.0` | feature | `dev` | `pending` | 新增数据源任务队列实时指标、AIS 大表分批删除和智能星球可插拔聚类策略,并让新设备启动前同步前端依赖 |
|
||||
| `0.67.0` | feature | `dev` | `pending` | 新增控制台日志实时跟随和运行时错误上报,重构智能星球 Interactable 聚合、wheel 缩放输入、国界壳半径和开发脚本锁文件保护 |
|
||||
| `0.66.3` | bugfix | `dev` | `pending` | 补上 Admin utility module 并放开前端源码 lib 例外,修复控制台动态导入 500 与 Mermaid 包解析失败 |
|
||||
| `0.66.2` | bugfix | `dev` | `pending` | 统一智能星球、控制台和文档的中文显示命名,清理 Docs catalog、使用手册、控制台入口和智能星球设置中的中英混排文案 |
|
||||
| `0.66.1` | bugfix | `dev` | `pending` | CelesTrak active 限频时复用持久下载缓存并完整 fallback group,destroy 保留原始下载缓存,同时修复采集失败 toast 重复弹出 |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "planet-frontend",
|
||||
"version": "0.66.3",
|
||||
"version": "0.69.0",
|
||||
"private": true,
|
||||
"packageManager": "bun@1",
|
||||
"dependencies": {
|
||||
|
||||
@@ -1086,6 +1086,12 @@
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.earth-mobile-news-filter-popover {
|
||||
position: relative;
|
||||
inset: auto;
|
||||
max-height: 34vh;
|
||||
}
|
||||
|
||||
.earth-mobile-news-board-list .news-story-card {
|
||||
margin: 0;
|
||||
}
|
||||
@@ -2822,6 +2828,7 @@ label.is-disabled.earth-mobile-settings-card {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.earth-settings-segmented {
|
||||
@@ -2908,10 +2915,17 @@ label.is-disabled.earth-mobile-settings-card {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
flex: 0 0 calc(30px * var(--hud-scale));
|
||||
width: calc(30px * var(--hud-scale));
|
||||
height: calc(30px * var(--hud-scale));
|
||||
min-width: calc(30px * var(--hud-scale));
|
||||
min-height: calc(30px * var(--hud-scale));
|
||||
max-width: calc(30px * var(--hud-scale));
|
||||
max-height: calc(30px * var(--hud-scale));
|
||||
aspect-ratio: 1 / 1;
|
||||
padding: 0;
|
||||
border: 1px solid rgba(122, 180, 255, 0.24);
|
||||
border-radius: 999px;
|
||||
border-radius: 50%;
|
||||
background:
|
||||
radial-gradient(circle at 35% 22%, rgba(255, 255, 255, 0.2), transparent 58%),
|
||||
linear-gradient(180deg, rgba(121, 159, 207, 0.18), rgba(72, 101, 139, 0.24));
|
||||
@@ -2926,12 +2940,16 @@ label.is-disabled.earth-mobile-settings-card {
|
||||
}
|
||||
|
||||
.earth-settings-reload-action--text {
|
||||
flex-basis: auto;
|
||||
width: auto;
|
||||
max-width: none;
|
||||
min-width: calc(46px * var(--hud-scale));
|
||||
padding: 0 calc(12px * var(--hud-scale));
|
||||
aspect-ratio: auto;
|
||||
font: inherit;
|
||||
font-size: calc(0.7rem * var(--hud-scale));
|
||||
font-weight: 600;
|
||||
border-radius: 999px;
|
||||
}
|
||||
|
||||
.earth-settings-reload-action[hidden] {
|
||||
@@ -3076,6 +3094,7 @@ label.is-disabled.earth-mobile-settings-card {
|
||||
|
||||
.earth-settings-slider {
|
||||
flex: 1 1 auto;
|
||||
min-width: calc(120px * var(--hud-scale));
|
||||
width: 100%;
|
||||
height: calc(4px * var(--hud-scale));
|
||||
appearance: none;
|
||||
|
||||
@@ -427,6 +427,43 @@
|
||||
text-wrap: balance;
|
||||
}
|
||||
|
||||
.info-card-news-meta-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
gap: calc(6px * var(--hud-scale));
|
||||
}
|
||||
|
||||
.info-card-news-meta-grid--mobile {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.info-card-news-meta-item {
|
||||
min-width: 0;
|
||||
padding: calc(6px * var(--hud-scale)) calc(8px * var(--hud-scale));
|
||||
border: 1px solid rgba(201, 225, 247, 0.08);
|
||||
background: rgba(255, 255, 255, 0.035);
|
||||
}
|
||||
|
||||
.info-card-news-meta-item span,
|
||||
.info-card-news-meta-item strong {
|
||||
display: block;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.info-card-news-meta-item span {
|
||||
color: rgba(188, 212, 238, 0.62);
|
||||
font-size: calc(0.58rem * var(--hud-scale));
|
||||
}
|
||||
|
||||
.info-card-news-meta-item strong {
|
||||
color: rgba(236, 246, 255, 0.9);
|
||||
font-size: calc(0.68rem * var(--hud-scale));
|
||||
font-weight: 650;
|
||||
margin-top: calc(2px * var(--hud-scale));
|
||||
}
|
||||
|
||||
.info-card-news-summary-shell {
|
||||
position: relative;
|
||||
padding: calc(10px * var(--hud-scale)) calc(12px * var(--hud-scale));
|
||||
|
||||
@@ -303,6 +303,121 @@
|
||||
font-size: calc(0.72rem * var(--hud-scale));
|
||||
}
|
||||
|
||||
.news-filter-bar {
|
||||
position: relative;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: calc(8px * var(--hud-scale));
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.news-filter-pill {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: calc(6px * var(--hud-scale));
|
||||
min-height: calc(30px * var(--hud-scale));
|
||||
border: 1px solid rgba(201, 225, 247, 0.1);
|
||||
border-radius: calc(12px * var(--hud-scale));
|
||||
padding: calc(5px * var(--hud-scale)) calc(9px * var(--hud-scale));
|
||||
color: var(--hud-text-soft);
|
||||
background:
|
||||
linear-gradient(180deg, rgba(255, 255, 255, 0.07), rgba(116, 166, 224, 0.04));
|
||||
box-shadow:
|
||||
inset 0 1px 0 rgba(255, 255, 255, 0.06),
|
||||
0 8px 18px rgba(2, 10, 22, 0.12);
|
||||
font: inherit;
|
||||
font-size: calc(0.72rem * var(--hud-scale));
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.news-filter-pill:hover,
|
||||
.news-filter-pill[aria-expanded="true"] {
|
||||
color: var(--hud-text);
|
||||
border-color: rgba(147, 202, 255, 0.22);
|
||||
background:
|
||||
linear-gradient(180deg, rgba(255, 255, 255, 0.1), rgba(116, 166, 224, 0.07));
|
||||
}
|
||||
|
||||
.news-filter-pill strong {
|
||||
color: var(--hud-accent-strong);
|
||||
font-size: calc(0.7rem * var(--hud-scale));
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.news-filter-pill--view {
|
||||
margin-left: auto;
|
||||
}
|
||||
|
||||
.news-filter-popover {
|
||||
position: absolute;
|
||||
top: calc(154px * var(--hud-scale));
|
||||
left: calc(14px * var(--hud-scale));
|
||||
right: calc(14px * var(--hud-scale));
|
||||
z-index: 30;
|
||||
display: grid;
|
||||
gap: calc(10px * var(--hud-scale));
|
||||
max-height: min(calc(280px * var(--hud-scale)), 44vh);
|
||||
overflow-y: auto;
|
||||
border: 1px solid rgba(205, 231, 255, 0.12);
|
||||
border-radius: calc(16px * var(--hud-scale));
|
||||
padding: calc(12px * var(--hud-scale));
|
||||
color: var(--hud-text);
|
||||
background:
|
||||
radial-gradient(circle at 18% 10%, rgba(122, 187, 255, 0.16), transparent 42%),
|
||||
linear-gradient(180deg, rgba(20, 35, 58, 0.96), rgba(10, 20, 35, 0.96));
|
||||
box-shadow:
|
||||
0 20px 50px rgba(2, 8, 20, 0.38),
|
||||
inset 0 1px 0 rgba(255, 255, 255, 0.07);
|
||||
backdrop-filter: blur(18px) saturate(125%);
|
||||
-webkit-backdrop-filter: blur(18px) saturate(125%);
|
||||
}
|
||||
|
||||
.news-filter-popover[hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.news-filter-popover__header {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
justify-content: space-between;
|
||||
gap: calc(10px * var(--hud-scale));
|
||||
}
|
||||
|
||||
.news-filter-popover__title {
|
||||
font-size: calc(0.8rem * var(--hud-scale));
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.news-filter-popover__hint {
|
||||
color: var(--hud-text-muted);
|
||||
font-size: calc(0.66rem * var(--hud-scale));
|
||||
}
|
||||
|
||||
.news-filter-chip-group {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: calc(8px * var(--hud-scale));
|
||||
}
|
||||
|
||||
.news-filter-chip {
|
||||
border: 1px solid rgba(201, 225, 247, 0.12);
|
||||
border-radius: calc(14px * var(--hud-scale));
|
||||
padding: calc(7px * var(--hud-scale)) calc(10px * var(--hud-scale));
|
||||
color: var(--hud-text-soft);
|
||||
background: rgba(255, 255, 255, 0.04);
|
||||
font: inherit;
|
||||
font-size: calc(0.74rem * var(--hud-scale));
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.news-filter-chip.is-active {
|
||||
color: var(--hud-text);
|
||||
border-color: rgba(120, 190, 255, 0.36);
|
||||
background:
|
||||
linear-gradient(180deg, rgba(79, 143, 232, 0.22), rgba(64, 111, 191, 0.12));
|
||||
box-shadow: inset 0 0 0 1px rgba(206, 232, 255, 0.08);
|
||||
}
|
||||
|
||||
.news-board {
|
||||
display: flex;
|
||||
flex: 1 1 auto;
|
||||
@@ -379,13 +494,21 @@
|
||||
.news-story-tags {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: calc(8px * var(--hud-scale));
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.news-story-meta {
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
.news-story-tags {
|
||||
justify-content: flex-start;
|
||||
}
|
||||
|
||||
.news-story-source,
|
||||
.news-story-time,
|
||||
.news-story-origin,
|
||||
.news-story-tag {
|
||||
color: var(--hud-text-soft);
|
||||
font-size: calc(0.66rem * var(--hud-scale));
|
||||
@@ -393,6 +516,17 @@
|
||||
|
||||
.news-story-source {
|
||||
color: var(--hud-accent-strong);
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.news-story-origin {
|
||||
color: rgba(188, 212, 238, 0.56);
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.news-story-title {
|
||||
|
||||
@@ -606,6 +606,22 @@
|
||||
<div id="news-source-count" class="news-source-count">0 路聚合源</div>
|
||||
</div>
|
||||
|
||||
<div class="news-filter-bar" data-news-filter-bar>
|
||||
<button class="news-filter-pill" type="button" data-news-filter-toggle="category" aria-expanded="false">
|
||||
<span>类型</span>
|
||||
<strong data-news-filter-summary="category">全部</strong>
|
||||
</button>
|
||||
<button class="news-filter-pill" type="button" data-news-filter-toggle="source" aria-expanded="false">
|
||||
<span>来源</span>
|
||||
<strong data-news-filter-summary="source">全部</strong>
|
||||
</button>
|
||||
<button id="news-view-all-toggle" class="news-filter-pill news-filter-pill--view" type="button">
|
||||
<span data-news-view-mode-label>查看全部</span>
|
||||
<strong data-news-filter-summary="limit">12 条</strong>
|
||||
</button>
|
||||
</div>
|
||||
<div class="news-filter-popover" data-news-filter-popover hidden></div>
|
||||
|
||||
<div class="news-board">
|
||||
<div id="news-board-status" class="news-board-status">正在准备全球态势新闻...</div>
|
||||
<div id="news-board-list" class="news-board-list"></div>
|
||||
@@ -768,6 +784,21 @@
|
||||
</div>
|
||||
<div id="mobile-news-source-count" class="earth-mobile-news-source-count">0 路聚合源</div>
|
||||
</div>
|
||||
<div class="news-filter-bar earth-mobile-news-filter-bar" data-news-filter-bar>
|
||||
<button class="news-filter-pill" type="button" data-news-filter-toggle="category" aria-expanded="false">
|
||||
<span>类型</span>
|
||||
<strong data-news-filter-summary="category">全部</strong>
|
||||
</button>
|
||||
<button class="news-filter-pill" type="button" data-news-filter-toggle="source" aria-expanded="false">
|
||||
<span>来源</span>
|
||||
<strong data-news-filter-summary="source">全部</strong>
|
||||
</button>
|
||||
<button id="mobile-news-view-all-toggle" class="news-filter-pill news-filter-pill--view" type="button">
|
||||
<span data-news-view-mode-label>查看全部</span>
|
||||
<strong data-news-filter-summary="limit">12 条</strong>
|
||||
</button>
|
||||
</div>
|
||||
<div class="news-filter-popover earth-mobile-news-filter-popover" data-news-filter-popover hidden></div>
|
||||
<div id="mobile-news-board-status" class="earth-mobile-news-board-status">正在准备全球态势新闻...</div>
|
||||
<div id="mobile-news-board-list" class="earth-mobile-news-board-list"></div>
|
||||
<div id="mobile-news-board-empty" class="earth-mobile-news-board-empty" hidden>正在准备全球态势新闻聚合源...</div>
|
||||
@@ -1049,6 +1080,27 @@
|
||||
<span class="earth-mobile-settings-switch-track"></span>
|
||||
</span>
|
||||
</label>
|
||||
<div class="earth-mobile-settings-subsection-title">新闻类型</div>
|
||||
<div class="earth-mobile-settings-card earth-mobile-settings-card--stacked">
|
||||
<div class="earth-mobile-settings-copy">
|
||||
<span class="earth-mobile-settings-label">新闻类型</span>
|
||||
<span class="earth-mobile-settings-subtitle">只筛选当前浏览器的新闻面板与新闻巡航,不改变后台新闻源。</span>
|
||||
</div>
|
||||
<div class="earth-mobile-settings-chip-group" role="group" aria-label="选择新闻类型">
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="politics" aria-pressed="true">政治</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="business" aria-pressed="true">商业</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="ecommerce" aria-pressed="true">电商</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="finance" aria-pressed="true">金融</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="sports" aria-pressed="true">体育</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="technology" aria-pressed="true">科技</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="military" aria-pressed="true">军事</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="disaster" aria-pressed="true">灾害</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="energy" aria-pressed="true">能源</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="society" aria-pressed="true">社会</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="culture" aria-pressed="true">文化</button>
|
||||
<button type="button" class="earth-mobile-settings-chip is-active" data-news-category-toggle="other" aria-pressed="true">其他</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="earth-mobile-settings-group" data-settings-tab-panel="panels" hidden>
|
||||
<div class="earth-mobile-settings-title">面板</div>
|
||||
@@ -1555,6 +1607,27 @@
|
||||
<span class="earth-settings-switch-track"></span>
|
||||
</span>
|
||||
</label>
|
||||
<div class="earth-settings-subsection-title">新闻类型</div>
|
||||
<div class="earth-settings-item earth-settings-item--stacked">
|
||||
<div class="earth-settings-copy">
|
||||
<span class="earth-settings-item-title">新闻类型</span>
|
||||
<span class="earth-settings-item-subtitle">只筛选当前浏览器的新闻面板与新闻巡航,不改变后台新闻源。</span>
|
||||
</div>
|
||||
<div class="earth-settings-chip-group" role="group" aria-label="选择新闻类型">
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="politics" aria-pressed="true">政治</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="business" aria-pressed="true">商业</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="ecommerce" aria-pressed="true">电商</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="finance" aria-pressed="true">金融</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="sports" aria-pressed="true">体育</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="technology" aria-pressed="true">科技</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="military" aria-pressed="true">军事</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="disaster" aria-pressed="true">灾害</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="energy" aria-pressed="true">能源</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="society" aria-pressed="true">社会</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="culture" aria-pressed="true">文化</button>
|
||||
<button type="button" class="earth-settings-chip is-active" data-news-category-toggle="other" aria-pressed="true">其他</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<section class="earth-settings-section" data-settings-tab-panel="panels" hidden>
|
||||
|
||||
@@ -333,6 +333,9 @@ const bgpEventIconLayer = createInteractableLayer({
|
||||
pulseOffset: Math.random() * Math.PI * 2,
|
||||
}),
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
},
|
||||
});
|
||||
|
||||
const bgpCollectorIconLayer = createInteractableLayer({
|
||||
@@ -402,6 +405,9 @@ const bgpCollectorIconLayer = createInteractableLayer({
|
||||
};
|
||||
},
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
},
|
||||
});
|
||||
|
||||
function clamp(value, min, max) {
|
||||
|
||||
@@ -2,6 +2,8 @@ import { PATHS } from "./constants.js";
|
||||
|
||||
const RECENT_EVENT_TTL_MS = 15_000;
|
||||
const recentEventMap = new Map();
|
||||
const pendingEventMap = new Map();
|
||||
let pendingFlushTimer = null;
|
||||
|
||||
function normalizeErrorDetail(detail) {
|
||||
if (!detail) return "";
|
||||
@@ -22,6 +24,37 @@ function dedupeKey(level, message, detail, category) {
|
||||
return `${level}::${category || ""}::${message}::${detail}`;
|
||||
}
|
||||
|
||||
function normalizeFingerprintText(value) {
|
||||
return String(value || "")
|
||||
.replace(/[?&](m|t|token|expires|signature|X-Amz-[^=]+)=[^&\s]+/gi, "")
|
||||
.replace(/(index|chunk|segment)[_-]?\d+(_\d+)?\.(ts|m4s|vtt)/gi, "<hls-fragment>")
|
||||
.replace(/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi, "<uuid>")
|
||||
.replace(/\bconn_[A-Za-z0-9:._-]+\b/g, "<connection>")
|
||||
.replace(/\b\d{5,}\b/g, "<number>")
|
||||
.replace(/\s+/g, " ")
|
||||
.trim();
|
||||
}
|
||||
|
||||
function hashFingerprint(value) {
|
||||
let hash = 5381;
|
||||
for (let index = 0; index < value.length; index += 1) {
|
||||
hash = ((hash << 5) + hash) ^ value.charCodeAt(index);
|
||||
}
|
||||
return `client-${(hash >>> 0).toString(16).padStart(8, "0")}`;
|
||||
}
|
||||
|
||||
function buildFingerprint(level, message, detail, category, module) {
|
||||
return hashFingerprint(
|
||||
[
|
||||
normalizeFingerprintText(level),
|
||||
normalizeFingerprintText(category),
|
||||
normalizeFingerprintText(module),
|
||||
normalizeFingerprintText(message),
|
||||
normalizeFingerprintText(detail),
|
||||
].join("|"),
|
||||
);
|
||||
}
|
||||
|
||||
function shouldSkip(level, message, detail, category) {
|
||||
const key = dedupeKey(level, message, detail, category);
|
||||
const now = Date.now();
|
||||
@@ -37,33 +70,14 @@ function shouldSkip(level, message, detail, category) {
|
||||
return lastSeenAt && now - lastSeenAt < RECENT_EVENT_TTL_MS;
|
||||
}
|
||||
|
||||
export async function reportEarthClientLog({
|
||||
level = "error",
|
||||
message,
|
||||
category = "runtime",
|
||||
module = "earth",
|
||||
detail = "",
|
||||
}) {
|
||||
if (!message) return;
|
||||
const normalizedDetail = normalizeErrorDetail(detail);
|
||||
if (shouldSkip(level, message, normalizedDetail, category)) {
|
||||
return;
|
||||
}
|
||||
|
||||
async function sendEarthClientLog(payload) {
|
||||
try {
|
||||
await fetch(PATHS.earthClientLogsApi, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
level,
|
||||
message,
|
||||
category,
|
||||
module,
|
||||
url: window.location.href,
|
||||
detail: normalizedDetail.slice(0, 4000),
|
||||
}),
|
||||
body: JSON.stringify(payload),
|
||||
keepalive: true,
|
||||
});
|
||||
} catch {
|
||||
@@ -71,6 +85,61 @@ export async function reportEarthClientLog({
|
||||
}
|
||||
}
|
||||
|
||||
function scheduleFlush() {
|
||||
if (pendingFlushTimer) return;
|
||||
pendingFlushTimer = window.setTimeout(() => {
|
||||
pendingFlushTimer = null;
|
||||
const pending = Array.from(pendingEventMap.values());
|
||||
pendingEventMap.clear();
|
||||
pending.forEach((entry) => {
|
||||
void sendEarthClientLog({
|
||||
level: entry.level,
|
||||
message: entry.message,
|
||||
category: entry.category,
|
||||
module: entry.module,
|
||||
url: window.location.href,
|
||||
detail: entry.detail.slice(0, 4000),
|
||||
fingerprint: entry.fingerprint,
|
||||
occurrence_count: entry.occurrenceCount,
|
||||
metadata: entry.metadata,
|
||||
});
|
||||
});
|
||||
}, 1000);
|
||||
}
|
||||
|
||||
export function reportEarthClientLog({
|
||||
level = "error",
|
||||
message,
|
||||
category = "runtime",
|
||||
module = "earth",
|
||||
detail = "",
|
||||
metadata = {},
|
||||
}) {
|
||||
if (!message) return;
|
||||
const normalizedDetail = normalizeErrorDetail(detail);
|
||||
const fingerprint = buildFingerprint(level, message, normalizedDetail, category, module);
|
||||
const key = dedupeKey(level, message, normalizedDetail, category);
|
||||
const existing = pendingEventMap.get(key);
|
||||
if (existing) {
|
||||
existing.occurrenceCount += 1;
|
||||
existing.metadata = { ...existing.metadata, ...metadata };
|
||||
} else {
|
||||
pendingEventMap.set(key, {
|
||||
level,
|
||||
message,
|
||||
category,
|
||||
module,
|
||||
detail: normalizedDetail,
|
||||
fingerprint,
|
||||
occurrenceCount: 1,
|
||||
metadata,
|
||||
});
|
||||
}
|
||||
|
||||
shouldSkip(level, message, normalizedDetail, category);
|
||||
scheduleFlush();
|
||||
}
|
||||
|
||||
export function registerEarthClientErrorHandlers() {
|
||||
window.addEventListener("error", (event) => {
|
||||
console.error("全局错误:", event.error);
|
||||
|
||||
@@ -214,7 +214,6 @@ const computeCenterIconLayer = createInteractableLayer({
|
||||
},
|
||||
icon: {
|
||||
coordinates: "canvas",
|
||||
colorable: false,
|
||||
fitSize: COMPUTE_CENTER_ICON_FIT_SIZE,
|
||||
glowBlur: 16,
|
||||
getSource({ marker, item }) {
|
||||
@@ -248,6 +247,9 @@ const computeCenterIconLayer = createInteractableLayer({
|
||||
pulseOffset: Math.random() * Math.PI * 2,
|
||||
}),
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
},
|
||||
});
|
||||
|
||||
export function formatComputeCenterTypeLabel(siteType) {
|
||||
|
||||
@@ -64,6 +64,8 @@ export const SURFACE_HOVER_INFO_MODES = {
|
||||
export const DEFAULT_SURFACE_HOVER_INFO_MODE =
|
||||
SURFACE_HOVER_INFO_MODES.FULL;
|
||||
|
||||
export const EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48;
|
||||
|
||||
export const CRUISE_CONFIG = {
|
||||
dwellMs: 7_000,
|
||||
focusDurationMs: 1_400,
|
||||
@@ -233,8 +235,12 @@ export const COUNTRY_BOUNDARY_CONFIG = {
|
||||
tilePrefetchRing: 1,
|
||||
tileDebounceMs: 180,
|
||||
tileCacheLimit: 150,
|
||||
lineAltitudeOffset: 0.115,
|
||||
hoverAltitudeOffset: 0.115,
|
||||
// Keep boundary/coastline strokes on the same shell as the high-res earth
|
||||
// texture overlay. A lower or higher radius creates visible parallax while
|
||||
// the globe rotates.
|
||||
lineAltitudeOffset: EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET,
|
||||
hoverAltitudeOffset: EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET,
|
||||
claimLineAltitudeOffset: 0,
|
||||
hoverMissStickyMs: 160,
|
||||
lineColor: 0x7fc7ff,
|
||||
lineOpacity: 0.58,
|
||||
@@ -540,7 +546,7 @@ export const EARTH_MATERIAL_CONFIG = {
|
||||
shininess: 12,
|
||||
emissive: 0x010609,
|
||||
opacity: 1,
|
||||
textureOverlayAltitudeOffset: 0.48,
|
||||
textureOverlayAltitudeOffset: EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET,
|
||||
textureOverlayOpacity: 0.88,
|
||||
textureOverlayRenderOrder: 0.96,
|
||||
textureOverlaySpecular: 0x05080d,
|
||||
|
||||
309
frontend/public/earth/js/controls.js
vendored
309
frontend/public/earth/js/controls.js
vendored
@@ -159,7 +159,7 @@ const SETTINGS_SHEET_MAX_SCALE_X = 0.22;
|
||||
const SETTINGS_SHEET_MAX_SCALE_Y = 0.18;
|
||||
const EARTH_SETTINGS_STORAGE_KEY = "planet.earth.settings.v2";
|
||||
const LEGACY_EARTH_SETTINGS_STORAGE_KEY = "planet.earth.settings.v1";
|
||||
const EARTH_SETTINGS_VERSION = 15;
|
||||
const EARTH_SETTINGS_VERSION = 16;
|
||||
const GRID_LINES_DEFAULT_VERSION = 3;
|
||||
const SATELLITE_DISPLAY_DEFAULT_VERSION = 4;
|
||||
const MEDIA_PANEL_DEFAULT_VERSION = 5;
|
||||
@@ -172,6 +172,21 @@ const SURFACE_HOVER_INFO_DEFAULT_VERSION = 11;
|
||||
const KEYBOARD_SHORTCUTS_DEFAULT_VERSION = 13;
|
||||
const CRUISE_QUEUE_DEFAULT_VERSION = 14;
|
||||
const AUTO_ROTATION_SPEED_DEFAULT_VERSION = 15;
|
||||
const NEWS_CATEGORY_FILTERS_DEFAULT_VERSION = 16;
|
||||
const DEFAULT_NEWS_CATEGORY_FILTERS = {
|
||||
politics: true,
|
||||
business: true,
|
||||
ecommerce: true,
|
||||
finance: true,
|
||||
sports: true,
|
||||
technology: true,
|
||||
military: true,
|
||||
disaster: true,
|
||||
energy: true,
|
||||
society: true,
|
||||
culture: true,
|
||||
other: true,
|
||||
};
|
||||
const AUTO_ROTATION_SPEED_MIN = 0.0001;
|
||||
const AUTO_ROTATION_SPEED_MAX = 0.0015;
|
||||
const AUTO_ROTATION_SPEED_STEP = 0.00005;
|
||||
@@ -185,6 +200,11 @@ const KEYBOARD_ROTATION_STOP_SPEED = 0.012;
|
||||
const KEYBOARD_ZOOM_STEP = 0.1;
|
||||
const WHEEL_ZOOM_STEP = 0.1;
|
||||
const WHEEL_ZOOM_DURATION_MS = 180;
|
||||
const WHEEL_TRACKPAD_PIXEL_THRESHOLD = 48;
|
||||
const WHEEL_TRACKPAD_DEADZONE = 0.35;
|
||||
const WHEEL_TRACKPAD_RESIDUAL_WINDOW_MS = 140;
|
||||
const WHEEL_TRACKPAD_RESIDUAL_RATIO = 0.65;
|
||||
const WHEEL_TRACKPAD_SENSITIVITY = 0.0024;
|
||||
const TARGET_SWITCH_ZOOM_IN_PHASE = 0.28;
|
||||
const TARGET_SWITCH_ROTATE_PHASE = 0.5;
|
||||
let settingsModalTimer = null;
|
||||
@@ -655,7 +675,7 @@ function stopKeyboardRotationControl({ actionId = null, restoreAutoRotate = fals
|
||||
}
|
||||
|
||||
function applyKeyboardZoom(direction) {
|
||||
setZoomLevel(zoomLevel + direction * KEYBOARD_ZOOM_STEP, activeCamera);
|
||||
setZoomLevel(getZoomLevelFromCamera(activeCamera) + direction * KEYBOARD_ZOOM_STEP, activeCamera);
|
||||
showZoomStatusCapsule({ force: true });
|
||||
}
|
||||
|
||||
@@ -1318,6 +1338,19 @@ function clampEarthZoomLevel(nextZoom) {
|
||||
return Math.min(CONFIG.maxZoom, Math.max(CONFIG.minZoom, parsedZoom));
|
||||
}
|
||||
|
||||
function getZoomLevelFromCamera(camera = activeCamera) {
|
||||
const cameraZ = Number(camera?.position?.z);
|
||||
if (!Number.isFinite(cameraZ) || cameraZ <= 0) {
|
||||
return clampEarthZoomLevel(zoomLevel);
|
||||
}
|
||||
return clampEarthZoomLevel(CONFIG.defaultCameraZ / cameraZ);
|
||||
}
|
||||
|
||||
function syncZoomLevelFromCamera(camera = activeCamera) {
|
||||
zoomLevel = getZoomLevelFromCamera(camera);
|
||||
return zoomLevel;
|
||||
}
|
||||
|
||||
function formatZoomPercent(zoom) {
|
||||
return `${Math.round(zoom * 100)}%`;
|
||||
}
|
||||
@@ -1349,6 +1382,23 @@ function canUseLocalStorage() {
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeNewsCategoryFilters(filters) {
|
||||
const normalized = { ...DEFAULT_NEWS_CATEGORY_FILTERS };
|
||||
if (!filters || typeof filters !== "object") return normalized;
|
||||
Object.keys(DEFAULT_NEWS_CATEGORY_FILTERS).forEach((category) => {
|
||||
if (typeof filters[category] === "boolean") {
|
||||
normalized[category] = filters[category];
|
||||
}
|
||||
});
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function isNewsCategoryFilterEnabled(filters, category) {
|
||||
const key = String(category || "").trim();
|
||||
if (!(key in DEFAULT_NEWS_CATEGORY_FILTERS)) return true;
|
||||
return normalizeNewsCategoryFilters(filters)[key] !== false;
|
||||
}
|
||||
|
||||
function getCurrentPanelVisibilitySnapshot() {
|
||||
return Object.fromEntries(
|
||||
HUD_PANEL_IDS.map((panelId) => {
|
||||
@@ -1382,6 +1432,9 @@ function getCurrentSharedSettingsSnapshot() {
|
||||
satelliteRealAltitudeEnabled: getSatelliteRealAltitudeEnabled(),
|
||||
interactableCompactDotsEnabled: getInteractableCompactDotsEnabled(),
|
||||
surfaceHoverInfoMode: getSurfaceHoverInfoMode(),
|
||||
newsCategoryFilters: normalizeNewsCategoryFilters(
|
||||
earthSettingsState?.shared?.newsCategoryFilters,
|
||||
),
|
||||
keyboardShortcuts: normalizeKeyboardShortcuts(keyboardShortcuts),
|
||||
};
|
||||
}
|
||||
@@ -1446,6 +1499,7 @@ function cloneEarthSettings(settings) {
|
||||
surfaceHoverInfoMode: normalizeSurfaceHoverInfoMode(
|
||||
settings.shared.surfaceHoverInfoMode,
|
||||
),
|
||||
newsCategoryFilters: normalizeNewsCategoryFilters(settings.shared.newsCategoryFilters),
|
||||
keyboardShortcuts: normalizeKeyboardShortcuts(settings.shared.keyboardShortcuts),
|
||||
layerVisibility: { ...(settings.shared.layerVisibility || {}) },
|
||||
},
|
||||
@@ -1593,6 +1647,13 @@ function normalizeEarthSettings(rawSettings, defaults) {
|
||||
(rawSettings?.version || 0) >= KEYBOARD_SHORTCUTS_DEFAULT_VERSION
|
||||
? normalizeKeyboardShortcuts(sharedSettings?.keyboardShortcuts)
|
||||
: defaults.shared.keyboardShortcuts;
|
||||
const legacyNewsCategoryFilters = sharedSettings?.["display" + "Types"]?.news;
|
||||
const nextNewsCategoryFilters =
|
||||
(rawSettings?.version || 0) >= NEWS_CATEGORY_FILTERS_DEFAULT_VERSION
|
||||
? normalizeNewsCategoryFilters(
|
||||
sharedSettings?.newsCategoryFilters || legacyNewsCategoryFilters,
|
||||
)
|
||||
: normalizeNewsCategoryFilters(defaults.shared.newsCategoryFilters);
|
||||
const nextCruiseQueueMode =
|
||||
(rawSettings?.version || 0) >= CRUISE_QUEUE_DEFAULT_VERSION
|
||||
? normalizeCruiseQueueMode(sharedSettings?.cruiseQueueMode)
|
||||
@@ -1633,6 +1694,7 @@ function normalizeEarthSettings(rawSettings, defaults) {
|
||||
satelliteRealAltitudeEnabled: nextSatelliteRealAltitudeEnabled,
|
||||
interactableCompactDotsEnabled: nextInteractableCompactDotsEnabled,
|
||||
surfaceHoverInfoMode: nextSurfaceHoverInfoMode,
|
||||
newsCategoryFilters: nextNewsCategoryFilters,
|
||||
keyboardShortcuts: nextKeyboardShortcuts,
|
||||
},
|
||||
views: {
|
||||
@@ -1919,6 +1981,71 @@ function syncInteractableCompactDotsToggle() {
|
||||
});
|
||||
}
|
||||
|
||||
function syncNewsCategoryFilterControls() {
|
||||
const filters = normalizeNewsCategoryFilters(
|
||||
earthSettingsState?.shared?.newsCategoryFilters,
|
||||
);
|
||||
document.querySelectorAll("[data-news-category-toggle]").forEach((button) => {
|
||||
if (!(button instanceof HTMLButtonElement)) return;
|
||||
const category = button.dataset.newsCategoryToggle || "";
|
||||
const active = isNewsCategoryFilterEnabled(filters, category);
|
||||
button.classList.toggle("is-active", active);
|
||||
button.setAttribute("aria-pressed", active ? "true" : "false");
|
||||
});
|
||||
}
|
||||
|
||||
function dispatchNewsCategoryFiltersChange(
|
||||
filters = earthSettingsState?.shared?.newsCategoryFilters,
|
||||
) {
|
||||
window.dispatchEvent(
|
||||
new CustomEvent("earth:news-category-filters-change", {
|
||||
detail: {
|
||||
categories: normalizeNewsCategoryFilters(filters),
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function applyNewsCategoryFilters(filters = earthSettingsState?.shared?.newsCategoryFilters) {
|
||||
const normalized = normalizeNewsCategoryFilters(filters);
|
||||
syncNewsCategoryFilterControls();
|
||||
dispatchNewsCategoryFiltersChange(normalized);
|
||||
}
|
||||
|
||||
export function getEarthNewsCategoryFilters() {
|
||||
return normalizeNewsCategoryFilters(earthSettingsState?.shared?.newsCategoryFilters);
|
||||
}
|
||||
|
||||
export function isEarthNewsCategoryEnabled(category) {
|
||||
return isNewsCategoryFilterEnabled(
|
||||
earthSettingsState?.shared?.newsCategoryFilters,
|
||||
category,
|
||||
);
|
||||
}
|
||||
|
||||
export function setEarthNewsCategoryEnabled(
|
||||
category,
|
||||
enabled,
|
||||
{ persist = true, suppressStatus = false } = {},
|
||||
) {
|
||||
const key = String(category || "").trim();
|
||||
if (!(key in DEFAULT_NEWS_CATEGORY_FILTERS)) return false;
|
||||
|
||||
ensureMutableEarthSettingsState();
|
||||
const nextFilters = normalizeNewsCategoryFilters(earthSettingsState.shared.newsCategoryFilters);
|
||||
nextFilters[key] = Boolean(enabled);
|
||||
earthSettingsState.shared.newsCategoryFilters = nextFilters;
|
||||
applyNewsCategoryFilters(nextFilters);
|
||||
|
||||
if (persist) {
|
||||
persistEarthSettings();
|
||||
}
|
||||
if (!suppressStatus) {
|
||||
showStatusMessage(Boolean(enabled) ? "新闻类型已显示" : "新闻类型已隐藏", "info");
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function syncSurfaceHoverInfoModeControls() {
|
||||
const activeMode = getSurfaceHoverInfoMode();
|
||||
document.querySelectorAll("[data-surface-hover-info-mode]").forEach((button) => {
|
||||
@@ -2188,8 +2315,7 @@ function setDefaultEarthZoom(nextZoom, { persist = true, applyToCurrentView = tr
|
||||
syncDefaultEarthZoomUi(defaultEarthZoom);
|
||||
|
||||
if (applyToCurrentView && activeCamera) {
|
||||
zoomLevel = defaultEarthZoom;
|
||||
applyZoom(activeCamera);
|
||||
setZoomLevel(defaultEarthZoom, activeCamera);
|
||||
}
|
||||
|
||||
if (persist) {
|
||||
@@ -2278,6 +2404,7 @@ async function applyEarthSettings(settings, { applyLayers = true } = {}) {
|
||||
const layerVisibility = { ...(settings.shared.layerVisibility || {}) };
|
||||
applyImmediateLayerVisibilityHints(layerVisibility);
|
||||
deferredLayerVisibilitySettings = layerVisibility;
|
||||
applyNewsCategoryFilters(settings.shared.newsCategoryFilters);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -2286,6 +2413,7 @@ async function applyEarthSettings(settings, { applyLayers = true } = {}) {
|
||||
persist: false,
|
||||
silent: true,
|
||||
});
|
||||
applyNewsCategoryFilters(settings.shared.newsCategoryFilters);
|
||||
}
|
||||
|
||||
export function getMotionDebugEnabled() {
|
||||
@@ -2398,6 +2526,7 @@ export async function applyDeferredLayerVisibilitySettings(options = {}) {
|
||||
silent: true,
|
||||
...options,
|
||||
});
|
||||
applyNewsCategoryFilters(earthSettingsState?.shared?.newsCategoryFilters);
|
||||
}
|
||||
|
||||
function resetEarthSettings() {
|
||||
@@ -3110,12 +3239,7 @@ export function applyImmediateView(targetEarthObj, camera, options = {}) {
|
||||
targetEarthObj.rotation.x = nextRotation.x;
|
||||
targetEarthObj.rotation.y = nextRotation.y;
|
||||
targetEarthObj.rotation.z = nextRotation.z;
|
||||
zoomLevel = zoom;
|
||||
|
||||
if (camera) {
|
||||
camera.position.z = CONFIG.defaultCameraZ / zoomLevel;
|
||||
updateZoomDisplay(zoomLevel, camera.position.z.toFixed(0));
|
||||
}
|
||||
setZoomLevel(zoom, camera);
|
||||
}
|
||||
|
||||
export function setZoomLevel(nextZoom, camera = activeCamera) {
|
||||
@@ -3127,13 +3251,16 @@ export function setZoomLevel(nextZoom, camera = activeCamera) {
|
||||
return zoomLevel;
|
||||
}
|
||||
|
||||
export function showZoomStatusCapsule({ force = false } = {}) {
|
||||
export function showZoomStatusCapsule({ force = false, zoom = null } = {}) {
|
||||
const now = Date.now();
|
||||
if (!force && now - lastZoomStatusUpdateTime < ZOOM_STATUS_UPDATE_INTERVAL_MS) {
|
||||
return;
|
||||
}
|
||||
lastZoomStatusUpdateTime = now;
|
||||
showGestureStatusMessage(`缩放 ${Math.round(zoomLevel * 100)}%`, "info");
|
||||
const currentZoom = Number.isFinite(Number(zoom))
|
||||
? clampEarthZoomLevel(zoom)
|
||||
: syncZoomLevelFromCamera(activeCamera);
|
||||
showGestureStatusMessage(`缩放 ${Math.round(currentZoom * 100)}%`, "info");
|
||||
}
|
||||
|
||||
function cancelSettingsSheetAnimation() {
|
||||
@@ -4266,6 +4393,19 @@ function setupSettingsControls() {
|
||||
});
|
||||
});
|
||||
|
||||
document.querySelectorAll("[data-news-category-toggle]").forEach((toggle) => {
|
||||
if (!(toggle instanceof HTMLButtonElement)) return;
|
||||
bindListener(toggle, "click", () => {
|
||||
const active = toggle.classList.contains("is-active");
|
||||
setEarthNewsCategoryEnabled(toggle.dataset.newsCategoryToggle, !active);
|
||||
});
|
||||
});
|
||||
|
||||
bindListener(window, "earth:set-news-category-enabled", (event) => {
|
||||
const detail = event.detail || {};
|
||||
setEarthNewsCategoryEnabled(detail.category, Boolean(detail.enabled));
|
||||
});
|
||||
|
||||
document.querySelectorAll("#toggle-daynight, [data-daynight-toggle]").forEach((dayNightToggle) => {
|
||||
if (!(dayNightToggle instanceof HTMLInputElement)) return;
|
||||
bindListener(dayNightToggle, "change", () => {
|
||||
@@ -4308,6 +4448,7 @@ function setupSettingsControls() {
|
||||
syncSatelliteIdleBreathingToggle();
|
||||
syncSatelliteRealAltitudeToggle();
|
||||
syncInteractableCompactDotsToggle();
|
||||
syncNewsCategoryFilterControls();
|
||||
syncSurfaceHoverInfoModeControls();
|
||||
syncDayNightToggle(dayNightEnabled);
|
||||
syncMotionDebugToggle(motionDebugEnabled);
|
||||
@@ -4692,27 +4833,25 @@ function setupZoomControls(camera) {
|
||||
const MAX_PERCENT = CONFIG.maxZoom * 100;
|
||||
|
||||
function doZoomStep(direction) {
|
||||
let currentPercent = Math.round(zoomLevel * 100);
|
||||
let currentPercent = Math.round(getZoomLevelFromCamera(camera) * 100);
|
||||
let newPercent =
|
||||
direction > 0 ? currentPercent + CLICK_STEP : currentPercent - CLICK_STEP;
|
||||
|
||||
if (newPercent > MAX_PERCENT) newPercent = MAX_PERCENT;
|
||||
if (newPercent < MIN_PERCENT) newPercent = MIN_PERCENT;
|
||||
|
||||
zoomLevel = newPercent / 100;
|
||||
applyZoom(camera);
|
||||
setZoomLevel(newPercent / 100, camera);
|
||||
showZoomStatusCapsule({ force: true });
|
||||
}
|
||||
|
||||
function doContinuousZoom(direction) {
|
||||
let currentPercent = Math.round(zoomLevel * 100);
|
||||
let currentPercent = Math.round(getZoomLevelFromCamera(camera) * 100);
|
||||
let newPercent = direction > 0 ? currentPercent + 1 : currentPercent - 1;
|
||||
|
||||
if (newPercent > MAX_PERCENT) newPercent = MAX_PERCENT;
|
||||
if (newPercent < MIN_PERCENT) newPercent = MIN_PERCENT;
|
||||
|
||||
zoomLevel = newPercent / 100;
|
||||
applyZoom(camera);
|
||||
setZoomLevel(newPercent / 100, camera);
|
||||
showZoomStatusCapsule();
|
||||
}
|
||||
|
||||
@@ -4775,10 +4914,8 @@ function setupZoomControls(camera) {
|
||||
bindListener(zoomOut, "touchend", () => handleMouseUp(-1));
|
||||
|
||||
bindListener(zoomValue, "click", () => {
|
||||
const startZoomVal = zoomLevel;
|
||||
const startZoomVal = getZoomLevelFromCamera(camera);
|
||||
const targetZoom = getDefaultEarthZoomLevel();
|
||||
const startDistance = CONFIG.defaultCameraZ / startZoomVal;
|
||||
const targetDistance = CONFIG.defaultCameraZ / targetZoom;
|
||||
|
||||
animateValue(
|
||||
0,
|
||||
@@ -4786,14 +4923,11 @@ function setupZoomControls(camera) {
|
||||
600,
|
||||
(progress) => {
|
||||
const ease = 1 - Math.pow(1 - progress, 3);
|
||||
zoomLevel = startZoomVal + (targetZoom - startZoomVal) * ease;
|
||||
camera.position.z = CONFIG.defaultCameraZ / zoomLevel;
|
||||
const distance =
|
||||
startDistance + (targetDistance - startDistance) * ease;
|
||||
updateZoomDisplay(zoomLevel, distance.toFixed(0));
|
||||
const nextZoom = startZoomVal + (targetZoom - startZoomVal) * ease;
|
||||
setZoomLevel(nextZoom, camera);
|
||||
},
|
||||
() => {
|
||||
zoomLevel = targetZoom;
|
||||
setZoomLevel(targetZoom, camera);
|
||||
showStatusMessage(getZoomResetStatusMessage(targetZoom), "info");
|
||||
},
|
||||
);
|
||||
@@ -4802,9 +4936,51 @@ function setupZoomControls(camera) {
|
||||
|
||||
function setupWheelZoom(camera, renderer) {
|
||||
let wheelZoomFrameId = null;
|
||||
let wheelZoomTarget = zoomLevel;
|
||||
let wheelZoomStart = zoomLevel;
|
||||
let wheelZoomTarget = getZoomLevelFromCamera(camera);
|
||||
let wheelZoomStart = wheelZoomTarget;
|
||||
let wheelZoomStartAt = 0;
|
||||
let lastTrackpadDirection = 0;
|
||||
let suppressedTrackpadDirection = 0;
|
||||
let suppressedTrackpadUntil = 0;
|
||||
let suppressedTrackpadMagnitude = 0;
|
||||
|
||||
function getWheelPixelDelta(event) {
|
||||
if (event.deltaMode === WheelEvent.DOM_DELTA_LINE) {
|
||||
return event.deltaY * 16;
|
||||
}
|
||||
if (event.deltaMode === WheelEvent.DOM_DELTA_PAGE) {
|
||||
return event.deltaY * window.innerHeight;
|
||||
}
|
||||
return event.deltaY;
|
||||
}
|
||||
|
||||
function isTrackpadWheel(event, pixelDelta) {
|
||||
return event.deltaMode === WheelEvent.DOM_DELTA_PIXEL &&
|
||||
Math.abs(pixelDelta) < WHEEL_TRACKPAD_PIXEL_THRESHOLD;
|
||||
}
|
||||
|
||||
function shouldSuppressTrackpadResidual(pixelDelta) {
|
||||
const direction = Math.sign(pixelDelta);
|
||||
const magnitude = Math.abs(pixelDelta);
|
||||
const now = performance.now();
|
||||
return direction !== 0 &&
|
||||
direction === suppressedTrackpadDirection &&
|
||||
now < suppressedTrackpadUntil &&
|
||||
magnitude < suppressedTrackpadMagnitude * WHEEL_TRACKPAD_RESIDUAL_RATIO;
|
||||
}
|
||||
|
||||
function recordTrackpadWheel(pixelDelta) {
|
||||
const direction = Math.sign(pixelDelta);
|
||||
const magnitude = Math.abs(pixelDelta);
|
||||
if (direction === 0) return;
|
||||
const now = performance.now();
|
||||
if (lastTrackpadDirection !== 0 && direction !== lastTrackpadDirection) {
|
||||
suppressedTrackpadDirection = lastTrackpadDirection;
|
||||
suppressedTrackpadUntil = now + WHEEL_TRACKPAD_RESIDUAL_WINDOW_MS;
|
||||
suppressedTrackpadMagnitude = magnitude;
|
||||
}
|
||||
lastTrackpadDirection = direction;
|
||||
}
|
||||
|
||||
function stopWheelZoomAnimation() {
|
||||
if (wheelZoomFrameId !== null) {
|
||||
@@ -4824,28 +5000,55 @@ function setupWheelZoom(camera, renderer) {
|
||||
1,
|
||||
);
|
||||
const ease = 1 - Math.pow(1 - progress, 3);
|
||||
zoomLevel = wheelZoomStart + (wheelZoomTarget - wheelZoomStart) * ease;
|
||||
applyZoom(camera);
|
||||
setZoomLevel(
|
||||
wheelZoomStart + (wheelZoomTarget - wheelZoomStart) * ease,
|
||||
camera,
|
||||
);
|
||||
|
||||
if (progress < 1) {
|
||||
wheelZoomFrameId = window.requestAnimationFrame(animateWheelZoom);
|
||||
return;
|
||||
}
|
||||
|
||||
zoomLevel = wheelZoomTarget;
|
||||
applyZoom(camera);
|
||||
setZoomLevel(wheelZoomTarget, camera);
|
||||
wheelZoomFrameId = null;
|
||||
wheelZoomStartAt = 0;
|
||||
}
|
||||
|
||||
function startWheelZoomAnimation() {
|
||||
wheelZoomStart = zoomLevel;
|
||||
wheelZoomStart = getZoomLevelFromCamera(camera);
|
||||
wheelZoomStartAt = 0;
|
||||
if (wheelZoomFrameId === null) {
|
||||
wheelZoomFrameId = window.requestAnimationFrame(animateWheelZoom);
|
||||
}
|
||||
}
|
||||
|
||||
function applyMouseWheelZoom(direction) {
|
||||
suppressedTrackpadDirection = 0;
|
||||
suppressedTrackpadUntil = 0;
|
||||
const baseZoom = wheelZoomFrameId === null
|
||||
? getZoomLevelFromCamera(camera)
|
||||
: wheelZoomTarget;
|
||||
wheelZoomTarget = clampEarthZoomLevel(
|
||||
baseZoom + direction * WHEEL_ZOOM_STEP,
|
||||
);
|
||||
stopWheelZoomAnimation();
|
||||
startWheelZoomAnimation();
|
||||
showZoomStatusCapsule({ force: true, zoom: wheelZoomTarget });
|
||||
}
|
||||
|
||||
function applyTrackpadWheelZoom(pixelDelta) {
|
||||
if (Math.abs(pixelDelta) < WHEEL_TRACKPAD_DEADZONE) return;
|
||||
if (shouldSuppressTrackpadResidual(pixelDelta)) return;
|
||||
stopWheelZoomAnimation();
|
||||
const currentZoom = getZoomLevelFromCamera(camera);
|
||||
const nextZoom = currentZoom * Math.exp(-pixelDelta * WHEEL_TRACKPAD_SENSITIVITY);
|
||||
wheelZoomTarget = setZoomLevel(nextZoom, camera);
|
||||
wheelZoomStart = wheelZoomTarget;
|
||||
recordTrackpadWheel(pixelDelta);
|
||||
showZoomStatusCapsule({ force: true, zoom: wheelZoomTarget });
|
||||
}
|
||||
|
||||
cleanupFns.push(stopWheelZoomAnimation);
|
||||
|
||||
bindListener(
|
||||
@@ -4853,25 +5056,17 @@ function setupWheelZoom(camera, renderer) {
|
||||
"wheel",
|
||||
(e) => {
|
||||
e.preventDefault();
|
||||
const direction = e.deltaY < 0 ? 1 : -1;
|
||||
const baseZoom =
|
||||
wheelZoomFrameId === null ? zoomLevel : wheelZoomTarget;
|
||||
wheelZoomTarget = clampEarthZoomLevel(
|
||||
baseZoom + direction * WHEEL_ZOOM_STEP,
|
||||
);
|
||||
startWheelZoomAnimation();
|
||||
showZoomStatusCapsule({ force: true });
|
||||
const pixelDelta = getWheelPixelDelta(e);
|
||||
if (isTrackpadWheel(e, pixelDelta)) {
|
||||
applyTrackpadWheelZoom(pixelDelta);
|
||||
return;
|
||||
}
|
||||
applyMouseWheelZoom(pixelDelta < 0 ? 1 : -1);
|
||||
},
|
||||
{ passive: false },
|
||||
);
|
||||
}
|
||||
|
||||
function applyZoom(camera) {
|
||||
camera.position.z = CONFIG.defaultCameraZ / zoomLevel;
|
||||
const distance = camera.position.z.toFixed(0);
|
||||
updateZoomDisplay(zoomLevel, distance);
|
||||
}
|
||||
|
||||
function animateValue(start, end, duration, onUpdate, onComplete) {
|
||||
const animationToken = ++focusViewAnimationToken;
|
||||
const startTime = performance.now();
|
||||
@@ -5800,7 +5995,7 @@ export function focusEarthView(camera, options = {}) {
|
||||
const startRotX = earthObj.rotation.x;
|
||||
const startRotY = earthObj.rotation.y;
|
||||
const startRotZ = earthObj.rotation.z;
|
||||
const startZoom = zoomLevel;
|
||||
const startZoom = getZoomLevelFromCamera(camera);
|
||||
const defaultZoom = getDefaultEarthZoomLevel();
|
||||
const shouldRestoreZoomViaDefault =
|
||||
zoomTransitionMode === "restore-current-via-default" &&
|
||||
@@ -5829,30 +6024,26 @@ export function focusEarthView(camera, options = {}) {
|
||||
if (progress < rotateStartProgress) {
|
||||
const zoomProgress = progress / rotateStartProgress;
|
||||
const zoomEase = 1 - Math.pow(1 - zoomProgress, 3);
|
||||
zoomLevel = startZoom + (defaultZoom - startZoom) * zoomEase;
|
||||
setZoomLevel(startZoom + (defaultZoom - startZoom) * zoomEase, camera);
|
||||
} else if (progress <= rotateEndProgress) {
|
||||
zoomLevel = defaultZoom;
|
||||
setZoomLevel(defaultZoom, camera);
|
||||
} else {
|
||||
const zoomProgress = (progress - rotateEndProgress) / (1 - rotateEndProgress);
|
||||
const zoomEase = 1 - Math.pow(1 - zoomProgress, 3);
|
||||
zoomLevel = defaultZoom + (startZoom - defaultZoom) * zoomEase;
|
||||
setZoomLevel(defaultZoom + (startZoom - defaultZoom) * zoomEase, camera);
|
||||
}
|
||||
} else {
|
||||
earthObj.rotation.x = startRotX + (nextRotation.x - startRotX) * ease;
|
||||
earthObj.rotation.y = startRotY + (nextRotation.y - startRotY) * ease;
|
||||
earthObj.rotation.z = startRotZ + (nextRotation.z - startRotZ) * ease;
|
||||
zoomLevel = startZoom + (zoom - startZoom) * ease;
|
||||
setZoomLevel(startZoom + (zoom - startZoom) * ease, camera);
|
||||
}
|
||||
camera.position.z = CONFIG.defaultCameraZ / zoomLevel;
|
||||
updateZoomDisplay(zoomLevel, camera.position.z.toFixed(0));
|
||||
},
|
||||
() => {
|
||||
zoomLevel = shouldRestoreZoomViaDefault ? startZoom : zoom;
|
||||
setZoomLevel(shouldRestoreZoomViaDefault ? startZoom : zoom, camera);
|
||||
earthObj.rotation.x = nextRotation.x;
|
||||
earthObj.rotation.y = nextRotation.y;
|
||||
earthObj.rotation.z = nextRotation.z;
|
||||
camera.position.z = CONFIG.defaultCameraZ / zoomLevel;
|
||||
updateZoomDisplay(zoomLevel, camera.position.z.toFixed(0));
|
||||
if (!suppressStatus) {
|
||||
showStatusMessage("视角已重置", "info");
|
||||
}
|
||||
@@ -5863,7 +6054,7 @@ export function focusEarthView(camera, options = {}) {
|
||||
}
|
||||
|
||||
export function getZoomLevel() {
|
||||
return zoomLevel;
|
||||
return syncZoomLevelFromCamera(activeCamera);
|
||||
}
|
||||
|
||||
export function getDefaultEarthZoomLevel() {
|
||||
|
||||
@@ -210,7 +210,7 @@ function boundaryLineRadius({ claim = false } = {}) {
|
||||
return (
|
||||
CONFIG.earthRadius +
|
||||
COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset +
|
||||
(claim ? 0.018 : 0)
|
||||
(claim ? COUNTRY_BOUNDARY_CONFIG.claimLineAltitudeOffset : 0)
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import { PATHS } from "./constants.js";
|
||||
import { createInteractableLayer, SURFACE_AVOIDANCE_PROFILES } from "./interactable.js";
|
||||
|
||||
let showEarthInteractables = true;
|
||||
const interactableRevisions = new Map();
|
||||
|
||||
function featureToInteractableItem(feature) {
|
||||
@@ -93,6 +92,9 @@ const earthInteractableLayer = createInteractableLayer({
|
||||
...item,
|
||||
type: "earth_interactable",
|
||||
}),
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
},
|
||||
});
|
||||
|
||||
export async function loadEarthInteractables(earth, { silent = false } = {}) {
|
||||
@@ -109,7 +111,7 @@ export async function loadEarthInteractables(earth, { silent = false } = {}) {
|
||||
items.forEach((item) => interactableRevisions.set(item.id, Number(item.revision || 0)));
|
||||
earthInteractableLayer.setData(items);
|
||||
earthInteractableLayer.attach(earth);
|
||||
earthInteractableLayer.setVisible(showEarthInteractables);
|
||||
earthInteractableLayer.setVisible(true);
|
||||
if (!silent) {
|
||||
console.info("Earth interactables loaded", { count: items.length });
|
||||
}
|
||||
@@ -156,7 +158,7 @@ export function applyEarthInteractableEvent(earth, payload = {}) {
|
||||
const changed = earthInteractableLayer.upsertItem(item);
|
||||
interactableRevisions.set(id, nextRevision || Number(item.revision || 0));
|
||||
earthInteractableLayer.attach(earth);
|
||||
earthInteractableLayer.setVisible(showEarthInteractables);
|
||||
earthInteractableLayer.setVisible(true);
|
||||
return changed;
|
||||
}
|
||||
|
||||
|
||||
@@ -287,10 +287,28 @@ function renderNewsCardContent(content, data) {
|
||||
if (!(content instanceof HTMLElement)) return;
|
||||
const summary = getNewsCardSummary(data);
|
||||
const title = getNewsCardTitle(data);
|
||||
const metaItems = [
|
||||
['媒体来源', data?.source],
|
||||
['RSS 来源', data?.feedName],
|
||||
['源类型', data?.feedSourceTypeLabel],
|
||||
['抓取通道', data?.fetchChannelLabel],
|
||||
['新闻类型', data?.categoryLabel],
|
||||
['区域', data?.regionLabel],
|
||||
['发布时间', data?.publishedAtDisplay],
|
||||
].filter(([, value]) => String(value ?? '').trim());
|
||||
const metaHtml = metaItems.length
|
||||
? `<div class="info-card-news-meta-grid">${metaItems.map(([label, value]) => `
|
||||
<div class="info-card-news-meta-item">
|
||||
<span>${escapeInfoCardHtml(label)}</span>
|
||||
<strong>${escapeInfoCardHtml(value)}</strong>
|
||||
</div>
|
||||
`).join('')}</div>`
|
||||
: '';
|
||||
content.innerHTML = `
|
||||
<div class="info-card-news-layout">
|
||||
<div class="info-card-news-kicker">新闻信号</div>
|
||||
<div class="info-card-news-title">${escapeInfoCardHtml(title)}</div>
|
||||
${metaHtml}
|
||||
<div class="info-card-news-summary-shell">
|
||||
<div class="info-card-news-summary-label">概要</div>
|
||||
<div class="info-card-news-summary" data-news-summary></div>
|
||||
@@ -305,10 +323,25 @@ function renderMobileNewsCardContent(content, data) {
|
||||
if (!(content instanceof HTMLElement)) return;
|
||||
const summary = getNewsCardSummary(data);
|
||||
const title = getNewsCardTitle(data);
|
||||
const metaItems = [
|
||||
['来源', data?.source],
|
||||
['RSS', data?.feedName],
|
||||
['类型', data?.categoryLabel],
|
||||
['区域', data?.regionLabel],
|
||||
].filter(([, value]) => String(value ?? '').trim());
|
||||
const metaHtml = metaItems.length
|
||||
? `<div class="info-card-news-meta-grid info-card-news-meta-grid--mobile">${metaItems.map(([label, value]) => `
|
||||
<div class="info-card-news-meta-item">
|
||||
<span>${escapeInfoCardHtml(label)}</span>
|
||||
<strong>${escapeInfoCardHtml(value)}</strong>
|
||||
</div>
|
||||
`).join('')}</div>`
|
||||
: '';
|
||||
content.innerHTML = `
|
||||
<div class="earth-mobile-news-detail">
|
||||
<div class="earth-mobile-news-detail-kicker">新闻信号</div>
|
||||
<div class="earth-mobile-news-detail-title">${escapeInfoCardHtml(title)}</div>
|
||||
${metaHtml}
|
||||
<div class="earth-mobile-news-detail-summary-shell">
|
||||
<div class="earth-mobile-news-detail-summary-label">概要</div>
|
||||
<div class="earth-mobile-news-detail-summary" data-news-summary></div>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -15,12 +15,18 @@ import {
|
||||
SCENE_LIGHT_CONFIG,
|
||||
SURFACE_HOVER_INFO_MODES,
|
||||
} from "./constants.js";
|
||||
import {
|
||||
canAttemptEarthRealtime,
|
||||
getEarthRealtimeCooldownMs,
|
||||
getEarthRealtimeUrl,
|
||||
recordEarthRealtimeFailure,
|
||||
recordEarthRealtimeOpen,
|
||||
} from "./realtime.js";
|
||||
import { vector3ToLatLon, screenToEarthCoords } from "./utils.js";
|
||||
import {
|
||||
showStatusMessage,
|
||||
queueStatusMessage,
|
||||
updateCoordinatesDisplay,
|
||||
updateZoomDisplay,
|
||||
updateEarthStats,
|
||||
setEarthStatValue,
|
||||
setLoading,
|
||||
@@ -301,6 +307,8 @@ export let scene;
|
||||
export let camera;
|
||||
export let renderer;
|
||||
|
||||
const WEBGL_GPU_DIAGNOSTICS_EVENT = "earth:gpu-diagnostics";
|
||||
|
||||
let isDragging = false;
|
||||
let previousMousePosition = { x: 0, y: 0 };
|
||||
let targetRotation = { x: 0, y: 0 };
|
||||
@@ -451,6 +459,29 @@ function getViewportAspect() {
|
||||
return window.innerWidth / window.innerHeight;
|
||||
}
|
||||
|
||||
function getWebGLGpuDiagnostics(activeRenderer) {
|
||||
const gl = activeRenderer?.getContext?.();
|
||||
if (!gl) return null;
|
||||
const debugInfo = gl.getExtension?.("WEBGL_debug_renderer_info");
|
||||
const attrs = gl.getContextAttributes?.() || {};
|
||||
return {
|
||||
requestedPowerPreference: "high-performance",
|
||||
contextPowerPreference: attrs.powerPreference || null,
|
||||
vendor: debugInfo ? gl.getParameter(debugInfo.UNMASKED_VENDOR_WEBGL) : gl.getParameter(gl.VENDOR),
|
||||
renderer: debugInfo ? gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL) : gl.getParameter(gl.RENDERER),
|
||||
antialias: Boolean(attrs.antialias),
|
||||
alpha: Boolean(attrs.alpha),
|
||||
};
|
||||
}
|
||||
|
||||
function publishWebGLGpuDiagnostics(activeRenderer) {
|
||||
const diagnostics = getWebGLGpuDiagnostics(activeRenderer);
|
||||
if (!diagnostics) return;
|
||||
window.__planetEarthGpu = diagnostics;
|
||||
console.info("[智能星球] WebGL GPU diagnostics", diagnostics);
|
||||
window.dispatchEvent(new CustomEvent(WEBGL_GPU_DIAGNOSTICS_EVENT, { detail: diagnostics }));
|
||||
}
|
||||
|
||||
function syncRendererViewport() {
|
||||
if (!camera || !renderer) return;
|
||||
camera.aspect = getViewportAspect();
|
||||
@@ -793,12 +824,30 @@ function isSameVessel(marker1, marker2) {
|
||||
return Boolean(marker1 && marker2 && marker1.userData?.mmsi === marker2.userData?.mmsi);
|
||||
}
|
||||
|
||||
function getFirstObjectIntersection(intersections) {
|
||||
return intersections.find((hit) => !hit?.cluster && hit?.object)?.object || null;
|
||||
}
|
||||
|
||||
function getPrimaryClusterHit(...intersectionGroups) {
|
||||
return intersectionGroups
|
||||
.flat()
|
||||
.filter((hit) => hit?.cluster && Number(hit.clusterCount) > 1)
|
||||
.sort((a, b) => a.distancePxSq - b.distancePxSq)[0] || null;
|
||||
}
|
||||
|
||||
function getClusterBriefHtml(hit) {
|
||||
const count = Number(hit?.clusterCount || hit?.clusterMarkers?.length || 0);
|
||||
return `<strong>共 ${count} 个对象</strong><br><span>放大后可查看单个图标</span>`;
|
||||
}
|
||||
|
||||
function getPrimaryBGPHoverTarget(bgpAnomalyIntersects, bgpCollectorIntersects) {
|
||||
if (bgpAnomalyIntersects.length > 0) {
|
||||
return bgpAnomalyIntersects[0].object;
|
||||
const anomalyMarker = getFirstObjectIntersection(bgpAnomalyIntersects);
|
||||
if (anomalyMarker) {
|
||||
return anomalyMarker;
|
||||
}
|
||||
if (bgpCollectorIntersects.length > 0) {
|
||||
return bgpCollectorIntersects[0].object;
|
||||
const collectorMarker = getFirstObjectIntersection(bgpCollectorIntersects);
|
||||
if (collectorMarker) {
|
||||
return collectorMarker;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -808,8 +857,8 @@ function getPrimaryBGPClickTarget(
|
||||
bgpAnomalyIntersects,
|
||||
bgpCollectorIntersects,
|
||||
) {
|
||||
const anomalyMarker = bgpAnomalyIntersects[0]?.object || null;
|
||||
const collectorMarker = bgpCollectorIntersects[0]?.object || null;
|
||||
const anomalyMarker = getFirstObjectIntersection(bgpAnomalyIntersects);
|
||||
const collectorMarker = getFirstObjectIntersection(bgpCollectorIntersects);
|
||||
if (!anomalyMarker && !collectorMarker) return null;
|
||||
if (!anomalyMarker) return collectorMarker;
|
||||
if (!collectorMarker) return anomalyMarker;
|
||||
@@ -952,19 +1001,23 @@ function getMotionIconCandidates() {
|
||||
const candidates = [];
|
||||
if (getShowBGP()) {
|
||||
getBGPEventIconPointerIntersections(sharedOptions).forEach((hit) => {
|
||||
if (hit.cluster || !hit.object) return;
|
||||
candidates.push({ type: "bgp", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq });
|
||||
});
|
||||
getBGPCollectorIconPointerIntersections(sharedOptions).forEach((hit) => {
|
||||
if (hit.cluster || !hit.object) return;
|
||||
candidates.push({ type: "bgp_collector", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq });
|
||||
});
|
||||
}
|
||||
if (getShowComputeCenters()) {
|
||||
getComputeCenterIconPointerIntersections(sharedOptions).forEach((hit) => {
|
||||
if (hit.cluster || !hit.object) return;
|
||||
candidates.push({ type: "compute_center", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq });
|
||||
});
|
||||
}
|
||||
if (getShowVessels()) {
|
||||
getVesselIconPointerIntersections(sharedOptions).forEach((hit) => {
|
||||
if (hit.cluster || !hit.object) return;
|
||||
candidates.push({ type: "vessel", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq });
|
||||
});
|
||||
}
|
||||
@@ -3961,7 +4014,7 @@ export function init() {
|
||||
0.1,
|
||||
5000,
|
||||
);
|
||||
camera.position.z = CONFIG.defaultCameraZ;
|
||||
setZoomLevel(getDefaultEarthZoomLevel(), camera);
|
||||
setSatelliteCamera(camera);
|
||||
|
||||
renderer = new THREE.WebGLRenderer({
|
||||
@@ -3972,6 +4025,7 @@ export function init() {
|
||||
syncRendererViewport();
|
||||
renderer.setClearColor(0x02040a, 1);
|
||||
renderer.setPixelRatio(window.devicePixelRatio);
|
||||
publishWebGLGpuDiagnostics(renderer);
|
||||
|
||||
const container = document.getElementById("container");
|
||||
if (container) {
|
||||
@@ -4240,17 +4294,13 @@ async function hydrateAllSatellitesInBackground(guardFn) {
|
||||
}
|
||||
}
|
||||
|
||||
function getEarthUpdatesRealtimeUrl() {
|
||||
const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
|
||||
return `${protocol}//${window.location.host}/ws`;
|
||||
}
|
||||
|
||||
function scheduleEarthUpdatesReconnect() {
|
||||
if (earthUpdatesReconnectTimer || destroyed) return;
|
||||
const delay = Math.max(EARTH_UPDATES_RECONNECT_DELAY_MS, getEarthRealtimeCooldownMs());
|
||||
earthUpdatesReconnectTimer = window.setTimeout(() => {
|
||||
earthUpdatesReconnectTimer = null;
|
||||
connectEarthUpdatesRealtime();
|
||||
}, EARTH_UPDATES_RECONNECT_DELAY_MS);
|
||||
}, delay);
|
||||
}
|
||||
|
||||
async function refreshBGPRealtimeLayer() {
|
||||
@@ -4519,10 +4569,16 @@ function startEarthDataReconciliation() {
|
||||
}
|
||||
|
||||
function connectEarthUpdatesRealtime() {
|
||||
if (earthUpdatesSocket || typeof WebSocket === "undefined" || destroyed) return;
|
||||
const socket = new WebSocket(getEarthUpdatesRealtimeUrl());
|
||||
if (earthUpdatesSocket || destroyed) return;
|
||||
if (!canAttemptEarthRealtime()) {
|
||||
scheduleEarthUpdatesReconnect();
|
||||
return;
|
||||
}
|
||||
const socket = new WebSocket(getEarthRealtimeUrl());
|
||||
earthUpdatesSocket = socket;
|
||||
socket.onopen = () => {
|
||||
socket.__planetOpened = true;
|
||||
recordEarthRealtimeOpen();
|
||||
if (earthUpdatesReconnectTimer) {
|
||||
window.clearTimeout(earthUpdatesReconnectTimer);
|
||||
earthUpdatesReconnectTimer = null;
|
||||
@@ -4552,6 +4608,9 @@ function connectEarthUpdatesRealtime() {
|
||||
if (earthUpdatesSocket === socket) {
|
||||
earthUpdatesSocket = null;
|
||||
}
|
||||
if (!socket.__planetOpened) {
|
||||
recordEarthRealtimeFailure();
|
||||
}
|
||||
scheduleEarthUpdatesReconnect();
|
||||
};
|
||||
socket.onerror = () => {
|
||||
@@ -5287,13 +5346,17 @@ function onMouseMove(event) {
|
||||
bgpAnomalyIntersects,
|
||||
bgpCollectorIntersects,
|
||||
);
|
||||
const hoveredClusterHit = getPrimaryClusterHit(
|
||||
bgpAnomalyIntersects,
|
||||
bgpCollectorIntersects,
|
||||
computeCenterIntersects,
|
||||
);
|
||||
|
||||
if (hoveredBGP && !isSameBGPMarker(hoveredBGP, hoveredBGPMarker)) {
|
||||
clearTransientHoverState();
|
||||
}
|
||||
|
||||
const hoveredComputeCenterMarker =
|
||||
computeCenterIntersects.length > 0 ? computeCenterIntersects[0].object : null;
|
||||
const hoveredComputeCenterMarker = getFirstObjectIntersection(computeCenterIntersects);
|
||||
const hoveredVesselMarker =
|
||||
vesselPick.checked && vesselIntersects.length > 0 ? vesselIntersects[0].object : null;
|
||||
const earthPoint = screenToEarthCoords(
|
||||
@@ -5357,6 +5420,20 @@ function onMouseMove(event) {
|
||||
let objectTooltipShown = false;
|
||||
|
||||
if (
|
||||
hoveredClusterHit &&
|
||||
!hoveredBGPMarker &&
|
||||
!hoveredComputeCenterMarker &&
|
||||
lockedObjectType !== "bgp" &&
|
||||
lockedObjectType !== "bgp_collector" &&
|
||||
lockedObjectType !== "compute_center"
|
||||
) {
|
||||
showTooltip(
|
||||
event.clientX + TOOLTIP_CURSOR_OFFSET,
|
||||
event.clientY + TOOLTIP_CURSOR_OFFSET,
|
||||
getClusterBriefHtml(hoveredClusterHit),
|
||||
);
|
||||
objectTooltipShown = true;
|
||||
} else if (
|
||||
hoveredBGPMarker &&
|
||||
getShowBGP() &&
|
||||
lockedObjectType !== "bgp" &&
|
||||
@@ -5663,9 +5740,7 @@ function onClick(event) {
|
||||
const clickedBGPMarker = getShowBGP()
|
||||
? getPrimaryBGPClickTarget(event, bgpAnomalyIntersects, bgpCollectorIntersects)
|
||||
: null;
|
||||
const clickedComputeCenterMarker = computeCenterIntersects.length > 0
|
||||
? computeCenterIntersects[0].object
|
||||
: null;
|
||||
const clickedComputeCenterMarker = getFirstObjectIntersection(computeCenterIntersects);
|
||||
const clickedVesselMarker = vesselIntersects.length > 0
|
||||
? vesselIntersects[0].object
|
||||
: null;
|
||||
|
||||
@@ -16,9 +16,12 @@ import {
|
||||
import {
|
||||
getNewsDisplaySummary,
|
||||
getNewsDisplayTitle,
|
||||
getNewsCategoryLabel,
|
||||
getNewsFetchChannelLabel,
|
||||
getNewsFeedLabel,
|
||||
getNewsLocationSourceLabel,
|
||||
getNewsRegionLabel,
|
||||
getNewsSourceTypeLabel,
|
||||
} from "./news-locale.js";
|
||||
|
||||
const CRUISE_PRESENTATION_HIDE_MS = 220;
|
||||
@@ -89,6 +92,9 @@ function mapNewsItemToCruiseEvent(item) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const rawFeedName = item.feed_name || "";
|
||||
const sourceType = item.source_type || (String(rawFeedName).startsWith("Global Monitor /") ? "aggregated" : "rss");
|
||||
|
||||
return {
|
||||
id: `news:${item.id}`,
|
||||
sourceId: item.id,
|
||||
@@ -96,7 +102,11 @@ function mapNewsItemToCruiseEvent(item) {
|
||||
title: getNewsDisplayTitle(item),
|
||||
summary: getNewsDisplaySummary(item),
|
||||
source: item.source || "",
|
||||
feedName: getNewsFeedLabel(item.feed_name),
|
||||
feedName: getNewsFeedLabel(rawFeedName),
|
||||
rawFeedName,
|
||||
feedSourceTypeLabel: getNewsSourceTypeLabel(sourceType),
|
||||
fetchChannelLabel: getNewsFetchChannelLabel(rawFeedName, sourceType),
|
||||
categoryLabel: getNewsCategoryLabel(item.category),
|
||||
region: item.region || "global",
|
||||
regionLabel: item.display_region || getNewsRegionLabel(item.region),
|
||||
url: item.url || "",
|
||||
@@ -275,6 +285,13 @@ export function createNewsCruiseAdapter({ camera, earth, connector, focusView })
|
||||
showInfoCard("news", {
|
||||
title: item.title,
|
||||
summary: item.summary || item.title || "",
|
||||
source: item.source,
|
||||
feedName: item.rawFeedName || item.feedName,
|
||||
feedSourceTypeLabel: item.feedSourceTypeLabel,
|
||||
fetchChannelLabel: item.fetchChannelLabel,
|
||||
categoryLabel: item.categoryLabel,
|
||||
regionLabel: item.regionLabel,
|
||||
publishedAtDisplay: item.publishedAtDisplay,
|
||||
}, {
|
||||
x: placement.x,
|
||||
y: placement.y,
|
||||
@@ -320,6 +337,13 @@ export function createNewsCruiseAdapter({ camera, earth, connector, focusView })
|
||||
showInfoCard("news", {
|
||||
title: item.title,
|
||||
summary: item.summary || item.title || "",
|
||||
source: item.source,
|
||||
feedName: item.rawFeedName || item.feedName,
|
||||
feedSourceTypeLabel: item.feedSourceTypeLabel,
|
||||
fetchChannelLabel: item.fetchChannelLabel,
|
||||
categoryLabel: item.categoryLabel,
|
||||
regionLabel: item.regionLabel,
|
||||
publishedAtDisplay: item.publishedAtDisplay,
|
||||
}, {
|
||||
x: placement.x,
|
||||
y: placement.y,
|
||||
|
||||
@@ -9,11 +9,33 @@ const REGION_LABELS = {
|
||||
};
|
||||
|
||||
const FEED_LABELS = {
|
||||
"Global Monitor / World": "全球监测",
|
||||
"Global Monitor / Americas": "美洲监测",
|
||||
"Global Monitor / Europe": "欧洲监测",
|
||||
"Global Monitor / MEA": "中东与非洲监测",
|
||||
"Global Monitor / APAC": "亚太监测",
|
||||
"Global Monitor / World": "区域监测",
|
||||
"Global Monitor / Americas": "区域监测",
|
||||
"Global Monitor / Europe": "区域监测",
|
||||
"Global Monitor / MEA": "区域监测",
|
||||
"Global Monitor / APAC": "区域监测",
|
||||
};
|
||||
|
||||
const CATEGORY_LABELS = {
|
||||
politics: "政治",
|
||||
business: "商业",
|
||||
ecommerce: "电商",
|
||||
finance: "金融",
|
||||
sports: "体育",
|
||||
technology: "科技",
|
||||
military: "军事",
|
||||
disaster: "灾害",
|
||||
energy: "能源",
|
||||
society: "社会",
|
||||
culture: "文化",
|
||||
other: "其他",
|
||||
};
|
||||
|
||||
const SOURCE_TYPE_LABELS = {
|
||||
rss: "RSS",
|
||||
atom: "Atom",
|
||||
aggregated: "Aggregated",
|
||||
reference: "Reference",
|
||||
};
|
||||
|
||||
const LOCATION_SOURCE_LABELS = {
|
||||
@@ -94,10 +116,7 @@ export function isNewsContentReady(item, locale = DEFAULT_LOCALE) {
|
||||
const localized = getLocalization(item, locale);
|
||||
const title = normalizeText(item?.display_title || localized.title);
|
||||
const summary = normalizeText(item?.display_summary || localized.summary);
|
||||
return (
|
||||
(item?.enrichment_status === "success" || item?.enrichment_status === "content_only")
|
||||
&& Boolean(title && summary)
|
||||
);
|
||||
return Boolean(title && summary);
|
||||
}
|
||||
|
||||
export function getNewsRegionLabel(region, fallback = "") {
|
||||
@@ -108,10 +127,39 @@ export function getNewsFeedLabel(feedName) {
|
||||
return FEED_LABELS[feedName] || feedName || "聚合源";
|
||||
}
|
||||
|
||||
export function getNewsCategoryLabel(category) {
|
||||
return CATEGORY_LABELS[category] || category || CATEGORY_LABELS.other;
|
||||
}
|
||||
|
||||
export function getNewsSourceTypeLabel(sourceType) {
|
||||
const normalized = normalizeText(sourceType).toLowerCase();
|
||||
return SOURCE_TYPE_LABELS[normalized] || sourceType || "RSS";
|
||||
}
|
||||
|
||||
export function isRegionalMonitorFeed(feedName) {
|
||||
return Object.prototype.hasOwnProperty.call(FEED_LABELS, feedName);
|
||||
}
|
||||
|
||||
export function getNewsFetchChannelLabel(feedName, sourceType = "") {
|
||||
const normalized = normalizeText(sourceType).toLowerCase();
|
||||
if (isRegionalMonitorFeed(feedName) || normalized === "aggregated") return "区域监测";
|
||||
if (normalized === "atom") return "单源 Atom";
|
||||
if (normalized === "reference") return "配置保留";
|
||||
return "单源 RSS";
|
||||
}
|
||||
|
||||
export function getNewsLocationSourceLabel(source) {
|
||||
return LOCATION_SOURCE_LABELS[source] || source || "位置来源";
|
||||
}
|
||||
|
||||
export function getNewsEnrichmentStatusLabel(status) {
|
||||
export function getNewsEnrichmentStatusLabel(statusOrItem) {
|
||||
const item = statusOrItem && typeof statusOrItem === "object" ? statusOrItem : null;
|
||||
const status = item ? item.enrichment_status : statusOrItem;
|
||||
const language = String(item?.content_language || "").toLowerCase();
|
||||
if (language.startsWith("zh") && status !== "success" && status !== "content_only") {
|
||||
if (status === "queued" || status === "attempted") return "英文补译中";
|
||||
if (status === "provider_error" || status === "parse_error" || status === "no_result") return "中文原文";
|
||||
return "中文原文";
|
||||
}
|
||||
return ENRICHMENT_STATUS_LABELS[status] || status || "增强状态";
|
||||
}
|
||||
|
||||
@@ -2,11 +2,20 @@ import { showStatusMessage } from "./ui.js";
|
||||
import {
|
||||
getNewsDisplaySummary,
|
||||
getNewsDisplayTitle,
|
||||
getNewsCategoryLabel,
|
||||
getNewsEnrichmentStatusLabel,
|
||||
getNewsFeedLabel,
|
||||
getNewsFetchChannelLabel,
|
||||
getNewsRegionLabel,
|
||||
getNewsSourceTypeLabel,
|
||||
isNewsContentReady,
|
||||
} from "./news-locale.js";
|
||||
import {
|
||||
canAttemptEarthRealtime,
|
||||
getEarthRealtimeCooldownMs,
|
||||
getEarthRealtimeUrl,
|
||||
recordEarthRealtimeFailure,
|
||||
recordEarthRealtimeOpen,
|
||||
} from "./realtime.js";
|
||||
|
||||
// Desktop news has two surfaces:
|
||||
// - a persistent top ticker
|
||||
@@ -23,6 +32,9 @@ const NEWS_HUD_MIN_WIDTH_PX = 420;
|
||||
const NEWS_HUD_MIN_HEIGHT_PX = 360;
|
||||
const NEWS_HUD_RESIZE_MARGIN_PX = 12;
|
||||
const NEWS_REALTIME_RECONNECT_MS = 5000;
|
||||
const NEWS_SUMMARY_LIMIT = 12;
|
||||
const NEWS_FULL_LIMIT = 50;
|
||||
const NEWS_SOURCE_FILTER_STORAGE_KEY = "planet.earth.newsSourceFilters.v1";
|
||||
|
||||
let initialized = false;
|
||||
let refreshPromise = null;
|
||||
@@ -34,6 +46,54 @@ let selectedCruiseStoryId = null;
|
||||
let morphTimer = null;
|
||||
let newsRealtimeSocket = null;
|
||||
let newsRealtimeReconnectTimer = null;
|
||||
let activeNewsCategoryFilters = null;
|
||||
let lastCategorySignature = "";
|
||||
let activeNewsSourceFilters = loadNewsSourceFilters();
|
||||
let lastSourceSignature = "";
|
||||
let newsFullListMode = false;
|
||||
let activeFilterPopover = null;
|
||||
|
||||
const NEWS_CATEGORY_ALIASES = {
|
||||
politics: ["politics", "political", "policy", "政府", "政治", "政策", "政务"],
|
||||
business: ["business", "economy", "economic", "commerce", "商业", "经济", "产业", "企业"],
|
||||
ecommerce: ["ecommerce", "e-commerce", "online_retail", "retail_online", "电商", "电子商务", "网上零售", "跨境电商"],
|
||||
finance: ["finance", "financial", "market", "stock", "金融", "财经", "市场", "证券"],
|
||||
sports: ["sports", "sport", "体育"],
|
||||
technology: ["technology", "tech", "science", "科技", "科学", "技术", "ai", "人工智能"],
|
||||
military: ["military", "defense", "war", "军事", "防务", "战争"],
|
||||
disaster: ["disaster", "emergency", "earthquake", "flood", "storm", "灾害", "灾难", "应急", "地震", "洪水"],
|
||||
energy: ["energy", "oil", "gas", "power", "能源", "石油", "天然气", "电力"],
|
||||
society: ["society", "social", "社会", "民生"],
|
||||
culture: ["culture", "arts", "entertainment", "文化", "艺术", "娱乐"],
|
||||
other: ["other", "general", "misc", "其他", "综合"],
|
||||
};
|
||||
|
||||
function loadNewsSourceFilters() {
|
||||
try {
|
||||
const raw = window.localStorage?.getItem(NEWS_SOURCE_FILTER_STORAGE_KEY);
|
||||
if (!raw) return null;
|
||||
const parsed = JSON.parse(raw);
|
||||
if (!Array.isArray(parsed?.enabled)) return null;
|
||||
return parsed.enabled.map((id) => String(id || "").trim()).filter(Boolean);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function persistNewsSourceFilters(enabledIds) {
|
||||
try {
|
||||
if (!Array.isArray(enabledIds)) {
|
||||
window.localStorage?.removeItem(NEWS_SOURCE_FILTER_STORAGE_KEY);
|
||||
return;
|
||||
}
|
||||
window.localStorage?.setItem(
|
||||
NEWS_SOURCE_FILTER_STORAGE_KEY,
|
||||
JSON.stringify({ enabled: enabledIds }),
|
||||
);
|
||||
} catch {
|
||||
// Ignore localStorage failures; source filters are display-only preferences.
|
||||
}
|
||||
}
|
||||
|
||||
function getElements() {
|
||||
const isMobile = document.body.classList.contains("layout-mode-mobile");
|
||||
@@ -53,6 +113,9 @@ function getElements() {
|
||||
tickerTrack: document.getElementById("news-ticker-track"),
|
||||
hud: document.getElementById("news-hud-panel"),
|
||||
hudCloseBtn: document.getElementById("news-hud-close"),
|
||||
filterPopovers: document.querySelectorAll("[data-news-filter-popover]"),
|
||||
filterToggles: document.querySelectorAll("[data-news-filter-toggle]"),
|
||||
viewAllToggles: document.querySelectorAll("#news-view-all-toggle, #mobile-news-view-all-toggle"),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -196,6 +259,11 @@ function closeNewsHud() {
|
||||
setNewsHudOpen(false);
|
||||
}
|
||||
|
||||
function isNewsHudOpen() {
|
||||
const { hud } = getElements();
|
||||
return hud instanceof HTMLElement && !hud.classList.contains("hud-panel-hidden");
|
||||
}
|
||||
|
||||
function setupNewsHudResize() {
|
||||
const { hud } = getElements();
|
||||
const container = document.getElementById("container");
|
||||
@@ -298,7 +366,221 @@ function escapeNewsHtml(value) {
|
||||
}
|
||||
|
||||
function getDisplayableNewsItems(items) {
|
||||
return Array.isArray(items) ? items.filter(isNewsContentReady) : [];
|
||||
return Array.isArray(items)
|
||||
? items.filter(isNewsContentReady)
|
||||
: [];
|
||||
}
|
||||
|
||||
function normalizeNewsSourceType(value) {
|
||||
const normalized = String(value ?? "").trim().toLowerCase();
|
||||
return normalized || "";
|
||||
}
|
||||
|
||||
function getNewsSourceDescriptor(item, sourcesByName, sourcesById) {
|
||||
const feedName = String(item?.feed_name || "").trim();
|
||||
const sourceName = String(item?.source || feedName || "NEWS").trim();
|
||||
const sourceId = String(item?.source_id || "").trim();
|
||||
const sourceConfig = sourcesById.get(sourceId) || sourcesByName.get(feedName) || null;
|
||||
const sourceType = normalizeNewsSourceType(item?.source_type || sourceConfig?.source_type)
|
||||
|| (feedName.startsWith("Global Monitor /") ? "aggregated" : "rss");
|
||||
const sourceTypeLabel = getNewsSourceTypeLabel(sourceType);
|
||||
const channelLabel = getNewsFetchChannelLabel(feedName, sourceType);
|
||||
const sourceGroupName = String(sourceConfig?.name || "").trim();
|
||||
const originLabel = sourceGroupName
|
||||
? `${sourceGroupName} · ${channelLabel}`
|
||||
: feedName && feedName !== sourceName
|
||||
? `${feedName} · ${sourceTypeLabel}`
|
||||
: `${channelLabel} · ${sourceTypeLabel}`;
|
||||
const tooltip = [
|
||||
`媒体来源:${sourceName}`,
|
||||
sourceGroupName ? `来源组:${sourceGroupName}` : "",
|
||||
feedName ? `RSS 来源:${feedName}` : "",
|
||||
`源类型:${sourceTypeLabel}`,
|
||||
`抓取通道:${channelLabel}`,
|
||||
].filter(Boolean).join("\n");
|
||||
return {
|
||||
sourceName,
|
||||
feedName,
|
||||
sourceType,
|
||||
sourceTypeLabel,
|
||||
channelLabel,
|
||||
originLabel,
|
||||
tooltip,
|
||||
};
|
||||
}
|
||||
|
||||
function getEnabledNewsCategoryKeys(filters = activeNewsCategoryFilters) {
|
||||
if (!filters || typeof filters !== "object") return [];
|
||||
return Object.entries(filters)
|
||||
.filter(([, enabled]) => enabled !== false)
|
||||
.map(([key]) => key)
|
||||
.filter((key) => Object.prototype.hasOwnProperty.call(NEWS_CATEGORY_ALIASES, key))
|
||||
.sort();
|
||||
}
|
||||
|
||||
function getNewsCategorySignature(filters = activeNewsCategoryFilters) {
|
||||
const enabled = getEnabledNewsCategoryKeys(filters);
|
||||
const total = Object.keys(NEWS_CATEGORY_ALIASES).length;
|
||||
if (enabled.length === 0) return "__none__";
|
||||
if (enabled.length === total) return "";
|
||||
return enabled.join(",");
|
||||
}
|
||||
|
||||
function getAvailableSourceIds(nextPayload = payload) {
|
||||
return (Array.isArray(nextPayload?.sources) ? nextPayload.sources : [])
|
||||
.map((source) => String(source?.id || "").trim())
|
||||
.filter(Boolean)
|
||||
.sort();
|
||||
}
|
||||
|
||||
function getEnabledNewsSourceIds(nextPayload = payload) {
|
||||
const available = getAvailableSourceIds(nextPayload);
|
||||
if (!available.length) return [];
|
||||
if (!Array.isArray(activeNewsSourceFilters)) return available;
|
||||
const allowed = new Set(activeNewsSourceFilters);
|
||||
return available.filter((id) => allowed.has(id));
|
||||
}
|
||||
|
||||
function getNewsSourceSignature(nextPayload = payload) {
|
||||
const available = getAvailableSourceIds(nextPayload);
|
||||
const enabled = getEnabledNewsSourceIds(nextPayload);
|
||||
if (available.length > 0 && enabled.length === 0) return "__none__";
|
||||
if (enabled.length === available.length) return "";
|
||||
return enabled.join(",");
|
||||
}
|
||||
|
||||
function getNewsLimit() {
|
||||
return newsFullListMode ? NEWS_FULL_LIMIT : NEWS_SUMMARY_LIMIT;
|
||||
}
|
||||
|
||||
function setNewsSourceFilters(enabledIds, { persist = true } = {}) {
|
||||
const available = getAvailableSourceIds();
|
||||
const next = Array.isArray(enabledIds)
|
||||
? enabledIds.map((id) => String(id || "").trim()).filter((id) => available.includes(id))
|
||||
: null;
|
||||
activeNewsSourceFilters = next && next.length === available.length ? null : next;
|
||||
if (persist) persistNewsSourceFilters(activeNewsSourceFilters);
|
||||
}
|
||||
|
||||
function summarizeSelection(enabledCount, totalCount) {
|
||||
if (totalCount <= 0) return "暂无";
|
||||
if (enabledCount <= 0) return "未选";
|
||||
if (enabledCount === totalCount) return "全部";
|
||||
return `${enabledCount} 项`;
|
||||
}
|
||||
|
||||
function syncFilterSummaries(nextPayload = payload) {
|
||||
const categories = getEnabledNewsCategoryKeys();
|
||||
const totalCategories = Object.keys(NEWS_CATEGORY_ALIASES).length;
|
||||
const sources = getAvailableSourceIds(nextPayload);
|
||||
const enabledSources = getEnabledNewsSourceIds(nextPayload);
|
||||
document.querySelectorAll('[data-news-filter-summary="category"]').forEach((el) => {
|
||||
el.textContent = summarizeSelection(categories.length, totalCategories);
|
||||
});
|
||||
document.querySelectorAll('[data-news-filter-summary="source"]').forEach((el) => {
|
||||
el.textContent = summarizeSelection(enabledSources.length, sources.length);
|
||||
});
|
||||
document.querySelectorAll('[data-news-filter-summary="limit"]').forEach((el) => {
|
||||
const total = Array.isArray(nextPayload?.items) ? nextPayload.items.length : 0;
|
||||
el.textContent = newsFullListMode ? "全部" : `${Math.min(NEWS_SUMMARY_LIMIT, total || NEWS_SUMMARY_LIMIT)} 条`;
|
||||
});
|
||||
document.querySelectorAll("[data-news-view-mode-label]").forEach((el) => {
|
||||
el.textContent = newsFullListMode ? "返回摘要" : "查看全部";
|
||||
});
|
||||
}
|
||||
|
||||
function closeNewsFilterPopover() {
|
||||
activeFilterPopover = null;
|
||||
document.querySelectorAll("[data-news-filter-popover]").forEach((popover) => {
|
||||
if (popover instanceof HTMLElement) popover.hidden = true;
|
||||
});
|
||||
document.querySelectorAll("[data-news-filter-toggle]").forEach((toggle) => {
|
||||
if (toggle instanceof HTMLElement) toggle.setAttribute("aria-expanded", "false");
|
||||
});
|
||||
}
|
||||
|
||||
function renderCategoryFilterChips() {
|
||||
const enabled = new Set(getEnabledNewsCategoryKeys());
|
||||
return Object.keys(NEWS_CATEGORY_ALIASES)
|
||||
.map((key) => `
|
||||
<button
|
||||
class="news-filter-chip${enabled.has(key) ? " is-active" : ""}"
|
||||
type="button"
|
||||
data-news-category-toggle="${escapeNewsHtml(key)}"
|
||||
aria-pressed="${enabled.has(key) ? "true" : "false"}"
|
||||
>${escapeNewsHtml(getNewsCategoryLabel(key))}</button>
|
||||
`)
|
||||
.join("");
|
||||
}
|
||||
|
||||
function renderSourceFilterChips() {
|
||||
const sources = Array.isArray(payload?.sources) ? payload.sources : [];
|
||||
const enabled = new Set(getEnabledNewsSourceIds());
|
||||
if (!sources.length) return `<span class="news-filter-popover__hint">暂无可筛选来源。</span>`;
|
||||
return sources
|
||||
.map((source) => {
|
||||
const id = String(source?.id || "").trim();
|
||||
if (!id) return "";
|
||||
const active = enabled.has(id);
|
||||
return `
|
||||
<button
|
||||
class="news-filter-chip${active ? " is-active" : ""}"
|
||||
type="button"
|
||||
data-news-source-toggle="${escapeNewsHtml(id)}"
|
||||
aria-pressed="${active ? "true" : "false"}"
|
||||
>${escapeNewsHtml(source?.name || id)}</button>
|
||||
`;
|
||||
})
|
||||
.join("");
|
||||
}
|
||||
|
||||
function renderFilterPopover(kind) {
|
||||
const title = kind === "source" ? "新闻来源" : "新闻类型";
|
||||
const hint = kind === "source" ? "按大来源筛选,不影响后台抓取。" : "按新闻内容分类筛选。";
|
||||
const content = kind === "source" ? renderSourceFilterChips() : renderCategoryFilterChips();
|
||||
|
||||
activeFilterPopover = kind;
|
||||
document.querySelectorAll("[data-news-filter-popover]").forEach((popover) => {
|
||||
if (!(popover instanceof HTMLElement)) return;
|
||||
popover.hidden = false;
|
||||
popover.innerHTML = `
|
||||
<div class="news-filter-popover__header">
|
||||
<div class="news-filter-popover__title">${escapeNewsHtml(title)}</div>
|
||||
<div class="news-filter-popover__hint">${escapeNewsHtml(hint)}</div>
|
||||
</div>
|
||||
<div class="news-filter-chip-group">${content}</div>
|
||||
`;
|
||||
});
|
||||
document.querySelectorAll("[data-news-filter-toggle]").forEach((toggle) => {
|
||||
if (!(toggle instanceof HTMLElement)) return;
|
||||
toggle.setAttribute("aria-expanded", toggle.dataset.newsFilterToggle === kind ? "true" : "false");
|
||||
});
|
||||
}
|
||||
|
||||
function toggleNewsSource(sourceId) {
|
||||
const available = getAvailableSourceIds();
|
||||
if (!available.includes(sourceId)) return;
|
||||
const current = new Set(getEnabledNewsSourceIds());
|
||||
if (current.has(sourceId)) current.delete(sourceId);
|
||||
else current.add(sourceId);
|
||||
setNewsSourceFilters([...current]);
|
||||
syncFilterSummaries();
|
||||
if (activeFilterPopover === "source") renderFilterPopover("source");
|
||||
lastFetchAt = 0;
|
||||
refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {});
|
||||
}
|
||||
|
||||
function toggleNewsCategory(category, enabled) {
|
||||
window.dispatchEvent(new CustomEvent("earth:set-news-category-enabled", {
|
||||
detail: { category, enabled },
|
||||
}));
|
||||
}
|
||||
|
||||
function toggleNewsListMode() {
|
||||
newsFullListMode = !newsFullListMode;
|
||||
syncFilterSummaries();
|
||||
lastFetchAt = 0;
|
||||
refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {});
|
||||
}
|
||||
|
||||
function renderTicker(nextPayload) {
|
||||
@@ -320,7 +602,7 @@ function renderTicker(nextPayload) {
|
||||
|
||||
const visibleItems = getDisplayableNewsItems(items).slice(0, 6);
|
||||
if (visibleItems.length === 0) {
|
||||
tickerTrack.textContent = "正在等待中文新闻...";
|
||||
tickerTrack.textContent = "当前新闻类型没有可显示新闻...";
|
||||
tickerTrack.style.removeProperty("--news-ticker-duration");
|
||||
return;
|
||||
}
|
||||
@@ -366,9 +648,20 @@ function renderPayload(nextPayload) {
|
||||
const items = Array.isArray(nextPayload?.items) ? nextPayload.items : [];
|
||||
const displayItems = getDisplayableNewsItems(items);
|
||||
const sources = Array.isArray(nextPayload?.sources) ? nextPayload.sources : [];
|
||||
const sourcesByName = new Map(
|
||||
sources
|
||||
.filter((source) => source && typeof source === "object" && source.name)
|
||||
.map((source) => [String(source.name), source]),
|
||||
);
|
||||
const sourcesById = new Map(
|
||||
sources
|
||||
.filter((source) => source && typeof source === "object" && source.id)
|
||||
.map((source) => [String(source.id), source]),
|
||||
);
|
||||
const focus = nextPayload?.focus || {};
|
||||
|
||||
renderTicker(nextPayload);
|
||||
syncFilterSummaries(nextPayload);
|
||||
|
||||
if (!board || !status || !focusLabel || !focusCoords || !sourceCount || !regionChip) {
|
||||
if (document.body.classList.contains("layout-mode-mobile")) {
|
||||
@@ -400,8 +693,11 @@ function renderPayload(nextPayload) {
|
||||
focusCoords.textContent = "跟随当前视角自动聚焦";
|
||||
}
|
||||
|
||||
sourceCount.textContent = `${sources.length} 路聚合源`;
|
||||
if (nextPayload?.stale) {
|
||||
const enabledSourceCount = getEnabledNewsSourceIds(nextPayload).length;
|
||||
sourceCount.textContent = `${enabledSourceCount || sources.length} / ${sources.length} 路来源`;
|
||||
if (displayItems.length !== items.length) {
|
||||
status.textContent = `展示 ${displayItems.length} / ${items.length} 条态势新闻`;
|
||||
} else if (nextPayload?.stale) {
|
||||
status.textContent = `当前显示最近一次可用新闻缓存,共 ${items.length} 条`;
|
||||
} else {
|
||||
status.textContent = nextPayload?.errors?.length
|
||||
@@ -424,7 +720,7 @@ function renderPayload(nextPayload) {
|
||||
empty.hidden = false;
|
||||
empty.textContent = items.length === 0
|
||||
? "当前未拉到可用新闻,请稍后刷新或切换视角区域。"
|
||||
: "正在等待中文新闻内容完成处理。";
|
||||
: "当前新闻类型没有可显示新闻。";
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -440,22 +736,24 @@ function renderPayload(nextPayload) {
|
||||
const summaryText = getNewsDisplaySummary(item);
|
||||
const leadText = summaryText || title;
|
||||
const regionLabel = item.display_region || getNewsRegionLabel(item.region);
|
||||
const feedLabel = getNewsFeedLabel(item.feed_name);
|
||||
const statusLabel = getNewsEnrichmentStatusLabel(item.enrichment_status);
|
||||
const categoryLabel = getNewsCategoryLabel(item.category);
|
||||
const statusLabel = getNewsEnrichmentStatusLabel(item);
|
||||
const sourceDescriptor = getNewsSourceDescriptor(item, sourcesByName, sourcesById);
|
||||
const summary = title && title !== leadText
|
||||
? `<div class="news-story-summary">${escapeNewsHtml(title)}</div>`
|
||||
: "";
|
||||
return `
|
||||
<a class="${cardClass}" data-news-id="${item.id}" href="${item.url}" target="_blank" rel="noreferrer noopener">
|
||||
<a class="${cardClass}" data-news-id="${item.id}" href="${item.url}" target="_blank" rel="noreferrer noopener" title="${escapeNewsHtml(sourceDescriptor.tooltip)}">
|
||||
<div class="news-story-meta">
|
||||
<span class="news-story-source">${escapeNewsHtml(item.source || "NEWS")}</span>
|
||||
<span class="news-story-source">${escapeNewsHtml(sourceDescriptor.sourceName)}</span>
|
||||
<span class="news-story-time">${formatRelativeTime(item.published_at)}</span>
|
||||
</div>
|
||||
<div class="news-story-origin">${escapeNewsHtml(sourceDescriptor.originLabel)}</div>
|
||||
<div class="news-story-title">${escapeNewsHtml(leadText)}</div>
|
||||
${summary}
|
||||
<div class="news-story-tags">
|
||||
<span class="news-story-tag">${escapeNewsHtml(categoryLabel)}</span>
|
||||
<span class="news-story-tag">${escapeNewsHtml(regionLabel)}</span>
|
||||
<span class="news-story-tag">${escapeNewsHtml(feedLabel)}</span>
|
||||
<span class="news-story-tag">${escapeNewsHtml(statusLabel)}</span>
|
||||
</div>
|
||||
</a>
|
||||
@@ -472,11 +770,6 @@ function renderPayload(nextPayload) {
|
||||
}));
|
||||
}
|
||||
|
||||
function getNewsRealtimeUrl() {
|
||||
const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
|
||||
return `${protocol}//${window.location.host}/ws`;
|
||||
}
|
||||
|
||||
function clearNewsRealtimeReconnectTimer() {
|
||||
if (!newsRealtimeReconnectTimer) return;
|
||||
window.clearTimeout(newsRealtimeReconnectTimer);
|
||||
@@ -485,10 +778,11 @@ function clearNewsRealtimeReconnectTimer() {
|
||||
|
||||
function scheduleNewsRealtimeReconnect() {
|
||||
if (newsRealtimeReconnectTimer) return;
|
||||
const delay = Math.max(NEWS_REALTIME_RECONNECT_MS, getEarthRealtimeCooldownMs());
|
||||
newsRealtimeReconnectTimer = window.setTimeout(() => {
|
||||
newsRealtimeReconnectTimer = null;
|
||||
connectNewsRealtime();
|
||||
}, NEWS_REALTIME_RECONNECT_MS);
|
||||
}, delay);
|
||||
}
|
||||
|
||||
function applyNewsRealtimePatch(updatePayload) {
|
||||
@@ -517,10 +811,15 @@ function applyNewsRealtimePatch(updatePayload) {
|
||||
}
|
||||
|
||||
function connectNewsRealtime() {
|
||||
if (newsRealtimeSocket || typeof WebSocket === "undefined") return;
|
||||
const socket = new WebSocket(getNewsRealtimeUrl());
|
||||
if (newsRealtimeSocket || !canAttemptEarthRealtime()) {
|
||||
scheduleNewsRealtimeReconnect();
|
||||
return;
|
||||
}
|
||||
const socket = new WebSocket(getEarthRealtimeUrl());
|
||||
newsRealtimeSocket = socket;
|
||||
socket.onopen = () => {
|
||||
socket.__planetOpened = true;
|
||||
recordEarthRealtimeOpen();
|
||||
clearNewsRealtimeReconnectTimer();
|
||||
socket.send(JSON.stringify({
|
||||
type: "subscribe",
|
||||
@@ -547,6 +846,9 @@ function connectNewsRealtime() {
|
||||
if (newsRealtimeSocket === socket) {
|
||||
newsRealtimeSocket = null;
|
||||
}
|
||||
if (!socket.__planetOpened) {
|
||||
recordEarthRealtimeFailure();
|
||||
}
|
||||
scheduleNewsRealtimeReconnect();
|
||||
};
|
||||
socket.onerror = () => {
|
||||
@@ -555,9 +857,34 @@ function connectNewsRealtime() {
|
||||
}
|
||||
|
||||
async function fetchNews(lat, lon) {
|
||||
const categorySignature = getNewsCategorySignature();
|
||||
const sourceSignature = getNewsSourceSignature();
|
||||
if (categorySignature === "__none__" || sourceSignature === "__none__") {
|
||||
return {
|
||||
...(payload || {}),
|
||||
generated_at: new Date().toISOString(),
|
||||
focus: payload?.focus || { lat, lon, region: "global", label: "全球焦点", display_region: "全球" },
|
||||
sources: payload?.sources || [],
|
||||
filters: {
|
||||
region: payload?.focus?.region || "global",
|
||||
categories: categorySignature === "__none__" ? [] : getEnabledNewsCategoryKeys(),
|
||||
sources: sourceSignature === "__none__" ? [] : getEnabledNewsSourceIds(),
|
||||
limit: getNewsLimit(),
|
||||
locale: "zh-CN",
|
||||
},
|
||||
items: [],
|
||||
cruise_items: [],
|
||||
errors: [],
|
||||
stale: false,
|
||||
};
|
||||
}
|
||||
const url = new URL(EARTH_NEWS_API, window.location.origin);
|
||||
if (typeof lat === "number") url.searchParams.set("lat", lat.toFixed(4));
|
||||
if (typeof lon === "number") url.searchParams.set("lon", lon.toFixed(4));
|
||||
if (categorySignature) url.searchParams.set("categories", categorySignature);
|
||||
if (sourceSignature) url.searchParams.set("sources", sourceSignature);
|
||||
url.searchParams.set("limit", String(getNewsLimit()));
|
||||
url.searchParams.set("locale", "zh-CN");
|
||||
|
||||
const controller = new AbortController();
|
||||
const timeoutId = window.setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
|
||||
@@ -585,6 +912,8 @@ async function refreshNews(lat, lon, { silent = false } = {}) {
|
||||
.then((nextPayload) => {
|
||||
renderPayload(nextPayload);
|
||||
lastFetchAt = Date.now();
|
||||
lastCategorySignature = getNewsCategorySignature();
|
||||
lastSourceSignature = getNewsSourceSignature(nextPayload);
|
||||
if (Array.isArray(nextPayload?.items) && nextPayload.items.length === 0) {
|
||||
const { status } = getElements();
|
||||
if (status) {
|
||||
@@ -618,6 +947,8 @@ export async function refreshEarthNews({ silent = true } = {}) {
|
||||
|
||||
function shouldRefreshForFocus(lat, lon, region) {
|
||||
const now = Date.now();
|
||||
if (getNewsCategorySignature() !== lastCategorySignature) return true;
|
||||
if (getNewsSourceSignature() !== lastSourceSignature) return true;
|
||||
if (!lastFocus) return true;
|
||||
if (region !== lastFocus.region && now - lastRegionSwitchAt > MIN_REGION_SWITCH_INTERVAL_MS) {
|
||||
lastRegionSwitchAt = now;
|
||||
@@ -650,6 +981,7 @@ export function updateNewsViewFocus(coords) {
|
||||
if (!coords || typeof coords.lat !== "number" || typeof coords.lon !== "number") return;
|
||||
|
||||
const region = inferRegion(coords.lat, coords.lon);
|
||||
const previousRegion = lastFocus?.region || null;
|
||||
const nextFocus = {
|
||||
lat: coords.lat,
|
||||
lon: coords.lon,
|
||||
@@ -658,6 +990,9 @@ export function updateNewsViewFocus(coords) {
|
||||
};
|
||||
|
||||
const shouldRefresh = shouldRefreshForFocus(coords.lat, coords.lon, region);
|
||||
if (!shouldRefresh && previousRegion && region !== previousRegion) {
|
||||
return;
|
||||
}
|
||||
lastFocus = nextFocus;
|
||||
if (shouldRefresh) {
|
||||
refreshNews(coords.lat, coords.lon, { silent: true }).catch(() => {});
|
||||
@@ -743,6 +1078,57 @@ export function initNewsPanel() {
|
||||
openNewsHud();
|
||||
});
|
||||
hudCloseBtn?.addEventListener("click", closeNewsHud);
|
||||
document.addEventListener("keydown", (event) => {
|
||||
if (event.key !== "Escape" || !isNewsHudOpen()) return;
|
||||
event.preventDefault();
|
||||
if (activeFilterPopover) {
|
||||
closeNewsFilterPopover();
|
||||
return;
|
||||
}
|
||||
closeNewsHud();
|
||||
});
|
||||
document.addEventListener("click", (event) => {
|
||||
const target = event.target instanceof Element ? event.target : null;
|
||||
if (!target) return;
|
||||
const filterToggle = target.closest("[data-news-filter-toggle]");
|
||||
if (filterToggle instanceof HTMLElement) {
|
||||
const kind = filterToggle.dataset.newsFilterToggle || "";
|
||||
if (activeFilterPopover === kind) closeNewsFilterPopover();
|
||||
else renderFilterPopover(kind);
|
||||
return;
|
||||
}
|
||||
|
||||
const categoryToggle = target.closest("[data-news-category-toggle]");
|
||||
if (categoryToggle instanceof HTMLElement && categoryToggle.closest("[data-news-filter-popover]")) {
|
||||
const category = categoryToggle.dataset.newsCategoryToggle || "";
|
||||
const active = categoryToggle.classList.contains("is-active");
|
||||
toggleNewsCategory(category, !active);
|
||||
return;
|
||||
}
|
||||
|
||||
const sourceToggle = target.closest("[data-news-source-toggle]");
|
||||
if (sourceToggle instanceof HTMLElement) {
|
||||
toggleNewsSource(sourceToggle.dataset.newsSourceToggle || "");
|
||||
return;
|
||||
}
|
||||
|
||||
const viewAllToggle = target.closest("#news-view-all-toggle, #mobile-news-view-all-toggle");
|
||||
if (viewAllToggle instanceof HTMLElement) {
|
||||
toggleNewsListMode();
|
||||
return;
|
||||
}
|
||||
|
||||
if (activeFilterPopover && !target.closest("[data-news-filter-popover]")) {
|
||||
closeNewsFilterPopover();
|
||||
}
|
||||
});
|
||||
window.addEventListener("earth:news-category-filters-change", (event) => {
|
||||
activeNewsCategoryFilters = event.detail?.categories || null;
|
||||
syncFilterSummaries();
|
||||
if (activeFilterPopover === "category") renderFilterPopover("category");
|
||||
lastFetchAt = 0;
|
||||
refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {});
|
||||
});
|
||||
setupNewsHudResize();
|
||||
connectNewsRealtime();
|
||||
|
||||
|
||||
40
frontend/public/earth/js/realtime.js
Normal file
40
frontend/public/earth/js/realtime.js
Normal file
@@ -0,0 +1,40 @@
|
||||
const WS_BACKOFF_BASE_MS = 5_000;
|
||||
const WS_BACKOFF_MAX_MS = 60_000;
|
||||
|
||||
const wsBackoffState = {
|
||||
failures: 0,
|
||||
nextAttemptAt: 0,
|
||||
};
|
||||
|
||||
export function getEarthRealtimeUrl() {
|
||||
const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
|
||||
const host = window.location.hostname;
|
||||
const port = window.location.port;
|
||||
if ((host === "localhost" || host === "127.0.0.1") && port === "3000") {
|
||||
return `${protocol}//${host}:8000/ws`;
|
||||
}
|
||||
return `${protocol}//${window.location.host}/ws`;
|
||||
}
|
||||
|
||||
export function getEarthRealtimeCooldownMs() {
|
||||
return Math.max(0, wsBackoffState.nextAttemptAt - Date.now());
|
||||
}
|
||||
|
||||
export function canAttemptEarthRealtime() {
|
||||
return typeof WebSocket !== "undefined" && getEarthRealtimeCooldownMs() <= 0;
|
||||
}
|
||||
|
||||
export function recordEarthRealtimeOpen() {
|
||||
wsBackoffState.failures = 0;
|
||||
wsBackoffState.nextAttemptAt = 0;
|
||||
}
|
||||
|
||||
export function recordEarthRealtimeFailure() {
|
||||
wsBackoffState.failures += 1;
|
||||
const delay = Math.min(
|
||||
WS_BACKOFF_MAX_MS,
|
||||
WS_BACKOFF_BASE_MS * 2 ** Math.min(wsBackoffState.failures - 1, 5),
|
||||
);
|
||||
wsBackoffState.nextAttemptAt = Date.now() + delay;
|
||||
return delay;
|
||||
}
|
||||
@@ -39,12 +39,85 @@ const TV_PANEL_MIN_HEIGHT_PX = 340;
|
||||
|
||||
const HLS_MAX_RECOVERY_ATTEMPTS = 3;
|
||||
const HLS_RETRY_CONFIG = {
|
||||
maxNumRetry: 4,
|
||||
retryDelayMs: 1500,
|
||||
maxRetryDelayMs: 8000,
|
||||
maxNumRetry: 1,
|
||||
retryDelayMs: 1200,
|
||||
maxRetryDelayMs: 3000,
|
||||
backoff: "exponential",
|
||||
};
|
||||
|
||||
const HLS_NON_PLAYBACK_ERROR_DETAILS = new Set([
|
||||
"subtitleTrackLoadError",
|
||||
"subtitleTrackParsingError",
|
||||
"subtitleTrackSwitchError",
|
||||
"audioTrackLoadError",
|
||||
"audioTrackSwitchError",
|
||||
"keyLoadError",
|
||||
]);
|
||||
|
||||
const HLS_SOURCE_FAILURE_DETAILS = new Set([
|
||||
"manifestLoadError",
|
||||
"manifestLoadTimeOut",
|
||||
"levelLoadError",
|
||||
"levelLoadTimeOut",
|
||||
"fragLoadError",
|
||||
"fragLoadTimeOut",
|
||||
]);
|
||||
|
||||
function isVideoActuallyPlaying(video) {
|
||||
return (
|
||||
video instanceof HTMLVideoElement
|
||||
&& !video.paused
|
||||
&& !video.ended
|
||||
&& video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA
|
||||
);
|
||||
}
|
||||
|
||||
function getHlsErrorMessage(data) {
|
||||
const details = data?.details || "";
|
||||
const response = data?.response || {};
|
||||
const status = response?.code || response?.status;
|
||||
const url = data?.url || response?.url || "";
|
||||
|
||||
if (details === "manifestLoadError" || details === "manifestLoadTimeOut") {
|
||||
return status
|
||||
? `HLS 主播放列表加载失败(HTTP ${status})`
|
||||
: "HLS 主播放列表加载失败";
|
||||
}
|
||||
if (details === "levelLoadError" || details === "levelLoadTimeOut") {
|
||||
return status
|
||||
? `HLS 清晰度播放列表加载失败(HTTP ${status})`
|
||||
: "HLS 清晰度播放列表加载失败";
|
||||
}
|
||||
if (details === "fragLoadError" || details === "fragLoadTimeOut") {
|
||||
return status
|
||||
? `HLS 分片加载失败(HTTP ${status})`
|
||||
: "HLS 分片加载失败";
|
||||
}
|
||||
if (details === "bufferStalledError") return "直播流缓冲停滞,正在等待数据";
|
||||
if (details === "bufferAppendError") return "直播流缓冲写入失败";
|
||||
if (details === "manifestParsingError") return "HLS 播放列表格式无法解析";
|
||||
if (details === "fragParsingError") return "HLS 分片格式无法解析";
|
||||
if (url) return `HLS 资源加载失败:${url}`;
|
||||
return "HLS 播放流不可用";
|
||||
}
|
||||
|
||||
function logHlsDiagnostic(data, source) {
|
||||
const payload = {
|
||||
source_id: source?.id,
|
||||
source_name: source?.name,
|
||||
type: data?.type,
|
||||
details: data?.details,
|
||||
fatal: Boolean(data?.fatal),
|
||||
url: data?.url || data?.response?.url,
|
||||
status: data?.response?.code || data?.response?.status,
|
||||
};
|
||||
if (data?.fatal) {
|
||||
console.error("HLS 播放失败:", payload, data);
|
||||
} else {
|
||||
console.debug("HLS 非致命事件:", payload, data);
|
||||
}
|
||||
}
|
||||
|
||||
function getElements() {
|
||||
const isMobile = document.body.classList.contains("layout-mode-mobile");
|
||||
return {
|
||||
@@ -398,6 +471,8 @@ function tryStartPlayback(video) {
|
||||
function attachVideoSource(video, source) {
|
||||
const sourceUrl = getVideoUrl(source);
|
||||
if (!(video instanceof HTMLVideoElement) || !sourceUrl) return;
|
||||
let hlsNetworkErrorCount = 0;
|
||||
let hlsHardFailureHandled = false;
|
||||
|
||||
destroyHlsPlayer();
|
||||
video.autoplay = true;
|
||||
@@ -414,16 +489,18 @@ function attachVideoSource(video, source) {
|
||||
if (Hls.isSupported()) {
|
||||
hlsPlayer = new Hls({
|
||||
enableWorker: true,
|
||||
enableWebVTT: false,
|
||||
subtitleDisplay: false,
|
||||
lowLatencyMode: false,
|
||||
manifestLoadingTimeOut: 20000,
|
||||
levelLoadingTimeOut: 20000,
|
||||
fragLoadingTimeOut: 25000,
|
||||
fragLoadingMaxRetry: 3,
|
||||
fragLoadingRetryDelay: 1500,
|
||||
levelLoadingMaxRetry: 3,
|
||||
levelLoadingRetryDelay: 1500,
|
||||
manifestLoadingMaxRetry: 2,
|
||||
manifestLoadingRetryDelay: 1500,
|
||||
fragLoadingMaxRetry: 1,
|
||||
fragLoadingRetryDelay: 1200,
|
||||
levelLoadingMaxRetry: 1,
|
||||
levelLoadingRetryDelay: 1200,
|
||||
manifestLoadingMaxRetry: 1,
|
||||
manifestLoadingRetryDelay: 1200,
|
||||
liveSyncDurationCount: 4,
|
||||
liveMaxLatencyDurationCount: 10,
|
||||
manifestLoadPolicy: {
|
||||
@@ -432,11 +509,11 @@ function attachVideoSource(video, source) {
|
||||
maxLoadTimeMs: 20000,
|
||||
timeoutRetry: {
|
||||
...HLS_RETRY_CONFIG,
|
||||
maxNumRetry: 2,
|
||||
maxNumRetry: 1,
|
||||
},
|
||||
errorRetry: {
|
||||
...HLS_RETRY_CONFIG,
|
||||
maxNumRetry: 2,
|
||||
maxNumRetry: 1,
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -465,14 +542,35 @@ function attachVideoSource(video, source) {
|
||||
tryStartPlayback(video);
|
||||
});
|
||||
hlsPlayer.on(Hls.Events.ERROR, (_event, data) => {
|
||||
console.error("HLS 播放失败:", data);
|
||||
logHlsDiagnostic(data, source);
|
||||
const status = Number(data?.response?.code || data?.response?.status || 0);
|
||||
const sourceFailure = HLS_SOURCE_FAILURE_DETAILS.has(data?.details);
|
||||
if (sourceFailure && (status >= 500 || data?.details === "manifestLoadError")) {
|
||||
hlsNetworkErrorCount += 1;
|
||||
}
|
||||
if (!hlsHardFailureHandled && !isVideoActuallyPlaying(video) && sourceFailure && hlsNetworkErrorCount >= 2) {
|
||||
hlsHardFailureHandled = true;
|
||||
const reasonMessage = getHlsErrorMessage(data);
|
||||
hlsPlayer?.stopLoad();
|
||||
if (!showEmbeddedFallback(source, reasonMessage) && !tryFallbackSource()) {
|
||||
setPanelMessage(reasonMessage);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (
|
||||
!data?.fatal
|
||||
&& HLS_NON_PLAYBACK_ERROR_DETAILS.has(data?.details)
|
||||
&& isVideoActuallyPlaying(video)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
if (!data?.fatal) {
|
||||
if (data?.type === Hls.ErrorTypes.NETWORK_ERROR) {
|
||||
setPanelMessage("直播流网络波动,正在重试...");
|
||||
setPanelMessage(isVideoActuallyPlaying(video) ? TV_STATUS_MESSAGE.videoReady : "直播流网络波动,正在重试...");
|
||||
return;
|
||||
}
|
||||
if (data?.type === Hls.ErrorTypes.MEDIA_ERROR) {
|
||||
setPanelMessage("直播流正在恢复...");
|
||||
setPanelMessage(isVideoActuallyPlaying(video) ? TV_STATUS_MESSAGE.videoReady : "直播流正在恢复...");
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -491,8 +589,9 @@ function attachVideoSource(video, source) {
|
||||
}
|
||||
}
|
||||
|
||||
if (!showEmbeddedFallback(source) && !tryFallbackSource()) {
|
||||
setPanelMessage(TV_STATUS_MESSAGE.videoError);
|
||||
const reasonMessage = getHlsErrorMessage(data);
|
||||
if (!showEmbeddedFallback(source, reasonMessage) && !tryFallbackSource()) {
|
||||
setPanelMessage(reasonMessage);
|
||||
}
|
||||
});
|
||||
return;
|
||||
|
||||
@@ -2,6 +2,13 @@ import * as THREE from "three";
|
||||
|
||||
import { CONFIG, PATHS, VESSEL_CONFIG } from "./constants.js";
|
||||
import { createInteractableLayer } from "./interactable.js";
|
||||
import {
|
||||
canAttemptEarthRealtime,
|
||||
getEarthRealtimeCooldownMs,
|
||||
getEarthRealtimeUrl,
|
||||
recordEarthRealtimeFailure,
|
||||
recordEarthRealtimeOpen,
|
||||
} from "./realtime.js";
|
||||
import { latLonToVector3 } from "./utils.js";
|
||||
|
||||
let showVessels = false;
|
||||
@@ -105,12 +112,6 @@ function applyVesselDeltas(earth, vessels = []) {
|
||||
return changed;
|
||||
}
|
||||
|
||||
function getVesselStreamUrl() {
|
||||
if (typeof window === "undefined") return "ws://localhost:8000/ws";
|
||||
const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
|
||||
return `${protocol}//${window.location.host}/ws`;
|
||||
}
|
||||
|
||||
function normalizeVesselType(value, code) {
|
||||
const type = String(value || "").trim().toLowerCase();
|
||||
const numericCode = Number(code);
|
||||
@@ -257,6 +258,11 @@ const vesselIconLayer = createInteractableLayer({
|
||||
vessel_kind: item.type,
|
||||
baseScale: VESSEL_CONFIG.marker.baseScale,
|
||||
}),
|
||||
cluster: {
|
||||
strategy: "dynamic-screen",
|
||||
enabled: true,
|
||||
maxMarkersPerDot: 10,
|
||||
},
|
||||
});
|
||||
|
||||
const DEFAULT_VESSEL_VIEWPORT = {
|
||||
@@ -381,13 +387,19 @@ function normalizeVesselViewportOptions(options = {}) {
|
||||
}
|
||||
|
||||
export function startVesselRealtime(earth, { onUpdate, bbox, zoom, limit } = {}) {
|
||||
if (vesselStreamSocket || typeof WebSocket === "undefined") return;
|
||||
if (vesselStreamSocket || vesselStreamReconnectTimer || typeof WebSocket === "undefined") return;
|
||||
const subscriptionOptions = normalizeVesselViewportOptions({ bbox, zoom, limit });
|
||||
const connect = () => {
|
||||
if (!showVessels || vesselStreamSocket) return;
|
||||
const socket = new WebSocket(getVesselStreamUrl());
|
||||
if (!canAttemptEarthRealtime()) {
|
||||
vesselStreamReconnectTimer = window.setTimeout(connect, Math.max(3000, getEarthRealtimeCooldownMs()));
|
||||
return;
|
||||
}
|
||||
const socket = new WebSocket(getEarthRealtimeUrl());
|
||||
vesselStreamSocket = socket;
|
||||
socket.onopen = () => {
|
||||
socket.__planetOpened = true;
|
||||
recordEarthRealtimeOpen();
|
||||
vesselRealtimeStats = {
|
||||
...vesselRealtimeStats,
|
||||
connected: true,
|
||||
@@ -452,7 +464,10 @@ export function startVesselRealtime(earth, { onUpdate, bbox, zoom, limit } = {})
|
||||
};
|
||||
onUpdate?.({ totalCount: getVesselCount(), stream: getVesselRealtimeStats() });
|
||||
if (showVessels) {
|
||||
vesselStreamReconnectTimer = window.setTimeout(connect, 3000);
|
||||
if (!socket.__planetOpened) {
|
||||
recordEarthRealtimeFailure();
|
||||
}
|
||||
vesselStreamReconnectTimer = window.setTimeout(connect, Math.max(3000, getEarthRealtimeCooldownMs()));
|
||||
}
|
||||
};
|
||||
socket.onerror = () => {
|
||||
|
||||
@@ -4,6 +4,7 @@ import { Routes, Route, Navigate, useLocation } from 'react-router-dom'
|
||||
|
||||
import { useAuthStore } from './stores/auth'
|
||||
import Login from './pages/Login/Login'
|
||||
import { AdminErrorBoundary } from './admin/components/AdminErrorBoundary'
|
||||
|
||||
const Register = lazy(() => import('./pages/Register/Register'))
|
||||
const VerifyEmail = lazy(() => import('./pages/VerifyEmail/VerifyEmail'))
|
||||
@@ -51,7 +52,7 @@ function App() {
|
||||
<Route path={DOCS_ROUTE} element={<Docs />} />
|
||||
<Route path={DOCS_ROUTE_PATTERN} element={<Docs />} />
|
||||
<Route path="/playground" element={<Navigate to="/ai?section=playground" replace />} />
|
||||
<Route path="/*" element={<AdminRoutes />} />
|
||||
<Route path="/*" element={<AdminErrorBoundary><AdminRoutes /></AdminErrorBoundary>} />
|
||||
</Routes>
|
||||
</Suspense>
|
||||
)
|
||||
|
||||
39
frontend/src/admin/components/AdminErrorBoundary.tsx
Normal file
39
frontend/src/admin/components/AdminErrorBoundary.tsx
Normal file
@@ -0,0 +1,39 @@
|
||||
import { Component, type ErrorInfo, type ReactNode } from 'react'
|
||||
import { reportAdminRuntimeLog } from '../runtimeLogs'
|
||||
|
||||
type AdminErrorBoundaryProps = {
|
||||
children: ReactNode
|
||||
}
|
||||
|
||||
type AdminErrorBoundaryState = {
|
||||
hasError: boolean
|
||||
}
|
||||
|
||||
export class AdminErrorBoundary extends Component<AdminErrorBoundaryProps, AdminErrorBoundaryState> {
|
||||
state: AdminErrorBoundaryState = { hasError: false }
|
||||
|
||||
static getDerivedStateFromError() {
|
||||
return { hasError: true }
|
||||
}
|
||||
|
||||
componentDidCatch(error: Error, errorInfo: ErrorInfo) {
|
||||
void reportAdminRuntimeLog({
|
||||
level: 'error',
|
||||
category: 'react-error-boundary',
|
||||
module: 'admin',
|
||||
message: error.message || '控制台渲染错误',
|
||||
detail: `${error.stack || error.message}\n${errorInfo.componentStack || ''}`,
|
||||
})
|
||||
}
|
||||
|
||||
render() {
|
||||
if (this.state.hasError) {
|
||||
return (
|
||||
<div className="app-route-loading">
|
||||
<div className="app-route-loading__message">控制台发生错误,请刷新页面重试。</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
return this.props.children
|
||||
}
|
||||
}
|
||||
@@ -15,7 +15,7 @@ import Scrollbar from '../../../components/Scrollbar/Scrollbar'
|
||||
import SegmentedControl from '../../../components/SegmentedControl/SegmentedControl'
|
||||
import { useAuthStore } from '../../../stores/auth'
|
||||
import { useAdminTheme, type AdminThemeMode } from '../../design/theme'
|
||||
import { cn } from '../../lib/utils'
|
||||
import { cn } from '../../utils'
|
||||
import { adminRouteGroups, getVisibleAdminRoutes } from '../../routes/manifest'
|
||||
import { useAdminSearch } from '../../search/AdminSearchContext'
|
||||
import { Button } from '../ui/button'
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { type HTMLAttributes } from 'react'
|
||||
import { cn } from '../../lib/utils'
|
||||
import { cn } from '../../utils'
|
||||
|
||||
type BadgeTone = 'default' | 'blue' | 'green' | 'amber' | 'red' | 'purple' | 'cyan' | 'slate'
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { type HTMLAttributes } from 'react'
|
||||
import { cn } from '../../lib/utils'
|
||||
import { cn } from '../../utils'
|
||||
|
||||
export function Card({ className, ...props }: HTMLAttributes<HTMLDivElement>) {
|
||||
return <div className={cn('an-card', className)} {...props} />
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { forwardRef, type InputHTMLAttributes, type TextareaHTMLAttributes } from 'react'
|
||||
import { cn } from '../../lib/utils'
|
||||
import { cn } from '../../utils'
|
||||
|
||||
export const Input = forwardRef<HTMLInputElement, InputHTMLAttributes<HTMLInputElement>>(
|
||||
({ className, ...props }, ref) => <input ref={ref} className={cn('an-input', className)} {...props} />,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import * as SelectPrimitive from '@radix-ui/react-select'
|
||||
import { Check, ChevronDown } from 'lucide-react'
|
||||
import { cn } from '../../lib/utils'
|
||||
import { cn } from '../../utils'
|
||||
|
||||
export interface SelectOption {
|
||||
value: string
|
||||
@@ -33,7 +33,7 @@ export function Select({ value, onValueChange, options, placeholder, disabled, c
|
||||
{options.map((option) => (
|
||||
<SelectPrimitive.Item key={option.value} value={option.value} className="an-select__item">
|
||||
<SelectPrimitive.ItemText>{option.label}</SelectPrimitive.ItemText>
|
||||
<SelectPrimitive.ItemIndicator>
|
||||
<SelectPrimitive.ItemIndicator className="an-select__indicator">
|
||||
<Check size={14} />
|
||||
</SelectPrimitive.ItemIndicator>
|
||||
</SelectPrimitive.Item>
|
||||
|
||||
@@ -23,7 +23,7 @@ import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '../co
|
||||
import { Dialog } from '../components/ui/dialog'
|
||||
import { Select } from '../components/ui/select'
|
||||
import { useToast } from '../components/ui/toast'
|
||||
import { formatNumber } from '../lib/utils'
|
||||
import { formatNumber } from '../utils'
|
||||
|
||||
interface Stats {
|
||||
total_datasources: number
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import axios from 'axios'
|
||||
import { Copy, RefreshCw, Search, X } from 'lucide-react'
|
||||
import { useEffect, useMemo, useState } from 'react'
|
||||
import { Copy, Pause, Play, RefreshCw, Search, X } from 'lucide-react'
|
||||
import { useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { useLocation, useNavigate } from 'react-router-dom'
|
||||
import Scrollbar from '../../components/Scrollbar/Scrollbar'
|
||||
import { useWebSocket } from '../../hooks/useWebSocket'
|
||||
import { useAuthStore } from '../../stores/auth'
|
||||
import { AdminLayout } from '../components/layout/AdminLayout'
|
||||
import { Badge } from '../components/ui/badge'
|
||||
@@ -20,6 +22,13 @@ const LOG_LEVEL_OPTIONS = [
|
||||
{ value: 'info', label: '信息' },
|
||||
{ value: 'debug', label: '调试' },
|
||||
]
|
||||
const LOG_TAIL_CHANNEL = 'logs_tail'
|
||||
const LOG_VIEW_OPTIONS = [
|
||||
{ value: 'grouped', label: '重复统计' },
|
||||
{ value: 'raw', label: '原始日志' },
|
||||
{ value: 'audit', label: '审计日志' },
|
||||
]
|
||||
type LogViewMode = 'grouped' | 'raw' | 'audit'
|
||||
|
||||
interface LogSourceSummary {
|
||||
source_id: string
|
||||
@@ -48,30 +57,107 @@ interface LogSnapshot {
|
||||
lines: string[]
|
||||
}
|
||||
|
||||
interface ObservabilityGroup {
|
||||
fingerprint: string
|
||||
source: string
|
||||
service?: string | null
|
||||
module?: string | null
|
||||
category?: string | null
|
||||
event?: string | null
|
||||
level: string
|
||||
message: string
|
||||
detail?: string | null
|
||||
affected_sources: string[]
|
||||
count: number
|
||||
first_seen_at?: string | null
|
||||
last_seen_at?: string | null
|
||||
}
|
||||
|
||||
interface ObservabilityEvent {
|
||||
id: number
|
||||
source: string
|
||||
service?: string | null
|
||||
module?: string | null
|
||||
category?: string | null
|
||||
event?: string | null
|
||||
level: string
|
||||
message: string
|
||||
fingerprint: string
|
||||
occurred_at?: string | null
|
||||
request_id?: string | null
|
||||
trace_id?: string | null
|
||||
task_id?: string | null
|
||||
source_id?: string | null
|
||||
provider?: string | null
|
||||
context: Record<string, unknown>
|
||||
occurrence_count: number
|
||||
}
|
||||
|
||||
interface ObservabilityGroupsResponse {
|
||||
groups: ObservabilityGroup[]
|
||||
line_count: number
|
||||
}
|
||||
|
||||
interface ObservabilityGroupEventsResponse {
|
||||
fingerprint: string
|
||||
group: ObservabilityGroup
|
||||
events: ObservabilityEvent[]
|
||||
line_count: number
|
||||
}
|
||||
|
||||
type StoredLogFilters = {
|
||||
selectedSource?: string
|
||||
mode?: LogViewMode
|
||||
lineLimit?: number
|
||||
level?: string
|
||||
startDate?: string
|
||||
endDate?: string
|
||||
searchQuery?: string
|
||||
follow?: boolean
|
||||
}
|
||||
|
||||
function readStoredFilters() {
|
||||
if (typeof window === 'undefined') return null
|
||||
try {
|
||||
const rawValue = window.localStorage.getItem(LOG_FILTER_STORAGE_KEY)
|
||||
return rawValue ? JSON.parse(rawValue) as {
|
||||
selectedSource?: string
|
||||
lineLimit?: number
|
||||
level?: string
|
||||
startDate?: string
|
||||
endDate?: string
|
||||
searchQuery?: string
|
||||
} : null
|
||||
return rawValue ? JSON.parse(rawValue) as StoredLogFilters : null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
function readUrlFilters(search: string): StoredLogFilters {
|
||||
const params = new URLSearchParams(search)
|
||||
const lineLimit = Number(params.get('limit') || '')
|
||||
const rawMode = params.get('mode')
|
||||
const mode = rawMode === 'raw' || rawMode === 'audit' || rawMode === 'grouped' ? rawMode : undefined
|
||||
return {
|
||||
selectedSource: params.get('source') || undefined,
|
||||
mode,
|
||||
lineLimit: Number.isFinite(lineLimit) && lineLimit > 0 ? lineLimit : undefined,
|
||||
level: params.get('level') || undefined,
|
||||
startDate: params.get('start_date') || undefined,
|
||||
endDate: params.get('end_date') || undefined,
|
||||
searchQuery: params.get('search') || undefined,
|
||||
follow: params.get('follow') === '1',
|
||||
}
|
||||
}
|
||||
|
||||
function statusTone(status: string) {
|
||||
if (status === 'ok') return 'success'
|
||||
if (status === 'missing' || status === 'empty') return 'warning'
|
||||
if (status === 'missing') return 'warning'
|
||||
if (status === 'empty') return 'neutral'
|
||||
if (status.includes('unavailable')) return 'danger'
|
||||
return 'neutral'
|
||||
}
|
||||
|
||||
function levelTone(level: string) {
|
||||
if (level === 'error') return 'danger'
|
||||
if (level === 'warning') return 'warning'
|
||||
if (level === 'info') return 'success'
|
||||
return 'neutral'
|
||||
}
|
||||
|
||||
function statusLabel(status: string) {
|
||||
if (status === 'ok') return '可用'
|
||||
if (status === 'missing') return '暂无日志'
|
||||
@@ -87,29 +173,98 @@ function getErrorMessage(error: unknown, fallback: string) {
|
||||
return typeof detail === 'string' ? detail : fallback
|
||||
}
|
||||
|
||||
function formatDateTime(value?: string | null) {
|
||||
if (!value) return '-'
|
||||
try {
|
||||
return new Date(value).toLocaleString()
|
||||
} catch {
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
function stringifyContext(context: Record<string, unknown>) {
|
||||
try {
|
||||
return JSON.stringify(context || {}, null, 2)
|
||||
} catch {
|
||||
return '{}'
|
||||
}
|
||||
}
|
||||
|
||||
export default function Logs() {
|
||||
const storedFilters = readStoredFilters()
|
||||
const location = useLocation()
|
||||
const navigate = useNavigate()
|
||||
const urlFilters = readUrlFilters(location.search)
|
||||
const { user } = useAuthStore()
|
||||
const { toast } = useToast()
|
||||
const isSuperAdmin = user?.role === 'super_admin'
|
||||
const [sources, setSources] = useState<LogSourceSummary[]>([])
|
||||
const [selectedSource, setSelectedSource] = useState(storedFilters?.selectedSource || 'backend')
|
||||
const [lineLimit, setLineLimit] = useState(storedFilters?.lineLimit || 200)
|
||||
const [level, setLevel] = useState(storedFilters?.level || 'all')
|
||||
const [startDate, setStartDate] = useState(storedFilters?.startDate || '')
|
||||
const [endDate, setEndDate] = useState(storedFilters?.endDate || '')
|
||||
const [searchQuery, setSearchQuery] = useState(storedFilters?.searchQuery || '')
|
||||
const [submittedSearch, setSubmittedSearch] = useState(storedFilters?.searchQuery || '')
|
||||
const [mode, setMode] = useState<LogViewMode>(urlFilters.mode || storedFilters?.mode || 'grouped')
|
||||
const [selectedSource, setSelectedSource] = useState(urlFilters.selectedSource || storedFilters?.selectedSource || 'backend')
|
||||
const [lineLimit, setLineLimit] = useState(urlFilters.lineLimit || storedFilters?.lineLimit || 200)
|
||||
const [level, setLevel] = useState(urlFilters.level || storedFilters?.level || 'all')
|
||||
const [startDate, setStartDate] = useState(urlFilters.startDate || storedFilters?.startDate || '')
|
||||
const [endDate, setEndDate] = useState(urlFilters.endDate || storedFilters?.endDate || '')
|
||||
const [searchQuery, setSearchQuery] = useState(urlFilters.searchQuery ?? storedFilters?.searchQuery ?? '')
|
||||
const [submittedSearch, setSubmittedSearch] = useState(urlFilters.searchQuery ?? storedFilters?.searchQuery ?? '')
|
||||
const [followEnabled, setFollowEnabled] = useState(Boolean(urlFilters.follow || storedFilters?.follow))
|
||||
const [snapshot, setSnapshot] = useState<LogSnapshot | null>(null)
|
||||
const [groups, setGroups] = useState<ObservabilityGroup[]>([])
|
||||
const [selectedGroup, setSelectedGroup] = useState<ObservabilityGroup | null>(null)
|
||||
const [groupEvents, setGroupEvents] = useState<ObservabilityEvent[]>([])
|
||||
const [sourcesLoading, setSourcesLoading] = useState(false)
|
||||
const [logLoading, setLogLoading] = useState(false)
|
||||
const [errorMessage, setErrorMessage] = useState<string | null>(null)
|
||||
const scrollContainerRef = useRef<HTMLDivElement | null>(null)
|
||||
const shouldStickToBottomRef = useRef(true)
|
||||
|
||||
const selectedSourceInfo = useMemo(
|
||||
() => sources.find((source) => source.source_id === selectedSource) || null,
|
||||
[selectedSource, sources],
|
||||
)
|
||||
const hasActiveFilters = lineLimit !== 200 || level !== 'all' || Boolean(startDate || endDate || submittedSearch.trim() || searchQuery.trim())
|
||||
const hasActiveFilters = lineLimit !== 200 || level !== 'all' || Boolean(startDate || endDate || submittedSearch.trim() || searchQuery.trim() || followEnabled)
|
||||
|
||||
const { connected: followConnected, sendMessage } = useWebSocket({
|
||||
autoConnect: followEnabled && isSuperAdmin,
|
||||
onMessage: (message) => {
|
||||
if (message.channel !== LOG_TAIL_CHANNEL || !message.payload) return
|
||||
const payload = message.payload as { mode?: string; source_id?: string; lines?: unknown; line_count?: number; status?: string }
|
||||
if (payload.source_id !== selectedSource) return
|
||||
const incomingLines = Array.isArray(payload.lines) ? payload.lines.filter((line): line is string => typeof line === 'string') : []
|
||||
setSnapshot((current) => {
|
||||
const base = current || {
|
||||
source_id: selectedSource,
|
||||
name: selectedSourceInfo?.name || selectedSource,
|
||||
kind: selectedSourceInfo?.kind || 'unknown',
|
||||
location: selectedSourceInfo?.location || '',
|
||||
description: selectedSourceInfo?.description || '',
|
||||
category: selectedSourceInfo?.category || '',
|
||||
status: payload.status || 'ok',
|
||||
level,
|
||||
selected_levels: level === 'all' ? [] : [level],
|
||||
search_query: submittedSearch,
|
||||
available_levels: ['all', 'error', 'warning', 'info', 'debug'],
|
||||
line_limit: lineLimit,
|
||||
line_count: 0,
|
||||
lines: [],
|
||||
}
|
||||
const nextLines = payload.mode === 'snapshot'
|
||||
? incomingLines
|
||||
: [...(base.lines || []), ...incomingLines].slice(-lineLimit)
|
||||
return {
|
||||
...base,
|
||||
status: payload.status || base.status,
|
||||
line_limit: lineLimit,
|
||||
line_count: nextLines.length,
|
||||
lines: nextLines,
|
||||
}
|
||||
})
|
||||
setErrorMessage(null)
|
||||
},
|
||||
onError: () => {
|
||||
setErrorMessage('日志跟随连接失败,可暂停后使用手动刷新。')
|
||||
},
|
||||
})
|
||||
|
||||
const fetchSources = async () => {
|
||||
if (!isSuperAdmin) return
|
||||
@@ -132,9 +287,10 @@ export default function Logs() {
|
||||
|
||||
const fetchSnapshot = async () => {
|
||||
if (!isSuperAdmin || !selectedSource) return
|
||||
const sourceForMode = mode === 'audit' ? 'audit-db' : selectedSource
|
||||
setLogLoading(true)
|
||||
try {
|
||||
const response = await axios.get<LogSnapshot>(`/api/v1/system/logs/${encodeURIComponent(selectedSource)}`, {
|
||||
const response = await axios.get<LogSnapshot>(`/api/v1/system/logs/${encodeURIComponent(sourceForMode)}`, {
|
||||
params: {
|
||||
limit: lineLimit,
|
||||
level,
|
||||
@@ -154,13 +310,55 @@ export default function Logs() {
|
||||
}
|
||||
}
|
||||
|
||||
const fetchGroups = async () => {
|
||||
if (!isSuperAdmin) return
|
||||
setLogLoading(true)
|
||||
try {
|
||||
const response = await axios.get<ObservabilityGroupsResponse>('/api/v1/system/logs/observability/groups', {
|
||||
params: {
|
||||
limit: lineLimit,
|
||||
level,
|
||||
levels: level === 'all' ? undefined : level,
|
||||
start_date: startDate || undefined,
|
||||
end_date: endDate || undefined,
|
||||
search: submittedSearch.trim() || undefined,
|
||||
},
|
||||
})
|
||||
setGroups(response.data.groups || [])
|
||||
setErrorMessage(null)
|
||||
} catch (error) {
|
||||
setGroups([])
|
||||
setErrorMessage(getErrorMessage(error, '加载重复日志统计失败'))
|
||||
} finally {
|
||||
setLogLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
const fetchGroupEvents = async (group: ObservabilityGroup) => {
|
||||
setSelectedGroup(group)
|
||||
try {
|
||||
const response = await axios.get<ObservabilityGroupEventsResponse>(
|
||||
`/api/v1/system/logs/observability/groups/${encodeURIComponent(group.fingerprint)}/events`,
|
||||
{ params: { limit: lineLimit } },
|
||||
)
|
||||
setGroupEvents(response.data.events || [])
|
||||
} catch (error) {
|
||||
setGroupEvents([])
|
||||
setErrorMessage(getErrorMessage(error, '加载重复日志详情失败'))
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
void fetchSources()
|
||||
}, [isSuperAdmin])
|
||||
|
||||
useEffect(() => {
|
||||
void fetchSnapshot()
|
||||
}, [isSuperAdmin, selectedSource, lineLimit, level, startDate, endDate, submittedSearch])
|
||||
if (mode === 'grouped') {
|
||||
void fetchGroups()
|
||||
} else if (!followEnabled) {
|
||||
void fetchSnapshot()
|
||||
}
|
||||
}, [isSuperAdmin, selectedSource, lineLimit, level, startDate, endDate, submittedSearch, followEnabled, mode])
|
||||
|
||||
useEffect(() => {
|
||||
const timer = window.setTimeout(() => {
|
||||
@@ -173,13 +371,80 @@ export default function Logs() {
|
||||
if (typeof window === 'undefined') return
|
||||
window.localStorage.setItem(LOG_FILTER_STORAGE_KEY, JSON.stringify({
|
||||
selectedSource,
|
||||
mode,
|
||||
lineLimit,
|
||||
level,
|
||||
startDate,
|
||||
endDate,
|
||||
searchQuery: submittedSearch,
|
||||
follow: followEnabled,
|
||||
}))
|
||||
}, [endDate, level, lineLimit, selectedSource, startDate, submittedSearch])
|
||||
}, [endDate, followEnabled, level, lineLimit, mode, selectedSource, startDate, submittedSearch])
|
||||
|
||||
useEffect(() => {
|
||||
const filters = readUrlFilters(location.search)
|
||||
if (filters.mode && filters.mode !== mode) setMode(filters.mode)
|
||||
if (filters.selectedSource && filters.selectedSource !== selectedSource) setSelectedSource(filters.selectedSource)
|
||||
if (filters.lineLimit && filters.lineLimit !== lineLimit) setLineLimit(filters.lineLimit)
|
||||
if (filters.level && filters.level !== level) setLevel(filters.level)
|
||||
if ((filters.startDate || '') !== startDate) setStartDate(filters.startDate || '')
|
||||
if ((filters.endDate || '') !== endDate) setEndDate(filters.endDate || '')
|
||||
if (filters.searchQuery !== undefined && filters.searchQuery !== searchQuery) {
|
||||
setSearchQuery(filters.searchQuery)
|
||||
setSubmittedSearch(filters.searchQuery)
|
||||
}
|
||||
if (filters.follow !== followEnabled && new URLSearchParams(location.search).has('follow')) {
|
||||
setFollowEnabled(Boolean(filters.follow))
|
||||
}
|
||||
}, [location.search])
|
||||
|
||||
useEffect(() => {
|
||||
if (typeof window === 'undefined') return
|
||||
const params = new URLSearchParams()
|
||||
if (mode !== 'grouped') params.set('mode', mode)
|
||||
if (selectedSource && selectedSource !== 'backend') params.set('source', selectedSource)
|
||||
if (lineLimit !== 200) params.set('limit', String(lineLimit))
|
||||
if (level !== 'all') params.set('level', level)
|
||||
if (startDate) params.set('start_date', startDate)
|
||||
if (endDate) params.set('end_date', endDate)
|
||||
if (submittedSearch.trim()) params.set('search', submittedSearch.trim())
|
||||
if (followEnabled) params.set('follow', '1')
|
||||
const nextSearch = params.toString()
|
||||
const currentSearch = location.search.replace(/^\?/, '')
|
||||
if (nextSearch !== currentSearch) {
|
||||
navigate({ pathname: location.pathname, search: nextSearch ? `?${nextSearch}` : '' }, { replace: true })
|
||||
}
|
||||
}, [endDate, followEnabled, level, lineLimit, location.pathname, location.search, mode, navigate, selectedSource, startDate, submittedSearch])
|
||||
|
||||
useEffect(() => {
|
||||
if (!followEnabled || !followConnected || !isSuperAdmin || !selectedSource) return
|
||||
sendMessage({
|
||||
type: 'subscribe',
|
||||
data: {
|
||||
channel: LOG_TAIL_CHANNEL,
|
||||
source_id: selectedSource,
|
||||
limit: lineLimit,
|
||||
level,
|
||||
levels: level === 'all' ? undefined : level,
|
||||
start_date: startDate || undefined,
|
||||
end_date: endDate || undefined,
|
||||
search: submittedSearch.trim() || undefined,
|
||||
},
|
||||
})
|
||||
}, [endDate, followConnected, followEnabled, isSuperAdmin, level, lineLimit, selectedSource, sendMessage, startDate, submittedSearch])
|
||||
|
||||
useEffect(() => {
|
||||
const viewport = scrollContainerRef.current
|
||||
if (!viewport || !shouldStickToBottomRef.current) return
|
||||
viewport.scrollTop = viewport.scrollHeight
|
||||
}, [snapshot?.lines])
|
||||
|
||||
useEffect(() => {
|
||||
const viewport = scrollContainerRef.current
|
||||
if (!viewport) return
|
||||
viewport.addEventListener('scroll', handleLogScroll, { passive: true })
|
||||
return () => viewport.removeEventListener('scroll', handleLogScroll)
|
||||
}, [snapshot?.source_id])
|
||||
|
||||
const resetFilters = () => {
|
||||
setLevel('all')
|
||||
@@ -188,10 +453,21 @@ export default function Logs() {
|
||||
setSearchQuery('')
|
||||
setSubmittedSearch('')
|
||||
setLineLimit(200)
|
||||
setFollowEnabled(false)
|
||||
}
|
||||
|
||||
const handleLogScroll = () => {
|
||||
const viewport = scrollContainerRef.current
|
||||
if (!viewport) return
|
||||
const distanceToBottom = viewport.scrollHeight - viewport.scrollTop - viewport.clientHeight
|
||||
shouldStickToBottomRef.current = distanceToBottom < 32
|
||||
}
|
||||
|
||||
const copyLogs = async () => {
|
||||
await navigator.clipboard.writeText(snapshot?.lines?.join('\n') || '')
|
||||
const text = mode === 'grouped'
|
||||
? groupEvents.map((event) => `${event.occurred_at || ''} ${event.level.toUpperCase()} ${event.message} ${stringifyContext(event.context)}`).join('\n')
|
||||
: snapshot?.lines?.join('\n') || ''
|
||||
await navigator.clipboard.writeText(text)
|
||||
toast({ tone: 'success', title: '日志已复制' })
|
||||
}
|
||||
|
||||
@@ -212,26 +488,78 @@ export default function Logs() {
|
||||
<PageFrame
|
||||
title="系统日志"
|
||||
description="查看日志源、按级别/日期/搜索条件读取快照,并复制原始输出。"
|
||||
className="an-logs-page"
|
||||
actions={(
|
||||
<>
|
||||
<Button size="icon" variant="subtle" onClick={() => void fetchSources()} loading={sourcesLoading} aria-label="刷新日志源" title="刷新日志源">
|
||||
<RefreshCw size={15} />
|
||||
</Button>
|
||||
<Button size="icon" variant="primary" onClick={() => void fetchSnapshot()} loading={logLoading} aria-label="刷新日志" title="刷新日志">
|
||||
{mode === 'raw' ? (
|
||||
<Button
|
||||
size="icon"
|
||||
variant={followEnabled ? 'primary' : 'subtle'}
|
||||
onClick={() => setFollowEnabled((enabled) => !enabled)}
|
||||
aria-label={followEnabled ? '暂停日志跟随' : '跟随日志'}
|
||||
title={followEnabled ? '暂停日志跟随' : '跟随日志'}
|
||||
>
|
||||
{followEnabled ? <Pause size={15} /> : <Play size={15} />}
|
||||
</Button>
|
||||
) : null}
|
||||
<Button size="icon" variant="primary" onClick={() => mode === 'grouped' ? void fetchGroups() : void fetchSnapshot()} loading={logLoading} aria-label="刷新日志" title="刷新日志">
|
||||
<RefreshCw size={15} />
|
||||
</Button>
|
||||
</>
|
||||
)}
|
||||
>
|
||||
<div className="an-section-tabs" role="tablist" aria-label="日志视图">
|
||||
{LOG_VIEW_OPTIONS.map((option) => {
|
||||
const active = mode === option.value
|
||||
return (
|
||||
<button
|
||||
key={option.value}
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected={active}
|
||||
className={active ? 'an-section-tab is-active' : 'an-section-tab'}
|
||||
onClick={() => {
|
||||
setMode(option.value as LogViewMode)
|
||||
setFollowEnabled(false)
|
||||
setSelectedGroup(null)
|
||||
setGroupEvents([])
|
||||
}}
|
||||
>
|
||||
<span className={active ? 'an-section-tab__dot an-section-tab__dot--success' : 'an-section-tab__dot an-section-tab__dot--neutral'} />
|
||||
<span>{option.label}</span>
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
<div className="an-logs-layout">
|
||||
<Panel className="an-logs-sidebar">
|
||||
<div className="an-panel-heading">
|
||||
<div>
|
||||
<h2>日志源</h2>
|
||||
<p>{sources.length} 个来源</p>
|
||||
<h2>{mode === 'grouped' ? '重复统计' : mode === 'audit' ? '审计来源' : '日志源'}</h2>
|
||||
<p>{mode === 'grouped' ? `${groups.length} 个聚合项` : `${sources.length} 个来源`}</p>
|
||||
</div>
|
||||
</div>
|
||||
<Scrollbar className="an-logs-source-list">
|
||||
{mode === 'grouped' ? (
|
||||
<Scrollbar className="an-logs-source-list">
|
||||
{groups.map((group) => (
|
||||
<button
|
||||
type="button"
|
||||
key={group.fingerprint}
|
||||
className={selectedGroup?.fingerprint === group.fingerprint ? 'an-logs-source is-active' : 'an-logs-source'}
|
||||
onClick={() => void fetchGroupEvents(group)}
|
||||
>
|
||||
<strong>{group.message}</strong>
|
||||
<span>{group.category || group.event || group.module || group.source}</span>
|
||||
<StatusText tone={levelTone(group.level)}>{group.count} 次</StatusText>
|
||||
</button>
|
||||
))}
|
||||
{!groups.length && !logLoading ? <EmptyState title="暂无重复日志" /> : null}
|
||||
</Scrollbar>
|
||||
) : (
|
||||
<Scrollbar className="an-logs-source-list">
|
||||
{sources.map((source) => (
|
||||
<button
|
||||
type="button"
|
||||
@@ -245,18 +573,20 @@ export default function Logs() {
|
||||
</button>
|
||||
))}
|
||||
{!sources.length && !sourcesLoading ? <EmptyState title="暂无日志源" /> : null}
|
||||
</Scrollbar>
|
||||
</Scrollbar>
|
||||
)}
|
||||
</Panel>
|
||||
|
||||
<Panel className="an-logs-main">
|
||||
<div className="an-panel-heading">
|
||||
<div>
|
||||
<h2>{selectedSourceInfo?.name || selectedSource}</h2>
|
||||
<p>{selectedSourceInfo?.description || '选择日志源后读取快照。'}</p>
|
||||
<h2>{mode === 'grouped' ? (selectedGroup ? '重复日志详情' : '重复日志统计') : mode === 'audit' ? '审计事件' : (selectedSourceInfo?.name || selectedSource)}</h2>
|
||||
<p>{mode === 'grouped' ? '点击左侧聚合项查看每次发生时间。' : mode === 'audit' ? '管理员敏感操作和安全审计记录。' : (selectedSourceInfo?.description || '选择日志源后读取快照。')}</p>
|
||||
</div>
|
||||
<div className="an-toolbar">
|
||||
{snapshot ? <Badge tone="blue">{snapshot.line_count} 行</Badge> : null}
|
||||
<Button size="icon" variant="subtle" onClick={copyLogs} disabled={!snapshot?.lines?.length} aria-label="复制日志" title="复制日志">
|
||||
{mode === 'grouped' ? <Badge tone="blue">{selectedGroup ? groupEvents.length : groups.length} 条</Badge> : snapshot ? <Badge tone="blue">{snapshot.line_count} 行</Badge> : null}
|
||||
{followEnabled ? <Badge tone={followConnected ? 'green' : 'amber'}>{followConnected ? '跟随中' : '连接中'}</Badge> : null}
|
||||
<Button size="icon" variant="subtle" onClick={copyLogs} disabled={mode === 'grouped' ? !groupEvents.length : !snapshot?.lines?.length} aria-label="复制日志" title="复制日志">
|
||||
<Copy size={15} />
|
||||
</Button>
|
||||
</div>
|
||||
@@ -288,8 +618,32 @@ export default function Logs() {
|
||||
<div className="an-log-reader">
|
||||
{logLoading ? (
|
||||
<div className="an-loading"><span className="an-spinner" />加载中</div>
|
||||
) : mode === 'grouped' && selectedGroup ? (
|
||||
<Scrollbar className="an-log-reader__scroll" viewportRef={scrollContainerRef}>
|
||||
<div className="an-log-group-detail">
|
||||
<div className="an-log-group-summary">
|
||||
<strong>{selectedGroup.message}</strong>
|
||||
<span>指纹 {selectedGroup.fingerprint}</span>
|
||||
<span>首次 {formatDateTime(selectedGroup.first_seen_at)} · 最近 {formatDateTime(selectedGroup.last_seen_at)}</span>
|
||||
</div>
|
||||
{groupEvents.map((event) => (
|
||||
<div className="an-log-occurrence" key={event.id}>
|
||||
<div>
|
||||
<strong>{formatDateTime(event.occurred_at)}</strong>
|
||||
<StatusText tone={levelTone(event.level)}>{event.level}</StatusText>
|
||||
{event.occurrence_count > 1 ? <Badge tone="blue">{event.occurrence_count} 次</Badge> : null}
|
||||
</div>
|
||||
<p>{event.message}</p>
|
||||
<pre>{stringifyContext(event.context)}</pre>
|
||||
</div>
|
||||
))}
|
||||
{!groupEvents.length ? <EmptyState title="暂无发生明细" /> : null}
|
||||
</div>
|
||||
</Scrollbar>
|
||||
) : mode === 'grouped' ? (
|
||||
<EmptyState title="选择一组重复日志" description="左侧展示按 fingerprint 聚合后的运行时错误。" />
|
||||
) : snapshot?.lines?.length ? (
|
||||
<Scrollbar className="an-log-reader__scroll">
|
||||
<Scrollbar className="an-log-reader__scroll" viewportRef={scrollContainerRef}>
|
||||
<pre>{snapshot.lines.join('\n')}</pre>
|
||||
</Scrollbar>
|
||||
) : (
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,6 @@
|
||||
import { type HTMLAttributes, type ReactNode } from 'react'
|
||||
import Scrollbar from '../../components/Scrollbar/Scrollbar'
|
||||
import { cn } from '../lib/utils'
|
||||
import { cn } from '../utils'
|
||||
|
||||
export function PageFrame({
|
||||
title,
|
||||
|
||||
@@ -48,7 +48,7 @@ export const adminRoutes: AdminRouteItem[] = [
|
||||
{ path: '/alerts/bgp', label: 'BGP 告警', group: 'alerts', icon: Network, keywords: ['alert', 'bgp', '风险'] },
|
||||
{ path: '/alerts/situational', label: '态势告警', group: 'alerts', icon: Globe2, keywords: ['situational', '态势', '研判'] },
|
||||
{ path: '/ai', label: 'AI', group: 'ops', icon: Bot, keywords: ['ai', 'provider', 'playground', 'prompt'] },
|
||||
{ path: '/earth-content', label: '智能星球内容', group: 'ops', icon: Globe2, keywords: ['earth', '地球', '智能星球', 'tv', 'boundary', 'brand'] },
|
||||
{ path: '/earth-content', label: '智能星球内容', group: 'ops', icon: Globe2, keywords: ['earth', '地球', '智能星球', 'tv', 'boundary', 'brand', 'news', 'rss', '新闻源'] },
|
||||
{ path: '/collection-management', label: '采集管理', group: 'ops', icon: Database, keywords: ['collector', 'mapping', 'custom source'] },
|
||||
{ path: '/logs', label: '系统日志', group: 'ops', icon: FileText, keywords: ['log', '日志', 'tail'], superAdminOnly: true },
|
||||
{ path: '/users', label: '用户管理', group: 'ops', icon: Users, keywords: ['users', 'role', 'gatekeeper'] },
|
||||
|
||||
167
frontend/src/admin/runtimeLogs.ts
Normal file
167
frontend/src/admin/runtimeLogs.ts
Normal file
@@ -0,0 +1,167 @@
|
||||
const RECENT_EVENT_TTL_MS = 15_000
|
||||
const MAX_DETAIL_LENGTH = 4000
|
||||
const recentEventMap = new Map<string, number>()
|
||||
const pendingEventMap = new Map<string, {
|
||||
level: string
|
||||
message: string
|
||||
category: string
|
||||
module: string
|
||||
detail: string
|
||||
fingerprint: string
|
||||
occurrenceCount: number
|
||||
}>()
|
||||
let pendingFlushTimer: number | null = null
|
||||
const NON_ADMIN_PATH_PREFIXES = ['/earth', '/docs', '/login', '/register', '/verify-email', '/forgot-password']
|
||||
|
||||
type RuntimeLogLevel = 'error' | 'warning' | 'info' | 'debug'
|
||||
|
||||
type AdminRuntimeLogInput = {
|
||||
level?: RuntimeLogLevel
|
||||
message: string
|
||||
category?: string
|
||||
module?: string
|
||||
detail?: unknown
|
||||
}
|
||||
|
||||
function normalizeErrorDetail(detail: unknown) {
|
||||
if (!detail) return ''
|
||||
if (detail instanceof Error) {
|
||||
return detail.stack || detail.message || String(detail)
|
||||
}
|
||||
if (typeof detail === 'string') return detail
|
||||
try {
|
||||
return JSON.stringify(detail)
|
||||
} catch {
|
||||
return String(detail)
|
||||
}
|
||||
}
|
||||
|
||||
function shouldSkip(level: string, message: string, detail: string, category: string) {
|
||||
const key = `${level}::${category}::${message}::${detail}`
|
||||
const now = Date.now()
|
||||
const lastSeenAt = recentEventMap.get(key)
|
||||
recentEventMap.set(key, now)
|
||||
for (const [entryKey, entryTime] of recentEventMap.entries()) {
|
||||
if (now - entryTime > RECENT_EVENT_TTL_MS) {
|
||||
recentEventMap.delete(entryKey)
|
||||
}
|
||||
}
|
||||
return Boolean(lastSeenAt && now - lastSeenAt < RECENT_EVENT_TTL_MS)
|
||||
}
|
||||
|
||||
function normalizeFingerprintText(value: unknown) {
|
||||
return String(value || '')
|
||||
.replace(/[?&](m|t|token|expires|signature|X-Amz-[^=]+)=[^&\s]+/gi, '')
|
||||
.replace(/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi, '<uuid>')
|
||||
.replace(/\bconn_[A-Za-z0-9:._-]+\b/g, '<connection>')
|
||||
.replace(/\b\d{5,}\b/g, '<number>')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
}
|
||||
|
||||
function hashFingerprint(value: string) {
|
||||
let hash = 5381
|
||||
for (let index = 0; index < value.length; index += 1) {
|
||||
hash = ((hash << 5) + hash) ^ value.charCodeAt(index)
|
||||
}
|
||||
return `admin-${(hash >>> 0).toString(16).padStart(8, '0')}`
|
||||
}
|
||||
|
||||
function buildFingerprint(level: string, message: string, detail: string, category: string, module: string) {
|
||||
return hashFingerprint([level, category, module, message, detail].map(normalizeFingerprintText).join('|'))
|
||||
}
|
||||
|
||||
function isAdminRoute() {
|
||||
if (typeof window === 'undefined') return false
|
||||
const pathname = window.location.pathname
|
||||
return pathname !== '/' && !NON_ADMIN_PATH_PREFIXES.some((prefix) => pathname === prefix || pathname.startsWith(`${prefix}/`))
|
||||
}
|
||||
|
||||
export async function reportAdminRuntimeLog({
|
||||
level = 'error',
|
||||
message,
|
||||
category = 'runtime',
|
||||
module = 'admin',
|
||||
detail = '',
|
||||
}: AdminRuntimeLogInput) {
|
||||
if (!message || typeof window === 'undefined' || !isAdminRoute()) return
|
||||
const normalizedDetail = normalizeErrorDetail(detail)
|
||||
const fingerprint = buildFingerprint(level, message, normalizedDetail, category, module)
|
||||
const key = `${level}::${category}::${message}::${normalizedDetail}`
|
||||
const existing = pendingEventMap.get(key)
|
||||
if (existing) {
|
||||
existing.occurrenceCount += 1
|
||||
} else {
|
||||
pendingEventMap.set(key, {
|
||||
level,
|
||||
message,
|
||||
category,
|
||||
module,
|
||||
detail: normalizedDetail,
|
||||
fingerprint,
|
||||
occurrenceCount: 1,
|
||||
})
|
||||
}
|
||||
shouldSkip(level, message, normalizedDetail, category)
|
||||
if (pendingFlushTimer) return
|
||||
pendingFlushTimer = window.setTimeout(() => {
|
||||
pendingFlushTimer = null
|
||||
const pending = Array.from(pendingEventMap.values())
|
||||
pendingEventMap.clear()
|
||||
pending.forEach((entry) => {
|
||||
void sendAdminRuntimeLog(entry)
|
||||
})
|
||||
}, 1000)
|
||||
}
|
||||
|
||||
async function sendAdminRuntimeLog(entry: {
|
||||
level: string
|
||||
message: string
|
||||
category: string
|
||||
module: string
|
||||
detail: string
|
||||
fingerprint: string
|
||||
occurrenceCount: number
|
||||
}) {
|
||||
try {
|
||||
await fetch('/api/v1/system/logs/admin-client', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
level: entry.level,
|
||||
message: entry.message,
|
||||
category: entry.category,
|
||||
module: entry.module,
|
||||
url: window.location.href,
|
||||
detail: entry.detail.slice(0, MAX_DETAIL_LENGTH),
|
||||
fingerprint: entry.fingerprint,
|
||||
occurrence_count: entry.occurrenceCount,
|
||||
}),
|
||||
keepalive: true,
|
||||
})
|
||||
} catch {
|
||||
// Runtime log reporting must never create more runtime noise.
|
||||
}
|
||||
}
|
||||
|
||||
export function registerAdminRuntimeErrorHandlers() {
|
||||
if (typeof window === 'undefined') return
|
||||
window.addEventListener('error', (event) => {
|
||||
void reportAdminRuntimeLog({
|
||||
level: 'error',
|
||||
category: 'window-error',
|
||||
module: 'admin',
|
||||
message: event.message || '控制台发生未捕获错误',
|
||||
detail: event.error || `${event.filename || ''}:${event.lineno || 0}:${event.colno || 0}`,
|
||||
})
|
||||
})
|
||||
window.addEventListener('unhandledrejection', (event) => {
|
||||
void reportAdminRuntimeLog({
|
||||
level: 'error',
|
||||
category: 'unhandledrejection',
|
||||
module: 'admin',
|
||||
message: '控制台发生未处理 Promise 错误',
|
||||
detail: event.reason,
|
||||
})
|
||||
})
|
||||
}
|
||||
@@ -81,6 +81,7 @@ const sectionTargets = [
|
||||
{ key: 'brand', label: '品牌标识', terms: ['logo', '标题', 'subtitle'] },
|
||||
{ key: 'earth_assets', label: '国界精度', terms: ['boundary', 'PMTiles', '边界'] },
|
||||
{ key: 'tv', label: '电视直播', terms: ['TV', '直播源', '频道'] },
|
||||
{ key: 'news_sources', label: '新闻源', terms: ['news', 'rss', '商业新闻', '电商', '财经', '新闻类型'] },
|
||||
] },
|
||||
{ routePath: '/collection-management', routeLabel: '采集管理', icon: Database, sections: [
|
||||
{ key: 'collector_credentials', label: '采集器', terms: ['collector', 'credential', '凭证教程'] },
|
||||
|
||||
@@ -444,8 +444,9 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
}
|
||||
|
||||
.an-collection-queue__item {
|
||||
position: relative;
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr) auto auto;
|
||||
grid-template-columns: minmax(0, 1fr) 44px;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 8px;
|
||||
@@ -455,6 +456,12 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
}
|
||||
|
||||
.an-collection-queue__item-actions {
|
||||
position: absolute;
|
||||
top: 6px;
|
||||
right: 6px;
|
||||
z-index: 2;
|
||||
display: flex;
|
||||
gap: 6px;
|
||||
opacity: 0;
|
||||
transform: translateX(4px);
|
||||
pointer-events: none;
|
||||
@@ -470,6 +477,7 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
|
||||
.an-collection-queue__item strong,
|
||||
.an-collection-queue__item p {
|
||||
max-width: 100%;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
@@ -522,6 +530,10 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
transform: none;
|
||||
pointer-events: auto;
|
||||
}
|
||||
|
||||
.an-collection-queue__item {
|
||||
padding-right: 104px;
|
||||
}
|
||||
}
|
||||
|
||||
.an-task-summary {
|
||||
@@ -577,6 +589,12 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.an-task-summary__actions {
|
||||
grid-column: 1 / -1;
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.admin__content {
|
||||
min-width: 0;
|
||||
height: 100vh;
|
||||
@@ -1523,6 +1541,10 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
container-type: inline-size;
|
||||
}
|
||||
|
||||
.an-hierarchy-list__header {
|
||||
padding: 10px 10px 0;
|
||||
}
|
||||
|
||||
.an-hierarchy-list__scroll,
|
||||
.an-hierarchy-form-scroll {
|
||||
min-height: 0;
|
||||
@@ -1544,6 +1566,33 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
padding: 10px 0 4px;
|
||||
}
|
||||
|
||||
.an-news-source-filters {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, minmax(140px, 1fr));
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.an-hierarchy-workspace {
|
||||
min-height: 0;
|
||||
display: grid;
|
||||
grid-template-rows: auto minmax(0, 1fr);
|
||||
gap: var(--an-section-gap, 16px);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.an-news-source-filters {
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 560px) {
|
||||
.an-news-source-filters {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
|
||||
.an-hierarchy-group-set {
|
||||
display: grid;
|
||||
gap: 6px;
|
||||
@@ -2322,6 +2371,35 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.an-news-feed-list {
|
||||
display: grid;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.an-news-feed-card {
|
||||
display: grid;
|
||||
gap: 10px;
|
||||
padding: 10px;
|
||||
border: 1px solid var(--an-border);
|
||||
border-radius: 7px;
|
||||
background: var(--an-surface-alt);
|
||||
}
|
||||
|
||||
.an-news-feed-card__header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.an-news-feed-card__actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.an-textarea {
|
||||
height: auto;
|
||||
min-height: 94px;
|
||||
@@ -2385,7 +2463,7 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
.an-select__content {
|
||||
z-index: 300;
|
||||
min-width: var(--radix-select-trigger-width);
|
||||
max-width: min(360px, calc(100vw - 24px));
|
||||
max-width: min(420px, calc(100vw - 24px));
|
||||
background: var(--an-surface, #ffffff);
|
||||
color: var(--an-text, #0f172a);
|
||||
border: 1px solid var(--an-border);
|
||||
@@ -2404,21 +2482,32 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
min-height: 30px;
|
||||
max-width: 100%;
|
||||
border-radius: 4px;
|
||||
padding: 0 8px;
|
||||
display: flex;
|
||||
padding: 6px 28px 6px 8px;
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr);
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
cursor: pointer;
|
||||
outline: none;
|
||||
line-height: 1.25;
|
||||
white-space: nowrap;
|
||||
white-space: normal;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.an-select__item span:first-child {
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
max-width: 100%;
|
||||
overflow: visible;
|
||||
text-overflow: clip;
|
||||
white-space: normal;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.an-select__indicator {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
right: 8px;
|
||||
display: inline-flex;
|
||||
transform: translateY(-50%);
|
||||
}
|
||||
|
||||
.an-select__item[data-highlighted] {
|
||||
@@ -2455,34 +2544,64 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
background: currentColor;
|
||||
}
|
||||
|
||||
.an-badge--green,
|
||||
.an-status-pill--success {
|
||||
.an-badge--green {
|
||||
border-color: color-mix(in srgb, var(--an-success) 34%, var(--an-border));
|
||||
background: color-mix(in srgb, var(--an-success) 12%, var(--an-surface-alt));
|
||||
color: var(--an-success);
|
||||
}
|
||||
|
||||
.an-badge--amber,
|
||||
.an-status-pill--warning {
|
||||
.an-badge--amber {
|
||||
border-color: color-mix(in srgb, var(--an-warning) 34%, var(--an-border));
|
||||
background: color-mix(in srgb, var(--an-warning) 12%, var(--an-surface-alt));
|
||||
color: var(--an-warning);
|
||||
}
|
||||
|
||||
.an-badge--red,
|
||||
.an-status-pill--danger {
|
||||
.an-badge--red {
|
||||
border-color: color-mix(in srgb, var(--an-danger) 34%, var(--an-border));
|
||||
background: color-mix(in srgb, var(--an-danger) 12%, var(--an-surface-alt));
|
||||
color: var(--an-danger);
|
||||
}
|
||||
|
||||
.an-badge--blue,
|
||||
.an-badge--cyan,
|
||||
.an-badge--cyan {
|
||||
border-color: color-mix(in srgb, var(--an-info) 34%, var(--an-border));
|
||||
background: color-mix(in srgb, var(--an-info) 12%, var(--an-surface-alt));
|
||||
color: var(--an-info);
|
||||
}
|
||||
|
||||
.an-badge--purple {
|
||||
border-color: color-mix(in srgb, #7c3aed 34%, var(--an-border));
|
||||
background: color-mix(in srgb, #7c3aed 12%, var(--an-surface-alt));
|
||||
color: #7c3aed;
|
||||
}
|
||||
|
||||
.an-badge--slate {
|
||||
border-color: color-mix(in srgb, var(--an-muted) 28%, var(--an-border));
|
||||
background: color-mix(in srgb, var(--an-muted) 10%, var(--an-surface-alt));
|
||||
color: var(--an-muted);
|
||||
}
|
||||
|
||||
.an-status-pill--success {
|
||||
color: var(--an-success);
|
||||
}
|
||||
|
||||
.an-status-pill--warning {
|
||||
color: var(--an-warning);
|
||||
}
|
||||
|
||||
.an-status-pill--danger {
|
||||
color: var(--an-danger);
|
||||
}
|
||||
|
||||
.an-status-pill--info,
|
||||
.an-status-pill--running {
|
||||
color: var(--an-info);
|
||||
}
|
||||
|
||||
.an-badge--purple,
|
||||
.an-status-pill--ai {
|
||||
color: #7c3aed;
|
||||
}
|
||||
|
||||
.an-badge--slate,
|
||||
.an-status-pill--neutral {
|
||||
color: var(--an-muted);
|
||||
}
|
||||
@@ -3276,9 +3395,15 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
line-height: 1.65;
|
||||
}
|
||||
|
||||
.an-logs-page .an-page__body {
|
||||
display: grid;
|
||||
grid-template-rows: auto minmax(0, 1fr);
|
||||
gap: var(--an-section-gap);
|
||||
}
|
||||
|
||||
.an-logs-layout {
|
||||
min-height: 0;
|
||||
height: min(760px, calc(100vh - 170px));
|
||||
height: 100%;
|
||||
display: grid;
|
||||
grid-template-columns: 280px minmax(0, 1fr);
|
||||
gap: var(--an-section-gap);
|
||||
@@ -3323,7 +3448,7 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
background: var(--an-surface-alt);
|
||||
}
|
||||
|
||||
.an-logs-source span {
|
||||
.an-logs-source > span:not(.an-status-pill) {
|
||||
color: var(--an-muted);
|
||||
font-size: 12px;
|
||||
}
|
||||
@@ -3366,6 +3491,54 @@ body:has(.admin-theme-root[data-theme='dark']) .an-toast {
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||||
}
|
||||
|
||||
.an-log-group-detail {
|
||||
display: grid;
|
||||
gap: 10px;
|
||||
padding: 10px;
|
||||
color: #dbeafe;
|
||||
}
|
||||
|
||||
.an-log-group-summary,
|
||||
.an-log-occurrence {
|
||||
border: 1px solid rgba(148, 163, 184, 0.28);
|
||||
border-radius: 8px;
|
||||
background: rgba(15, 23, 42, 0.72);
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.an-log-group-summary {
|
||||
display: grid;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.an-log-group-summary span,
|
||||
.an-log-occurrence p {
|
||||
color: #94a3b8;
|
||||
}
|
||||
|
||||
.an-log-occurrence {
|
||||
display: grid;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.an-log-occurrence > div {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.an-log-occurrence pre {
|
||||
min-width: 0;
|
||||
border-radius: 6px;
|
||||
background: rgba(2, 6, 23, 0.82);
|
||||
padding: 10px;
|
||||
max-height: 220px;
|
||||
overflow: auto;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.an-playground {
|
||||
min-height: 0;
|
||||
height: 100%;
|
||||
|
||||
@@ -2,8 +2,11 @@ import React from 'react'
|
||||
import ReactDOM from 'react-dom/client'
|
||||
import { BrowserRouter } from 'react-router-dom'
|
||||
import App from './App'
|
||||
import { registerAdminRuntimeErrorHandlers } from './admin/runtimeLogs'
|
||||
import './index.css'
|
||||
|
||||
registerAdminRuntimeErrorHandlers()
|
||||
|
||||
ReactDOM.createRoot(document.getElementById('root')!).render(
|
||||
<React.StrictMode>
|
||||
<BrowserRouter future={{ v7_startTransition: true, v7_relativeSplatPath: true }}>
|
||||
|
||||
@@ -111,9 +111,17 @@ export const DOCS_METADATA: Record<string, DocsMetadataEntry> = {
|
||||
zh: { title: '智能星球可交互图标接入', group: 'Earth', order: 16 },
|
||||
en: { title: 'Intelligent Planet Interactable Usage', group: 'Earth', order: 16 },
|
||||
},
|
||||
'earth-interactable-clustering.md': {
|
||||
zh: { title: '智能星球可交互图标聚类策略', group: 'Earth', order: 17 },
|
||||
en: { title: 'Intelligent Planet Interactable Clustering', group: 'Earth', order: 17 },
|
||||
},
|
||||
'earth-toolbar-overlay-coordination.md': {
|
||||
zh: { title: '智能星球工具栏与浮层协同', group: 'Earth', order: 17 },
|
||||
en: { title: 'Intelligent Planet Toolbar and Overlay Coordination', group: 'Earth', order: 17 },
|
||||
zh: { title: '智能星球工具栏与浮层协同', group: 'Earth', order: 18 },
|
||||
en: { title: 'Intelligent Planet Toolbar and Overlay Coordination', group: 'Earth', order: 18 },
|
||||
},
|
||||
'earth-news-sources.md': {
|
||||
zh: { title: '智能星球新闻源配置', group: 'Earth', order: 19 },
|
||||
en: { title: 'Intelligent Planet News Source Configuration', group: 'Earth', order: 19 },
|
||||
},
|
||||
'frontend-admin-frontend-context.md': {
|
||||
zh: { title: '控制台前端结构', group: 'Frontend', order: 20 },
|
||||
|
||||
90
planet.sh
90
planet.sh
@@ -138,6 +138,7 @@ AI_PROVIDER_IMAGE_NAME="${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}"
|
||||
AI_PROVIDER_CONTAINER_NAME="${AI_PROVIDER_CONTAINER_NAME:-planet_aiprovider}"
|
||||
PLANET_AI_PROVIDER_RUNTIME_ENV_FILE="${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-$PLANET_STATE_DIR/aiprovider_runtime.env}"
|
||||
PLANET_EMPTY_UV_CONFIG_FILE="$PLANET_STATE_DIR/uv.empty.toml"
|
||||
PLANET_TUNA_UV_CONFIG_FILE="$PLANET_STATE_DIR/uv.tuna.toml"
|
||||
PLANET_UV_CONFIG_FILE="${PLANET_UV_CONFIG_FILE:-}"
|
||||
AI_PROVIDER_RECREATE_REQUIRED=0
|
||||
START_RUN_ACTIVE=0
|
||||
@@ -1111,55 +1112,94 @@ run_with_retry() {
|
||||
|
||||
run_uv_sync_with_mirror_fallback() {
|
||||
local log_file="$1"
|
||||
local lock_digest_before
|
||||
local sync_status
|
||||
|
||||
lock_digest_before="$(uv_lock_digest)"
|
||||
if run_with_retry \
|
||||
"$DEPENDENCY_INSTALL_MAX_RETRIES" \
|
||||
"$DEPENDENCY_INSTALL_RETRY_INTERVAL" \
|
||||
"uv sync 默认源失败,已重试 ${DEPENDENCY_INSTALL_MAX_RETRIES} 次,准备切换清华源" \
|
||||
"uv sync --frozen" \
|
||||
run_command_quiet_unless_verbose "$log_file" uv sync --frozen --group dev; then
|
||||
assert_uv_lock_unchanged "$lock_digest_before" "uv sync --frozen"
|
||||
return 0
|
||||
fi
|
||||
assert_uv_lock_unchanged "$lock_digest_before" "uv sync --frozen"
|
||||
|
||||
configure_uv_tuna_index
|
||||
set_wait_detail "已写入 uv.toml 清华源,重新执行 uv sync"
|
||||
set_wait_detail "已准备临时清华源配置,重新执行 uv sync"
|
||||
|
||||
lock_digest_before="$(uv_lock_digest)"
|
||||
run_with_retry \
|
||||
"$DEPENDENCY_INSTALL_MAX_RETRIES" \
|
||||
"$DEPENDENCY_INSTALL_RETRY_INTERVAL" \
|
||||
"uv 环境初始化失败,清华源重试 ${DEPENDENCY_INSTALL_MAX_RETRIES} 次后仍失败" \
|
||||
"uv sync --frozen (清华源)" \
|
||||
run_uv_sync_with_tuna_config "$log_file"
|
||||
sync_status=$?
|
||||
assert_uv_lock_unchanged "$lock_digest_before" "uv sync --frozen (清华源)"
|
||||
return "$sync_status"
|
||||
}
|
||||
|
||||
run_uv_sync_with_tuna_config() {
|
||||
local log_file="$1"
|
||||
|
||||
UV_CONFIG_FILE="$PLANET_TUNA_UV_CONFIG_FILE" \
|
||||
run_command_quiet_unless_verbose "$log_file" uv sync --frozen --group dev
|
||||
}
|
||||
|
||||
configure_uv_tuna_index() {
|
||||
local uv_config="$SCRIPT_DIR/uv.toml"
|
||||
local uv_config="$PLANET_TUNA_UV_CONFIG_FILE"
|
||||
|
||||
if [ -f "$uv_config" ] &&
|
||||
grep -Fq 'name = "tsinghua"' "$uv_config" 2>/dev/null &&
|
||||
grep -Fq "$PLANET_UV_TUNA_INDEX_URL" "$uv_config" 2>/dev/null &&
|
||||
grep -Eq '^[[:space:]]*default[[:space:]]*=[[:space:]]*true' "$uv_config" 2>/dev/null; then
|
||||
log_note "uv.toml 已配置清华源,直接重试 uv sync"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ -f "$uv_config" ] && grep -Eq '^[[:space:]]*default[[:space:]]*=[[:space:]]*true' "$uv_config" 2>/dev/null; then
|
||||
log_warn "uv.toml 已存在默认 index,未覆盖用户配置"
|
||||
log_note "如需手动切换清华源,可添加 [[index]] name=\"tsinghua\" 并设置 default=true"
|
||||
log_note "临时 uv 清华源配置已存在,直接重试 uv sync"
|
||||
return 0
|
||||
fi
|
||||
|
||||
{
|
||||
if [ -s "$uv_config" ]; then
|
||||
printf "\n"
|
||||
fi
|
||||
printf '[[index]]\n'
|
||||
printf 'name = "tsinghua"\n'
|
||||
printf 'url = "%s"\n' "$PLANET_UV_TUNA_INDEX_URL"
|
||||
printf 'default = true\n'
|
||||
} >> "$uv_config"
|
||||
} > "$uv_config"
|
||||
|
||||
log_note "已向 uv.toml 添加清华 PyPI 源: ${PLANET_UV_TUNA_INDEX_URL}"
|
||||
chmod 600 "$uv_config" 2>/dev/null || true
|
||||
log_note "已写入临时 uv 清华源配置: ${uv_config}"
|
||||
}
|
||||
|
||||
uv_lock_digest() {
|
||||
local lock_file="$SCRIPT_DIR/uv.lock"
|
||||
|
||||
if [ ! -f "$lock_file" ]; then
|
||||
printf "missing\n"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if command -v sha256sum >/dev/null 2>&1; then
|
||||
sha256sum "$lock_file" | awk '{print $1}'
|
||||
return 0
|
||||
fi
|
||||
|
||||
cksum "$lock_file" | awk '{print $1 ":" $2}'
|
||||
}
|
||||
|
||||
assert_uv_lock_unchanged() {
|
||||
local expected_digest="$1"
|
||||
local action_label="$2"
|
||||
local current_digest
|
||||
|
||||
current_digest="$(uv_lock_digest)"
|
||||
if [ "$current_digest" = "$expected_digest" ]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_error "${action_label} 修改了 uv.lock;新环境依赖安装不允许污染 lockfile"
|
||||
log_note "请还原 uv.lock,并只在明确升级依赖时手动运行 uv lock"
|
||||
exit 1
|
||||
}
|
||||
|
||||
install_system_package() {
|
||||
@@ -1763,19 +1803,15 @@ ensure_frontend_deps() {
|
||||
cd "$SCRIPT_DIR/frontend"
|
||||
: > "$log_file"
|
||||
|
||||
set_wait_detail "检查 Vite Bun 入口是否已安装"
|
||||
if [ ! -f "$FRONTEND_VITE_ENTRY" ]; then
|
||||
log_warn "前端依赖缺失,正在执行 bun install (${FRONTEND_RUNTIME_SOURCE})"
|
||||
set_wait_detail "执行 ${FRONTEND_RUNTIME_SOURCE} bun install"
|
||||
if ! run_with_retry \
|
||||
"$DEPENDENCY_INSTALL_MAX_RETRIES" \
|
||||
"$DEPENDENCY_INSTALL_RETRY_INTERVAL" \
|
||||
"前端依赖安装失败,已重试 ${DEPENDENCY_INSTALL_MAX_RETRIES} 次" \
|
||||
"bun install" \
|
||||
run_command_quiet_unless_verbose "$log_file" "$FRONTEND_RUNTIME_BIN" install; then
|
||||
tail -20 "$log_file" 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
set_wait_detail "同步前端依赖"
|
||||
if ! run_with_retry \
|
||||
"$DEPENDENCY_INSTALL_MAX_RETRIES" \
|
||||
"$DEPENDENCY_INSTALL_RETRY_INTERVAL" \
|
||||
"前端依赖安装失败,已重试 ${DEPENDENCY_INSTALL_MAX_RETRIES} 次" \
|
||||
"bun install" \
|
||||
run_command_quiet_unless_verbose "$log_file" "$FRONTEND_RUNTIME_BIN" install; then
|
||||
tail -20 "$log_file" 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$FRONTEND_VITE_ENTRY" ]; then
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "planet"
|
||||
version = "0.66.3"
|
||||
version = "0.69.0"
|
||||
description = "智能星球计划 - 态势感知系统"
|
||||
requires-python = ">=3.14"
|
||||
dependencies = [
|
||||
|
||||
Reference in New Issue
Block a user