diff --git a/.claude/commands/cleanup.md b/.claude/commands/cleanup.md deleted file mode 100644 index 97e0d0b1..00000000 --- a/.claude/commands/cleanup.md +++ /dev/null @@ -1,139 +0,0 @@ ---- -description: 审查当前工作区未提交代码中的垃圾代码，并在不影响逻辑的前提下自动清理 -argument-hint: 可选：指定要检查的文件或目录（默认检查所有未提交修改） -allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"] ---- - -# /cleanup — 垃圾代码审查与清理 - -分析当前工作区（git diff）中的未提交代码，找出并修复常见垃圾代码，**不得改变任何运行逻辑**。 - -## 检查范围 - -若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。 - -## 节省上下文规则 - -优先用确定性的 CLI 检查缩小范围，不要一上来把完整文件或大 diff 读入上下文： - -```bash -git diff --name-only HEAD -git diff --unified=0 HEAD -- -git diff --check -rg -n "TODO|FIXME|console\.log|debugger|print\(" -``` - -只有 focused diff 不足以安全判断或修改时，才读取完整文件。 - -## 审查清单 - -按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）： - -### 1. 重复逻辑 (Duplicate Logic) -- 完全相同或高度相似的代码块在多处出现 -- 同一函数/方法被多个地方各自实现，已有公共版本未被复用 -- 相同的 DOM 查询、正则、模板字符串在同一文件重复 - -### 2. Magic Numbers / Magic Strings -- 裸数字直接参与计算（如偏移量、时间、尺寸、阈值），没有命名常量 -- 硬编码字符串（如 id 名、状态值、URL 片段）散落在逻辑中 -- 例外：`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算 - -### 3. 命名问题 -- 含义不明的缩写变量（如 `or_`, `tmp2`, `x2`） -- 命名与实际用途不符 -- 同一概念在不同地方用不同名字表达 - -### 4. 死代码 / 无效代码 -- 注释掉的旧代码块（3行以上） -- 声明后从未使用的变量/参数/导入 -- 永远不会执行的条件分支 - -### 5. 代码风格问题 -- 尾部空白字符（trailing whitespace） -- 同一文件内风格不一致（如混用单双引号、缩进不统一） -- 空行使用不一致（连续多个空行等） - -### 6. 其他常见问题 -- 私有辅助函数应被 export 但没有，导致调用方重复实现 -- 类型/接口重复定义 -- 过于冗长的条件表达式可以简化（不改逻辑） - -## 执行步骤 - -### Step 1 — 获取待检查文件列表 - -```bash -# 无参数时：获取所有未提交修改 -git diff HEAD --name-only - -# 有参数时：用 $ARGUMENTS 过滤 -``` - -### Step 2 — 逐文件阅读并分析 - -先从 focused diff 开始： - -```bash -git diff --unified=0 HEAD -- -``` - -用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式。 - -### Step 3 — 报告问题清单 - -在修改前，先以列表形式输出所有发现的问题： - -``` -发现 N 个问题： - -[文件] js/foo.js - · L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询，可提取到 getPanel() - · L91: Magic number — 硬编码 14 作为偏移量，应命名为 TOOLTIP_OFFSET - -[文件] js/bar.js - · L12: 命名问题 — 变量 `or_` 语义不明，应命名为 outerR/outerG/outerB - ... -``` - -如果没有发现问题，直接输出"未发现垃圾代码，当前代码质量良好。"并停止。 - -### Step 4 — 执行修复 - -对每个问题，使用 Edit 工具进行**最小化修改**： - -- **重复逻辑**：提取为共享常量/函数，更新所有调用点 -- **Magic number**：在文件顶部或逻辑附近声明 `const NAME = value`，替换所有引用 -- **命名问题**：重命名变量，更新所有使用处 -- **死代码**：直接删除 -- **尾部空白/风格**：修正 -- **未 export 的函数**：添加 `export`，在调用方改为导入（不重复实现） - -**修复原则：** -- 只改在审查清单中发现的问题，不做额外优化 -- 每次 Edit 只修改确实有问题的行，保持 diff 最小 -- 改完后用 `grep` 验证旧的坏代码已消失 -- 优先做精确补丁；只有仓库已有对应格式化流程时，才运行格式化工具 - -### Step 5 — 输出总结 - -``` -清理完成： - -修复了 N 个问题： - ✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量 - ✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14（4处引用） - ✓ controls.js — export updateLayerButtonState，移除 main.js 中的重复实现 - ... - -未修改的问题（需人工确认）： - ! foo.js L45 — 注释代码块较长，建议手动确认是否可删除 -``` - -## 约束 - -- **禁止**改变函数签名、接口定义、导出 API（除非问题正是私有函数应被 export） -- **禁止**添加新功能、新抽象、新参数 -- **禁止**修改注释内容（只删除注释掉的死代码） -- **禁止**修改测试文件逻辑 -- 如果一个 Magic number 的语义不完全确定，**跳过**，在总结中标记为"需人工确认" diff --git a/.claude/commands/docs.md b/.claude/commands/docs.md deleted file mode 100644 index 0be584f4..00000000 --- a/.claude/commands/docs.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -description: Create or update repository documentation from current code changes -argument-hint: Optional: topic to document, or leave empty to infer from git diff -allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"] ---- - -# /docs — Documentation Workflow - -## Goal - -Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately. - -## Repository Rules - -Before deciding scope, check whether the repository has a documentation rules file: - -```bash -test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md -``` - -If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below. - -## Workflow - -### Step 1 — Understand The Change - -```bash -git diff HEAD --stat -git diff HEAD --name-only -git log --oneline -10 -rg --files docs -``` - -If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only: - -```bash -git diff HEAD -- -rg -n "class |def |function |export |router|@router|interface |type " -``` - -### Step 2 — Decide Scope - -- Prefer updating an existing relevant document over creating a duplicate. -- Use one document for one coherent topic. -- Split documents only when the change crosses meaningful domains. -- Keep filenames lowercase and hyphenated. -- Apply the repository-specific rules file before writing. - -#### Document Audience Routing (Planet) - -In this repository, classify the action's performer before picking a target file: - -- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`. -- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`). -- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files. - -Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence. - -For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly. - -### Step 3 — Write - -Explain: - -- Background/problem: what was wrong or missing before. -- Core design decisions and rationale. -- Operational or user-facing impact. -- Relevant code paths, only when useful for future maintainers. - -Style: - -- Follow the repository’s existing language and heading conventions. -- Use fenced code blocks with language tags. -- Prefer tables for comparisons or parameter lists. -- Keep snippets concise and relevant. -- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change. - -### Step 4 — Verify - -- Read the completed docs once for clarity and stale statements. -- Verify referenced paths exist with `test -e` or `rg --files`. -- Run applicable checks from `docs/documentation-coverage-rules.md`. -- Check Markdown links use readable user-facing titles unless repository rules allow otherwise. - -### Step 5 — Report - -Summarize changed docs and verification: - -```md -Updated: -- path/to/doc.md — what changed - -Verified: -- checks that passed -- checks that could not be run, if any -``` - -## Hard Constraints - -- Do not leave placeholder docs. -- Do not duplicate bilingual files byte-for-byte. -- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested. -- Do not write changelog-style lists without the reasoning and tradeoffs behind the change. -- Keep docs maintainable and concise. diff --git a/.claude/commands/goal-driven.md b/.claude/commands/goal-driven.md deleted file mode 100644 index 4757a94c..00000000 --- a/.claude/commands/goal-driven.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -description: 用 goal-driven 方法推动一个复杂任务持续执行，直到明确成功标准被满足 -argument-hint: 建议填写任务目标；若同时给出成功标准更好 -allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"] ---- - -# /goal-driven — 目标驱动执行模式 - -使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务：先固定目标与成功标准，再持续执行和反复验收，直到标准真正满足。 - -适用场景： - -- 长周期实现任务 -- 高复杂度工程任务 -- 可被明确验收的研究、实现、迁移、验证类工作 - -不适用场景： - -- 纯脑暴 -- 无法定义成功标准的模糊任务 -- 很小的一次性修改 - -## 输入要求 - -若 `$ARGUMENTS` 只包含目标，没有成功标准，先补全一版可执行的成功标准再开始。 - -启动时先输出： - -```md -Goal -- ... - -Criteria for success -- ... - -Plan -1. ... -2. ... -3. ... - -Verification -- ... -``` - -## 执行规则 - -1. 先把任务固化为两个核心块： - - `Goal` - - `Criteria for success` - -2. 成功标准必须尽量客观，可验证，可落地。 - 优先写成： - - 需要交付什么 - - 需要通过哪些测试或验证 - - 如何判断结果真的完成 - -3. 进入持续执行循环： - - 完成一个阶段 - - 检查当前结果是否满足成功标准 - - 若未满足，明确剩余差距并继续推进 - -4. 任何“完成了”“差不多了”“已实现”之类的结论，都必须经过验证，不能直接接受。 - -5. 如果验证失败： - - 明确指出哪条成功标准没满足 - - 继续工作，不要把阶段性进展误判为完成 - -6. 只有在以下情况之一才能停止： - - 成功标准已满足 - - 用户明确要求停止 - -## 执行风格 - -- 重证据，轻口头判断 -- 优先使用确定性工具证据：`rg`、`git diff --stat`、`git diff -- `、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式 -- 不把大段命令输出粘进回复；保留在工具调用里，回复只总结关键证据 -- 重验收，轻自我感觉 -- 优先用测试、日志、产物、对比结果来证明完成 -- 对长期任务保持“未达标就继续”的节奏 - -## 简版模板 - -```md -Goal: [[[[[在此填写最终目标]]]]] - -Criteria for success: [[[[[在此填写成功标准]]]]] - -循环执行： -1. 推进任务 -2. 检查是否满足成功标准 -3. 若未满足，继续工作 -4. 直到满足标准或用户明确停止 -``` diff --git a/.claude/commands/release.md b/.claude/commands/release.md deleted file mode 100644 index 43ff3813..00000000 --- a/.claude/commands/release.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -description: 发版工作流：根据变更类型决定版本号，更新所有版本文件和 changelog，运行验证，commit 并 push -argument-hint: 可选：feature | bugfix | 或直接描述本次发布内容 -allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"] ---- - -# /release — Planet 发版工作流 - -## 版本号规则 - -| 变更类型 | 版本跳动 | 适用场景 | -|---------|---------|---------| -| `feature` | `+0.1.0` | 纯新功能，无 bugfix | -| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合，或以 UI/体验改进为主的迭代 | -| `bugfix` | `+0.0.1` | 纯 bug 修复，无新功能 | -| `docs` / `maintenance` / `refactor` | 默认不发版，除非用户明确要求 | - -意图混合时以用户明确描述为准；bugfix + 小 feature 混合默认判定为 `improvement`（`+0.0.1`）。 - -## 必须同步更新的文件 - -使用 `git rev-parse --show-toplevel` 获取仓库根目录，以下路径均相对于根目录： - -- `VERSION` -- `frontend/package.json`（`"version"` 字段） -- `pyproject.toml`（`version =` 字段） -- `uv.lock`（**不要手动编辑**，通过 `uv lock` 重新生成） -- `docs/CHANGELOG.md` -- `docs/version-history.md` - -## 节省上下文规则 - -发版判断应以确定性 CLI 证据为主，优先使用紧凑命令和定点读取： - -```bash -git status --short -git diff --stat HEAD -git diff --name-only HEAD -rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md -``` - -除非需要判断某个代码变更是否属于本次发版，否则不要读取完整 diff。 - -## 执行步骤 - -### Step 1 — 环境检查 - -```bash -git branch --show-current # 确认在 dev 分支 -git status --short # 检查是否有无关的未暂存修改 -cat VERSION # 读取当前版本 -``` - -若当前**不在 `dev` 分支**，停下来告知用户，不要继续。 - -若存在无关的未暂存修改，列出并询问用户是否一并提交，或先 stash。 - -### Step 2 — 确定发版类型与新版本号 - -- 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用 -- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断 -- 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`） -- **先输出发版计划供用户确认**： - -``` -发版计划： - 类型：bugfix - 版本：0.26.2 → 0.26.3 - 分支：dev - 将更新：VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md -``` - -### Step 3 — 更新版本号文件 - -按顺序更新（每步用 Edit 工具，精确替换，不要重写整个文件）： - -1. `VERSION` — 直接替换全部内容为新版本号 -2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行 -3. `pyproject.toml` — 替换 `version = "x.x.x"` 行 -4. 运行 `uv lock` 重新生成 `uv.lock`（在仓库根目录下执行） - -### Step 4 — 更新 CHANGELOG.md - -在文件顶部插入新条目，格式： - -```markdown -## [x.x.x] — YYYY-MM-DD - -### ✨ Features / 🐛 Fixes / 🔧 Improvements -- ...（只列高信号条目，最多 5 条） -- ... - ---- -``` - -日期使用 `date +%Y-%m-%d` 获取今天的日期。 - -### Step 5 — 更新 docs/version-history.md - -- 更新文件头部的"当前开发版本"字段 -- 在时间线表格顶部插入新行：`| vx.x.x | YYYY-MM-DD | 一句话摘要 |` - -### Step 6 — 验证 - -针对本次变更范围做最小验证： - -- Python 文件有修改：先用 `git diff --name-only HEAD -- '*.py'` 列出，再运行 `python3 -m py_compile ` -- Frontend 文件有修改：先用 `git diff --name-only HEAD -- frontend` 判断范围，再运行项目标准检查（若无则跳过并说明） -- 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致 - -```bash -cat VERSION -rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock -``` - -### Step 7 — 提交前预览 - -展示将要提交的文件列表： - -```bash -git diff --stat HEAD -``` - -再次确认所有必须文件都在变更列表中，**不包含**非预期文件（如调试文件、.env 等）。 - -### Step 8 — Commit & Push（用户确认后） - -```bash -git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md -# 若有代码变更也一并 stage -git add - -git commit -m "release: bump version to x.x.x" -git tag vx.x.x -git push origin dev -git push origin vx.x.x -``` - -commit message 固定格式：`release: bump version to x.x.x` - -### Step 9 — 完成确认 - -输出摘要： - -``` -✓ 版本号已更新：0.26.2 → 0.26.3 -✓ CHANGELOG 已更新 -✓ version-history 已更新 -✓ uv.lock 已重新生成 -✓ 验证通过 -✓ commit: release: bump version to 0.26.3 -✓ tag: v0.26.3 -✓ 已 push 到 origin/dev -``` - -## 注意事项 - -- `uv.lock` 只能通过 `uv lock` 生成，绝不手动编辑 -- 发版 commit 只包含版本文件 + 本次功能代码，不混入无关改动 -- 若环境中 `uv` 不可用，说明原因并跳过 lockfile 更新，提醒用户手动运行 diff --git a/.codex/screenshots/earth-i18n-current-page.png b/.codex/screenshots/earth-i18n-current-page.png new file mode 100644 index 00000000..a9802312 Binary files /dev/null and b/.codex/screenshots/earth-i18n-current-page.png differ diff --git a/.codex/screenshots/earth-i18n-hd-texture-status.png b/.codex/screenshots/earth-i18n-hd-texture-status.png new file mode 100644 index 00000000..5dc60fd1 Binary files /dev/null and b/.codex/screenshots/earth-i18n-hd-texture-status.png differ diff --git a/.codex/screenshots/i18n-admin-data-sidebar.png b/.codex/screenshots/i18n-admin-data-sidebar.png new file mode 100644 index 00000000..4eb6b6ef Binary files /dev/null and b/.codex/screenshots/i18n-admin-data-sidebar.png differ diff --git a/.codex/screenshots/i18n-ai-tool-calls.png b/.codex/screenshots/i18n-ai-tool-calls.png new file mode 100644 index 00000000..16fef4bb Binary files /dev/null and b/.codex/screenshots/i18n-ai-tool-calls.png differ diff --git a/.codex/screenshots/i18n-earth-brand-config.png b/.codex/screenshots/i18n-earth-brand-config.png new file mode 100644 index 00000000..3808e77a Binary files /dev/null and b/.codex/screenshots/i18n-earth-brand-config.png differ diff --git a/.codex/screenshots/i18n-earth-hud-brand.png b/.codex/screenshots/i18n-earth-hud-brand.png new file mode 100644 index 00000000..c8d8571c Binary files /dev/null and b/.codex/screenshots/i18n-earth-hud-brand.png differ diff --git a/.codex/screenshots/i18n-settings-notifications.png b/.codex/screenshots/i18n-settings-notifications.png new file mode 100644 index 00000000..795bbe45 Binary files /dev/null and b/.codex/screenshots/i18n-settings-notifications.png differ diff --git a/.codex/screenshots/i18n-settings-system.png b/.codex/screenshots/i18n-settings-system.png new file mode 100644 index 00000000..4ff47b13 Binary files /dev/null and b/.codex/screenshots/i18n-settings-system.png differ diff --git a/.codex/screenshots/sidebar-left-align.png b/.codex/screenshots/sidebar-left-align.png new file mode 100644 index 00000000..13d019f7 Binary files /dev/null and b/.codex/screenshots/sidebar-left-align.png differ diff --git a/.gitignore b/.gitignore index 7f7520e8..bf4cc62d 100644 --- a/.gitignore +++ b/.gitignore @@ -25,11 +25,14 @@ __pycache__/ build/ develop-eggs/ dist/ -downloads/ +downloads/* +!downloads/usbipd-win/ +downloads/usbipd-win/* +!downloads/usbipd-win/usbipd-win-5.3.0.msi eggs/ .eggs/ -lib/ -lib64/ +/lib/ +/lib64/ parts/ sdist/ var/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..67d1ec7f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,180 @@ +# AGENTS.md + +**Planet agent harness. Defines behavior for coding agents working in this repository.** + +--- + +## Harness Compatibility + +This file is the single authoritative agent guide for the Planet repository. +The older lowercase `agents.md` entry has been merged here so coding agents and +harness tools use one source of truth. + +### Source Of Truth + +- `rules.md` is the mandatory repository rule source. Always load `core`, + `security`, and `workflow`; load only task-relevant modules after that. +- `AGENTS.md` defines the local agent operating mode and evidence gates. +- `project_context.md` is background, not a rule source. Prefer newer + implementation docs when it disagrees with current code. +- `.codex/skills/` is the active specialized workflow layer for cleanup, docs, + goal-driven work, and release. +- Do not duplicate long workflow text across harness files. Durable constraints + belong in `rules.md`; task procedures belong in skills or scripts. + +Read these files before changing code: + +1. `rules.md` +2. `AGENTS.md` +3. `project_context.md` +4. `README.md` +5. `docs/HARNESS.md` +6. `CODEMAP.md` + +For documentation work, also read `docs/documentation-coverage-rules.md`. + +### Start Safely + +Before broad edits: + +```bash +git status --short +scripts/harness/doctor.sh +``` + +Use focused context commands before reading large files: + +```bash +rg -n "" +git diff --stat HEAD +git diff --name-only HEAD +git diff --unified=0 HEAD -- +``` + +Preserve user changes already present in the worktree. + +### Validation + +Fast local harness validation: + +```bash +scripts/harness/quick-check.sh +``` + +Full local validation: + +```bash +scripts/harness/validate.sh +``` + +`validate.sh` includes quick checks, frontend Bun build, and frontend smoke +unless disabled by its documented environment flags. Docker image smoke builds +are intentionally opt-in: + +```bash +PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh +``` + +Harness scripts resolve `bun`, `uv`, and optional delivery tools from the +current non-interactive environment first. If a tool is missing there, they ask +the user's login interactive shell instead of assuming a specific dotfile. + +### High-Risk Areas + +- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive + `destroy` cleanup. +- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn. +- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use + rendered smoke evidence for public pages, auth guards, authenticated admin + route/section availability, safe navigation/search/tab interactions, mobile + layout, and 125% / 150% zoom, not only a build. +- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the + one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`, + intermediate wrappers keep `min-height: 0`, and only the intended child owns + scrolling. +- `aiprovider` is a protocol/provider adapter; keep business prompts and product + workflows in the backend. +- Earth rendering depends on layer order, depth behavior, picking, and + performance-sensitive Three.js code. +- Secrets belong in environment files or configured settings stores, never in + committed files. +- Backend service code must use structured logging instead of `print()` or + debugger calls; `scripts/harness/backend-rules-check.sh` enforces this. + +### Conflict Policy + +Existing project rules and workflows win. If new harness guidance conflicts with +`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing +behavior and document the compatibility note in `docs/harness-audit.md` or +`docs/HARNESS.md`. + +--- + +## Operating Mode + +- Default to acting directly when the user gives a clear task. +- Ask before acting only when the missing decision is risky, cannot be + discovered from repository context, and no conservative assumption is safe. +- Read relevant files before editing. +- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`, + focused file reads, tests, builds, linters, and harness scripts. +- Keep changes scoped to the requested area. Do not mix cleanup, feature work, + release work, and documentation unless the task requires it. + +--- + +## Evidence Gates + +- Visual inputs are blocking evidence. If the user provides a screenshot, image, + mock, browser capture, or visual reference, obtain evidence from the artifact + before interpreting intent or editing code. +- Path resolution is part of the task. If the path cannot be opened, first try + reasonable local equivalents such as WSL/Windows path conversion, + workspace-relative lookup, absolute paths, and attached-file locations. +- Never guess from prompt text, filenames, previous context, logs, OCR, or + memory when a visual artifact was provided but cannot be accessed. +- OCR is acceptable evidence for text-only visual questions or non-multimodal + environments; state that OCR was used as the fallback. Layout, color, spacing, + pixel, and rendering issues need real visual inspection or a clear limitation + note. +- If a visual artifact still cannot be inspected, say so and pause that + visual-dependent part of the work. +- Claims of completion need evidence: a relevant test, build, lint, screenshot, + diff, direct file check, or harness result. +- For UI and rendering changes, verify the rendered result when local tooling + allows it. + +--- + +## Communication + +- Match the user's language. Use Chinese for Chinese requests unless the user + asks otherwise. +- Keep updates short and specific: what is being inspected, edited, or verified. +- Final responses should summarize changed files and verification, with blockers + stated plainly. +- Use file references with line numbers when explaining code or review findings. + +--- + +## Quality Bar + +- Prefer existing project patterns over new abstractions. +- Remove stale branches, mocks, compatibility paths, and duplicated helpers once + a stable path exists. +- Centralize prompts, constants, defaults, and shared request/response handling. +- Do not add secrets, generated runtime output, or local environment files. +- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`. +- Run the smallest relevant verification for the changed scope and report + anything skipped. + +--- + +## Prohibited + +- Do not skip visual evidence handling when a visual artifact was provided. +- Do not preserve obsolete harness files just because they already exist. +- Do not invent behavior not present in code, docs, or verified external + sources. +- Do not rewrite unrelated files during cleanup. +- Do not mark a task complete without checking concrete success criteria. diff --git a/CODEMAP.md b/CODEMAP.md new file mode 100644 index 00000000..8619fff3 --- /dev/null +++ b/CODEMAP.md @@ -0,0 +1,109 @@ +# Code Map + +This map gives agents and maintainers a quick orientation without replacing the +deeper architecture docs. Current implementation docs under `docs/technical/` +are the source of detail for specific subsystems. + +## Top-Level Areas + +| Path | Role | Notes | +| --- | --- | --- | +| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. | +| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. | +| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. | +| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. | +| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. | +| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. | +| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. | +| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. | +| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. | + +## Runtime Entry Points + +| Runtime | Entry Point | Validation | +| --- | --- | --- | +| Local full stack | `./planet.sh start` | `./planet.sh health` | +| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` | +| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` | +| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup | +| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup | +| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build | + +## Ownership Boundaries + +- Backend owns business state, auth, evidence collection, prompt selection, AI + task orchestration, and database persistence. +- `aiprovider` owns provider identity, request adapter style, model gateway + retries, and health/status endpoints only. +- Frontend owns operator workflows, Docs presentation, Web Earth orchestration, + and client-side state that mirrors backend truth. +- Web Earth rendering changes must preserve documented layer order, altitude + offsets, picking behavior, legend semantics, and performance constraints. +- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer + wrapping it from harness scripts instead of duplicating its internals. +- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot + see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login + interactive command path without hardcoding `.zshrc`. + +## Validation Commands + +```bash +scripts/harness/doctor.sh +scripts/harness/security-check.sh +scripts/harness/backend-rules-check.sh +scripts/harness/frontend-rules-check.sh +scripts/harness/docs-consistency-check.sh +scripts/harness/quick-check.sh +scripts/harness/validate.sh +./planet.sh health +``` + +CI-equivalent local checks: + +```bash +cd backend +uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q + +cd frontend +bun install --frozen-lockfile +bun run build +PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs +``` + +The frontend smoke covers public routes, unauthenticated admin guards, +login-error handling, the Earth iframe entry, and authenticated `super_admin` +admin route/section rendering with mocked API data. Authenticated admin checks +run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also +check for accidental global horizontal overflow. A second smoke layer exercises +safe desktop/mobile navigation, admin search, section tab switching, dialog +opening, and non-destructive shortcut links. + +Optional delivery smoke, when Docker and Helm are available: + +```bash +PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh +``` + +## Deeper Docs + +| Topic | Start Here | +| --- | --- | +| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` | +| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` | +| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` | +| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` | +| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` | +| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` | +| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` | +| Documentation rules | `docs/documentation-coverage-rules.md` | +| Harness workflow | `docs/HARNESS.md` | + +## Known Sharp Edges + +- `project_context.md` is static background for agents. It now labels future + stack directions separately, but current code and technical docs still win + when details diverge. +- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the + active local development shape. +- Local `destroy` is intentionally destructive for Planet-owned Docker and build + state. Never run it as a validation shortcut. diff --git a/README.md b/README.md index de98845a..d1775aa1 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ | 组件 | 用途 | |------|------| | React 18 | UI 框架 | -| Ant Design Pro | 管理后台组件 | +| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 | | Axios | HTTP 客户端 | | Socket.io-client | WebSocket 客户端 | | ECharts | 统计图表 | @@ -168,10 +168,12 @@ ## 快速启动 +入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上，`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx，启动 Docker 服务并配置当前用户的访问权限；需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。 + ```bash # 新机器或空项目首次初始化 ./planet.sh init -# 会自动安装/检查 uv、bun，同步 Python/前端依赖 +# 会先准备 Docker / Compose / Buildx，再安装/检查 uv、bun 并同步 Python/前端依赖 # 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local # 会启动 PostgreSQL/Redis，并创建表、默认数据源和本地默认用户 diff --git a/TODO.md b/TODO.md index 0749ee13..191996d9 100644 --- a/TODO.md +++ b/TODO.md @@ -4,20 +4,16 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL ## Earth +- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md). - [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md). - [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior. - [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables. -- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md). -- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries. -- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder. - [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available. - [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data. - [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes. - [ ] Earth high-resolution basemap tiles: implement the viewport-loaded imagery layer described in [Earth High Resolution Basemap Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-resolution-basemap-tiles-plan.md), using high-precision coastline as the alignment reference instead of replacing the globe with one huge texture. -- [ ] Presentation controller ownership: replace the singleton card fallback in [presentation-controller.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/presentation-controller.js) with a presentation/card token check before BGP/News migrate onto the shared controller, so connectors only attach to their owning card. - [ ] BGP frontend maintainability: split [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by responsibility into data loading, marker rendering, overlays, and animation once the current interaction behavior is stable. - [ ] Optional BGP marker experiment: evaluate HTML markers for BGP incident/collector points if WebGL marker density or fixed screen-size clickability becomes a real blocker. -- [ ] Earth news cruise: connect Earth news to the generic cruise queue via a news adapter rather than coupling news-specific sequencing into `main.js`. ## Compute Centers And Location @@ -48,19 +44,15 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL - [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches. - [ ] BGP geography fallback: evaluate `inetnum` / `inet6num` whois as a finer fallback layer after `prefix_geography`, `OpenGeoFeed`, and RIR delegated data. -## Platform - -- [ ] Earth preferences scope: keep current device-local Earth preferences in `localStorage`; only design backend user preferences if account-level synchronization becomes a real product requirement. -- [ ] System logs: finish a usable Planet log viewing flow that covers backend, frontend, AI Provider, and collector/task logs, with filtering and tailing. -- [ ] Console UI modernization: gradually replace Ant Design with Planet-owned components and a consistent Tabler Icons based icon system. -- [ ] Earth live sync: design a unified realtime invalidation path for summary/BGP/satellite updates if polling and current WebSocket channels become insufficient. - ## Archive Archived items stay here so old context is not lost. Completed items remain checked; obsolete, invalid, or superseded items stay unchecked and include the reason. ### Completed +- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache. +- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries. +- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder. - [x] Refined BGP observer and anomaly `hover/click` feel. - [x] Added BGP anomaly relationship display with cables / regions. - [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low. @@ -75,6 +67,11 @@ Archived items stay here so old context is not lost. Completed items remain chec - [x] Added OpenGeoFeed as a high-quality prefix geography override source. - [x] Made RIR delegated data a prefix geography fallback rather than the primary source. - [x] Added route leak and path instability / flap detectors after the activity layer work. +- [x] Console UI modernization. Admin is now the only console, legacy Ant Design / Admin Next code paths and dependencies have been removed, and current console UI uses Planet-owned components. +- [x] Earth news cruise adapter. News cruise now uses `news-cruise-adapter.js` and is wired from `main.js` instead of keeping news-specific sequencing directly in the main Earth loop. +- [x] Presentation controller ownership. `PresentationController` now guards async ownership through active request identity checks, and current callers pass per-request card targets so stale connector/card work cannot overwrite the active presentation. +- [x] Earth live sync. Database writes now flow through `earth_data_change_events`, `earth_db_change_listener`, layer adapters, cache invalidation, and the `earth_updates` WebSocket channel; the Earth frontend debounces updates and refreshes BGP, cables, compute centers, satellites, vessels, news, and interactables by layer. +- [x] System logs. Log sources now normalize into `LogEvent`, Admin supports snapshot filtering plus WebSocket tail/follow, task/detail views deep-link into prefiltered logs, and Admin runtime errors report through the `admin-client` log source. ### Obsolete Or Superseded @@ -85,3 +82,4 @@ Archived items stay here so old context is not lost. Completed items remain chec - [ ] Earth surface material overlay for boundary calibration. Superseded by the high-precision boundary tile plan; future work must use source-faithful boundary/coastline data rather than overlay calibration against the coarse base map. - [ ] Hardcoded Earth news source extraction as a standalone task. Superseded by the broader Earth news source configuration and collector plans. - [ ] Country-level compute-center fallback placement as a standalone task. Superseded by the shared location pipeline and registry/manual-review backlog. +- [ ] Earth preferences backend sync scope. Superseded by the current product decision to keep Earth preferences device-local in `localStorage` until account-level synchronization becomes a real requirement. diff --git a/VERSION b/VERSION index afed694e..a30a1640 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.65.0 +0.74.3 diff --git a/agents.md b/agents.md deleted file mode 100644 index 2331ec16..00000000 --- a/agents.md +++ /dev/null @@ -1,231 +0,0 @@ -# agents.md - -**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。** - ---- - -## Identity - -You are **opencode**, an AI coding assistant specialized in enterprise-level systems. - -You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring: -- Python FastAPI backend -- React Admin dashboard -- Unreal Engine 5 3D visualization -- Multi-source data collection -- Polarized 3D large display (4K, 120Hz) - ---- - -## Communication Style - -### Tone -- **Professional but concise** -- Technical accuracy with clarity -- No unnecessary verbosity -- Use code comments sparingly (explain **why**, not **what**) - -### When Responding -1. **Answer directly** - 1-3 sentences for simple questions -2. **Use code blocks** for all code snippets -3. **Include file:line_number** references when discussing code -4. **Never** start with "I am an AI assistant" or similar phrases -5. **Never** add unnecessary preambles/postambles - -### Examples - -**Good:** -``` -GPU clusters are stored in `backend/app/services/collectors/top500.py:45`. -``` - -**Bad:** -``` -Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this... -``` - ---- - -## Operational Mode - -### Plan Mode (default for complex tasks) -- Analyze requirements -- Propose architecture -- Confirm with user before execution -- **DO NOT** write code until approved - -### Build Mode (after user approval) -- Execute the approved plan -- Write code, run commands -- Verify results -- Report completion concisely - -### Read-Only Mode -- Analyze code -- Explain functionality -- Answer questions -- **DO NOT** modify files - ---- - -## Decision Framework - -### When to Ask Before Acting -- Unclear requirements -- Multiple implementation approaches -- Architecture changes -- Dependency additions -- Anything that could break existing functionality - -### When to Act Directly -- Clear, approved requirements -- Routine tasks (linting, formatting, running tests) -- Following established patterns -- Fixing obvious bugs - -### When to Refuse -- Malicious code requests -- Security violations (secrets, credentials) -- Anything that violates `rules.md` - ---- - -## Working Principles - -### 1. First Understand, Then Act -- Read relevant files before editing -- Understand existing patterns and conventions -- Follow the code style in the codebase -- Match the project's technology choices - -### 2. Incremental Progress -- Break large tasks into smaller PRs -- Complete one feature before starting the next -- Run tests after each significant change -- Commit frequently with clear messages - -### 3. Quality First -- Write tests for new functionality -- Run linters before committing -- Fix warnings, don't ignore them -- Document non-obvious decisions - -### 4. Communication Clarity -- Use precise technical language -- Show relevant code, not explanations -- Report errors with context -- Confirm understanding of requirements - ---- - -## Code Review Checklist - -Before marking a task complete: - -- [ ] Code follows `rules.md` style guidelines -- [ ] Type hints are correct and complete -- [ ] Error handling is proper (no silent failures) -- [ ] Tests pass locally -- [ ] Linting passes -- [ ] No TODO comments left behind -- [ ] Documentation updated if needed -- [ ] Commit message is clear - ---- - -## Common Workflows - -### Feature Development -``` -1. Understand requirements -2. Check existing patterns in codebase -3. Design solution (brief mental model) -4. Write code following rules.md -5. Write/run tests -6. Lint and format -7. Commit with clear message -8. Report completion -``` - -### Bug Fix -``` -1. Reproduce the bug (write failing test) -2. Locate the source -3. Fix the issue -4. Verify test passes -5. Check for regressions -6. Commit fix -``` - -### Refactoring -``` -1. Understand current behavior -2. Design target state -3. Make incremental changes -4. Preserve tests -5. Verify functionality -6. Clean up dead code -``` - ---- - -## Special Considerations - -### WebSocket Services -- Implement heartbeat mechanism (30-second intervals) -- Handle disconnection gracefully -- Include camera position in control frames -- Support both update and full sync modes - -### Data Collectors -- Inherit from BaseCollector -- Implement fetch() and transform() methods -- Support incremental updates -- Handle API changes gracefully - -### UE5 Integration -- Communicate via WebSocket -- Send data frames at configurable intervals (default 5 min) -- Support auto-cruise and manual modes -- Optimize for 4K@120Hz rendering - -### Multi-User Security -- JWT tokens with 15-minute expiration -- Redis token blacklist for logout -- Role-based access control (RBAC) -- Audit logging for all actions - ---- - -## Output Format - -### When Writing Code -```python -# File: backend/app/services/collectors/top500.py -from typing import List, Dict - -class TOP500Collector: - async def fetch(self) -> List[Dict]: - ... -``` - -### When Explaining -- Use concise paragraphs -- Include code references -- No conversational filler - -### When Reporting Progress -- What was done -- What remains -- Any blockers -- Next action - ---- - -## Remember - -1. **Rules are hard constraints** - follow `rules.md` absolutely -2. **Context provides understanding** - use `project_context.md` for background -3. **Role defines behavior** - follow `agents.md` for how to work -4. **Quality over speed** - Enterprise systems require precision -5. **Communicate clearly** - Precision in, precision out diff --git a/aiprovider/Dockerfile b/aiprovider/Dockerfile index 6b6f6f69..91ea40df 100644 --- a/aiprovider/Dockerfile +++ b/aiprovider/Dockerfile @@ -2,10 +2,14 @@ ARG PYTHON_IMAGE=python:3.14-slim ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest +ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown FROM ${UV_IMAGE} AS uv FROM ${PYTHON_IMAGE} +ARG AI_PROVIDER_BUILD_FINGERPRINT +LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}" + COPY --from=uv /uv /uvx /bin/ WORKDIR /app @@ -15,12 +19,15 @@ ENV PYTHONUNBUFFERED=1 ENV UV_COMPILE_BYTECODE=1 ENV UV_LINK_MODE=copy +RUN mkdir -p /root/.config/uv + RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ && rm -rf /var/lib/apt/lists/* COPY pyproject.toml uv.lock /app/ RUN --mount=type=cache,target=/root/.cache/uv \ + --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \ uv sync --frozen --no-dev COPY aiprovider /app/aiprovider diff --git a/backend/Dockerfile b/backend/Dockerfile index 8546fe7a..b98c7a24 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -1,3 +1,5 @@ +# syntax=docker/dockerfile:1.7 + ARG PYTHON_IMAGE=python:3.14-slim ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest @@ -14,12 +16,16 @@ ENV UV_COMPILE_BYTECODE=1 ENV UV_LINK_MODE=copy ENV PYTHONPATH=/app/backend +RUN mkdir -p /root/.config/uv + RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ && rm -rf /var/lib/apt/lists/* COPY pyproject.toml uv.lock /app/ -RUN uv sync --frozen --no-dev +RUN --mount=type=cache,target=/root/.cache/uv \ + --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \ + uv sync --frozen --no-dev COPY backend /app/backend COPY VERSION /app/VERSION diff --git a/backend/app/api/main.py b/backend/app/api/main.py index 9a4b3de8..97ecf971 100644 --- a/backend/app/api/main.py +++ b/backend/app/api/main.py @@ -19,6 +19,7 @@ from app.api.v1 import ( vessels, bgp, news, + interactables, realtime_sources, system_control, tv, @@ -53,4 +54,5 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"]) api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"]) api_router.include_router(tv.router, prefix="/tv", tags=["tv"]) api_router.include_router(news.router, prefix="/news", tags=["news"]) +api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"]) api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"]) diff --git a/backend/app/api/v1/ai.py b/backend/app/api/v1/ai.py index 2900ceb7..d33b4b3f 100644 --- a/backend/app/api/v1/ai.py +++ b/backend/app/api/v1/ai.py @@ -3,6 +3,7 @@ from uuid import uuid4 from fastapi import APIRouter, Depends, HTTPException, Request, Response from sqlalchemy.ext.asyncio import AsyncSession +from app.core.logging import get_logger from app.core.security import get_current_user from app.db.session import get_db from app.models.user import User @@ -47,8 +48,10 @@ from app.services.playground_chat_service import ( stop_message, ) from app.services.situational_alert_ai_brief import build_situational_alert_brief_request +from app.services.business_logs import emit_business_log, exception_context router = APIRouter() +logger = get_logger(__name__, service="api") @router.get("/provider/status", response_model=AIProviderStatusResponse) @@ -122,6 +125,16 @@ async def create_playground_message( provider_client: AIProviderClient = Depends(get_ai_provider_client), db: AsyncSession = Depends(get_db), ): + await emit_business_log( + logger, + event="ai.playground.message.create", + message="Playground message creation requested", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={"session_key": payload.session_key, "preset": payload.selected_preset_key}, + ) return await create_turn( db, user_id=current_user.id, @@ -136,6 +149,16 @@ async def stop_playground_message( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): + await emit_business_log( + logger, + event="ai.playground.message.stop", + message="Playground message stop requested", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={"session_key": payload.session_key, "message_id": payload.message_id}, + ) return await stop_message( db, user_id=current_user.id, @@ -150,6 +173,16 @@ async def resend_playground_message( provider_client: AIProviderClient = Depends(get_ai_provider_client), db: AsyncSession = Depends(get_db), ): + await emit_business_log( + logger, + event="ai.playground.message.resend", + message="Playground message resend requested", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={"session_key": payload.session_key, "user_message_id": payload.user_message_id}, + ) return await resend_turn( db, user_id=current_user.id, @@ -214,16 +247,70 @@ async def analyze_bgp_brief( anomaly_limit=payload.anomaly_limit, collector_limit=payload.collector_limit, ) + await emit_business_log( + logger, + event="ai.brief.bgp.facts_collected", + message="BGP brief facts collected", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={ + "incident_limit": payload.incident_limit, + "anomaly_limit": payload.anomaly_limit, + "collector_limit": payload.collector_limit, + "fact_count": len(facts or []), + }, + ) brief_request.preferred_model = payload.preferred_model brief_request.thinking = payload.thinking - analysis = await provider_client.analyze(brief_request, request_id=request_id) - return save_bgp_brief_record( - analysis, + await emit_business_log( + logger, + event="ai.brief.bgp.start", + message="BGP brief AI analysis started", + category="ai", + service="api", + module=__name__, request_id=request_id, - facts=facts, - context=context, + user_id=current_user.id, + context={"preferred_model": payload.preferred_model}, ) + try: + analysis = await provider_client.analyze(brief_request, request_id=request_id) + record = save_bgp_brief_record( + analysis, + request_id=request_id, + facts=facts, + context=context, + ) + await emit_business_log( + logger, + event="ai.brief.bgp.completed", + message="BGP brief AI analysis saved", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id}, + ) + return record + except Exception as exc: + await emit_business_log( + logger, + event="ai.brief.bgp.failed", + message="BGP brief AI analysis failed", + category="ai", + level="error", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context=exception_context(exc, {"preferred_model": payload.preferred_model}), + ) + raise @router.post("/alerts/brief", response_model=AlertBriefResponse) @@ -242,17 +329,65 @@ async def analyze_alert_brief( db, alert_limit=payload.alert_limit, ) + await emit_business_log( + logger, + event="ai.brief.alerts.facts_collected", + message="Alert brief facts collected", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])}, + ) brief_request.preferred_model = payload.preferred_model brief_request.thinking = payload.thinking - analysis = await provider_client.analyze(brief_request, request_id=request_id) - return AlertBriefResponse( - **analysis.model_dump(), - title=brief_request.title, - objective=brief_request.objective, - facts=facts, - context=context, + await emit_business_log( + logger, + event="ai.brief.alerts.start", + message="Alert brief AI analysis started", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"preferred_model": payload.preferred_model}, ) + try: + analysis = await provider_client.analyze(brief_request, request_id=request_id) + await emit_business_log( + logger, + event="ai.brief.alerts.completed", + message="Alert brief AI analysis completed", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"provider": analysis.provider, "model": analysis.model}, + ) + return AlertBriefResponse( + **analysis.model_dump(), + title=brief_request.title, + objective=brief_request.objective, + facts=facts, + context=context, + ) + except Exception as exc: + await emit_business_log( + logger, + event="ai.brief.alerts.failed", + message="Alert brief AI analysis failed", + category="ai", + level="error", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context=exception_context(exc, {"preferred_model": payload.preferred_model}), + ) + raise @router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse) @@ -268,14 +403,62 @@ async def analyze_situational_alert_brief( response.headers["X-Request-ID"] = request_id brief_request, facts, context = await build_situational_alert_brief_request(db) + await emit_business_log( + logger, + event="ai.brief.situational_alerts.facts_collected", + message="Situational alert brief facts collected", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"fact_count": len(facts or [])}, + ) brief_request.preferred_model = payload.preferred_model brief_request.thinking = payload.thinking - analysis = await provider_client.analyze(brief_request, request_id=request_id) - return SituationalAlertBriefResponse( - **analysis.model_dump(), - title=brief_request.title, - objective=brief_request.objective, - facts=facts, - context=context, + await emit_business_log( + logger, + event="ai.brief.situational_alerts.start", + message="Situational alert brief AI analysis started", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"preferred_model": payload.preferred_model}, ) + try: + analysis = await provider_client.analyze(brief_request, request_id=request_id) + await emit_business_log( + logger, + event="ai.brief.situational_alerts.completed", + message="Situational alert brief AI analysis completed", + category="ai", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context={"provider": analysis.provider, "model": analysis.model}, + ) + return SituationalAlertBriefResponse( + **analysis.model_dump(), + title=brief_request.title, + objective=brief_request.objective, + facts=facts, + context=context, + ) + except Exception as exc: + await emit_business_log( + logger, + event="ai.brief.situational_alerts.failed", + message="Situational alert brief AI analysis failed", + category="ai", + level="error", + service="api", + module=__name__, + request_id=request_id, + user_id=current_user.id, + context=exception_context(exc, {"preferred_model": payload.preferred_model}), + ) + raise diff --git a/backend/app/api/v1/auth.py b/backend/app/api/v1/auth.py index ebd23d8e..e2c5f1fe 100644 --- a/backend/app/api/v1/auth.py +++ b/backend/app/api/v1/auth.py @@ -4,6 +4,7 @@ from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy import text from app.core.config import settings +from app.core.enums import OtpPurpose, UserRole from app.core.logging import get_logger from app.core.security import ( create_access_token, @@ -170,7 +171,7 @@ async def get_me(current_user: User = Depends(get_current_user)): } -async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: str) -> None: +async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: OtpPurpose) -> None: try: await send_verification_email(db, to=email, code=code, purpose=purpose) except EmailNotConfiguredError as exc: @@ -207,7 +208,7 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)): username=payload.username, email=payload.email, password_hash=get_password_hash(payload.password), - role="viewer", + role=UserRole.VIEWER.value, is_active=True, email_verified=False, ) @@ -215,13 +216,13 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)): await db.commit() try: - code = otp.issue_code(payload.email, "register") + code = otp.issue_code(payload.email, OtpPurpose.REGISTER) except otp.OtpResendRateLimited as exc: raise HTTPException( status_code=status.HTTP_429_TOO_MANY_REQUESTS, detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds}, ) from exc - await _send_code_or_raise(db, payload.email, code, "register") + await _send_code_or_raise(db, payload.email, code, OtpPurpose.REGISTER) return {"status": "pending_verification", "email": payload.email} @@ -266,7 +267,7 @@ async def resend_code(payload: ResendCodeRequest, db: AsyncSession = Depends(get if user is None: # Avoid email enumeration; pretend success. return {"status": "ok"} - if payload.purpose == "register" and user.email_verified: + if payload.purpose is OtpPurpose.REGISTER and user.email_verified: raise HTTPException( status_code=status.HTTP_409_CONFLICT, detail={"code": "ALREADY_VERIFIED"}, @@ -289,12 +290,17 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep # Don't leak whether an email is registered. return {"status": "ok"} try: - code = otp.issue_code(payload.email, "reset_password") + code = otp.issue_code(payload.email, OtpPurpose.RESET_PASSWORD) except otp.OtpResendRateLimited: # Silently accept; the user can retry after the cooldown. return {"status": "ok"} try: - await send_verification_email(db, to=payload.email, code=code, purpose="reset_password") + await send_verification_email( + db, + to=payload.email, + code=code, + purpose=OtpPurpose.RESET_PASSWORD, + ) except EmailNotConfiguredError as exc: raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, @@ -304,7 +310,11 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep logger.warning_event( "SMTP send failed", event="auth.email.send_failed", - context={"email": payload.email, "purpose": "reset_password", "error": str(exc)}, + context={ + "email": payload.email, + "purpose": OtpPurpose.RESET_PASSWORD.value, + "error": str(exc), + }, ) return {"status": "ok"} @@ -318,7 +328,7 @@ async def reset_password(payload: ResetPasswordRequest, db: AsyncSession = Depen detail={"code": "OTP_INVALID"}, ) try: - otp.verify_code(payload.email, "reset_password", payload.code) + otp.verify_code(payload.email, OtpPurpose.RESET_PASSWORD, payload.code) except otp.OtpExpired as exc: raise HTTPException( status_code=status.HTTP_410_GONE, diff --git a/backend/app/api/v1/datasource_config.py b/backend/app/api/v1/datasource_config.py index 3f7ce76d..958a868d 100644 --- a/backend/app/api/v1/datasource_config.py +++ b/backend/app/api/v1/datasource_config.py @@ -13,6 +13,7 @@ import httpx from app.core.target_schema_registry import get_target_schema, list_target_schemas from app.core.datasource_defaults import DEFAULT_DATASOURCES +from app.core.enums import AuthType, MappingValidationStatus, UserRole from app.db.session import get_db from app.models.user import User from app.models.datasource_config import DataSourceConfig @@ -34,7 +35,6 @@ from app.services.datasource_mapping import ( ) from app.services.custom_datasource_runtime import ( CustomDatasourceRuntimeError, - fetch_rest_payload, get_custom_stream_status, run_mapped_rest_config, run_mapped_websocket_config, @@ -42,8 +42,6 @@ from app.services.custom_datasource_runtime import ( stop_custom_stream, test_websocket_config, ) - -DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping" from app.services.datasource_connectivity import ( _resolve_aisstream_api_key, _resolve_spacetrack_credentials_with_override, @@ -57,7 +55,8 @@ from app.services.persistent_logs import record_audit_log router = APIRouter() -SECRET_REVEAL_ROLES = {"admin", "super_admin"} +DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping" +SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value} def _user_role_value(user: User) -> str: @@ -124,7 +123,7 @@ class DataSourceConfigCreate(BaseModel): description: Optional[str] = None source_type: str = Field(..., description="rest, websocket, http, api, database") endpoint: str = Field(..., max_length=500) - auth_type: str = Field(default="none", description="none, bearer, api_key, basic") + auth_type: AuthType = Field(default=AuthType.NONE, description="none, bearer, api_key, basic") auth_config: dict = Field(default={}) headers: dict = Field(default={}) config: dict = Field(default={"timeout": 30, "retry": 3}) @@ -135,7 +134,7 @@ class DataSourceConfigUpdate(BaseModel): description: Optional[str] = None source_type: Optional[str] = None endpoint: Optional[str] = Field(None, max_length=500) - auth_type: Optional[str] = None + auth_type: Optional[AuthType] = None auth_config: Optional[dict] = None headers: Optional[dict] = None config: Optional[dict] = None @@ -210,7 +209,7 @@ class MappingTemplateCreate(BaseModel): mapping_json: dict sample_payload: Any | None = None sample_payload_hash: Optional[str] = None - validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$") + validation_status: MappingValidationStatus = MappingValidationStatus.DRAFT is_active: bool = False @@ -219,7 +218,7 @@ class MappingTemplateUpdate(BaseModel): mapping_json: Optional[dict] = None sample_payload: Any | None = None sample_payload_hash: Optional[str] = None - validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$") + validation_status: Optional[MappingValidationStatus] = None is_active: Optional[bool] = None @@ -918,6 +917,7 @@ async def get_datasource_target_schemas( @router.post("/mappings/propose") async def propose_datasource_mapping( payload: MappingProposeRequest, + db: AsyncSession = Depends(get_db), current_user: User = Depends(get_current_user), ai_client: AIProviderClient = Depends(get_ai_provider_client), ): diff --git a/backend/app/api/v1/datasources.py b/backend/app/api/v1/datasources.py index 99094b3a..305612b9 100644 --- a/backend/app/api/v1/datasources.py +++ b/backend/app/api/v1/datasources.py @@ -1,4 +1,3 @@ -import asyncio from datetime import datetime, timedelta, timezone from typing import Optional @@ -7,7 +6,8 @@ from pydantic import BaseModel, Field from sqlalchemy import func, or_, select, text from sqlalchemy.ext.asyncio import AsyncSession -from app.core.cache import cache +from app.core.logging import get_logger +from app.core.enums import JobStatus, SnapshotStatus from app.core.time import to_iso8601_utc from app.core.security import get_current_user from app.core.data_sources import get_data_sources_config @@ -20,18 +20,27 @@ from app.models.datasource_config import DataSourceConfig from app.models.task import CollectionTask from app.models.user import User from app.models.vessel import AISRawObservation -from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA from app.services.scheduler import ( - cancel_running_collector_now, - get_latest_task_id_for_datasource, - run_collector_now, sync_datasource_job, ) -from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source +from app.services.data_jobs import ( + JOB_STATUS_CANCELLING, + JOB_STATUS_QUEUED, + JOB_STATUS_RUNNING, + JOB_TYPE_CLEAR_CACHE, + JOB_TYPE_CLEAR_DATA, + JOB_TYPE_COLLECT, + enqueue_datasource_job, + get_active_datasource_job, + request_cancel_datasource_task, +) +from app.services.business_logs import emit_business_log router = APIRouter() +logger = get_logger(__name__, service="api") STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90 + PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = ( ("vessels", ("vessel", "ais")), ("cables", ("cable", "landing", "telegeography", "arcgis", "fao")), @@ -115,7 +124,7 @@ async def _load_latest_running_tasks( _task_rank_column(CollectionTask.started_at), ) .where(CollectionTask.datasource_id.in_(datasource_ids)) - .where(CollectionTask.status == "running") + .where(CollectionTask.status.in_((JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING))) .subquery() ) result = await db.execute( @@ -126,32 +135,6 @@ async def _load_latest_running_tasks( return {task.datasource_id: task for task in result.scalars().all()} -async def _load_latest_task_ids( - db: AsyncSession, - datasource_ids: list[int], -) -> dict[int, int]: - if not datasource_ids: - return {} - - ranked_tasks = ( - select( - CollectionTask.id.label("task_id"), - CollectionTask.datasource_id.label("datasource_id"), - func.row_number().over( - partition_by=CollectionTask.datasource_id, - order_by=CollectionTask.id.desc(), - ).label("row_num"), - ) - .where(CollectionTask.datasource_id.in_(datasource_ids)) - .subquery() - ) - result = await db.execute( - select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id) - .where(ranked_tasks.c.row_num == 1) - ) - return {datasource_id: task_id for datasource_id, task_id in result.all()} - - async def _load_latest_tasks( db: AsyncSession, datasource_ids: list[int], @@ -182,6 +165,8 @@ async def _load_latest_tasks( async def _load_collected_record_counts( db: AsyncSession, sources: list[str], + *, + exact_vessel_counts: bool = False, ) -> dict[str, int]: if not sources: return {} @@ -202,14 +187,46 @@ async def _load_collected_record_counts( or "ais" in source ] if vessel_sources: - raw_result = await db.execute( - select(AISRawObservation.source, func.count(AISRawObservation.id)) - .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA) - .where(AISRawObservation.source.in_(vessel_sources)) - .group_by(AISRawObservation.source) + if exact_vessel_counts: + exact_result = await db.execute( + select(AISRawObservation.source, func.count(AISRawObservation.id)) + .where(AISRawObservation.source.in_(vessel_sources)) + .group_by(AISRawObservation.source) + ) + for source, count in exact_result.all(): + counts[source] = max(counts.get(source, 0), int(count or 0)) + return counts + + # AIS raw observations can be tens of millions of rows. Use planner + # statistics for the datasource list instead of blocking page load on + # source-level count(*) scans. + stats_result = await db.execute( + text( + """ + SELECT + COALESCE(pg_class.reltuples, 0)::bigint AS total_rows, + pg_stats.most_common_vals::text AS source_values, + pg_stats.most_common_freqs::text AS source_freqs + FROM pg_class + LEFT JOIN pg_stats + ON pg_stats.schemaname = 'public' + AND pg_stats.tablename = 'ais_raw_observations' + AND pg_stats.attname = 'source' + WHERE pg_class.relname = 'ais_raw_observations' + LIMIT 1 + """ + ) ) - for source, count in raw_result.all(): - counts[source] = max(counts.get(source, 0), int(count or 0)) + stats = stats_result.mappings().first() + if stats: + total_rows = int(stats["total_rows"] or 0) + values = str(stats["source_values"] or "").strip("{}") + freqs = str(stats["source_freqs"] or "").strip("{}") + source_values = [value.strip('"') for value in values.split(",") if value] + source_freqs = [float(value) for value in freqs.split(",") if value] + for source, freq in zip(source_values, source_freqs): + if source in vessel_sources: + counts[source] = max(counts.get(source, 0), int(round(total_rows * freq))) return counts @@ -303,8 +320,11 @@ def serialize_datasource_row( "last_run": to_iso8601_utc(last_run_at), "last_run_at": to_iso8601_utc(last_run_at), "last_status": last_status, - "is_running": running_task is not None, + "is_running": running_task is not None and running_task.task_type == JOB_TYPE_COLLECT, + "is_task_active": running_task is not None, + "task_status": running_task.status if running_task else None, "task_id": display_task.id if display_task else None, + "task_type": display_task.task_type if display_task else None, "progress": display_task.progress if display_task else None, "phase": display_task.phase if display_task else None, "phase_progress": display_task.phase_progress if display_task else None, @@ -400,9 +420,25 @@ async def _trigger_datasource_batch( datasources: list[DataSource], *, force: bool, + actor_id: int | None = None, + trigger_kind: str = "batch", ) -> dict: + await emit_business_log( + logger, + event=f"collector.trigger.{trigger_kind}.start", + message="Datasource batch trigger started", + category="collector", + service="api", + module=__name__, + user_id=actor_id, + context={ + "trigger_kind": trigger_kind, + "force": force, + "requested_count": len(datasources), + }, + ) if not datasources: - return { + result = { "status": "noop", "message": "No matching data sources to trigger", "force": force, @@ -410,8 +446,18 @@ async def _trigger_datasource_batch( "skipped": [], "failed": [], } + await emit_business_log( + logger, + event=f"collector.trigger.{trigger_kind}.completed", + message="Datasource batch trigger completed with no matching sources", + category="collector", + service="api", + module=__name__, + user_id=actor_id, + context={"trigger_kind": trigger_kind, "force": force, "status": "noop", "triggered_count": 0}, + ) + return result - previous_task_ids: dict[int, Optional[int]] = {} triggered_sources: list[dict] = [] skipped_sources: list[dict] = [] failed_sources: list[dict] = [] @@ -446,9 +492,11 @@ async def _trigger_datasource_batch( } ) continue - cancelled = await cancel_running_collector_now(datasource.source) - if not cancelled: - await rollback_orphaned_running_task(db, datasource, running_task) + await request_cancel_datasource_task( + db, + running_task, + reason="superseded_by_forced_collection", + ) if not force and not is_due_for_collection(datasource, now): skipped_sources.append( @@ -465,57 +513,51 @@ async def _trigger_datasource_batch( ) continue - previous_task_ids[datasource.id] = None - success = run_collector_now(datasource.source) - if not success: - failed_sources.append( - { - "id": datasource.id, - "source": datasource.source, - "name": datasource.name, - "reason": "trigger_failed", - } - ) - continue + task = await enqueue_datasource_job( + db, + datasource, + JOB_TYPE_COLLECT, + payload={"force": force, "trigger": "batch"}, + ) triggered_sources.append( { "id": datasource.id, "source": datasource.source, "name": datasource.name, - "task_id": None, + "task_id": task.id, } ) - latest_task_ids = await _load_latest_task_ids( - db, - [datasource.id for datasource in datasources], - ) - for datasource_id in previous_task_ids: - previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id) - - for _ in range(20): - await asyncio.sleep(0.1) - pending = [item for item in triggered_sources if item["task_id"] is None] - if not pending: - break - latest_task_ids = await _load_latest_task_ids( - db, - [item["id"] for item in pending], - ) - for item in pending: - task_id = latest_task_ids.get(item["id"]) - if task_id is not None and task_id != previous_task_ids.get(item["id"]): - item["task_id"] = task_id - - return { - "status": "triggered" if triggered_sources else "partial", - "message": f"Triggered {len(triggered_sources)} data sources", + result = { + "status": "queued" if triggered_sources else "partial", + "message": f"Queued {len(triggered_sources)} data source jobs", "force": force, "triggered": triggered_sources, "skipped": skipped_sources, "failed": failed_sources, } + await emit_business_log( + logger, + event=f"collector.trigger.{trigger_kind}.completed", + message="Datasource batch trigger completed", + category="collector", + service="api", + module=__name__, + user_id=actor_id, + context={ + "trigger_kind": trigger_kind, + "force": force, + "status": result["status"], + "requested_count": len(datasources), + "triggered_count": len(triggered_sources), + "skipped_count": len(skipped_sources), + "failed_count": len(failed_sources), + "triggered_sources": [item["source"] for item in triggered_sources], + "skipped_reasons": [item["reason"] for item in skipped_sources], + }, + ) + return result async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]: @@ -540,7 +582,7 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col result = await db.execute( select(CollectionTask) .where(CollectionTask.datasource_id == datasource_id) - .where(CollectionTask.status == "running") + .where(CollectionTask.status == JobStatus.RUNNING.value) .order_by(CollectionTask.started_at.desc()) .limit(1) ) @@ -568,8 +610,8 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col f"Marked failed automatically after stale running timeout " f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)" ) - task.status = "failed" - task.phase = "failed" + task.status = JobStatus.FAILED.value + task.phase = JobStatus.FAILED.value task.completed_at = now task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason await db.commit() @@ -606,7 +648,7 @@ async def rollback_orphaned_running_task( ) if snapshot is not None: - snapshot.status = "cancelled" + snapshot.status = SnapshotStatus.CANCELLED.value snapshot.is_current = False snapshot.completed_at = datetime.now(timezone.utc) summary = dict(snapshot.summary or {}) @@ -629,13 +671,13 @@ async def rollback_orphaned_running_task( {"snapshot_id": snapshot.parent_snapshot_id}, ) - running_task.status = "cancelled" - running_task.phase = "cancelled" + running_task.status = JobStatus.CANCELLED.value + running_task.phase = JobStatus.CANCELLED.value running_task.completed_at = datetime.now(timezone.utc) existing_error = (running_task.error_message or "").strip() cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back" running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason - datasource.last_status = "cancelled" + datasource.last_status = JobStatus.CANCELLED.value datasource.last_run_at = datetime.now(timezone.utc) await db.commit() @@ -670,7 +712,7 @@ async def fail_and_rollback_stale_running_task( ) if snapshot is not None: - snapshot.status = "failed" + snapshot.status = SnapshotStatus.FAILED.value snapshot.is_current = False snapshot.completed_at = datetime.now(timezone.utc) summary = dict(snapshot.summary or {}) @@ -698,11 +740,11 @@ async def fail_and_rollback_stale_running_task( f"Marked failed automatically after stale running timeout " f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back" ) - running_task.status = "failed" - running_task.phase = "failed" + running_task.status = JobStatus.FAILED.value + running_task.phase = JobStatus.FAILED.value running_task.completed_at = datetime.now(timezone.utc) running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason - datasource.last_status = "failed" + datasource.last_status = JobStatus.FAILED.value datasource.last_run_at = datetime.now(timezone.utc) await db.commit() @@ -780,7 +822,13 @@ async def trigger_all_datasources( .order_by(DataSource.module, DataSource.id) ) datasources = result.scalars().all() - return await _trigger_datasource_batch(db, datasources, force=force) + return await _trigger_datasource_batch( + db, + datasources, + force=force, + actor_id=current_user.id, + trigger_kind="all", + ) @router.post("/trigger-batch") @@ -816,7 +864,13 @@ async def trigger_datasource_batch( collected=None if payload.source_ids else payload.collected, credential_status=None if payload.source_ids else payload.credential_status, ) - return await _trigger_datasource_batch(db, datasources, force=payload.force) + return await _trigger_datasource_batch( + db, + datasources, + force=payload.force, + actor_id=current_user.id, + trigger_kind="batch", + ) @router.get("/snapshots") @@ -909,7 +963,7 @@ async def get_datasource_row( [datasource], include_endpoint=include_endpoint, ) - record_counts = await _load_collected_record_counts(db, [datasource.source]) + record_counts = await _load_collected_record_counts(db, [datasource.source], exact_vessel_counts=True) return { "data": serialize_datasource_row( datasource, @@ -992,8 +1046,24 @@ async def trigger_datasource( if not datasource.is_active: raise HTTPException(status_code=400, detail="Data source is disabled") - running_task = await get_running_task(db, datasource.id) + running_task = await get_active_datasource_job(db, datasource.id, task_types=(JOB_TYPE_COLLECT,)) if running_task is not None and not force: + await emit_business_log( + logger, + event="collector.trigger.single.skipped_already_running", + message="Datasource trigger skipped because a task is already running", + category="collector", + level="warning", + service="api", + module=__name__, + user_id=current_user.id, + context={ + "collector_name": datasource.source, + "datasource_id": datasource.id, + "task_id": running_task.id, + "status": "skipped", + }, + ) raise HTTPException( status_code=409, detail={ @@ -1013,31 +1083,42 @@ async def trigger_datasource( ) if running_task is not None and force: - cancelled = await cancel_running_collector_now(datasource.source) - if not cancelled: - await rollback_orphaned_running_task(db, datasource, running_task) + await request_cancel_datasource_task( + db, + running_task, + reason="superseded_by_forced_collection", + ) - previous_task_id = await get_latest_task_id_for_datasource(datasource.id) - success = run_collector_now(datasource.source) - if not success: - raise HTTPException(status_code=500, detail=f"Failed to trigger collector '{datasource.source}'") - - task_id = None - for _ in range(20): - await asyncio.sleep(0.1) - task_id = await get_latest_task_id_for_datasource(datasource.id) - if task_id is not None and task_id != previous_task_id: - break - if task_id == previous_task_id: - task_id = None + task = await enqueue_datasource_job( + db, + datasource, + JOB_TYPE_COLLECT, + payload={"force": force, "trigger": "single"}, + ) + await emit_business_log( + logger, + event="collector.trigger.single.completed", + message="Datasource trigger queued", + category="collector", + service="api", + module=__name__, + user_id=current_user.id, + context={ + "collector_name": datasource.source, + "datasource_id": datasource.id, + "task_id": task.id, + "force": force, + "status": "queued", + }, + ) return { - "status": "triggered", + "status": "queued", "source_id": datasource.id, - "task_id": task_id, + "task_id": task.id, "collector_name": datasource.source, "force": force, - "message": f"Collector '{datasource.source}' has been triggered", + "message": f"Collector '{datasource.source}' has been queued", } @@ -1051,22 +1132,30 @@ async def clear_datasource_data( if not datasource: raise HTTPException(status_code=404, detail="Data source not found") - result = await db.execute( - select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source) + active_task = await get_active_datasource_job(db, datasource.id) + if active_task is not None: + raise HTTPException( + status_code=409, + detail={ + "reason": "datasource_job_in_progress", + "message": "当前数据源已有任务在执行，请等待完成或先取消任务。", + "task_id": active_task.id, + "task_type": active_task.task_type, + "status": active_task.status, + }, + ) + task = await enqueue_datasource_job( + db, + datasource, + JOB_TYPE_CLEAR_DATA, + payload={"source": datasource.source}, ) - count = result.scalar() or 0 - - if count == 0: - return {"status": "success", "message": "No data to clear", "deleted_count": 0} - - delete_query = CollectedData.__table__.delete().where(CollectedData.source == datasource.source) - await db.execute(delete_query) - await db.commit() return { - "status": "success", - "message": f"Cleared {count} records for data source '{datasource.name}'", - "deleted_count": count, + "status": "queued", + "message": f"Queued data clearing for data source '{datasource.name}'", + "task_id": task.id, + "deleted_count": None, } @@ -1080,16 +1169,44 @@ async def clear_datasource_cache( if not datasource: raise HTTPException(status_code=404, detail="Data source not found") - earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source) - dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary")) - deleted_count = earth_deleted_count + dashboard_deleted_count + task = await enqueue_datasource_job( + db, + datasource, + JOB_TYPE_CLEAR_CACHE, + payload={"source": datasource.source}, + dedupe_key=f"clear_cache:{datasource.source}", + ) return { - "status": "success", - "message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'", - "deleted_count": deleted_count, - "earth_layer_deleted_count": earth_deleted_count, - "dashboard_deleted_count": dashboard_deleted_count, + "status": "queued", + "message": f"Queued cache clearing for data source '{datasource.name}'", + "task_id": task.id, + "deleted_count": None, + } + + +@router.post("/{source_id}/tasks/{task_id}/cancel") +async def cancel_datasource_task( + source_id: str, + task_id: int, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + datasource = await get_datasource_record(db, source_id) + if not datasource: + raise HTTPException(status_code=404, detail="Data source not found") + + task = await db.get(CollectionTask, task_id) + if not task or task.datasource_id != datasource.id: + raise HTTPException(status_code=404, detail="Task not found") + + task = await request_cancel_datasource_task(db, task) + return { + "status": "cancelled" if task.completed_at else "cancelling", + "task_id": task.id, + "task_type": task.task_type, + "phase": task.phase, + "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at), } @@ -1109,7 +1226,7 @@ async def get_task_status( if not task or task.datasource_id != datasource.id: raise HTTPException(status_code=404, detail="Task not found") else: - task = await get_running_task(db, datasource.id) + task = await get_active_datasource_job(db, datasource.id) if task is None: result = await db.execute( select(CollectionTask) @@ -1134,8 +1251,12 @@ async def get_task_status( } return { - "is_running": task.status == "running", + "is_running": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING} + and task.task_type == JOB_TYPE_COLLECT, + "is_task_active": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING}, "task_id": task.id, + "task_type": task.task_type, + "task_status": task.status, "progress": task.progress, "phase": task.phase, "phase_progress": task.phase_progress, @@ -1146,5 +1267,6 @@ async def get_task_status( "records_processed": task.records_processed, "total_records": task.total_records, "status": task.status, + "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at), "error_message": task.error_message, } diff --git a/backend/app/api/v1/earth.py b/backend/app/api/v1/earth.py index fb8571c7..16474964 100644 --- a/backend/app/api/v1/earth.py +++ b/backend/app/api/v1/earth.py @@ -6,12 +6,13 @@ from pathlib import Path from typing import Any from uuid import uuid4 -from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status +from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer from pydantic import BaseModel, Field from sqlalchemy import delete, func, select, text from sqlalchemy.ext.asyncio import AsyncSession +from app.core.config import settings as app_settings from app.core.security import decode_token, get_current_user, redis_client from app.db.session import get_db from app.models.collected_data import CollectedData @@ -20,6 +21,26 @@ from app.models.datasource_config import DataSourceConfig from app.models.system_setting import SystemSetting from app.models.user import User from app.services.tv_streams import get_tv_settings_payload +from app.services.earth_news import ( + get_earth_news_sources_payload, + reset_earth_news_sources_payload, + save_earth_news_sources_payload, + test_news_source_config, +) +from app.services.earth_news_manual import ( + broadcast_manual_news_changed, + create_manual_news_group, + delete_manual_news_item, + get_news_record_or_404, + import_manual_news_items, + list_news_groups, + list_news_records, + parse_manual_news_import_upload, + rename_manual_news_group, + reprocess_manual_news_item, + serialize_news_record, + upsert_manual_news_item, +) from app.services.earth_boundaries import ( EarthBoundaryBuildError, get_boundary_build_status, @@ -36,9 +57,16 @@ EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand" EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets" EARTH_BRAND_CATEGORY = "earth_brand" EARTH_ABOUT_CATEGORY = "earth_about" +SYSTEM_SETTINGS_CATEGORY = "system" MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024 ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"} + +def _app_version_label() -> str: + version = str(app_settings.VERSION or "").strip() or "0.0.0" + return version if version.startswith("v") else f"v{version}" + + DEFAULT_EARTH_BRAND = { "logo_src": "/earth/assets/brand/earth-logo.png", "title_src": "/earth/assets/brand/title-zh.png", @@ -53,14 +81,15 @@ DEFAULT_EARTH_ABOUT = { "logo_src": "/earth/assets/brand/lim-logo.png", "kicker": "About", "title": "智能星球计划", - "version": "v0.64.0", + "version": _app_version_label(), "description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航，提供可视化观测、事件聚合与交互式探索能力。", "meta": [ {"label": "出品方", "value": "浙江大学临空智能媒体研究院"}, - {"label": "策划人", "value": "黄柳青"}, + {"label": "策划人", "value": "方兴东、黄柳青"}, {"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"}, ], } +EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青" class EarthBoundaryConfigPayload(BaseModel): @@ -91,6 +120,39 @@ class EarthAboutPayload(BaseModel): meta: list[EarthAboutMetaItem] = Field(default_factory=list) +class EarthNewsSourcesPayload(BaseModel): + cache_version: int | None = None + source_tags: list[dict[str, Any]] = Field(default_factory=list) + categories: list[dict[str, Any]] = Field(default_factory=list) + item_tag_rules: list[dict[str, Any]] = Field(default_factory=list) + sources: list[dict[str, Any]] = Field(default_factory=list) + health: dict[str, Any] = Field(default_factory=dict) + + +class EarthNewsSourceTestPayload(BaseModel): + source: dict[str, Any] = Field(default_factory=dict) + + +class EarthNewsManualItemPayload(BaseModel): + title: str = Field(default="", max_length=500) + summary: str = Field(default="", max_length=1200) + content: str = Field(default="", max_length=12000) + url: str = Field(default="", max_length=2000) + source: str = Field(default="", max_length=255) + region: str = Field(default="global", max_length=80) + published_at: str | None = None + category: str = Field(default="other", max_length=80) + tags: list[str] = Field(default_factory=list) + location: dict[str, Any] | None = None + homepage_url: str = Field(default="", max_length=2000) + content_language: str = Field(default="", max_length=32) + group_id: str | None = Field(default=None, max_length=120) + + +class EarthNewsManualGroupPayload(BaseModel): + name: str = Field(default="", max_length=120) + + def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]: merged = DEFAULT_EARTH_BRAND.copy() if payload: @@ -116,11 +178,12 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, } raw_meta = DEFAULT_EARTH_ABOUT["meta"] if payload: - for key in ("logo_src", "kicker", "title", "version", "description"): + for key in ("logo_src", "kicker", "title", "description"): value = payload.get(key) if value is not None: merged[key] = str(value).strip() raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta + merged["version"] = _app_version_label() for key, default_value in DEFAULT_EARTH_ABOUT.items(): if key == "meta": @@ -134,6 +197,8 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, continue label = str(item.get("label") or "").strip() value = str(item.get("value") or "").strip() + if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE: + value = "方兴东、黄柳青" if label or value: normalized_meta.append({"label": label, "value": value}) if not normalized_meta: @@ -142,6 +207,10 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, return merged +def _is_demo_mode_enabled(payload: Any) -> bool: + return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False + + async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None: result = await db.execute( select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY) @@ -308,6 +377,194 @@ async def reset_earth_about( return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True} +@router.get("/news-sources") +async def get_earth_news_sources(db: AsyncSession = Depends(get_db)): + return await get_earth_news_sources_payload(db) + + +@router.put("/news-sources") +async def update_earth_news_sources( + payload: EarthNewsSourcesPayload, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + return await save_earth_news_sources_payload(db, payload.model_dump()) + + +@router.delete("/news-sources") +@router.post("/news-sources/reset") +async def reset_earth_news_sources( + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + return await reset_earth_news_sources_payload(db) + + +@router.post("/news-sources/test") +async def test_earth_news_source( + payload: EarthNewsSourceTestPayload, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + return await test_news_source_config(payload.source, db=db) + + +@router.get("/news-groups") +async def list_earth_news_groups_admin( + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + return await list_news_groups(db) + + +@router.post("/news-groups") +async def create_earth_news_group_admin( + payload: EarthNewsManualGroupPayload, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + try: + group = await create_manual_news_group(db, payload.name) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + await db.commit() + return {"status": "ok", "group": group} + + +@router.put("/news-groups/{group_id:path}") +async def rename_earth_news_group_admin( + group_id: str, + payload: EarthNewsManualGroupPayload, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + try: + group = await rename_manual_news_group(db, group_id, payload.name) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + await db.commit() + await broadcast_manual_news_changed() + return {"status": "ok", "group": group} + + +@router.get("/news-items") +async def list_earth_news_items_admin( + page: int = Query(1, ge=1), + page_size: int = Query(50, ge=1, le=100), + source_type: str | None = Query(None), + region: str | None = Query(None), + category: str | None = Query(None), + status_filter: str | None = Query(None, alias="status"), + group_id: str | None = Query(None), + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + return await list_news_records( + db, + page=page, + page_size=page_size, + source_type=source_type, + region=region, + category=category, + status_filter=status_filter, + group_id=group_id, + ) + + +@router.post("/news-items") +async def create_earth_news_item_admin( + payload: EarthNewsManualItemPayload, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + try: + result = await upsert_manual_news_item(db, payload.model_dump()) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except PermissionError as exc: + raise HTTPException(status_code=403, detail=str(exc)) from exc + await db.commit() + await broadcast_manual_news_changed() + return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)} + + +@router.post("/news-items/import") +async def import_earth_news_items_admin( + file: UploadFile = File(...), + group_id: str | None = Form(default=None), + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + try: + payload = await parse_manual_news_import_upload(await file.read()) + result = await import_manual_news_items(db, payload, group_id=group_id) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + await db.commit() + await broadcast_manual_news_changed() + return {"status": "ok", **result} + + +@router.put("/news-items/{item_id:path}") +async def update_earth_news_item_admin( + item_id: str, + payload: EarthNewsManualItemPayload, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + existing = await get_news_record_or_404(db, item_id) + if existing is None: + raise HTTPException(status_code=404, detail="News item not found.") + try: + result = await upsert_manual_news_item( + db, + payload.model_dump(), + item_id_override=item_id, + ) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except PermissionError as exc: + raise HTTPException(status_code=403, detail=str(exc)) from exc + await db.commit() + await broadcast_manual_news_changed() + return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)} + + +@router.delete("/news-items/{item_id:path}") +async def delete_earth_news_item_admin( + item_id: str, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + try: + deleted = await delete_manual_news_item(db, item_id) + except PermissionError as exc: + raise HTTPException(status_code=403, detail=str(exc)) from exc + if not deleted: + raise HTTPException(status_code=404, detail="News item not found.") + await db.commit() + await broadcast_manual_news_changed() + return {"status": "deleted", "id": item_id} + + +@router.post("/news-items/{item_id:path}/reprocess") +async def reprocess_earth_news_item_admin( + item_id: str, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + existing = await get_news_record_or_404(db, item_id) + if existing is None: + raise HTTPException(status_code=404, detail="News item not found.") + try: + queued = await reprocess_manual_news_item(db, item_id) + except PermissionError as exc: + raise HTTPException(status_code=403, detail=str(exc)) from exc + await db.commit() + await broadcast_manual_news_changed() + return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id} + + @router.get("/oobe-status") async def get_earth_oobe_status( current_user: User | None = Depends(_get_optional_current_user), @@ -317,6 +574,11 @@ async def get_earth_oobe_status( select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True)) ) current_record_count = int(current_count_result.scalar() or 0) + system_result = await db.execute( + select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY) + ) + system_record = system_result.scalar_one_or_none() + demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None) datasource_count_result = await db.execute(select(func.count(DataSource.id))) datasource_count = int(datasource_count_result.scalar() or 0) @@ -337,6 +599,8 @@ async def get_earth_oobe_status( ready = has_collected_data suggestions: list[str] = [] + if demo_mode: + suggestions.append("演示模式已开启") if not current_user: suggestions.append("登录控制台") if not has_collected_data: @@ -348,8 +612,9 @@ async def get_earth_oobe_status( return { "ready": ready, + "demo_mode": demo_mode, "authenticated": current_user is not None, - "needs_login": current_user is None and not ready, + "needs_login": current_user is None and not ready and not demo_mode, "has_collected_data": has_collected_data, "has_tv_sources": tv_source_count > 0, "has_core_layers": has_core_layers, diff --git a/backend/app/api/v1/interactables.py b/backend/app/api/v1/interactables.py new file mode 100644 index 00000000..055ff9a7 --- /dev/null +++ b/backend/app/api/v1/interactables.py @@ -0,0 +1,190 @@ +"""CRUD APIs for persistent Earth interactables.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +from fastapi import APIRouter, Depends, HTTPException, Query, Response, status +from pydantic import BaseModel, Field, field_validator +from sqlalchemy.ext.asyncio import AsyncSession + +from app.core.security import get_current_user +from app.db.session import get_db +from app.models.earth_interactable import EarthInteractable +from app.models.user import User +from app.services.earth_interactables import ( + build_interactable_event, + interactables_to_geojson, + invalidate_interactable_cache, + list_interactables, + normalize_interactable_id, + publish_interactable_event, + serialize_interactable, +) +from app.services.earth_layer_cache import ( + EarthLayerCachePolicy, + earth_layer_cache, + get_or_build_layer_payload, +) + +router = APIRouter() +INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy( + fresh_ttl_seconds=60, + stale_ttl_seconds=10 * 60, + max_features=5000, +) + + +class InteractableCreate(BaseModel): + id: str | None = Field(default=None, max_length=160) + layer: str = Field(default="default", min_length=1, max_length=80) + kind: str = Field(default="default", min_length=1, max_length=80) + label: str = Field(default="", max_length=255) + description: str = Field(default="", max_length=4000) + latitude: float = Field(ge=-90, le=90) + longitude: float = Field(ge=-180, le=180) + altitude: float | None = None + properties: dict[str, Any] = Field(default_factory=dict) + + @field_validator("layer", "kind") + @classmethod + def normalize_key(cls, value: str) -> str: + normalized = str(value or "").strip() + if not normalized: + raise ValueError("must not be empty") + return normalized + + +class InteractableUpdate(BaseModel): + layer: str | None = Field(default=None, min_length=1, max_length=80) + kind: str | None = Field(default=None, min_length=1, max_length=80) + label: str | None = Field(default=None, max_length=255) + description: str | None = Field(default=None, max_length=4000) + latitude: float | None = Field(default=None, ge=-90, le=90) + longitude: float | None = Field(default=None, ge=-180, le=180) + altitude: float | None = None + properties: dict[str, Any] | None = None + + +@router.get("") +async def get_interactables( + response: Response, + layer: str | None = Query(default=None), + include_deleted: bool = Query(default=False), + db: AsyncSession = Depends(get_db), +): + items = await list_interactables(db, layer=layer, include_deleted=include_deleted) + response.headers["X-Planet-Interactables-Count"] = str(len(items)) + return {"items": [serialize_interactable(item) for item in items]} + + +@router.get("/geojson") +async def get_interactables_geojson( + response: Response, + layer: str | None = Query(default=None), + db: AsyncSession = Depends(get_db), +): + async def build_payload() -> dict[str, Any]: + items = await list_interactables(db, layer=layer) + return interactables_to_geojson(items) + + payload = await get_or_build_layer_payload( + key=earth_layer_cache.key("interactables", interactable_layer=layer or "all"), + policy=INTERACTABLE_CACHE_POLICY, + builder=build_payload, + response=response, + ) + response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or [])) + return payload + + +@router.post("", status_code=status.HTTP_201_CREATED) +async def create_interactable( + payload: InteractableCreate, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + record_id = normalize_interactable_id(payload.id) + existing = await db.get(EarthInteractable, record_id) + if existing and not existing.is_deleted: + raise HTTPException(status_code=409, detail="Interactable already exists") + + if existing is None: + record = EarthInteractable(id=record_id) + db.add(record) + else: + record = existing + record.is_deleted = False + record.deleted_at = None + record.revision += 1 + + record.layer = payload.layer + record.kind = payload.kind + record.label = payload.label + record.description = payload.description + record.latitude = payload.latitude + record.longitude = payload.longitude + record.altitude = payload.altitude + record.properties = payload.properties + + await db.commit() + await db.refresh(record) + invalidate_interactable_cache(record.layer) + await publish_interactable_event("created", record) + return {"item": serialize_interactable(record)} + + +@router.get("/{interactable_id}") +async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)): + record = await db.get(EarthInteractable, interactable_id) + if record is None or record.is_deleted: + raise HTTPException(status_code=404, detail="Interactable not found") + return {"item": serialize_interactable(record)} + + +@router.patch("/{interactable_id}") +async def update_interactable( + interactable_id: str, + payload: InteractableUpdate, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + record = await db.get(EarthInteractable, interactable_id) + if record is None or record.is_deleted: + raise HTTPException(status_code=404, detail="Interactable not found") + + previous_layer = record.layer + patch = payload.model_dump(exclude_unset=True) + for key, value in patch.items(): + setattr(record, key, value) + record.revision += 1 + + await db.commit() + await db.refresh(record) + invalidate_interactable_cache(previous_layer) + if record.layer != previous_layer: + invalidate_interactable_cache(record.layer) + await publish_interactable_event("updated", record) + return {"item": serialize_interactable(record)} + + +@router.delete("/{interactable_id}") +async def delete_interactable( + interactable_id: str, + _current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + record = await db.get(EarthInteractable, interactable_id) + if record is None or record.is_deleted: + raise HTTPException(status_code=404, detail="Interactable not found") + + record.is_deleted = True + record.deleted_at = datetime.now(UTC) + record.revision += 1 + await db.commit() + await db.refresh(record) + invalidate_interactable_cache(record.layer) + await publish_interactable_event("deleted", record) + event = build_interactable_event(action="deleted", record=record, include_item=True) + return {"deleted": True, "event": event} diff --git a/backend/app/api/v1/layers.py b/backend/app/api/v1/layers.py index 991ff5a2..57891d85 100644 --- a/backend/app/api/v1/layers.py +++ b/backend/app/api/v1/layers.py @@ -105,7 +105,7 @@ def _parse_layer_bbox(bbox: str) -> tuple[float, float, float, float]: @router.get("/vessels/snapshot") async def get_vessel_layer_snapshot( bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"), - zoom: int = Query(..., ge=1, le=20), + zoom: float = Query(..., ge=1, le=20), limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1), vessel_type: Optional[str] = Query(None, alias="type"), since_minutes: int = Query(60, ge=1, le=1440), diff --git a/backend/app/api/v1/news.py b/backend/app/api/v1/news.py index 12cc7d0b..47389e63 100644 --- a/backend/app/api/v1/news.py +++ b/backend/app/api/v1/news.py @@ -1,16 +1,92 @@ -from fastapi import APIRouter, Depends, Query +from fastapi import APIRouter, Depends, HTTPException, Query from sqlalchemy.ext.asyncio import AsyncSession from app.db.session import get_db -from app.services.earth_news import get_earth_news_payload +from app.services.earth_news import ( + ALLOWED_NEWS_CATEGORY_KEYS, + SUPPORTED_NEWS_LOCALES, + REGION_ANCHORS, + get_earth_news_payload, +) router = APIRouter() +def _parse_categories(raw: str | None) -> set[str] | None: + if raw is None or not raw.strip(): + return None + requested = {item.strip().lower() for item in raw.split(",") if item.strip()} + invalid = sorted(requested - set(ALLOWED_NEWS_CATEGORY_KEYS)) + if invalid: + raise HTTPException( + status_code=422, + detail={ + "message": "Unsupported news categories.", + "invalid_categories": invalid, + "allowed_categories": list(ALLOWED_NEWS_CATEGORY_KEYS), + }, + ) + return requested or None + + +def _parse_source_ids(raw: str | None) -> set[str] | None: + if raw is None or not raw.strip(): + return None + return {item.strip() for item in raw.split(",") if item.strip()} or None + + +def _parse_limit(raw: int | None) -> int: + if raw is None: + return 12 + if raw < 1: + raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."}) + return min(raw, 100) + + +def _parse_locale(raw: str | None) -> str: + if raw is None or not raw.strip(): + return "zh-CN" + requested = raw.strip() + if requested not in SUPPORTED_NEWS_LOCALES: + raise HTTPException( + status_code=422, + detail={ + "message": "Unsupported news locale.", + "invalid_locale": requested, + "allowed_locales": sorted(SUPPORTED_NEWS_LOCALES), + }, + ) + return requested + + @router.get("/earth-feed") async def get_earth_feed( lat: float | None = Query(None, description="Current Earth view center latitude"), lon: float | None = Query(None, description="Current Earth view center longitude"), + region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"), + categories: str | None = Query(None, description="Comma-separated news category keys"), + sources: str | None = Query(None, description="Comma-separated news source ids"), + limit: int | None = Query(None, description="Maximum news items to return, capped at 100"), + locale: str | None = Query(None, description="Display locale, zh-CN or en-US"), db: AsyncSession = Depends(get_db), ): - return await get_earth_news_payload(lat=lat, lon=lon, db=db) + normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None + if normalized_region is not None and normalized_region not in REGION_ANCHORS: + raise HTTPException( + status_code=422, + detail={ + "message": "Unsupported news region.", + "invalid_region": normalized_region, + "allowed_regions": list(REGION_ANCHORS.keys()), + }, + ) + return await get_earth_news_payload( + lat=lat, + lon=lon, + region=normalized_region, + categories=_parse_categories(categories), + source_ids=_parse_source_ids(sources), + limit=_parse_limit(limit), + locale=_parse_locale(locale), + db=db, + ) diff --git a/backend/app/api/v1/settings.py b/backend/app/api/v1/settings.py index 5a8cf3f6..c1dd55aa 100644 --- a/backend/app/api/v1/settings.py +++ b/backend/app/api/v1/settings.py @@ -11,6 +11,8 @@ from dotenv import dotenv_values from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession +from app.core.logging import get_logger +from app.core.enums import ProviderApi, TVSourceType, UserRole from app.core.security import get_current_user from app.core.time import to_iso8601_utc from app.core.config import settings as app_settings @@ -66,11 +68,13 @@ from app.services.llm_provider_catalog import ( from app.services.scheduler import sync_datasource_job from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings from app.services.persistent_logs import record_audit_log +from app.services.business_logs import emit_business_log, exception_context router = APIRouter() +logger = get_logger(__name__, service="api") AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5 AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test" -SECRET_REVEAL_ROLES = {"admin", "super_admin"} +SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value} DEFAULT_SETTINGS = { "system": { @@ -79,6 +83,7 @@ DEFAULT_SETTINGS = { "auto_refresh": True, "data_retention_days": 30, "max_concurrent_tasks": 5, + "demo_mode": False, }, "notifications": { "email_enabled": False, @@ -203,6 +208,7 @@ class SystemSettingsUpdate(BaseModel): auto_refresh: bool = True data_retention_days: int = Field(default=30, ge=1, le=3650) max_concurrent_tasks: int = Field(default=5, ge=1, le=50) + demo_mode: bool = False class NotificationSettingsUpdate(BaseModel): @@ -231,7 +237,7 @@ class TVStreamSourceUpdate(BaseModel): provider: str = Field(default="Unknown", max_length=100) region: str = Field(default="Global", max_length=100) language: str = Field(default="und", max_length=32) - source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$") + source_type: TVSourceType = TVSourceType.IFRAME embed_url: str = "" stream_url: str = "" homepage_url: str = "" @@ -274,7 +280,7 @@ class AIProviderIntegrationUpdate(BaseModel): service_token: Optional[str] = None default_provider: Optional[str] = None provider: str = Field(default="minimax", max_length=80) - provider_api: str = Field(default="anthropic-messages", max_length=80) + provider_api: ProviderApi = ProviderApi.ANTHROPIC_MESSAGES base_url: str = Field(default="", max_length=500) model: str = Field(default="", max_length=200) api_key: Optional[str] = None @@ -418,7 +424,7 @@ def _get_provider_preset(provider: str) -> dict: except ValueError: return { "provider": provider, - "provider_api": "openai-completions", + "provider_api": ProviderApi.OPENAI_COMPLETIONS.value, "base_url": "", "model": "", "models": [], @@ -482,12 +488,12 @@ def _provider_defaults(provider: str) -> dict: preset = _get_provider_preset(provider) return { "provider": provider, - "provider_api": preset.get("provider_api") or "openai-completions", + "provider_api": preset.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value, "base_url": preset.get("base_url") or "", "model": preset.get("model") or "", "api_key": "", "max_tokens": ( - 1200 if preset.get("provider_api") == "anthropic-messages" else 4096 + 1200 if preset.get("provider_api") == ProviderApi.ANTHROPIC_MESSAGES.value else 4096 ), "anthropic_version": "2023-06-01", "model_provider_apis": preset.get("model_provider_apis") or {}, @@ -660,7 +666,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict: ), "llm_config": { "provider": default_provider, - "provider_api": provider_config.get("provider_api") or "anthropic-messages", + "provider_api": provider_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value, "base_url": provider_config.get("base_url") or "", "model": provider_config.get("model") or "", "api_key": api_key, @@ -699,6 +705,18 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict: retry_attempts=runtime_config["retry_attempts"], llm_config=runtime_config.get("llm_config") or {}, ) + await emit_business_log( + logger, + event="settings.ai_provider.full_connection.start", + message="AI provider full connection validation started", + category="ai", + service="api", + module=__name__, + context={ + "provider": runtime_config.get("llm_config", {}).get("provider"), + "model": runtime_config.get("llm_config", {}).get("model"), + }, + ) status_result = await client.get_status() if not status_result.configured: raise HTTPException( @@ -715,6 +733,19 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict: constraints=["回复尽量简短。"], ) ) + await emit_business_log( + logger, + event="settings.ai_provider.full_connection.success", + message="AI provider full connection validation completed", + category="ai", + service="api", + module=__name__, + context={ + "provider": analysis_result.provider, + "model": analysis_result.model, + "configured": status_result.configured, + }, + ) return { "status": status_result.model_dump(), "provider": analysis_result.provider, @@ -751,7 +782,10 @@ def _contains_model(model_ids: list[str], model: str) -> bool: async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict: provider = _normalize_provider_id(llm_config.get("provider") or "") - configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions" + configured_api = ( + str(llm_config.get("provider_api") or "").strip() + or ProviderApi.OPENAI_COMPLETIONS.value + ) model = str(llm_config.get("model") or "").strip() base_url = str(llm_config.get("base_url") or "").strip().rstrip("/") api_key = str(llm_config.get("api_key") or "").strip() @@ -772,7 +806,7 @@ async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) "message": "当前 provider/base_url/model 未完整配置。", "mode": "lightweight_config", } - if provider_api != "ollama-generate" and not api_key: + if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key: return { "success": False, "connected": False, @@ -783,13 +817,13 @@ async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) if provider == "opencode-go": url = _join_provider_url(base_url, "/models") headers = {"Authorization": f"Bearer {api_key}"} - elif provider_api == "ollama-generate": + elif provider_api == ProviderApi.OLLAMA_GENERATE.value: url = _join_provider_url(base_url, "/api/tags") headers: dict[str, str] = {} - elif provider_api == "openai-completions": + elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value: url = _join_provider_url(base_url, "/models") headers = {"Authorization": f"Bearer {api_key}"} - elif provider_api == "anthropic-messages": + elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value: url = _join_provider_url(base_url, "/models") headers = { "x-api-key": api_key, @@ -1135,7 +1169,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict: api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config) providers_payload[provider_id] = { "provider": provider_id, - "provider_api": provider_config.get("provider_api") or "openai-completions", + "provider_api": provider_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value, "base_url": provider_config.get("base_url") or "", "model": provider_config.get("model") or "", "api_key": _mask_secret(api_key, api_key_source), @@ -1185,7 +1219,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict: "service_token": _mask_secret(*_resolve_service_token(normalized_ai)), "default_provider": default_provider, "provider": default_provider, - "provider_api": display_llm_config.get("provider_api") or "anthropic-messages", + "provider_api": display_llm_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value, "base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic", "model": display_llm_config.get("model") or "MiniMax-M2.7", "api_key": display_llm_config.get("api_key") or _mask_secret(None), @@ -1428,7 +1462,7 @@ async def update_smtp_settings( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if current_user.role not in ("admin", "super_admin"): + if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value): raise HTTPException(status_code=403, detail="Only administrators can change SMTP settings") current = await get_setting_payload(db, "smtp") merged = _build_smtp_payload(current, payload) @@ -1442,7 +1476,7 @@ async def test_smtp_settings( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if current_user.role not in ("admin", "super_admin"): + if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value): raise HTTPException(status_code=403, detail="Only administrators can test SMTP settings") from app.services.email import EmailError, send_email @@ -1612,9 +1646,34 @@ async def connect_ai_provider_integration( llm_config=quick_llm_config, ) + await emit_business_log( + logger, + event="settings.ai_provider.connect.start", + message="AI provider connection test started", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={ + "provider": payload.provider, + "model": payload.model, + "timeout_seconds": min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS), + }, + ) try: status_result = await client.get_status() if not status_result.configured: + await emit_business_log( + logger, + event="settings.ai_provider.connect.failed", + message="AI provider connection test failed because provider is incomplete", + category="ai", + level="warning", + service="api", + module=__name__, + user_id=current_user.id, + context={"provider": payload.provider, "model": payload.model, "configured": False}, + ) return { "success": False, "connected": False, @@ -1628,6 +1687,21 @@ async def connect_ai_provider_integration( AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS, ), ) + await emit_business_log( + logger, + event="settings.ai_provider.connect.success", + message="AI provider connection test completed", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={ + "provider": payload.provider, + "model": payload.model, + "configured": True, + "lightweight_status": lightweight_result.get("status"), + }, + ) return { **lightweight_result, "status": status_result.model_dump(), @@ -1639,6 +1713,17 @@ async def connect_ai_provider_integration( "message": str(exc.detail), } except Exception as exc: + await emit_business_log( + logger, + event="settings.ai_provider.connect.failed", + message="AI provider connection test failed", + category="ai", + level="error", + service="api", + module=__name__, + user_id=current_user.id, + context=exception_context(exc, {"provider": payload.provider, "model": payload.model}), + ) return { "success": False, "connected": False, @@ -1786,8 +1871,28 @@ async def connect_web_search_integration( runtime_config = _runtime_config_from_web_search_payload(draft_web_search_payload) client = WebSearchClient(runtime_config) + await emit_business_log( + logger, + event="settings.web_search.connect.start", + message="WebSearch connection test started", + category="ai_tool", + service="api", + module=__name__, + user_id=current_user.id, + context={"provider": runtime_config.default_provider}, + ) try: results = await client.test_connection() + await emit_business_log( + logger, + event="settings.web_search.connect.success", + message="WebSearch connection test completed", + category="ai_tool", + service="api", + module=__name__, + user_id=current_user.id, + context={"provider": runtime_config.default_provider, "result_count": len(results)}, + ) return { "success": True, "connected": True, @@ -1796,18 +1901,51 @@ async def connect_web_search_integration( "results": [item.model_dump(mode="json") for item in results[:3]], } except WebSearchConfigurationError as exc: + await emit_business_log( + logger, + event="settings.web_search.connect.failed", + message="WebSearch connection test failed because configuration is incomplete", + category="ai_tool", + level="warning", + service="api", + module=__name__, + user_id=current_user.id, + context=exception_context(exc, {"provider": runtime_config.default_provider}), + ) return { "success": False, "connected": False, "message": str(exc), } except WebSearchError as exc: + await emit_business_log( + logger, + event="settings.web_search.connect.failed", + message="WebSearch connection test failed", + category="ai_tool", + level="error", + service="api", + module=__name__, + user_id=current_user.id, + context=exception_context(exc, {"provider": runtime_config.default_provider}), + ) return { "success": False, "connected": False, "message": str(exc), } except Exception as exc: + await emit_business_log( + logger, + event="settings.web_search.connect.failed", + message="WebSearch connection test failed", + category="ai_tool", + level="error", + service="api", + module=__name__, + user_id=current_user.id, + context=exception_context(exc, {"provider": runtime_config.default_provider}), + ) return { "success": False, "connected": False, @@ -1835,17 +1973,60 @@ async def generate_provider_credential_guide( ai_client: AIProviderClient = Depends(get_ai_provider_client), ): try: + await emit_business_log( + logger, + event="settings.credential_guide.generate.start", + message="Credential guide generation started", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={"provider": provider}, + ) web_search_client = await get_web_search_client(db) - return { - "guide": await generate_credential_guide( + guide = await generate_credential_guide( db, provider, ai_client, web_search_client, ) - } + await emit_business_log( + logger, + event="settings.credential_guide.generate.success", + message="Credential guide generation completed", + category="ai", + service="api", + module=__name__, + user_id=current_user.id, + context={"provider": provider}, + ) + return {"guide": guide} except ValueError as exc: + await emit_business_log( + logger, + event="settings.credential_guide.generate.failed", + message="Credential guide generation failed", + category="ai", + level="warning", + service="api", + module=__name__, + user_id=current_user.id, + context=exception_context(exc, {"provider": provider}), + ) raise HTTPException(status_code=404, detail=str(exc)) from exc + except Exception as exc: + await emit_business_log( + logger, + event="settings.credential_guide.generate.failed", + message="Credential guide generation failed", + category="ai", + level="error", + service="api", + module=__name__, + user_id=current_user.id, + context=exception_context(exc, {"provider": provider}), + ) + raise @router.post("/credential-guides/{provider}/reset") diff --git a/backend/app/api/v1/system_control.py b/backend/app/api/v1/system_control.py index 3cc01032..f614c553 100644 --- a/backend/app/api/v1/system_control.py +++ b/backend/app/api/v1/system_control.py @@ -1,20 +1,19 @@ from __future__ import annotations import os +import secrets import subprocess import sys from datetime import datetime -from fastapi import APIRouter, Depends, HTTPException, Query, Request, status +from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, status from pydantic import BaseModel -from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from app.core.config import ROOT_DIR +from app.core.config import ROOT_DIR, settings from app.core.security import get_current_user from app.db.session import get_db -from app.models.system_log import AuditLog, SystemLog from app.models.user import User from app.services.persistent_logs import record_audit_log, record_system_log from app.services.system_control import ( @@ -37,7 +36,11 @@ from app.services.system_logs import ( append_buffer_log, list_log_sources, normalize_log_level, + read_database_log_snapshot, read_log_snapshot, + read_observability_group_events, + read_observability_groups, + read_observability_raw_events, ) from app.services.earth_layer_cache import earth_layer_cache @@ -109,12 +112,104 @@ class EarthClientLogEventCreate(BaseModel): url: str | None = None module: str | None = None detail: str | None = None + fingerprint: str | None = None + occurrence_count: int = 1 + metadata: dict[str, object] | None = None class EarthClientLogEventResponse(BaseModel): accepted: bool source_id: str level: str + fingerprint: str | None = None + + +class ServiceLogEventCreate(BaseModel): + source: str = "ai-provider" + service: str = "ai-provider" + module: str | None = None + category: str | None = None + event: str = "service.runtime_log" + level: str = "error" + message: str + fingerprint: str | None = None + occurrence_count: int = 1 + request_id: str | None = None + trace_id: str | None = None + task_id: str | None = None + source_id: int | str | None = None + provider: str | None = None + context: dict[str, object] | None = None + + +async def ingest_client_log_event( + source_id: str, + *, + service: str, + event: str, + default_module: str, + default_category: str, + payload: EarthClientLogEventCreate, + request: Request, +) -> EarthClientLogEventResponse: + normalized_level = normalize_log_level(payload.level) + append_buffer_log( + source_id, + level=normalized_level, + message=payload.message, + context={ + "category": payload.category or "", + "url": payload.url or "", + "module": payload.module or "", + "detail": payload.detail or "", + "fingerprint": payload.fingerprint or "", + "occurrence_count": max(1, int(payload.occurrence_count or 1)), + "metadata": payload.metadata or {}, + }, + ) + await record_system_log( + source=source_id, + service=service, + module=payload.module or default_module, + event=event, + level=normalized_level, + message=payload.message, + category=payload.category or default_category, + context={ + "url": payload.url or "", + "detail": payload.detail or "", + "module": payload.module or "", + "client_ip": request.client.host if request.client else "", + "metadata": payload.metadata or {}, + }, + fingerprint=payload.fingerprint, + occurrence_count=max(1, int(payload.occurrence_count or 1)), + ) + return EarthClientLogEventResponse(accepted=True, source_id=source_id, level=normalized_level, fingerprint=payload.fingerprint) + + +def require_observability_ingest_token( + authorization: str | None, + ingest_token: str | None, +) -> None: + expected_token = settings.OBSERVABILITY_INGEST_TOKEN.strip() + if not expected_token: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="Observability service ingestion is not configured", + ) + provided = "" + if ingest_token: + provided = ingest_token.strip() + elif authorization: + scheme, _, token = authorization.partition(" ") + if scheme.lower() == "bearer": + provided = token.strip() + if not provided or not secrets.compare_digest(provided, expected_token): + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Invalid observability ingestion token", + ) class EarthLayerCacheStatusResponse(BaseModel): @@ -339,95 +434,116 @@ async def get_system_log_sources( } -async def read_database_log_snapshot( - source_id: str, - *, - limit: int, - level: str, - levels: str | None, - start_date: str | None, - end_date: str | None, - search: str | None, - db: AsyncSession, -) -> dict | None: - selected_levels = set(normalize_log_level(item) for item in (levels or level).split(",") if item.strip()) - selected_levels.discard("all") - search_query = (search or "").strip().lower() - lines: list[str] = [] +@router.get("/logs/observability/groups") +async def get_observability_log_groups( + limit: int = DEFAULT_LOG_LINE_LIMIT, + level: str = "all", + levels: str | None = Query(None, description="Comma-separated log levels"), + start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"), + end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"), + search: str | None = Query(None, description="Case-insensitive substring search"), + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + ensure_super_admin(current_user) + if limit < 1 or limit > MAX_LOG_LINE_LIMIT: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}") + normalized_start_date = validate_log_date(start_date, "start_date") + normalized_end_date = validate_log_date(end_date, "end_date") + if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date") + return await read_observability_groups( + limit=limit, + level=level, + levels=levels, + start_date=normalized_start_date, + end_date=normalized_end_date, + search=search, + db=db, + ) - if source_id == "system-db": - query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(limit * 5) - result = await db.execute(query) - records = result.scalars().all() - for record in records: - record_level = normalize_log_level(record.level) - if selected_levels and record_level not in selected_levels: - continue - occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else "" - if start_date and occurred_at and occurred_at < start_date: - continue - if end_date and occurred_at and occurred_at > end_date: - continue - line = " ".join( - part - for part in [ - record.occurred_at.isoformat() if record.occurred_at else "", - record_level.upper(), - record.source, - record.event or "", - record.message, - ] - if part - ) - if search_query and search_query not in line.lower(): - continue - lines.append(line) - elif source_id == "audit-db": - query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(limit * 5) - result = await db.execute(query) - records = result.scalars().all() - for record in records: - occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else "" - if start_date and occurred_at and occurred_at < start_date: - continue - if end_date and occurred_at and occurred_at > end_date: - continue - line = " ".join( - part - for part in [ - record.occurred_at.isoformat() if record.occurred_at else "", - "INFO", - record.action, - record.target_type or "", - record.target_id or "", - record.result or "", - ] - if part - ) - if search_query and search_query not in line.lower(): - continue - lines.append(line) - else: - return None - lines = list(reversed(lines[:limit])) - return { - "source_id": source_id, - "name": "系统事件" if source_id == "system-db" else "审计事件", - "kind": "database", - "location": "table://system_logs" if source_id == "system-db" else "table://audit_logs", - "description": "数据库持久化日志", - "category": "database" if source_id == "system-db" else "audit", - "status": "ok" if lines else "empty", - "level": level, - "selected_levels": sorted(selected_levels), - "search_query": search or "", - "available_levels": ["all", "error", "warning", "info", "debug"], - "daily_markers": [], - "line_limit": limit, - "line_count": len(lines), - "lines": lines, - } +@router.get("/logs/observability/groups/{fingerprint}/events") +async def get_observability_group_events( + fingerprint: str, + limit: int = DEFAULT_LOG_LINE_LIMIT, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + ensure_super_admin(current_user) + if limit < 1 or limit > MAX_LOG_LINE_LIMIT: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}") + payload = await read_observability_group_events(fingerprint, limit=limit, db=db) + if payload is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Observability group not found") + return payload + + +@router.get("/logs/observability/raw") +async def get_observability_raw_events( + limit: int = DEFAULT_LOG_LINE_LIMIT, + level: str = "all", + levels: str | None = Query(None, description="Comma-separated log levels"), + start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"), + end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"), + search: str | None = Query(None, description="Case-insensitive substring search"), + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + ensure_super_admin(current_user) + if limit < 1 or limit > MAX_LOG_LINE_LIMIT: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}") + normalized_start_date = validate_log_date(start_date, "start_date") + normalized_end_date = validate_log_date(end_date, "end_date") + return await read_observability_raw_events( + limit=limit, + level=level, + levels=levels, + start_date=normalized_start_date, + end_date=normalized_end_date, + search=search, + db=db, + ) + + +@router.post("/logs/service", response_model=EarthClientLogEventResponse) +async def ingest_service_log( + payload: ServiceLogEventCreate, + authorization: str | None = Header(default=None), + ingest_token: str | None = Header(default=None, alias="X-Planet-Observability-Token"), +): + require_observability_ingest_token(authorization, ingest_token) + normalized_level = normalize_log_level(payload.level) + source = (payload.source or "ai-provider").strip() or "ai-provider" + context = dict(payload.context or {}) + if payload.request_id: + context["request_id"] = payload.request_id + if payload.trace_id: + context["trace_id"] = payload.trace_id + if payload.task_id: + context["task_id"] = payload.task_id + if payload.source_id is not None: + context["source_id"] = payload.source_id + if payload.provider: + context["provider"] = payload.provider + await record_system_log( + source=source, + service=(payload.service or source).strip() or source, + module=payload.module or source, + event=(payload.event or "service.runtime_log").strip() or "service.runtime_log", + level=normalized_level, + message=payload.message, + category=payload.category or "service-runtime", + context=context, + fingerprint=payload.fingerprint, + occurrence_count=max(1, int(payload.occurrence_count or 1)), + ) + return EarthClientLogEventResponse( + accepted=True, + source_id=source, + level=normalized_level, + fingerprint=payload.fingerprint, + ) @router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse) @@ -493,31 +609,28 @@ async def ingest_earth_client_log( payload: EarthClientLogEventCreate, request: Request, ): - normalized_level = normalize_log_level(payload.level) - append_buffer_log( + return await ingest_client_log_event( "earth-client", - level=normalized_level, - message=payload.message, - context={ - "category": payload.category or "", - "url": payload.url or "", - "module": payload.module or "", - "detail": payload.detail or "", - }, - ) - await record_system_log( - source="earth-client", service="earth", - module=payload.module or "earth-client", event="earth.client.runtime_log", - level=normalized_level, - message=payload.message, - category=payload.category or "client-runtime", - context={ - "url": payload.url or "", - "detail": payload.detail or "", - "module": payload.module or "", - "client_ip": request.client.host if request.client else "", - }, + default_module="earth-client", + default_category="client-runtime", + payload=payload, + request=request, + ) + + +@router.post("/logs/admin-client", response_model=EarthClientLogEventResponse) +async def ingest_admin_client_log( + payload: EarthClientLogEventCreate, + request: Request, +): + return await ingest_client_log_event( + "admin-client", + service="admin", + event="admin.client.runtime_log", + default_module="admin-client", + default_category="client-runtime", + payload=payload, + request=request, ) - return {"accepted": True, "source_id": "earth-client", "level": normalized_level} diff --git a/backend/app/api/v1/tasks.py b/backend/app/api/v1/tasks.py index 2ad29304..e3b99c8f 100644 --- a/backend/app/api/v1/tasks.py +++ b/backend/app/api/v1/tasks.py @@ -29,7 +29,8 @@ async def list_tasks( SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status, ct.started_at, ct.completed_at, ct.records_processed, ct.error_message, ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current, - ct.phase_total, ct.phase_unit, ct.total_records, ct.progress + ct.phase_total, ct.phase_unit, ct.total_records, ct.progress, + ct.task_type, ct.source, ds.source as datasource_source FROM collection_tasks ct JOIN data_sources ds ON ct.datasource_id = ds.id WHERE 1=1 @@ -39,12 +40,19 @@ async def list_tasks( if datasource_id: query += " AND ct.datasource_id = :datasource_id" - count_query += " WHERE ct.datasource_id = :datasource_id" + count_query += " AND ct.datasource_id = :datasource_id" params["datasource_id"] = datasource_id if status: - query += " AND ct.status = :status" - count_query += " AND ct.status = :status" - params["status"] = status + statuses = [item.strip() for item in status.split(",") if item.strip()] + if len(statuses) > 1: + placeholders = ", ".join(f":status_{index}" for index, _item in enumerate(statuses)) + query += f" AND ct.status IN ({placeholders})" + count_query += f" AND ct.status IN ({placeholders})" + params.update({f"status_{index}": item for index, item in enumerate(statuses)}) + else: + query += " AND ct.status = :status" + count_query += " AND ct.status = :status" + params["status"] = statuses[0] if statuses else status query += f" ORDER BY ct.created_at DESC LIMIT {page_size} OFFSET {offset}" @@ -76,6 +84,9 @@ async def list_tasks( "phase_unit": t[13], "total_records": t[14], "progress": t[15], + "task_type": t[16], + "source": t[17] or t[18], + "datasource_source": t[18], } for t in tasks ], diff --git a/backend/app/api/v1/tv.py b/backend/app/api/v1/tv.py index 75f01ea5..78687a5c 100644 --- a/backend/app/api/v1/tv.py +++ b/backend/app/api/v1/tv.py @@ -1,3 +1,4 @@ +import re from urllib.parse import quote, urljoin import httpx @@ -10,6 +11,26 @@ from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_u router = APIRouter() +_HLS_URI_ATTRIBUTE_RE = re.compile(r'URI="([^"]+)"') + + +def _proxied_tv_url(url: str) -> str: + return f"/api/v1/tv/proxy?url={quote(url, safe='')}" + + +def _rewrite_hls_uri_attributes(line: str, *, base_url: str) -> str: + def replace(match: re.Match[str]) -> str: + uri = match.group(1) + absolute_url = urljoin(base_url, uri) + return f'URI="{_proxied_tv_url(absolute_url)}"' + + return _HLS_URI_ATTRIBUTE_RE.sub(replace, line) + + +def _should_strip_hls_metadata_line(line: str) -> bool: + normalized = line.strip().upper() + return normalized.startswith("#EXT-X-MEDIA:") and "TYPE=SUBTITLES" in normalized + @router.get("/streams") async def list_public_tv_streams( @@ -56,11 +77,16 @@ async def proxy_tv_stream( rewritten_lines: list[str] = [] for line in manifest_text.splitlines(): stripped = line.strip() - if not stripped or stripped.startswith("#"): + if not stripped: rewritten_lines.append(line) continue + if stripped.startswith("#"): + if _should_strip_hls_metadata_line(line): + continue + rewritten_lines.append(_rewrite_hls_uri_attributes(line, base_url=response_url)) + continue absolute_url = urljoin(response_url, stripped) - rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}") + rewritten_lines.append(_proxied_tv_url(absolute_url)) return Response( content="\n".join(rewritten_lines), media_type="application/vnd.apple.mpegurl", diff --git a/backend/app/api/v1/users.py b/backend/app/api/v1/users.py index fdcfca95..cc174126 100644 --- a/backend/app/api/v1/users.py +++ b/backend/app/api/v1/users.py @@ -5,6 +5,7 @@ from fastapi import APIRouter, Depends, HTTPException, status from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy import text +from app.core.enums import UserRole from app.core.security import get_current_user, get_password_hash from app.db.session import get_db from app.models.user import User @@ -13,6 +14,8 @@ from app.schemas.user import UserCreate, UserUpdate router = APIRouter() VALID_GATEKEEPER_GROUPS = {"docs_user", "docs_developer", "docs_admin"} +ADMIN_ROLES = [UserRole.SUPER_ADMIN.value, UserRole.ADMIN.value] +SUPER_ADMIN_ROLES = [UserRole.SUPER_ADMIN.value] def check_permission(current_user: User, required_roles: List[str]) -> bool: @@ -32,7 +35,7 @@ async def list_users( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if not check_permission(current_user, ["super_admin", "admin"]): + if not check_permission(current_user, ADMIN_ROLES): raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Insufficient permissions", @@ -91,7 +94,7 @@ async def get_user( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id: + if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id: raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Insufficient permissions", @@ -128,7 +131,7 @@ async def create_user( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if not check_permission(current_user, ["super_admin"]): + if not check_permission(current_user, SUPER_ADMIN_ROLES): raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Only super_admin can create users", @@ -196,18 +199,18 @@ async def update_user( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id: + if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id: raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Insufficient permissions", ) - if not check_permission(current_user, ["super_admin"]) and user_data.role is not None: + if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.role is not None: raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Only super_admin can change user role", ) - if not check_permission(current_user, ["super_admin"]) and user_data.gatekeeper_groups is not None: + if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.gatekeeper_groups is not None: raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Only super_admin can change Gatekeeper groups", @@ -260,7 +263,7 @@ async def delete_user( current_user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db), ): - if not check_permission(current_user, ["super_admin"]): + if not check_permission(current_user, SUPER_ADMIN_ROLES): raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail="Only super_admin can delete users", diff --git a/backend/app/api/v1/vessels.py b/backend/app/api/v1/vessels.py index 2779e440..5dc6ded0 100644 --- a/backend/app/api/v1/vessels.py +++ b/backend/app/api/v1/vessels.py @@ -1,4 +1,4 @@ -"""Bounded vessel snapshot APIs for viewport-first consumers.""" +"""Bounded vessel snapshot APIs backed by the latest vessel state table.""" from typing import Optional @@ -14,8 +14,8 @@ router = APIRouter() @router.get("/snapshot") async def get_vessel_snapshot( - bbox: Optional[str] = Query(None, description="Viewport bbox as lon_min,lat_min,lon_max,lat_max"), - zoom: int = Query(..., ge=1, le=20, description="Current map zoom level"), + bbox: Optional[str] = Query(None, description="Snapshot bbox as lon_min,lat_min,lon_max,lat_max"), + zoom: float = Query(..., ge=1, le=20, description="Current map zoom level"), type: Optional[str] = Query( None, description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other", diff --git a/backend/app/api/v1/visualization.py b/backend/app/api/v1/visualization.py index 23c369b9..12c2a4a8 100644 --- a/backend/app/api/v1/visualization.py +++ b/backend/app/api/v1/visualization.py @@ -18,6 +18,7 @@ from sqlalchemy import select, func from typing import List, Dict, Any, Optional from app.core.collected_data_fields import get_record_field +from app.core.enums import BGPStatus from app.core.satellite_tle import build_tle_lines_from_elements from app.core.time import to_iso8601_utc from app.db.session import get_db @@ -25,7 +26,7 @@ from app.models.bgp_anomaly import BGPAnomaly from app.models.bgp_incident import BGPIncident from app.models.bgp_observation import BGPObservation from app.models.collected_data import CollectedData -from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic +from app.models.vessel import AISSourceHealth, VesselCurrentState, VesselPosition, VesselStatic from app.services.bgp_collectors import build_bgp_collector_coverage from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance from app.services.compute_center_locations import ( @@ -38,7 +39,7 @@ from app.services.compute_center_locations import ( upsert_compute_center_location, ) from app.services.ai_client import get_ai_provider_client -from app.api.v1.settings import get_web_search_client +from app.api.v1.settings import get_runtime_web_search_config, get_web_search_client from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS from app.services.location.llm_fallback import ( collect_llm_location_fallback_candidate, @@ -47,11 +48,10 @@ from app.services.location.llm_fallback import ( from app.services.persistent_logs import record_system_log from app.services.vessel_ais_aggregation import ( build_field_conflict_candidates, - count_unique_raw_vessel_mmsi, get_aggregated_vessel, get_aggregated_vessel_track, get_aggregated_vessels, - get_aggregated_vessels_snapshot, + get_current_vessels_snapshot, get_vessel_conflict_records, get_vessel_raw_observations, MAX_SNAPSHOT_LIMIT, @@ -75,7 +75,6 @@ TERRAIN_TILE_BATCH_MAX_ITEMS = 128 TERRAIN_TILE_BATCH_CONCURRENCY = 16 _terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict() VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE) -VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True SECONDS_PER_MINUTE = 60 BYTES_PER_MIB = 1024 * 1024 CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE @@ -105,8 +104,8 @@ LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy( SATELLITE_CACHE_POLICY = EarthLayerCachePolicy( SATELLITE_CACHE_FRESH_SECONDS, SATELLITE_CACHE_STALE_SECONDS, - max_features=8000, - max_bytes=10 * BYTES_PER_MIB, + max_features=25000, + max_bytes=32 * BYTES_PER_MIB, ) COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy( COMPUTE_CENTER_CACHE_FRESH_SECONDS, @@ -839,9 +838,14 @@ def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict continue source_summary = {} for source, summary in (vessel.get("source_summary") or {}).items(): + latest_observed_at = summary.get("latest_observed_at") source_summary[source] = { **summary, - "latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")), + "latest_observed_at": ( + to_iso8601_utc(latest_observed_at) + if isinstance(latest_observed_at, datetime) + else latest_observed_at + ), } props = { "mmsi": vessel["mmsi"], @@ -1075,7 +1079,7 @@ async def build_vessel_snapshot_response( db: AsyncSession, *, bbox: tuple[float, float, float, float] | None, - zoom: int | None, + zoom: float | None, type_filter: str | None, limit: int | None, since_minutes: int = 60, @@ -1563,15 +1567,7 @@ async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]: try: records = await _load_current_collected_data(db, "arcgis_cables") - if not records: - raise HTTPException( - status_code=404, - detail="No cable data found. Please run the arcgis_cables collector first.", - ) - return convert_cable_to_geojson(records) - except HTTPException: - raise except Exception as e: logger.exception_event( "Failed to build cables GeoJSON response", @@ -1625,16 +1621,8 @@ async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]: relation_records, cable_records, ) - - if not records: - raise HTTPException( - status_code=404, - detail="No landing point data found. Please run the arcgis_landing_points collector first.", - ) - + return convert_landing_point_to_geojson(records, city_to_cable_ids_map, cable_id_to_name_map) - except HTTPException: - raise except Exception as e: logger.exception_event( "Failed to build landing points GeoJSON response", @@ -2071,6 +2059,44 @@ class SaveComputeCenterLocationRequest(BaseModel): model_config = {"populate_by_name": True} +async def _compute_center_location_web_search_capability(db: AsyncSession) -> Dict[str, Any]: + try: + config = await get_runtime_web_search_config(db) + except Exception as exc: + return { + "enabled": False, + "provider": None, + "reason": f"WebSearch 配置读取失败：{exc}", + } + provider_config = config.active_provider_config + has_api_key = bool((provider_config.api_key or "").strip()) + if not config.enabled: + return { + "enabled": False, + "provider": config.default_provider, + "reason": "WebSearch 未开启，无法进行事实核查定位。", + } + if not has_api_key: + return { + "enabled": False, + "provider": config.default_provider, + "reason": f"WebSearch Provider {config.default_provider} 未配置 API Key。", + } + return { + "enabled": True, + "provider": config.default_provider, + "reason": "", + } + + +@router.get("/compute-centers/location-capability") +async def get_compute_center_location_capability( + db: AsyncSession = Depends(get_db), +): + """Return whether fact-checked compute-center location collection can run.""" + return await _compute_center_location_web_search_capability(db) + + @router.post("/compute-centers/{source_id}/collect-location") async def collect_compute_center_location( source_id: str, @@ -2089,6 +2115,9 @@ async def collect_compute_center_location( """ if not source_id or not source_id.strip(): raise HTTPException(status_code=400, detail="source_id is required") + capability = await _compute_center_location_web_search_capability(db) + if not capability.get("enabled"): + raise HTTPException(status_code=409, detail=capability) record = await _load_compute_center_record(db, source_id) name = payload.name or (record.name if record else None) @@ -2315,58 +2344,35 @@ async def _load_raw_vessel_snapshot_features( observed_since: datetime, ) -> tuple[list[dict[str, Any]], dict[str, Any]]: if bbox is None: - aggregated_vessels = await get_aggregated_vessels( - db, - limit=limit, - observed_since=observed_since, - ) - else: - aggregated_vessels = await get_aggregated_vessels_snapshot( - db, - bbox=bbox, - limit=limit, - observed_since=observed_since, - ) - raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels) - raw_features = raw_geojson.get("features", []) - features = raw_features - legacy_features: list[dict[str, Any]] = [] - legacy_fallback_used = False - if not raw_features and VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED: - legacy_features = await _load_legacy_vessel_snapshot_features( - db, - bbox=bbox, - limit=limit, - ) - features, _merge_diagnostics = _merge_vessel_features(raw_features, legacy_features) - legacy_fallback_used = bool(legacy_features) - + return [], { + "source": "vessel_current_state", + "current_state_count": 0, + "final_unique_mmsi": 0, + } + current_vessels = await get_current_vessels_snapshot( + db, + bbox=bbox, + limit=limit, + observed_since=observed_since, + ) + features = convert_aggregated_vessels_to_geojson(current_vessels).get("features", []) + unique_mmsi = len( + { + key + for key in (_feature_mmsi_key(feature) for feature in features) + if key is not None + } + ) return features, { - "raw_feature_count": len(raw_features), - "raw_unique_mmsi": len( - { - key - for key in (_feature_mmsi_key(feature) for feature in raw_features) - if key is not None - } - ), - "legacy_feature_count": len(legacy_features), - "legacy_backfilled_mmsi": len( - { - key - for key in (_feature_mmsi_key(feature) for feature in legacy_features) - if key is not None - } - ), - "legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED, - "legacy_fallback_used": legacy_fallback_used, - "final_unique_mmsi": len( - { - key - for key in (_feature_mmsi_key(feature) for feature in features) - if key is not None - } - ), + "source": "vessel_current_state", + "current_state_count": len(features), + "final_unique_mmsi": unique_mmsi, + "raw_feature_count": 0, + "raw_unique_mmsi": 0, + "legacy_feature_count": 0, + "legacy_backfilled_mmsi": 0, + "legacy_fallback_enabled": False, + "legacy_fallback_used": False, } @router.get("/vessels/custom-supplements") @@ -2696,6 +2702,8 @@ async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]: db, source_filter=("ris_live_bgp", "bgpstream_bgp"), ) + if not any(int(item.get("observation_count") or 0) > 0 for item in coverage): + return {"type": "FeatureCollection", "features": [], "count": 0} coverage_by_collector = { item["collector"]: item for item in coverage @@ -2732,10 +2740,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]: compute_center_count = supercomputer_count + gpu_cluster_count active_incident_result = await db.execute( - select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"), + select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value), ) active_anomaly_result = await db.execute( - select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"), + select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value), ) active_incident_count = int(active_incident_result.scalar() or 0) active_anomaly_count = int(active_anomaly_result.scalar() or 0) @@ -2756,21 +2764,14 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]: ) else: bgp_collector_count = int(bgp_collector_scalar or 0) - raw_unique_window_hours = 24 - raw_unique_mmsi = await count_unique_raw_vessel_mmsi( - db, - observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours), + vessel_current_window_minutes = 60 + vessel_current_result = await db.execute( + select(func.count(VesselCurrentState.mmsi)).where( + VesselCurrentState.observed_at + >= datetime.now(UTC) - timedelta(minutes=vessel_current_window_minutes) + ) ) - legacy_unique_result = await db.execute( - select(func.count(func.distinct(VesselPosition.mmsi))) - ) - legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0) - legacy_fallback_active = ( - VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED - and raw_unique_mmsi == 0 - and legacy_unique_mmsi > 0 - ) - vessel_count = legacy_unique_mmsi if legacy_fallback_active else raw_unique_mmsi + vessel_count = int(vessel_current_result.scalar() or 0) aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels") return { @@ -2781,11 +2782,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]: "satellite_count": satellite_count, "compute_center_count": compute_center_count, "vessel_count": vessel_count, - "vessel_count_source": "legacy_fallback" if legacy_fallback_active else "raw_recent", - "vessel_legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED, - "vessel_raw_unique_mmsi": raw_unique_mmsi, - "vessel_raw_unique_window_hours": raw_unique_window_hours, - "vessel_legacy_unique_mmsi": legacy_unique_mmsi, + "vessel_count_source": "vessel_current_state", + "vessel_current_window_minutes": vessel_current_window_minutes, + "vessel_raw_unique_mmsi": 0, + "vessel_legacy_unique_mmsi": 0, "aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None, "aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None, "aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None, diff --git a/backend/app/api/v1/websocket.py b/backend/app/api/v1/websocket.py index 3e512a85..4fa22eee 100644 --- a/backend/app/api/v1/websocket.py +++ b/backend/app/api/v1/websocket.py @@ -6,11 +6,15 @@ from typing import Optional from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query from jose import jwt, JWTError +from sqlalchemy import text from app.core.config import settings +from app.core.enums import UserRole from app.core.logging import get_logger from app.core.time import to_iso8601_utc from app.core.websocket.manager import manager +from app.db.session import async_session_factory +from app.services.log_tail import LOG_TAIL_CHANNEL, log_tail_manager logger = get_logger(__name__, service="api") router = APIRouter() @@ -37,6 +41,28 @@ async def authenticate_token(token: str) -> Optional[dict]: return None +async def load_websocket_user_role(user_id: str | None) -> str | None: + if not user_id: + return None + try: + async with async_session_factory() as db: + result = await db.execute( + text("SELECT role, is_active FROM users WHERE id = :id"), + {"id": int(user_id)}, + ) + row = result.fetchone() + except Exception as exc: + logger.warning_event( + "WebSocket user role lookup failed", + event="auth.websocket.role_lookup_failed", + context={"user_id": user_id, "error": str(exc)}, + ) + return None + if row is None or not row[1]: + return None + return str(row[0] or "") + + @router.websocket("/ws") async def websocket_endpoint( websocket: WebSocket, @@ -59,6 +85,7 @@ async def websocket_endpoint( is_anonymous = payload is None user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}" + user_role = await load_websocket_user_role(user_id) if payload else None supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [ "gpu_clusters", "submarine_cables", @@ -70,6 +97,8 @@ async def websocket_endpoint( "earth_news", EARTH_UPDATES_CHANNEL, ] + if user_role == UserRole.SUPER_ADMIN.value: + supported_channels = [*supported_channels, LOG_TAIL_CHANNEL] await manager.connect(websocket, user_id) try: @@ -100,6 +129,7 @@ async def websocket_endpoint( payload_data = data.get("data", {}) if not isinstance(payload_data, dict): payload_data = {} + log_tail_config = None channels = payload_data.get("channels", []) if isinstance(channels, str): channels = [channels] @@ -108,6 +138,26 @@ async def websocket_endpoint( channel = payload_data.get("channel") if channel and channel not in channels: channels = [*channels, channel] + if LOG_TAIL_CHANNEL in channels: + if user_role != UserRole.SUPER_ADMIN.value: + await websocket.send_json( + { + "type": "subscription_error", + "data": {"channel": LOG_TAIL_CHANNEL, "detail": "Only super_admin can subscribe logs"}, + } + ) + channels = [item for item in channels if item != LOG_TAIL_CHANNEL] + else: + try: + log_tail_config = await log_tail_manager.subscribe(websocket, payload_data) + except ValueError as exc: + await websocket.send_json( + { + "type": "subscription_error", + "data": {"channel": LOG_TAIL_CHANNEL, "detail": str(exc)}, + } + ) + channels = [item for item in channels if item != LOG_TAIL_CHANNEL] if is_anonymous: channels = [channel for channel in channels if channel in supported_channels] vessel_subscription = None @@ -131,14 +181,20 @@ async def websocket_endpoint( "action": "subscribe", "channels": [ *channels, + *([LOG_TAIL_CHANNEL] if log_tail_config else []), *(["vessels"] if vessel_subscription else []), ], "vessels": vessel_subscription, + "logs_tail": log_tail_config.__dict__ if log_tail_config else None, }, } ) elif data.get("type") == "unsubscribe": channels = data.get("data", {}).get("channels", []) + if isinstance(channels, str): + channels = [channels] + if LOG_TAIL_CHANNEL in channels: + await log_tail_manager.unsubscribe(websocket) manager.unsubscribe(websocket, channels) await websocket.send_json( { @@ -159,4 +215,5 @@ async def websocket_endpoint( except WebSocketDisconnect: pass finally: + await log_tail_manager.disconnect(websocket) manager.disconnect(websocket, user_id) diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 24e3559b..1210d881 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -41,6 +41,7 @@ class Settings(BaseSettings): AI_PROVIDER_SERVICE_TOKEN: str = "" AI_PROVIDER_TIMEOUT_SECONDS: int = 60 AI_PROVIDER_RETRY_ATTEMPTS: int = 2 + OBSERVABILITY_INGEST_TOKEN: str = "" @property def REDIS_URL(self) -> str: diff --git a/backend/app/core/enums.py b/backend/app/core/enums.py new file mode 100644 index 00000000..4b04ad77 --- /dev/null +++ b/backend/app/core/enums.py @@ -0,0 +1,226 @@ +"""Stable backend protocol enums. + +Database columns and JSON payloads continue to store the enum string values. +Configurable identifiers, user-authored values, and open-ended taxonomies do +not belong in this module. +""" + +from __future__ import annotations + +import logging +from enum import StrEnum +from typing import TypeVar + +logger = logging.getLogger(__name__) + +EnumT = TypeVar("EnumT", bound=StrEnum) + + +def parse_enum(enum_type: type[EnumT], value: object, default: EnumT) -> EnumT: + """Parse an external value without breaking reads of legacy data.""" + + if value is None or str(value).strip() == "": + return default + if isinstance(value, enum_type): + return value + try: + return enum_type(str(value).strip().lower()) + except (TypeError, ValueError): + logger.warning( + "Unknown %s value %r; falling back to %s", + enum_type.__name__, + value, + default.value, + ) + return default + + +class NewsImportanceLevel(StrEnum): + LOW = "low" + MEDIUM = "medium" + HIGH = "high" + CRITICAL = "critical" + + +class BreakingLevel(StrEnum): + NONE = "none" + WATCH = "watch" + BREAKING = "breaking" + CRITICAL = "critical" + + +class BreakingScope(StrEnum): + REGIONAL = "regional" + GLOBAL = "global" + + +class BreakingSource(StrEnum): + RULES = "rules" + AI = "ai" + MANUAL = "manual" + MULTI_SOURCE = "multi_source" + + +class NewsSourceType(StrEnum): + RSS = "rss" + ATOM = "atom" + AGGREGATED = "aggregated" + REFERENCE = "reference" + MANUAL = "manual" + + +class NewsEnrichmentStatus(StrEnum): + PENDING = "pending" + QUEUED = "queued" + ATTEMPTED = "attempted" + SUCCESS = "success" + CONTENT_ONLY = "content_only" + LOCATION_ONLY = "location_only" + UNAVAILABLE = "unavailable" + PROVIDER_ERROR = "provider_error" + PARSE_ERROR = "parse_error" + NO_RESULT = "no_result" + + +class NewsMarketImpact(StrEnum): + NONE = "none" + SECTOR = "sector" + NATIONAL = "national" + GLOBAL = "global" + + +class NewsTaggingSource(StrEnum): + RULES = "rules" + AI = "ai" + MANUAL = "manual" + + +class JobType(StrEnum): + COLLECT = "collect" + CLEAR_DATA = "clear_data" + CLEAR_CACHE = "clear_cache" + EARTH_REFRESH = "earth_refresh" + + +class JobStatus(StrEnum): + QUEUED = "queued" + RUNNING = "running" + CANCELLING = "cancelling" + SUCCESS = "success" + FAILED = "failed" + CANCELLED = "cancelled" + + +class RollbackPolicy(StrEnum): + KEEP_COMMITTED_BATCHES = "keep_committed_batches" + + +class MappingValidationStatus(StrEnum): + DRAFT = "draft" + VALID = "valid" + INVALID = "invalid" + + +class SnapshotStatus(StrEnum): + RUNNING = "running" + SUCCESS = "success" + FAILED = "failed" + CANCELLED = "cancelled" + + +class DatasourceRunStatus(StrEnum): + RUNNING = "running" + NOT_RUN = "not_run" + COLLECTED = "collected" + UNCOLLECTED = "uncollected" + + +class ProviderApi(StrEnum): + ANTHROPIC_MESSAGES = "anthropic-messages" + OPENAI_COMPLETIONS = "openai-completions" + OLLAMA_GENERATE = "ollama-generate" + + +class PlaygroundMessageRole(StrEnum): + SYSTEM = "system" + USER = "user" + ASSISTANT = "assistant" + TOOL = "tool" + + +class PlaygroundMessageKind(StrEnum): + MESSAGE = "message" + THINKING = "thinking" + ERROR = "error" + STATUS = "status" + + +class PlaygroundMessageStatus(StrEnum): + PENDING = "pending" + THINKING = "thinking" + ANSWERING = "answering" + DONE = "done" + FAILED = "failed" + CANCELLED = "cancelled" + ERROR = "error" + STOPPED = "stopped" + + +class OtpPurpose(StrEnum): + REGISTER = "register" + VERIFY_EMAIL = "verify_email" + RESET_PASSWORD = "reset_password" + + +class UserRole(StrEnum): + VIEWER = "viewer" + ADMIN = "admin" + SUPER_ADMIN = "super_admin" + + +class AlertSeverity(StrEnum): + CRITICAL = "critical" + WARNING = "warning" + INFO = "info" + + +class AlertStatus(StrEnum): + ACTIVE = "active" + ACKNOWLEDGED = "acknowledged" + RESOLVED = "resolved" + + +class BGPStatus(StrEnum): + ACTIVE = "active" + ACKNOWLEDGED = "acknowledged" + RESOLVED = "resolved" + + +class LogLevel(StrEnum): + ALL = "all" + ERROR = "error" + WARNING = "warning" + INFO = "info" + DEBUG = "debug" + + +class ConnectionState(StrEnum): + DISCONNECTED = "disconnected" + CONNECTING = "connecting" + CONNECTED = "connected" + ERROR = "error" + + +class AuthType(StrEnum): + NONE = "none" + BEARER = "bearer" + API_KEY = "api_key" + BASIC = "basic" + + +class TVSourceType(StrEnum): + IFRAME = "iframe" + HLS = "hls" + VIDEO = "video" + EXTERNAL = "external" + YOUTUBE = "youtube" diff --git a/backend/app/core/websocket/broadcaster.py b/backend/app/core/websocket/broadcaster.py index a80af079..2e1f1cd8 100644 --- a/backend/app/core/websocket/broadcaster.py +++ b/backend/app/core/websocket/broadcaster.py @@ -150,7 +150,7 @@ class DataBroadcaster: "timestamp": to_iso8601_utc(datetime.now(UTC)), "payload": data, }, - channel="all", + channel="datasource_tasks", ) def start(self): diff --git a/backend/app/db/session.py b/backend/app/db/session.py index 751003a4..57aecbb4 100644 --- a/backend/app/db/session.py +++ b/backend/app/db/session.py @@ -203,6 +203,7 @@ async def init_db(): import app.models.vessel_enrichment # noqa: F401 import app.models.datasource_mapping # noqa: F401 import app.models.earth_news # noqa: F401 + import app.models.earth_interactable # noqa: F401 logger.warning_event( "Database pool settings active", @@ -258,6 +259,407 @@ async def init_db(): """ ) ) + await conn.execute( + text( + """ + CREATE TABLE IF NOT EXISTS earth_data_change_events ( + id BIGSERIAL PRIMARY KEY, + table_name VARCHAR(128) NOT NULL, + operation VARCHAR(16) NOT NULL, + source VARCHAR(128), + entity_key VARCHAR(255), + payload JSONB NOT NULL, + occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + consumed_at TIMESTAMPTZ + ) + """ + ) + ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed + ON earth_data_change_events (consumed_at, id) + WHERE consumed_at IS NULL + """ + ) + ) + await conn.execute( + text( + """ + CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement( + change_table TEXT, + change_operation TEXT, + change_source TEXT, + source_record_count INTEGER, + source_entity_keys TEXT[] + ) + RETURNS VOID AS $$ + DECLARE + change_event_id BIGINT; + change_payload JSONB; + BEGIN + change_payload := jsonb_build_object( + 'event', 'earth.layer.changed', + 'table', change_table, + 'operation', change_operation, + 'source', change_source, + 'entity_key', NULL, + 'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb), + 'records_processed', COALESCE(source_record_count, 0), + 'occurred_at', NOW() + ); + + INSERT INTO earth_data_change_events ( + table_name, + operation, + source, + entity_key, + payload, + occurred_at + ) VALUES ( + change_table, + change_operation, + change_source, + NULL, + change_payload, + NOW() + ) + RETURNING id INTO change_event_id; + + change_payload := change_payload || jsonb_build_object( + 'event_id', change_event_id + ); + + UPDATE earth_data_change_events + SET payload = change_payload + WHERE id = change_event_id; + + PERFORM pg_notify( + 'planet_earth_data_changes', + change_payload::text + ); + END; + $$ LANGUAGE plpgsql; + """ + ) + ) + await conn.execute( + text( + """ + CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement( + change_operation TEXT, + change_source TEXT, + source_record_count INTEGER, + source_entity_keys TEXT[] + ) + RETURNS VOID AS $$ + BEGIN + PERFORM planet_emit_earth_data_changed_statement( + 'collected_data', + change_operation, + change_source, + source_record_count, + source_entity_keys + ); + END; + $$ LANGUAGE plpgsql; + """ + ) + ) + await conn.execute( + text( + """ + CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement() + RETURNS trigger AS $$ + DECLARE + change_source TEXT; + source_record_count INTEGER; + source_entity_keys TEXT[]; + BEGIN + IF TG_OP = 'INSERT' THEN + FOR change_source IN + SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) + FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows + LOOP + SELECT + COUNT(*), + ARRAY( + SELECT DISTINCT COALESCE( + NULLIF(row_data->>'entity_key', ''), + NULLIF(row_data->>'source_id', ''), + NULLIF(row_data->>'incident_key', ''), + NULLIF(row_data->>'id', ''), + NULLIF(row_data->>'mmsi', '') + ) + FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys + WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source + LIMIT 20 + ) + INTO source_record_count, source_entity_keys + FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count + WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source; + + PERFORM planet_emit_earth_data_changed_statement( + TG_TABLE_NAME, + TG_OP, + change_source, + source_record_count, + source_entity_keys + ); + END LOOP; + ELSIF TG_OP = 'DELETE' THEN + FOR change_source IN + SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) + FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows + LOOP + SELECT + COUNT(*), + ARRAY( + SELECT DISTINCT COALESCE( + NULLIF(row_data->>'entity_key', ''), + NULLIF(row_data->>'source_id', ''), + NULLIF(row_data->>'incident_key', ''), + NULLIF(row_data->>'id', ''), + NULLIF(row_data->>'mmsi', '') + ) + FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys + WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source + LIMIT 20 + ) + INTO source_record_count, source_entity_keys + FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count + WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source; + + PERFORM planet_emit_earth_data_changed_statement( + TG_TABLE_NAME, + TG_OP, + change_source, + source_record_count, + source_entity_keys + ); + END LOOP; + ELSIF TG_OP = 'UPDATE' THEN + FOR change_source IN + SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) + FROM ( + SELECT to_jsonb(t) AS row_data FROM new_rows AS t + UNION ALL + SELECT to_jsonb(t) AS row_data FROM old_rows AS t + ) changed_rows + LOOP + SELECT + COUNT(*), + ARRAY( + SELECT DISTINCT COALESCE( + NULLIF(row_data->>'entity_key', ''), + NULLIF(row_data->>'source_id', ''), + NULLIF(row_data->>'incident_key', ''), + NULLIF(row_data->>'id', ''), + NULLIF(row_data->>'mmsi', '') + ) + FROM ( + SELECT to_jsonb(t) AS row_data FROM new_rows AS t + UNION ALL + SELECT to_jsonb(t) AS row_data FROM old_rows AS t + ) rows_for_keys + WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source + LIMIT 20 + ) + INTO source_record_count, source_entity_keys + FROM ( + SELECT to_jsonb(t) AS row_data FROM new_rows AS t + UNION ALL + SELECT to_jsonb(t) AS row_data FROM old_rows AS t + ) rows_for_count + WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source; + + PERFORM planet_emit_earth_data_changed_statement( + TG_TABLE_NAME, + TG_OP, + change_source, + source_record_count, + source_entity_keys + ); + END LOOP; + END IF; + + RETURN NULL; + END; + $$ LANGUAGE plpgsql; + """ + ) + ) + await conn.execute( + text( + """ + CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement() + RETURNS trigger AS $$ + DECLARE + change_source TEXT; + source_record_count INTEGER; + source_entity_keys TEXT[]; + BEGIN + IF TG_OP = 'INSERT' THEN + FOR change_source IN + SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL + LOOP + SELECT + COUNT(*), + ARRAY( + SELECT DISTINCT COALESCE(entity_key, source_id, id::text) + FROM new_rows + WHERE source = change_source + LIMIT 20 + ) + INTO source_record_count, source_entity_keys + FROM new_rows + WHERE source = change_source; + + PERFORM planet_emit_collected_data_changed_statement( + TG_OP, + change_source, + source_record_count, + source_entity_keys + ); + END LOOP; + ELSIF TG_OP = 'DELETE' THEN + FOR change_source IN + SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL + LOOP + SELECT + COUNT(*), + ARRAY( + SELECT DISTINCT COALESCE(entity_key, source_id, id::text) + FROM old_rows + WHERE source = change_source + LIMIT 20 + ) + INTO source_record_count, source_entity_keys + FROM old_rows + WHERE source = change_source; + + PERFORM planet_emit_collected_data_changed_statement( + TG_OP, + change_source, + source_record_count, + source_entity_keys + ); + END LOOP; + ELSIF TG_OP = 'UPDATE' THEN + FOR change_source IN + SELECT DISTINCT source FROM ( + SELECT source FROM new_rows + UNION + SELECT source FROM old_rows + ) changed_sources + WHERE source IS NOT NULL + LOOP + SELECT + COUNT(*), + ARRAY( + SELECT DISTINCT COALESCE(entity_key, source_id, id::text) + FROM ( + SELECT id, source_id, entity_key, source FROM new_rows + UNION ALL + SELECT id, source_id, entity_key, source FROM old_rows + ) changed_rows + WHERE source = change_source + LIMIT 20 + ) + INTO source_record_count, source_entity_keys + FROM ( + SELECT id, source_id, entity_key, source FROM new_rows + UNION ALL + SELECT id, source_id, entity_key, source FROM old_rows + ) changed_rows + WHERE source = change_source; + + PERFORM planet_emit_collected_data_changed_statement( + TG_OP, + change_source, + source_record_count, + source_entity_keys + ); + END LOOP; + END IF; + + RETURN NULL; + END; + $$ LANGUAGE plpgsql; + """ + ) + ) + for statement in ( + "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data", + "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data", + "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data", + "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data", + "DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()", + """ + CREATE TRIGGER tr_planet_collected_data_changed_insert + AFTER INSERT ON collected_data + REFERENCING NEW TABLE AS new_rows + FOR EACH STATEMENT + EXECUTE FUNCTION planet_notify_collected_data_changed_statement() + """, + """ + CREATE TRIGGER tr_planet_collected_data_changed_update + AFTER UPDATE ON collected_data + REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows + FOR EACH STATEMENT + EXECUTE FUNCTION planet_notify_collected_data_changed_statement() + """, + """ + CREATE TRIGGER tr_planet_collected_data_changed_delete + AFTER DELETE ON collected_data + REFERENCING OLD TABLE AS old_rows + FOR EACH STATEMENT + EXECUTE FUNCTION planet_notify_collected_data_changed_statement() + """, + ): + await conn.execute(text(statement)) + for table_name in ( + "bgp_observations", + "bgp_anomalies", + "bgp_incidents", + "bgp_collector_locations", + "vessel_static", + "vessel_position", + "vessel_current_state", + "ais_raw_observations", + "ais_source_health", + "compute_center_locations", + "earth_interactables", + "earth_news_items", + ): + for statement in ( + f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}", + f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}", + f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}", + f""" + CREATE TRIGGER tr_planet_{table_name}_changed_insert + AFTER INSERT ON {table_name} + REFERENCING NEW TABLE AS new_rows + FOR EACH STATEMENT + EXECUTE FUNCTION planet_notify_earth_table_changed_statement() + """, + f""" + CREATE TRIGGER tr_planet_{table_name}_changed_update + AFTER UPDATE ON {table_name} + REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows + FOR EACH STATEMENT + EXECUTE FUNCTION planet_notify_earth_table_changed_statement() + """, + f""" + CREATE TRIGGER tr_planet_{table_name}_changed_delete + AFTER DELETE ON {table_name} + REFERENCING OLD TABLE AS old_rows + FOR EACH STATEMENT + EXECUTE FUNCTION planet_notify_earth_table_changed_statement() + """, + ): + await conn.execute(text(statement)) await conn.execute( text( """ @@ -267,7 +669,16 @@ async def init_db(): ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255), ADD COLUMN IF NOT EXISTS phase_current BIGINT, ADD COLUMN IF NOT EXISTS phase_total BIGINT, - ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30) + ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30), + ADD COLUMN IF NOT EXISTS source VARCHAR(100), + ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect', + ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb, + ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches', + ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180), + ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120), + ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ, + ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ, + ADD COLUMN IF NOT EXISTS cancel_reason TEXT """ ) ) @@ -283,6 +694,17 @@ async def init_db(): """ ) ) + await conn.execute( + text( + """ + ALTER TABLE earth_interactables + ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION, + ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1, + ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE, + ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ + """ + ) + ) await conn.execute( text( """ @@ -307,6 +729,48 @@ async def init_db(): """ ) ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted + ON earth_interactables (layer, is_deleted) + """ + ) + ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at + ON earth_interactables (updated_at) + """ + ) + ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status + ON collection_tasks (source, status) + """ + ) + ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue + ON collection_tasks (status, created_at, id) + WHERE status = 'queued' + """ + ) + ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe + ON collection_tasks (dedupe_key) + WHERE dedupe_key IS NOT NULL + """ + ) + ) await conn.execute( text( """ @@ -323,6 +787,22 @@ async def init_db(): """ ) ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_vessel_current_bbox + ON vessel_current_state (lon, lat) + """ + ) + ) + await conn.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_vessel_current_observed + ON vessel_current_state (observed_at DESC) + """ + ) + ) await conn.execute( text( """ diff --git a/backend/app/main.py b/backend/app/main.py index 5d0b0a1d..f4ca64fd 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -24,6 +24,11 @@ from app.services.earth_news_worker import ( start_earth_news_target_worker, stop_earth_news_target_worker, ) +from app.services.earth_db_change_listener import ( + start_earth_db_change_listener, + stop_earth_db_change_listener, +) +from app.services.data_jobs import start_data_job_worker, stop_data_job_worker configure_logging() @@ -59,9 +64,13 @@ async def lifespan(app: FastAPI): start_scheduler() await sync_scheduler_with_datasources() broadcaster.start() + start_data_job_worker() + start_earth_db_change_listener() start_earth_news_target_worker() yield await stop_earth_news_target_worker() + await stop_earth_db_change_listener() + await stop_data_job_worker() broadcaster.stop() stop_scheduler() diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index eaa8dbdc..563b6223 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -13,10 +13,11 @@ from app.models.compute_center_location import ComputeCenterLocationRecord from app.models.system_setting import SystemSetting from app.models.playground_session import PlaygroundSession from app.models.playground_message import PlaygroundMessage -from app.models.system_log import SystemLog, AuditLog +from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic from app.models.datasource_mapping import DataSourceMappingTemplate from app.models.earth_news import EarthNewsItem +from app.models.earth_interactable import EarthInteractable __all__ = [ "User", @@ -36,6 +37,8 @@ __all__ = [ "ComputeCenterLocationRecord", "SystemLog", "AuditLog", + "ObservabilityEvent", + "ObservabilityEventGroup", "PlaygroundSession", "PlaygroundMessage", "VesselPosition", @@ -45,4 +48,5 @@ __all__ = [ "AISSourceHealth", "DataSourceMappingTemplate", "EarthNewsItem", + "EarthInteractable", ] diff --git a/backend/app/models/alert.py b/backend/app/models/alert.py index 9c141e4c..90b3ea8d 100644 --- a/backend/app/models/alert.py +++ b/backend/app/models/alert.py @@ -1,26 +1,12 @@ from datetime import datetime -from enum import Enum -from typing import Optional -from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum -from sqlalchemy.orm import relationship +from sqlalchemy import Column, Integer, String, DateTime, Text, Enum as SQLEnum +from app.core.enums import AlertSeverity, AlertStatus from app.core.time import to_iso8601_utc from app.db.session import Base -class AlertSeverity(str, Enum): - CRITICAL = "critical" - WARNING = "warning" - INFO = "info" - - -class AlertStatus(str, Enum): - ACTIVE = "active" - ACKNOWLEDGED = "acknowledged" - RESOLVED = "resolved" - - class Alert(Base): __tablename__ = "alerts" diff --git a/backend/app/models/bgp_anomaly.py b/backend/app/models/bgp_anomaly.py index 013aa8fa..b54d987e 100644 --- a/backend/app/models/bgp_anomaly.py +++ b/backend/app/models/bgp_anomaly.py @@ -4,6 +4,7 @@ from datetime import datetime from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text +from app.core.enums import BGPStatus from app.core.time import to_iso8601_utc from app.db.session import Base @@ -17,7 +18,7 @@ class BGPAnomaly(Base): source = Column(String(100), nullable=False, index=True) anomaly_type = Column(String(50), nullable=False, index=True) severity = Column(String(20), nullable=False, index=True) - status = Column(String(20), nullable=False, default="active", index=True) + status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True) entity_key = Column(String(255), nullable=False, index=True) prefix = Column(String(64), nullable=True, index=True) origin_asn = Column(Integer, nullable=True, index=True) diff --git a/backend/app/models/bgp_incident.py b/backend/app/models/bgp_incident.py index 4e901c7a..8e3e573f 100644 --- a/backend/app/models/bgp_incident.py +++ b/backend/app/models/bgp_incident.py @@ -4,6 +4,7 @@ from datetime import datetime from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text +from app.core.enums import BGPStatus from app.core.time import to_iso8601_utc from app.db.session import Base @@ -20,7 +21,7 @@ class BGPIncident(Base): title = Column(String(255), nullable=False) summary = Column(Text, nullable=False) severity = Column(String(20), nullable=False, index=True) - status = Column(String(20), nullable=False, default="active", index=True) + status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True) confidence = Column(Float, nullable=False, default=0.5) started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True) ended_at = Column(DateTime(timezone=True), nullable=True) diff --git a/backend/app/models/data_snapshot.py b/backend/app/models/data_snapshot.py index f70b4f12..fd226d39 100644 --- a/backend/app/models/data_snapshot.py +++ b/backend/app/models/data_snapshot.py @@ -1,6 +1,7 @@ from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String from sqlalchemy.sql import func +from app.core.enums import SnapshotStatus from app.db.session import Base @@ -16,7 +17,7 @@ class DataSnapshot(Base): started_at = Column(DateTime(timezone=True), server_default=func.now()) completed_at = Column(DateTime(timezone=True), nullable=True) record_count = Column(Integer, default=0) - status = Column(String(20), nullable=False, default="running") + status = Column(String(20), nullable=False, default=SnapshotStatus.RUNNING.value) is_current = Column(Boolean, default=True, index=True) parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True) summary = Column(JSON, default={}) diff --git a/backend/app/models/datasource_mapping.py b/backend/app/models/datasource_mapping.py index eee2c269..6c7ddf3a 100644 --- a/backend/app/models/datasource_mapping.py +++ b/backend/app/models/datasource_mapping.py @@ -3,6 +3,7 @@ from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String from sqlalchemy.sql import func +from app.core.enums import MappingValidationStatus from app.db.session import Base @@ -19,7 +20,7 @@ class DataSourceMappingTemplate(Base): target_schema = Column(String(80), nullable=False, index=True) mapping_json = Column(JSON, nullable=False, default={}) sample_payload_hash = Column(String(64), nullable=True) - validation_status = Column(String(30), nullable=False, default="draft") + validation_status = Column(String(30), nullable=False, default=MappingValidationStatus.DRAFT.value) version = Column(Integer, nullable=False, default=1) is_active = Column(Boolean, nullable=False, default=False, index=True) created_at = Column(DateTime(timezone=True), server_default=func.now()) diff --git a/backend/app/models/earth_interactable.py b/backend/app/models/earth_interactable.py new file mode 100644 index 00000000..c5547ff0 --- /dev/null +++ b/backend/app/models/earth_interactable.py @@ -0,0 +1,30 @@ +"""Persistent Earth interactable objects.""" + +from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text +from sqlalchemy.sql import func + +from app.db.session import Base + + +class EarthInteractable(Base): + __tablename__ = "earth_interactables" + + id = Column(String(160), primary_key=True) + layer = Column(String(80), nullable=False, default="interactables", index=True) + kind = Column(String(80), nullable=False, default="default", index=True) + label = Column(String(255), nullable=False, default="") + description = Column(Text, nullable=False, default="") + latitude = Column(Float, nullable=False) + longitude = Column(Float, nullable=False) + altitude = Column(Float, nullable=True) + revision = Column(Integer, nullable=False, default=1) + properties = Column(JSON, nullable=False, default=dict) + is_deleted = Column(Boolean, nullable=False, default=False, index=True) + created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False) + updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now()) + deleted_at = Column(DateTime(timezone=True), nullable=True, index=True) + + __table_args__ = ( + Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"), + Index("idx_earth_interactables_updated_at", "updated_at"), + ) diff --git a/backend/app/models/playground_message.py b/backend/app/models/playground_message.py index ce85de8c..ed454137 100644 --- a/backend/app/models/playground_message.py +++ b/backend/app/models/playground_message.py @@ -1,6 +1,7 @@ from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text from sqlalchemy.sql import func +from app.core.enums import PlaygroundMessageKind, PlaygroundMessageStatus from app.db.session import Base @@ -13,8 +14,8 @@ class PlaygroundMessage(Base): user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True) parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True) role = Column(String(20), nullable=False) - kind = Column(String(20), nullable=False, default="message") - status = Column(String(20), nullable=False, default="done") + kind = Column(String(20), nullable=False, default=PlaygroundMessageKind.MESSAGE.value) + status = Column(String(20), nullable=False, default=PlaygroundMessageStatus.DONE.value) title = Column(String(255), nullable=True) content = Column(Text, nullable=False, default="") thinking_content = Column(Text, nullable=False, default="") diff --git a/backend/app/models/system_log.py b/backend/app/models/system_log.py index 5024c9b4..9b98f82a 100644 --- a/backend/app/models/system_log.py +++ b/backend/app/models/system_log.py @@ -38,3 +38,46 @@ class AuditLog(Base): ip = Column(String(64), nullable=True) details = Column(JSON, nullable=False, default=dict) created_at = Column(DateTime(timezone=True), server_default=func.now()) + + +class ObservabilityEvent(Base): + __tablename__ = "observability_events" + + id = Column(Integer, primary_key=True, autoincrement=True) + occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True) + source = Column(String(50), nullable=False, index=True) + service = Column(String(50), nullable=True, index=True) + module = Column(String(120), nullable=True, index=True) + category = Column(String(80), nullable=True, index=True) + event = Column(String(160), nullable=True, index=True) + level = Column(String(20), nullable=False, index=True) + message = Column(Text, nullable=False) + fingerprint = Column(String(80), nullable=False, index=True) + request_id = Column(String(64), nullable=True, index=True) + trace_id = Column(String(64), nullable=True, index=True) + task_id = Column(String(120), nullable=True, index=True) + source_ref_id = Column(String(120), nullable=True, index=True) + provider = Column(String(120), nullable=True, index=True) + user_id = Column(Integer, nullable=True, index=True) + context = Column(JSON, nullable=False, default=dict) + occurrence_count = Column(Integer, nullable=False, default=1) + created_at = Column(DateTime(timezone=True), server_default=func.now()) + + +class ObservabilityEventGroup(Base): + __tablename__ = "observability_event_groups" + + fingerprint = Column(String(80), primary_key=True) + source = Column(String(50), nullable=False, index=True) + service = Column(String(50), nullable=True, index=True) + module = Column(String(120), nullable=True, index=True) + category = Column(String(80), nullable=True, index=True) + event = Column(String(160), nullable=True, index=True) + last_level = Column(String(20), nullable=False, index=True) + sample_message = Column(Text, nullable=False) + sample_detail = Column(Text, nullable=True) + affected_sources = Column(JSON, nullable=False, default=list) + count = Column(Integer, nullable=False, default=0) + first_seen_at = Column(DateTime(timezone=True), nullable=False, index=True) + last_seen_at = Column(DateTime(timezone=True), nullable=False, index=True) + updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now()) diff --git a/backend/app/models/task.py b/backend/app/models/task.py index 0e29c299..f35ca540 100644 --- a/backend/app/models/task.py +++ b/backend/app/models/task.py @@ -1,8 +1,9 @@ -"""Collection Task model""" +"""Datasource job model.""" -from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float +from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text from sqlalchemy.sql import func +from app.core.enums import JobStatus, JobType, RollbackPolicy from app.db.session import Base @@ -11,8 +12,10 @@ class CollectionTask(Base): id = Column(Integer, primary_key=True, autoincrement=True) datasource_id = Column(Integer, nullable=False, index=True) - status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled - phase = Column(String(30), default="queued") + source = Column(String(100), nullable=True, index=True) + task_type = Column(String(30), nullable=False, default=JobType.COLLECT.value, index=True) + status = Column(String(20), nullable=False) # queued, running, cancelling, success, failed, cancelled + phase = Column(String(30), default=JobStatus.QUEUED.value) phase_progress = Column(Float) phase_message = Column(String(255)) phase_current = Column(BigInteger) @@ -24,6 +27,13 @@ class CollectionTask(Base): total_records = Column(Integer, default=0) # Total records to process progress = Column(Float, default=0.0) # Progress percentage (0-100) error_message = Column(Text) + payload = Column(JSON, default=dict) + rollback_policy = Column(String(40), nullable=False, default=RollbackPolicy.KEEP_COMMITTED_BATCHES.value) + dedupe_key = Column(String(180), nullable=True, index=True) + worker_id = Column(String(120), nullable=True, index=True) + locked_at = Column(DateTime(timezone=True), nullable=True, index=True) + requested_cancel_at = Column(DateTime(timezone=True), nullable=True) + cancel_reason = Column(Text) created_at = Column(DateTime(timezone=True), server_default=func.now()) def __repr__(self): diff --git a/backend/app/models/user.py b/backend/app/models/user.py index e06407b7..3b188b17 100644 --- a/backend/app/models/user.py +++ b/backend/app/models/user.py @@ -1,6 +1,7 @@ from sqlalchemy import Boolean, Column, DateTime, Integer, JSON, String from sqlalchemy.sql import func +from app.core.enums import UserRole from app.db.session import Base @@ -11,7 +12,7 @@ class User(Base): username = Column(String(50), unique=True, index=True, nullable=False) email = Column(String(255), unique=True, index=True, nullable=False) password_hash = Column(String(255), nullable=False) - role = Column(String(20), default="viewer") + role = Column(String(20), default=UserRole.VIEWER.value) gatekeeper_groups = Column(JSON, default=list) is_active = Column(Boolean, default=True) email_verified = Column(Boolean, default=False, nullable=False) diff --git a/backend/app/models/vessel.py b/backend/app/models/vessel.py index 35e93b1c..70992eb6 100644 --- a/backend/app/models/vessel.py +++ b/backend/app/models/vessel.py @@ -3,6 +3,7 @@ from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String from sqlalchemy.sql import func +from app.core.enums import ConnectionState from app.core.time import to_iso8601_utc from app.db.session import Base @@ -75,6 +76,67 @@ class VesselPosition(Base): } +class VesselCurrentState(Base): + """Latest renderable state for one vessel, independent from AIS history.""" + + __tablename__ = "vessel_current_state" + + mmsi = Column(BigInteger, primary_key=True) + lat = Column(Float, nullable=False) + lon = Column(Float, nullable=False) + sog = Column(Float, nullable=True) + cog = Column(Float, nullable=True) + heading = Column(SmallInteger, nullable=True) + nav_status = Column(SmallInteger, nullable=True, index=True) + name = Column(String(128), nullable=True) + callsign = Column(String(16), nullable=True) + vessel_type = Column(SmallInteger, nullable=True, index=True) + vessel_type_name = Column(String(64), nullable=True, index=True) + flag = Column(String(4), nullable=True, index=True) + length = Column(Float, nullable=True) + width = Column(Float, nullable=True) + draught = Column(Float, nullable=True) + imo = Column(BigInteger, nullable=True) + source = Column(String(100), nullable=False, index=True) + observed_at = Column(DateTime(timezone=True), nullable=False) + field_sources = Column(JSON, default=dict) + selected_reasons = Column(JSON, default=dict) + source_summary = Column(JSON, default=dict) + quality_flags = Column(JSON, default=list) + updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) + + __table_args__ = ( + Index("idx_vessel_current_bbox", "lon", "lat"), + Index("idx_vessel_current_observed", "observed_at"), + ) + + def to_dict(self) -> dict: + return { + "mmsi": self.mmsi, + "lat": self.lat, + "lon": self.lon, + "sog": self.sog, + "cog": self.cog, + "heading": self.heading, + "nav_status": self.nav_status, + "name": self.name, + "callsign": self.callsign, + "vessel_type": self.vessel_type, + "vessel_type_name": self.vessel_type_name, + "flag": self.flag, + "length": self.length, + "width": self.width, + "draught": self.draught, + "imo": self.imo, + "source": self.source, + "received_at": self.observed_at, + "field_sources": self.field_sources or {}, + "selected_reasons": self.selected_reasons or {}, + "source_summary": self.source_summary or {}, + "quality_flags": self.quality_flags or [], + } + + class AISRawObservation(Base): """Source-level AIS fact before aggregation and conflict resolution.""" @@ -165,7 +227,7 @@ class AISSourceHealth(Base): __tablename__ = "ais_source_health" source = Column(String(100), primary_key=True) - connection_state = Column(String(32), nullable=False, default="disconnected", index=True) + connection_state = Column(String(32), nullable=False, default=ConnectionState.DISCONNECTED.value, index=True) last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True) last_success_at = Column(DateTime(timezone=True), nullable=True, index=True) last_error = Column(String(500), nullable=True) diff --git a/backend/app/schemas/ai.py b/backend/app/schemas/ai.py index 037354f3..938a6820 100644 --- a/backend/app/schemas/ai.py +++ b/backend/app/schemas/ai.py @@ -2,6 +2,7 @@ from typing import Any from pydantic import BaseModel, Field +from app.core.enums import PlaygroundMessageKind, PlaygroundMessageRole, PlaygroundMessageStatus class AIContentBlock(BaseModel): type: str @@ -110,9 +111,9 @@ class PlaygroundSessionUpsertRequest(BaseModel): class PlaygroundMessageRecord(BaseModel): id: str - role: str - kind: str = "message" - status: str = "done" + role: PlaygroundMessageRole + kind: PlaygroundMessageKind = PlaygroundMessageKind.MESSAGE + status: PlaygroundMessageStatus = PlaygroundMessageStatus.DONE title: str | None = None content: str = "" thinking_content: str = "" diff --git a/backend/app/schemas/user.py b/backend/app/schemas/user.py index eaa0c272..34cd43f5 100644 --- a/backend/app/schemas/user.py +++ b/backend/app/schemas/user.py @@ -3,6 +3,7 @@ from typing import Optional from pydantic import BaseModel, EmailStr, Field +from app.core.enums import OtpPurpose, UserRole class UserBase(BaseModel): username: str @@ -11,13 +12,13 @@ class UserBase(BaseModel): class UserCreate(UserBase): password: str = Field(..., min_length=8) - role: str = "viewer" + role: UserRole = UserRole.VIEWER gatekeeper_groups: list[str] = Field(default_factory=list) class UserUpdate(BaseModel): email: Optional[EmailStr] = None - role: Optional[str] = None + role: Optional[UserRole] = None gatekeeper_groups: Optional[list[str]] = None is_active: Optional[bool] = None @@ -59,7 +60,7 @@ class VerifyEmailRequest(BaseModel): class ResendCodeRequest(BaseModel): email: EmailStr - purpose: str = Field(default="register", pattern="^(register|verify_email|reset_password)$") + purpose: OtpPurpose = OtpPurpose.REGISTER class ForgotPasswordRequest(BaseModel): diff --git a/backend/app/services/ai_client.py b/backend/app/services/ai_client.py index f648af3b..49644e57 100644 --- a/backend/app/services/ai_client.py +++ b/backend/app/services/ai_client.py @@ -2,18 +2,24 @@ from __future__ import annotations import asyncio import json +from time import perf_counter import httpx from fastapi import Depends, HTTPException, status from sqlalchemy.ext.asyncio import AsyncSession from app.core.config import settings +from app.core.logging import get_logger from app.db.session import get_db from app.schemas.ai import ( AIProviderStatusResponse, SituationalAnalysisRequest, SituationalAnalysisResponse, ) +from app.services.business_logs import emit_business_log, exception_context + + +logger = get_logger(__name__, service="ai") class AIProviderClient: @@ -64,7 +70,19 @@ class AIProviderClient: return headers async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse: + context = self._base_log_context(operation="status") if not self.service_url: + await emit_business_log( + logger, + event="ai.provider.status.failed", + message="AI provider status skipped because service URL is not configured", + category="ai", + level="warning", + service="ai", + module=__name__, + request_id=request_id, + context={**context, "status": "unconfigured"}, + ) return AIProviderStatusResponse( provider="unconfigured", enabled=False, @@ -73,27 +91,133 @@ class AIProviderClient: base_url=None, ) - data = await self._request("GET", "/v1/provider/status", request_id=request_id) - return AIProviderStatusResponse.model_validate(data) + started_at = perf_counter() + await emit_business_log( + logger, + event="ai.provider.status.start", + message="AI provider status request started", + category="ai", + service="ai", + module=__name__, + request_id=request_id, + context=context, + ) + try: + data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status") + result = AIProviderStatusResponse.model_validate(data) + await emit_business_log( + logger, + event="ai.provider.status.success", + message="AI provider status request completed", + category="ai", + service="ai", + module=__name__, + request_id=request_id, + context={ + **context, + "status": "success", + "duration_ms": self._duration_ms(started_at), + "result_provider": result.provider, + "result_model": result.model, + "configured": result.configured, + "enabled": result.enabled, + }, + ) + return result + except Exception as exc: + await emit_business_log( + logger, + event="ai.provider.status.failed", + message="AI provider status request failed", + category="ai", + level="error", + service="ai", + module=__name__, + request_id=request_id, + context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}), + ) + raise async def analyze( self, payload: SituationalAnalysisRequest, request_id: str | None = None, ) -> SituationalAnalysisResponse: + context = self._base_log_context( + operation="analyze", + preferred_model=payload.preferred_model, + input_summary=self._summarize_analysis_payload(payload), + ) if not self.service_url: + await emit_business_log( + logger, + event="ai.provider.analyze.failed", + message="AI provider analyze skipped because service URL is not configured", + category="ai", + level="warning", + service="ai", + module=__name__, + request_id=request_id, + context={**context, "status": "unconfigured"}, + ) raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="AI provider service URL is not configured.", ) - data = await self._request( - "POST", - "/v1/analyze", - json=payload.model_dump(), + started_at = perf_counter() + await emit_business_log( + logger, + event="ai.provider.analyze.start", + message="AI provider analyze request started", + category="ai", + service="ai", + module=__name__, request_id=request_id, + context=context, ) - return SituationalAnalysisResponse.model_validate(data) + try: + data = await self._request( + "POST", + "/v1/analyze", + json=payload.model_dump(), + request_id=request_id, + operation="analyze", + payload_summary=context["input_summary"], + ) + result = SituationalAnalysisResponse.model_validate(data) + await emit_business_log( + logger, + event="ai.provider.analyze.success", + message="AI provider analyze request completed", + category="ai", + service="ai", + module=__name__, + request_id=request_id, + context={ + **context, + "status": "success", + "duration_ms": self._duration_ms(started_at), + "result_provider": result.provider, + "result_model": result.model, + "content_block_count": len(result.content_blocks or []), + "thinking_block_count": len(result.thinking_blocks or []), + }, + ) + return result + except Exception as exc: + await emit_business_log( + logger, + event="ai.provider.analyze.failed", + message="AI provider analyze request failed", + category="ai", + level="error", + service="ai", + module=__name__, + request_id=request_id, + context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}), + ) + raise async def _request( self, @@ -101,9 +225,12 @@ class AIProviderClient: path: str, json: dict | None = None, request_id: str | None = None, + operation: str = "request", + payload_summary: dict | None = None, ) -> dict: last_error: Exception | None = None for attempt in range(1, self.retry_attempts + 1): + attempt_started_at = perf_counter() try: async with httpx.AsyncClient(timeout=self.timeout) as client: response = await client.request( @@ -117,6 +244,15 @@ class AIProviderClient: except httpx.HTTPStatusError as exc: last_error = exc if attempt < self.retry_attempts and exc.response.status_code >= 500: + await self._log_retry( + operation=operation, + request_id=request_id, + attempt=attempt, + status_code=exc.response.status_code, + duration_ms=self._duration_ms(attempt_started_at), + error=exc, + payload_summary=payload_summary, + ) await asyncio.sleep(0.3 * attempt) continue detail = exc.response.text or "AI provider service returned an error" @@ -127,6 +263,14 @@ class AIProviderClient: except httpx.HTTPError as exc: last_error = exc if attempt < self.retry_attempts: + await self._log_retry( + operation=operation, + request_id=request_id, + attempt=attempt, + duration_ms=self._duration_ms(attempt_started_at), + error=exc, + payload_summary=payload_summary, + ) await asyncio.sleep(0.3 * attempt) continue raise HTTPException( @@ -139,6 +283,71 @@ class AIProviderClient: detail=f"AI provider service request failed: {last_error}", ) + def _base_log_context(self, **extra: object) -> dict: + llm_provider_apis = self.llm_config.get("model_provider_apis") + return { + "provider": self.llm_config.get("provider") or "", + "provider_api": self.llm_config.get("provider_api") or "", + "model": self.llm_config.get("model") or "", + "base_url_configured": bool(self.llm_config.get("base_url")), + "service_url_configured": bool(self.service_url), + "timeout_seconds": self.timeout, + "retry_attempts": self.retry_attempts, + "model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0, + **extra, + } + + @staticmethod + def _duration_ms(started_at: float) -> int: + return int((perf_counter() - started_at) * 1000) + + @staticmethod + def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict: + context = payload.context if isinstance(payload.context, dict) else {} + thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking + return { + "title_length": len(payload.title or ""), + "objective_length": len(payload.objective or ""), + "observation_count": len(payload.observations or []), + "constraint_count": len(payload.constraints or []), + "has_system_prompt": bool(payload.system_prompt), + "thinking_enabled": bool(thinking), + "context_keys": sorted(str(key) for key in context.keys()), + } + + async def _log_retry( + self, + *, + operation: str, + request_id: str | None, + attempt: int, + duration_ms: int, + error: BaseException, + status_code: int | None = None, + payload_summary: dict | None = None, + ) -> None: + await emit_business_log( + logger, + event=f"ai.provider.{operation}.retry", + message="AI provider request will retry", + category="ai", + level="warning", + service="ai", + module=__name__, + request_id=request_id, + context=exception_context( + error, + { + **self._base_log_context(operation=operation), + "attempt": attempt, + "next_attempt": attempt + 1, + "status_code": status_code, + "duration_ms": duration_ms, + "input_summary": payload_summary, + }, + ), + ) + async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient: from app.api.v1.settings import get_runtime_ai_provider_config diff --git a/backend/app/services/ai_tools/web_fetch.py b/backend/app/services/ai_tools/web_fetch.py index 53f69cc4..e490dfe7 100644 --- a/backend/app/services/ai_tools/web_fetch.py +++ b/backend/app/services/ai_tools/web_fetch.py @@ -1,11 +1,18 @@ from __future__ import annotations import hashlib +from time import perf_counter +from urllib.parse import urlparse import httpx from bs4 import BeautifulSoup +from app.core.logging import get_logger from app.services.ai_tools.schemas import FetchedEvidence +from app.services.business_logs import emit_business_log, exception_context + + +logger = get_logger(__name__, service="ai_tool") class WebFetchError(RuntimeError): @@ -29,8 +36,33 @@ async def fetch_url_evidence( timeout_seconds: int = 20, max_bytes: int = 1_500_000, ) -> FetchedEvidence: + started_at = perf_counter() if not url: + await emit_business_log( + logger, + event="ai_tool.web_fetch.failed", + message="WebFetch failed because URL is empty", + category="ai_tool", + level="warning", + service="ai_tool", + module=__name__, + context={"reason": "empty_url"}, + ) raise WebFetchError("url is required") + request_host = urlparse(url).netloc + await emit_business_log( + logger, + event="ai_tool.web_fetch.start", + message="WebFetch request started", + category="ai_tool", + service="ai_tool", + module=__name__, + context={ + "url_host": request_host, + "timeout_seconds": timeout_seconds, + "max_bytes": max_bytes, + }, + ) try: async with httpx.AsyncClient( timeout=timeout_seconds, @@ -41,10 +73,45 @@ async def fetch_url_evidence( response.raise_for_status() content = response.content[:max_bytes] except httpx.HTTPError as exc: + await emit_business_log( + logger, + event="ai_tool.web_fetch.failed", + message="WebFetch request failed", + category="ai_tool", + level="error", + service="ai_tool", + module=__name__, + context=exception_context( + exc, + { + "url_host": request_host, + "status": "failed", + "duration_ms": int((perf_counter() - started_at) * 1000), + }, + ), + ) raise WebFetchError(f"failed to fetch page: {exc}") from exc title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore")) content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest() + await emit_business_log( + logger, + event="ai_tool.web_fetch.success", + message="WebFetch request completed", + category="ai_tool", + service="ai_tool", + module=__name__, + context={ + "url_host": request_host, + "final_url_host": urlparse(str(response.url)).netloc, + "status": "success", + "status_code": response.status_code, + "bytes_read": len(content), + "content_hash": content_hash, + "duration_ms": int((perf_counter() - started_at) * 1000), + "extractor": "beautifulsoup_basic", + }, + ) return FetchedEvidence( url=url, final_url=str(response.url), @@ -53,4 +120,3 @@ async def fetch_url_evidence( content_hash=content_hash, extractor="beautifulsoup_basic", ) - diff --git a/backend/app/services/ai_tools/web_search.py b/backend/app/services/ai_tools/web_search.py index 1ab13dd0..efad9946 100644 --- a/backend/app/services/ai_tools/web_search.py +++ b/backend/app/services/ai_tools/web_search.py @@ -1,11 +1,18 @@ from __future__ import annotations from copy import deepcopy +import hashlib +from time import perf_counter from typing import Any import httpx +from app.core.logging import get_logger from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig +from app.services.business_logs import emit_business_log, exception_context + + +logger = get_logger(__name__, service="ai_tool") WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = { @@ -120,29 +127,108 @@ class WebSearchClient: domains: list[str] | None = None, freshness_days: int | None = None, ) -> list[SearchEvidence]: + started_at = perf_counter() if not self.config.enabled: + await emit_business_log( + logger, + event="ai_tool.web_search.unavailable", + message="WebSearch skipped because integration is disabled", + category="ai_tool", + level="warning", + service="ai_tool", + module=__name__, + context={"provider": self.config.default_provider, "reason": "disabled"}, + ) raise WebSearchConfigurationError("WebSearch is disabled.") provider_config = self.config.active_provider_config provider = normalize_web_search_provider(provider_config.provider) if provider != "searxng" and not provider_config.api_key: + await emit_business_log( + logger, + event="ai_tool.web_search.unavailable", + message="WebSearch skipped because API key is not configured", + category="ai_tool", + level="warning", + service="ai_tool", + module=__name__, + context={"provider": provider, "reason": "missing_api_key"}, + ) raise WebSearchConfigurationError(f"{provider} API key is not configured.") query = " ".join(str(query or "").split()) if not query: + await emit_business_log( + logger, + event="ai_tool.web_search.failed", + message="WebSearch failed because query is empty", + category="ai_tool", + level="warning", + service="ai_tool", + module=__name__, + context={"provider": provider, "reason": "empty_query"}, + ) raise WebSearchConfigurationError("search query is required.") limit = max_results or provider_config.max_results - if provider == "tavily": - return await self._search_tavily(provider_config, query, limit, domains, freshness_days) - if provider == "brave": - return await self._search_brave(provider_config, query, limit, domains) - if provider == "serpapi": - return await self._search_serpapi(provider_config, query, limit) - if provider == "exa": - return await self._search_exa(provider_config, query, limit, domains) - if provider == "firecrawl": - return await self._search_firecrawl(provider_config, query, limit) - if provider == "searxng": - return await self._search_searxng(provider_config, query, limit, domains) - raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}") + context = { + "provider": provider, + "query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(), + "query_length": len(query), + "max_results": limit, + "domain_count": len(domains or []), + "freshness_days": freshness_days, + } + await emit_business_log( + logger, + event="ai_tool.web_search.start", + message="WebSearch request started", + category="ai_tool", + service="ai_tool", + module=__name__, + context=context, + ) + try: + if provider == "tavily": + results = await self._search_tavily(provider_config, query, limit, domains, freshness_days) + elif provider == "brave": + results = await self._search_brave(provider_config, query, limit, domains) + elif provider == "serpapi": + results = await self._search_serpapi(provider_config, query, limit) + elif provider == "exa": + results = await self._search_exa(provider_config, query, limit, domains) + elif provider == "firecrawl": + results = await self._search_firecrawl(provider_config, query, limit) + elif provider == "searxng": + results = await self._search_searxng(provider_config, query, limit, domains) + else: + raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}") + event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty" + await emit_business_log( + logger, + event=event, + message="WebSearch request completed" if results else "WebSearch returned no results", + category="ai_tool", + level="info" if results else "warning", + service="ai_tool", + module=__name__, + context={ + **context, + "status": "success" if results else "empty", + "result_count": len(results), + "duration_ms": int((perf_counter() - started_at) * 1000), + }, + ) + return results + except Exception as exc: + await emit_business_log( + logger, + event="ai_tool.web_search.failed", + message="WebSearch request failed", + category="ai_tool", + level="error", + service="ai_tool", + module=__name__, + context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}), + ) + raise async def test_connection(self) -> list[SearchEvidence]: return await self.search("Planet WebSearch connectivity test", max_results=1) @@ -388,4 +474,3 @@ def _float_or_none(value: Any) -> float | None: return float(value) except (TypeError, ValueError): return None - diff --git a/backend/app/services/bgp_detectors.py b/backend/app/services/bgp_detectors.py index c1263ff8..f501ed69 100644 --- a/backend/app/services/bgp_detectors.py +++ b/backend/app/services/bgp_detectors.py @@ -6,6 +6,7 @@ from collections import Counter, defaultdict from datetime import UTC, datetime from typing import Any +from app.core.enums import BGPStatus from app.models.bgp_anomaly import BGPAnomaly @@ -127,7 +128,7 @@ def detect_origin_change_anomalies( source=source, anomaly_type=anomaly_type, severity=severity, - status="active", + status=BGPStatus.ACTIVE.value, entity_key=f"{anomaly_type}:{prefix}:{new_origin}", prefix=prefix, origin_asn=sorted(historic)[0] if historic else None, @@ -197,7 +198,7 @@ def detect_more_specific_burst_anomalies( source=source, anomaly_type="more_specific_burst", severity="high", - status="active", + status=BGPStatus.ACTIVE.value, entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}", prefix=sample.get("prefix"), origin_asn=sample.get("origin_asn"), @@ -267,7 +268,7 @@ def detect_mass_withdrawal_anomalies( source=source, anomaly_type="mass_withdrawal", severity=severity, - status="active", + status=BGPStatus.ACTIVE.value, entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}", prefix=prefix, origin_asn=origin_asn, @@ -354,7 +355,7 @@ def detect_route_leak_anomalies( source=source, anomaly_type="route_leak_candidate", severity="high" if max_path_length >= dominant_length + 3 else "medium", - status="active", + status=BGPStatus.ACTIVE.value, entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}", prefix=prefix, origin_asn=sample_metadata.get("origin_asn"), @@ -435,7 +436,7 @@ def detect_path_flap_anomalies( source=source, anomaly_type="path_flap", severity=severity, - status="active", + status=BGPStatus.ACTIVE.value, entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}", prefix=prefix, origin_asn=sample_metadata.get("origin_asn"), diff --git a/backend/app/services/bgp_incidents.py b/backend/app/services/bgp_incidents.py index 91e454f1..af15c2ce 100644 --- a/backend/app/services/bgp_incidents.py +++ b/backend/app/services/bgp_incidents.py @@ -9,6 +9,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from app.core.collected_data_fields import get_record_field +from app.core.enums import BGPStatus from app.models.bgp_anomaly import BGPAnomaly from app.models.bgp_incident import BGPIncident from app.models.collected_data import CollectedData @@ -290,7 +291,7 @@ async def create_bgp_incidents_for_anomalies( existing.title = title existing.summary = summary existing.severity = severity - existing.status = "active" + existing.status = BGPStatus.ACTIVE.value existing.confidence = confidence existing.started_at = primary.started_at or existing.started_at or datetime.now(UTC) existing.ended_at = None @@ -313,7 +314,7 @@ async def create_bgp_incidents_for_anomalies( title=title, summary=summary, severity=severity, - status="active", + status=BGPStatus.ACTIVE.value, confidence=confidence, started_at=primary.started_at or datetime.now(UTC), affected_prefixes=prefixes, diff --git a/backend/app/services/business_logs.py b/backend/app/services/business_logs.py new file mode 100644 index 00000000..4fd5fcac --- /dev/null +++ b/backend/app/services/business_logs.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +import asyncio +from collections.abc import Mapping +from typing import Any + +from app.core.logging import PlanetLoggerAdapter, sanitize_log_value +from app.core.request_context import get_request_id +from app.services.persistent_logs import record_system_log + + +LEVEL_METHODS = { + "debug": "debug_event", + "info": "info_event", + "warning": "warning_event", + "error": "error_event", +} + + +def normalize_business_level(level: str | None) -> str: + normalized = str(level or "info").strip().lower() + if normalized in {"warn", "warning"}: + return "warning" + if normalized in {"err", "error", "critical", "fatal"}: + return "error" + if normalized == "debug": + return "debug" + return "info" + + +def build_business_context( + context: Mapping[str, Any] | None = None, + **fields: Any, +) -> dict[str, Any]: + payload = dict(context or {}) + for key, value in fields.items(): + if value is not None: + payload[key] = value + return sanitize_log_value(payload) + + +async def emit_business_log( + logger: PlanetLoggerAdapter, + *, + event: str, + message: str, + category: str, + level: str = "info", + source: str = "backend", + service: str | None = None, + module: str | None = None, + request_id: str | None = None, + user_id: int | None = None, + context: Mapping[str, Any] | None = None, +) -> None: + normalized_level = normalize_business_level(level) + safe_context = build_business_context(context) + log_method = getattr(logger, LEVEL_METHODS[normalized_level]) + log_method(message, event=event, context=safe_context) + await record_system_log( + source=source, + level=normalized_level, + message=message, + service=service, + module=module, + event=event, + request_id=request_id or get_request_id(), + user_id=user_id, + category=category, + context=safe_context, + ) + + +def emit_business_log_background( + logger: PlanetLoggerAdapter, + *, + event: str, + message: str, + category: str, + level: str = "info", + source: str = "backend", + service: str | None = None, + module: str | None = None, + request_id: str | None = None, + user_id: int | None = None, + context: Mapping[str, Any] | None = None, +) -> None: + normalized_level = normalize_business_level(level) + safe_context = build_business_context(context) + log_method = getattr(logger, LEVEL_METHODS[normalized_level]) + log_method(message, event=event, context=safe_context) + try: + loop = asyncio.get_running_loop() + except RuntimeError: + return + loop.create_task( + record_system_log( + source=source, + level=normalized_level, + message=message, + service=service, + module=module, + event=event, + request_id=request_id or get_request_id(), + user_id=user_id, + category=category, + context=safe_context, + ) + ) + + +def exception_context(exc: BaseException, context: Mapping[str, Any] | None = None) -> dict[str, Any]: + return build_business_context( + context, + error_type=type(exc).__name__, + error=str(exc), + ) diff --git a/backend/app/services/collectors/aisstream.py b/backend/app/services/collectors/aisstream.py index eb9956f1..78cb78c9 100644 --- a/backend/app/services/collectors/aisstream.py +++ b/backend/app/services/collectors/aisstream.py @@ -322,6 +322,21 @@ class AISStreamCollector(BaseCollector): last_success_at=now if data else None, lag_seconds=max((now - latest_observed_at).total_seconds(), 0), ) + if snapshot_id is not None: + from app.models.data_snapshot import DataSnapshot + + snapshot = await db.get(DataSnapshot, snapshot_id) + if snapshot: + snapshot.record_count = records_added + snapshot.status = "success" + snapshot.completed_at = now + snapshot.summary = { + "created": records_added, + "updated": 0, + "unchanged": 0, + "deleted": 0, + "storage": "ais_raw_observations", + } await db.commit() await self.update_progress(records_added, force=True) return records_added diff --git a/backend/app/services/collectors/base.py b/backend/app/services/collectors/base.py index 2dcb090d..81627290 100644 --- a/backend/app/services/collectors/base.py +++ b/backend/app/services/collectors/base.py @@ -4,42 +4,22 @@ import asyncio from abc import ABC, abstractmethod from typing import Dict, List, Any, Optional from datetime import UTC, datetime +from time import perf_counter +from urllib.parse import urlparse import httpx from sqlalchemy import select, text from sqlalchemy.ext.asyncio import AsyncSession from app.core.collected_data_fields import build_dynamic_metadata, get_record_field from app.core.countries import normalize_country +from app.core.enums import JobStatus, SnapshotStatus +from app.core.logging import get_logger from app.core.time import to_iso8601_utc from app.core.websocket.broadcaster import broadcaster -from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source - -EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = { - "ris_live_bgp": ["bgp"], - "bgpstream_bgp": ["bgp"], - "top500_supercomputers": ["computeCenters"], - "epoch_ai_gpu": ["computeCenters"], - "huggingface_models": ["computeCenters"], - "huggingface_datasets": ["computeCenters"], - "huggingface_spaces": ["computeCenters"], - "telegeography_cables": ["cables"], - "telegeography_landing_points": ["cables"], - "telegeography_cable_systems": ["cables"], - "arcgis_cables": ["cables"], - "fao_landing_points": ["cables"], - "arcgis_landing_points": ["cables"], - "arcgis_cable_landing_relations": ["cables"], - "spacetrack_tle": ["satellites"], - "celestrak_tle": ["satellites"], - "barentswatch_vessels": ["vessels"], - "aisstream_vessels": ["vessels"], - "news_live_streams": ["media"], - "media_news_archive": ["news"], -} +from app.services.business_logs import emit_business_log, exception_context -def get_earth_update_layers_for_source(source: str) -> list[str]: - return EARTH_UPDATE_LAYER_HINTS.get(source, []) +logger = get_logger(__name__, service="collector") class BaseCollector(ABC): @@ -58,6 +38,7 @@ class BaseCollector(ABC): self._datasource_id = 1 self._resolved_url: Optional[str] = None self._last_broadcast_progress: Optional[int] = None + self._last_save_summary: dict[str, int] = {} async def resolve_url(self, db: AsyncSession) -> None: from app.core.data_sources import get_data_sources_config @@ -96,29 +77,6 @@ class BaseCollector(ABC): ) self._last_broadcast_progress = rounded_progress - async def _publish_earth_update( - self, - *, - action: str, - records_processed: int, - task_id: int | None = None, - ) -> None: - layers = get_earth_update_layers_for_source(self.name) - if not layers: - return - await broadcaster.broadcast_earth_update( - { - "action": action, - "source": self.name, - "data_type": self.data_type, - "layers": layers, - "datasource_id": getattr(self, "_datasource_id", None), - "task_id": task_id, - "records_processed": records_processed, - "timestamp": to_iso8601_utc(datetime.now(UTC)), - } - ) - async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False): """Update task progress - call this during data processing""" if self._current_task and self._db_session: @@ -280,7 +238,7 @@ class BaseCollector(ABC): snapshot = await db.get(DataSnapshot, snapshot_id) if snapshot: parent_snapshot_id = snapshot.parent_snapshot_id - snapshot.status = "cancelled" + snapshot.status = SnapshotStatus.CANCELLED.value snapshot.is_current = False snapshot.completed_at = datetime.now(UTC) summary = dict(snapshot.summary or {}) @@ -322,19 +280,39 @@ class BaseCollector(ABC): from app.models.data_snapshot import DataSnapshot start_time = datetime.now(UTC) + started_at = perf_counter() datasource_id = getattr(self, "_datasource_id", 1) snapshot_id: Optional[int] = None if not collector_registry.is_active(self.name): + await self._log_collection_event( + "collector.run.skipped_disabled", + "Collector skipped because it is disabled", + level="info", + context={"status": "skipped", "reason": "disabled"}, + ) return {"status": "skipped", "reason": "Collector is disabled"} - task = CollectionTask( - datasource_id=datasource_id, - status="running", - phase="queued", - started_at=start_time, - ) - db.add(task) + task = self._current_task if isinstance(self._current_task, CollectionTask) else None + if task is None: + task = CollectionTask( + datasource_id=datasource_id, + source=self.name, + task_type="collect", + status="running", + phase="queued", + started_at=start_time, + ) + db.add(task) + else: + task.datasource_id = datasource_id + task.source = task.source or self.name + task.task_type = task.task_type or "collect" + task.status = JobStatus.RUNNING.value + task.phase = "queued" + task.started_at = task.started_at or start_time + task.completed_at = None + task.error_message = None await db.commit() task_id = task.id @@ -344,25 +322,78 @@ class BaseCollector(ABC): await self.resolve_url(db) await self._publish_task_update(force=True) + await self._log_collection_event( + "collector.run.started", + "Collector run started", + context={"status": "running", "task_id": task_id}, + ) try: + phase_started_at = perf_counter() await self.set_phase("fetching", message="正在拉取原始数据") + await self._log_collection_event( + "collector.phase.fetching.start", + "Collector fetch phase started", + context={"task_id": task_id, "snapshot_id": snapshot_id}, + ) raw_data = await self.fetch() task.total_records = len(raw_data) await db.commit() await self._publish_task_update(force=True) + await self._log_collection_event( + "collector.phase.fetching.success", + "Collector fetch phase completed", + context={ + "task_id": task_id, + "raw_count": len(raw_data), + "duration_ms": self._duration_ms(phase_started_at), + }, + ) if self.fail_on_empty and not raw_data: raise RuntimeError(f"Collector {self.name} returned no data") + phase_started_at = perf_counter() await self.set_phase("transforming", message="正在转换采集数据") + await self._log_collection_event( + "collector.phase.transforming.start", + "Collector transform phase started", + context={"task_id": task_id, "raw_count": len(raw_data)}, + ) data = self.transform(raw_data) + await self._log_collection_event( + "collector.phase.transforming.success", + "Collector transform phase completed", + context={ + "task_id": task_id, + "raw_count": len(raw_data), + "transformed_count": len(data), + "duration_ms": self._duration_ms(phase_started_at), + }, + ) snapshot_id = await self._create_snapshot(db, task_id, data, start_time) + phase_started_at = perf_counter() await self.set_phase("saving", message="正在保存采集数据") + await self._log_collection_event( + "collector.phase.saving.start", + "Collector save phase started", + context={"task_id": task_id, "snapshot_id": snapshot_id, "transformed_count": len(data)}, + ) records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id) + await self._log_collection_event( + "collector.phase.saving.success", + "Collector save phase completed", + context={ + "task_id": task_id, + "snapshot_id": snapshot_id, + "saved_count": records_count, + **self._last_save_summary, + "duration_ms": self._duration_ms(phase_started_at), + }, + ) - task.status = "success" + task.status = JobStatus.SUCCESS.value task.phase = "completed" task.phase_progress = 100.0 task.phase_message = "采集完成" @@ -374,10 +405,19 @@ class BaseCollector(ABC): task.completed_at = datetime.now(UTC) await db.commit() await self._publish_task_update(force=True) - await self._publish_earth_update( - action="collector_completed", - records_processed=records_count, - task_id=task_id, + await self._log_collection_event( + "collector.run.completed", + "Collector run completed", + context={ + "status": "success", + "task_id": task_id, + "snapshot_id": snapshot_id, + "raw_count": len(raw_data), + "transformed_count": len(data), + "saved_count": records_count, + **self._last_save_summary, + "duration_ms": self._duration_ms(started_at), + }, ) return { @@ -388,7 +428,7 @@ class BaseCollector(ABC): } except asyncio.CancelledError: await db.rollback() - task.status = "cancelled" + task.status = JobStatus.CANCELLED.value task.phase = "cancelled" task.phase_message = "采集已取消" task.error_message = "Collection cancelled by operator and rolled back" @@ -402,10 +442,21 @@ class BaseCollector(ABC): ) await db.commit() await self._publish_task_update(force=True) + await self._log_collection_event( + "collector.run.cancelled", + "Collector run cancelled", + level="warning", + context={ + "status": "cancelled", + "task_id": task_id, + "snapshot_id": snapshot_id, + "duration_ms": self._duration_ms(started_at), + }, + ) raise except Exception as e: await db.rollback() - task.status = "failed" + task.status = JobStatus.FAILED.value task.phase = "failed" task.phase_message = str(e) task.error_message = str(e) @@ -413,11 +464,25 @@ class BaseCollector(ABC): if snapshot_id is not None: snapshot = await db.get(DataSnapshot, snapshot_id) if snapshot: - snapshot.status = "failed" + snapshot.status = SnapshotStatus.FAILED.value snapshot.completed_at = datetime.now(UTC) snapshot.summary = {"error": str(e)} await db.commit() await self._publish_task_update(force=True) + await self._log_collection_event( + "collector.run.failed", + "Collector run failed", + level="error", + context=exception_context( + e, + { + "status": "failed", + "task_id": task_id, + "snapshot_id": snapshot_id, + "duration_ms": self._duration_ms(started_at), + }, + ), + ) return { "status": "failed", @@ -438,12 +503,13 @@ class BaseCollector(ABC): from app.models.data_snapshot import DataSnapshot if not data: + self._last_save_summary = {"created": 0, "updated": 0, "unchanged": 0, "deleted": 0} if snapshot_id is not None: snapshot = await db.get(DataSnapshot, snapshot_id) if snapshot: snapshot.record_count = 0 snapshot.summary = {"created": 0, "updated": 0, "unchanged": 0} - snapshot.status = "success" + snapshot.status = SnapshotStatus.SUCCESS.value snapshot.completed_at = datetime.now(UTC) await db.commit() return 0 @@ -576,7 +642,7 @@ class BaseCollector(ABC): snapshot = await db.get(DataSnapshot, snapshot_id) if snapshot: snapshot.record_count = records_added - snapshot.status = "success" + snapshot.status = SnapshotStatus.SUCCESS.value snapshot.completed_at = datetime.now(UTC) snapshot.summary = { "created": created_count, @@ -584,12 +650,51 @@ class BaseCollector(ABC): "unchanged": unchanged_count, "deleted": len(deleted_keys), } + self._last_save_summary = { + "created": created_count, + "updated": updated_count, + "unchanged": unchanged_count, + "deleted": len(deleted_keys), + } + else: + self._last_save_summary = { + "created": created_count, + "updated": updated_count, + "unchanged": unchanged_count, + "deleted": 0, + } await db.commit() - invalidate_earth_layer_cache_for_source(self.name) await self.update_progress(len(data), force=True) return records_added + @staticmethod + def _duration_ms(started_at: float) -> int: + return int((perf_counter() - started_at) * 1000) + + async def _log_collection_event( + self, + event: str, + message: str, + *, + level: str = "info", + context: Dict[str, Any] | None = None, + ) -> None: + await emit_business_log( + logger, + event=event, + message=message, + category="collector", + level=level, + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + **(context or {}), + }, + ) + async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int: """Save data to database (legacy method, use _save_data instead)""" return await self._save_data(db, data) @@ -602,10 +707,65 @@ class HTTPCollector(BaseCollector): headers: Dict[str, str] = {} async def fetch(self) -> List[Dict[str, Any]]: + started_at = perf_counter() + request_host = urlparse(self.base_url).netloc + await emit_business_log( + logger, + event="collector.http.fetch.start", + message="Collector HTTP request started", + category="collector", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "url_host": request_host, + }, + ) async with httpx.AsyncClient(timeout=60.0) as client: - response = await client.get(self.base_url, headers=self.headers) - response.raise_for_status() - return self.parse_response(response.json()) + try: + response = await client.get(self.base_url, headers=self.headers) + response.raise_for_status() + payload = response.json() + parsed = self.parse_response(payload) + await emit_business_log( + logger, + event="collector.http.fetch.success", + message="Collector HTTP request completed", + category="collector", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "url_host": request_host, + "status_code": response.status_code, + "response_bytes": len(response.content or b""), + "parsed_count": len(parsed), + "duration_ms": BaseCollector._duration_ms(started_at), + }, + ) + return parsed + except Exception as exc: + await emit_business_log( + logger, + event="collector.http.fetch.failed", + message="Collector HTTP request failed", + category="collector", + level="error", + service="collector", + module=__name__, + context=exception_context( + exc, + { + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "url_host": request_host, + "duration_ms": BaseCollector._duration_ms(started_at), + }, + ), + ) + raise @abstractmethod def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]: diff --git a/backend/app/services/collectors/celestrak.py b/backend/app/services/collectors/celestrak.py index 49038b42..9a7dded9 100644 --- a/backend/app/services/collectors/celestrak.py +++ b/backend/app/services/collectors/celestrak.py @@ -1,15 +1,45 @@ -"""CelesTrak TLE Collector +"""CelesTrak TLE Collector. -Collects satellite TLE (Two-Line Element) data from CelesTrak.org. -Free, no authentication required. +Collects the full active satellite GP element set from CelesTrak. """ +import asyncio import json -from typing import Dict, Any, List +from pathlib import Path +from time import perf_counter +from typing import Any, Dict, List +from urllib.parse import urlencode, urlparse + import httpx +from app.core.logging import get_logger from app.core.satellite_tle import build_tle_lines_from_elements +from app.services.business_logs import emit_business_log, exception_context from app.services.collectors.base import BaseCollector +from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader + + +logger = get_logger(__name__, service="collector") +ACTIVE_GROUP = "active" +FALLBACK_GROUPS = ( + "starlink", + "gps-ops", + "galileo", + "glo-ops", + "beidou", + "geo", + "iridium-next", + "stations", + "visual", + "weather", + "science", + "cubesat", + "amateur", + "last-30-days", +) +FETCH_RETRY_ATTEMPTS = 3 +FETCH_RETRY_BASE_DELAY_SECONDS = 0.8 +CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful" class CelesTrakTLECollector(BaseCollector): @@ -18,55 +48,360 @@ class CelesTrakTLECollector(BaseCollector): module = "L3" frequency_hours = 24 data_type = "satellite_tle" + _downloader = ResumableFileDownloader( + cache_namespace="celestrak", + default_accept="application/json", + ) @property def base_url(self) -> str: return self._resolved_url or "" + def _active_url(self) -> str: + return self._group_url(ACTIVE_GROUP) + + def _group_url(self, group: str) -> str: + if not self.base_url: + raise RuntimeError("CelesTrak base URL is not configured") + return f"{self.base_url}?{urlencode({'GROUP': group, 'FORMAT': 'json'})}" + async def fetch(self) -> List[Dict[str, Any]]: - satellite_groups = [ - "starlink", - "gps-ops", - "galileo", - "glonass", - "beidou", - "leo", - "geo", - "iridium-next", - ] + url = self._active_url() + last_error: Exception | None = None - all_satellites = [] - - async with httpx.AsyncClient(timeout=120.0) as client: - for group in satellite_groups: + async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client: + for attempt in range(1, FETCH_RETRY_ATTEMPTS + 1): + started_at = perf_counter() try: - url = f"{self.base_url}?GROUP={group}&FORMAT=json" - response = await client.get(url) + await emit_business_log( + logger, + event="collector.celestrak.download.start", + message="CelesTrak active satellite download started", + category="collector", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "group": ACTIVE_GROUP, + "attempt": attempt, + "url_host": urlparse(url).netloc, + }, + ) + body_path = await self._downloader.download_file( + client, + url, + extension=".json", + accept="application/json", + progress_callback=self._report_download_progress, + validate_existing=self._validate_json_file, + ) + data = await self._load_downloaded_payload(body_path, url) + await emit_business_log( + logger, + event="collector.celestrak.download.success", + message="CelesTrak active satellite download completed", + category="collector", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "group": ACTIVE_GROUP, + "attempt": attempt, + "record_count": len(data), + "duration_ms": self._duration_ms(started_at), + }, + ) + return data + except DownloadHTTPStatusError as exc: + if self._is_not_updated_response(exc): + cached_path = self._downloader.get_cached_file( + url, + ".json", + validate_existing=self._validate_json_file, + ) + if cached_path is not None: + data = await self._load_downloaded_payload(cached_path, url) + await emit_business_log( + logger, + event="collector.celestrak.download.cached_not_updated", + message="CelesTrak active satellite data has not changed; using cached download", + category="collector", + level="warning", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "group": ACTIVE_GROUP, + "attempt": attempt, + "record_count": len(data), + "duration_ms": self._duration_ms(started_at), + }, + ) + return data + await emit_business_log( + logger, + event="collector.celestrak.download.not_updated_no_cache", + message="CelesTrak active satellite data has not changed; trying fallback groups", + category="collector", + level="warning", + service="collector", + module=__name__, + context=exception_context( + exc, + { + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "group": ACTIVE_GROUP, + "attempt": attempt, + "duration_ms": self._duration_ms(started_at), + }, + ), + ) + return await self._fetch_fallback_groups(client, active_error=exc) + raise + except Exception as exc: + last_error = exc + is_final_attempt = attempt >= FETCH_RETRY_ATTEMPTS + await emit_business_log( + logger, + event=( + "collector.celestrak.download.failed" + if is_final_attempt + else "collector.celestrak.download.retry" + ), + message=( + "CelesTrak active satellite download failed" + if is_final_attempt + else "CelesTrak active satellite download will retry" + ), + category="collector", + level="error" if is_final_attempt else "warning", + service="collector", + module=__name__, + context=exception_context( + exc, + { + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "group": ACTIVE_GROUP, + "attempt": attempt, + "duration_ms": self._duration_ms(started_at), + }, + ), + ) + if not is_final_attempt: + await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt) - if response.status_code == 200: - data = response.json() - if isinstance(data, list): - for item in data: - if isinstance(item, dict): - item["_celestrak_group"] = group - all_satellites.extend(data) - print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'") - except Exception as e: - print(f"CelesTrak: Error fetching group '{group}': {e}") + raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}") - if not all_satellites: - return self._get_sample_data() + async def _fetch_fallback_groups( + self, + client: httpx.AsyncClient, + *, + active_error: DownloadHTTPStatusError, + ) -> List[Dict[str, Any]]: + started_at = perf_counter() + records_by_norad: dict[str, Dict[str, Any]] = {} + group_counts: dict[str, int] = {} - print(f"CelesTrak: Total satellites fetched: {len(all_satellites)}") + await emit_business_log( + logger, + event="collector.celestrak.fallback_groups.start", + message="CelesTrak fallback group download started", + category="collector", + level="warning", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "groups": list(FALLBACK_GROUPS), + "reason": "active_not_updated_without_cache", + }, + ) - # Return raw data - base.run() will call transform() - return all_satellites + try: + for group in FALLBACK_GROUPS: + group_url = self._group_url(group) + cached_path = self._downloader.get_cached_file( + group_url, + ".json", + validate_existing=self._validate_json_file, + ) + if cached_path is not None: + body_path = cached_path + else: + try: + body_path = await self._downloader.download_file( + client, + group_url, + extension=".json", + accept="application/json", + validate_existing=self._validate_json_file, + ) + except DownloadHTTPStatusError as exc: + if not self._is_not_updated_response(exc): + raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc + raise RuntimeError( + f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available" + ) from exc + + group_records = await self._load_downloaded_payload( + body_path, + group_url, + query_group=group, + constellation_group=group, + ) + group_counts[group] = len(group_records) + for item in group_records: + norad_cat_id = item.get("NORAD_CAT_ID") + if norad_cat_id is None: + continue + records_by_norad.setdefault(str(norad_cat_id), item) + except Exception as exc: + await emit_business_log( + logger, + event="collector.celestrak.fallback_groups.failed", + message="CelesTrak fallback group download failed", + category="collector", + level="error", + service="collector", + module=__name__, + context=exception_context( + exc, + { + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "groups": list(FALLBACK_GROUPS), + "completed_groups": list(group_counts), + "duration_ms": self._duration_ms(started_at), + }, + ), + ) + raise RuntimeError( + "CelesTrak active data has not updated since this network's last successful download, " + "no active cache is available, and fallback group mode failed. Wait until CelesTrak " + "publishes the next GP update, restore the Planet download cache, or use Space-Track." + ) from active_error + + records = list(records_by_norad.values()) + if not records: + raise RuntimeError("CelesTrak fallback group mode produced no satellite records") + + await emit_business_log( + logger, + event="collector.celestrak.fallback_groups.success", + message="CelesTrak fallback group download completed", + category="collector", + level="warning", + service="collector", + module=__name__, + context={ + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "groups": list(FALLBACK_GROUPS), + "group_counts": group_counts, + "record_count": len(records), + "duration_ms": self._duration_ms(started_at), + }, + ) + return records + + async def _load_downloaded_payload( + self, + body_path: Path, + url: str, + *, + query_group: str = ACTIVE_GROUP, + constellation_group: str | None = None, + ) -> List[Dict[str, Any]]: + try: + data = self._load_active_payload(body_path) + except RuntimeError as exc: + await self._log_parse_failure(exc) + raise + for item in data: + item["_celestrak_query_group"] = query_group + item["_celestrak_source_url"] = url + if constellation_group: + item["_celestrak_group"] = constellation_group + return data + + @staticmethod + def _is_not_updated_response(exc: DownloadHTTPStatusError) -> bool: + return exc.status_code == 403 and CELESTRAK_NOT_UPDATED_MARKER in exc.body + + @staticmethod + def _duration_ms(started_at: float) -> int: + return int((perf_counter() - started_at) * 1000) + + async def _report_download_progress(self, downloaded: int, total: int | None) -> None: + if total and total > 0: + await self.update_phase_progress( + current=min(downloaded, total), + total=total, + unit="bytes", + message=f"正在下载 CelesTrak active 卫星数据 {downloaded}/{total} bytes", + commit=True, + ) + + @staticmethod + def _validate_json_file(path: Path) -> bool: + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError, UnicodeDecodeError): + return False + return isinstance(data, list) + + async def _log_parse_failure(self, exc: Exception) -> None: + await emit_business_log( + logger, + event="collector.celestrak.parse.failed", + message="CelesTrak active satellite JSON parsing failed", + category="collector", + level="error", + service="collector", + module=__name__, + context=exception_context( + exc, + { + "collector_name": self.name, + "datasource_id": getattr(self, "_datasource_id", None), + "group": ACTIVE_GROUP, + }, + ), + ) + + def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]: + try: + raw = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc: + raise RuntimeError(f"CelesTrak active payload is not valid JSON: {exc}") from exc + if not isinstance(raw, list): + raise RuntimeError("CelesTrak active payload is not a JSON array") + + records: List[Dict[str, Any]] = [] + invalid_count = 0 + for item in raw: + if isinstance(item, dict) and item.get("NORAD_CAT_ID") is not None: + records.append(item) + else: + invalid_count += 1 + if invalid_count: + raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)") + if not records: + raise RuntimeError("CelesTrak active payload contains no satellite records") + return records def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]: transformed = [] for item in raw_data: + norad_cat_id = item.get("NORAD_CAT_ID") tle_line1, tle_line2 = build_tle_lines_from_elements( - norad_cat_id=item.get("NORAD_CAT_ID"), + norad_cat_id=norad_cat_id, epoch=item.get("EPOCH"), inclination=item.get("INCLINATION"), raan=item.get("RA_OF_ASC_NODE"), @@ -75,14 +410,18 @@ class CelesTrakTLECollector(BaseCollector): mean_anomaly=item.get("MEAN_ANOMALY"), mean_motion=item.get("MEAN_MOTION"), ) + constellation_group = self._infer_constellation_group(item) transformed.append( { + "source_id": str(norad_cat_id), "name": item.get("OBJECT_NAME", "Unknown"), "reference_date": item.get("EPOCH", ""), "metadata": { - "constellation_group": item.get("_celestrak_group"), - "norad_cat_id": item.get("NORAD_CAT_ID"), + "constellation_group": constellation_group, + "celestrak_query_group": item.get("_celestrak_query_group") or ACTIVE_GROUP, + "celestrak_source_url": item.get("_celestrak_source_url"), + "norad_cat_id": norad_cat_id, "international_designator": item.get("OBJECT_ID"), "epoch": item.get("EPOCH"), "mean_motion": item.get("MEAN_MOTION"), @@ -105,6 +444,19 @@ class CelesTrakTLECollector(BaseCollector): ) return transformed + @staticmethod + def _infer_constellation_group(item: Dict[str, Any]) -> str | None: + explicit_group = str(item.get("_celestrak_group") or "").strip().lower() + if explicit_group and explicit_group != ACTIVE_GROUP: + return explicit_group + + name = str(item.get("OBJECT_NAME") or "").strip().upper() + if name.startswith("STARLINK"): + return "starlink" + if name.startswith("IRIDIUM"): + return "iridium-next" + return None + def _get_sample_data(self) -> List[Dict[str, Any]]: return [ { diff --git a/backend/app/services/collectors/downloads.py b/backend/app/services/collectors/downloads.py index 5f604176..95660d39 100644 --- a/backend/app/services/collectors/downloads.py +++ b/backend/app/services/collectors/downloads.py @@ -4,7 +4,7 @@ from __future__ import annotations import hashlib import json -import tempfile +import os import time from datetime import UTC, datetime from pathlib import Path @@ -17,6 +17,31 @@ ProgressCallback = Callable[[int, int | None], Awaitable[None]] ValidateCallback = Callable[[Path], bool] +class DownloadHTTPStatusError(RuntimeError): + """HTTP status error that keeps the upstream response body for caller-specific handling.""" + + def __init__(self, *, url: str, status_code: int, body: str) -> None: + self.url = url + self.status_code = status_code + self.body = body + preview = body.strip().replace("\r", " ").replace("\n", " ")[:240] + suffix = f": {preview}" if preview else "" + super().__init__(f"HTTP {status_code} while downloading {url}{suffix}") + + +def default_download_cache_root() -> Path: + configured = os.getenv("PLANET_DOWNLOAD_CACHE_DIR") + if configured: + return Path(configured).expanduser() + planet_cache = os.getenv("PLANET_CACHE_DIR") + if planet_cache: + return Path(planet_cache).expanduser() / "downloads" + xdg_cache = os.getenv("XDG_CACHE_HOME") + if xdg_cache: + return Path(xdg_cache).expanduser() / "planet" / "downloads" + return Path.home() / ".cache" / "planet" / "downloads" + + class ResumableFileDownloader: """Download files with cache validators and byte-range resume support.""" @@ -26,8 +51,9 @@ class ResumableFileDownloader: cache_namespace: str, user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)", default_accept: str = "*/*", + cache_root: Path | None = None, ) -> None: - self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace + self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace self._user_agent = user_agent self._default_accept = default_accept @@ -43,6 +69,25 @@ class ResumableFileDownloader: meta_path = self._cache_dir / f"{key}.meta.json" return final_path, part_path, meta_path + def cached_file_path(self, url: str, extension: str) -> Path: + final_path, _, _ = self._cache_paths(url, extension) + return final_path + + def get_cached_file( + self, + url: str, + extension: str, + *, + validate_existing: ValidateCallback | None = None, + ) -> Path | None: + final_path = self.cached_file_path(url, extension) + if not final_path.exists(): + return None + if validate_existing and not validate_existing(final_path): + final_path.unlink(missing_ok=True) + return None + return final_path + @staticmethod def _load_meta(meta_path: Path) -> dict[str, Any]: if not meta_path.exists(): @@ -140,7 +185,9 @@ class ResumableFileDownloader: if progress_callback and expected_size and expected_size > 0: await progress_callback(expected_size, expected_size) return final_path - response.raise_for_status() + if response.status_code >= 400: + body = (await response.aread()).decode("utf-8", errors="replace") + raise DownloadHTTPStatusError(url=url, status_code=response.status_code, body=body) if response.status_code == 206 and resume_from > 0: mode = "ab" diff --git a/backend/app/services/collectors/peeringdb.py b/backend/app/services/collectors/peeringdb.py index e9ae2819..ef307030 100644 --- a/backend/app/services/collectors/peeringdb.py +++ b/backend/app/services/collectors/peeringdb.py @@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable. """ import asyncio -import os -from typing import Dict, Any, List from datetime import UTC, datetime +import os +from typing import Any, Dict, List +from urllib.parse import urlencode import httpx -from urllib.parse import urlencode + +from app.core.logging import get_logger from app.services.collectors.base import HTTPCollector # PeeringDB API key - read from environment variable PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "") +logger = get_logger(__name__, service="collector") class PeeringDBIXPCollector(HTTPCollector): @@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector): "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)", "Accept": "application/json", } + @property def request_url(self) -> str: base = self._resolved_url or self.base_url @@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector): if response.status_code == 429: # Rate limited - wait and retry with exponential backoff delay = base_delay * (2**attempt) - print(f"PeeringDB rate limited, waiting {delay}s before retry...") + logger.warning_event( + "PeeringDB rate limited; retrying after delay", + event="collector.peeringdb.rate_limited", + context={"delay_seconds": delay, "attempt": attempt + 1}, + ) await asyncio.sleep(delay) last_error = "Rate limited" continue @@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector): except httpx.HTTPStatusError as e: if e.response.status_code == 429: delay = base_delay * (2**attempt) - print(f"PeeringDB rate limited, waiting {delay}s before retry...") + logger.warning_event( + "PeeringDB rate limited; retrying after delay", + event="collector.peeringdb.rate_limited", + context={"delay_seconds": delay, "attempt": attempt + 1}, + ) await asyncio.sleep(delay) last_error = "Rate limited" continue raise - print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}") + logger.warning_event( + "PeeringDB collection failed after retries", + event="collector.peeringdb.retries_exhausted", + context={"max_retries": max_retries, "last_error": last_error}, + ) return {} async def fetch(self) -> List[Dict[str, Any]]: @@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector): "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)", "Accept": "application/json", } + @property def request_url(self) -> str: base = self._resolved_url or self.base_url @@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector): if response.status_code == 429: delay = base_delay * (2**attempt) - print(f"PeeringDB rate limited, waiting {delay}s before retry...") + logger.warning_event( + "PeeringDB rate limited; retrying after delay", + event="collector.peeringdb.rate_limited", + context={"delay_seconds": delay, "attempt": attempt + 1}, + ) await asyncio.sleep(delay) last_error = "Rate limited" continue @@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector): except httpx.HTTPStatusError as e: if e.response.status_code == 429: delay = base_delay * (2**attempt) - print(f"PeeringDB rate limited, waiting {delay}s before retry...") + logger.warning_event( + "PeeringDB rate limited; retrying after delay", + event="collector.peeringdb.rate_limited", + context={"delay_seconds": delay, "attempt": attempt + 1}, + ) await asyncio.sleep(delay) last_error = "Rate limited" continue raise - print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}") + logger.warning_event( + "PeeringDB collection failed after retries", + event="collector.peeringdb.retries_exhausted", + context={"max_retries": max_retries, "last_error": last_error}, + ) return {} async def fetch(self) -> List[Dict[str, Any]]: @@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector): "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)", "Accept": "application/json", } + @property def request_url(self) -> str: base = self._resolved_url or self.base_url @@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector): if response.status_code == 429: delay = base_delay * (2**attempt) - print(f"PeeringDB rate limited, waiting {delay}s before retry...") + logger.warning_event( + "PeeringDB rate limited; retrying after delay", + event="collector.peeringdb.rate_limited", + context={"delay_seconds": delay, "attempt": attempt + 1}, + ) await asyncio.sleep(delay) last_error = "Rate limited" continue @@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector): except httpx.HTTPStatusError as e: if e.response.status_code == 429: delay = base_delay * (2**attempt) - print(f"PeeringDB rate limited, waiting {delay}s before retry...") + logger.warning_event( + "PeeringDB rate limited; retrying after delay", + event="collector.peeringdb.rate_limited", + context={"delay_seconds": delay, "attempt": attempt + 1}, + ) await asyncio.sleep(delay) last_error = "Rate limited" continue raise - print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}") + logger.warning_event( + "PeeringDB collection failed after retries", + event="collector.peeringdb.retries_exhausted", + context={"max_retries": max_retries, "last_error": last_error}, + ) return {} async def fetch(self) -> List[Dict[str, Any]]: diff --git a/backend/app/services/collectors/spacetrack.py b/backend/app/services/collectors/spacetrack.py index 3104f083..c5c4f9f5 100644 --- a/backend/app/services/collectors/spacetrack.py +++ b/backend/app/services/collectors/spacetrack.py @@ -1,17 +1,21 @@ -"""Space-Track TLE Collector +"""Space-Track TLE Collector. Collects satellite TLE (Two-Line Element) data from Space-Track.org. API documentation: https://www.space-track.org/documentation """ -import json -from typing import Dict, Any, List -import httpx +from typing import Any, Dict, List from urllib.parse import urlparse -from app.services.collectors.base import BaseCollector +import httpx + from app.core.data_sources import get_data_sources_config +from app.core.logging import get_logger from app.core.satellite_tle import build_tle_lines_from_elements +from app.services.collectors.base import BaseCollector + + +logger = get_logger(__name__, service="collector") class SpaceTrackTLECollector(BaseCollector): @@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector): password = settings.SPACETRACK_PASSWORD if not username or not password: - print("SPACETRACK: No credentials configured, using sample data") + logger.warning_event( + "Space-Track credentials are not configured; using sample data", + event="collector.spacetrack.credentials_missing", + ) return self._get_sample_data() - print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}") + logger.info_event( + "Space-Track TLE fetch started", + event="collector.spacetrack.fetch.start", + ) try: async with httpx.AsyncClient( @@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector): "password": password, }, ) - print(f"SPACETRACK: Login response status: {login_response.status_code}") - print(f"SPACETRACK: Login response URL: {login_response.url}") + logger.info_event( + "Space-Track login response received", + event="collector.spacetrack.login.response", + context={"status_code": login_response.status_code}, + ) if login_response.status_code == 403: - print("SPACETRACK: Trying alternate login method...") + logger.warning_event( + "Space-Track login returned forbidden; trying alternate method", + event="collector.spacetrack.login.forbidden", + ) async with httpx.AsyncClient( timeout=120.0, @@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector): ) as alt_client: await alt_client.get(f"{self.site_root}/") - form_data = { - "username": username, - "password": password, - "query": "class/gp/NORAD_CAT_ID/25544/format/json", - } alt_login = await alt_client.post( self.login_url, data={ @@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector): "password": password, }, ) - print(f"SPACETRACK: Alt login status: {alt_login.status_code}") + logger.info_event( + "Space-Track alternate login response received", + event="collector.spacetrack.alt_login.response", + context={"status_code": alt_login.status_code}, + ) if alt_login.status_code == 200: tle_response = await alt_client.get(self.probe_url) if tle_response.status_code == 200: data = tle_response.json() - print(f"SPACETRACK: Received {len(data)} records via alt method") + logger.info_event( + "Space-Track alternate query completed", + event="collector.spacetrack.alt_query.completed", + context={"record_count": len(data)}, + ) return data if login_response.status_code != 200: - print(f"SPACETRACK: Login failed, using sample data") + logger.warning_event( + "Space-Track login failed; using sample data", + event="collector.spacetrack.login.failed", + context={"status_code": login_response.status_code}, + ) return self._get_sample_data() tle_response = await client.get(self.probe_url) - print(f"SPACETRACK: TLE query status: {tle_response.status_code}") - - if tle_response.status_code != 200: - print(f"SPACETRACK: Query failed, using sample data") - return self._get_sample_data() - - data = tle_response.json() - print(f"SPACETRACK: Received {len(data)} records") - return data - except Exception as e: - print(f"SPACETRACK: Error - {e}, using sample data") - return self._get_sample_data() - - print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}") - - try: - async with httpx.AsyncClient( - timeout=120.0, - follow_redirects=True, - headers={ - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", - "Accept": "application/json, text/html, */*", - "Accept-Language": "en-US,en;q=0.9", - }, - ) as client: - # First, visit the main page to get any cookies - await client.get(f"{self.site_root}/") - - # Login to get session cookie - login_response = await client.post( - self.login_url, - data={ - "identity": username, - "password": password, - }, + logger.info_event( + "Space-Track TLE query response received", + event="collector.spacetrack.query.response", + context={"status_code": tle_response.status_code}, ) - print(f"SPACETRACK: Login response status: {login_response.status_code}") - print(f"SPACETRACK: Login response URL: {login_response.url}") - print(f"SPACETRACK: Login response body: {login_response.text[:500]}") - - if login_response.status_code != 200: - print(f"SPACETRACK: Login failed, using sample data") - return self._get_sample_data() - - # Query for TLE data (get first 1000 satellites) - tle_response = await client.get(self.query_url) - print(f"SPACETRACK: TLE query status: {tle_response.status_code}") if tle_response.status_code != 200: - print(f"SPACETRACK: Query failed, using sample data") + logger.warning_event( + "Space-Track TLE query failed; using sample data", + event="collector.spacetrack.query.failed", + context={"status_code": tle_response.status_code}, + ) return self._get_sample_data() data = tle_response.json() - print(f"SPACETRACK: Received {len(data)} records") + logger.info_event( + "Space-Track TLE fetch completed", + event="collector.spacetrack.fetch.completed", + context={"record_count": len(data)}, + ) return data except Exception as e: - print(f"SPACETRACK: Error - {e}, using sample data") + logger.warning_event( + "Space-Track TLE fetch failed; using sample data", + event="collector.spacetrack.fetch.failed", + context={"error": str(e)}, + ) return self._get_sample_data() def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]: diff --git a/backend/app/services/collectors/telegeography.py b/backend/app/services/collectors/telegeography.py index d6af4e21..f27d487c 100644 --- a/backend/app/services/collectors/telegeography.py +++ b/backend/app/services/collectors/telegeography.py @@ -163,32 +163,64 @@ class TeleGeographyLandingPointCollector(BaseCollector): data_type = "landing_point" async def fetch(self) -> List[Dict[str, Any]]: - """Fetch landing point data from GitHub mirror""" - url = self._resolved_url or "" + """Fetch landing point data, falling back when the old mirror disappears.""" + config = get_data_sources_config() + sources = [ + self._resolved_url or "", + str(config.get_yaml_value("telegeography.landing_point_url") or ""), + str(config.get_yaml_value("arcgis.landing_point_url") or ""), + ] - async with httpx.AsyncClient(timeout=60.0) as client: - response = await client.get(url) - response.raise_for_status() - return self.parse_response(response.json()) + last_error: Exception | None = None + async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client: + for url in dict.fromkeys(source for source in sources if source): + try: + params = ( + {"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"} + if "FeatureServer" in url or url.endswith("/query") + else None + ) + response = await client.get(url, params=params) + response.raise_for_status() + records = self.parse_response(response.json()) + if records: + return records + except Exception as exc: + last_error = exc + continue - def parse_response(self, data: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + if last_error: + raise last_error + return self._get_sample_data() + + def parse_response(self, data: List[Dict[str, Any]] | Dict[str, Any]) -> List[Dict[str, Any]]: """Parse landing point data""" result = [] + items = data.get("features", []) if isinstance(data, dict) else data - for item in data: + for item in items: + props = item.get("properties", {}) if isinstance(item, dict) else {} + geometry = item.get("geometry", {}) if isinstance(item, dict) else {} + source = props or item + coords = geometry.get("coordinates", []) if isinstance(geometry, dict) else [] + longitude = coords[0] if len(coords) > 0 else source.get("longitude") + latitude = coords[1] if len(coords) > 1 else source.get("latitude") + source_id = source.get("id") or source.get("OBJECTID") or source.get("city_id") or "" try: entry = { - "source_id": f"telegeo_lp_{item.get('id', '')}", - "name": item.get("name", "Unknown"), - "country": item.get("country", "Unknown"), - "city": item.get("city", item.get("name", "")), - "latitude": str(item.get("latitude", "")), - "longitude": str(item.get("longitude", "")), + "source_id": f"telegeo_lp_{source_id}", + "name": source.get("name", source.get("Name", "Unknown")), + "country": source.get("country", "Unknown"), + "city": source.get("city", source.get("Name", source.get("name", ""))), + "latitude": str(latitude or ""), + "longitude": str(longitude or ""), "value": "", "unit": "", "metadata": { - "cable_count": len(item.get("cables", [])), - "url": item.get("url"), + "cable_count": len(source.get("cables", [])), + "url": source.get("url"), + "objectid": source.get("OBJECTID"), + "city_id": source.get("city_id"), }, "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"), } diff --git a/backend/app/services/collectors/vessel_ais.py b/backend/app/services/collectors/vessel_ais.py index 810b91c2..52c931f6 100644 --- a/backend/app/services/collectors/vessel_ais.py +++ b/backend/app/services/collectors/vessel_ais.py @@ -6,6 +6,7 @@ from typing import Any import httpx from sqlalchemy.ext.asyncio import AsyncSession +from app.core.enums import SnapshotStatus from app.core.time import to_iso8601_utc from app.core.websocket.broadcaster import broadcaster from app.services.barentswatch import ( @@ -119,6 +120,21 @@ class VesselAISCollector(BaseCollector): last_success_at=now if data else None, lag_seconds=max((now - latest_observed_at).total_seconds(), 0), ) + if snapshot_id is not None: + from app.models.data_snapshot import DataSnapshot + + snapshot = await db.get(DataSnapshot, snapshot_id) + if snapshot: + snapshot.record_count = records_added + snapshot.status = SnapshotStatus.SUCCESS.value + snapshot.completed_at = now + snapshot.summary = { + "created": records_added, + "updated": 0, + "unchanged": 0, + "deleted": 0, + "storage": "ais_raw_observations", + } await db.commit() await self._broadcast_vessel_snapshot(data) await self.update_progress(records_added, force=True) diff --git a/backend/app/services/compute_center_locations.py b/backend/app/services/compute_center_locations.py index d2ba9acd..8c91df74 100644 --- a/backend/app/services/compute_center_locations.py +++ b/backend/app/services/compute_center_locations.py @@ -11,7 +11,7 @@ startup, so it must stay local and deterministic. For the full design and the reason behind the abstraction (compute centers, BGP collectors, BGP events, and future entities all share one pipeline), -see ``docs/plans/location-resolver-shared-pipeline-plan.md``. +see ``docs/technical/zh/location-pipeline-development.md``. The ``ComputeCenterLocation`` dataclass and the public function signatures are preserved verbatim so existing callers and tests do not need to change. diff --git a/backend/app/services/data_jobs.py b/backend/app/services/data_jobs.py new file mode 100644 index 00000000..a10ab057 --- /dev/null +++ b/backend/app/services/data_jobs.py @@ -0,0 +1,720 @@ +"""Kafka-ready datasource job queue backed by PostgreSQL for v1.""" + +from __future__ import annotations + +import asyncio +from datetime import UTC, datetime, timedelta +from uuid import uuid4 +from typing import Any + +from sqlalchemy import bindparam, select, text +from sqlalchemy.ext.asyncio import AsyncSession + +from app.core.cache import cache +from app.core.config import settings +from app.core.enums import JobStatus, JobType, RollbackPolicy +from app.core.logging import get_logger +from app.core.time import to_iso8601_utc +from app.core.websocket.broadcaster import broadcaster +from app.db.session import async_session_factory +from app.models.collected_data import CollectedData +from app.models.data_snapshot import DataSnapshot +from app.models.datasource import DataSource +from app.models.task import CollectionTask +from app.services.collectors.registry import collector_registry +from app.services.datasource_connectivity import ( + build_builtin_connectivity_checksum, + get_builtin_effective_candidate, + save_connectivity_success, +) +from app.services.earth_layer_adapters import ( + clear_derived_datasource_data, + get_earth_refresh_strategy_for_change, + get_earth_update_layers_for_source, +) +from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source +from app.services.scheduler import sync_datasource_job + +logger = get_logger(__name__) + +JOB_TYPE_COLLECT = JobType.COLLECT.value +JOB_TYPE_CLEAR_DATA = JobType.CLEAR_DATA.value +JOB_TYPE_CLEAR_CACHE = JobType.CLEAR_CACHE.value +JOB_TYPE_EARTH_REFRESH = JobType.EARTH_REFRESH.value + +JOB_STATUS_QUEUED = JobStatus.QUEUED.value +JOB_STATUS_RUNNING = JobStatus.RUNNING.value +JOB_STATUS_CANCELLING = JobStatus.CANCELLING.value +JOB_STATUS_SUCCESS = JobStatus.SUCCESS.value +JOB_STATUS_FAILED = JobStatus.FAILED.value +JOB_STATUS_CANCELLED = JobStatus.CANCELLED.value + +ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING) +TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED) +DATA_WRITE_JOB_TYPES = (JOB_TYPE_COLLECT, JOB_TYPE_CLEAR_DATA, JOB_TYPE_CLEAR_CACHE) +SOURCE_LOCK_JOB_STATUSES = (JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING) +QUEUE_POLL_SECONDS = 0.35 +JOB_STALE_LOCK_MINUTES = 90 +ORPHAN_CANCELLING_GRACE_SECONDS = 30 +JOB_RECOVERY_SWEEP_SECONDS = 15 +DATA_DELETE_BATCH_SIZE = 50_000 +DEFAULT_WORKER_CONCURRENCY = 2 + +RUNNING_DATA_JOB_TASKS: dict[int, asyncio.Task[Any]] = {} + + +def _utcnow() -> datetime: + return datetime.now(UTC) + + +def _job_worker_id() -> str: + return f"{settings.PROJECT_NAME}:data-job-worker:{uuid4().hex[:8]}" + + +def is_terminal_job_status(status: str | None) -> bool: + return status in TERMINAL_JOB_STATUSES + + +async def enqueue_datasource_job( + db: AsyncSession, + datasource: DataSource, + task_type: str, + *, + payload: dict[str, Any] | None = None, + rollback_policy: str = RollbackPolicy.KEEP_COMMITTED_BATCHES.value, + dedupe_key: str | None = None, +) -> CollectionTask: + if dedupe_key: + existing = await _get_active_job_by_dedupe_key(db, dedupe_key) + if existing is not None: + return existing + + task = CollectionTask( + datasource_id=datasource.id, + source=datasource.source, + task_type=task_type, + status=JOB_STATUS_QUEUED, + phase="queued", + phase_message="任务已进入队列", + payload=payload or {}, + rollback_policy=rollback_policy, + dedupe_key=dedupe_key, + ) + db.add(task) + await db.commit() + await db.refresh(task) + await _broadcast_task_update(task) + return task + + +async def enqueue_earth_refresh_job( + db: AsyncSession, + *, + source: str, + payload: dict[str, Any] | None = None, +) -> CollectionTask | None: + layers = list((payload or {}).get("layers") or get_earth_update_layers_for_source(source)) + if not layers: + return None + + datasource = await _get_or_create_virtual_datasource(db, source) + refresh_payload = { + "source": source, + "layers": layers, + "refresh_strategy": (payload or {}).get("refresh_strategy") + or get_earth_refresh_strategy_for_change((payload or {}).get("table"), source) + or "clear_then_reload", + **(payload or {}), + } + return await enqueue_datasource_job( + db, + datasource, + JOB_TYPE_EARTH_REFRESH, + payload=refresh_payload, + dedupe_key=f"earth_refresh:{source}", + ) + + +async def enqueue_earth_refresh_from_update(payload: dict[str, Any]) -> None: + source = str(payload.get("source") or "").strip() + if not source: + return + async with async_session_factory() as db: + await enqueue_earth_refresh_job(db, source=source, payload=payload) + + +async def request_cancel_datasource_task( + db: AsyncSession, + task: CollectionTask, + *, + reason: str = "cancelled_by_operator", +) -> CollectionTask: + if is_terminal_job_status(task.status): + return task + + running_task = RUNNING_DATA_JOB_TASKS.get(task.id) + if task.status == JOB_STATUS_QUEUED or ( + running_task is None + and ( + task.status == JOB_STATUS_CANCELLING + or (task.status == JOB_STATUS_RUNNING and task.task_type != JOB_TYPE_COLLECT) + ) + ): + return await _cancel_task_without_runner(db, task, reason=reason) + + task.status = JOB_STATUS_CANCELLING + task.phase = JOB_STATUS_CANCELLING + task.phase_message = "正在停止任务" + task.requested_cancel_at = _utcnow() + task.cancel_reason = reason + await db.commit() + await db.refresh(task) + + if running_task is not None and not running_task.done(): + running_task.cancel() + + await _broadcast_task_update(task) + return task + + +async def _cancel_task_without_runner( + db: AsyncSession, + task: CollectionTask, + *, + reason: str, +) -> CollectionTask: + if task.task_type == JOB_TYPE_COLLECT: + await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task.id)) + snapshot_result = await db.execute(select(DataSnapshot).where(DataSnapshot.task_id == task.id)) + for snapshot in snapshot_result.scalars().all(): + snapshot.status = JOB_STATUS_CANCELLED + snapshot.completed_at = _utcnow() + snapshot.error_message = "Cancelled after operator stop request; no active worker handle remained" + datasource = await db.get(DataSource, task.datasource_id) + if datasource is not None: + datasource.last_status = JOB_STATUS_CANCELLED + + task.status = JOB_STATUS_CANCELLED + task.phase = JOB_STATUS_CANCELLED + task.phase_message = "任务已停止" + task.completed_at = _utcnow() + task.requested_cancel_at = task.requested_cancel_at or _utcnow() + task.cancel_reason = reason + await db.commit() + await db.refresh(task) + await _broadcast_task_update(task) + return task + + +async def get_active_datasource_job( + db: AsyncSession, + datasource_id: int, + *, + task_types: tuple[str, ...] = DATA_WRITE_JOB_TYPES, +) -> CollectionTask | None: + result = await db.execute( + select(CollectionTask) + .where(CollectionTask.datasource_id == datasource_id) + .where(CollectionTask.task_type.in_(task_types)) + .where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES)) + .order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc()) + .limit(1) + ) + return result.scalar_one_or_none() + + +async def _get_active_job_by_dedupe_key(db: AsyncSession, dedupe_key: str) -> CollectionTask | None: + result = await db.execute( + select(CollectionTask) + .where(CollectionTask.dedupe_key == dedupe_key) + .where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES)) + .order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc()) + .limit(1) + ) + return result.scalar_one_or_none() + + +async def _get_or_create_virtual_datasource(db: AsyncSession, source: str) -> DataSource: + result = await db.execute(select(DataSource).where(DataSource.source == source)) + datasource = result.scalar_one_or_none() + if datasource is not None: + return datasource + + datasource = DataSource( + name=f"Earth refresh: {source}", + source=source, + module="SYS", + collector_class="EarthRefreshJob", + is_active=True, + ) + db.add(datasource) + await db.commit() + await db.refresh(datasource) + return datasource + + +async def _broadcast_task_update(task: CollectionTask) -> None: + await broadcaster.broadcast_datasource_task_update( + { + "datasource_id": task.datasource_id, + "collector_name": task.source, + "task_id": task.id, + "task_type": task.task_type, + "status": task.status, + "phase": task.phase, + "phase_progress": task.phase_progress, + "phase_message": task.phase_message, + "phase_current": task.phase_current, + "phase_total": task.phase_total, + "phase_unit": task.phase_unit, + "progress": task.progress, + "records_processed": task.records_processed, + "total_records": task.total_records, + "started_at": to_iso8601_utc(task.started_at), + "completed_at": to_iso8601_utc(task.completed_at), + "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at), + "error_message": task.error_message, + } + ) + + +class DataJobWorker: + def __init__(self, *, concurrency: int = DEFAULT_WORKER_CONCURRENCY) -> None: + self.worker_id = _job_worker_id() + self.concurrency = max(1, concurrency) + self._task: asyncio.Task[None] | None = None + self._stop_event: asyncio.Event | None = None + self._running: set[asyncio.Task[Any]] = set() + self._last_recovery_sweep_at: datetime | None = None + + def start(self) -> None: + if self._task and not self._task.done(): + return + self._stop_event = asyncio.Event() + self._task = asyncio.create_task(self._run(), name="data-job-worker") + + async def stop(self) -> None: + if self._stop_event: + self._stop_event.set() + for task in list(self._running): + task.cancel() + if self._task: + await asyncio.gather(self._task, return_exceptions=True) + if self._running: + await asyncio.gather(*self._running, return_exceptions=True) + + async def _run(self) -> None: + assert self._stop_event is not None + await self._recover_stale_running_jobs() + while not self._stop_event.is_set(): + self._running = {task for task in self._running if not task.done()} + if ( + self._last_recovery_sweep_at is None + or (_utcnow() - self._last_recovery_sweep_at).total_seconds() >= JOB_RECOVERY_SWEEP_SECONDS + ): + await self._recover_stale_running_jobs() + if len(self._running) >= self.concurrency: + await asyncio.sleep(QUEUE_POLL_SECONDS) + continue + + task_id = await self._claim_next_job() + if task_id is None: + await asyncio.sleep(QUEUE_POLL_SECONDS) + continue + + runner = asyncio.create_task(self._run_claimed_job(task_id), name=f"data-job:{task_id}") + self._running.add(runner) + + async def _recover_stale_running_jobs(self) -> None: + self._last_recovery_sweep_at = _utcnow() + cutoff = _utcnow() - timedelta(minutes=JOB_STALE_LOCK_MINUTES) + orphan_cancelling_cutoff = _utcnow() - timedelta(seconds=ORPHAN_CANCELLING_GRACE_SECONDS) + async with async_session_factory() as db: + result = await db.execute( + select(CollectionTask) + .where(CollectionTask.status.in_((JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING))) + .where(CollectionTask.locked_at.is_not(None)) + .where(CollectionTask.locked_at < cutoff) + ) + stale_jobs = list(result.scalars().all()) + for job in stale_jobs: + job.status = JOB_STATUS_FAILED + job.phase = JOB_STATUS_FAILED + job.completed_at = _utcnow() + job.error_message = "Marked failed after stale data job lock timeout" + if stale_jobs: + await db.commit() + orphan_result = await db.execute( + select(CollectionTask) + .where(CollectionTask.status == JOB_STATUS_CANCELLING) + .where(CollectionTask.locked_at.is_(None)) + .where(CollectionTask.requested_cancel_at.is_not(None)) + .where(CollectionTask.requested_cancel_at < orphan_cancelling_cutoff) + ) + for job in orphan_result.scalars().all(): + if job.id in RUNNING_DATA_JOB_TASKS: + continue + await _cancel_task_without_runner( + db, + job, + reason=job.cancel_reason or "cancelled_after_orphaned_runner", + ) + + async def _claim_next_job(self) -> int | None: + async with async_session_factory() as db: + row = await db.execute( + text( + """ + SELECT queued.id + FROM collection_tasks AS queued + WHERE queued.status = :queued_status + AND NOT EXISTS ( + SELECT 1 + FROM collection_tasks AS active + WHERE active.source = queued.source + AND active.id <> queued.id + AND active.status IN :active_statuses + ) + ORDER BY queued.created_at ASC NULLS FIRST, queued.id ASC + LIMIT 1 + FOR UPDATE SKIP LOCKED + """ + ).bindparams(bindparam("active_statuses", expanding=True)), + { + "queued_status": JOB_STATUS_QUEUED, + "active_statuses": SOURCE_LOCK_JOB_STATUSES, + }, + ) + task_id = row.scalar_one_or_none() + if task_id is None: + return None + + task = await db.get(CollectionTask, int(task_id)) + if task is None: + return None + task.status = JOB_STATUS_RUNNING + task.phase = "starting" + task.phase_message = "任务开始执行" + task.started_at = task.started_at or _utcnow() + task.worker_id = self.worker_id + task.locked_at = _utcnow() + await db.commit() + await _broadcast_task_update(task) + return int(task_id) + + async def _run_claimed_job(self, task_id: int) -> None: + current_task = asyncio.current_task() + if current_task is not None: + RUNNING_DATA_JOB_TASKS[task_id] = current_task + try: + async with async_session_factory() as db: + task = await db.get(CollectionTask, task_id) + if task is None: + return + await self._execute_job(db, task) + except asyncio.CancelledError: + async with async_session_factory() as db: + task = await db.get(CollectionTask, task_id) + if task is not None and not is_terminal_job_status(task.status): + task.status = JOB_STATUS_CANCELLED + task.phase = JOB_STATUS_CANCELLED + task.phase_message = "任务已停止" + task.completed_at = _utcnow() + await db.commit() + await _broadcast_task_update(task) + raise + except Exception as exc: + logger.exception_event( + "Data job failed", + event="data_jobs.job_failed", + context={"task_id": task_id, "error": str(exc)}, + ) + async with async_session_factory() as db: + task = await db.get(CollectionTask, task_id) + if task is not None: + task.status = JOB_STATUS_FAILED + task.phase = JOB_STATUS_FAILED + task.phase_message = str(exc) + task.error_message = str(exc) + task.completed_at = _utcnow() + await db.commit() + await _broadcast_task_update(task) + finally: + RUNNING_DATA_JOB_TASKS.pop(task_id, None) + + async def _execute_job(self, db: AsyncSession, task: CollectionTask) -> None: + if task.task_type == JOB_TYPE_COLLECT: + await _run_collect_job(db, task) + elif task.task_type == JOB_TYPE_CLEAR_DATA: + await _run_clear_data_job(db, task) + elif task.task_type == JOB_TYPE_CLEAR_CACHE: + await _run_clear_cache_job(db, task) + elif task.task_type == JOB_TYPE_EARTH_REFRESH: + await _run_earth_refresh_job(db, task) + else: + raise RuntimeError(f"Unsupported data job type: {task.task_type}") + + +async def _run_collect_job(db: AsyncSession, task: CollectionTask) -> None: + datasource = await db.get(DataSource, task.datasource_id) + if datasource is None: + raise RuntimeError("Data source not found") + + collector = collector_registry.get(datasource.source) + if collector is None: + raise RuntimeError(f"Collector '{datasource.source}' not found") + if not datasource.is_active: + raise RuntimeError("Data source is disabled") + + collector._datasource_id = datasource.id + collector._current_task = task + collector._db_session = db + result = await collector.run(db) + + datasource.last_run_at = _utcnow() + datasource.last_status = result.get("status") + if datasource.last_status == JOB_STATUS_SUCCESS: + effective_candidate = await get_builtin_effective_candidate(db, datasource.source) + checksum, _credential_context = await build_builtin_connectivity_checksum( + datasource.source, + effective_candidate["endpoint"], + effective_candidate["auth_type"], + effective_candidate["headers"], + effective_candidate["config"], + db, + ) + await save_connectivity_success( + db, + datasource.source, + checksum, + {"status_code": None}, + connected_by="collection", + ) + await db.commit() + await sync_datasource_job(datasource.id) + + +async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None: + source = str(task.source or (task.payload or {}).get("source") or "").strip() + if not source: + raise RuntimeError("Clear data job has no source") + + task.phase = "clearing_data" + task.phase_message = "正在删除数据库数据" + await db.commit() + await _broadcast_task_update(task) + + deleted_count = await _delete_table_rows_by_source( + db, + task, + table_name="collected_data", + source_column="source", + source=source, + ) + derived_deleted_counts = await _clear_derived_datasource_data_in_batches( + db, + task, + source, + progress_offset=deleted_count, + ) + derived_deleted_count = sum(derived_deleted_counts.values()) + if any(key.startswith("ais_") for key in derived_deleted_counts): + await db.execute(text("ANALYZE ais_raw_observations")) + + task.records_processed = deleted_count + derived_deleted_count + task.total_records = task.records_processed + task.progress = 100.0 + task.phase_progress = 100.0 + task.phase_current = task.records_processed + task.phase_total = task.records_processed + task.phase_unit = "records" + task.payload = { + **(task.payload or {}), + "deleted_count": deleted_count, + "derived_deleted_count": derived_deleted_count, + "derived_deleted_counts": derived_deleted_counts, + } + task.status = JOB_STATUS_SUCCESS + task.phase = "completed" + task.phase_message = "数据库数据已清理" + task.completed_at = _utcnow() + datasource = await db.get(DataSource, task.datasource_id) + if datasource is not None: + datasource.last_status = JOB_STATUS_SUCCESS + datasource.last_run_at = task.completed_at + await db.execute( + DataSnapshot.__table__.update() + .where(DataSnapshot.source == source) + .values(is_current=False) + ) + await db.commit() + await _broadcast_task_update(task) + + +async def _delete_table_rows_by_source( + db: AsyncSession, + task: CollectionTask, + *, + table_name: str, + source_column: str, + source: str, + progress_offset: int = 0, +) -> int: + deleted = 0 + while True: + result = await db.execute( + text( + f""" + WITH doomed AS ( + SELECT ctid + FROM {table_name} + WHERE {source_column} = :source + LIMIT :batch_size + ), + deleted_rows AS ( + DELETE FROM {table_name} + USING doomed + WHERE {table_name}.ctid = doomed.ctid + RETURNING 1 + ) + SELECT COUNT(*) FROM deleted_rows + """ + ), + {"source": source, "batch_size": DATA_DELETE_BATCH_SIZE}, + ) + batch_deleted = max(int(result.scalar_one() or 0), 0) + if batch_deleted <= 0: + break + deleted += batch_deleted + task.records_processed = progress_offset + deleted + task.phase_current = task.records_processed + task.phase_unit = "records" + task.phase_message = f"正在删除数据：{task.records_processed} 条" + await db.commit() + await _broadcast_task_update(task) + return deleted + + +async def _clear_derived_datasource_data_in_batches( + db: AsyncSession, + task: CollectionTask, + source: str, + progress_offset: int = 0, +) -> dict[str, int]: + deleted_counts: dict[str, int] = {} + if source in {"barentswatch_vessels", "aisstream_vessels"}: + deleted_counts["ais_conflict_records"] = await _delete_table_rows_by_source( + db, + task, + table_name="ais_conflict_records", + source_column="selected_source", + source=source, + progress_offset=progress_offset + sum(deleted_counts.values()), + ) + deleted_counts["ais_source_health"] = await _delete_table_rows_by_source( + db, + task, + table_name="ais_source_health", + source_column="source", + source=source, + progress_offset=progress_offset + sum(deleted_counts.values()), + ) + deleted_counts["ais_raw_observations"] = await _delete_table_rows_by_source( + db, + task, + table_name="ais_raw_observations", + source_column="source", + source=source, + progress_offset=progress_offset + sum(deleted_counts.values()), + ) + return deleted_counts + return await clear_derived_datasource_data(db, source) + + +async def _run_clear_cache_job(db: AsyncSession, task: CollectionTask) -> None: + source = str(task.source or (task.payload or {}).get("source") or "").strip() + if not source: + raise RuntimeError("Clear cache job has no source") + + earth_deleted_count = invalidate_earth_layer_cache_for_source(source) + dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary")) + deleted_count = earth_deleted_count + dashboard_deleted_count + + task.records_processed = deleted_count + task.total_records = deleted_count + task.progress = 100.0 + task.phase_progress = 100.0 + task.phase = "completed" + task.phase_message = "缓存已清理" + task.payload = { + **(task.payload or {}), + "earth_layer_deleted_count": earth_deleted_count, + "dashboard_deleted_count": dashboard_deleted_count, + } + task.status = JOB_STATUS_SUCCESS + task.completed_at = _utcnow() + await db.commit() + await _broadcast_task_update(task) + await enqueue_earth_refresh_job(db, source=source, payload={"operation": "CACHE_INVALIDATED"}) + + +async def _run_earth_refresh_job(db: AsyncSession, task: CollectionTask) -> None: + payload = task.payload or {} + source = str(payload.get("source") or task.source or "").strip() + layers = list(payload.get("layers") or get_earth_update_layers_for_source(source)) + if not source or not layers: + task.status = JOB_STATUS_SUCCESS + task.phase = "completed" + task.phase_message = "没有需要刷新的 Earth 图层" + task.completed_at = _utcnow() + await db.commit() + await _broadcast_task_update(task) + return + + deleted_cache_entries = invalidate_earth_layer_cache_for_source(source) + update_payload = { + "event": "earth.layer.changed", + "action": "database_changed", + "source": source, + "table": payload.get("table"), + "data_type": source, + "layers": layers, + "refresh_strategy": payload.get("refresh_strategy") or "clear_then_reload", + "records_processed": payload.get("records_processed", 0), + "operations": payload.get("operations") or [payload.get("operation") or "CHANGE"], + "operation": payload.get("operation"), + "cache_entries_invalidated": deleted_cache_entries, + "timestamp": to_iso8601_utc(_utcnow()), + } + if payload.get("entity") == "interactable": + update_payload.update( + { + "entity": "interactable", + "action": payload.get("action") or "changed", + "ids": payload.get("ids") or payload.get("entity_keys") or [], + "item": payload.get("item"), + } + ) + await broadcaster.broadcast_earth_update(update_payload) + + task.records_processed = int(payload.get("records_processed") or 0) + task.progress = 100.0 + task.phase_progress = 100.0 + task.phase = "completed" + task.phase_message = "Earth 图层刷新通知已发送" + task.status = JOB_STATUS_SUCCESS + task.completed_at = _utcnow() + task.payload = {**payload, "cache_entries_invalidated": deleted_cache_entries} + await db.commit() + await _broadcast_task_update(task) + + +_worker = DataJobWorker() + + +def start_data_job_worker() -> None: + _worker.start() + + +async def stop_data_job_worker() -> None: + await _worker.stop() diff --git a/backend/app/services/datasource_connectivity.py b/backend/app/services/datasource_connectivity.py index f6c7fb3a..db18342d 100644 --- a/backend/app/services/datasource_connectivity.py +++ b/backend/app/services/datasource_connectivity.py @@ -13,6 +13,7 @@ from sqlalchemy import func, select from app.core.data_sources import get_data_sources_config from app.core.datasource_defaults import DEFAULT_DATASOURCES +from app.core.enums import JobStatus from app.models.collected_data import CollectedData from app.models.datasource import DataSource from app.models.datasource_config import DataSourceConfig @@ -398,7 +399,7 @@ async def has_collected_data(db, source: str) -> bool: datasource_result = await db.execute(select(DataSource).where(DataSource.source == source)) datasource = datasource_result.scalar_one_or_none() - return bool(datasource and datasource.last_status == "success") + return bool(datasource and datasource.last_status == JobStatus.SUCCESS.value) async def get_builtin_connection_status( diff --git a/backend/app/services/docs_gatekeeper.py b/backend/app/services/docs_gatekeeper.py index 1ebbdf89..66f774f2 100644 --- a/backend/app/services/docs_gatekeeper.py +++ b/backend/app/services/docs_gatekeeper.py @@ -6,6 +6,7 @@ from dataclasses import dataclass from pathlib import Path from typing import Literal +from app.core.enums import UserRole from app.models.user import User DocsAccess = Literal["public", "docs_user", "docs_developer", "docs_admin"] @@ -32,30 +33,34 @@ class DocsMetadata: DOCS_METADATA: tuple[DocsMetadata, ...] = ( DocsMetadata(DOCS_README_FILENAME, DEFAULT_DOCS_SLUG, "public", "Overview", 0, "技术文档", "Technical Docs"), - DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 1, "快速开始", "Quickstart"), - DocsMetadata("manual.md", "manual", "public", "Manual", 2, "Planet 使用手册", "Planet Manual"), + DocsMetadata("manual.md", "manual", "public", "Manual", 1, "智能星球使用手册", "Intelligent Planet Manual"), + DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 2, "快速开始", "Quickstart"), DocsMetadata("faq.md", "faq", "public", "Manual", 3, "常见问题", "FAQ"), - DocsMetadata("location-pipeline-user.md", "location-pipeline-user", "public", "Manual", 4, "Earth 位置候选采集使用手册", "Earth Location Candidate Collection User Guide"), - DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "Earth 前端结构", "Earth Frontend Context"), - DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "Earth 图层样式属性索引", "Earth Layer Style Reference"), - DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "Earth 渲染图层顺序", "Earth Render Layer Order"), - DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "Earth 卫星覆盖策略", "Earth Satellite Footprint Policy"), + DocsMetadata("platform-data-flows.md", "platform-data-flows", "docs_developer", "Architecture", 5, "业务架构与数据流转", "Business Architecture and Data Flows"), + DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Architecture", 6, "命名与术语对照", "Naming Glossary"), + DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "智能星球前端结构", "Intelligent Planet Frontend Context"), + DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "智能星球图层样式属性索引", "Intelligent Planet Layer Style Reference"), + DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "智能星球渲染图层顺序", "Intelligent Planet Render Layer Order"), + DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "智能星球卫星覆盖策略", "Intelligent Planet Satellite Footprint Policy"), DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"), - DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Earth", 15, "新闻直播采集格式", "News Live Streams Collector Format"), - DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "Earth 可交互图标接入", "Earth Interactable Usage"), - DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "Earth 工具栏与浮层协同", "Earth Toolbar and Overlay Coordination"), + DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"), + DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"), + DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"), + DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"), DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"), DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"), - DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Frontend", 22, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"), - DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Frontend", 23, "命名与术语对照", "Naming Glossary"), DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"), DocsMetadata("backend-collectors.md", "backend-collectors", "docs_developer", "Backend", 30, "数据采集系统", "Data Collectors"), DocsMetadata("backend-system-service-control.md", "backend-system-service-control", "docs_admin", "Backend", 31, "系统服务控制", "System Service Control"), DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"), DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"), - DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 34, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"), + DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"), + DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"), + DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"), + DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"), + DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"), DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"), - DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "Planet 运维手册", "Planet Ops Runbook"), + DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "智能星球运维手册", "Intelligent Planet Ops Runbook"), DocsMetadata("ops-docker-compose-buildx-upgrade.md", "ops-docker-compose-buildx-upgrade", "docs_admin", "Ops", 50, "Docker + Compose + Buildx 升级", "Docker + Compose + Buildx Upgrade"), DocsMetadata("ops-planet-sh-startup.md", "ops-planet-sh-startup", "docs_admin", "Ops", 51, "planet.sh 启动机制", "planet.sh Startup"), ) @@ -68,9 +73,9 @@ def get_user_gatekeeper_groups(user: User | None) -> set[str]: return set() role = user.role.value if hasattr(user.role, "value") else str(user.role or "") - if role == "super_admin": + if role == UserRole.SUPER_ADMIN.value: return {"docs_user", "docs_developer", "docs_admin"} - if role == "admin": + if role == UserRole.ADMIN.value: return {"docs_user", "docs_developer", "docs_admin"} groups = set() diff --git a/backend/app/services/earth_db_change_listener.py b/backend/app/services/earth_db_change_listener.py new file mode 100644 index 00000000..4bd61946 --- /dev/null +++ b/backend/app/services/earth_db_change_listener.py @@ -0,0 +1,482 @@ +"""PostgreSQL LISTEN/NOTIFY bridge for Earth layer refresh events.""" + +from __future__ import annotations + +import asyncio +import json +from collections import deque +from dataclasses import dataclass, field +from datetime import UTC, datetime +from time import monotonic +from typing import Any, Awaitable, Callable + +import asyncpg + +from app.core.config import settings +from app.core.logging import get_logger +from app.core.time import to_iso8601_utc +from app.core.websocket.broadcaster import broadcaster +from app.services.earth_layer_adapters import ( + get_earth_refresh_strategy_for_change, + get_earth_update_layers_for_change, + get_earth_update_layers_for_source, +) +from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source + +logger = get_logger(__name__) + +EARTH_DATA_CHANGES_CHANNEL = "planet_earth_data_changes" +DEFAULT_DEBOUNCE_SECONDS = 0.25 +DEFAULT_MAX_WAIT_SECONDS = 1.5 +DELETE_FAST_FLUSH_SECONDS = 0.05 +LISTEN_KEEPALIVE_SECONDS = 5.0 +OUTBOX_POLL_LIMIT = 5000 +MAX_ENTITY_KEY_SAMPLES = 20 +MAX_SEEN_EVENT_IDS = 20000 + +BroadcastFn = Callable[[dict[str, Any]], Awaitable[None]] +InvalidateFn = Callable[[str], int] + + +def normalize_asyncpg_dsn(dsn: str) -> str: + """Convert SQLAlchemy asyncpg URLs into asyncpg-compatible URLs.""" + return dsn.replace("postgresql+asyncpg://", "postgresql://", 1) + + +def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any] | None: + table = payload.get("table") + source = str(payload.get("source") or "").strip() + table_name = str(table or "").strip() + if not source and not table_name: + return None + layers = get_earth_update_layers_for_change(table_name, source) + if not layers: + return None + refresh_strategy = get_earth_refresh_strategy_for_change(table_name, source) or "clear_then_reload" + source_has_adapter = bool(get_earth_update_layers_for_source(source)) + effective_source = source if source_has_adapter else (table_name if table_name else source) + operation = payload.get("operation") + update: dict[str, Any] = { + "event": "earth.layer.changed", + "action": "database_changed", + "source": effective_source, + "original_source": source or None, + "table": table_name or None, + "data_type": effective_source, + "layers": layers, + "refresh_strategy": refresh_strategy, + "operation": operation, + "entity_key": payload.get("entity_key"), + "timestamp": to_iso8601_utc(datetime.now(UTC)), + } + if table_name == "earth_interactables" and refresh_strategy == "delta": + ids = payload.get("entity_keys") + if not isinstance(ids, list): + ids = [payload.get("entity_key")] if payload.get("entity_key") else [] + update.update( + { + "entity": "interactable", + "action": "deleted" if operation == "DELETE" else "changed", + "ids": [str(item) for item in ids if item], + "item": None, + } + ) + return { + **update, + } + + +@dataclass +class PendingEarthDbChange: + source: str + layers: list[str] + table: str | None = None + refresh_strategy: str = "clear_then_reload" + entity: str | None = None + action: str = "database_changed" + records_processed: int = 0 + operations: set[str] = field(default_factory=set) + entity_keys: list[str] = field(default_factory=list) + first_occurred_at: str | None = None + last_occurred_at: str | None = None + first_seen_monotonic: float = field(default_factory=monotonic) + last_seen_monotonic: float = field(default_factory=monotonic) + + def add(self, payload: dict[str, Any]) -> None: + self.last_seen_monotonic = monotonic() + records_processed = payload.get("records_processed", 1) + try: + records_processed = int(records_processed) + except (TypeError, ValueError): + records_processed = 1 + self.records_processed += max(records_processed, 1) + operation = payload.get("operation") + if operation: + self.operations.add(str(operation)) + entity_keys = payload.get("entity_keys") + if not isinstance(entity_keys, list): + entity_key = payload.get("entity_key") + entity_keys = [entity_key] if entity_key else [] + for entity_key in entity_keys: + if entity_key and len(self.entity_keys) < MAX_ENTITY_KEY_SAMPLES: + self.entity_keys.append(str(entity_key)) + occurred_at = payload.get("occurred_at") + if occurred_at: + occurred_at = str(occurred_at) + self.first_occurred_at = self.first_occurred_at or occurred_at + self.last_occurred_at = occurred_at + + +class EarthDbChangeDispatcher: + """Debounces database notifications and broadcasts Earth refresh hints.""" + + def __init__( + self, + *, + broadcast_earth_update: BroadcastFn | None = None, + invalidate_cache: InvalidateFn | None = None, + debounce_seconds: float = DEFAULT_DEBOUNCE_SECONDS, + max_wait_seconds: float = DEFAULT_MAX_WAIT_SECONDS, + ) -> None: + self._broadcast_earth_update = broadcast_earth_update or broadcaster.broadcast_earth_update + self._invalidate_cache = invalidate_cache or invalidate_earth_layer_cache_for_source + self._debounce_seconds = debounce_seconds + self._max_wait_seconds = max(max_wait_seconds, debounce_seconds) + self._pending: dict[str, PendingEarthDbChange] = {} + self._flush_tasks: dict[str, asyncio.Task[None]] = {} + self._seen_event_ids: set[int] = set() + self._seen_event_order: deque[int] = deque() + + def handle_notification(self, payload_text: str) -> bool: + try: + payload = json.loads(payload_text) + except json.JSONDecodeError: + logger.warning_event( + "Ignoring malformed Earth database change notification", + event="earth.db_changes.notification_malformed", + ) + return False + if not isinstance(payload, dict): + return False + return self.handle_payload(payload) + + def handle_payload(self, payload: dict[str, Any]) -> bool: + event_id = payload.get("event_id") + if event_id is not None: + try: + normalized_event_id = int(event_id) + except (TypeError, ValueError): + normalized_event_id = None + if normalized_event_id is not None: + if normalized_event_id in self._seen_event_ids: + return False + self._remember_event_id(normalized_event_id) + + update = build_earth_update_from_db_payload(payload) + if not update: + return False + + source = update["source"] + pending = self._pending.get(source) + if pending is None: + pending = PendingEarthDbChange( + source=source, + layers=list(update["layers"]), + table=update.get("table"), + refresh_strategy=str(update.get("refresh_strategy") or "clear_then_reload"), + entity=update.get("entity"), + action=str(update.get("action") or "database_changed"), + ) + self._pending[source] = pending + pending.add(payload) + + task = self._flush_tasks.pop(source, None) + if task and not task.done(): + task.cancel() + self._flush_tasks[source] = asyncio.create_task( + self._flush_later(source, delay_seconds=self._next_flush_delay(pending)) + ) + return True + + def _next_flush_delay(self, pending: PendingEarthDbChange) -> float: + if "DELETE" in pending.operations and pending.refresh_strategy == "clear_then_reload": + return DELETE_FAST_FLUSH_SECONDS + elapsed = max(0.0, monotonic() - pending.first_seen_monotonic) + remaining = self._max_wait_seconds - elapsed + if remaining <= 0: + return 0.0 + return min(self._debounce_seconds, remaining) + + def _remember_event_id(self, event_id: int) -> None: + self._seen_event_ids.add(event_id) + self._seen_event_order.append(event_id) + while len(self._seen_event_order) > MAX_SEEN_EVENT_IDS: + expired_event_id = self._seen_event_order.popleft() + self._seen_event_ids.discard(expired_event_id) + + async def _flush_later(self, source: str, *, delay_seconds: float) -> None: + try: + if delay_seconds > 0: + await asyncio.sleep(delay_seconds) + await self.flush_source(source) + except asyncio.CancelledError: + raise + except Exception as exc: + logger.exception_event( + "Failed to broadcast debounced Earth database change", + event="earth.db_changes.flush_failed", + context={"source": source, "error": str(exc)}, + ) + finally: + current = self._flush_tasks.get(source) + if current is asyncio.current_task(): + self._flush_tasks.pop(source, None) + + async def flush_source(self, source: str) -> None: + pending = self._pending.get(source) + if pending is None: + return + + flushed_at = datetime.now(UTC) + deleted_cache_entries = self._invalidate_cache(source) + operations = sorted(pending.operations) + payload: dict[str, Any] = { + "event": "earth.layer.changed", + "action": "database_changed", + "source": source, + "table": pending.table, + "data_type": source, + "layers": pending.layers, + "refresh_strategy": pending.refresh_strategy, + "records_processed": pending.records_processed, + "operations": operations, + "operation": operations[-1] if len(operations) == 1 else None, + "entity_keys": pending.entity_keys, + "entity_key_sample_size": len(pending.entity_keys), + "cache_entries_invalidated": deleted_cache_entries, + "first_occurred_at": pending.first_occurred_at, + "last_occurred_at": pending.last_occurred_at, + "debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000), + "total_latency_ms": self._total_latency_ms(pending, flushed_at), + "timestamp": to_iso8601_utc(flushed_at), + } + if pending.entity == "interactable": + payload.update( + { + "entity": "interactable", + "action": "deleted" if "DELETE" in pending.operations else "changed", + "ids": pending.entity_keys, + "item": None, + } + ) + await self._broadcast_earth_update(payload) + self._pending.pop(source, None) + logger.info_event( + "Broadcasted Earth database change", + event="earth.db_changes.broadcasted", + context={ + "source": source, + "layers": pending.layers, + "records_processed": pending.records_processed, + "cache_entries_invalidated": deleted_cache_entries, + "debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000), + "total_latency_ms": payload["total_latency_ms"], + }, + ) + + @staticmethod + def _total_latency_ms(pending: PendingEarthDbChange, flushed_at: datetime) -> int | None: + occurred_at = pending.first_occurred_at + if not occurred_at: + return None + try: + normalized = occurred_at.replace("Z", "+00:00") + occurred = datetime.fromisoformat(normalized) + if occurred.tzinfo is None: + occurred = occurred.replace(tzinfo=UTC) + return max(0, int((flushed_at - occurred.astimezone(UTC)).total_seconds() * 1000)) + except ValueError: + return None + + async def flush_all(self) -> None: + sources = list(self._pending) + for source in sources: + task = self._flush_tasks.pop(source, None) + if task and not task.done(): + task.cancel() + await self.flush_source(source) + + async def stop(self) -> None: + tasks = [task for task in self._flush_tasks.values() if not task.done()] + for task in tasks: + task.cancel() + if tasks: + await asyncio.gather(*tasks, return_exceptions=True) + self._flush_tasks.clear() + await self.flush_all() + + +class EarthDbChangeListener: + def __init__( + self, + *, + dsn: str, + dispatcher: EarthDbChangeDispatcher, + channel: str = EARTH_DATA_CHANGES_CHANNEL, + ) -> None: + self._dsn = normalize_asyncpg_dsn(dsn) + self._dispatcher = dispatcher + self._channel = channel + self._task: asyncio.Task[None] | None = None + self._stop_event: asyncio.Event | None = None + self._connection: asyncpg.Connection | None = None + self._loop: asyncio.AbstractEventLoop | None = None + + def start(self) -> None: + if self._task and not self._task.done(): + return + self._loop = asyncio.get_running_loop() + self._stop_event = asyncio.Event() + self._task = asyncio.create_task(self._run()) + + async def stop(self) -> None: + if self._stop_event: + self._stop_event.set() + if self._connection: + await self._connection.close() + if self._task: + await asyncio.gather(self._task, return_exceptions=True) + await self._dispatcher.stop() + + async def _run(self) -> None: + backoff_seconds = 1.0 + assert self._stop_event is not None + + while not self._stop_event.is_set(): + try: + self._connection = await asyncpg.connect(self._dsn) + await self._connection.add_listener(self._channel, self._on_notification) + logger.info_event( + "Earth database change listener connected", + event="earth.db_changes.connected", + context={"channel": self._channel}, + ) + backoff_seconds = 1.0 + while not self._stop_event.is_set(): + try: + await asyncio.wait_for( + self._stop_event.wait(), + timeout=LISTEN_KEEPALIVE_SECONDS, + ) + except asyncio.TimeoutError: + await self._poll_outbox() + except asyncio.CancelledError: + raise + except Exception as exc: + logger.exception_event( + "Earth database change listener failed", + event="earth.db_changes.listener_failed", + context={"channel": self._channel, "error": str(exc)}, + ) + try: + await asyncio.wait_for(self._stop_event.wait(), timeout=backoff_seconds) + except asyncio.TimeoutError: + pass + backoff_seconds = min(backoff_seconds * 2, 30.0) + finally: + if self._connection: + try: + await self._connection.remove_listener(self._channel, self._on_notification) + except Exception: + pass + try: + await self._connection.close() + except Exception: + pass + self._connection = None + + async def _poll_outbox(self) -> None: + if self._connection is None: + return + + rows = await self._connection.fetch( + """ + SELECT id, payload + FROM earth_data_change_events + WHERE consumed_at IS NULL + ORDER BY id + LIMIT $1 + """, + OUTBOX_POLL_LIMIT, + ) + accepted_count = 0 + consumed_ids: list[int] = [] + for row in rows: + payload = row["payload"] + if isinstance(payload, str): + try: + payload = json.loads(payload) + except json.JSONDecodeError: + consumed_ids.append(int(row["id"])) + continue + if isinstance(payload, dict): + if self._dispatcher.handle_payload(payload): + accepted_count += 1 + consumed_ids.append(int(row["id"])) + else: + consumed_ids.append(int(row["id"])) + if consumed_ids: + await self._dispatcher.flush_all() + if consumed_ids: + await self._connection.execute( + """ + UPDATE earth_data_change_events + SET consumed_at = NOW() + WHERE id = ANY($1::bigint[]) + AND consumed_at IS NULL + """, + consumed_ids, + ) + if rows: + logger.info_event( + "Polled Earth database change outbox", + event="earth.db_changes.outbox_polled", + context={"events": len(rows), "accepted": accepted_count}, + ) + + def _on_notification( + self, + _connection: asyncpg.Connection, + _pid: int, + _channel: str, + payload: str, + ) -> None: + if self._loop and self._loop.is_running(): + self._loop.call_soon_threadsafe(self._dispatcher.handle_notification, payload) + return + self._dispatcher.handle_notification(payload) + + +_dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcaster.broadcast_earth_update, + invalidate_cache=invalidate_earth_layer_cache_for_source, +) +_listener: EarthDbChangeListener | None = None + + +def start_earth_db_change_listener() -> None: + global _listener + if _listener is not None: + return + _listener = EarthDbChangeListener(dsn=settings.DATABASE_URL, dispatcher=_dispatcher) + _listener.start() + + +async def stop_earth_db_change_listener() -> None: + global _listener + if _listener is None: + await _dispatcher.stop() + return + listener = _listener + _listener = None + await listener.stop() diff --git a/backend/app/services/earth_interactables.py b/backend/app/services/earth_interactables.py new file mode 100644 index 00000000..f1b5f365 --- /dev/null +++ b/backend/app/services/earth_interactables.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any +from uuid import uuid4 + +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.core.time import to_iso8601_utc +from app.core.websocket.broadcaster import broadcaster +from app.models.earth_interactable import EarthInteractable +from app.services.earth_layer_cache import EARTH_LAYER_CACHE_PREFIX, earth_layer_cache + +INTERACTABLE_ENTITY = "interactable" +INTERACTABLE_LAYER = "interactables" + + +def normalize_interactable_id(value: str | None = None) -> str: + raw = str(value or "").strip() + return raw or f"interactable-{uuid4().hex}" + + +def serialize_interactable(record: EarthInteractable) -> dict[str, Any]: + return { + "id": record.id, + "layer": record.layer, + "kind": record.kind, + "label": record.label, + "description": record.description, + "latitude": record.latitude, + "longitude": record.longitude, + "altitude": record.altitude, + "revision": record.revision, + "properties": record.properties or {}, + "is_deleted": bool(record.is_deleted), + "created_at": to_iso8601_utc(record.created_at), + "updated_at": to_iso8601_utc(record.updated_at), + "deleted_at": to_iso8601_utc(record.deleted_at), + } + + +def interactables_to_geojson(items: list[EarthInteractable]) -> dict[str, Any]: + return { + "type": "FeatureCollection", + "features": [ + { + "type": "Feature", + "id": item.id, + "geometry": { + "type": "Point", + "coordinates": [item.longitude, item.latitude], + }, + "properties": serialize_interactable(item), + } + for item in items + if not item.is_deleted + ], + } + + +def invalidate_interactable_cache(layer: str | None = None) -> int: + layer_key = str(layer or "*").strip() or "*" + deleted = earth_layer_cache.delete_pattern( + f"{EARTH_LAYER_CACHE_PREFIX}:interactables:interactable_layer:{layer_key}*" + ) + if layer_key != "all": + deleted += earth_layer_cache.delete_pattern( + f"{EARTH_LAYER_CACHE_PREFIX}:interactables:interactable_layer:all*" + ) + return deleted + + +def build_interactable_event( + *, + action: str, + record: EarthInteractable, + include_item: bool = True, +) -> dict[str, Any]: + item = serialize_interactable(record) + return { + "entity": INTERACTABLE_ENTITY, + "action": action, + "layer": record.layer, + "layers": [INTERACTABLE_LAYER], + "ids": [record.id], + "revision": record.revision, + "changed_at": item["deleted_at"] or item["updated_at"] or to_iso8601_utc(datetime.now(UTC)), + "item": item if include_item else None, + "source": "earth_interactables", + } + + +async def publish_interactable_event(action: str, record: EarthInteractable, *, include_item: bool = True) -> None: + await broadcaster.broadcast_earth_update( + build_interactable_event(action=action, record=record, include_item=include_item) + ) + + +async def list_interactables( + db: AsyncSession, + *, + layer: str | None = None, + include_deleted: bool = False, +) -> list[EarthInteractable]: + stmt = select(EarthInteractable) + if layer: + stmt = stmt.where(EarthInteractable.layer == layer) + if not include_deleted: + stmt = stmt.where(EarthInteractable.is_deleted.is_(False)) + stmt = stmt.order_by(EarthInteractable.updated_at.desc(), EarthInteractable.id.asc()) + result = await db.execute(stmt) + return list(result.scalars().all()) diff --git a/backend/app/services/earth_layer_adapters.py b/backend/app/services/earth_layer_adapters.py new file mode 100644 index 00000000..078e6cd7 --- /dev/null +++ b/backend/app/services/earth_layer_adapters.py @@ -0,0 +1,187 @@ +"""Earth layer adapter registry for datasource-backed refresh behavior.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from sqlalchemy.ext.asyncio import AsyncSession + + +@dataclass(frozen=True) +class EarthLayerAdapter: + sources: frozenset[str] + layers: tuple[str, ...] + cache_patterns: tuple[str, ...] + tables: frozenset[str] = field(default_factory=frozenset) + derived_models: tuple[str, ...] = field(default_factory=tuple) + refresh_strategy: str = "clear_then_reload" + + +EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = ( + EarthLayerAdapter( + sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}), + tables=frozenset({"vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}), + layers=("vessels",), + cache_patterns=("vessels*", "summary*"), + derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"), + ), + EarthLayerAdapter( + sources=frozenset( + { + "telegeography_cables", + "telegeography_landing", + "telegeography_landing_points", + "telegeography_systems", + "telegeography_cable_systems", + "arcgis_cables", + "arcgis_landing_points", + "arcgis_cable_landing_relation", + "arcgis_cable_landing_relations", + "fao_landing_points", + } + ), + tables=frozenset({"collected_data"}), + layers=("cables",), + cache_patterns=("cables*", "landing-points*", "summary*"), + ), + EarthLayerAdapter( + sources=frozenset({"celestrak_tle", "spacetrack_tle"}), + tables=frozenset({"collected_data"}), + layers=("satellites",), + cache_patterns=("satellites*", "summary*"), + ), + EarthLayerAdapter( + sources=frozenset( + { + "top500", + "top500_supercomputers", + "epoch_ai_gpu", + "huggingface_models", + "huggingface_datasets", + "huggingface_spaces", + "compute_center_locations", + } + ), + tables=frozenset({"compute_center_locations"}), + layers=("computeCenters",), + cache_patterns=("compute-centers*", "summary*"), + refresh_strategy="reload", + ), + EarthLayerAdapter( + sources=frozenset( + { + "ris_live_bgp", + "bgpstream_bgp", + "iptoasn_prefix_geo", + "opengeofeed_prefix_geo", + "nro_delegated_prefix_geo", + "bgp_observations", + "bgp_anomalies", + "bgp_incidents", + "bgp_collector_locations", + } + ), + tables=frozenset({"bgp_observations", "bgp_anomalies", "bgp_incidents", "bgp_collector_locations"}), + layers=("bgp",), + cache_patterns=("bgp*", "summary*"), + derived_models=("bgp_observations", "bgp_anomalies", "bgp_incidents"), + ), + EarthLayerAdapter( + sources=frozenset({"news_live_streams"}), + tables=frozenset({"collected_data"}), + layers=("media",), + cache_patterns=("summary*",), + refresh_strategy="reload", + ), + EarthLayerAdapter( + sources=frozenset({"media_news_archive", "earth_news_items"}), + tables=frozenset({"earth_news_items"}), + layers=("news",), + cache_patterns=("summary*",), + refresh_strategy="reload", + ), + EarthLayerAdapter( + sources=frozenset({"earth_interactables"}), + tables=frozenset({"earth_interactables"}), + layers=("interactables",), + cache_patterns=("interactables*", "summary*"), + refresh_strategy="delta", + ), +) + +_ADAPTERS_BY_SOURCE = { + source: adapter + for adapter in EARTH_LAYER_ADAPTERS + for source in adapter.sources +} +_ADAPTERS_BY_TABLE = { + table: adapter + for adapter in EARTH_LAYER_ADAPTERS + for table in adapter.tables +} + + +def get_earth_layer_adapter_for_source(source: str | None) -> EarthLayerAdapter | None: + return _ADAPTERS_BY_SOURCE.get(str(source or "").strip()) + + +def get_earth_layer_adapter_for_change(table: str | None, source: str | None) -> EarthLayerAdapter | None: + table_key = str(table or "").strip() + source_key = str(source or "").strip() + if table_key and table_key != "collected_data": + adapter = _ADAPTERS_BY_TABLE.get(table_key) + if adapter is not None: + return adapter + return get_earth_layer_adapter_for_source(source_key) + + +def get_earth_update_layers_for_source(source: str | None) -> list[str]: + adapter = get_earth_layer_adapter_for_source(source) + return list(adapter.layers) if adapter else [] + + +def get_earth_update_layers_for_change(table: str | None, source: str | None) -> list[str]: + adapter = get_earth_layer_adapter_for_change(table, source) + return list(adapter.layers) if adapter else [] + + +def get_earth_refresh_strategy_for_change(table: str | None, source: str | None) -> str | None: + adapter = get_earth_layer_adapter_for_change(table, source) + return adapter.refresh_strategy if adapter else None + + +def get_earth_cache_patterns_for_source(source: str | None) -> list[str]: + adapter = get_earth_layer_adapter_for_source(source) + return list(adapter.cache_patterns) if adapter else [] + + +async def clear_derived_datasource_data(db: AsyncSession, source: str) -> dict[str, int]: + adapter = get_earth_layer_adapter_for_source(source) + if adapter is None or not adapter.derived_models: + return {} + + from app.models.bgp_anomaly import BGPAnomaly + from app.models.bgp_incident import BGPIncident + from app.models.bgp_observation import BGPObservation + from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth + + model_by_key: dict[str, Any] = { + "bgp_observations": BGPObservation, + "bgp_anomalies": BGPAnomaly, + "bgp_incidents": BGPIncident, + "ais_raw_observations": AISRawObservation, + "ais_conflict_records": AISConflictRecord, + "ais_source_health": AISSourceHealth, + } + deleted_counts: dict[str, int] = {} + for key in adapter.derived_models: + model = model_by_key.get(key) + if model is None: + continue + if key == "ais_conflict_records": + result = await db.execute(model.__table__.delete().where(model.selected_source == source)) + else: + result = await db.execute(model.__table__.delete().where(model.source == source)) + deleted_counts[key] = int(result.rowcount or 0) + return deleted_counts diff --git a/backend/app/services/earth_layer_cache.py b/backend/app/services/earth_layer_cache.py index 913f60c5..253960e8 100644 --- a/backend/app/services/earth_layer_cache.py +++ b/backend/app/services/earth_layer_cache.py @@ -268,34 +268,10 @@ def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) def invalidate_earth_layer_cache_for_source(source: str) -> int: + from app.services.earth_layer_adapters import get_earth_cache_patterns_for_source + source_key = str(source or "").strip() - patterns = { - "barentswatch_vessels": ["vessels*", "summary*"], - "aisstream_vessels": ["vessels*", "summary*"], - "telegeography_cables": ["cables*", "landing-points*", "summary*"], - "telegeography_landing": ["landing-points*", "summary*"], - "telegeography_landing_points": ["landing-points*", "summary*"], - "telegeography_systems": ["cables*", "summary*"], - "telegeography_cable_systems": ["cables*", "summary*"], - "arcgis_cables": ["cables*", "landing-points*", "summary*"], - "arcgis_landing_points": ["landing-points*", "summary*"], - "arcgis_cable_landing_relation": ["landing-points*", "summary*"], - "arcgis_cable_landing_relations": ["landing-points*", "summary*"], - "fao_landing_points": ["landing-points*", "summary*"], - "celestrak_tle": ["satellites*", "summary*"], - "spacetrack_tle": ["satellites*", "summary*"], - "top500": ["compute-centers*", "summary*"], - "top500_supercomputers": ["compute-centers*", "summary*"], - "epoch_ai_gpu": ["compute-centers*", "summary*"], - "huggingface_models": ["compute-centers*", "summary*"], - "huggingface_datasets": ["compute-centers*", "summary*"], - "huggingface_spaces": ["compute-centers*", "summary*"], - "ris_live_bgp": ["bgp*", "summary*"], - "bgpstream_bgp": ["bgp*", "summary*"], - "iptoasn_prefix_geo": ["bgp*", "summary*"], - "opengeofeed_prefix_geo": ["bgp*", "summary*"], - "nro_delegated_prefix_geo": ["bgp*", "summary*"], - }.get(source_key, []) + patterns = get_earth_cache_patterns_for_source(source_key) deleted = 0 for layer_pattern in patterns: deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}") diff --git a/backend/app/services/earth_news.py b/backend/app/services/earth_news.py index baa1080c..9de927b7 100644 --- a/backend/app/services/earth_news.py +++ b/backend/app/services/earth_news.py @@ -2,25 +2,45 @@ from __future__ import annotations import asyncio from dataclasses import dataclass, field -from datetime import UTC, datetime +from datetime import UTC, datetime, timedelta from email.utils import parsedate_to_datetime import hashlib import html import json import math import re +from time import perf_counter from typing import Any from urllib.parse import quote import xml.etree.ElementTree as ET import httpx from bs4 import BeautifulSoup +from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from app.core.countries import COUNTRY_VARIANTS_MAP, get_country_centroid, normalize_country +from app.core.enums import ( + BreakingLevel, + BreakingScope, + BreakingSource, + NewsEnrichmentStatus, + NewsImportanceLevel, + NewsMarketImpact, + NewsSourceType, + NewsTaggingSource, +) +from app.models.system_setting import SystemSetting from app.ai_tasks.prompts import EffectiveAIPrompt, get_effective_prompt from app.schemas.ai import SituationalAnalysisRequest from app.services.ai_client import AIProviderClient +from app.services.earth_news_classification import ( + apply_news_classification as _apply_news_classification, + breaking_sort_rank as _breaking_sort_rank, + highest_breaking_level as _highest_breaking_level, + normalize_breaking_level as _normalize_breaking_level_enum, + normalize_breaking_scope as _normalize_breaking_scope_enum, +) from app.services.location.resolvers.nominatim import build_default_nominatim_geocoder @@ -33,7 +53,16 @@ RSS_SUPPLEMENT_MAX_AGE_SECONDS = STALE_CACHE_MAX_AGE_SECONDS MAX_TARGET_INFERENCE_CONCURRENCY = 3 TARGET_INFERENCE_TIMEOUT_SECONDS = 6.0 DEFAULT_NEWS_LOCALE = "zh-CN" +SUPPORTED_NEWS_LOCALES = frozenset({"zh-CN", "en-US"}) NEWS_ENRICH_PROMPT_KEY = "earth.news.enrich" +EARTH_NEWS_SOURCES_CATEGORY = "earth_news_sources" +DEFAULT_NEWS_HEALTH_POLICY = { + "timeout_seconds": REQUEST_TIMEOUT, + "failure_threshold": 3, + "cooldown_minutes": 30, + "fetch_interval_minutes": 45, + "circuit_breaker": True, +} @dataclass(frozen=True) @@ -52,6 +81,19 @@ class RegionAnchor: longitude: float +@dataclass(frozen=True) +class NewsFeedEndpoint: + id: str + name: str + url: str + type: str = NewsSourceType.RSS.value + region: str = "" + enabled: bool = True + default_category: str = "other" + tags: tuple[str, ...] = () + priority: int = 100 + + @dataclass(frozen=True) class NewsFeedSource: id: str @@ -59,8 +101,16 @@ class NewsFeedSource: region: str feed_url: str homepage_url: str - source_type: str = "rss" + feed_directory_url: str = "" + source_type: str = NewsSourceType.RSS.value + feed_urls: tuple[str, ...] = () + feeds: tuple[NewsFeedEndpoint, ...] = () priority: int = 100 + enabled: bool = True + source_tags: tuple[str, ...] = () + default_category: str = "other" + importance_weight: int = 0 + health_policy: dict[str, Any] = field(default_factory=dict) @dataclass(frozen=True) @@ -87,7 +137,7 @@ class ParsedNewsItem: published_at: datetime | None content_language: str = "en" localizations: dict[str, dict[str, str]] = field(default_factory=dict) - enrichment_status: str = "pending" + enrichment_status: str = NewsEnrichmentStatus.PENDING.value enrichment_error: str | None = None enriched_at: datetime | None = None target_location: NewsTargetLocation | None = None @@ -97,6 +147,24 @@ class ParsedNewsItem: target_ai_error: str | None = None target_debug_note: str | None = None location_patch: dict[str, Any] | None = None + source_tags: list[str] = field(default_factory=list) + feed_id: str = "" + feed_type: str = NewsSourceType.RSS.value + feed_default_category: str = "other" + category: str = "other" + item_tags: list[str] = field(default_factory=list) + tagging_source: str = NewsTaggingSource.RULES.value + tagging_confidence: float = 0.0 + importance_score: int = 0 + importance_level: str = NewsImportanceLevel.LOW.value + importance_reasons: list[str] = field(default_factory=list) + market_impact: str = NewsMarketImpact.NONE.value + breaking_level: str = BreakingLevel.NONE.value + breaking_scope: str = BreakingScope.REGIONAL.value + breaking_reasons: list[str] = field(default_factory=list) + breaking_source: str = BreakingSource.RULES.value + breaking_confidence: float = 0.0 + breaking_expires_at: datetime | None = None @dataclass @@ -189,7 +257,15 @@ NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = ( region="global", feed_url="https://feeds.bbci.co.uk/news/world/rss.xml", homepage_url="https://www.bbc.com/news/world", + feeds=( + NewsFeedEndpoint(id="world", name="World", url="https://feeds.bbci.co.uk/news/world/rss.xml", default_category="politics", priority=1), + ), + source_type="rss", priority=10, + source_tags=("official_media", "business_news", "global"), + default_category="politics", + importance_weight=12, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, ), NewsFeedSource( id="dw-top", @@ -197,11 +273,304 @@ NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = ( region="europe", feed_url="https://rss.dw.com/rdf/rss-en-top", homepage_url="https://www.dw.com/en/top-stories/s-9097", + feeds=( + NewsFeedEndpoint(id="top", name="Top Stories", url="https://rss.dw.com/rdf/rss-en-top", default_category="politics", priority=1), + ), + source_type="rss", priority=20, + source_tags=("official_media", "business_news", "global", "europe"), + default_category="politics", + importance_weight=12, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, ), NewsFeedSource( - id="global-scan", - name="Global Monitor / World", + id="cnbc-business", + name="CNBC Business", + region="global", + feed_url="https://www.cnbc.com/id/10001147/device/rss/rss.html", + homepage_url="https://www.cnbc.com/business/", + feeds=( + NewsFeedEndpoint(id="business", name="Business", url="https://www.cnbc.com/id/10001147/device/rss/rss.html", default_category="business", priority=1), + ), + source_type="rss", + priority=21, + source_tags=("business_news", "global", "us"), + default_category="business", + importance_weight=16, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="bbc-business", + name="BBC Business", + region="global", + feed_url="https://feeds.bbci.co.uk/news/business/rss.xml", + homepage_url="https://www.bbc.com/news/business", + feeds=( + NewsFeedEndpoint(id="business", name="Business", url="https://feeds.bbci.co.uk/news/business/rss.xml", default_category="business", priority=1), + ), + source_type="rss", + priority=22, + source_tags=("official_media", "business_news", "global"), + default_category="business", + importance_weight=14, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="guardian-business", + name="The Guardian Business", + region="europe", + feed_url="https://www.theguardian.com/uk/business/rss", + homepage_url="https://www.theguardian.com/uk/business", + feeds=( + NewsFeedEndpoint(id="business", name="Business", url="https://www.theguardian.com/uk/business/rss", default_category="business", priority=1), + ), + source_type="rss", + priority=24, + source_tags=("business_news", "global", "europe"), + default_category="business", + importance_weight=12, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="npr-business", + name="NPR Business", + region="americas", + feed_url="https://feeds.npr.org/1006/rss.xml", + homepage_url="https://www.npr.org/sections/business/", + feeds=( + NewsFeedEndpoint(id="business", name="Business", url="https://feeds.npr.org/1006/rss.xml", default_category="business", priority=1), + ), + source_type="rss", + priority=25, + source_tags=("business_news", "global", "us"), + default_category="business", + importance_weight=12, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="marketwatch-top", + name="MarketWatch Top Stories", + region="americas", + feed_url="https://feeds.content.dowjones.io/public/rss/mw_topstories", + homepage_url="https://www.marketwatch.com/", + feeds=( + NewsFeedEndpoint(id="top", name="Top Stories", url="https://feeds.content.dowjones.io/public/rss/mw_topstories", default_category="finance", priority=1), + ), + source_type="rss", + priority=26, + source_tags=("business_news", "finance", "global", "us"), + default_category="finance", + importance_weight=12, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="techcrunch", + name="TechCrunch", + region="global", + feed_url="https://techcrunch.com/feed/", + homepage_url="https://techcrunch.com/", + feeds=( + NewsFeedEndpoint(id="main", name="Main Feed", url="https://techcrunch.com/feed/", default_category="technology", priority=1), + ), + source_type="rss", + priority=32, + source_tags=("business_news", "ecommerce", "global"), + default_category="technology", + importance_weight=8, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="retaildive", + name="Retail Dive", + region="global", + feed_url="https://www.retaildive.com/feeds/news/", + homepage_url="https://www.retaildive.com/", + feeds=( + NewsFeedEndpoint(id="news", name="News", url="https://www.retaildive.com/feeds/news/", default_category="business", priority=1), + ), + source_type="rss", + priority=34, + source_tags=("business_news", "retail", "ecommerce", "global"), + default_category="business", + importance_weight=10, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="prnewswire-retail", + name="PR Newswire Consumer Products & Retail", + region="global", + feed_url="https://www.prnewswire.com/rss/consumer-products-retail-latest-news/consumer-products-retail-latest-news-list.rss", + homepage_url="https://www.prnewswire.com/news-releases/consumer-products-retail-latest-news/", + feeds=( + NewsFeedEndpoint(id="consumer-retail", name="Consumer Products & Retail", url="https://www.prnewswire.com/rss/consumer-products-retail-latest-news/consumer-products-retail-latest-news-list.rss", default_category="business", priority=1), + ), + source_type="rss", + priority=48, + source_tags=("press_release", "retail", "ecommerce", "global"), + default_category="business", + importance_weight=-4, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "failure_threshold": 2}, + ), + NewsFeedSource( + id="36kr", + name="36氪", + region="asia-pacific", + feed_url="https://36kr.com/feed-article", + homepage_url="https://www.36kr.com/", + feed_directory_url="https://www.36kr.com/rss-center", + source_type="rss", + feeds=( + NewsFeedEndpoint(id="feed", name="综合资讯", url="https://36kr.com/feed", default_category="business", priority=1), + NewsFeedEndpoint(id="article", name="文章资讯", url="https://36kr.com/feed-article", default_category="business", priority=2), + NewsFeedEndpoint(id="newsflash", name="最新快讯", url="https://36kr.com/feed-newsflash", default_category="business", priority=3), + NewsFeedEndpoint(id="moment", name="动态内容", url="https://36kr.com/feed-moment", default_category="business", priority=4), + ), + priority=35, + source_tags=("business_news", "ecommerce", "china"), + default_category="business", + importance_weight=12, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="businesswire-ecommerce", + name="BusinessWire Electronic Commerce", + region="global", + feed_url="https://www.businesswire.com/newsroom/industry/technology/ecommerce", + homepage_url="https://www.businesswire.com/newsroom/industry/technology/ecommerce", + source_type="reference", + priority=62, + enabled=False, + source_tags=("press_release", "ecommerce", "global", "low_stability"), + default_category="ecommerce", + importance_weight=-6, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "failure_threshold": 2}, + ), + NewsFeedSource( + id="mckinsey-retail", + name="McKinsey Retail Insights", + region="global", + feed_url="https://www.mckinsey.com/industries/retail/our-insights", + homepage_url="https://www.mckinsey.com/industries/retail/our-insights", + source_type="reference", + priority=64, + enabled=False, + source_tags=("industry_insight", "retail", "global"), + default_category="business", + importance_weight=18, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 720}, + ), + NewsFeedSource( + id="deloitte-retail", + name="Deloitte Retail", + region="global", + feed_url="https://www.deloitte.com/us/en/Industries/retail/about.html", + homepage_url="https://www.deloitte.com/us/en/Industries/retail/about.html", + source_type="reference", + priority=66, + enabled=False, + source_tags=("industry_insight", "retail", "global", "us"), + default_category="business", + importance_weight=16, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 720}, + ), + NewsFeedSource( + id="us-census-ecommerce", + name="US Census Retail / Quarterly E-Commerce", + region="americas", + feed_url="https://www.census.gov/retail/index.html#ecommerce", + homepage_url="https://www.census.gov/retail/index.html#ecommerce", + source_type="reference", + priority=18, + enabled=False, + source_tags=("official_data", "ecommerce", "retail", "us"), + default_category="ecommerce", + importance_weight=34, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 1440}, + ), + NewsFeedSource( + id="mofcom-data", + name="商务数据中心", + region="asia-pacific", + feed_url="https://data.mofcom.gov.cn/index.shtml", + homepage_url="https://data.mofcom.gov.cn/index.shtml", + source_type="reference", + priority=18, + enabled=False, + source_tags=("official_data", "business_news", "china"), + default_category="business", + importance_weight=34, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 1440}, + ), + NewsFeedSource( + id="mofcom-ecommerce", + name="商务部电商动态", + region="asia-pacific", + feed_url="https://www.mofcom.gov.cn/", + homepage_url="https://www.mofcom.gov.cn/", + source_type="reference", + priority=19, + enabled=False, + source_tags=("official_data", "ecommerce", "china"), + default_category="ecommerce", + importance_weight=34, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 1440}, + ), + NewsFeedSource( + id="stats-china-online-retail", + name="国家统计局数据发布", + region="asia-pacific", + feed_url="https://www.stats.gov.cn/sj/zxfb/rss.xml", + homepage_url="https://www.stats.gov.cn/sj/zxfb/", + feeds=( + NewsFeedEndpoint(id="release", name="数据发布", url="https://www.stats.gov.cn/sj/zxfb/rss.xml", default_category="ecommerce", priority=1), + ), + source_type="rss", + priority=19, + source_tags=("official_data", "ecommerce", "retail", "china"), + default_category="ecommerce", + importance_weight=36, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 1440}, + ), + NewsFeedSource( + id="china-ecommerce-logistics-index", + name="电商物流指数", + region="asia-pacific", + feed_url="https://www.gov.cn/", + homepage_url="https://www.gov.cn/", + source_type="reference", + priority=20, + enabled=False, + source_tags=("official_data", "ecommerce", "retail", "china"), + default_category="ecommerce", + importance_weight=36, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, "fetch_interval_minutes": 1440}, + ), + NewsFeedSource( + id="ebrun", + name="亿邦动力", + region="asia-pacific", + feed_url="https://www.ebrun.com/rss/news_b2c.xml", + homepage_url="https://www.ebrun.com/", + feed_directory_url="https://www.ebrun.com/rss/", + source_type="rss", + feeds=( + NewsFeedEndpoint(id="b2c", name="B2C", url="https://www.ebrun.com/rss/news_b2c.xml", default_category="ecommerce", priority=1), + NewsFeedEndpoint(id="b2b", name="B2B", url="https://www.ebrun.com/rss/news_b2b.xml", default_category="ecommerce", priority=2), + NewsFeedEndpoint(id="retail", name="零售", url="https://www.ebrun.com/rss/news_retail.xml", default_category="ecommerce", priority=3), + NewsFeedEndpoint(id="o2o", name="O2O", url="https://www.ebrun.com/rss/news_o2o.xml", default_category="ecommerce", priority=4), + NewsFeedEndpoint(id="service", name="服务", url="https://www.ebrun.com/rss/news_service.xml", default_category="ecommerce", priority=5), + NewsFeedEndpoint(id="data", name="数据", url="https://www.ebrun.com/rss/news_data.xml", default_category="ecommerce", priority=6), + NewsFeedEndpoint(id="policy", name="政策", url="https://www.ebrun.com/rss/news_policy.xml", default_category="ecommerce", priority=7), + ), + priority=38, + source_tags=("business_news", "ecommerce", "retail", "china"), + default_category="ecommerce", + importance_weight=14, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, + ), + NewsFeedSource( + id="google-news", + name="Google News", region="global", feed_url=_google_news_feed( REGION_PROFILES["global"].query, @@ -210,66 +579,88 @@ NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = ( ceid="US:en", ), homepage_url="https://news.google.com/", - source_type="aggregated", - priority=30, - ), - NewsFeedSource( - id="google-americas", - name="Global Monitor / Americas", - region="americas", - feed_url=_google_news_feed( - REGION_PROFILES["americas"].query, - hl="en-US", - gl="US", - ceid="US:en", + feed_directory_url="https://news.google.com/rss", + feeds=( + NewsFeedEndpoint(id="world", name="全球", url=_google_news_feed(REGION_PROFILES["global"].query, hl="en-US", gl="US", ceid="US:en"), type="aggregated", region="global", default_category="politics", priority=1), + NewsFeedEndpoint(id="americas", name="美洲", url=_google_news_feed(REGION_PROFILES["americas"].query, hl="en-US", gl="US", ceid="US:en"), type="aggregated", region="americas", default_category="politics", priority=2), + NewsFeedEndpoint(id="europe", name="欧洲", url=_google_news_feed(REGION_PROFILES["europe"].query, hl="en-GB", gl="GB", ceid="GB:en"), type="aggregated", region="europe", default_category="politics", priority=3), + NewsFeedEndpoint(id="middle-east-africa", name="中东与非洲", url=_google_news_feed(REGION_PROFILES["middle-east-africa"].query, hl="en-US", gl="US", ceid="US:en"), type="aggregated", region="middle-east-africa", default_category="politics", priority=4), + NewsFeedEndpoint(id="asia-pacific", name="亚太", url=_google_news_feed(REGION_PROFILES["asia-pacific"].query, hl="en-SG", gl="SG", ceid="SG:en"), type="aggregated", region="asia-pacific", default_category="politics", priority=5), ), - homepage_url="https://news.google.com/", source_type="aggregated", - priority=40, - ), - NewsFeedSource( - id="google-europe", - name="Global Monitor / Europe", - region="europe", - feed_url=_google_news_feed( - REGION_PROFILES["europe"].query, - hl="en-GB", - gl="GB", - ceid="GB:en", - ), - homepage_url="https://news.google.com/", - source_type="aggregated", - priority=40, - ), - NewsFeedSource( - id="google-mea", - name="Global Monitor / MEA", - region="middle-east-africa", - feed_url=_google_news_feed( - REGION_PROFILES["middle-east-africa"].query, - hl="en-US", - gl="US", - ceid="US:en", - ), - homepage_url="https://news.google.com/", - source_type="aggregated", - priority=40, - ), - NewsFeedSource( - id="google-apac", - name="Global Monitor / APAC", - region="asia-pacific", - feed_url=_google_news_feed( - REGION_PROFILES["asia-pacific"].query, - hl="en-SG", - gl="SG", - ceid="SG:en", - ), - homepage_url="https://news.google.com/", - source_type="aggregated", - priority=40, + priority=90, + source_tags=("aggregated", "business_news", "global", "europe", "low_stability"), + default_category="politics", + importance_weight=-2, + health_policy=DEFAULT_NEWS_HEALTH_POLICY, ), ) +DEFAULT_NEWS_SOURCE_BY_ID: dict[str, NewsFeedSource] = {source.id: source for source in NEWS_FEED_SOURCES} +LEGACY_GOOGLE_NEWS_SOURCE_IDS = { + "global-scan", + "google-americas", + "google-europe", + "google-mea", + "google-apac", +} + +DEFAULT_SOURCE_TAGS: tuple[dict[str, Any], ...] = ( + {"key": "official_media", "label": "官方媒体", "color": "#2563eb", "enabled": True, "sort_order": 10}, + {"key": "official_data", "label": "官方数据", "color": "#059669", "enabled": True, "sort_order": 20}, + {"key": "business_news", "label": "商业新闻", "color": "#0f766e", "enabled": True, "sort_order": 30}, + {"key": "ecommerce", "label": "电商", "color": "#7c3aed", "enabled": True, "sort_order": 40}, + {"key": "finance", "label": "金融", "color": "#1d4ed8", "enabled": True, "sort_order": 45}, + {"key": "retail", "label": "零售", "color": "#ea580c", "enabled": True, "sort_order": 50}, + {"key": "logistics", "label": "物流", "color": "#16a34a", "enabled": True, "sort_order": 55}, + {"key": "industry_insight", "label": "行业洞察", "color": "#0891b2", "enabled": True, "sort_order": 60}, + {"key": "press_release", "label": "企业公告", "color": "#64748b", "enabled": True, "sort_order": 70}, + {"key": "china", "label": "中国", "color": "#dc2626", "enabled": True, "sort_order": 80}, + {"key": "global", "label": "全球", "color": "#475569", "enabled": True, "sort_order": 90}, + {"key": "us", "label": "美国", "color": "#1d4ed8", "enabled": True, "sort_order": 100}, + {"key": "europe", "label": "欧洲", "color": "#0284c7", "enabled": True, "sort_order": 110}, + {"key": "aggregated", "label": "聚合源", "color": "#9333ea", "enabled": True, "sort_order": 120}, + {"key": "low_stability", "label": "低稳定性", "color": "#f59e0b", "enabled": True, "sort_order": 130}, +) + +DEFAULT_CATEGORIES: tuple[dict[str, Any], ...] = ( + {"key": "politics", "label": "政治", "color": "#2563eb", "enabled": True, "sort_order": 10, "keywords": ["election", "government", "minister", "parliament", "sanction", "diplomatic", "policy", "summit", "选举", "政府", "制裁", "外交", "政策"]}, + {"key": "business", "label": "商业", "color": "#0f766e", "enabled": True, "sort_order": 20, "keywords": ["market", "company", "earnings", "trade", "supply chain", "merger", "retail", "consumer", "商业", "公司", "贸易", "供应链", "消费", "零售"]}, + {"key": "ecommerce", "label": "电商", "color": "#7c3aed", "enabled": True, "sort_order": 30, "keywords": ["e-commerce", "ecommerce", "online retail", "gmv", "marketplace", "shopify", "amazon", "tiktok shop", "网上零售", "电商", "直播电商", "跨境电商", "订单量", "物流指数", "履约"]}, + {"key": "finance", "label": "金融", "color": "#0369a1", "enabled": True, "sort_order": 40, "keywords": ["stock", "bond", "inflation", "central bank", "rate cut", "rate hike", "bank", "金融", "股市", "通胀", "央行", "利率"]}, + {"key": "sports", "label": "体育", "color": "#16a34a", "enabled": True, "sort_order": 50, "keywords": ["football", "basketball", "tennis", "match", "league", "world cup", "olympics", "体育", "足球", "篮球", "世界杯", "奥运"]}, + {"key": "technology", "label": "科技", "color": "#0891b2", "enabled": True, "sort_order": 60, "keywords": ["ai", "semiconductor", "chip", "cyber", "satellite", "software", "科技", "人工智能", "半导体", "芯片", "网络安全"]}, + {"key": "military", "label": "军事", "color": "#4b5563", "enabled": True, "sort_order": 70, "keywords": ["military", "missile", "airstrike", "defense", "warship", "军事", "导弹", "空袭", "防务", "军舰"]}, + {"key": "disaster", "label": "灾害", "color": "#dc2626", "enabled": True, "sort_order": 80, "keywords": ["earthquake", "flood", "wildfire", "typhoon", "hurricane", "灾害", "地震", "洪水", "山火", "台风"]}, + {"key": "energy", "label": "能源", "color": "#ca8a04", "enabled": True, "sort_order": 90, "keywords": ["oil", "gas", "opec", "energy", "power grid", "能源", "石油", "天然气", "电网"]}, + {"key": "society", "label": "社会", "color": "#64748b", "enabled": True, "sort_order": 100, "keywords": ["health", "education", "crime", "migration", "社会", "医疗", "教育", "犯罪", "移民"]}, + {"key": "culture", "label": "文化", "color": "#db2777", "enabled": True, "sort_order": 110, "keywords": ["film", "music", "art", "culture", "文化", "电影", "音乐", "艺术"]}, + {"key": "other", "label": "其他", "color": "#64748b", "enabled": True, "sort_order": 999, "keywords": []}, +) +ALLOWED_NEWS_CATEGORY_KEYS = tuple(item["key"] for item in DEFAULT_CATEGORIES) + +DEFAULT_ITEM_TAG_RULES: tuple[dict[str, Any], ...] = ( + {"key": "cross_border_ecommerce", "label": "跨境电商", "category": "ecommerce", "keywords": ["cross-border e-commerce", "cross border ecommerce", "跨境电商"]}, + {"key": "live_commerce", "label": "直播电商", "category": "ecommerce", "keywords": ["live commerce", "livestream shopping", "直播电商", "直播带货"]}, + {"key": "retail_data", "label": "零售数据", "category": "ecommerce", "keywords": ["online retail sales", "retail sales", "网上零售额", "社零", "社会消费品零售总额"]}, + {"key": "logistics_fulfillment", "label": "物流履约", "category": "ecommerce", "keywords": ["logistics", "fulfillment", "delivery", "物流指数", "履约", "配送"]}, + {"key": "platform_governance", "label": "平台治理", "category": "ecommerce", "keywords": ["platform regulation", "marketplace rules", "平台治理", "平台监管"]}, + {"key": "ai", "label": "AI", "category": "technology", "keywords": ["ai", "artificial intelligence", "人工智能", "大模型"]}, + {"key": "semiconductor", "label": "半导体", "category": "technology", "keywords": ["semiconductor", "chip", "半导体", "芯片"]}, + {"key": "election", "label": "选举", "category": "politics", "keywords": ["election", "vote", "campaign", "选举", "投票"]}, + {"key": "oil_price", "label": "油价", "category": "energy", "keywords": ["oil price", "crude", "油价", "原油"]}, + {"key": "football", "label": "足球", "category": "sports", "keywords": ["football", "soccer", "premier league", "足球"]}, + {"key": "supply_chain", "label": "供应链", "category": "business", "keywords": ["supply chain", "供应链"]}, +) + +def default_earth_news_sources_payload() -> dict[str, Any]: + return { + "cache_version": 1, + "source_tags": [dict(item) for item in DEFAULT_SOURCE_TAGS], + "categories": [dict(item) for item in DEFAULT_CATEGORIES], + "item_tag_rules": [dict(item) for item in DEFAULT_ITEM_TAG_RULES], + "sources": [serialize_news_source_config(source) for source in NEWS_FEED_SOURCES], + "health": {}, + } _REGION_CACHE: dict[str, CachedRegionFeed] = {} @@ -393,6 +784,68 @@ def _normalize_localizations(value: Any) -> dict[str, dict[str, str]]: return normalized +def _is_chinese_language(language: str | None) -> bool: + return str(language or "").lower() in {"zh", "zh-cn", "zh-hans", "chinese"} + + +def _is_english_language(language: str | None) -> bool: + return str(language or "").lower() in {"en", "en-us", "english"} + + +def _contains_cjk_text(value: str) -> bool: + return bool(re.search(r"[\u3400-\u9fff]", value or "")) + + +def _detect_content_language( + *, + title: str, + summary: str, + source: NewsFeedSource, + feed: NewsFeedEndpoint | None = None, +) -> str: + text = f"{title}\n{summary}" + if _contains_cjk_text(text): + return "zh-CN" + identity = {source.id.lower(), *(tag.lower() for tag in source.source_tags)} + if feed is not None: + identity.add(feed.id.lower()) + identity.update(tag.lower() for tag in feed.tags) + if identity & {"china", "chinese", "36kr", "ebrun", "stats-china-online-retail"}: + return "zh-CN" + return "en" + + +def _source_language_localizations( + *, + title: str, + summary: str, + content_language: str, +) -> dict[str, dict[str, str]]: + if _is_chinese_language(content_language): + return {"zh-CN": {"title": title, "summary": summary or title}} + return {} + + +def _target_localization_locale(item: ParsedNewsItem) -> str: + return "en-US" if _is_chinese_language(item.content_language) else DEFAULT_NEWS_LOCALE + + +def _target_locale_schema(locale: str) -> dict[str, dict[str, str]]: + if locale == "en-US": + return { + "en-US": { + "title": "faithful English title", + "summary": "one-sentence newswire-style English lead summary", + } + } + return { + "zh-CN": { + "title": "faithful Simplified Chinese title", + "summary": "one-sentence newswire-style Simplified Chinese lead summary", + } + } + + def _get_locale_text( item: ParsedNewsItem, key: str, @@ -404,16 +857,35 @@ def _get_locale_text( value = _coerce_str(localized.get(key)) if value: return value + if locale == "zh-CN" and _is_chinese_language(item.content_language): + return item.title if key == "title" else item.summary + if locale == "en-US" and _is_english_language(item.content_language): + return item.title if key == "title" else item.summary + fallback = item.localizations.get(DEFAULT_NEWS_LOCALE) + if isinstance(fallback, dict): + value = _coerce_str(fallback.get(key)) + if value: + if locale == "en-US" and _contains_cjk_text(value): + return "" + return value return "" -def _has_default_localization(item: ParsedNewsItem) -> bool: - localized = item.localizations.get(DEFAULT_NEWS_LOCALE) +def _has_localization(item: ParsedNewsItem, locale: str) -> bool: + localized = item.localizations.get(locale) if not isinstance(localized, dict): return False return bool(_coerce_str(localized.get("title")) and _coerce_str(localized.get("summary"))) +def _has_default_localization(item: ParsedNewsItem) -> bool: + return _has_localization(item, DEFAULT_NEWS_LOCALE) + + +def _has_required_localization(item: ParsedNewsItem) -> bool: + return _has_localization(item, _target_localization_locale(item)) + + def apply_enrichment_patch_to_item( item: ParsedNewsItem, patch: dict[str, Any], @@ -630,16 +1102,21 @@ async def _infer_news_enrichment( item.enrichment_error = None prompt = prompt or await get_effective_prompt(None, NEWS_ENRICH_PROMPT_KEY) + target_locale = _target_localization_locale(item) + target_locale_name = "English" if target_locale == "en-US" else "Simplified Chinese" request = SituationalAnalysisRequest( - title="Enrich Earth news item with event location and zh-CN content", + title=f"Enrich Earth news item with event location and {target_locale} content", objective=prompt.prompt, system_prompt=prompt.system_prompt or None, context={ "news_item": { "title": item.title, "summary": item.summary, + "content_language": item.content_language, "source": item.source, "feed_name": item.feed_name, + "feed_id": item.feed_id, + "feed_type": item.feed_type, "feed_region": item.feed_region, "url": item.url, "published_at": ( @@ -658,20 +1135,16 @@ async def _infer_news_enrichment( "confidence": "number from 0 to 1", "reasoning_summary": "short string", }, - "localizations": { - "zh-CN": { - "title": "faithful Simplified Chinese title", - "summary": "one-sentence newswire-style Simplified Chinese lead summary", - } - }, + "localizations": _target_locale_schema(target_locale), }, }, constraints=[ "Return only strict JSON. Do not wrap it in markdown.", "For localizations, do not add facts that are absent from the RSS headline, description, source, or date.", - "Write zh-CN summary as one concise newswire-style sentence, like a breaking-news lead.", + f"Write {target_locale} summary as one concise {target_locale_name} newswire-style sentence, like a breaking-news lead.", "If the RSS description is thin, write a conservative one-sentence summary that says only what is supported.", - "Keep zh-CN summary factual, non-promotional, and avoid colon-heavy keyword labels.", + f"Keep {target_locale} summary factual, non-promotional, and avoid colon-heavy keyword labels.", + "If the source content is Chinese and target locale is en-US, translate faithfully into English instead of rewriting the story.", "Prefer the event location, not the newsroom or publisher headquarters.", "When a country visit or summit is the clear topic but the city is omitted, use the most likely host city only if it is broadly public knowledge.", "Use null for unknown fields instead of inventing details.", @@ -724,7 +1197,7 @@ async def _infer_news_enrichment( item.target_ai_error = None item.target_debug_note = f"ai inferred {target.label}" - item.localizations = localizations + item.localizations = {**dict(item.localizations or {}), **localizations} if localizations and item.target_ai_status in {"success", "skipped_text_hint"}: item.enrichment_status = "success" item.enrichment_error = None @@ -764,11 +1237,517 @@ async def _enrich_items_with_target_locations( def get_sources_for_region(region: str) -> list[NewsFeedSource]: return sorted( - [source for source in NEWS_FEED_SOURCES if source.region in {"global", region}], + [ + source + for source in NEWS_FEED_SOURCES + if source.enabled + and source.source_type in {"rss", "atom", "aggregated"} + and _source_matches_active_region(source, region) + ], key=lambda source: (source.priority, source.name), ) +def _clean_feed_urls(*values: Any) -> tuple[str, ...]: + urls: list[str] = [] + for value in values: + if isinstance(value, str): + parts = re.split(r"[\n,]+", value) + elif isinstance(value, (list, tuple)): + parts = [str(item) for item in value] + else: + parts = [] + for part in parts: + url = str(part or "").strip() + if url and url not in urls: + urls.append(url) + return tuple(urls) + + +def _clean_feed_tags(value: Any) -> tuple[str, ...]: + if isinstance(value, str): + parts = re.split(r"[\n,，]+", value) + elif isinstance(value, (list, tuple)): + parts = [str(item) for item in value] + else: + parts = [] + tags: list[str] = [] + for part in parts: + tag = str(part or "").strip() + if tag and tag not in tags: + tags.append(tag) + return tuple(tags) + + +def _slug_feed_id(value: str, fallback: str) -> str: + text_value = str(value or "").strip().lower() + slug = re.sub(r"[^a-z0-9_-]+", "-", text_value).strip("-") + return slug or fallback + + +def _feed_from_config(raw: Any, *, source: NewsFeedSource | None = None, index: int = 0) -> NewsFeedEndpoint | None: + if not isinstance(raw, dict): + return None + url = str(raw.get("url") or raw.get("feed_url") or "").strip() + if not url: + return None + feed_id = _slug_feed_id(str(raw.get("id") or raw.get("key") or raw.get("name") or ""), f"feed-{index + 1}") + fallback_type = source.source_type if source else "rss" + feed_type = str(raw.get("type") or raw.get("source_type") or fallback_type).strip().lower() or "rss" + default_category = str(raw.get("default_category") or (source.default_category if source else "other") or "other").strip() or "other" + try: + priority = int(raw.get("priority", index + 1)) + except (TypeError, ValueError): + priority = index + 1 + return NewsFeedEndpoint( + id=feed_id, + name=str(raw.get("name") or raw.get("label") or feed_id).strip() or feed_id, + url=url, + type=feed_type, + region=str(raw.get("region") or (source.region if source else "") or "").strip(), + enabled=raw.get("enabled") is not False and feed_type in {"rss", "atom", "aggregated"}, + default_category=default_category, + tags=_clean_feed_tags(raw.get("tags")), + priority=priority, + ) + + +def _source_feed_urls(source: NewsFeedSource) -> tuple[str, ...]: + return tuple(feed.url for feed in _source_feeds(source)) + + +def _source_feeds(source: NewsFeedSource) -> tuple[NewsFeedEndpoint, ...]: + if source.source_type == "reference": + return tuple() + feeds = tuple(feed for feed in source.feeds if feed.url) + if feeds: + return feeds + urls = _clean_feed_urls(source.feed_urls, source.feed_url) + return tuple( + NewsFeedEndpoint( + id=f"feed-{index + 1}", + name=source.name if len(urls) == 1 else f"{source.name} {index + 1}", + url=url, + type=source.source_type, + region=source.region, + enabled=source.enabled and source.source_type in {"rss", "atom", "aggregated"}, + default_category=source.default_category, + priority=index + 1, + ) + for index, url in enumerate(urls) + ) + + +def _feed_matches_active_region(feed: NewsFeedEndpoint, active_region: str | None) -> bool: + return ( + active_region is None + or active_region == "global" + or not feed.region + or feed.region in {"global", active_region} + ) + + +def _source_matches_active_region(source: NewsFeedSource, active_region: str) -> bool: + return active_region == "global" or source.region in {"global", active_region} + + +def _expected_feed_keys(sources: list[NewsFeedSource], *, active_region: str) -> set[tuple[str, str]]: + return { + (source.id, feed.id) + for source in sources + for feed in _source_feeds(source) + if source.enabled + and source.source_type in {"rss", "atom", "aggregated"} + and feed.enabled + and feed.type in {"rss", "atom", "aggregated"} + and _feed_matches_active_region(feed, active_region) + } + + +def _serialize_feed_endpoint(feed: NewsFeedEndpoint) -> dict[str, Any]: + return { + "id": feed.id, + "name": feed.name, + "url": feed.url, + "type": feed.type, + "region": feed.region, + "enabled": feed.enabled and feed.type in {"rss", "atom", "aggregated"}, + "default_category": feed.default_category, + "tags": list(feed.tags), + "priority": feed.priority, + } + + +def _should_repair_builtin_source_feeds( + source_id: str, + *, + raw_feeds: list[Any], + feed_urls: tuple[str, ...], + feed_url: str, + homepage_url: str, + feed_directory_url: str, +) -> bool: + default_source = DEFAULT_NEWS_SOURCE_BY_ID.get(source_id) + if not default_source or not default_source.feeds: + return False + if not raw_feeds: + return True + + current_urls: set[str] = set(feed_urls) + if feed_url: + current_urls.add(feed_url) + for raw_feed in raw_feeds: + if isinstance(raw_feed, dict): + current_url = str(raw_feed.get("url") or raw_feed.get("feed_url") or "").strip() + if current_url: + current_urls.add(current_url) + + non_feed_urls = { + url + for url in ( + homepage_url, + feed_directory_url, + default_source.homepage_url, + default_source.feed_directory_url, + ) + if url + } + return bool(current_urls & non_feed_urls) + + +def serialize_news_source_config(source: NewsFeedSource) -> dict[str, Any]: + feeds = _source_feeds(source) + feed_urls = tuple(feed.url for feed in feeds) + return { + "id": source.id, + "name": source.name, + "region": source.region, + "feed_url": source.feed_url or (feed_urls[0] if feed_urls else ""), + "feed_urls": list(feed_urls), + "feeds": [_serialize_feed_endpoint(feed) for feed in feeds], + "homepage_url": source.homepage_url, + "feed_directory_url": source.feed_directory_url, + "source_type": source.source_type, + "priority": source.priority, + "enabled": source.enabled and source.source_type in {"rss", "atom", "aggregated"}, + "source_tags": list(source.source_tags), + "default_category": source.default_category, + "importance_weight": source.importance_weight, + "health_policy": dict(source.health_policy or DEFAULT_NEWS_HEALTH_POLICY), + } + + +def _source_from_config(payload: dict[str, Any]) -> NewsFeedSource | None: + source_id = str(payload.get("id") or "").strip() + name = str(payload.get("name") or "").strip() + source_type = str(payload.get("source_type") or payload.get("type") or "rss").strip().lower() or "rss" + default_source = DEFAULT_NEWS_SOURCE_BY_ID.get(source_id) + homepage_url = str(payload.get("homepage_url") or "").strip() + feed_directory_url = str(payload.get("feed_directory_url") or "").strip() + raw_feeds = payload.get("feeds") if isinstance(payload.get("feeds"), list) else [] + feed_urls = _clean_feed_urls(payload.get("feed_urls"), payload.get("feed_url")) + feed_url = str(payload.get("feed_url") or "").strip() or (feed_urls[0] if feed_urls else "") + enabled_override: bool | None = None + if default_source: + homepage_url = homepage_url or default_source.homepage_url + feed_directory_url = feed_directory_url or default_source.feed_directory_url + if source_type == "reference" and default_source.source_type in {"rss", "atom", "aggregated"}: + source_type = default_source.source_type + if not raw_feeds: + enabled_override = default_source.enabled + if _should_repair_builtin_source_feeds( + source_id, + raw_feeds=raw_feeds, + feed_urls=feed_urls, + feed_url=feed_url, + homepage_url=homepage_url, + feed_directory_url=feed_directory_url, + ): + raw_feeds = [_serialize_feed_endpoint(feed) for feed in default_source.feeds] + feed_urls = tuple(feed.url for feed in default_source.feeds) + feed_url = default_source.feed_url or (feed_urls[0] if feed_urls else "") + if not feed_url and source_type == "reference": + feed_url = homepage_url + feed_urls = _clean_feed_urls(feed_url) + if not source_id or not name or (not feed_url and not raw_feeds): + return None + try: + priority = int(payload.get("priority", default_source.priority if default_source else 100)) + except (TypeError, ValueError): + priority = default_source.priority if default_source else 100 + try: + importance_weight = int(payload.get("importance_weight", default_source.importance_weight if default_source else 0)) + except (TypeError, ValueError): + importance_weight = default_source.importance_weight if default_source else 0 + raw_tags = payload.get("source_tags") + source_tags = ( + tuple(str(tag).strip() for tag in raw_tags if str(tag).strip()) + if isinstance(raw_tags, list) + else (default_source.source_tags if default_source else ()) + ) + health_policy = payload.get("health_policy") if isinstance(payload.get("health_policy"), dict) else {} + base_source = NewsFeedSource( + id=source_id, + name=name, + region=str(payload.get("region") or "global").strip() or "global", + feed_url=feed_url, + homepage_url=homepage_url, + feed_directory_url=feed_directory_url, + source_type=source_type, + priority=priority, + enabled=(enabled_override if enabled_override is not None else payload.get("enabled") is not False) + and source_type in {"rss", "atom", "aggregated"}, + source_tags=source_tags, + default_category=str(payload.get("default_category") or (default_source.default_category if default_source else "other") or "other").strip() or "other", + importance_weight=importance_weight, + health_policy={**DEFAULT_NEWS_HEALTH_POLICY, **health_policy}, + ) + feeds = tuple( + feed + for index, raw_feed in enumerate(raw_feeds) + for feed in [_feed_from_config(raw_feed, source=base_source, index=index)] + if feed is not None + ) + return NewsFeedSource( + **{ + **base_source.__dict__, + "feed_urls": feed_urls or ((feed_url,) if feed_url else tuple()), + "feeds": feeds, + } + ) + + +def _merge_legacy_google_news_sources(sources: list[dict[str, Any]]) -> list[dict[str, Any]]: + has_legacy_google = any(str(source.get("id") or "") in LEGACY_GOOGLE_NEWS_SOURCE_IDS for source in sources) + if not has_legacy_google: + return sources + + google_default = DEFAULT_NEWS_SOURCE_BY_ID.get("google-news") + if google_default is None: + return sources + + merged = [source for source in sources if str(source.get("id") or "") not in LEGACY_GOOGLE_NEWS_SOURCE_IDS] + if not any(str(source.get("id") or "") == "google-news" for source in merged): + merged.append(serialize_news_source_config(google_default)) + return sorted(merged, key=lambda source: (int(source.get("priority") or 100), str(source.get("name") or ""))) + + +def normalize_earth_news_sources_payload(payload: dict[str, Any] | None) -> dict[str, Any]: + defaults = default_earth_news_sources_payload() + if not isinstance(payload, dict): + return defaults + + normalized = { + "cache_version": int(payload.get("cache_version") or defaults["cache_version"]), + "source_tags": payload.get("source_tags") if isinstance(payload.get("source_tags"), list) else defaults["source_tags"], + "categories": payload.get("categories") if isinstance(payload.get("categories"), list) else defaults["categories"], + "item_tag_rules": payload.get("item_tag_rules") if isinstance(payload.get("item_tag_rules"), list) else defaults["item_tag_rules"], + "health": payload.get("health") if isinstance(payload.get("health"), dict) else {}, + "sources": [], + } + for source in payload.get("sources") if isinstance(payload.get("sources"), list) else defaults["sources"]: + if isinstance(source, dict): + coerced = _source_from_config(source) + if coerced: + normalized["sources"].append(serialize_news_source_config(coerced)) + if not normalized["sources"]: + normalized["sources"] = defaults["sources"] + else: + normalized["sources"] = _merge_legacy_google_news_sources(normalized["sources"]) + return normalized + + +async def get_earth_news_sources_payload(db: AsyncSession | None) -> dict[str, Any]: + if db is None: + payload = default_earth_news_sources_payload() + payload["is_default"] = True + return payload + result = await db.execute(select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_SOURCES_CATEGORY)) + record = result.scalar_one_or_none() + payload = normalize_earth_news_sources_payload(record.payload if record else None) + payload["is_default"] = record is None + return payload + + +async def save_earth_news_sources_payload(db: AsyncSession, payload: dict[str, Any]) -> dict[str, Any]: + normalized = normalize_earth_news_sources_payload(payload) + normalized["cache_version"] = int(normalized.get("cache_version") or 1) + 1 + result = await db.execute(select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_SOURCES_CATEGORY)) + record = result.scalar_one_or_none() + if record is None: + record = SystemSetting(category=EARTH_NEWS_SOURCES_CATEGORY, payload=normalized) + db.add(record) + else: + record.payload = normalized + await db.commit() + await db.refresh(record) + clear_earth_news_region_cache() + return {**normalize_earth_news_sources_payload(record.payload), "is_default": False} + + +async def reset_earth_news_sources_payload(db: AsyncSession) -> dict[str, Any]: + result = await db.execute(select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_SOURCES_CATEGORY)) + record = result.scalar_one_or_none() + if record is not None: + await db.delete(record) + await db.commit() + clear_earth_news_region_cache() + payload = default_earth_news_sources_payload() + payload["is_default"] = True + return payload + + +async def record_earth_news_source_health( + db: AsyncSession | None, + source_id: str, + health: dict[str, Any], +) -> None: + if db is None or not source_id or not callable(getattr(db, "execute", None)): + return + result = await db.execute(select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_SOURCES_CATEGORY)) + record = result.scalar_one_or_none() + payload = normalize_earth_news_sources_payload(record.payload if record else None) + payload_health = payload.get("health") if isinstance(payload.get("health"), dict) else {} + payload_health[source_id] = health + payload["health"] = payload_health + if record is None: + record = SystemSetting(category=EARTH_NEWS_SOURCES_CATEGORY, payload=payload) + db.add(record) + else: + record.payload = payload + await db.commit() + + +async def record_earth_news_sources_health( + db: AsyncSession | None, + health_by_source: dict[str, dict[str, Any]], +) -> None: + if db is None or not health_by_source or not callable(getattr(db, "execute", None)): + return + result = await db.execute(select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_SOURCES_CATEGORY)) + record = result.scalar_one_or_none() + payload = normalize_earth_news_sources_payload(record.payload if record else None) + payload_health = payload.get("health") if isinstance(payload.get("health"), dict) else {} + payload_health.update(health_by_source) + payload["health"] = payload_health + if record is None: + record = SystemSetting(category=EARTH_NEWS_SOURCES_CATEGORY, payload=payload) + db.add(record) + else: + record.payload = payload + await db.commit() + + +async def get_configured_sources_for_region(db: AsyncSession | None, region: str) -> list[NewsFeedSource]: + if db is None: + return get_sources_for_region(region) + payload = await get_earth_news_sources_payload(db) + sources = [ + source + for raw in payload.get("sources", []) + if isinstance(raw, dict) + for source in [_source_from_config(raw)] + if source + ] + enabled = [ + source + for source in sources + if source.enabled + and source.source_type in {"rss", "atom", "aggregated"} + and _source_matches_active_region(source, region) + ] + return sorted(enabled, key=lambda source: (source.priority, source.name)) + + +def _source_health_result( + source: NewsFeedSource, + *, + ok: bool, + count: int, + error: str | None = None, + status: str | None = None, + status_code: int | None = None, + content_type: str | None = None, + latency_ms: int | None = None, + final_url: str | None = None, + feed: NewsFeedEndpoint | None = None, +) -> dict[str, Any]: + return { + "source_id": source.id, + "feed_id": feed.id if feed else "", + "feed_name": feed.name if feed else "", + "feed_type": feed.type if feed else source.source_type, + "feed_url": feed.url if feed else source.feed_url, + "ok": ok, + "status": status or ("ok" if ok else "failed"), + "count": int(count or 0), + "item_count": int(count or 0), + "error": error, + "status_code": status_code, + "content_type": content_type or "", + "latency_ms": latency_ms, + "final_url": final_url or (feed.url if feed else source.feed_url), + "fetched_at": datetime.now(UTC).isoformat().replace("+00:00", "Z"), + } + + +def _format_source_fetch_error( + *, + status_code: int | None, + content_type: str, + body: str, + parsed_count: int, +) -> tuple[str | None, str]: + if status_code is not None and status_code >= 400: + return f"HTTP {status_code}", "http_error" + lower_content_type = content_type.lower() + body_prefix = body[:500].lower() + looks_like_html = "text/html" in lower_content_type or " dict[str, Any]: + source = _source_from_config(raw_source) + if source is None: + return {"ok": False, "count": 0, "error": "新闻源缺少 id、名称或 URL"} + if source.source_type not in {"rss", "atom", "aggregated"}: + health = _source_health_result( + source, + ok=False, + count=0, + error="参考链接仅用于记录官网、报告页或未来采集器线索，不参与 RSS/Atom 抓取。", + status="reference", + ) + await record_earth_news_source_health(db, source.id, health) + return { + "ok": False, + "count": 0, + "error": health["error"], + "health": health, + "source": serialize_news_source_config(source), + } + async with httpx.AsyncClient( + timeout=float(source.health_policy.get("timeout_seconds", REQUEST_TIMEOUT)), + follow_redirects=True, + headers={"User-Agent": USER_AGENT}, + ) as client: + fetched_source, items, error, health = await _fetch_source(client, source) + del fetched_source + await record_earth_news_source_health(db, source.id, health) + return { + "ok": error is None, + "count": len(items), + "item_count": len(items), + "error": error, + "health": health, + "source": serialize_news_source_config(source), + } + + def _strip_html(value: str) -> str: if not value: return "" @@ -814,37 +1793,49 @@ def _parse_datetime(raw: str | None) -> datetime | None: def _extract_item_text(element: ET.Element, *names: str) -> str: for name in names: node = element.find(name) + if node is None and not name.startswith("{"): + node = element.find(f"{{*}}{name}") if node is not None and node.text: return node.text.strip() return "" -def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]: +def _parse_feed_entries( + xml_text: str, + source: NewsFeedSource, + *, + feed: NewsFeedEndpoint | None = None, + config_payload: dict[str, Any] | None = None, +) -> list[ParsedNewsItem]: root = ET.fromstring(xml_text) items: list[ParsedNewsItem] = [] - rss_items = root.findall("./channel/item") - atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry") + rss_items = root.findall("./channel/item") or root.findall(".//item") or root.findall(".//{*}item") + atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry") or root.findall(".//{*}entry") nodes = rss_items or atom_entries for node in nodes[:MAX_ITEMS_PER_SOURCE]: if node.tag.endswith("entry"): - title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title") + title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title", "title") summary = _extract_item_text( node, "{http://www.w3.org/2005/Atom}summary", "{http://www.w3.org/2005/Atom}content", + "summary", + "content", ) - link_node = node.find("{http://www.w3.org/2005/Atom}link") + link_node = node.find("{http://www.w3.org/2005/Atom}link") or node.find("{*}link") link = link_node.get("href", "").strip() if link_node is not None else "" published = _extract_item_text( node, "{http://www.w3.org/2005/Atom}updated", "{http://www.w3.org/2005/Atom}published", + "updated", + "published", ) else: title = _extract_item_text(node, "title") - summary = _extract_item_text(node, "description", "content") + summary = _extract_item_text(node, "description", "content", "encoded") link = _extract_item_text(node, "link") published = _extract_item_text(node, "pubDate", "published", "updated") @@ -855,35 +1846,82 @@ def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNew item_source = source.name display_title = clean_title - if source.source_type == "aggregated" and " - " in clean_title: + feed_type = feed.type if feed else source.source_type + if feed_type == "aggregated" and " - " in clean_title: parts = clean_title.rsplit(" - ", 1) display_title = parts[0].strip() item_source = _normalize_source_name(parts[1], source.name) - items.append( - ParsedNewsItem( - id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}", + content_language = _detect_content_language( + title=display_title, + summary=clean_summary, + source=source, + feed=feed, + ) + feed_id_segment = f"{feed.id}:" if feed is not None and len(source.feeds or ()) > 1 else "" + item = ParsedNewsItem( + id=f"{source.id}:{feed_id_segment}{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}", + title=display_title, + summary=clean_summary, + url=link, + source=item_source, + feed_name=feed.name if feed else source.name, + feed_region=(feed.region if feed and feed.region else source.region), + homepage_url=source.homepage_url, + published_at=_parse_datetime(published), + content_language=content_language, + localizations=_source_language_localizations( title=display_title, summary=clean_summary, - url=link, - source=item_source, - feed_name=source.name, - feed_region=source.region, - homepage_url=source.homepage_url, - published_at=_parse_datetime(published), - ) + content_language=content_language, + ), + feed_id=feed.id if feed else "", + feed_type=feed_type, + feed_default_category=(feed.default_category if feed else source.default_category) or "other", + source_tags=list(source.source_tags), ) + apply_news_classification(item, source, feed=feed, config_payload=config_payload) + items.append(item) return items +def _normalize_breaking_level(value: str | None) -> str: + return _normalize_breaking_level_enum(value).value + + +def _normalize_breaking_scope(value: str | None) -> str: + return _normalize_breaking_scope_enum(value).value + + +def apply_news_classification( + item: ParsedNewsItem, + source: NewsFeedSource, + *, + feed: NewsFeedEndpoint | None = None, + config_payload: dict[str, Any] | None = None, +) -> ParsedNewsItem: + config = normalize_earth_news_sources_payload(config_payload) + return _apply_news_classification(item, source, feed=feed, config=config) + + def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]: return [ { "id": source.id, "name": source.name, "region": source.region, + "feed_url": source.feed_url, + "feed_urls": list(_source_feed_urls(source)), + "feeds": [_serialize_feed_endpoint(feed) for feed in _source_feeds(source)], "homepage_url": source.homepage_url, + "feed_directory_url": source.feed_directory_url, + "source_type": source.source_type, + "priority": source.priority, + "enabled": source.enabled, + "source_tags": list(source.source_tags), + "default_category": source.default_category, + "importance_weight": source.importance_weight, } for source in sources ] @@ -916,6 +1954,10 @@ def _serialize_enriched_at(value: datetime | None) -> str | None: return value.isoformat().replace("+00:00", "Z") if value else None +def _serialize_breaking_expires_at(value: datetime | None) -> str | None: + return value.isoformat().replace("+00:00", "Z") if value else None + + def _content_patch(item: ParsedNewsItem) -> dict[str, Any]: return { "content_language": item.content_language, @@ -926,6 +1968,32 @@ def _content_patch(item: ParsedNewsItem) -> dict[str, Any]: } +def _news_meta_patch(item: ParsedNewsItem) -> dict[str, Any]: + source_id = item.id.split(":", 1)[0] if ":" in item.id else "" + return { + "source_id": source_id, + "source_tags": list(item.source_tags or []), + "feed_id": item.feed_id, + "feed_name": item.feed_name, + "feed_type": item.feed_type, + "feed_default_category": item.feed_default_category, + "category": item.category, + "item_tags": list(item.item_tags or []), + "tagging_source": item.tagging_source, + "tagging_confidence": item.tagging_confidence, + "importance_score": item.importance_score, + "importance_level": item.importance_level, + "importance_reasons": list(item.importance_reasons or []), + "market_impact": item.market_impact, + "breaking_level": _normalize_breaking_level(item.breaking_level), + "breaking_scope": _normalize_breaking_scope(item.breaking_scope), + "breaking_reasons": list(item.breaking_reasons or []), + "breaking_source": item.breaking_source, + "breaking_confidence": item.breaking_confidence, + "breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at), + } + + def build_anchor_location_patch( item: ParsedNewsItem, *, @@ -959,6 +2027,7 @@ def build_anchor_location_patch( "queue_available": queue_available, "target": None, "anchor": _serialize_anchor(anchor), + "news_meta": _news_meta_patch(item), }, **content_patch, } @@ -982,6 +2051,7 @@ def build_target_location_patch(item: ParsedNewsItem, target: NewsTargetLocation "debug_note": item.target_debug_note, "target": _serialize_target(target), "anchor": _serialize_anchor(anchor), + "news_meta": _news_meta_patch(item), }, **_content_patch(item), } @@ -1001,9 +2071,27 @@ def build_target_location_job_payload(item: ParsedNewsItem) -> dict[str, Any]: "url": item.url, "source": item.source, "feed_name": item.feed_name, + "feed_id": item.feed_id, + "feed_type": item.feed_type, + "feed_default_category": item.feed_default_category, "feed_region": item.feed_region, "homepage_url": item.homepage_url, "published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None, + "source_tags": list(item.source_tags or []), + "category": item.category, + "item_tags": list(item.item_tags or []), + "tagging_source": item.tagging_source, + "tagging_confidence": item.tagging_confidence, + "importance_score": item.importance_score, + "importance_level": item.importance_level, + "importance_reasons": list(item.importance_reasons or []), + "market_impact": item.market_impact, + "breaking_level": _normalize_breaking_level(item.breaking_level), + "breaking_scope": _normalize_breaking_scope(item.breaking_scope), + "breaking_reasons": list(item.breaking_reasons or []), + "breaking_source": item.breaking_source, + "breaking_confidence": item.breaking_confidence, + "breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at), } @@ -1020,26 +2108,49 @@ def parsed_news_item_from_job_payload(payload: dict[str, Any]) -> ParsedNewsItem url=str(payload.get("url") or ""), source=str(payload.get("source") or ""), feed_name=str(payload.get("feed_name") or ""), + feed_id=str(payload.get("feed_id") or ""), + feed_type=str(payload.get("feed_type") or "rss"), + feed_default_category=str(payload.get("feed_default_category") or payload.get("category") or "other"), feed_region=str(payload.get("feed_region") or "global"), homepage_url=str(payload.get("homepage_url") or ""), published_at=_parse_datetime(_coerce_str(payload.get("published_at"))), + source_tags=list(payload.get("source_tags") or []), + category=str(payload.get("category") or "other"), + item_tags=list(payload.get("item_tags") or []), + tagging_source=str(payload.get("tagging_source") or "rules"), + tagging_confidence=float(payload.get("tagging_confidence") or 0), + importance_score=int(payload.get("importance_score") or 0), + importance_level=str(payload.get("importance_level") or "low"), + importance_reasons=list(payload.get("importance_reasons") or []), + market_impact=str(payload.get("market_impact") or "none"), + breaking_level=_normalize_breaking_level(str(payload.get("breaking_level") or "none")), + breaking_scope=_normalize_breaking_scope(str(payload.get("breaking_scope") or "regional")), + breaking_reasons=list(payload.get("breaking_reasons") or []), + breaking_source=str(payload.get("breaking_source") or "rules"), + breaking_confidence=float(payload.get("breaking_confidence") or 0), + breaking_expires_at=_parse_datetime(_coerce_str(payload.get("breaking_expires_at"))), ) -def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]: +def _serialize_item(item: ParsedNewsItem, *, active_region: str, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]: published_at = item.published_at location_patch = item.location_patch or build_target_location_patch(item, item.target_location) + source_id = item.id.split(":", 1)[0] if ":" in item.id else "" return { "id": item.id, + "source_id": source_id, "title": item.title, "summary": item.summary, "content_language": item.content_language, "localizations": item.localizations, - "display_title": _get_locale_text(item, "title"), - "display_summary": _get_locale_text(item, "summary"), + "display_title": _get_locale_text(item, "title", locale=locale), + "display_summary": _get_locale_text(item, "summary", locale=locale), "url": item.url, "source": item.source, "feed_name": item.feed_name, + "feed_id": item.feed_id, + "feed_type": item.feed_type, + "feed_default_category": item.feed_default_category, "region": item.feed_region, "display_region": get_region_anchor(item.feed_region).label, "homepage_url": item.homepage_url, @@ -1054,6 +2165,21 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, An "enrichment_error": item.enrichment_error, "enriched_at": _serialize_enriched_at(item.enriched_at), "is_focus_match": item.feed_region == active_region, + "source_tags": list(item.source_tags or []), + "category": item.category, + "item_tags": list(item.item_tags or []), + "tagging_source": item.tagging_source, + "tagging_confidence": item.tagging_confidence, + "importance_score": item.importance_score, + "importance_level": item.importance_level, + "importance_reasons": list(item.importance_reasons or []), + "market_impact": item.market_impact, + "breaking_level": _normalize_breaking_level(item.breaking_level), + "breaking_scope": _normalize_breaking_scope(item.breaking_scope), + "breaking_reasons": list(item.breaking_reasons or []), + "breaking_source": item.breaking_source, + "breaking_confidence": item.breaking_confidence, + "breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at), } @@ -1063,13 +2189,21 @@ def _build_payload( lon: float | None, active_region: str, items: list[ParsedNewsItem], + cruise_items: list[ParsedNewsItem] | None = None, sources: list[NewsFeedSource], errors: list[str], stale: bool, + categories: set[str] | None = None, + source_ids: set[str] | None = None, + limit: int = MAX_ITEMS_TOTAL, + locale: str = DEFAULT_NEWS_LOCALE, generated_at: datetime | None = None, ) -> dict[str, Any]: profile = get_region_profile(active_region) timestamp = generated_at or datetime.now(UTC) + visible_items = list(items) + cruise_visible_items = list(cruise_items if cruise_items is not None else items) + highest_breaking_level = _highest_breaking_level(visible_items + cruise_visible_items) return { "generated_at": timestamp.isoformat().replace("+00:00", "Z"), "focus": { @@ -1081,13 +2215,31 @@ def _build_payload( "accent": profile.accent, }, "sources": _serialize_sources(sources), - "items": [_serialize_item(item, active_region=active_region) for item in items], + "filters": { + "region": active_region, + "categories": sorted(categories or []), + "sources": sorted(source_ids or []), + "limit": limit, + "locale": locale, + "has_breaking": highest_breaking_level != "none", + "highest_breaking_level": highest_breaking_level, + }, + "items": [_serialize_item(item, active_region=active_region, locale=locale) for item in visible_items], + "cruise_items": [ + _serialize_item(item, active_region=active_region, locale=locale) + for item in cruise_visible_items + ], "errors": errors, "stale": stale, } -def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]: +def _rank_and_trim_items( + items: list[ParsedNewsItem], + *, + active_region: str, + limit: int = MAX_ITEMS_TOTAL, +) -> list[ParsedNewsItem]: deduped: dict[str, ParsedNewsItem] = {} for item in items: key = item.url.strip() or item.title.strip().lower() @@ -1097,12 +2249,95 @@ def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> return sorted( deduped.values(), key=lambda item: ( - item.feed_region != active_region, + -_breaking_sort_rank(item), + False + if active_region == "global" + or (_breaking_sort_rank(item) > 0 and _normalize_breaking_scope(item.breaking_scope) == "global") + else item.feed_region != active_region, item.published_at is None, -(item.published_at.timestamp() if item.published_at else 0), item.feed_name, ), - )[:MAX_ITEMS_TOTAL] + )[:limit] + + +def _filter_news_items_by_categories( + items: list[ParsedNewsItem], + categories: set[str] | None, +) -> list[ParsedNewsItem]: + if not categories: + return items + return [item for item in items if (item.category or "other") in categories] + + +def _filter_news_items_by_source_ids( + items: list[ParsedNewsItem], + source_ids: set[str] | None, +) -> list[ParsedNewsItem]: + if not source_ids: + return items + return [ + item for item in items + if (item.id.split(":", 1)[0] if ":" in item.id else "") in source_ids + ] + + +def _news_item_source_id(item: ParsedNewsItem) -> str: + return item.id.split(":", 1)[0] if ":" in item.id else "" + + +def _is_news_item_display_ready(item: ParsedNewsItem, *, locale: str) -> bool: + return bool( + _get_locale_text(item, "title", locale=locale) + and _get_locale_text(item, "summary", locale=locale) + ) + + +def _diversify_news_items_for_locale( + items: list[ParsedNewsItem], + *, + active_region: str, + limit: int, + locale: str, +) -> list[ParsedNewsItem]: + ranked = sorted( + _rank_and_trim_items(items, active_region=active_region, limit=max(len(items), limit)), + key=lambda item: (not _is_news_item_display_ready(item, locale=locale),), + ) + buckets: dict[str, list[ParsedNewsItem]] = {} + order: list[str] = [] + for item in ranked: + if active_region == "global": + key = item.feed_region or "global" + else: + key = _news_item_source_id(item) or item.source or item.feed_name or item.id + if key not in buckets: + buckets[key] = [] + order.append(key) + buckets[key].append(item) + + diversified: list[ParsedNewsItem] = [] + while len(diversified) < limit and order: + next_order: list[str] = [] + for source_id in order: + bucket = buckets.get(source_id) or [] + if bucket and len(diversified) < limit: + diversified.append(bucket.pop(0)) + if bucket: + next_order.append(source_id) + order = next_order + return diversified + + +async def _call_store_list_items(list_fn, db: AsyncSession, **kwargs): + try: + return await list_fn(db, **kwargs) + except TypeError as error: + if "source_ids" not in str(error): + raise + legacy_kwargs = dict(kwargs) + legacy_kwargs.pop("source_ids", None) + return await list_fn(db, **legacy_kwargs) def _get_cached_region_feed(region: str) -> CachedRegionFeed | None: @@ -1115,6 +2350,10 @@ def _get_cached_region_feed(region: str) -> CachedRegionFeed | None: return cached +def clear_earth_news_region_cache() -> None: + _REGION_CACHE.clear() + + def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None: _REGION_CACHE[region] = CachedRegionFeed( region=region, @@ -1140,7 +2379,7 @@ async def _apply_cached_locations_and_enqueue(items: list[ParsedNewsItem]) -> li cached_patch = await get_cached_target_location_patch(item.id) if cached_patch: apply_enrichment_patch_to_item(item, cached_patch) - if not _has_default_localization(item): + if not _has_required_localization(item): queued = await enqueue_item(item, force=True) if queued and item.enrichment_status in { "pending", @@ -1180,42 +2419,192 @@ async def _enqueue_unverified_locations(items: list[ParsedNewsItem]) -> None: if ( item.location_patch is None or item.location_patch.get("verified") is False - or not _has_default_localization(item) + or not _has_required_localization(item) ) ) ) +async def _fetch_single_feed_url( + client: httpx.AsyncClient, + source: NewsFeedSource, + feed: NewsFeedEndpoint, + *, + config_payload: dict[str, Any] | None = None, +) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]: + started_at = perf_counter() + try: + response = await client.get(feed.url) + latency_ms = int((perf_counter() - started_at) * 1000) + status_code = response.status_code + content_type = response.headers.get("content-type", "") + if status_code >= 400: + error = f"HTTP {status_code}" + return source, [], error, _source_health_result( + source, + ok=False, + count=0, + error=error, + status="http_error", + status_code=status_code, + content_type=content_type, + latency_ms=latency_ms, + final_url=str(response.url), + feed=feed, + ) + try: + items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload) + except Exception as exc: + error, status = _format_source_fetch_error( + status_code=status_code, + content_type=content_type, + body=response.text, + parsed_count=0, + ) + if error is None: + error = str(exc) + status = "format_error" + return source, [], error, _source_health_result( + source, + ok=False, + count=0, + error=error, + status=status, + status_code=status_code, + content_type=content_type, + latency_ms=latency_ms, + final_url=str(response.url), + feed=feed, + ) + error, status = _format_source_fetch_error( + status_code=status_code, + content_type=content_type, + body=response.text, + parsed_count=len(items), + ) + return source, items, error, _source_health_result( + source, + ok=error is None, + count=len(items), + error=error, + status=status, + status_code=status_code, + content_type=content_type, + latency_ms=latency_ms, + final_url=str(response.url), + feed=feed, + ) + except httpx.TimeoutException as exc: + error = f"请求超时: {exc}" + return source, [], error, _source_health_result( + source, + ok=False, + count=0, + error=error, + status="timeout", + latency_ms=int((perf_counter() - started_at) * 1000), + feed=feed, + ) + except Exception as exc: + error = str(exc) + return source, [], error, _source_health_result( + source, + ok=False, + count=0, + error=error, + status="network_error", + latency_ms=int((perf_counter() - started_at) * 1000), + feed=feed, + ) + + async def _fetch_source( client: httpx.AsyncClient, source: NewsFeedSource, -) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]: - try: - response = await client.get(source.feed_url) - response.raise_for_status() - return source, _parse_feed_entries(response.text, source), None - except Exception as exc: - return source, [], str(exc) + *, + active_region: str | None = None, + config_payload: dict[str, Any] | None = None, +) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]: + feeds = tuple( + feed + for feed in _source_feeds(source) + if feed.enabled and feed.type in {"rss", "atom", "aggregated"} + and _feed_matches_active_region(feed, active_region) + ) + if not feeds: + health = _source_health_result(source, ok=False, count=0, error="新闻源缺少已启用的 Feed 子项。", status="format_error") + return source, [], health["error"], health + + results = await asyncio.gather(*( + _fetch_single_feed_url(client, source, feed, config_payload=config_payload) + for feed in sorted(feeds, key=lambda item: (item.priority, item.name)) + )) + merged_items: list[ParsedNewsItem] = [] + feed_results: list[dict[str, Any]] = [] + errors: list[str] = [] + for _source, items, error, health in results: + feed_results.append(health) + if error: + errors.append(f"{health.get('final_url') or source.feed_url}: {error}") + continue + merged_items.extend(items) + + deduped: dict[str, ParsedNewsItem] = {} + for item in merged_items: + key = item.url.strip() or item.title.strip().lower() + if key and key not in deduped: + deduped[key] = item + items = list(deduped.values()) + ok = bool(items) + error = None if ok else "; ".join(errors) or "未解析到 RSS/Atom 条目，请检查 Feed 地址或源格式。" + latency_ms = sum(int(result.get("latency_ms") or 0) for result in feed_results) + status = "ok" if ok else (feed_results[0].get("status") if len(feed_results) == 1 else "failed") + status_code = next((result.get("status_code") for result in feed_results if result.get("status_code")), None) + content_type = ", ".join(sorted({str(result.get("content_type") or "") for result in feed_results if result.get("content_type")})) + health = _source_health_result( + source, + ok=ok, + count=len(items), + error=error, + status=status, + status_code=int(status_code) if isinstance(status_code, int) else None, + content_type=content_type, + latency_ms=latency_ms, + final_url=source.feed_url, + ) + health["feed_results"] = feed_results + return source, items, error, health async def _fetch_rss_items_for_sources( sources: list[NewsFeedSource], -) -> tuple[list[ParsedNewsItem], list[str]]: + *, + active_region: str | None = None, + config_payload: dict[str, Any] | None = None, +) -> tuple[list[ParsedNewsItem], list[str], dict[str, dict[str, Any]]]: errors: list[str] = [] async with httpx.AsyncClient( timeout=REQUEST_TIMEOUT, follow_redirects=True, headers={"User-Agent": USER_AGENT}, ) as client: - results = await asyncio.gather(*(_fetch_source(client, source) for source in sources)) + if config_payload is None: + results = await asyncio.gather(*(_fetch_source(client, source, active_region=active_region) for source in sources)) + else: + results = await asyncio.gather(*( + _fetch_source(client, source, active_region=active_region, config_payload=config_payload) + for source in sources + )) fetched_items: list[ParsedNewsItem] = [] - for source, items, error in results: + health_by_source: dict[str, dict[str, Any]] = {} + for source, items, error, health in results: + health_by_source[source.id] = health if error: errors.append(f"{source.name}: {error}") continue fetched_items.extend(items) - return fetched_items, errors + return fetched_items, errors, health_by_source def _needs_rss_supplement(*, item_count: int, newest_at: datetime | None) -> bool: @@ -1233,9 +2622,19 @@ async def _get_earth_news_payload_from_rss_only( lon: float | None, active_region: str, sources: list[NewsFeedSource], + categories: set[str] | None = None, + source_ids: set[str] | None = None, + limit: int = MAX_ITEMS_TOTAL, + locale: str = DEFAULT_NEWS_LOCALE, ) -> dict[str, Any]: - fetched_items, errors = await _fetch_rss_items_for_sources(sources) - ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region) + fetched_items, errors, _health_by_source = await _fetch_rss_items_for_sources(sources, active_region=active_region) + ranked_items = _filter_news_items_by_source_ids( + _filter_news_items_by_categories( + _rank_and_trim_items(fetched_items, active_region=active_region, limit=limit), + categories, + ), + source_ids, + )[:limit] if ranked_items: ranked_items = await _apply_cached_locations_and_enqueue(ranked_items) _store_region_cache(active_region, items=ranked_items, sources=sources) @@ -1247,11 +2646,19 @@ async def _get_earth_news_payload_from_rss_only( sources=sources, errors=errors, stale=False, + categories=categories, + source_ids=source_ids, + limit=limit, + locale=locale, ) cached = _get_cached_region_feed(active_region) if cached: - cached.items = await _apply_cached_locations_and_enqueue(cached.items) + cached_items = _filter_news_items_by_source_ids( + _filter_news_items_by_categories(cached.items, categories), + source_ids, + )[:limit] + cached.items = await _apply_cached_locations_and_enqueue(cached_items) return _build_payload( lat=lat, lon=lon, @@ -1260,6 +2667,10 @@ async def _get_earth_news_payload_from_rss_only( sources=cached.sources, errors=errors, stale=True, + categories=categories, + source_ids=source_ids, + limit=limit, + locale=locale, generated_at=cached.fetched_at, ) @@ -1271,6 +2682,10 @@ async def _get_earth_news_payload_from_rss_only( sources=sources, errors=errors, stale=False, + categories=categories, + source_ids=source_ids, + limit=limit, + locale=locale, ) @@ -1278,12 +2693,23 @@ async def get_earth_news_payload( lat: float | None = None, lon: float | None = None, *, + region: str | None = None, + categories: set[str] | None = None, + source_ids: set[str] | None = None, + limit: int = MAX_ITEMS_TOTAL, + locale: str = DEFAULT_NEWS_LOCALE, provider_client: AIProviderClient | None = None, db: AsyncSession | None = None, ) -> dict[str, Any]: del provider_client - active_region = determine_focus_region(lat, lon) - sources = get_sources_for_region(active_region) + active_region = region if region in REGION_ANCHORS else determine_focus_region(lat, lon) + has_settings_db = db is not None and callable(getattr(db, "execute", None)) + source_config_payload = await get_earth_news_sources_payload(db) if has_settings_db else None + sources = ( + await get_configured_sources_for_region(db, active_region) + if has_settings_db + else get_sources_for_region(active_region) + ) if db is None: return await _get_earth_news_payload_from_rss_only( @@ -1291,10 +2717,16 @@ async def get_earth_news_payload( lon=lon, active_region=active_region, sources=sources, + categories=categories, + source_ids=source_ids, + limit=limit, + locale=locale, ) from app.services.earth_news_store import ( + get_earth_news_feed_coverage, get_earth_news_freshness, + list_earth_news_cruise_items, list_earth_news_items, upsert_earth_news_items, ) @@ -1302,17 +2734,80 @@ async def get_earth_news_payload( errors: list[str] = [] item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region) should_supplement = _needs_rss_supplement(item_count=item_count, newest_at=newest_at) + if not should_supplement and callable(getattr(db, "execute", None)): + expected_feed_keys = _expected_feed_keys(sources, active_region=active_region) + if expected_feed_keys: + covered_feed_keys = await get_earth_news_feed_coverage( + db, + active_region=active_region, + recent_after=datetime.now(UTC) - timedelta(seconds=RSS_SUPPLEMENT_MAX_AGE_SECONDS), + ) + should_supplement = bool(expected_feed_keys - covered_feed_keys) if should_supplement: - fetched_items, errors = await _fetch_rss_items_for_sources(sources) + if source_config_payload is None: + fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(sources, active_region=active_region) + else: + fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources( + sources, + active_region=active_region, + config_payload=source_config_payload, + ) + await record_earth_news_sources_health(db, health_by_source) ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region) + await _enqueue_unverified_locations(ranked_fetched_items) await upsert_earth_news_items(db, ranked_fetched_items) - items = await list_earth_news_items( + items = await _call_store_list_items( + list_earth_news_items, db, active_region=active_region, - limit=MAX_ITEMS_TOTAL, + limit=limit, + categories=categories, + source_ids=source_ids, ) - await _enqueue_unverified_locations(items) + if not source_ids: + ready_sources = { + _news_item_source_id(item) + for item in items + if _is_news_item_display_ready(item, locale=locale) + } + missing_ready_sources = [ + source.id + for source in sources + if source.id and source.id not in ready_sources + ] + if missing_ready_sources: + extra_items: list[ParsedNewsItem] = [] + for missing_source_id in missing_ready_sources: + extra_items.extend( + await _call_store_list_items( + list_earth_news_items, + db, + active_region=active_region, + limit=3, + categories=categories, + source_ids={missing_source_id}, + ) + ) + if extra_items: + items = _diversify_news_items_for_locale( + [*items, *extra_items], + active_region=active_region, + limit=limit, + locale=locale, + ) + if hasattr(db, "execute"): + cruise_items = await _call_store_list_items( + list_earth_news_cruise_items, + db, + limit=min(max(limit, MAX_ITEMS_TOTAL) * len(REGION_ANCHORS), 500), + categories=categories, + source_ids=source_ids, + ) + else: + cruise_items = items + enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id} + await _enqueue_unverified_locations(list(enqueue_candidates.values())) stale = bool(errors and items) return _build_payload( @@ -1320,7 +2815,12 @@ async def get_earth_news_payload( lon=lon, active_region=active_region, items=items, + cruise_items=cruise_items, sources=sources, errors=errors, stale=stale, + categories=categories, + source_ids=source_ids, + limit=limit, + locale=locale, ) diff --git a/backend/app/services/earth_news_classification.py b/backend/app/services/earth_news_classification.py new file mode 100644 index 00000000..fd887b48 --- /dev/null +++ b/backend/app/services/earth_news_classification.py @@ -0,0 +1,258 @@ +"""Classification, importance, and breaking-news policy for Earth news.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import UTC, datetime, timedelta +import re +from typing import Any, Protocol + +from app.core.enums import ( + BreakingLevel, + BreakingScope, + BreakingSource, + NewsImportanceLevel, + NewsMarketImpact, + NewsTaggingSource, + parse_enum, +) + + +class NewsItemLike(Protocol): + title: str + summary: str + source: str + feed_name: str + published_at: datetime | None + feed_default_category: str + category: str + item_tags: list[str] + tagging_source: str + tagging_confidence: float + importance_score: int + importance_level: str + importance_reasons: list[str] + market_impact: str + source_tags: list[str] + breaking_level: str + breaking_scope: str + breaking_reasons: list[str] + breaking_source: str + breaking_confidence: float + breaking_expires_at: datetime | None + + +class NewsSourceLike(Protocol): + default_category: str + importance_weight: int + source_tags: tuple[str, ...] + + +class NewsFeedLike(Protocol): + default_category: str + + +@dataclass(frozen=True) +class BreakingRule: + level: BreakingLevel + scope: BreakingScope + reason: str + keywords: tuple[str, ...] + + +IMPORTANCE_THRESHOLDS: tuple[tuple[int, NewsImportanceLevel], ...] = ( + (80, NewsImportanceLevel.CRITICAL), + (60, NewsImportanceLevel.HIGH), + (35, NewsImportanceLevel.MEDIUM), + (0, NewsImportanceLevel.LOW), +) + +BREAKING_LEVEL_RANK: dict[BreakingLevel, int] = { + BreakingLevel.NONE: 0, + BreakingLevel.WATCH: 1, + BreakingLevel.BREAKING: 2, + BreakingLevel.CRITICAL: 3, +} + +BREAKING_TTL: dict[BreakingLevel, timedelta] = { + BreakingLevel.WATCH: timedelta(hours=6), + BreakingLevel.BREAKING: timedelta(hours=12), + BreakingLevel.CRITICAL: timedelta(hours=24), +} + +BREAKING_RULES: tuple[BreakingRule, ...] = ( + BreakingRule(BreakingLevel.CRITICAL, BreakingScope.GLOBAL, "核事故或核风险", ("nuclear accident", "nuclear emergency", "radiation leak", "核事故", "核泄漏", "辐射泄漏")), + BreakingRule(BreakingLevel.CRITICAL, BreakingScope.REGIONAL, "重大军事冲突升级", ("airstrike", "missile strike", "invasion", "martial law", "空袭", "导弹袭击", "入侵", "戒严")), + BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "战争或安全事件", ("war escalates", "terror attack", "coup", "hostage", "战争升级", "恐袭", "政变", "人质")), + BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "重大灾害应急", ("major earthquake", "tsunami", "volcanic eruption", "state of emergency", "强震", "海啸", "火山喷发", "紧急状态")), + BreakingRule(BreakingLevel.BREAKING, BreakingScope.GLOBAL, "金融市场异常", ("market halt", "trading halt", "flash crash", "bank run", "金融熔断", "交易暂停", "银行挤兑")), + BreakingRule(BreakingLevel.WATCH, BreakingScope.GLOBAL, "大规模网络安全事件", ("massive cyberattack", "ransomware attack", "data breach", "大规模网络攻击", "勒索软件", "数据泄露")), + BreakingRule(BreakingLevel.WATCH, BreakingScope.REGIONAL, "航天或卫星事故", ("rocket explosion", "satellite collision", "space station emergency", "火箭爆炸", "卫星碰撞", "空间站事故")), +) + + +def contains_keyword(text: str, keyword: str) -> bool: + keyword_text = str(keyword or "").strip().lower() + if not keyword_text: + return False + if re.search(r"[\u4e00-\u9fff]", keyword_text): + return keyword_text in text + return re.search(rf"(? int: + score = 0 + keywords = category.get("keywords") if isinstance(category.get("keywords"), list) else [] + for keyword in keywords: + if contains_keyword(title_text, keyword): + score += 3 + elif contains_keyword(text, keyword): + score += 1 + return score + + +def importance_level(score: int) -> NewsImportanceLevel: + normalized_score = max(0, min(100, int(score))) + for threshold, level in IMPORTANCE_THRESHOLDS: + if normalized_score >= threshold: + return level + return NewsImportanceLevel.LOW + + +def normalize_breaking_level(value: object) -> BreakingLevel: + return parse_enum(BreakingLevel, value, BreakingLevel.NONE) + + +def normalize_breaking_scope(value: object) -> BreakingScope: + return parse_enum(BreakingScope, value, BreakingScope.REGIONAL) + + +def breaking_expires_at(level: object, published_at: datetime | None) -> datetime | None: + normalized = normalize_breaking_level(level) + if normalized is BreakingLevel.NONE: + return None + base = published_at or datetime.now(UTC) + base = base.replace(tzinfo=UTC) if base.tzinfo is None else base.astimezone(UTC) + return base + BREAKING_TTL[normalized] + + +def is_breaking_active(item: NewsItemLike, *, now: datetime | None = None) -> bool: + if normalize_breaking_level(item.breaking_level) is BreakingLevel.NONE: + return False + expires_at = item.breaking_expires_at + if expires_at is None: + return True + expires_at = expires_at.replace(tzinfo=UTC) if expires_at.tzinfo is None else expires_at.astimezone(UTC) + return expires_at > (now or datetime.now(UTC)) + + +def breaking_sort_rank(item: NewsItemLike) -> int: + if not is_breaking_active(item): + return 0 + return BREAKING_LEVEL_RANK[normalize_breaking_level(item.breaking_level)] + + +def highest_breaking_level(items: list[NewsItemLike]) -> BreakingLevel: + active = [normalize_breaking_level(item.breaking_level) for item in items if is_breaking_active(item)] + return max(active, key=BREAKING_LEVEL_RANK.get) if active else BreakingLevel.NONE + + +def apply_breaking_rules(item: NewsItemLike) -> None: + combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower() + best_level = BreakingLevel.NONE + best_scope = BreakingScope.REGIONAL + reasons: list[str] = [] + confidence = 0.0 + for rule in BREAKING_RULES: + if not any(contains_keyword(combined_text, keyword) for keyword in rule.keywords): + continue + if BREAKING_LEVEL_RANK[rule.level] > BREAKING_LEVEL_RANK[best_level]: + best_level = rule.level + best_scope = rule.scope + if rule.reason not in reasons: + reasons.append(rule.reason) + confidence = max(confidence, 0.72 if rule.level is BreakingLevel.CRITICAL else 0.64 if rule.level is BreakingLevel.BREAKING else 0.52) + + item.breaking_level = best_level.value + item.breaking_scope = (best_scope if best_level is not BreakingLevel.NONE else BreakingScope.REGIONAL).value + item.breaking_reasons = reasons + item.breaking_source = BreakingSource.RULES.value + item.breaking_confidence = round(confidence, 2) + item.breaking_expires_at = breaking_expires_at(best_level, item.published_at) + + +def apply_news_classification( + item: NewsItemLike, + source: NewsSourceLike, + *, + feed: NewsFeedLike | None, + config: dict[str, Any], +) -> NewsItemLike: + title_text = item.title.lower() + combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower() + feed_default_category = (feed.default_category if feed else item.feed_default_category) or source.default_category or "other" + best_key = feed_default_category + best_score = second_score = 0 + for category in config["categories"]: + if not isinstance(category, dict) or category.get("enabled") is False: + continue + score = score_category(combined_text, title_text, category) + if score > best_score: + second_score, best_score = best_score, score + best_key = str(category.get("key") or "other") + elif score > second_score: + second_score = score + + item_tags: list[str] = [] + for rule in config["item_tag_rules"]: + if not isinstance(rule, dict): + continue + keywords = rule.get("keywords") if isinstance(rule.get("keywords"), list) else [] + if any(contains_keyword(combined_text, keyword) for keyword in keywords): + tag_key = str(rule.get("key") or "").strip() + if tag_key and tag_key not in item_tags: + item_tags.append(tag_key) + if best_score < 3 and rule.get("category"): + best_key, best_score = str(rule["category"]), 3 + + confidence = round(best_score / (best_score + second_score + 1), 2) if best_score else 0.35 + if best_score < 3 and feed_default_category: + best_key, confidence = feed_default_category, 0.45 + + score = max(0, min(100, 18 + source.importance_weight + best_score * 6)) + reasons: list[str] = [] + source_tags = set(source.source_tags) + if "official_data" in source_tags: + score += 20 + reasons.append("官方数据源") + if "press_release" in source_tags: + score = max(0, score - 12) + reasons.append("企业公告基础权重较低") + if any(contains_keyword(combined_text, term) for term in ("网上零售额", "电商物流指数", "gmv", "订单量", "物流指数", "履约", "直播电商", "跨境电商")): + score += 25 + reasons.append("命中电商数据指标") + if any(contains_keyword(combined_text, term) for term in ("amazon", "shopify", "walmart", "alibaba", "jd.com", "pinduoduo", "tiktok shop", "shein", "阿里", "京东", "拼多多", "抖音")): + score += 15 + reasons.append("涉及大型平台") + if any(term in combined_text for term in ("同比", "环比", "%", "billion", "million", "增长", "下降")): + score += 10 + reasons.append("包含量化指标") + + score = max(0, min(100, score)) + item.category = best_key or "other" + item.item_tags = item_tags + item.tagging_source = NewsTaggingSource.RULES.value + item.tagging_confidence = confidence + item.importance_score = score + item.importance_level = importance_level(score).value + item.importance_reasons = reasons or ["按来源权重和分类规则计算"] + item.market_impact = ( + NewsMarketImpact.GLOBAL.value + if "global" in source_tags + else NewsMarketImpact.NATIONAL.value + if {"china", "us"} & source_tags + else NewsMarketImpact.SECTOR.value + ) + item.source_tags = list(source.source_tags) + apply_breaking_rules(item) + return item diff --git a/backend/app/services/earth_news_manual.py b/backend/app/services/earth_news_manual.py new file mode 100644 index 00000000..b8da4941 --- /dev/null +++ b/backend/app/services/earth_news_manual.py @@ -0,0 +1,693 @@ +from __future__ import annotations + +from dataclasses import dataclass +from datetime import UTC, datetime +import hashlib +import html +import json +import re +from typing import Any + +from bs4 import BeautifulSoup +from sqlalchemy import delete, func, select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.core.enums import NewsEnrichmentStatus, NewsSourceType, NewsTaggingSource +from app.core.websocket.broadcaster import broadcaster +from app.models.earth_news import EarthNewsItem +from app.models.system_setting import SystemSetting +from app.services.earth_news import ( + ALLOWED_NEWS_CATEGORY_KEYS, + DEFAULT_NEWS_LOCALE, + REGION_ANCHORS, + NewsFeedEndpoint, + NewsFeedSource, + NewsTargetLocation, + ParsedNewsItem, + apply_news_classification, + build_anchor_location_patch, + build_target_location_job_payload, + build_target_location_patch, + _serialize_item, +) +from app.services.earth_news_queue import enqueue_target_location_job +from app.services.earth_news_store import record_to_parsed_news_item + + +MANUAL_NEWS_SOURCE_ID = "manual" +MANUAL_NEWS_SOURCE_LABEL = "手动添加" +MANUAL_NEWS_MAX_IMPORT_ITEMS = 500 +MANUAL_NEWS_MAX_TITLE_LENGTH = 500 +MANUAL_NEWS_MAX_SUMMARY_LENGTH = 1200 +MANUAL_NEWS_MAX_CONTENT_LENGTH = 12000 +EARTH_NEWS_MANUAL_GROUPS_CATEGORY = "earth_news_manual_groups" +DEFAULT_MANUAL_NEWS_GROUP_ID = "manual-default" +DEFAULT_MANUAL_NEWS_GROUP_NAME = "新建新闻组" + + +@dataclass(frozen=True) +class ManualNewsWriteResult: + item: EarthNewsItem + created: bool + queued: bool + + +@dataclass(frozen=True) +class ManualNewsGroup: + id: str + name: str + sort_order: int = 0 + + +def _clean_text(value: object, *, max_length: int) -> str: + raw = "" if value is None else str(value) + text = BeautifulSoup(html.unescape(raw), "html.parser").get_text(" ", strip=True) + text = re.sub(r"\s+", " ", text).strip() + if len(text) > max_length: + return text[: max_length - 1].rstrip() + "…" + return text + + +def _parse_datetime(value: object) -> datetime | None: + if value is None or str(value).strip() == "": + return None + if isinstance(value, datetime): + parsed = value + else: + try: + parsed = datetime.fromisoformat(str(value).strip().replace("Z", "+00:00")) + except ValueError as exc: + raise ValueError("published_at 必须是 ISO8601 时间。") from exc + if parsed.tzinfo is None: + return parsed.replace(tzinfo=UTC) + return parsed.astimezone(UTC) + + +def _detect_language(*parts: str) -> str: + text = " ".join(part for part in parts if part) + cjk_count = len(re.findall(r"[\u4e00-\u9fff]", text)) + latin_count = len(re.findall(r"[A-Za-z]", text)) + return "zh-CN" if cjk_count >= max(4, latin_count // 3) else "en-US" + + +def _manual_item_id(*, title: str, published_at: datetime | None, url: str, source: str) -> str: + published = published_at.isoformat() if published_at else "" + basis = "\n".join([title.strip().lower(), published, url.strip().lower(), source.strip().lower()]) + return f"manual:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:16]}" + + +def _manual_group_id(name: str) -> str: + basis = f"{name.strip().lower()}\n{datetime.now(UTC).isoformat()}" + return f"manual-group:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:10]}" + + +def _news_meta(record: EarthNewsItem) -> dict[str, Any]: + location_meta = record.location_meta if isinstance(record.location_meta, dict) else {} + news_meta = location_meta.get("news_meta") + return dict(news_meta) if isinstance(news_meta, dict) else {} + + +def _record_source_type(record: EarthNewsItem) -> str: + return str(_news_meta(record).get("feed_type") or _news_meta(record).get("source_type") or "rss") + + +def _record_manual_group_id(record: EarthNewsItem) -> str: + return str(_news_meta(record).get("manual_group_id") or DEFAULT_MANUAL_NEWS_GROUP_ID) + + +def _rss_group_id(record: EarthNewsItem) -> str: + basis = "\n".join( + [ + _record_source_type(record), + str(record.feed_name or ""), + str(record.source or ""), + ] + ) + return f"rss:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:12]}" + + +def _default_manual_group() -> dict[str, Any]: + return { + "id": DEFAULT_MANUAL_NEWS_GROUP_ID, + "name": DEFAULT_MANUAL_NEWS_GROUP_NAME, + "sort_order": 0, + } + + +def _normalize_manual_groups_payload(payload: Any) -> list[dict[str, Any]]: + raw_groups = payload.get("groups") if isinstance(payload, dict) else None + normalized: list[dict[str, Any]] = [] + seen: set[str] = set() + for index, item in enumerate(raw_groups if isinstance(raw_groups, list) else []): + if not isinstance(item, dict): + continue + group_id = str(item.get("id") or "").strip() + name = _clean_text(item.get("name"), max_length=120) + if not group_id or not name or group_id in seen: + continue + normalized.append( + { + "id": group_id, + "name": name, + "sort_order": int(item.get("sort_order") or index), + } + ) + seen.add(group_id) + if DEFAULT_MANUAL_NEWS_GROUP_ID not in seen: + normalized.insert(0, _default_manual_group()) + return sorted(normalized, key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or ""))) + + +async def _get_manual_groups_record(db: AsyncSession) -> SystemSetting | None: + result = await db.execute( + select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_MANUAL_GROUPS_CATEGORY) + ) + return result.scalar_one_or_none() + + +async def get_manual_news_groups(db: AsyncSession) -> list[dict[str, Any]]: + record = await _get_manual_groups_record(db) + return _normalize_manual_groups_payload(record.payload if record else None) + + +async def _save_manual_news_groups(db: AsyncSession, groups: list[dict[str, Any]]) -> list[dict[str, Any]]: + normalized = _normalize_manual_groups_payload({"groups": groups}) + record = await _get_manual_groups_record(db) + payload = {"groups": normalized} + if record is None: + db.add(SystemSetting(category=EARTH_NEWS_MANUAL_GROUPS_CATEGORY, payload=payload)) + else: + record.payload = payload + await db.flush() + return normalized + + +async def resolve_manual_news_group(db: AsyncSession, group_id: str | None) -> ManualNewsGroup: + normalized_id = str(group_id or DEFAULT_MANUAL_NEWS_GROUP_ID).strip() or DEFAULT_MANUAL_NEWS_GROUP_ID + groups = await get_manual_news_groups(db) + match = next((item for item in groups if item.get("id") == normalized_id), None) + if match is None and normalized_id != DEFAULT_MANUAL_NEWS_GROUP_ID: + raise ValueError(f"手动新闻组不存在：{normalized_id}") + match = match or _default_manual_group() + return ManualNewsGroup( + id=str(match["id"]), + name=str(match["name"]), + sort_order=int(match.get("sort_order") or 0), + ) + + +async def create_manual_news_group(db: AsyncSession, name: str) -> dict[str, Any]: + group_name = _clean_text(name, max_length=120) + if not group_name: + raise ValueError("新闻组名称不能为空。") + groups = await get_manual_news_groups(db) + group = {"id": _manual_group_id(group_name), "name": group_name, "sort_order": len(groups)} + groups.append(group) + await _save_manual_news_groups(db, groups) + return group + + +async def rename_manual_news_group(db: AsyncSession, group_id: str, name: str) -> dict[str, Any]: + group_name = _clean_text(name, max_length=120) + if not group_name: + raise ValueError("新闻组名称不能为空。") + groups = await get_manual_news_groups(db) + match = next((item for item in groups if item.get("id") == group_id), None) + if match is None: + raise ValueError(f"手动新闻组不存在：{group_id}") + match["name"] = group_name + await _save_manual_news_groups(db, groups) + + result = await db.execute( + select(EarthNewsItem).where( + EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == NewsSourceType.MANUAL.value + ) + ) + for record in result.scalars().all(): + if _record_manual_group_id(record) != group_id: + continue + location_meta = dict(record.location_meta or {}) + news_meta = dict(location_meta.get("news_meta") or {}) + news_meta["manual_group_name"] = group_name + location_meta["news_meta"] = news_meta + record.location_meta = location_meta + await db.flush() + return match + + +def _normalize_region(value: object) -> str: + region = str(value or "global").strip().lower() or "global" + if region not in REGION_ANCHORS: + raise ValueError(f"region 不支持：{region}") + return region + + +def _normalize_tags(value: object) -> list[str]: + if value is None: + return [] + if isinstance(value, str): + parts = re.split(r"[,，\n]", value) + elif isinstance(value, list): + parts = [str(item) for item in value] + else: + raise ValueError("tags 必须是字符串数组或逗号分隔字符串。") + return [item.strip() for item in parts if item.strip()][:20] + + +def _normalize_location(value: object) -> NewsTargetLocation | None: + if value in (None, ""): + return None + if not isinstance(value, dict): + raise ValueError("location 必须是对象。") + lat = value.get("latitude") + lon = value.get("longitude") + if lat in (None, "") and lon in (None, ""): + return None + try: + latitude = float(lat) + longitude = float(lon) + except (TypeError, ValueError) as exc: + raise ValueError("location.latitude / longitude 必须是数字。") from exc + if not -90 <= latitude <= 90 or not -180 <= longitude <= 180: + raise ValueError("location 经纬度超出范围。") + label = _clean_text(value.get("label"), max_length=255) + if not label: + label = f"{latitude:.4f}, {longitude:.4f}" + return NewsTargetLocation( + latitude=latitude, + longitude=longitude, + label=label, + source="manual_location", + confidence=1.0, + country=_clean_text(value.get("country"), max_length=100) or None, + city=_clean_text(value.get("city"), max_length=100) or None, + ) + + +def _manual_source(source_name: str, *, region: str) -> NewsFeedSource: + return NewsFeedSource( + id=MANUAL_NEWS_SOURCE_ID, + name=source_name or MANUAL_NEWS_SOURCE_LABEL, + region=region, + feed_url="", + homepage_url="", + source_type=NewsSourceType.MANUAL.value, + default_category="other", + source_tags=("manual",), + ) + + +def _manual_feed(category: str) -> NewsFeedEndpoint: + return NewsFeedEndpoint( + id=MANUAL_NEWS_SOURCE_ID, + name=MANUAL_NEWS_SOURCE_LABEL, + url="", + type=NewsSourceType.MANUAL.value, + default_category=category or "other", + tags=("manual",), + priority=1, + ) + + +def parsed_manual_news_item( + payload: dict[str, Any], + *, + item_id_override: str | None = None, +) -> tuple[ParsedNewsItem, NewsTargetLocation | None, str]: + title = _clean_text(payload.get("title"), max_length=MANUAL_NEWS_MAX_TITLE_LENGTH) + if not title: + raise ValueError("title 不能为空。") + content = _clean_text(payload.get("content"), max_length=MANUAL_NEWS_MAX_CONTENT_LENGTH) + summary = _clean_text(payload.get("summary"), max_length=MANUAL_NEWS_MAX_SUMMARY_LENGTH) + if not summary: + summary = _clean_text(content, max_length=240) if content else title + source = _clean_text(payload.get("source"), max_length=255) or MANUAL_NEWS_SOURCE_LABEL + region = _normalize_region(payload.get("region")) + published_at = _parse_datetime(payload.get("published_at")) or datetime.now(UTC) + url = str(payload.get("url") or "").strip() + category = str(payload.get("category") or "other").strip().lower() or "other" + if category not in ALLOWED_NEWS_CATEGORY_KEYS: + raise ValueError(f"category 不支持：{category}") + tags = _normalize_tags(payload.get("tags")) + target = _normalize_location(payload.get("location")) + language = str(payload.get("content_language") or "").strip() or _detect_language(title, summary, content) + localizations = { + language: { + "title": title, + "summary": summary, + } + } + item = ParsedNewsItem( + id=item_id_override + or _manual_item_id(title=title, published_at=published_at, url=url, source=source), + title=title, + summary=summary, + url=url, + source=source, + feed_name=MANUAL_NEWS_SOURCE_LABEL, + feed_region=region, + homepage_url=str(payload.get("homepage_url") or ""), + published_at=published_at, + content_language=language, + localizations=localizations, + enrichment_status=NewsEnrichmentStatus.PENDING.value, + source_tags=["manual"], + feed_id=MANUAL_NEWS_SOURCE_ID, + feed_type=NewsSourceType.MANUAL.value, + feed_default_category=category, + category=category, + item_tags=tags, + tagging_source=NewsTaggingSource.MANUAL.value if payload.get("category") else NewsTaggingSource.RULES.value, + tagging_confidence=0.9 if payload.get("category") else 0.0, + ) + source_config = _manual_source(source, region=region) + feed = _manual_feed(category) + apply_news_classification(item, source_config, feed=feed) + if payload.get("category"): + item.category = category + item.tagging_source = NewsTaggingSource.MANUAL.value + item.tagging_confidence = 0.9 + if tags: + item.item_tags = sorted(set([*item.item_tags, *tags])) + return item, target, content + + +def _manual_editable(record: EarthNewsItem) -> bool: + if record.id.startswith("manual:"): + return True + news_meta = (record.location_meta or {}).get("news_meta") if isinstance(record.location_meta, dict) else None + return isinstance(news_meta, dict) and news_meta.get("feed_type") == NewsSourceType.MANUAL.value + + +async def _broadcast_news_reload() -> None: + await broadcaster.broadcast_earth_update( + { + "action": "database_changed", + "source": "earth_news_items", + "layers": ["news"], + "refresh_strategy": "reload", + } + ) + + +async def upsert_manual_news_item( + db: AsyncSession, + payload: dict[str, Any], + *, + item_id_override: str | None = None, + group_id: str | None = None, +) -> ManualNewsWriteResult: + item, target, content = parsed_manual_news_item(payload, item_id_override=item_id_override) + group = await resolve_manual_news_group(db, group_id or payload.get("group_id")) + existing = await db.get(EarthNewsItem, item.id) + created = existing is None + patch = build_target_location_patch(item, target) if target else build_anchor_location_patch(item) + patch_meta = dict(patch.get("location_meta") or {}) + patch_news_meta = dict(patch_meta.get("news_meta") or {}) + patch_news_meta["feed_type"] = NewsSourceType.MANUAL.value + patch_news_meta["source_type"] = NewsSourceType.MANUAL.value + patch_news_meta["manual_group_id"] = group.id + patch_news_meta["manual_group_name"] = group.name + patch_meta["news_meta"] = patch_news_meta + patch["location_meta"] = patch_meta + now = datetime.now(UTC) + record = existing or EarthNewsItem( + id=item.id, + title=item.title, + summary=item.summary, + content_language=item.content_language, + localizations=dict(item.localizations or {}), + url=item.url, + source=item.source, + feed_name=item.feed_name, + region=item.feed_region, + homepage_url=item.homepage_url, + published_at=item.published_at, + latitude=patch["latitude"], + longitude=patch["longitude"], + location_label=patch["location_label"], + location_source=patch["location_source"], + verified=patch["verified"], + location_meta=patch["location_meta"], + first_seen_at=now, + last_seen_at=now, + resolved_at=now if patch["verified"] else None, + enrichment_status=item.enrichment_status, + ) + if existing is None: + db.add(record) + else: + if not _manual_editable(record): + raise PermissionError("RSS 新闻不允许通过手动新闻接口编辑。") + record.title = item.title + record.summary = item.summary + record.content_language = item.content_language + record.localizations = dict(item.localizations or {}) + record.url = item.url + record.source = item.source + record.feed_name = item.feed_name + record.region = item.feed_region + record.homepage_url = item.homepage_url + record.published_at = item.published_at + record.last_seen_at = now + if target is None and record.location_source == "manual_location": + merged_meta = dict(record.location_meta or {}) + patch_meta = patch.get("location_meta") if isinstance(patch, dict) else None + patch_news_meta = patch_meta.get("news_meta") if isinstance(patch_meta, dict) else None + if isinstance(patch_news_meta, dict): + merged_meta["news_meta"] = patch_news_meta + record.location_meta = merged_meta + else: + record.location_meta = patch["location_meta"] + if target: + record.latitude = patch["latitude"] + record.longitude = patch["longitude"] + record.location_label = patch["location_label"] + record.location_source = patch["location_source"] + record.verified = patch["verified"] + record.resolved_at = now + elif record.location_source != "manual_location": + record.latitude = patch["latitude"] + record.longitude = patch["longitude"] + record.location_label = patch["location_label"] + record.location_source = patch["location_source"] + record.verified = patch["verified"] + record.resolved_at = None + record.enrichment_status = NewsEnrichmentStatus.PENDING.value + record.enrichment_error = None + record.enriched_at = None + if content: + meta = dict(record.location_meta or {}) + meta["manual_content"] = content + record.location_meta = meta + await db.flush() + + queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True) + if queued: + record.enrichment_status = NewsEnrichmentStatus.QUEUED.value + await db.flush() + return ManualNewsWriteResult(item=record, created=created, queued=queued) + + +async def import_manual_news_items( + db: AsyncSession, + payload: list[Any], + *, + group_id: str | None = None, +) -> dict[str, Any]: + if len(payload) > MANUAL_NEWS_MAX_IMPORT_ITEMS: + raise ValueError(f"单次最多导入 {MANUAL_NEWS_MAX_IMPORT_ITEMS} 条。") + created = 0 + updated = 0 + queued = 0 + errors: list[dict[str, Any]] = [] + for index, raw_item in enumerate(payload): + if not isinstance(raw_item, dict): + errors.append({"index": index, "error": "条目必须是 JSON 对象。"}) + continue + try: + result = await upsert_manual_news_item(db, raw_item, group_id=group_id) + created += 1 if result.created else 0 + updated += 0 if result.created else 1 + queued += 1 if result.queued else 0 + except Exception as exc: + errors.append({"index": index, "error": str(exc)}) + if errors and created == 0 and updated == 0: + raise ValueError("导入失败，未写入任何新闻。") + return {"created": created, "updated": updated, "queued": queued, "failed": len(errors), "errors": errors} + + +async def parse_manual_news_import_upload(raw_bytes: bytes) -> list[Any]: + try: + payload = json.loads(raw_bytes.decode("utf-8-sig")) + except UnicodeDecodeError as exc: + raise ValueError("JSON 文件必须使用 UTF-8 编码。") from exc + except json.JSONDecodeError as exc: + raise ValueError(f"JSON 解析失败：第 {exc.lineno} 行第 {exc.colno} 列。") from exc + if not isinstance(payload, list): + raise ValueError("JSON 顶层必须是数组。") + return payload + + +def serialize_news_record(record: EarthNewsItem, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]: + item = record_to_parsed_news_item(record) + payload = _serialize_item(item, active_region=item.feed_region, locale=locale) + news_meta = _news_meta(record) + payload["editable"] = _manual_editable(record) + payload["source_type"] = payload.get("feed_type") + payload["status"] = record.enrichment_status + payload["translated"] = bool((record.localizations or {}).get("zh-CN") and (record.localizations or {}).get("en-US")) + payload["manual_content"] = (record.location_meta or {}).get("manual_content") if isinstance(record.location_meta, dict) else None + payload["manual_group_id"] = news_meta.get("manual_group_id") + payload["manual_group_name"] = news_meta.get("manual_group_name") + return payload + + +def _record_matches_group(record: EarthNewsItem, group_id: str) -> bool: + source_type = _record_source_type(record) + if source_type == NewsSourceType.MANUAL.value: + return _record_manual_group_id(record) == group_id + return _rss_group_id(record) == group_id + + +async def list_news_records( + db: AsyncSession, + *, + page: int, + page_size: int, + source_type: str | None = None, + region: str | None = None, + category: str | None = None, + status_filter: str | None = None, + group_id: str | None = None, +) -> dict[str, Any]: + page = max(page, 1) + page_size = min(max(page_size, 1), 100) + query = select(EarthNewsItem) + count_query = select(func.count(EarthNewsItem.id)) + filters = [] + if region and region != "all": + filters.append(EarthNewsItem.region == region) + if status_filter and status_filter != "all": + filters.append(EarthNewsItem.enrichment_status == status_filter) + if source_type and source_type != "all": + filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == source_type) + if category and category != "all": + filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category") == category) + for clause in filters: + query = query.where(clause) + count_query = count_query.where(clause) + ordered_query = query.order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc()) + if group_id: + result = await db.execute(ordered_query) + all_records = [record for record in result.scalars().all() if _record_matches_group(record, group_id)] + total = len(all_records) + records = all_records[(page - 1) * page_size : page * page_size] + else: + total_result = await db.execute(count_query) + result = await db.execute( + ordered_query.offset((page - 1) * page_size).limit(page_size) + ) + records = list(result.scalars().all()) + total = int(total_result.scalar() or 0) + return { + "items": [serialize_news_record(record) for record in records], + "page": page, + "page_size": page_size, + "total": total, + } + + +async def list_news_groups(db: AsyncSession, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]: + manual_groups = await get_manual_news_groups(db) + manual_by_id: dict[str, dict[str, Any]] = { + str(group["id"]): { + "id": str(group["id"]), + "name": str(group["name"]), + "group_type": "manual", + "source_type": NewsSourceType.MANUAL.value, + "editable": True, + "sort_order": int(group.get("sort_order") or 0), + "count": 0, + "items": [], + } + for group in manual_groups + } + rss_by_id: dict[str, dict[str, Any]] = {} + result = await db.execute( + select(EarthNewsItem).order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc()) + ) + for record in result.scalars().all(): + serialized = serialize_news_record(record, locale=locale) + source_type = _record_source_type(record) + if source_type == NewsSourceType.MANUAL.value: + group_id = _record_manual_group_id(record) + group = manual_by_id.setdefault( + group_id, + { + "id": group_id, + "name": str(_news_meta(record).get("manual_group_name") or DEFAULT_MANUAL_NEWS_GROUP_NAME), + "group_type": "manual", + "source_type": NewsSourceType.MANUAL.value, + "editable": True, + "sort_order": len(manual_by_id), + "count": 0, + "items": [], + }, + ) + else: + group_id = _rss_group_id(record) + group = rss_by_id.setdefault( + group_id, + { + "id": group_id, + "name": record.feed_name or record.source or "RSS 新闻", + "group_type": "rss", + "source_type": source_type, + "editable": False, + "region": record.region, + "source": record.source, + "feed_name": record.feed_name, + "count": 0, + "items": [], + }, + ) + group["count"] = int(group.get("count") or 0) + 1 + group.setdefault("items", []).append(serialized) + manual_items = sorted(manual_by_id.values(), key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or ""))) + rss_items = sorted(rss_by_id.values(), key=lambda item: str(item.get("name") or "")) + return {"groups": [*manual_items, *rss_items], "manual_groups": manual_items, "rss_groups": rss_items} + + +async def get_news_record_or_404(db: AsyncSession, item_id: str) -> EarthNewsItem | None: + return await db.get(EarthNewsItem, item_id) + + +async def delete_manual_news_item(db: AsyncSession, item_id: str) -> bool: + record = await db.get(EarthNewsItem, item_id) + if record is None: + return False + if not _manual_editable(record): + raise PermissionError("RSS 新闻不允许通过手动新闻接口删除。") + await db.execute(delete(EarthNewsItem).where(EarthNewsItem.id == item_id)) + await db.flush() + return True + + +async def reprocess_manual_news_item(db: AsyncSession, item_id: str) -> bool: + record = await db.get(EarthNewsItem, item_id) + if record is None: + return False + if not _manual_editable(record): + raise PermissionError("RSS 新闻不允许通过手动新闻接口重新处理。") + item = record_to_parsed_news_item(record) + queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True) + if queued: + record.enrichment_status = NewsEnrichmentStatus.QUEUED.value + record.enrichment_error = None + await db.flush() + return queued + + +async def broadcast_manual_news_changed() -> None: + await _broadcast_news_reload() diff --git a/backend/app/services/earth_news_queue.py b/backend/app/services/earth_news_queue.py index c4a0e213..83915ea6 100644 --- a/backend/app/services/earth_news_queue.py +++ b/backend/app/services/earth_news_queue.py @@ -14,10 +14,14 @@ from app.core.logging import get_logger logger = get_logger(__name__, service="earth_news") TARGET_LOCATION_STREAM = "earth_news:target_location:jobs" +TARGET_LOCATION_PRIORITY_STREAM = "earth_news:target_location:priority" TARGET_LOCATION_GROUP = "earth_news_target_location" TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead" TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12 TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6 +TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS = 60 * 5 +TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS = 2 * 60 * 1000 +TARGET_LOCATION_PRIORITY_READ_BLOCK_MS = 1 TARGET_LOCATION_MAX_ATTEMPTS = 3 _redis_client: redis.Redis | None = None @@ -28,6 +32,7 @@ class NewsTargetLocationMessage: message_id: str item_id: str payload: dict[str, Any] + stream_name: str = TARGET_LOCATION_STREAM attempts: int = 0 @@ -44,7 +49,7 @@ class NewsTargetLocationQueue(Protocol): ) -> list[NewsTargetLocationMessage]: ... - async def ack(self, message_id: str) -> None: + async def ack(self, message: NewsTargetLocationMessage) -> None: ... async def retry_or_dead_letter( @@ -71,6 +76,10 @@ def _queued_key(item_id: str) -> str: return f"earth_news:target_location:queued:{item_id}" +def _priority_queued_key(item_id: str) -> str: + return f"earth_news:target_location:priority_queued:{item_id}" + + class RedisStreamsNewsTargetLocationQueue: def __init__(self, client: redis.Redis | None = None) -> None: self.client = client or _get_redis_client() @@ -79,34 +88,44 @@ class RedisStreamsNewsTargetLocationQueue: async def _ensure_group(self) -> None: if self._group_ready: return - try: - await self.client.xgroup_create( - TARGET_LOCATION_STREAM, - TARGET_LOCATION_GROUP, - id="0", - mkstream=True, - ) - except ResponseError as exc: - if "BUSYGROUP" not in str(exc): - raise + for stream_name in (TARGET_LOCATION_PRIORITY_STREAM, TARGET_LOCATION_STREAM): + try: + await self.client.xgroup_create( + stream_name, + TARGET_LOCATION_GROUP, + id="0", + mkstream=True, + ) + except ResponseError as exc: + if "BUSYGROUP" not in str(exc): + raise self._group_ready = True async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool: await self._ensure_group() if force: - await self.client.delete(_result_key(item_id), _queued_key(item_id)) + await self.client.delete(_result_key(item_id)) + queued_key = _priority_queued_key(item_id) elif await self.client.exists(_result_key(item_id)): return False + else: + queued_key = _queued_key(item_id) + dedup_ttl = ( + TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS + if force + else TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS + ) queued = await self.client.set( - _queued_key(item_id), + queued_key, "1", nx=True, - ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS, + ex=dedup_ttl, ) if not queued: - return bool(await self.client.exists(_queued_key(item_id))) + return bool(await self.client.exists(queued_key)) + stream_name = TARGET_LOCATION_PRIORITY_STREAM if force else TARGET_LOCATION_STREAM await self.client.xadd( - TARGET_LOCATION_STREAM, + stream_name, { "item_id": item_id, "attempts": "0", @@ -123,39 +142,104 @@ class RedisStreamsNewsTargetLocationQueue: block_ms: int, ) -> list[NewsTargetLocationMessage]: await self._ensure_group() - streams = await self.client.xreadgroup( + streams = [] + priority_claimed = await self._claim_stale_messages( + stream_name=TARGET_LOCATION_PRIORITY_STREAM, + consumer_name=consumer_name, + count=count, + ) + if priority_claimed: + return priority_claimed + + priority_messages = await self.client.xreadgroup( TARGET_LOCATION_GROUP, consumer_name, - {TARGET_LOCATION_STREAM: ">"}, + {TARGET_LOCATION_PRIORITY_STREAM: ">"}, count=count, - block=block_ms, + block=TARGET_LOCATION_PRIORITY_READ_BLOCK_MS, ) + if priority_messages: + streams = priority_messages + else: + regular_claimed = await self._claim_stale_messages( + stream_name=TARGET_LOCATION_STREAM, + consumer_name=consumer_name, + count=count, + ) + if regular_claimed: + return regular_claimed + streams = await self.client.xreadgroup( + TARGET_LOCATION_GROUP, + consumer_name, + {TARGET_LOCATION_STREAM: ">"}, + count=count, + block=block_ms, + ) messages: list[NewsTargetLocationMessage] = [] - for _stream_name, stream_messages in streams: + for stream_name, stream_messages in streams: for message_id, fields in stream_messages: - raw_payload = fields.get("payload") - item_id = fields.get("item_id") - if not raw_payload or not item_id: - await self.ack(message_id) - continue - try: - payload = json.loads(raw_payload) - except json.JSONDecodeError: - await self.ack(message_id) - continue - attempts = int(fields.get("attempts") or 0) - messages.append( - NewsTargetLocationMessage( - message_id=message_id, - item_id=item_id, - payload=payload, - attempts=attempts, - ) - ) + message = await self._message_from_fields(stream_name, message_id, fields) + if message is not None: + messages.append(message) return messages - async def ack(self, message_id: str) -> None: - await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id) + async def _claim_stale_messages( + self, + *, + stream_name: str, + consumer_name: str, + count: int, + ) -> list[NewsTargetLocationMessage]: + try: + _next_id, claimed, _deleted = await self.client.xautoclaim( + stream_name, + TARGET_LOCATION_GROUP, + consumer_name, + TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS, + start_id="0-0", + count=count, + ) + except ResponseError: + return [] + messages: list[NewsTargetLocationMessage] = [] + for message_id, fields in claimed: + message = await self._message_from_fields(stream_name, message_id, fields) + if message is not None: + messages.append(message) + return messages + + async def _message_from_fields( + self, + stream_name: str, + message_id: str, + fields: dict[str, str], + ) -> NewsTargetLocationMessage | None: + raw_payload = fields.get("payload") + item_id = fields.get("item_id") + if not raw_payload or not item_id: + await self._discard_message(stream_name, message_id) + return None + try: + payload = json.loads(raw_payload) + except json.JSONDecodeError: + await self._discard_message(stream_name, message_id) + return None + attempts = int(fields.get("attempts") or 0) + return NewsTargetLocationMessage( + message_id=message_id, + item_id=item_id, + payload=payload, + stream_name=stream_name, + attempts=attempts, + ) + + async def ack(self, message: NewsTargetLocationMessage) -> None: + await self.client.xack(message.stream_name, TARGET_LOCATION_GROUP, message.message_id) + await self.client.xdel(message.stream_name, message.message_id) + + async def _discard_message(self, stream_name: str, message_id: str) -> None: + await self.client.xack(stream_name, TARGET_LOCATION_GROUP, message_id) + await self.client.xdel(stream_name, message_id) async def retry_or_dead_letter( self, @@ -163,7 +247,7 @@ class RedisStreamsNewsTargetLocationQueue: *, error: str, ) -> None: - await self.ack(message.message_id) + await self.ack(message) if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS: await self.client.xadd( TARGET_LOCATION_DEAD_LETTER_STREAM, @@ -176,7 +260,7 @@ class RedisStreamsNewsTargetLocationQueue: ) return await self.client.xadd( - TARGET_LOCATION_STREAM, + message.stream_name, { "item_id": message.item_id, "attempts": str(message.attempts + 1), @@ -231,4 +315,4 @@ async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> Non TARGET_LOCATION_RESULT_TTL_SECONDS, json.dumps(patch, ensure_ascii=False), ) - await client.delete(_queued_key(item_id)) + await client.delete(_queued_key(item_id), _priority_queued_key(item_id)) diff --git a/backend/app/services/earth_news_store.py b/backend/app/services/earth_news_store.py index 910d7791..fa8ad4d2 100644 --- a/backend/app/services/earth_news_store.py +++ b/backend/app/services/earth_news_store.py @@ -3,7 +3,7 @@ from __future__ import annotations from datetime import UTC, datetime from typing import Any -from sqlalchemy import func, select +from sqlalchemy import func, or_, select from sqlalchemy.ext.asyncio import AsyncSession from app.models.earth_news import EarthNewsItem @@ -11,7 +11,24 @@ from app.services.earth_news import ( ParsedNewsItem, apply_enrichment_patch_to_item, build_anchor_location_patch, + _news_meta_patch, ) +from app.services.earth_news_classification import ( + breaking_sort_rank, + normalize_breaking_level, + normalize_breaking_scope, +) + +CRUISE_REGION_ORDER = ( + "americas", + "europe", + "middle-east-africa", + "asia-pacific", + "global", +) +CRUISE_REGION_QUERY_MULTIPLIER = 12 +CRUISE_REGION_QUERY_MIN_LIMIT = 240 +CRUISE_REGION_QUERY_MAX_LIMIT = 1000 def _coerce_datetime(value: datetime | None) -> datetime | None: @@ -22,6 +39,17 @@ def _coerce_datetime(value: datetime | None) -> datetime | None: return value.astimezone(UTC) +def _coerce_meta_datetime(value: Any) -> datetime | None: + if isinstance(value, datetime): + return _coerce_datetime(value) + if not isinstance(value, str) or not value.strip(): + return None + try: + return _coerce_datetime(datetime.fromisoformat(value.replace("Z", "+00:00"))) + except ValueError: + return None + + def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]: return { "latitude": record.latitude, @@ -34,6 +62,8 @@ def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]: def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem: + location_meta = dict(record.location_meta or {}) + news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {} item = ParsedNewsItem( id=record.id, title=record.title, @@ -49,11 +79,86 @@ def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem: enrichment_status=record.enrichment_status or "pending", enrichment_error=record.enrichment_error, enriched_at=_coerce_datetime(record.enriched_at), + source_tags=list(news_meta.get("source_tags") or []), + feed_id=str(news_meta.get("feed_id") or ""), + feed_type=str(news_meta.get("feed_type") or "rss"), + feed_default_category=str(news_meta.get("feed_default_category") or "other"), + category=str(news_meta.get("category") or "other"), + item_tags=list(news_meta.get("item_tags") or []), + tagging_source=str(news_meta.get("tagging_source") or "rules"), + tagging_confidence=float(news_meta.get("tagging_confidence") or 0), + importance_score=int(news_meta.get("importance_score") or 0), + importance_level=str(news_meta.get("importance_level") or "low"), + importance_reasons=list(news_meta.get("importance_reasons") or []), + market_impact=str(news_meta.get("market_impact") or "none"), + breaking_level=normalize_breaking_level(news_meta.get("breaking_level")).value, + breaking_scope=normalize_breaking_scope(news_meta.get("breaking_scope")).value, + breaking_reasons=list(news_meta.get("breaking_reasons") or []), + breaking_source=str(news_meta.get("breaking_source") or "rules"), + breaking_confidence=float(news_meta.get("breaking_confidence") or 0), + breaking_expires_at=_coerce_meta_datetime(news_meta.get("breaking_expires_at")), ) return apply_enrichment_patch_to_item(item, _location_patch_from_record(record)) +def _sort_parsed_news_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]: + return sorted( + items, + key=lambda item: ( + -breaking_sort_rank(item), + False + if active_region == "global" + or (breaking_sort_rank(item) > 0 and normalize_breaking_scope(item.breaking_scope).value == "global") + else item.feed_region != active_region, + item.published_at is None, + -(item.published_at.timestamp() if item.published_at else 0), + item.feed_name, + ), + ) + + +def _diversify_parsed_news_items_by_region( + items: list[ParsedNewsItem], + *, + limit: int, +) -> list[ParsedNewsItem]: + if limit <= 0: + return [] + sorted_items = _sort_parsed_news_items(items, active_region="global") + buckets: dict[str, list[ParsedNewsItem]] = {} + for item in sorted_items: + region = item.feed_region or "global" + buckets.setdefault(region, []).append(item) + + ordered_regions = [ + *[region for region in CRUISE_REGION_ORDER if buckets.get(region)], + *sorted(region for region in buckets if region not in CRUISE_REGION_ORDER), + ] + diversified: list[ParsedNewsItem] = [] + cursor = 0 + while len(diversified) < limit: + added = False + for region in ordered_regions: + bucket = buckets.get(region) or [] + if cursor >= len(bucket): + continue + diversified.append(bucket[cursor]) + added = True + if len(diversified) >= limit: + break + if not added: + break + cursor += 1 + return diversified + + def _query_sort_key(active_region: str): + if active_region == "global": + return ( + EarthNewsItem.published_at.is_(None), + EarthNewsItem.published_at.desc().nullslast(), + EarthNewsItem.feed_name.asc(), + ) return ( EarthNewsItem.region != active_region, EarthNewsItem.published_at.is_(None), @@ -62,20 +167,90 @@ def _query_sort_key(active_region: str): ) +def _category_filter_clause(categories: set[str] | None): + if not categories: + return None + return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category").in_(sorted(categories)) + + +def _source_filter_clause(source_ids: set[str] | None): + if not source_ids: + return None + return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id").in_(sorted(source_ids)) + + async def list_earth_news_items( db: AsyncSession, *, active_region: str, limit: int, + categories: set[str] | None = None, + source_ids: set[str] | None = None, ) -> list[ParsedNewsItem]: - regions = {"global", active_region} - result = await db.execute( + query_limit = limit if source_ids else min(max(limit * 20, limit), 500) + query = ( select(EarthNewsItem) - .where(EarthNewsItem.region.in_(regions)) .order_by(*_query_sort_key(active_region)) - .limit(limit) + .limit(query_limit) + ) + if active_region != "global": + news_meta = EarthNewsItem.location_meta.op("->")("news_meta") + query = query.where( + or_( + EarthNewsItem.region.in_({"global", active_region}), + news_meta.op("->>")("breaking_scope") == "global", + ) + ) + category_clause = _category_filter_clause(categories) + if category_clause is not None: + query = query.where(category_clause) + source_clause = _source_filter_clause(source_ids) + if source_clause is not None: + query = query.where(source_clause) + result = await db.execute(query) + records = list(result.scalars().all()) + items = _sort_parsed_news_items( + [record_to_parsed_news_item(record) for record in records], + active_region=active_region, + ) + if active_region == "global" and not source_ids: + return _diversify_parsed_news_items_by_region(items, limit=limit) + return items[:limit] + + +async def list_earth_news_cruise_items( + db: AsyncSession, + *, + limit: int, + categories: set[str] | None = None, + source_ids: set[str] | None = None, +) -> list[ParsedNewsItem]: + query_limit = min( + max(limit * CRUISE_REGION_QUERY_MULTIPLIER, CRUISE_REGION_QUERY_MIN_LIMIT), + CRUISE_REGION_QUERY_MAX_LIMIT, + ) + query = ( + select(EarthNewsItem) + .order_by( + EarthNewsItem.published_at.is_(None), + EarthNewsItem.published_at.desc().nullslast(), + EarthNewsItem.last_seen_at.desc(), + EarthNewsItem.region.asc(), + EarthNewsItem.feed_name.asc(), + ) + .limit(query_limit) + ) + category_clause = _category_filter_clause(categories) + if category_clause is not None: + query = query.where(category_clause) + source_clause = _source_filter_clause(source_ids) + if source_clause is not None: + query = query.where(source_clause) + result = await db.execute(query) + return _diversify_parsed_news_items_by_region( + [record_to_parsed_news_item(record) for record in result.scalars().all()], + limit=limit, ) - return [record_to_parsed_news_item(record) for record in result.scalars().all()] async def get_earth_news_freshness( @@ -83,13 +258,13 @@ async def get_earth_news_freshness( *, active_region: str, ) -> tuple[int, datetime | None]: - regions = {"global", active_region} - result = await db.execute( - select( - func.count(EarthNewsItem.id), - func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)), - ).where(EarthNewsItem.region.in_(regions)) + query = select( + func.count(EarthNewsItem.id), + func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)), ) + if active_region != "global": + query = query.where(EarthNewsItem.region.in_({"global", active_region})) + result = await db.execute(query) count, newest = result.one() item_count = int(count or 0) if item_count == 0: @@ -97,6 +272,33 @@ async def get_earth_news_freshness( return item_count, _coerce_datetime(newest) +async def get_earth_news_feed_coverage( + db: AsyncSession, + *, + active_region: str, + recent_after: datetime | None = None, +) -> set[tuple[str, str]]: + query = select( + EarthNewsItem.id, + EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"), + EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"), + ) + if active_region != "global": + query = query.where(EarthNewsItem.region.in_({"global", active_region})) + if recent_after is not None: + query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after) + result = await db.execute(query) + coverage: set[tuple[str, str]] = set() + for item_id, source_id, feed_id in result.all(): + normalized_source_id = str(source_id or "").strip() + normalized_feed_id = str(feed_id or "").strip() + if not normalized_source_id and isinstance(item_id, str) and ":" in item_id: + normalized_source_id = item_id.split(":", 1)[0] + if normalized_source_id and normalized_feed_id: + coverage.add((normalized_source_id, normalized_feed_id)) + return coverage + + async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem]) -> int: if not items: return 0 @@ -147,12 +349,20 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem]) record.homepage_url = item.homepage_url record.published_at = item.published_at record.last_seen_at = now + location_meta = dict(record.location_meta or {}) + location_meta["news_meta"] = _news_meta_patch(item) + record.location_meta = location_meta if item.localizations: + merged_localizations = { + **dict(record.localizations or {}), + **dict(item.localizations or {}), + } record.content_language = item.content_language - record.localizations = dict(item.localizations or {}) - record.enrichment_status = item.enrichment_status - record.enrichment_error = item.enrichment_error - record.enriched_at = item.enriched_at + record.localizations = merged_localizations + if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at: + record.enrichment_status = item.enrichment_status + record.enrichment_error = item.enrichment_error + record.enriched_at = item.enriched_at changed += 1 await db.flush() return changed @@ -188,13 +398,28 @@ async def update_earth_news_item_enrichment( if record is None: return False if "latitude" in patch: - record.latitude = float(patch["latitude"]) - record.longitude = float(patch["longitude"]) - record.location_label = str(patch["location_label"]) - record.location_source = str(patch["location_source"]) - record.verified = bool(patch["verified"]) - record.location_meta = dict(patch.get("location_meta") or {}) - record.resolved_at = datetime.now(UTC) if record.verified else None + patch_meta = dict(patch.get("location_meta") or {}) + if record.location_source == "manual_location": + current_meta = dict(record.location_meta or {}) + patch_news_meta = patch_meta.get("news_meta") + if isinstance(patch_news_meta, dict): + current_meta["news_meta"] = patch_news_meta + current_meta["manual_enrichment"] = { + "resolution_stage": patch_meta.get("resolution_stage"), + "ai_attempted": patch_meta.get("ai_attempted"), + "ai_status": patch_meta.get("ai_status"), + "ai_error": patch_meta.get("ai_error"), + "debug_note": patch_meta.get("debug_note"), + } + record.location_meta = current_meta + else: + record.latitude = float(patch["latitude"]) + record.longitude = float(patch["longitude"]) + record.location_label = str(patch["location_label"]) + record.location_source = str(patch["location_source"]) + record.verified = bool(patch["verified"]) + record.location_meta = patch_meta + record.resolved_at = datetime.now(UTC) if record.verified else None if "content_language" in patch: record.content_language = str(patch.get("content_language") or "en") if "localizations" in patch: diff --git a/backend/app/services/earth_news_worker.py b/backend/app/services/earth_news_worker.py index b53442f3..bb2a319d 100644 --- a/backend/app/services/earth_news_worker.py +++ b/backend/app/services/earth_news_worker.py @@ -29,6 +29,9 @@ logger = get_logger(__name__, service="earth_news") WORKER_BATCH_SIZE = 4 WORKER_BLOCK_MS = 5000 WORKER_BACKOFF_SECONDS = 5.0 +WORKER_JOB_TIMEOUT_MIN_SECONDS = 20.0 +WORKER_JOB_TIMEOUT_MAX_SECONDS = 90.0 +WORKER_JOB_TIMEOUT_GRACE_SECONDS = 10.0 _worker_task: asyncio.Task | None = None @@ -109,12 +112,25 @@ async def _run_target_location_worker() -> None: if not messages: continue provider_client = await _build_provider_client() - for message in messages: + job_timeout = _get_worker_job_timeout(provider_client) + + async def handle_message(message: NewsTargetLocationMessage) -> None: try: - await process_target_location_message(message, provider_client=provider_client) - await queue.ack(message.message_id) + await asyncio.wait_for( + process_target_location_message(message, provider_client=provider_client), + timeout=job_timeout, + ) + await queue.ack(message) except asyncio.CancelledError: raise + except TimeoutError as exc: + logger.warning_event( + "Earth news target location worker job timed out", + event="earth_news.target_location.worker_job_timeout", + context={"item_id": message.item_id, "timeout_seconds": job_timeout}, + ) + with suppress(Exception): + await queue.retry_or_dead_letter(message, error=str(exc) or "job timed out") except Exception as exc: logger.warning_event( "Earth news target location worker job failed", @@ -124,6 +140,8 @@ async def _run_target_location_worker() -> None: with suppress(Exception): await queue.retry_or_dead_letter(message, error=str(exc)) + await asyncio.gather(*(handle_message(message) for message in messages)) + def start_earth_news_target_worker() -> None: global _worker_task @@ -140,3 +158,15 @@ async def stop_earth_news_target_worker() -> None: with suppress(asyncio.CancelledError): await task _worker_task = None + + +def _get_worker_job_timeout(provider_client: AIProviderClient | None) -> float: + timeout = float(getattr(provider_client, "timeout", 0) or WORKER_JOB_TIMEOUT_MIN_SECONDS) + retry_attempts = float(getattr(provider_client, "retry_attempts", 1) or 1) + return min( + max( + timeout * retry_attempts + WORKER_JOB_TIMEOUT_GRACE_SECONDS, + WORKER_JOB_TIMEOUT_MIN_SECONDS, + ), + WORKER_JOB_TIMEOUT_MAX_SECONDS, + ) diff --git a/backend/app/services/email.py b/backend/app/services/email.py index 6db4fe32..c744c68a 100644 --- a/backend/app/services/email.py +++ b/backend/app/services/email.py @@ -9,12 +9,12 @@ yet to keep behavior obvious after settings changes). from __future__ import annotations from email.message import EmailMessage -from typing import Literal, Optional +from typing import Optional import aiosmtplib from sqlalchemy.ext.asyncio import AsyncSession -OtpPurpose = Literal["register", "verify_email", "reset_password"] +from app.core.enums import OtpPurpose class EmailError(Exception): @@ -81,15 +81,15 @@ async def send_email( _SUBJECTS: dict[OtpPurpose, str] = { - "register": "Confirm your Planet account", - "verify_email": "Verify your Planet email", - "reset_password": "Reset your Planet password", + OtpPurpose.REGISTER: "Confirm your Planet account", + OtpPurpose.VERIFY_EMAIL: "Verify your Planet email", + OtpPurpose.RESET_PASSWORD: "Reset your Planet password", } _HEADLINES: dict[OtpPurpose, str] = { - "register": "Welcome to Planet — confirm your email to activate your account.", - "verify_email": "Confirm your new email address to keep your Planet account active.", - "reset_password": "Use this code to set a new password for your Planet account.", + OtpPurpose.REGISTER: "Welcome to Planet — confirm your email to activate your account.", + OtpPurpose.VERIFY_EMAIL: "Confirm your new email address to keep your Planet account active.", + OtpPurpose.RESET_PASSWORD: "Use this code to set a new password for your Planet account.", } diff --git a/backend/app/services/log_tail.py b/backend/app/services/log_tail.py new file mode 100644 index 00000000..14797027 --- /dev/null +++ b/backend/app/services/log_tail.py @@ -0,0 +1,161 @@ +from __future__ import annotations + +import asyncio +from dataclasses import dataclass, field +from datetime import UTC, datetime +from typing import Any + +from fastapi import WebSocket + +from app.db.session import async_session_factory +from app.services.system_logs import ( + DEFAULT_LOG_LINE_LIMIT, + LOG_SOURCES, + MAX_LOG_LINE_LIMIT, + read_database_log_events, + read_log_events, +) + +DATABASE_LOG_SOURCE_IDS = {"system-db", "audit-db"} +LOG_TAIL_CHANNEL = "logs_tail" +LOG_TAIL_INTERVAL_SECONDS = 1.5 +LOG_TAIL_SCAN_MULTIPLIER = 5 + + +@dataclass(frozen=True) +class LogTailConfig: + source_id: str + limit: int = DEFAULT_LOG_LINE_LIMIT + level: str = "all" + levels: str | None = None + start_date: str | None = None + end_date: str | None = None + search: str | None = None + + +@dataclass +class LogTailSubscription: + config: LogTailConfig + emitted_cursors: set[str] = field(default_factory=set) + task: asyncio.Task | None = None + + +class LogTailManager: + def __init__(self) -> None: + self._subscriptions: dict[WebSocket, LogTailSubscription] = {} + + def normalize_config(self, payload: dict[str, Any]) -> LogTailConfig: + source_id = str(payload.get("source_id") or payload.get("source") or "").strip() + if not source_id: + raise ValueError("source_id is required") + if source_id not in LOG_SOURCES and source_id not in DATABASE_LOG_SOURCE_IDS: + raise ValueError("Log source not found") + try: + limit = int(payload.get("limit") or DEFAULT_LOG_LINE_LIMIT) + except (TypeError, ValueError) as exc: + raise ValueError("limit must be a number") from exc + if limit < 1 or limit > MAX_LOG_LINE_LIMIT: + raise ValueError(f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}") + return LogTailConfig( + source_id=source_id, + limit=limit, + level=str(payload.get("level") or "all"), + levels=str(payload.get("levels")).strip() if payload.get("levels") else None, + start_date=str(payload.get("start_date")).strip() if payload.get("start_date") else None, + end_date=str(payload.get("end_date")).strip() if payload.get("end_date") else None, + search=str(payload.get("search")).strip() if payload.get("search") else None, + ) + + async def subscribe(self, websocket: WebSocket, payload: dict[str, Any]) -> LogTailConfig: + config = self.normalize_config(payload) + await self.unsubscribe(websocket) + subscription = LogTailSubscription(config=config) + subscription.task = asyncio.create_task(self._run_tail(websocket, subscription)) + self._subscriptions[websocket] = subscription + return config + + async def unsubscribe(self, websocket: WebSocket) -> None: + subscription = self._subscriptions.pop(websocket, None) + if subscription and subscription.task: + subscription.task.cancel() + try: + await subscription.task + except asyncio.CancelledError: + pass + + async def disconnect(self, websocket: WebSocket) -> None: + await self.unsubscribe(websocket) + + async def _run_tail(self, websocket: WebSocket, subscription: LogTailSubscription) -> None: + first_frame = True + while True: + events = await self._read_events(subscription.config) + if first_frame: + visible_events = events[-subscription.config.limit :] + subscription.emitted_cursors.update(event.cursor for event in visible_events) + await self._send_frame(websocket, subscription.config, "snapshot", visible_events) + first_frame = False + else: + new_events = [ + event + for event in events + if event.cursor not in subscription.emitted_cursors + ] + if new_events: + visible_events = new_events[-subscription.config.limit :] + subscription.emitted_cursors.update(event.cursor for event in visible_events) + await self._send_frame(websocket, subscription.config, "append", visible_events) + await asyncio.sleep(LOG_TAIL_INTERVAL_SECONDS) + + async def _read_events(self, config: LogTailConfig): + scan_limit = max(config.limit * LOG_TAIL_SCAN_MULTIPLIER, config.limit) + if config.source_id in DATABASE_LOG_SOURCE_IDS: + async with async_session_factory() as db: + events = await read_database_log_events( + config.source_id, + scan_limit=scan_limit, + level=config.level, + levels=config.levels, + start_date=config.start_date, + end_date=config.end_date, + search=config.search, + db=db, + ) + return events or [] + events = read_log_events( + config.source_id, + scan_limit=scan_limit, + level=config.level, + levels=config.levels, + start_date=config.start_date, + end_date=config.end_date, + search=config.search, + ) + return events or [] + + async def _send_frame(self, websocket: WebSocket, config: LogTailConfig, mode: str, events) -> None: + await websocket.send_json( + { + "type": "data_frame", + "channel": LOG_TAIL_CHANNEL, + "timestamp": datetime.now(UTC).isoformat(), + "payload": { + "mode": mode, + "source_id": config.source_id, + "line_count": len(events), + "lines": [event.line for event in events], + "filters": { + "limit": config.limit, + "level": config.level, + "levels": config.levels, + "start_date": config.start_date, + "end_date": config.end_date, + "search": config.search, + }, + "status": "ok", + }, + } + ) + + +log_tail_manager = LogTailManager() diff --git a/backend/app/services/otp.py b/backend/app/services/otp.py index 4d9cafcf..e144809e 100644 --- a/backend/app/services/otp.py +++ b/backend/app/services/otp.py @@ -9,14 +9,12 @@ from __future__ import annotations import json import secrets -from typing import Literal import bcrypt +from app.core.enums import OtpPurpose from app.core.security import redis_client -OtpPurpose = Literal["register", "verify_email", "reset_password"] - CODE_TTL_SECONDS = 600 # 10 minutes RESEND_COOLDOWN_SECONDS = 60 MAX_ATTEMPTS = 5 diff --git a/backend/app/services/persistent_logs.py b/backend/app/services/persistent_logs.py index 838c9f7c..8f326ecc 100644 --- a/backend/app/services/persistent_logs.py +++ b/backend/app/services/persistent_logs.py @@ -1,14 +1,185 @@ from __future__ import annotations +import hashlib +import re + +from datetime import UTC, datetime from typing import Any from app.core.logging import get_logger, sanitize_log_value from app.core.request_context import get_request_id from app.db.session import async_session_factory -from app.models.system_log import AuditLog, SystemLog +from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog logger = get_logger(__name__) +HLS_TRANSIENT_RE = re.compile(r"(index|chunk|segment)[_-]?\d+(?:_\d+)?\.(?:ts|m4s|vtt)", re.IGNORECASE) +QUERY_RE = re.compile(r"([?&](?:m|t|token|expires|signature|X-Amz-[^=]+)=[^&\\s]+)", re.IGNORECASE) +UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.IGNORECASE) +CONNECTION_RE = re.compile(r"\bconn_[A-Za-z0-9:._-]+\b") +NUMBER_RE = re.compile(r"\b\d{5,}\b") + + +def normalize_observability_text(value: Any) -> str: + text = str(sanitize_log_value(value or "")).strip() + text = QUERY_RE.sub("", text) + text = HLS_TRANSIENT_RE.sub("", text) + text = UUID_RE.sub("", text) + text = CONNECTION_RE.sub("", text) + text = NUMBER_RE.sub("", text) + return re.sub(r"\s+", " ", text).strip() + + +def build_observability_fingerprint( + *, + source: str, + service: str | None = None, + module: str | None = None, + category: str | None = None, + event: str | None = None, + message: str, + context: dict[str, Any] | None = None, +) -> str: + context = context or {} + stable_context = { + key: context.get(key) + for key in ( + "task_type", + "source_id", + "source", + "provider", + "status_code", + "error_type", + "details", + ) + if context.get(key) not in (None, "") + } + raw = "|".join( + [ + normalize_observability_text(source), + normalize_observability_text(service), + normalize_observability_text(module), + normalize_observability_text(category), + normalize_observability_text(event), + normalize_observability_text(message), + normalize_observability_text(stable_context), + ] + ) + return hashlib.sha1(raw.encode("utf-8", errors="replace")).hexdigest() + + +def _context_text(context: dict[str, Any] | None, key: str) -> str | None: + value = (context or {}).get(key) + if value in (None, ""): + return None + return str(value) + + +async def record_observability_event( + *, + source: str, + level: str, + message: str, + service: str | None = None, + module: str | None = None, + event: str | None = None, + request_id: str | None = None, + trace_id: str | None = None, + user_id: int | None = None, + category: str | None = None, + context: dict[str, Any] | None = None, + fingerprint: str | None = None, + occurred_at: datetime | None = None, + occurrence_count: int = 1, +) -> None: + normalized_context = sanitize_log_value(context or {}) + if not isinstance(normalized_context, dict): + normalized_context = {"value": normalized_context} + safe_message = str(sanitize_log_value(message)) + normalized_level = str(level or "info").lower() + count = max(1, int(occurrence_count or 1)) + event_time = occurred_at or datetime.now(UTC) + event_fingerprint = fingerprint or build_observability_fingerprint( + source=source, + service=service, + module=module, + category=category, + event=event, + message=safe_message, + context=normalized_context, + ) + detail = _context_text(normalized_context, "detail") or _context_text(normalized_context, "error") + affected_sources = sorted( + { + item + for item in ( + source, + service, + module, + _context_text(normalized_context, "source_id"), + _context_text(normalized_context, "source"), + ) + if item + } + ) + try: + async with async_session_factory() as session: + session.add( + ObservabilityEvent( + source=source, + service=service, + module=module, + category=category, + event=event, + level=normalized_level, + message=safe_message, + fingerprint=event_fingerprint, + occurred_at=event_time, + request_id=request_id or get_request_id(), + trace_id=trace_id, + user_id=user_id, + task_id=_context_text(normalized_context, "task_id"), + source_ref_id=_context_text(normalized_context, "source_id") or _context_text(normalized_context, "source"), + provider=_context_text(normalized_context, "provider"), + context=normalized_context, + occurrence_count=count, + ) + ) + group = await session.get(ObservabilityEventGroup, event_fingerprint) + if group is None: + session.add( + ObservabilityEventGroup( + fingerprint=event_fingerprint, + source=source, + service=service, + module=module, + category=category, + event=event, + last_level=normalized_level, + sample_message=safe_message, + sample_detail=detail, + affected_sources=affected_sources, + count=count, + first_seen_at=event_time, + last_seen_at=event_time, + ) + ) + else: + group.count = int(group.count or 0) + count + group.last_seen_at = event_time + group.last_level = normalized_level + group.sample_message = safe_message + group.sample_detail = detail + merged_sources = sorted(set(group.affected_sources or []) | set(affected_sources)) + group.affected_sources = merged_sources + await session.commit() + except Exception: + logger.exception_event( + "Failed to persist observability event", + event="observability_event.persist.failed", + context={"event_name": event, "source": source}, + ) + async def record_system_log( *, @@ -23,6 +194,8 @@ async def record_system_log( user_id: int | None = None, category: str | None = None, context: dict[str, Any] | None = None, + fingerprint: str | None = None, + occurrence_count: int = 1, ) -> None: try: async with async_session_factory() as session: @@ -48,6 +221,21 @@ async def record_system_log( event="system_log.persist.failed", context={"event_name": event, "source": source}, ) + await record_observability_event( + source=source, + service=service, + module=module, + event=event, + level=level, + message=message, + request_id=request_id, + trace_id=trace_id, + user_id=user_id, + category=category, + context=context, + fingerprint=fingerprint, + occurrence_count=occurrence_count, + ) async def record_audit_log( diff --git a/backend/app/services/playground_chat_service.py b/backend/app/services/playground_chat_service.py index 0edcc3b2..41674896 100644 --- a/backend/app/services/playground_chat_service.py +++ b/backend/app/services/playground_chat_service.py @@ -7,9 +7,15 @@ from time import perf_counter from uuid import uuid4 from fastapi import HTTPException, status -from sqlalchemy import func, select, update +from sqlalchemy import func, select from sqlalchemy.ext.asyncio import AsyncSession +from app.core.enums import ( + PlaygroundMessageKind, + PlaygroundMessageRole, + PlaygroundMessageStatus, +) +from app.core.logging import get_logger from app.db.session import async_session_factory from app.models.playground_message import PlaygroundMessage from app.models.playground_session import PlaygroundSession @@ -20,20 +26,28 @@ from app.schemas.ai import ( PlaygroundMessageRecord, PlaygroundMessageResendRequest, PlaygroundMessageStopRequest, - PlaygroundSessionResponse, PlaygroundSessionState, PlaygroundSessionUpsertRequest, PlaygroundThreadResponse, SituationalAnalysisRequest, ) from app.services.ai_client import AIProviderClient +from app.services.business_logs import emit_business_log, exception_context from app.services.playground_session_store import _to_response as session_to_response from app.services.playground_session_store import upsert_playground_session +logger = get_logger(__name__, service="ai") STREAM_CHUNK_SIZE = 24 STREAM_INTERVAL_SECONDS = 0.08 THINKING_PREVIEW_SECONDS = 2.6 ORPHANED_RUN_MESSAGE = "后台生成任务已中断，请点击上一条用户消息的重试按钮重新生成。" +ACTIVE_MESSAGE_STATUSES = frozenset( + { + PlaygroundMessageStatus.PENDING.value, + PlaygroundMessageStatus.THINKING.value, + PlaygroundMessageStatus.ANSWERING.value, + } +) class _ActiveRun: @@ -90,7 +104,11 @@ async def _require_visible_message( result = await db.execute(select(PlaygroundMessage).where(*conditions)) message = result.scalar_one_or_none() if message is None: - detail = "User message not found" if role == "user" else "Playground message not found" + detail = ( + "User message not found" + if role == PlaygroundMessageRole.USER.value + else "Playground message not found" + ) raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail) return message @@ -105,7 +123,7 @@ def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None content=message.content or "", thinking_content=message.thinking_content or "", meta=list(message.meta or []), - markdown=message.role != "system", + markdown=message.role != PlaygroundMessageRole.SYSTEM.value, provider=message.provider, model=message.model, request_id=message.request_id, @@ -194,11 +212,11 @@ async def _reconcile_orphaned_active_messages( ) -> list[PlaygroundMessage]: changed = False for item in messages: - if item.status not in {"pending", "thinking", "answering"}: + if item.status not in ACTIVE_MESSAGE_STATUSES: continue if item.public_id in _ACTIVE_RUNS: continue - item.status = "error" + item.status = PlaygroundMessageStatus.ERROR.value item.content = item.content or ORPHANED_RUN_MESSAGE orphan_meta = "错误: 后台任务已中断" if orphan_meta not in (item.meta or []): @@ -327,9 +345,9 @@ async def create_turn( public_id=uuid4().hex, session_id=session.id, user_id=user_id, - role="user", - kind="message", - status="done", + role=PlaygroundMessageRole.USER.value, + kind=PlaygroundMessageKind.MESSAGE.value, + status=PlaygroundMessageStatus.DONE.value, title=payload.selected_preset_key, content=payload.input, meta=[payload.title], @@ -340,9 +358,9 @@ async def create_turn( session_id=session.id, user_id=user_id, parent_message_id=None, - role="assistant", - kind="thinking", - status="pending", + role=PlaygroundMessageRole.ASSISTANT.value, + kind=PlaygroundMessageKind.THINKING.value, + status=PlaygroundMessageStatus.PENDING.value, title="AI 回应", content="", thinking_content="", @@ -394,9 +412,9 @@ async def _create_assistant_retry_turn( session_id=session.id, user_id=user_id, parent_message_id=user_message.id, - role="assistant", - kind="thinking", - status="pending", + role=PlaygroundMessageRole.ASSISTANT.value, + kind=PlaygroundMessageKind.THINKING.value, + status=PlaygroundMessageStatus.PENDING.value, title="AI 回应", content="", thinking_content="", @@ -436,7 +454,7 @@ async def stop_message( session = await _require_session(db, user_id=user_id, session_key=payload.session_key) message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id) - if message.status not in {"pending", "thinking", "answering"}: + if message.status not in ACTIVE_MESSAGE_STATUSES: return await _build_action_response(db, session=session) active_run = _ACTIVE_RUNS.get(message.public_id) @@ -444,7 +462,7 @@ async def stop_message( active_run.stop_requested.set() active_run.task.cancel() - message.status = "stopped" + message.status = PlaygroundMessageStatus.STOPPED.value if "已手动停止生成" not in (message.meta or []): message.meta = [*(message.meta or []), "已手动停止生成"] await db.flush() @@ -466,7 +484,7 @@ async def resend_turn( db, user_id=user_id, public_id=payload.user_message_id, - role="user", + role=PlaygroundMessageRole.USER.value, ) later_messages = await db.execute( @@ -478,7 +496,7 @@ async def resend_turn( ) for item in later_messages.scalars().all(): item.is_visible = False - if item.status in {"pending", "thinking", "answering"}: + if item.status in ACTIVE_MESSAGE_STATUSES: active_run = _ACTIVE_RUNS.get(item.public_id) if active_run is not None: active_run.stop_requested.set() @@ -517,7 +535,7 @@ async def edit_user_message( db, user_id=user_id, public_id=payload.user_message_id, - role="user", + role=PlaygroundMessageRole.USER.value, ) user_message.content = payload.content.strip() @@ -563,12 +581,12 @@ def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_u for item in messages: if item.id >= current_user_message_id: break - if item.role == "system": + if item.role == PlaygroundMessageRole.SYSTEM.value: continue history.append( { "role": item.role, - "kind": item.kind or "message", + "kind": item.kind or PlaygroundMessageKind.MESSAGE.value, "title": item.title, "content": item.content or "", } @@ -624,13 +642,52 @@ async def _run_assistant_message( thinking={"type": "enabled"}, ) + await emit_business_log( + logger, + event="ai.playground.run.start", + message="Playground AI run started", + category="ai", + service="ai", + module=__name__, + request_id=request_id, + user_id=user_id, + context={ + "session_id": session_id, + "session_key": session_key, + "user_message_id": user_message_id, + "assistant_message_id": assistant_message_id, + "preset": payload.selected_preset_key, + }, + ) analysis = await provider_client.analyze(request_payload, request_id=request_id) + await emit_business_log( + logger, + event="ai.playground.run.success", + message="Playground AI run completed", + category="ai", + service="ai", + module=__name__, + request_id=request_id, + user_id=user_id, + context={ + "session_id": session_id, + "session_key": session_key, + "provider": analysis.provider, + "model": analysis.model, + "content_block_count": len(analysis.content_blocks or []), + "thinking_block_count": len(analysis.thinking_blocks or []), + }, + ) async with async_session_factory() as db: assistant_message = await _mark_message_state( db, message_id=assistant_message_id, - status="thinking" if analysis.thinking_blocks else "answering", + status=( + PlaygroundMessageStatus.THINKING.value + if analysis.thinking_blocks + else PlaygroundMessageStatus.ANSWERING.value + ), title=f"{analysis.provider} / {analysis.model}", provider=analysis.provider, model=analysis.model, @@ -668,7 +725,7 @@ async def _run_assistant_message( await _mark_message_state( db, message_id=assistant_message_id, - status="answering", + status=PlaygroundMessageStatus.ANSWERING.value, content=content[:cursor], ) await db.commit() @@ -679,7 +736,7 @@ async def _run_assistant_message( assistant_message = await _mark_message_state( db, message_id=assistant_message_id, - status="done", + status=PlaygroundMessageStatus.DONE.value, content=content, meta=[ f"Request ID: {request_id}", @@ -704,23 +761,60 @@ async def _run_assistant_message( await db.flush() await db.commit() except asyncio.CancelledError: + await emit_business_log( + logger, + event="ai.playground.run.cancelled", + message="Playground AI run cancelled", + category="ai", + level="warning", + service="ai", + module=__name__, + request_id=request_id, + user_id=user_id, + context={ + "session_id": session_id, + "session_key": session_key, + "assistant_message_id": assistant_message_id, + "duration_ms": round((perf_counter() - started_at) * 1000), + }, + ) async with async_session_factory() as db: result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id)) message = result.scalar_one_or_none() - if message is not None and message.status in {"pending", "thinking", "answering"}: - message.status = "stopped" + if message is not None and message.status in ACTIVE_MESSAGE_STATUSES: + message.status = PlaygroundMessageStatus.STOPPED.value if "已手动停止生成" not in (message.meta or []): message.meta = [*(message.meta or []), "已手动停止生成"] await db.flush() await db.commit() raise except Exception as exc: + await emit_business_log( + logger, + event="ai.playground.run.failed", + message="Playground AI run failed", + category="ai", + level="error", + service="ai", + module=__name__, + request_id=request_id, + user_id=user_id, + context=exception_context( + exc, + { + "session_id": session_id, + "session_key": session_key, + "assistant_message_id": assistant_message_id, + "duration_ms": round((perf_counter() - started_at) * 1000), + }, + ), + ) error_message = _format_run_exception(exc) async with async_session_factory() as db: result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id)) message = result.scalar_one_or_none() if message is not None: - message.status = "error" + message.status = PlaygroundMessageStatus.ERROR.value message.content = message.content or f"分析失败：{error_message}" message.meta = [ *(message.meta or []), diff --git a/backend/app/services/scheduler.py b/backend/app/services/scheduler.py index ae3a6b3b..056ad827 100644 --- a/backend/app/services/scheduler.py +++ b/backend/app/services/scheduler.py @@ -8,11 +8,13 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler from apscheduler.triggers.interval import IntervalTrigger from sqlalchemy import select +from app.core.enums import JobStatus from app.core.logging import get_logger from app.db.session import async_session_factory from app.core.time import to_iso8601_utc from app.models.datasource import DataSource from app.models.task import CollectionTask +from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context from app.services.collectors.registry import collector_registry from app.services.datasource_connectivity import ( build_builtin_connectivity_checksum, @@ -124,13 +126,22 @@ async def run_collector_task(collector_name: str): event="collector.run.skipped_disabled", context={"collector_name": collector_name}, ) + await emit_business_log( + logger, + event="collector.run.skipped_disabled", + message="Skipping disabled collector", + category="collector", + service="scheduler", + module=__name__, + context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"}, + ) return running_result = await db.execute( select(CollectionTask) .where( CollectionTask.datasource_id == datasource.id, - CollectionTask.status == "running", + CollectionTask.status == JobStatus.RUNNING.value, ) .order_by(CollectionTask.started_at.desc(), CollectionTask.id.desc()) .limit(1) @@ -152,6 +163,21 @@ async def run_collector_task(collector_name: str): event="collector.run.skipped_already_running", context={"collector_name": collector_name, "task_id": existing_running.id}, ) + await emit_business_log( + logger, + event="collector.run.skipped_already_running", + message="Skipping collector trigger because task is already running", + category="collector", + level="warning", + service="scheduler", + module=__name__, + context={ + "collector_name": collector_name, + "datasource_id": datasource.id, + "task_id": existing_running.id, + "status": "skipped", + }, + ) return existing_error = (existing_running.error_message or "").strip() @@ -159,7 +185,7 @@ async def run_collector_task(collector_name: str): f"Marked failed automatically after stale running timeout " f"({RUNNING_TASK_GUARD_TIMEOUT_MINUTES}m) in scheduler guard" ) - existing_running.status = "failed" + existing_running.status = JobStatus.FAILED.value existing_running.phase = "failed" existing_running.completed_at = now existing_running.error_message = ( @@ -173,6 +199,21 @@ async def run_collector_task(collector_name: str): event="collector.run.stale_task_failed", context={"collector_name": collector_name, "task_id": existing_running.id}, ) + await emit_business_log( + logger, + event="collector.run.stale_task_failed", + message="Marked stale running task as failed before rerun", + category="collector", + level="warning", + service="scheduler", + module=__name__, + context={ + "collector_name": collector_name, + "datasource_id": datasource.id, + "task_id": existing_running.id, + "status": "failed", + }, + ) try: datasource_id = datasource.id @@ -183,6 +224,15 @@ async def run_collector_task(collector_name: str): event="collector.run.started", context={"collector_name": collector_name, "datasource_id": datasource_id}, ) + await emit_business_log( + logger, + event="collector.run.scheduled_started", + message="Scheduler started collector run", + category="collector", + service="scheduler", + module=__name__, + context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"}, + ) task_result = await collector.run(db) datasource = await db.get(DataSource, datasource_id) if datasource is None: @@ -194,7 +244,7 @@ async def run_collector_task(collector_name: str): return datasource.last_run_at = datetime.now(UTC) datasource.last_status = task_result.get("status") - if datasource.last_status == "success": + if datasource.last_status == JobStatus.SUCCESS.value: effective_candidate = await get_builtin_effective_candidate(db, datasource_source) checksum, _credential_context = await build_builtin_connectivity_checksum( datasource_source, @@ -217,29 +267,66 @@ async def run_collector_task(collector_name: str): event="collector.run.completed", context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result}, ) + await emit_business_log( + logger, + event="collector.run.scheduled_completed", + message="Scheduler completed collector run", + category="collector", + service="scheduler", + module=__name__, + context={ + "collector_name": collector_name, + "datasource_id": datasource_id, + "status": task_result.get("status"), + "result": task_result, + }, + ) except asyncio.CancelledError: await db.rollback() datasource = await db.get(DataSource, datasource_id) datasource.last_run_at = datetime.now(UTC) - datasource.last_status = "cancelled" + datasource.last_status = JobStatus.CANCELLED.value await db.commit() logger.warning_event( "Collector cancelled by operator", event="collector.run.cancelled", context={"collector_name": collector_name, "datasource_id": datasource.id}, ) + await emit_business_log( + logger, + event="collector.run.cancelled", + message="Collector cancelled by operator", + category="collector", + level="warning", + service="scheduler", + module=__name__, + context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"}, + ) raise except Exception as exc: await db.rollback() datasource = await db.get(DataSource, datasource_id) datasource.last_run_at = datetime.now(UTC) - datasource.last_status = "failed" + datasource.last_status = JobStatus.FAILED.value await db.commit() logger.exception_event( "Collector failed", event="collector.run.failed", context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)}, ) + await emit_business_log( + logger, + event="collector.run.failed", + message="Collector failed", + category="collector", + level="error", + service="scheduler", + module=__name__, + context=exception_context( + exc, + {"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"}, + ), + ) async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int: @@ -249,7 +336,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int: async with async_session_factory() as db: result = await db.execute( select(CollectionTask).where( - CollectionTask.status == "running", + CollectionTask.status == JobStatus.RUNNING.value, CollectionTask.started_at.is_not(None), CollectionTask.started_at < cutoff, ) @@ -257,7 +344,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int: stale_tasks = result.scalars().all() for task in stale_tasks: - task.status = "failed" + task.status = JobStatus.FAILED.value task.phase = "failed" task.completed_at = datetime.now(UTC) existing_error = (task.error_message or "").strip() @@ -361,6 +448,16 @@ def run_collector_now(collector_name: str) -> bool: event="collector.trigger.skipped_already_running", context={"collector_name": collector_name}, ) + emit_business_log_background( + logger, + event="collector.trigger.skipped_already_running", + message="Collector is already running in-memory; skipping duplicate trigger", + category="collector", + level="warning", + service="scheduler", + module=__name__, + context={"collector_name": collector_name, "status": "skipped"}, + ) return False try: @@ -378,6 +475,15 @@ def run_collector_now(collector_name: str) -> bool: event="collector.trigger.started", context={"collector_name": collector_name}, ) + emit_business_log_background( + logger, + event="collector.trigger.started", + message="Triggered collector", + category="collector", + service="scheduler", + module=__name__, + context={"collector_name": collector_name, "status": "queued"}, + ) return True except Exception as exc: logger.error_event( @@ -385,6 +491,16 @@ def run_collector_now(collector_name: str) -> bool: event="collector.trigger.failed", context={"collector_name": collector_name, "error": str(exc)}, ) + emit_business_log_background( + logger, + event="collector.trigger.failed", + message="Failed to trigger collector", + category="collector", + level="error", + service="scheduler", + module=__name__, + context=exception_context(exc, {"collector_name": collector_name, "status": "failed"}), + ) return False diff --git a/backend/app/services/situational_alert_ai_brief.py b/backend/app/services/situational_alert_ai_brief.py index 2046e6fc..1ea102b2 100644 --- a/backend/app/services/situational_alert_ai_brief.py +++ b/backend/app/services/situational_alert_ai_brief.py @@ -6,6 +6,7 @@ from typing import Any from sqlalchemy import func, select from sqlalchemy.ext.asyncio import AsyncSession +from app.core.enums import BGPStatus from app.models.alert import Alert, AlertSeverity, AlertStatus from app.models.bgp_anomaly import BGPAnomaly from app.models.bgp_incident import BGPIncident @@ -49,11 +50,11 @@ async def build_situational_alert_brief_request( total_incidents_result = await db.execute(select(func.count(BGPIncident.id))) active_incidents_result = await db.execute( - select(func.count(BGPIncident.id)).where(BGPIncident.status == "active") + select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value) ) bgp_severity_result = await db.execute( select(BGPIncident.severity, func.count(BGPIncident.id)) - .where(BGPIncident.status == "active") + .where(BGPIncident.status == BGPStatus.ACTIVE.value) .group_by(BGPIncident.severity) ) bgp_region_counter: Counter[str] = Counter() @@ -65,11 +66,11 @@ async def build_situational_alert_brief_request( total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id))) active_anomalies_result = await db.execute( - select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active") + select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value) ) anomaly_type_result = await db.execute( select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id)) - .where(BGPAnomaly.status == "active") + .where(BGPAnomaly.status == BGPStatus.ACTIVE.value) .group_by(BGPAnomaly.anomaly_type) .order_by(func.count(BGPAnomaly.id).desc()) .limit(6) diff --git a/backend/app/services/system_control.py b/backend/app/services/system_control.py index 45de6a3e..45dc6428 100644 --- a/backend/app/services/system_control.py +++ b/backend/app/services/system_control.py @@ -7,6 +7,7 @@ from pathlib import Path from typing import Any from app.core.config import ROOT_DIR +from app.core.enums import UserRole from app.core.security import redis_client SYSTEM_TASK_TTL_SECONDS = 24 * 60 * 60 @@ -47,7 +48,7 @@ def normalize_user_role(role: Any) -> str: def require_super_admin(user_role: Any) -> bool: - return normalize_user_role(user_role) == "super_admin" + return normalize_user_role(user_role) == UserRole.SUPER_ADMIN.value def build_task_id(prefix: str = "restart") -> str: diff --git a/backend/app/services/system_logs.py b/backend/app/services/system_logs.py index b1de198a..973c59d9 100644 --- a/backend/app/services/system_logs.py +++ b/backend/app/services/system_logs.py @@ -5,6 +5,7 @@ import os import re import shutil import subprocess +import hashlib from collections import Counter, deque from dataclasses import dataclass @@ -12,7 +13,11 @@ from datetime import UTC, datetime from pathlib import Path from typing import Any +from app.core.enums import LogLevel from app.core.security import redis_client +from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession DEFAULT_LOG_LINE_LIMIT = 200 MAX_LOG_LINE_LIMIT = 1000 @@ -20,11 +25,11 @@ BUFFER_LOG_LIMIT = 1000 BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60 LOG_BUFFER_KEY_PREFIX = "planet:system_logs" -LOG_LEVEL_ERROR = "error" -LOG_LEVEL_WARNING = "warning" -LOG_LEVEL_INFO = "info" -LOG_LEVEL_DEBUG = "debug" -LOG_LEVEL_ALL = "all" +LOG_LEVEL_ERROR = LogLevel.ERROR.value +LOG_LEVEL_WARNING = LogLevel.WARNING.value +LOG_LEVEL_INFO = LogLevel.INFO.value +LOG_LEVEL_DEBUG = LogLevel.DEBUG.value +LOG_LEVEL_ALL = LogLevel.ALL.value SUPPORTED_LOG_LEVELS = { LOG_LEVEL_ALL, @@ -99,6 +104,16 @@ class StructuredLogEntry: search_text: str +@dataclass(frozen=True) +class LogEvent: + source_id: str + cursor: str + timestamp: datetime | None + level: str | None + line: str + search_text: str + + @dataclass class DailyLogMarker: date_token: str @@ -106,6 +121,10 @@ class DailyLogMarker: dominant_level: str +def _normalize_search_query(search: str | None) -> str: + return (search or "").strip().lower() + + def _planet_state_dir() -> Path: configured = os.getenv("PLANET_STATE_DIR") if configured: @@ -135,7 +154,7 @@ LOG_SOURCES: dict[str, LogSource] = { name="前端开发服务", kind="file", location=_state_log_path("frontend.log"), - description="控制台与 Earth 前端开发服务输出。", + description="控制台与智能星球前端开发服务输出。", category="service", fallback_locations=("/tmp/planet_frontend.log",), ), @@ -150,13 +169,22 @@ LOG_SOURCES: dict[str, LogSource] = { ), "earth-client": LogSource( source_id="earth-client", - name="Earth 浏览器端", + name="智能星球浏览器端", kind="buffer", location="redis://planet:system_logs:earth-client", - description="Earth 浏览器端上报的运行时错误与关键业务日志。", + description="智能星球浏览器端上报的运行时错误与关键业务日志。", category="client", buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client", ), + "admin-client": LogSource( + source_id="admin-client", + name="控制台浏览器端", + kind="buffer", + location="redis://planet:system_logs:admin-client", + description="控制台浏览器端上报的运行时错误。", + category="client", + buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:admin-client", + ), } @@ -365,6 +393,59 @@ def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry: ) +def compact_log_context(context: dict | None) -> str: + if not context: + return "" + allowed = { + key: value + for key, value in (context or {}).items() + if key + in { + "status", + "duration_ms", + "provider", + "model", + "result_provider", + "result_model", + "collector_name", + "datasource_id", + "task_id", + "snapshot_id", + "raw_count", + "transformed_count", + "saved_count", + "created", + "updated", + "unchanged", + "deleted", + "result_count", + "status_code", + "error_type", + "error", + "route", + "module", + } + } + if not allowed: + return "" + return json.dumps(allowed, ensure_ascii=False, sort_keys=True) + + +def context_search_aliases(context: dict | None) -> str: + if not context: + return "" + aliases: list[str] = [] + for key, value in sorted((context or {}).items()): + if value is None or isinstance(value, (dict, list, tuple, set)): + continue + normalized_key = str(key).strip() + normalized_value = str(value).strip() + if not normalized_key or not normalized_value: + continue + aliases.append(f"{normalized_key}={normalized_value}") + return " ".join(aliases) + + def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]: path = resolve_file_log_path(source) if not path.exists(): @@ -437,6 +518,400 @@ def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLo return [] +def _database_event_from_system_record(record: SystemLog) -> LogEvent: + record_level = normalize_log_level(record.level) + line = " ".join( + part + for part in [ + record.occurred_at.isoformat() if record.occurred_at else "", + record_level.upper(), + record.source, + record.category or "", + record.event or "", + f"request_id={record.request_id}" if record.request_id else "", + record.message, + compact_log_context(record.context), + ] + if part + ) + search_text = " ".join( + [ + line, + f"id={record.id}", + f"user_id={record.user_id}" if record.user_id else "", + context_search_aliases(record.context), + json.dumps(record.context or {}, ensure_ascii=False, sort_keys=True), + ] + ).lower() + return LogEvent( + source_id="system-db", + cursor=f"system-db:{record.id}", + timestamp=record.occurred_at, + level=None if record_level == LOG_LEVEL_ALL else record_level, + line=line, + search_text=search_text, + ) + + +def _database_event_from_audit_record(record: AuditLog) -> LogEvent: + line = " ".join( + part + for part in [ + record.occurred_at.isoformat() if record.occurred_at else "", + "INFO", + record.action, + record.target_type or "", + record.target_id or "", + record.result or "", + f"request_id={record.request_id}" if record.request_id else "", + ] + if part + ) + search_text = " ".join( + [ + line, + f"id={record.id}", + f"actor_id={record.actor_id}" if record.actor_id else "", + record.actor_name or "", + context_search_aliases(record.details), + json.dumps(record.details or {}, ensure_ascii=False, sort_keys=True), + ] + ).lower() + return LogEvent( + source_id="audit-db", + cursor=f"audit-db:{record.id}", + timestamp=record.occurred_at, + level=LOG_LEVEL_INFO, + line=line, + search_text=search_text, + ) + + +async def read_database_log_events( + source_id: str, + *, + scan_limit: int, + level: str = LOG_LEVEL_ALL, + levels: str | None = None, + start_date: str | None = None, + end_date: str | None = None, + search: str | None = None, + db: AsyncSession, +) -> list[LogEvent] | None: + selected_levels = normalize_log_levels(level, levels) + search_query = (search or "").strip() + if source_id == "system-db": + query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(scan_limit) + result = await db.execute(query) + events = [_database_event_from_system_record(record) for record in result.scalars().all()] + elif source_id == "audit-db": + query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(scan_limit) + result = await db.execute(query) + events = [_database_event_from_audit_record(record) for record in result.scalars().all()] + else: + return None + + events = list(reversed(events)) + return [ + event + for event in events + if event_matches_levels(event, selected_levels) + and event_matches_search(event, search_query) + and event_matches_date_range(event, start_date, end_date) + ] + + +async def read_database_log_snapshot( + source_id: str, + *, + limit: int, + level: str, + levels: str | None, + start_date: str | None, + end_date: str | None, + search: str | None, + db: AsyncSession, +) -> dict[str, Any] | None: + events = await read_database_log_events( + source_id, + scan_limit=limit * 5, + level=level, + levels=levels, + start_date=start_date, + end_date=end_date, + search=search, + db=db, + ) + if events is None: + return None + visible_events = events[-limit:] + selected_levels = normalize_log_levels(level, levels) + return { + "source_id": source_id, + "name": "系统事件" if source_id == "system-db" else "审计事件", + "kind": "database", + "location": "table://system_logs" if source_id == "system-db" else "table://audit_logs", + "description": "数据库持久化日志", + "category": "database" if source_id == "system-db" else "audit", + "status": "ok" if visible_events else "empty", + "level": level, + "selected_levels": list(selected_levels), + "search_query": search or "", + "available_levels": ["all", "error", "warning", "info", "debug"], + "daily_markers": build_daily_log_markers_from_events(events), + "line_limit": limit, + "line_count": len(visible_events), + "lines": [event.line for event in visible_events], + } + + +def _observability_group_matches( + group: ObservabilityEventGroup, + *, + selected_levels: tuple[str, ...], + start_date: str | None, + end_date: str | None, + search: str | None, +) -> bool: + if selected_levels and group.last_level not in selected_levels: + return False + if start_date or end_date: + if group.last_seen_at is None: + return False + date_token = group.last_seen_at.astimezone(UTC).date().isoformat() + if start_date and date_token < start_date: + return False + if end_date and date_token > end_date: + return False + query = _normalize_search_query(search) + if not query: + return True + haystack = " ".join( + [ + group.fingerprint or "", + group.source or "", + group.service or "", + group.module or "", + group.category or "", + group.event or "", + group.last_level or "", + group.sample_message or "", + group.sample_detail or "", + json.dumps(group.affected_sources or [], ensure_ascii=False, sort_keys=True), + ] + ).lower() + return query in haystack + + +def _serialize_observability_group(group: ObservabilityEventGroup) -> dict[str, Any]: + return { + "fingerprint": group.fingerprint, + "source": group.source, + "service": group.service, + "module": group.module, + "category": group.category, + "event": group.event, + "level": group.last_level, + "message": group.sample_message, + "detail": group.sample_detail, + "affected_sources": group.affected_sources or [], + "count": group.count or 0, + "first_seen_at": group.first_seen_at.isoformat() if group.first_seen_at else None, + "last_seen_at": group.last_seen_at.isoformat() if group.last_seen_at else None, + } + + +def _serialize_observability_event(record: ObservabilityEvent) -> dict[str, Any]: + return { + "id": record.id, + "source": record.source, + "service": record.service, + "module": record.module, + "category": record.category, + "event": record.event, + "level": record.level, + "message": record.message, + "fingerprint": record.fingerprint, + "occurred_at": record.occurred_at.isoformat() if record.occurred_at else None, + "request_id": record.request_id, + "trace_id": record.trace_id, + "task_id": record.task_id, + "source_id": record.source_ref_id, + "provider": record.provider, + "user_id": record.user_id, + "context": record.context or {}, + "occurrence_count": record.occurrence_count or 1, + } + + +async def read_observability_groups( + *, + limit: int, + level: str = LOG_LEVEL_ALL, + levels: str | None = None, + start_date: str | None = None, + end_date: str | None = None, + search: str | None = None, + db: AsyncSession, +) -> dict[str, Any]: + selected_levels = normalize_log_levels(level, levels) + scan_limit = max(limit * 5, limit, DEFAULT_LOG_LINE_LIMIT) + result = await db.execute( + select(ObservabilityEventGroup) + .order_by(ObservabilityEventGroup.last_seen_at.desc().nullslast()) + .limit(scan_limit) + ) + groups = [ + group + for group in result.scalars().all() + if _observability_group_matches( + group, + selected_levels=selected_levels, + start_date=start_date, + end_date=end_date, + search=search, + ) + ][:limit] + return { + "mode": "grouped", + "line_limit": limit, + "line_count": len(groups), + "groups": [_serialize_observability_group(group) for group in groups], + "filters": { + "level": level, + "levels": list(selected_levels), + "start_date": start_date, + "end_date": end_date, + "search": search or "", + }, + } + + +async def read_observability_group_events( + fingerprint: str, + *, + limit: int, + db: AsyncSession, +) -> dict[str, Any] | None: + group = await db.get(ObservabilityEventGroup, fingerprint) + if group is None: + return None + result = await db.execute( + select(ObservabilityEvent) + .where(ObservabilityEvent.fingerprint == fingerprint) + .order_by(ObservabilityEvent.occurred_at.desc().nullslast(), ObservabilityEvent.id.desc()) + .limit(limit) + ) + events = list(reversed(result.scalars().all())) + return { + "fingerprint": fingerprint, + "group": _serialize_observability_group(group), + "line_limit": limit, + "line_count": len(events), + "events": [_serialize_observability_event(record) for record in events], + } + + +async def read_observability_raw_events( + *, + limit: int, + level: str = LOG_LEVEL_ALL, + levels: str | None = None, + start_date: str | None = None, + end_date: str | None = None, + search: str | None = None, + db: AsyncSession, +) -> dict[str, Any]: + selected_levels = normalize_log_levels(level, levels) + query = select(ObservabilityEvent).order_by(ObservabilityEvent.occurred_at.desc().nullslast(), ObservabilityEvent.id.desc()) + if selected_levels: + query = query.where(ObservabilityEvent.level.in_(selected_levels)) + result = await db.execute(query.limit(max(limit * 5, limit))) + records = result.scalars().all() + search_query = _normalize_search_query(search) + visible: list[ObservabilityEvent] = [] + for record in records: + if start_date or end_date: + if record.occurred_at is None: + continue + date_token = record.occurred_at.astimezone(UTC).date().isoformat() + if start_date and date_token < start_date: + continue + if end_date and date_token > end_date: + continue + if search_query: + haystack = " ".join( + [ + record.source or "", + record.service or "", + record.module or "", + record.category or "", + record.event or "", + record.message or "", + record.fingerprint or "", + record.request_id or "", + record.trace_id or "", + record.task_id or "", + record.source_ref_id or "", + record.provider or "", + json.dumps(record.context or {}, ensure_ascii=False, sort_keys=True), + ] + ).lower() + if search_query not in haystack: + continue + visible.append(record) + if len(visible) >= limit: + break + visible = list(reversed(visible)) + return { + "mode": "raw", + "line_limit": limit, + "line_count": len(visible), + "events": [_serialize_observability_event(record) for record in visible], + "lines": [ + " ".join( + part + for part in [ + record.occurred_at.isoformat() if record.occurred_at else "", + record.level.upper(), + record.source, + record.category or "", + record.event or "", + f"fingerprint={record.fingerprint}", + record.message, + ] + if part + ) + for record in visible + ], + } + + +def _stable_hash(value: str) -> str: + return hashlib.sha1(value.encode("utf-8", errors="replace")).hexdigest()[:16] + + +def build_log_events(source_id: str, entries: list[StructuredLogEntry]) -> list[LogEvent]: + events: list[LogEvent] = [] + seen: dict[str, int] = {} + for entry in entries: + stable_value = entry.raw_line or entry.display_line + digest = _stable_hash(stable_value) + occurrence = seen.get(digest, 0) + 1 + seen[digest] = occurrence + events.append( + LogEvent( + source_id=source_id, + cursor=f"{source_id}:{digest}:{occurrence}", + timestamp=entry.timestamp, + level=entry.level, + line=entry.display_line, + search_text=entry.search_text, + ) + ) + return events + + def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool: if not selected_levels: return True @@ -469,6 +944,34 @@ def matches_search(entry: StructuredLogEntry, search: str | None) -> bool: return query in entry.search_text +def event_matches_levels(event: LogEvent, selected_levels: tuple[str, ...]) -> bool: + if not selected_levels: + return True + return event.level in selected_levels + + +def event_matches_date_range(event: LogEvent, start_date: str | None, end_date: str | None) -> bool: + if not start_date and not end_date: + return True + if event.timestamp is None: + return False + date_token = event.timestamp.astimezone(UTC).date().isoformat() + if start_date and date_token < start_date: + return False + if end_date and date_token > end_date: + return False + return True + + +def event_matches_search(event: LogEvent, search: str | None) -> bool: + if search is None: + return True + query = search.strip().lower() + if not query: + return True + return query in event.search_text + + def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]: grouped: dict[str, list[StructuredLogEntry]] = {} for entry in entries: @@ -503,6 +1006,69 @@ def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, return [marker.__dict__ for marker in markers] +def build_daily_log_markers_from_events(events: list[LogEvent]) -> list[dict[str, Any]]: + grouped: dict[str, list[LogEvent]] = {} + for event in events: + if event.timestamp is None: + continue + date_token = event.timestamp.astimezone(UTC).date().isoformat() + grouped.setdefault(date_token, []).append(event) + + markers: list[DailyLogMarker] = [] + for date_token, group in sorted(grouped.items()): + level_counts = Counter( + event.level + for event in group + if event.level in SUPPORTED_LOG_LEVELS and event.level != LOG_LEVEL_ALL + ) + dominant_level = LOG_LEVEL_INFO + if level_counts: + dominant_level = sorted( + level_counts.items(), + key=lambda item: ( + -item[1], + ("error", "warning", "info", "debug").index(item[0]), + ), + )[0][0] + markers.append( + DailyLogMarker( + date_token=date_token, + total=len(group), + dominant_level=dominant_level, + ) + ) + return [marker.__dict__ for marker in markers] + + +def read_log_events( + source_id: str, + scan_limit: int, + *, + level: str = LOG_LEVEL_ALL, + levels: str | None = None, + start_date: str | None = None, + end_date: str | None = None, + search: str | None = None, +) -> list[LogEvent] | None: + source = LOG_SOURCES.get(source_id) + if source is None: + return None + + selected_levels = normalize_log_levels(level, levels) + search_query = (search or "").strip() + events = build_log_events(source_id, read_source_entries(source, scan_limit)) + marker_events = [ + event + for event in events + if event_matches_levels(event, selected_levels) and event_matches_search(event, search_query) + ] + return [ + event + for event in marker_events + if event_matches_date_range(event, start_date, end_date) + ] + + def read_log_snapshot( source_id: str, limit: int, @@ -520,18 +1086,18 @@ def read_log_snapshot( selected_levels = normalize_log_levels(level, levels) search_query = (search or "").strip() scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000) - all_entries = read_source_entries(source, scan_limit) - marker_entries = [ - entry - for entry in all_entries - if matches_levels(entry, selected_levels) and matches_search(entry, search_query) + all_events = build_log_events(source_id, read_source_entries(source, scan_limit)) + marker_events = [ + event + for event in all_events + if event_matches_levels(event, selected_levels) and event_matches_search(event, search_query) ] - filtered_entries = [ - entry - for entry in marker_entries - if matches_date_range(entry, start_date, end_date) + filtered_events = [ + event + for event in marker_events + if event_matches_date_range(event, start_date, end_date) ] - visible_entries = filtered_entries[-limit:] + visible_events = filtered_events[-limit:] compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL return { @@ -552,8 +1118,8 @@ def read_log_snapshot( LOG_LEVEL_INFO, LOG_LEVEL_DEBUG, ], - "daily_markers": build_daily_log_markers(marker_entries), + "daily_markers": build_daily_log_markers_from_events(marker_events), "line_limit": limit, - "line_count": len(visible_entries), - "lines": [entry.display_line for entry in visible_entries], + "line_count": len(visible_events), + "lines": [event.line for event in visible_events], } diff --git a/backend/app/services/vessel_ais_aggregation.py b/backend/app/services/vessel_ais_aggregation.py index bb6f2c60..3298fca2 100644 --- a/backend/app/services/vessel_ais_aggregation.py +++ b/backend/app/services/vessel_ais_aggregation.py @@ -9,7 +9,12 @@ from sqlalchemy import select from sqlalchemy import Float from sqlalchemy.ext.asyncio import AsyncSession -from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth +from app.models.vessel import ( + AISConflictRecord, + AISRawObservation, + AISSourceHealth, + VesselCurrentState, +) from app.services.vessel_aggregation_strategy import ( DEFAULT_STRATEGY, load_strategy, @@ -44,6 +49,17 @@ CONFLICT_FIELDS = ( "width", "draught", ) +CURRENT_STATE_STATIC_FIELDS = ( + "name", + "callsign", + "vessel_type", + "vessel_type_name", + "flag", + "length", + "width", + "draught", + "imo", +) def _json_default(value: Any) -> Any: @@ -489,9 +505,115 @@ async def record_vessel_ais_observation( quality_flags=quality_flags or [], ) db.add(observation) + await upsert_vessel_current_state( + db, + source=source, + normalized_payload=normalized_json, + observed_at=observed_at, + quality_flags=quality_flags or [], + ) return observation +async def upsert_vessel_current_state( + db: AsyncSession, + *, + source: str, + normalized_payload: dict[str, Any], + observed_at: datetime, + quality_flags: list[str] | None = None, +) -> VesselCurrentState | None: + """Keep one latest renderable row per MMSI while preserving useful static fields.""" + + if not _has_valid_position(normalized_payload): + return None + mmsi = int(normalized_payload["mmsi"]) + current = await db.get(VesselCurrentState, mmsi) + if current is not None and current.observed_at is not None: + current_observed_at = _coerce_datetime(current.observed_at) + if current_observed_at is not None and observed_at < current_observed_at: + return current + + if current is None: + current = VesselCurrentState(mmsi=mmsi) + db.add(current) + + current.lat = float(normalized_payload["lat"]) + current.lon = float(normalized_payload["lon"]) + current.source = source + current.observed_at = observed_at + current.updated_at = datetime.now(UTC) + updated_fields: set[str] = {"lat", "lon"} + for field in DYNAMIC_FIELDS: + if field in {"lat", "lon"}: + continue + value = _payload_value(normalized_payload, field) + if value is not None: + setattr(current, field, value) + updated_fields.add(field) + field_sources = dict(current.field_sources or {}) + for field in CURRENT_STATE_STATIC_FIELDS: + value = _payload_value(normalized_payload, field) + if value is None: + continue + existing_source = field_sources.get(field) + existing_value = getattr(current, field, None) + if ( + existing_value in (None, "") + or _strategy_source_rank(source, DEFAULT_STRATEGY) + >= _strategy_source_rank(str(existing_source or ""), DEFAULT_STRATEGY) + ): + setattr(current, field, value) + updated_fields.add(field) + + current.vessel_type_name = current.vessel_type_name or normalize_vessel_type_name( + current.vessel_type + ) + selected_reasons = dict(current.selected_reasons or {}) + for field in updated_fields: + field_sources[field] = source + selected_reasons[field] = ( + "newest_observation" if field in DYNAMIC_FIELDS else "source_priority" + ) + current.field_sources = field_sources + current.selected_reasons = selected_reasons + current.source_summary = { + **dict(current.source_summary or {}), + source: { + "latest_observed_at": observed_at.isoformat(), + }, + } + current.quality_flags = sorted(set((current.quality_flags or []) + (quality_flags or []))) + return current + + +async def get_current_vessels_snapshot( + db: AsyncSession, + *, + bbox: tuple[float, float, float, float], + limit: int = 1000, + observed_since: datetime, +) -> list[dict[str, Any]]: + """Read the bounded latest-state table used by Earth rendering.""" + + safe_limit = min(max(int(limit or 1000), 1), MAX_SNAPSHOT_LIMIT) + lon_min, lat_min, lon_max, lat_max = bbox + stmt = ( + select(VesselCurrentState) + .where(VesselCurrentState.observed_at >= observed_since) + .where(VesselCurrentState.lon >= lon_min) + .where(VesselCurrentState.lon <= lon_max) + .where(VesselCurrentState.lat >= lat_min) + .where(VesselCurrentState.lat <= lat_max) + .order_by(VesselCurrentState.observed_at.desc(), VesselCurrentState.mmsi.asc()) + .limit(safe_limit) + ) + result = await db.execute(stmt) + if not hasattr(result, "scalars"): + return [] + return [item.to_dict() for item in result.scalars().all()] + + async def aggregate_vessel_observations( db: AsyncSession, observations: Iterable[AISRawObservation], diff --git a/backend/pytest.ini b/backend/pytest.ini index 450e2ff2..bea70978 100644 --- a/backend/pytest.ini +++ b/backend/pytest.ini @@ -1,4 +1,5 @@ [pytest] +pythonpath = .. asyncio_mode = auto testpaths = tests python_files = test_*.py diff --git a/backend/tests/test_ai_observability.py b/backend/tests/test_ai_observability.py new file mode 100644 index 00000000..73bc0d67 --- /dev/null +++ b/backend/tests/test_ai_observability.py @@ -0,0 +1,128 @@ +import pytest +from fastapi import HTTPException + +from app.schemas.ai import SituationalAnalysisRequest +from app.services.ai_tools import web_search as web_search_module +from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig +from app.services.ai_tools.web_search import WebSearchClient +from app.services import ai_client as ai_client_module +from app.services.ai_client import AIProviderClient + + +@pytest.mark.asyncio +async def test_ai_client_analyze_logs_summary_without_prompt(monkeypatch): + events = [] + + async def fake_emit_business_log(_logger, **payload): + events.append(payload) + + async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None): + return { + "provider": "test-provider", + "model": "test-model", + "content": "ok", + "content_blocks": [], + "text_blocks": ["ok"], + "thinking_blocks": [], + "raw_response": {}, + } + + monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log) + monkeypatch.setattr(AIProviderClient, "_request", fake_request) + + client = AIProviderClient( + service_url="http://provider.test", + llm_config={"provider": "openai", "provider_api": "openai-completions", "model": "gpt-test", "api_key": "sk-secret"}, + ) + result = await client.analyze( + SituationalAnalysisRequest( + title="Sensitive title", + objective="Do not store this full prompt", + observations=["secret observation"], + constraints=["secret constraint"], + context={"source": "test", "private": "value"}, + ), + request_id="req-ai-test", + ) + + assert result.model == "test-model" + assert [event["event"] for event in events] == [ + "ai.provider.analyze.start", + "ai.provider.analyze.success", + ] + serialized = str(events) + assert "Do not store this full prompt" not in serialized + assert "secret observation" not in serialized + assert "sk-secret" not in serialized + start_context = events[0]["context"] + assert start_context["model"] == "gpt-test" + assert start_context["input_summary"]["objective_length"] == len("Do not store this full prompt") + assert start_context["input_summary"]["observation_count"] == 1 + assert start_context["input_summary"]["context_keys"] == ["private", "source"] + + +@pytest.mark.asyncio +async def test_ai_client_analyze_logs_failure(monkeypatch): + events = [] + + async def fake_emit_business_log(_logger, **payload): + events.append(payload) + + async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None): + raise HTTPException(status_code=502, detail="provider failed") + + monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log) + monkeypatch.setattr(AIProviderClient, "_request", fake_request) + + client = AIProviderClient(service_url="http://provider.test", llm_config={"provider": "openai", "model": "gpt-test"}) + + with pytest.raises(HTTPException): + await client.analyze( + SituationalAnalysisRequest(title="T", objective="O", observations=["one"]), + request_id="req-ai-fail", + ) + + assert events[-1]["event"] == "ai.provider.analyze.failed" + assert events[-1]["level"] == "error" + assert events[-1]["context"]["error_type"] == "HTTPException" + + +@pytest.mark.asyncio +async def test_web_search_logs_query_hash_without_query(monkeypatch): + events = [] + + async def fake_emit_business_log(_logger, **payload): + events.append(payload) + + async def fake_search_tavily(self, config, query, max_results, domains, freshness_days): + return [ + SearchEvidence( + title="Example", + url="https://example.test", + snippet="result", + source_provider="tavily", + ) + ] + + monkeypatch.setattr(web_search_module, "emit_business_log", fake_emit_business_log) + monkeypatch.setattr(WebSearchClient, "_search_tavily", fake_search_tavily) + + client = WebSearchClient( + WebSearchConfig( + enabled=True, + default_provider="tavily", + providers={"tavily": WebSearchProviderConfig(provider="tavily", api_key="secret-key")}, + ) + ) + results = await client.search("secret query text", max_results=1) + + assert len(results) == 1 + assert [event["event"] for event in events] == [ + "ai_tool.web_search.start", + "ai_tool.web_search.success", + ] + serialized = str(events) + assert "secret query text" not in serialized + assert "secret-key" not in serialized + assert events[0]["context"]["query_length"] == len("secret query text") + assert events[1]["context"]["result_count"] == 1 diff --git a/backend/tests/test_api.py b/backend/tests/test_api.py index 7f5963ac..8b35e54f 100644 --- a/backend/tests/test_api.py +++ b/backend/tests/test_api.py @@ -655,6 +655,8 @@ async def test_ingest_earth_client_log_accepts_public_events(): "message": "登陆点加载失败: 登陆点接口返回 HTTP 500", "category": "startup-load", "module": "layer-startup", + "fingerprint": "client-test", + "occurrence_count": 3, }, ) assert response.status_code == 200 @@ -668,10 +670,98 @@ async def test_ingest_earth_client_log_accepts_public_events(): assert persisted_kwargs["event"] == "earth.client.runtime_log" assert persisted_kwargs["category"] == "startup-load" assert persisted_kwargs["level"] == "error" + assert persisted_kwargs["fingerprint"] == "client-test" + assert persisted_kwargs["occurrence_count"] == 3 finally: app.dependency_overrides.clear() +@pytest.mark.asyncio +async def test_ingest_admin_client_log_accepts_public_events(): + transport = ASGITransport(app=app) + try: + with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log: + with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log: + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.post( + "/api/v1/system/logs/admin-client", + json={ + "level": "error", + "message": "控制台发生未处理 Promise 错误", + "category": "unhandledrejection", + "module": "admin", + "url": "http://test/logs", + "detail": "stack preview", + }, + ) + assert response.status_code == 200 + data = response.json() + assert data["accepted"] is True + assert data["source_id"] == "admin-client" + mock_append_buffer_log.assert_called_once() + mock_record_system_log.assert_awaited_once() + persisted_kwargs = mock_record_system_log.await_args.kwargs + assert persisted_kwargs["source"] == "admin-client" + assert persisted_kwargs["event"] == "admin.client.runtime_log" + assert persisted_kwargs["category"] == "unhandledrejection" + assert persisted_kwargs["context"]["url"] == "http://test/logs" + finally: + app.dependency_overrides.clear() + + +@pytest.mark.asyncio +async def test_ingest_service_log_requires_configured_token(monkeypatch): + monkeypatch.setattr(settings, "OBSERVABILITY_INGEST_TOKEN", "") + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.post( + "/api/v1/system/logs/service", + json={"message": "AI provider failed"}, + headers={"X-Planet-Observability-Token": "secret"}, + ) + + assert response.status_code == 503 + + +@pytest.mark.asyncio +async def test_ingest_service_log_accepts_internal_token(monkeypatch): + monkeypatch.setattr(settings, "OBSERVABILITY_INGEST_TOKEN", "service-secret") + transport = ASGITransport(app=app) + with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log: + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.post( + "/api/v1/system/logs/service", + json={ + "source": "ai-provider", + "service": "ai-provider", + "module": "provider", + "category": "connectivity", + "event": "ai.provider.test.failed", + "level": "error", + "message": "Provider connectivity failed", + "fingerprint": "ai-provider-test", + "occurrence_count": 4, + "provider": "minimax", + "trace_id": "trace-123", + "context": {"status_code": 502}, + }, + headers={"Authorization": "Bearer service-secret"}, + ) + + assert response.status_code == 200 + data = response.json() + assert data["accepted"] is True + assert data["source_id"] == "ai-provider" + mock_record_system_log.assert_awaited_once() + persisted_kwargs = mock_record_system_log.await_args.kwargs + assert persisted_kwargs["event"] == "ai.provider.test.failed" + assert persisted_kwargs["fingerprint"] == "ai-provider-test" + assert persisted_kwargs["occurrence_count"] == 4 + assert persisted_kwargs["context"]["provider"] == "minimax" + assert persisted_kwargs["context"]["trace_id"] == "trace-123" + assert persisted_kwargs["context"]["status_code"] == 502 + + @pytest.mark.asyncio async def test_earth_layer_cache_status_requires_super_admin(auth_headers, monkeypatch): def override_get_current_user(): diff --git a/backend/tests/test_collectors.py b/backend/tests/test_collectors.py index d747f3a6..b800283a 100644 --- a/backend/tests/test_collectors.py +++ b/backend/tests/test_collectors.py @@ -1,9 +1,13 @@ """Unit tests for data collectors""" +import json + import pytest from unittest.mock import AsyncMock, patch from app.core.datasource_defaults import DEFAULT_DATASOURCES +from app.services.collectors.celestrak import CelesTrakTLECollector +from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES from app.services.collectors.top500 import TOP500Collector from app.services.collectors.registry import collector_registry @@ -149,6 +153,172 @@ class TestHTTPCollector: assert callable(collector.parse_response) +class TestCelesTrakTLECollector: + def test_transform_uses_norad_as_source_id_and_preserves_starlink_group(self): + collector = CelesTrakTLECollector() + result = collector.transform([ + { + "NORAD_CAT_ID": 44720, + "OBJECT_NAME": "STARLINK-1000", + "OBJECT_ID": "2019-029AZ", + "EPOCH": "2026-03-13T00:00:00Z", + "MEAN_MOTION": 15.79234567, + "ECCENTRICITY": 0.0001234, + "INCLINATION": 53.0, + "RA_OF_ASC_NODE": 10.0, + "ARG_OF_PERICENTER": 20.0, + "MEAN_ANOMALY": 30.0, + "_celestrak_query_group": "active", + "_celestrak_source_url": "https://celestrak.example/gp.php?GROUP=active&FORMAT=json", + } + ]) + + assert result[0]["source_id"] == "44720" + assert result[0]["metadata"]["constellation_group"] == "starlink" + assert result[0]["metadata"]["celestrak_query_group"] == "active" + assert result[0]["metadata"]["norad_cat_id"] == 44720 + assert result[0]["metadata"]["tle_line1"] + assert result[0]["metadata"]["tle_line2"] + + def test_load_active_payload_rejects_invalid_records(self, tmp_path): + collector = CelesTrakTLECollector() + payload_path = tmp_path / "active.json" + payload_path.write_text(json.dumps([{"OBJECT_NAME": "missing norad"}]), encoding="utf-8") + + with pytest.raises(RuntimeError, match="invalid record"): + collector._load_active_payload(payload_path) + + def test_load_active_payload_accepts_complete_array(self, tmp_path): + collector = CelesTrakTLECollector() + payload_path = tmp_path / "active.json" + payload_path.write_text( + json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]), + encoding="utf-8", + ) + + records = collector._load_active_payload(payload_path) + + assert records == [{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}] + + @pytest.mark.asyncio + async def test_fetch_retries_and_raises_instead_of_returning_partial_data(self, monkeypatch, tmp_path): + collector = CelesTrakTLECollector() + collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php" + attempts = 0 + + async def fake_download_file(*args, **kwargs): + nonlocal attempts + attempts += 1 + raise RuntimeError("network interrupted") + + async def fake_emit_business_log(*args, **kwargs): + return None + + monkeypatch.setattr(collector._downloader, "download_file", fake_download_file) + monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log) + monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock()) + + with pytest.raises(RuntimeError, match="failed after retries"): + await collector.fetch() + + assert attempts == 3 + + @pytest.mark.asyncio + async def test_fetch_uses_cache_when_celestrak_reports_not_updated(self, monkeypatch, tmp_path): + collector = CelesTrakTLECollector() + collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php" + collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path) + url = collector._active_url() + cached_path = collector._downloader.cached_file_path(url, ".json") + cached_path.parent.mkdir(parents=True, exist_ok=True) + cached_path.write_text( + json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]), + encoding="utf-8", + ) + + async def fake_download_file(*args, **kwargs): + raise DownloadHTTPStatusError( + url=url, + status_code=403, + body="GP data has not updated since your last successful download of GROUP=active.", + ) + + async def fake_emit_business_log(*args, **kwargs): + return None + + monkeypatch.setattr(collector._downloader, "download_file", fake_download_file) + monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log) + + records = await collector.fetch() + + assert records[0]["NORAD_CAT_ID"] == 25544 + assert records[0]["_celestrak_query_group"] == "active" + + @pytest.mark.asyncio + async def test_fetch_not_updated_without_cache_does_not_retry(self, monkeypatch, tmp_path): + collector = CelesTrakTLECollector() + collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php" + collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path) + attempts = 0 + + async def fake_download_file(*args, **kwargs): + nonlocal attempts + attempts += 1 + raise DownloadHTTPStatusError( + url=collector._active_url(), + status_code=403, + body="GP data has not updated since your last successful download of GROUP=active.", + ) + + async def fake_emit_business_log(*args, **kwargs): + return None + + monkeypatch.setattr(collector._downloader, "download_file", fake_download_file) + monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",)) + monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log) + monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock()) + + with pytest.raises(RuntimeError, match="fallback group mode failed"): + await collector.fetch() + + assert attempts == 2 + + @pytest.mark.asyncio + async def test_fetch_falls_back_to_all_groups_when_active_not_updated_without_cache(self, monkeypatch, tmp_path): + collector = CelesTrakTLECollector() + collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php" + collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path) + payload_by_group = { + "starlink": [{"NORAD_CAT_ID": 100, "OBJECT_NAME": "STARLINK-100"}], + "gps-ops": [{"NORAD_CAT_ID": 200, "OBJECT_NAME": "GPS BIIR-2"}], + } + + async def fake_download_file(_client, url, **_kwargs): + if "GROUP=active" in url: + raise DownloadHTTPStatusError( + url=url, + status_code=403, + body="GP data has not updated since your last successful download of GROUP=active.", + ) + group = "starlink" if "GROUP=starlink" in url else "gps-ops" + path = tmp_path / f"{group}.json" + path.write_text(json.dumps(payload_by_group[group]), encoding="utf-8") + return path + + async def fake_emit_business_log(*args, **kwargs): + return None + + monkeypatch.setattr(collector._downloader, "download_file", fake_download_file) + monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", tuple(payload_by_group)) + monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log) + + records = await collector.fetch() + + assert [item["NORAD_CAT_ID"] for item in records] == [100, 200] + assert records[0]["_celestrak_query_group"] == "starlink" + assert records[1]["_celestrak_group"] == "gps-ops" + + def test_aisstream_collector_is_registered(): collector = collector_registry.get("aisstream_vessels") diff --git a/backend/tests/test_datasources_batch.py b/backend/tests/test_datasources_batch.py index fa8b7b0a..0fa30d21 100644 --- a/backend/tests/test_datasources_batch.py +++ b/backend/tests/test_datasources_batch.py @@ -1,9 +1,11 @@ +import asyncio +import inspect from datetime import datetime, timedelta, timezone -import pytest - from app.api.v1 import datasources as datasources_api from app.models.datasource import DataSource +from app.models.task import CollectionTask +from app.services import data_jobs from app.services import earth_layer_cache as earth_cache @@ -100,6 +102,71 @@ def test_serialize_datasource_row_includes_endpoint_when_requested(): assert row["endpoint"] == "https://example.test/arcgis_landing_points" +def test_serialize_datasource_row_separates_collector_running_from_delete_task(): + datasource = make_datasource(9, "top500", last_status="success") + task = CollectionTask( + id=44, + datasource_id=9, + source="top500", + task_type="clear_data", + status="running", + phase="clearing_data", + ) + + row = datasources_api.serialize_datasource_row( + datasource, + running_tasks={9: task}, + latest_tasks={9: task}, + record_counts={"top500": 500}, + endpoint_overrides={}, + config=object(), + include_endpoint=False, + ) + + assert row["is_task_active"] is True + assert row["is_running"] is False + assert row["task_type"] == "clear_data" + assert row["task_status"] == "running" + + +def test_cancel_queued_delete_task_finishes_immediately(monkeypatch): + task = CollectionTask( + id=45, + datasource_id=9, + source="top500", + task_type="clear_data", + status="queued", + phase="queued", + ) + + class FakeDb: + async def commit(self): + return None + + async def refresh(self, _task): + return None + + async def fake_broadcast(_task): + return None + + monkeypatch.setattr(data_jobs, "_broadcast_task_update", fake_broadcast) + + async def run(): + return await data_jobs.request_cancel_datasource_task(FakeDb(), task) + + result = asyncio.run(run()) + + assert result.status == "cancelled" + assert result.phase == "cancelled" + assert result.completed_at is not None + + +def test_clear_data_job_relies_on_db_outbox_instead_of_extra_earth_refresh_task(): + source = inspect.getsource(data_jobs._run_clear_data_job) + + assert "enqueue_earth_refresh_job" not in source + + def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch): patterns: list[str] = [] @@ -119,34 +186,36 @@ def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monke ] -@pytest.mark.asyncio -async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch): - now = datetime.now(timezone.utc) - disabled = make_datasource(1, "aisstream_vessels", is_active=False) - not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120) - due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2)) - triggered_sources: list[str] = [] +def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch): + async def run(): + now = datetime.now(timezone.utc) + disabled = make_datasource(1, "aisstream_vessels", is_active=False) + not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120) + due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2)) + queued_sources: list[str] = [] - async def fake_running_tasks(_db, _ids): - return {} + async def fake_running_tasks(_db, _ids): + return {} - async def fake_latest_task_ids(_db, _ids): - return {} + async def fake_enqueue(_db, datasource, task_type, **_kwargs): + queued_sources.append(datasource.source) + return CollectionTask(id=100 + datasource.id, datasource_id=datasource.id, source=datasource.source, task_type=task_type, status="queued") - monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks) - monkeypatch.setattr(datasources_api, "_load_latest_task_ids", fake_latest_task_ids) - monkeypatch.setattr( - datasources_api, - "run_collector_now", - lambda source: triggered_sources.append(source) or True, - ) + monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks) + monkeypatch.setattr( + datasources_api, + "enqueue_datasource_job", + fake_enqueue, + ) - result = await datasources_api._trigger_datasource_batch( - object(), - [disabled, not_due, due], - force=False, - ) + result = await datasources_api._trigger_datasource_batch( + object(), + [disabled, not_due, due], + force=False, + ) - assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"] - assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"} - assert triggered_sources == ["ris_live_bgp"] + assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"] + assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"} + assert queued_sources == ["ris_live_bgp"] + + asyncio.run(run()) diff --git a/backend/tests/test_docs_gatekeeper.py b/backend/tests/test_docs_gatekeeper.py index 16a0d41c..bf3e7804 100644 --- a/backend/tests/test_docs_gatekeeper.py +++ b/backend/tests/test_docs_gatekeeper.py @@ -1,11 +1,15 @@ """Docs Gatekeeper API tests.""" +import re +from pathlib import Path + import pytest from httpx import ASGITransport, AsyncClient from app.api.v1 import docs as docs_api from app.main import app from app.models.user import User +from app.services.docs_gatekeeper import DOCS_METADATA def make_user(role: str = "viewer", groups: list[str] | None = None) -> User: @@ -43,17 +47,17 @@ async def test_public_catalog_only_for_anonymous_user(): assert response.status_code == 200 items = response.json()["items"] assert {item["access"] for item in items} == {"public"} - assert {item["slug"] for item in items if item["lang"] == "zh"} == { + zh_items = [item for item in items if item["lang"] == "zh"] + assert [item["slug"] for item in zh_items] == [ "overview", - "quickstart", "manual", + "quickstart", "faq", - "location-pipeline-user", - } + ] @pytest.mark.asyncio -async def test_developer_catalog_includes_frontend_reference_docs(): +async def test_developer_catalog_includes_architecture_and_frontend_reference_docs(): response = await get_json( "/api/v1/docs/catalog", make_user(role="viewer", groups=["docs_developer"]), @@ -61,6 +65,10 @@ async def test_developer_catalog_includes_frontend_reference_docs(): assert response.status_code == 200 zh_slugs = {item["slug"] for item in response.json()["items"] if item["lang"] == "zh"} + zh_items = [item for item in response.json()["items"] if item["lang"] == "zh"] + assert [item["group"] for item in zh_items[:4]] == ["Overview", "Manual", "Manual", "Manual"] + assert zh_items[4]["group"] == "Architecture" + assert "platform-data-flows" in zh_slugs assert "naming-glossary" in zh_slugs assert "tactile-ui-components" in zh_slugs @@ -68,10 +76,16 @@ async def test_developer_catalog_includes_frontend_reference_docs(): @pytest.mark.asyncio async def test_anonymous_can_read_public_doc(): response = await get_json("/api/v1/docs/zh/quickstart") + manual_response = await get_json("/api/v1/docs/zh/manual") + overview_response = await get_json("/api/v1/docs/zh/overview") assert response.status_code == 200 assert response.json()["access"] == "public" assert "快速开始" in response.json()["markdown"] + assert manual_response.status_code == 200 + assert manual_response.json()["access"] == "public" + assert overview_response.status_code == 200 + assert overview_response.json()["access"] == "public" @pytest.mark.asyncio @@ -133,3 +147,29 @@ async def test_unknown_language_slug_and_path_traversal_do_not_read_files(): assert bad_lang.status_code == 404 assert bad_slug.status_code == 404 assert traversal.status_code == 404 + + +def test_public_docs_markdown_links_do_not_create_missing_docs_routes(): + repo_root = Path(__file__).resolve().parents[2] + technical_root = repo_root / "docs" / "technical" + registered_filenames = {entry.filename for entry in DOCS_METADATA} + problems: list[str] = [] + + for markdown_path in sorted(technical_root.glob("*/*.md")): + markdown = markdown_path.read_text(encoding="utf-8") + for match in re.finditer(r"\[([^\]]+)]\(([^)]+\.md(?:#[^)]+)?)\)", markdown): + label, href = match.group(1), match.group(2) + if href.startswith(("http://", "https://", "mailto:")): + continue + + href_without_hash = href.split("#", 1)[0].replace("\\", "/") + filename = Path(href_without_hash).name + if "/docs/technical/" in href_without_hash: + if filename not in registered_filenames: + problems.append(f"{markdown_path.relative_to(repo_root)} links unregistered public doc {href!r} ({label})") + continue + + if href_without_hash.endswith(".md"): + problems.append(f"{markdown_path.relative_to(repo_root)} links non-public markdown {href!r} ({label})") + + assert problems == [] diff --git a/backend/tests/test_earth_db_change_listener.py b/backend/tests/test_earth_db_change_listener.py new file mode 100644 index 00000000..1100c90d --- /dev/null +++ b/backend/tests/test_earth_db_change_listener.py @@ -0,0 +1,454 @@ +import asyncio +import json + +from app.services.earth_db_change_listener import ( + EarthDbChangeDispatcher, + EarthDbChangeListener, + build_earth_update_from_db_payload, + normalize_asyncpg_dsn, +) + + +def test_normalize_asyncpg_dsn_strips_sqlalchemy_driver(): + assert ( + normalize_asyncpg_dsn("postgresql+asyncpg://postgres:postgres@localhost:5432/planet_db") + == "postgresql://postgres:postgres@localhost:5432/planet_db" + ) + + +def test_build_earth_update_maps_known_sources_to_layers(): + satellite_update = build_earth_update_from_db_payload( + {"source": "celestrak_tle", "operation": "DELETE", "entity_key": "25544"} + ) + cable_update = build_earth_update_from_db_payload( + {"source": "arcgis_cables", "operation": "UPDATE", "entity_key": "cable-1"} + ) + compute_update = build_earth_update_from_db_payload( + {"source": "top500", "operation": "DELETE", "entity_key": None} + ) + landing_update = build_earth_update_from_db_payload( + {"source": "telegeography_landing", "operation": "DELETE", "entity_key": None} + ) + + assert satellite_update is not None + assert satellite_update["layers"] == ["satellites"] + assert cable_update is not None + assert cable_update["layers"] == ["cables"] + assert compute_update is not None + assert compute_update["layers"] == ["computeCenters"] + assert landing_update is not None + assert landing_update["layers"] == ["cables"] + assert landing_update["refresh_strategy"] == "clear_then_reload" + + +def test_build_earth_update_maps_derived_tables_to_layers(): + bgp_update = build_earth_update_from_db_payload( + {"table": "bgp_anomalies", "source": "ris_live_bgp", "operation": "DELETE", "entity_key": "a1"} + ) + compute_update = build_earth_update_from_db_payload( + {"table": "compute_center_locations", "source": "top500", "operation": "UPDATE", "entity_key": "top500_1"} + ) + vessel_update = build_earth_update_from_db_payload( + {"table": "vessel_position", "operation": "DELETE", "entity_key": "123456789"} + ) + + assert bgp_update is not None + assert bgp_update["source"] == "ris_live_bgp" + assert bgp_update["table"] == "bgp_anomalies" + assert bgp_update["layers"] == ["bgp"] + assert bgp_update["refresh_strategy"] == "clear_then_reload" + assert compute_update is not None + assert compute_update["layers"] == ["computeCenters"] + assert compute_update["refresh_strategy"] == "reload" + assert vessel_update is not None + assert vessel_update["source"] == "vessel_position" + assert vessel_update["layers"] == ["vessels"] + + +def test_build_earth_update_maps_interactable_delete_to_delta(): + update = build_earth_update_from_db_payload( + { + "table": "earth_interactables", + "operation": "DELETE", + "entity_keys": ["note-1"], + "records_processed": 1, + } + ) + + assert update is not None + assert update["entity"] == "interactable" + assert update["action"] == "deleted" + assert update["ids"] == ["note-1"] + assert update["layers"] == ["interactables"] + assert update["refresh_strategy"] == "delta" + + +def test_build_earth_update_ignores_unmapped_sources(): + assert build_earth_update_from_db_payload({"source": "not_for_earth"}) is None + + +def test_dispatcher_debounces_same_source_notifications(): + async def run(): + broadcasts = [] + invalidated = [] + + async def broadcast(payload): + broadcasts.append(payload) + + def invalidate(source): + invalidated.append(source) + return 2 + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=invalidate, + debounce_seconds=10, + ) + dispatcher.handle_payload( + { + "table": "collected_data", + "operation": "INSERT", + "source": "arcgis_cables", + "entity_key": "cable-1", + "occurred_at": "2026-05-22T00:00:00Z", + } + ) + dispatcher.handle_payload( + { + "table": "collected_data", + "operation": "UPDATE", + "source": "arcgis_cables", + "entity_key": "cable-2", + "occurred_at": "2026-05-22T00:00:01Z", + } + ) + + await dispatcher.flush_all() + + assert invalidated == ["arcgis_cables"] + assert len(broadcasts) == 1 + assert broadcasts[0]["action"] == "database_changed" + assert broadcasts[0]["source"] == "arcgis_cables" + assert broadcasts[0]["layers"] == ["cables"] + assert broadcasts[0]["records_processed"] == 2 + assert broadcasts[0]["operations"] == ["INSERT", "UPDATE"] + assert broadcasts[0]["entity_keys"] == ["cable-1", "cable-2"] + assert broadcasts[0]["cache_entries_invalidated"] == 2 + + asyncio.run(run()) + + +def test_dispatcher_handles_delete_notification_as_earth_update(): + async def run(): + broadcasts = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 1, + debounce_seconds=10, + ) + accepted = dispatcher.handle_notification( + json.dumps( + { + "table": "collected_data", + "operation": "DELETE", + "source": "celestrak_tle", + "entity_key": "sat:25544", + "occurred_at": "2026-05-22T00:00:00Z", + } + ) + ) + + await dispatcher.flush_all() + + assert accepted is True + assert len(broadcasts) == 1 + assert broadcasts[0]["action"] == "database_changed" + assert broadcasts[0]["source"] == "celestrak_tle" + assert broadcasts[0]["layers"] == ["satellites"] + assert broadcasts[0]["operation"] == "DELETE" + assert broadcasts[0]["refresh_strategy"] == "clear_then_reload" + + asyncio.run(run()) + + +def test_dispatcher_debounces_bgp_derived_table_events_by_source(): + async def run(): + broadcasts = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + ) + dispatcher.handle_payload( + { + "event_id": 200, + "table": "bgp_observations", + "operation": "DELETE", + "source": "ris_live_bgp", + "records_processed": 2, + } + ) + dispatcher.handle_payload( + { + "event_id": 201, + "table": "bgp_anomalies", + "operation": "DELETE", + "source": "ris_live_bgp", + "records_processed": 3, + } + ) + + await dispatcher.flush_all() + + assert len(broadcasts) == 1 + assert broadcasts[0]["source"] == "ris_live_bgp" + assert broadcasts[0]["layers"] == ["bgp"] + assert broadcasts[0]["records_processed"] == 5 + assert broadcasts[0]["refresh_strategy"] == "clear_then_reload" + + asyncio.run(run()) + + +def test_dispatcher_deduplicates_notify_and_outbox_by_event_id(): + async def run(): + broadcasts = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + ) + payload = { + "event_id": 42, + "table": "collected_data", + "operation": "INSERT", + "source": "celestrak_tle", + "entity_key": "sat:42", + "occurred_at": "2026-05-23T00:00:00Z", + } + + assert dispatcher.handle_payload(payload) is True + assert dispatcher.handle_payload(dict(payload)) is False + await dispatcher.flush_all() + + assert len(broadcasts) == 1 + assert broadcasts[0]["records_processed"] == 1 + assert broadcasts[0]["entity_keys"] == ["sat:42"] + + asyncio.run(run()) + + +def test_dispatcher_flushes_continuous_events_at_max_wait(): + async def run(): + broadcasts = [] + sleeps = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + max_wait_seconds=0.01, + ) + dispatcher.handle_payload( + { + "event_id": 300, + "table": "collected_data", + "operation": "INSERT", + "source": "arcgis_cables", + "records_processed": 1, + } + ) + await asyncio.sleep(0.02) + dispatcher.handle_payload( + { + "event_id": 301, + "table": "collected_data", + "operation": "UPDATE", + "source": "arcgis_cables", + "records_processed": 1, + } + ) + await asyncio.sleep(0) + await dispatcher.flush_all() + + assert len(broadcasts) == 1 + assert broadcasts[0]["records_processed"] == 2 + assert broadcasts[0]["debounce_ms"] >= 0 + + asyncio.run(run()) + + +def test_dispatcher_fast_flushes_delete_clear_then_reload(): + async def run(): + broadcasts = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + max_wait_seconds=10, + ) + dispatcher.handle_payload( + { + "event_id": 310, + "table": "collected_data", + "operation": "DELETE", + "source": "celestrak_tle", + "records_processed": 10, + } + ) + await asyncio.sleep(0.08) + + assert len(broadcasts) == 1 + assert broadcasts[0]["source"] == "celestrak_tle" + assert broadcasts[0]["refresh_strategy"] == "clear_then_reload" + + asyncio.run(run()) + + +def test_outbox_rows_are_consumed_after_successful_flush(): + async def run(): + class FakeConnection: + def __init__(self): + self.consumed_ids = [] + + async def fetch(self, _query, _limit): + return [ + { + "id": 501, + "payload": { + "event_id": 501, + "table": "collected_data", + "operation": "DELETE", + "source": "celestrak_tle", + "records_processed": 1, + }, + } + ] + + async def execute(self, _query, consumed_ids): + self.consumed_ids.extend(consumed_ids) + + broadcasts = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + ) + listener = EarthDbChangeListener( + dsn="postgresql://example/db", + dispatcher=dispatcher, + ) + connection = FakeConnection() + listener._connection = connection + + await listener._poll_outbox() + + assert connection.consumed_ids == [501] + assert len(broadcasts) == 1 + + asyncio.run(run()) + + +def test_outbox_rows_remain_unconsumed_when_flush_fails(): + async def run(): + class FakeConnection: + def __init__(self): + self.consumed_ids = [] + + async def fetch(self, _query, _limit): + return [ + { + "id": 601, + "payload": { + "event_id": 601, + "table": "collected_data", + "operation": "DELETE", + "source": "celestrak_tle", + "records_processed": 1, + }, + } + ] + + async def execute(self, _query, consumed_ids): + self.consumed_ids.extend(consumed_ids) + + async def broadcast(_payload): + raise RuntimeError("ws down") + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + ) + listener = EarthDbChangeListener( + dsn="postgresql://example/db", + dispatcher=dispatcher, + ) + connection = FakeConnection() + listener._connection = connection + + try: + await listener._poll_outbox() + except RuntimeError: + pass + + assert connection.consumed_ids == [] + + asyncio.run(run()) + + +def test_dispatcher_uses_aggregated_statement_record_count(): + async def run(): + broadcasts = [] + + async def broadcast(payload): + broadcasts.append(payload) + + dispatcher = EarthDbChangeDispatcher( + broadcast_earth_update=broadcast, + invalidate_cache=lambda source: 0, + debounce_seconds=10, + ) + dispatcher.handle_payload( + { + "event_id": 99, + "table": "collected_data", + "operation": "DELETE", + "source": "top500", + "records_processed": 100, + "entity_keys": ["top500:1", "top500:2"], + "occurred_at": "2026-05-23T00:00:00Z", + } + ) + + await dispatcher.flush_all() + + assert len(broadcasts) == 1 + assert broadcasts[0]["source"] == "top500" + assert broadcasts[0]["layers"] == ["computeCenters"] + assert broadcasts[0]["records_processed"] == 100 + assert broadcasts[0]["entity_keys"] == ["top500:1", "top500:2"] + + asyncio.run(run()) diff --git a/backend/tests/test_earth_interactables.py b/backend/tests/test_earth_interactables.py new file mode 100644 index 00000000..bf464cec --- /dev/null +++ b/backend/tests/test_earth_interactables.py @@ -0,0 +1,79 @@ +from datetime import UTC, datetime + +from app.models.earth_interactable import EarthInteractable +from app.services import earth_interactables + + +def make_interactable(**overrides): + values = { + "id": "poi-1", + "layer": "places", + "kind": "note", + "label": "Test POI", + "description": "A point on Earth", + "latitude": 30.25, + "longitude": 120.15, + "altitude": None, + "revision": 3, + "properties": {"owner": "test"}, + "is_deleted": False, + "created_at": datetime(2026, 5, 22, 1, 0, tzinfo=UTC), + "updated_at": datetime(2026, 5, 22, 1, 5, tzinfo=UTC), + "deleted_at": None, + } + values.update(overrides) + return EarthInteractable(**values) + + +def test_interactable_event_uses_object_delta_contract(): + record = make_interactable() + + event = earth_interactables.build_interactable_event( + action="updated", + record=record, + ) + + assert event["entity"] == "interactable" + assert event["action"] == "updated" + assert event["layer"] == "places" + assert event["layers"] == ["interactables"] + assert event["ids"] == ["poi-1"] + assert event["revision"] == 3 + assert event["item"]["latitude"] == 30.25 + assert event["item"]["properties"] == {"owner": "test"} + + +def test_interactable_geojson_omits_deleted_records(): + active = make_interactable(id="active") + deleted = make_interactable(id="deleted", is_deleted=True) + + payload = earth_interactables.interactables_to_geojson([active, deleted]) + + assert payload["type"] == "FeatureCollection" + assert [feature["id"] for feature in payload["features"]] == ["active"] + assert payload["features"][0]["geometry"] == { + "type": "Point", + "coordinates": [120.15, 30.25], + } + + +def test_interactable_cache_invalidation_clears_layer_and_all(monkeypatch): + patterns = [] + + def fake_delete_pattern(pattern): + patterns.append(pattern) + return 1 + + monkeypatch.setattr( + earth_interactables.earth_layer_cache, + "delete_pattern", + fake_delete_pattern, + ) + + deleted = earth_interactables.invalidate_interactable_cache("places") + + assert deleted == 2 + assert patterns == [ + "earth:layer:v1:interactables:interactable_layer:places*", + "earth:layer:v1:interactables:interactable_layer:all*", + ] diff --git a/backend/tests/test_earth_news.py b/backend/tests/test_earth_news.py index e989c5f9..8bbfc868 100644 --- a/backend/tests/test_earth_news.py +++ b/backend/tests/test_earth_news.py @@ -1,21 +1,30 @@ -from datetime import UTC, datetime +from datetime import UTC, datetime, timedelta from types import SimpleNamespace import pytest from app.services.earth_news import ( + NewsFeedEndpoint, NewsFeedSource, NewsTargetLocation, ParsedNewsItem, + apply_news_classification, + default_earth_news_sources_payload, + normalize_earth_news_sources_payload, + _fetch_source, + _diversify_news_items_for_locale, _enrich_items_with_target_locations, _extract_target_location_from_text, _parse_feed_entries, + _rank_and_trim_items, _serialize_item, get_earth_news_payload, + test_news_source_config as run_news_source_config_test, ) from app.services.earth_news_queue import NewsTargetLocationMessage from app.services.earth_news_worker import process_target_location_message from app.services.collectors.media_news_archive import MediaNewsArchiveCollector +from app.services.earth_news_store import _diversify_parsed_news_items_by_region def test_serialize_item_includes_region_anchor_for_cruise(): @@ -44,6 +53,188 @@ def test_serialize_item_includes_region_anchor_for_cruise(): assert payload["published_at"] == "2026-04-23T02:30:00Z" +def test_serialize_item_includes_breaking_fields(): + item = ParsedNewsItem( + id="breaking:test", + title="Major market halt", + summary="Trading halt after flash crash", + url="https://example.com/breaking", + source="Example Source", + feed_name="Example Feed", + feed_region="global", + homepage_url="https://example.com", + published_at=datetime(2026, 5, 15, 2, 0, tzinfo=UTC), + breaking_level="critical", + breaking_scope="global", + breaking_reasons=["重大金融市场异常"], + breaking_source="rules", + breaking_confidence=0.72, + breaking_expires_at=datetime(2026, 5, 16, 2, 0, tzinfo=UTC), + ) + + payload = _serialize_item(item, active_region="europe") + + assert payload["breaking_level"] == "critical" + assert payload["breaking_scope"] == "global" + assert payload["breaking_reasons"] == ["重大金融市场异常"] + assert payload["breaking_source"] == "rules" + assert payload["breaking_confidence"] == 0.72 + assert payload["breaking_expires_at"] == "2026-05-16T02:00:00Z" + + +def test_rank_and_trim_items_prioritizes_active_breaking(): + older_breaking = ParsedNewsItem( + id="global:critical", + title="Nuclear accident reported", + summary="A nuclear accident has been reported.", + url="https://example.com/critical", + source="Global Source", + feed_name="Global Feed", + feed_region="global", + homepage_url="https://example.com", + published_at=datetime.now(UTC) - timedelta(hours=2), + breaking_level="critical", + breaking_scope="global", + breaking_expires_at=datetime.now(UTC) + timedelta(hours=6), + ) + newer_regular = ParsedNewsItem( + id="europe:regular", + title="Regular Europe story", + summary="A newer regular story.", + url="https://example.com/regular", + source="Europe Source", + feed_name="Europe Feed", + feed_region="europe", + homepage_url="https://example.com", + published_at=datetime.now(UTC), + ) + expired_breaking = ParsedNewsItem( + id="europe:expired", + title="Expired breaking", + summary="Expired breaking story.", + url="https://example.com/expired", + source="Europe Source", + feed_name="Europe Feed", + feed_region="europe", + homepage_url="https://example.com", + published_at=datetime.now(UTC) + timedelta(minutes=1), + breaking_level="critical", + breaking_scope="regional", + breaking_expires_at=datetime.now(UTC) - timedelta(minutes=1), + ) + + ranked = _rank_and_trim_items( + [newer_regular, expired_breaking, older_breaking], + active_region="europe", + limit=3, + ) + + assert [item.id for item in ranked] == ["global:critical", "europe:expired", "europe:regular"] + + +def test_diversify_news_items_prefers_display_ready_content_across_sources(): + published_at = datetime(2026, 6, 11, 3, 0, tzinfo=UTC) + + def make_item(source_id: str, suffix: str, *, zh_ready: bool) -> ParsedNewsItem: + return ParsedNewsItem( + id=f"{source_id}:{suffix}", + title=f"{source_id} title {suffix}", + summary=f"{source_id} summary {suffix}", + url=f"https://example.com/{source_id}/{suffix}", + source=source_id, + feed_name=source_id, + feed_region="global", + homepage_url="https://example.com", + published_at=published_at, + content_language="en", + localizations={ + "zh-CN": { + "title": f"{source_id} 中文标题 {suffix}", + "summary": f"{source_id} 中文摘要 {suffix}", + } + } if zh_ready else {}, + ) + + items = [ + make_item("source-a", "1", zh_ready=False), + make_item("source-a", "2", zh_ready=False), + make_item("source-a", "3", zh_ready=False), + make_item("source-b", "1", zh_ready=True), + make_item("source-c", "1", zh_ready=True), + ] + + result = _diversify_news_items_for_locale( + items, + active_region="global", + limit=3, + locale="zh-CN", + ) + + assert [item.id.split(":", 1)[0] for item in result] == ["source-b", "source-c", "source-a"] + + +def test_cruise_news_diversity_keeps_regions_from_being_starved(): + published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC) + + def make_item(region: str, index: int) -> ParsedNewsItem: + return ParsedNewsItem( + id=f"{region}:{index}", + title=f"{region} story {index}", + summary=f"{region} summary {index}", + url=f"https://example.com/{region}/{index}", + source=region, + feed_name=region, + feed_region=region, + homepage_url="https://example.com", + published_at=published_at - timedelta(minutes=index), + ) + + items = [ + *[make_item("asia-pacific", index) for index in range(40)], + make_item("europe", 1), + make_item("middle-east-africa", 1), + make_item("americas", 1), + make_item("global", 1), + ] + + result = _diversify_parsed_news_items_by_region(items, limit=8) + regions = [item.feed_region for item in result] + + assert "europe" in regions + assert "middle-east-africa" in regions + assert "americas" in regions + assert regions.count("asia-pacific") < len(regions) + + +def test_global_news_diversity_uses_same_region_balance(): + published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC) + + def make_item(region: str, index: int) -> ParsedNewsItem: + return ParsedNewsItem( + id=f"{region}:global:{index}", + title=f"{region} story {index}", + summary=f"{region} summary {index}", + url=f"https://example.com/{region}/global/{index}", + source=region, + feed_name=region, + feed_region=region, + homepage_url="https://example.com", + published_at=published_at - timedelta(minutes=index), + ) + + items = [ + *[make_item("asia-pacific", index) for index in range(24)], + *[make_item("europe", index) for index in range(2)], + *[make_item("middle-east-africa", index) for index in range(2)], + *[make_item("americas", index) for index in range(2)], + ] + + result = _diversify_parsed_news_items_by_region(items, limit=6) + regions = {item.feed_region for item in result} + + assert {"europe", "middle-east-africa", "americas"}.issubset(regions) + + def test_serialize_item_falls_back_to_global_anchor(): item = ParsedNewsItem( id="custom:test", @@ -164,6 +355,486 @@ def test_parse_aggregated_rss_splits_publisher_from_title(): assert items[0].source == "Reuters" +def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization(): + source = NewsFeedSource( + id="36kr", + name="36氪", + region="asia-pacific", + feed_url="https://36kr.com/feed", + homepage_url="https://www.36kr.com/", + source_tags=("china", "business_news"), + default_category="business", + ) + xml = """ + + + + 中国电商平台发布季度增长数据 + 平台表示，跨境电商订单量同比增长。 + https://36kr.com/p/example + + + + """ + + items = _parse_feed_entries(xml, source) + payload_zh = _serialize_item(items[0], active_region="global", locale="zh-CN") + payload_en = _serialize_item(items[0], active_region="global", locale="en-US") + + assert items[0].content_language == "zh-CN" + assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据" + assert payload_zh["display_title"] == "中国电商平台发布季度增长数据" + assert payload_en["display_title"] == "" + + items[0].localizations["en-US"] = { + "title": "Chinese e-commerce platform reports quarterly growth", + "summary": "The platform said cross-border orders rose year over year.", + } + payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US") + assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth" + + +def test_default_news_sources_include_business_and_ecommerce_sources(): + payload = default_earth_news_sources_payload() + sources_by_id = {source["id"]: source for source in payload["sources"]} + source_ids = {source["id"] for source in payload["sources"]} + category_keys = {category["key"] for category in payload["categories"]} + tag_keys = {tag["key"] for tag in payload["source_tags"]} + + assert "cnbc-business" in source_ids + assert "36kr" in source_ids + assert "techcrunch" in source_ids + assert "retaildive" in source_ids + assert "prnewswire-retail" in source_ids + assert "google-news" in source_ids + assert "global-scan" not in source_ids + assert "google-americas" not in source_ids + assert "google-europe" not in source_ids + assert "google-mea" not in source_ids + assert "google-apac" not in source_ids + assert "businesswire-ecommerce" in source_ids + assert "us-census-ecommerce" in source_ids + assert "mofcom-data" in source_ids + assert "stats-china-online-retail" in source_ids + assert "ebrun" in source_ids + assert sources_by_id["36kr"]["source_type"] == "rss" + assert sources_by_id["36kr"]["homepage_url"] == "https://www.36kr.com/" + assert sources_by_id["36kr"]["feed_directory_url"] == "https://www.36kr.com/rss-center" + kr_feeds = {feed["id"]: feed for feed in sources_by_id["36kr"]["feeds"]} + assert set(kr_feeds) == {"feed", "article", "newsflash", "moment"} + assert kr_feeds["feed"]["url"] == "https://36kr.com/feed" + assert kr_feeds["article"]["url"] == "https://36kr.com/feed-article" + assert kr_feeds["newsflash"]["url"] == "https://36kr.com/feed-newsflash" + assert kr_feeds["moment"]["url"] == "https://36kr.com/feed-moment" + assert all(feed["enabled"] is True for feed in kr_feeds.values()) + assert all(feed["default_category"] == "business" for feed in kr_feeds.values()) + assert "https://36kr.com/feed-article" in sources_by_id["36kr"]["feed_urls"] + assert "https://36kr.com/feed-newsflash" in sources_by_id["36kr"]["feed_urls"] + assert "https://36kr.com/feed-moment" in sources_by_id["36kr"]["feed_urls"] + assert sources_by_id["ebrun"]["source_type"] == "rss" + assert sources_by_id["ebrun"]["homepage_url"] == "https://www.ebrun.com/" + assert sources_by_id["ebrun"]["feed_directory_url"] == "https://www.ebrun.com/rss/" + ebrun_feeds = {feed["id"]: feed for feed in sources_by_id["ebrun"]["feeds"]} + assert {"b2c", "b2b", "retail", "o2o", "service", "data", "policy"}.issubset(ebrun_feeds) + assert all(feed["enabled"] is True for feed in ebrun_feeds.values()) + assert all(feed["default_category"] == "ecommerce" for feed in ebrun_feeds.values()) + assert "https://www.ebrun.com/rss/news_b2c.xml" in sources_by_id["ebrun"]["feed_urls"] + assert "https://www.ebrun.com/rss/news_retail.xml" in sources_by_id["ebrun"]["feed_urls"] + assert sources_by_id["businesswire-ecommerce"]["source_type"] == "reference" + assert sources_by_id["businesswire-ecommerce"]["enabled"] is False + assert sources_by_id["google-news"]["source_type"] == "aggregated" + assert sources_by_id["google-news"]["homepage_url"] == "https://news.google.com/" + assert sources_by_id["google-news"]["feed_directory_url"] == "https://news.google.com/rss" + google_feeds = {feed["id"]: feed for feed in sources_by_id["google-news"]["feeds"]} + assert set(google_feeds) == {"world", "americas", "europe", "middle-east-africa", "asia-pacific"} + assert all(feed["type"] == "aggregated" for feed in google_feeds.values()) + assert all(feed["enabled"] is True for feed in google_feeds.values()) + assert google_feeds["world"]["region"] == "global" + assert google_feeds["europe"]["region"] == "europe" + assert sources_by_id["stats-china-online-retail"]["source_type"] == "rss" + assert sources_by_id["stats-china-online-retail"]["enabled"] is True + assert "https://www.stats.gov.cn/sj/zxfb/rss.xml" in sources_by_id["stats-china-online-retail"]["feed_urls"] + assert {"business", "ecommerce", "finance"}.issubset(category_keys) + assert {"official_data", "business_news", "ecommerce", "press_release", "finance", "logistics"}.issubset(tag_keys) + + +def test_default_enabled_fetchable_sources_have_explicit_types_and_urls(): + payload = default_earth_news_sources_payload() + for source in payload["sources"]: + source_type = source["source_type"] + assert source_type in {"rss", "atom", "aggregated", "reference"} + if source_type == "reference": + assert source["enabled"] is False + assert source["feeds"] == [] + continue + if source["enabled"]: + assert source["feed_url"] + assert source["feed_urls"] + assert source["feeds"] + assert any(feed["enabled"] for feed in source["feeds"]) + for feed in source["feeds"]: + assert feed["url"] != source["homepage_url"] + assert feed["url"] != source.get("feed_directory_url", "") + + +def test_legacy_news_source_urls_migrate_to_feed_children(): + payload = normalize_earth_news_sources_payload( + { + "sources": [ + { + "id": "legacy-source", + "name": "Legacy Source", + "region": "global", + "source_type": "rss", + "feed_urls": ["https://example.com/a.xml", "https://example.com/b.xml"], + "default_category": "business", + } + ] + } + ) + + source = payload["sources"][0] + + assert source["feed_urls"] == ["https://example.com/a.xml", "https://example.com/b.xml"] + assert [feed["url"] for feed in source["feeds"]] == ["https://example.com/a.xml", "https://example.com/b.xml"] + assert [feed["id"] for feed in source["feeds"]] == ["feed-1", "feed-2"] + assert all(feed["default_category"] == "business" for feed in source["feeds"]) + + +def test_builtin_news_source_legacy_directory_url_is_repaired(): + payload = normalize_earth_news_sources_payload( + { + "sources": [ + { + "id": "36kr", + "name": "36氪", + "region": "asia-pacific", + "source_type": "rss", + "homepage_url": "https://www.36kr.com/", + "feed_url": "https://www.36kr.com/rss-center", + "feed_urls": ["https://www.36kr.com/rss-center"], + "feeds": [ + { + "id": "feed-1", + "name": "36氪", + "url": "https://www.36kr.com/rss-center", + "type": "rss", + "enabled": True, + "default_category": "business", + } + ], + "default_category": "business", + } + ] + } + ) + + source = payload["sources"][0] + feed_urls = {feed["url"] for feed in source["feeds"]} + + assert source["homepage_url"] == "https://www.36kr.com/" + assert source["feed_directory_url"] == "https://www.36kr.com/rss-center" + assert "https://www.36kr.com/rss-center" not in feed_urls + assert { + "https://36kr.com/feed", + "https://36kr.com/feed-article", + "https://36kr.com/feed-newsflash", + "https://36kr.com/feed-moment", + }.issubset(feed_urls) + + +def test_builtin_news_source_without_feed_children_gets_explicit_defaults(): + payload = normalize_earth_news_sources_payload( + { + "sources": [ + { + "id": "ebrun", + "name": "亿邦动力", + "region": "asia-pacific", + "source_type": "rss", + "homepage_url": "https://www.ebrun.com/", + "feed_url": "https://www.ebrun.com/rss/news_b2c.xml", + "feed_urls": ["https://www.ebrun.com/rss/news_b2c.xml"], + "default_category": "ecommerce", + } + ] + } + ) + + source = payload["sources"][0] + feed_urls = {feed["url"] for feed in source["feeds"]} + + assert source["feed_directory_url"] == "https://www.ebrun.com/rss/" + assert "https://www.ebrun.com/rss/" not in feed_urls + assert { + "https://www.ebrun.com/rss/news_b2c.xml", + "https://www.ebrun.com/rss/news_b2b.xml", + "https://www.ebrun.com/rss/news_retail.xml", + "https://www.ebrun.com/rss/news_o2o.xml", + "https://www.ebrun.com/rss/news_service.xml", + "https://www.ebrun.com/rss/news_data.xml", + "https://www.ebrun.com/rss/news_policy.xml", + }.issubset(feed_urls) + + +def test_builtin_fetchable_source_saved_as_reference_is_repaired(): + payload = normalize_earth_news_sources_payload( + { + "sources": [ + { + "id": "stats-china-online-retail", + "name": "国家统计局数据发布", + "region": "asia-pacific", + "source_type": "reference", + "enabled": False, + "homepage_url": "https://www.stats.gov.cn/sj/zxfb/", + "feed_url": "https://www.stats.gov.cn/sj/zxfb/", + "default_category": "ecommerce", + } + ] + } + ) + + source = payload["sources"][0] + + assert source["source_type"] == "rss" + assert source["enabled"] is True + assert source["priority"] == 19 + assert source["source_tags"] == ["official_data", "ecommerce", "retail", "china"] + assert source["default_category"] == "ecommerce" + assert source["importance_weight"] == 36 + assert source["feed_directory_url"] == "" + assert source["feeds"] == [ + { + "id": "release", + "name": "数据发布", + "url": "https://www.stats.gov.cn/sj/zxfb/rss.xml", + "type": "rss", + "region": "asia-pacific", + "enabled": True, + "default_category": "ecommerce", + "tags": [], + "priority": 1, + } + ] + + +def test_legacy_google_sources_merge_into_google_news_source(): + payload = normalize_earth_news_sources_payload( + { + "sources": [ + { + "id": "global-scan", + "name": "Global Monitor / World", + "region": "global", + "source_type": "aggregated", + "feed_url": "https://news.google.com/rss/search?q=world", + "homepage_url": "https://news.google.com/", + }, + { + "id": "google-europe", + "name": "Global Monitor / Europe", + "region": "europe", + "source_type": "aggregated", + "feed_url": "https://news.google.com/rss/search?q=europe", + "homepage_url": "https://news.google.com/", + }, + ] + } + ) + + sources_by_id = {source["id"]: source for source in payload["sources"]} + + assert "global-scan" not in sources_by_id + assert "google-europe" not in sources_by_id + assert "google-news" in sources_by_id + assert {feed["id"] for feed in sources_by_id["google-news"]["feeds"]} == { + "world", + "americas", + "europe", + "middle-east-africa", + "asia-pacific", + } + + +def test_feed_child_default_category_overrides_source_default(): + source = NewsFeedSource( + id="multi-feed", + name="Multi Feed", + region="global", + feed_url="https://example.com/source.xml", + homepage_url="https://example.com", + default_category="business", + ) + feed = NewsFeedEndpoint( + id="ecommerce-feed", + name="Ecommerce Feed", + url="https://example.com/ecommerce.xml", + default_category="ecommerce", + ) + xml = """ + + + + Quarterly results released + Company update. + https://example.com/results + + + + """ + + items = _parse_feed_entries(xml, source, feed=feed) + + assert items[0].feed_id == "ecommerce-feed" + assert items[0].feed_name == "Ecommerce Feed" + assert items[0].feed_default_category == "ecommerce" + assert items[0].category == "ecommerce" + + +@pytest.mark.asyncio +async def test_fetch_source_only_requests_enabled_feed_children(monkeypatch): + source = NewsFeedSource( + id="multi-feed", + name="Multi Feed", + region="global", + feed_url="https://example.com/source.xml", + homepage_url="https://example.com", + feeds=( + NewsFeedEndpoint(id="enabled", name="Enabled", url="https://example.com/enabled.xml", enabled=True), + NewsFeedEndpoint(id="disabled", name="Disabled", url="https://example.com/disabled.xml", enabled=False), + ), + ) + calls = [] + + async def fake_fetch_single(_client, feed_source, feed, *, config_payload=None): + calls.append(feed.id) + item = ParsedNewsItem( + id=f"{feed_source.id}:{feed.id}:1", + title="Fetched story", + summary="Fetched summary", + url=f"https://example.com/{feed.id}", + source="Example", + feed_name=feed.name, + feed_region="global", + homepage_url="https://example.com", + published_at=None, + feed_id=feed.id, + ) + return feed_source, [item], None, {"source_id": feed_source.id, "feed_id": feed.id, "ok": True, "status": "ok", "item_count": 1, "count": 1} + + monkeypatch.setattr("app.services.earth_news._fetch_single_feed_url", fake_fetch_single) + + source_result, items, error, health = await _fetch_source(object(), source) + + assert source_result.id == "multi-feed" + assert calls == ["enabled"] + assert error is None + assert [item.feed_id for item in items] == ["enabled"] + assert health["ok"] is True + assert [result["feed_id"] for result in health["feed_results"]] == ["enabled"] + + +@pytest.mark.asyncio +async def test_fetch_source_filters_google_feed_children_by_active_region(monkeypatch): + source = NewsFeedSource( + id="google-news", + name="Google News", + region="global", + feed_url="https://news.google.com/rss", + homepage_url="https://news.google.com/", + source_type="aggregated", + feeds=( + NewsFeedEndpoint(id="world", name="全球", url="https://example.com/world.xml", type="aggregated", region="global"), + NewsFeedEndpoint(id="europe", name="欧洲", url="https://example.com/europe.xml", type="aggregated", region="europe"), + NewsFeedEndpoint(id="americas", name="美洲", url="https://example.com/americas.xml", type="aggregated", region="americas"), + ), + ) + calls = [] + + async def fake_fetch_single(_client, feed_source, feed, *, config_payload=None): + calls.append(feed.id) + item = ParsedNewsItem( + id=f"{feed_source.id}:{feed.id}:1", + title=f"{feed.name} headline", + summary="Fetched summary", + url=f"https://example.com/{feed.id}", + source="Example", + feed_name=feed.name, + feed_region=feed.region, + homepage_url="https://example.com", + published_at=None, + feed_id=feed.id, + ) + return feed_source, [item], None, {"source_id": feed_source.id, "feed_id": feed.id, "ok": True, "status": "ok", "item_count": 1, "count": 1} + + monkeypatch.setattr("app.services.earth_news._fetch_single_feed_url", fake_fetch_single) + + _source_result, items, error, health = await _fetch_source(object(), source, active_region="europe") + + assert error is None + assert calls == ["world", "europe"] + assert [item.feed_region for item in items] == ["global", "europe"] + assert [result["feed_id"] for result in health["feed_results"]] == ["world", "europe"] + + +def test_parse_rdf_rss_items_with_namespaces(): + source = NewsFeedSource( + id="dw-top", + name="DW Top Stories", + region="europe", + feed_url="https://rss.dw.com/rdf/rss-en-top", + homepage_url="https://www.dw.com/en/top-stories/s-9097", + ) + xml = """ + + + German retail sales rise + https://example.com/dw + Retail summary + + + """ + + items = _parse_feed_entries(xml, source) + + assert len(items) == 1 + assert items[0].title == "German retail sales rise" + + +def test_news_classification_marks_ecommerce_and_importance(): + source = NewsFeedSource( + id="ebrun", + name="亿邦动力", + region="asia-pacific", + feed_url="https://www.ebrun.com/rss/", + homepage_url="https://www.ebrun.com/", + source_tags=("business_news", "ecommerce", "china"), + default_category="ecommerce", + importance_weight=14, + ) + item = ParsedNewsItem( + id="ebrun:test", + title="跨境电商平台 GMV 同比增长，物流履约效率提升", + summary="订单量和网上零售额继续增长。", + url="https://example.com/ecommerce", + source="亿邦动力", + feed_name="亿邦动力", + feed_region="asia-pacific", + homepage_url="https://www.ebrun.com/", + published_at=None, + ) + + apply_news_classification(item, source) + + assert item.category == "ecommerce" + assert "cross_border_ecommerce" in item.item_tags + assert "logistics_fulfillment" in item.item_tags + assert item.importance_level in {"high", "critical"} + assert "命中电商数据指标" in item.importance_reasons + + @pytest.mark.asyncio async def test_enrich_items_with_target_locations_uses_ai_and_geocode(monkeypatch): item = ParsedNewsItem( @@ -338,8 +1009,8 @@ async def test_earth_news_payload_returns_anchor_items_and_enqueues_location_job published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC), ) - async def fake_fetch_source(_client, feed_source): - return feed_source, [item], None + async def fake_fetch_source(_client, feed_source, **_kwargs): + return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1} async def fake_get_cached_target_location_patch(_item_id): return None @@ -402,9 +1073,11 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa async def fake_get_earth_news_freshness(_db, *, active_region): return 12, datetime.now(UTC) - async def fake_list_earth_news_items(_db, *, active_region, limit): - assert limit == 12 - return [item] + async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None): + if source_ids is None: + assert limit == 12 + return [item] + return [] async def fail_fetch(_sources): raise AssertionError("fresh database items should not fetch RSS") @@ -421,6 +1094,148 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa assert payload["stale"] is False +@pytest.mark.asyncio +async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monkeypatch): + class FakeDb: + execute = object() + + current_item = ParsedNewsItem( + id="db:current", + title="Current region story", + summary="Current summary", + url="https://example.com/current", + source="Stored Source", + feed_name="Stored Feed", + feed_region="americas", + homepage_url="https://example.com", + published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC), + ) + cruise_item = ParsedNewsItem( + id="db:apac", + title="APAC story", + summary="APAC summary", + url="https://example.com/apac", + source="Stored Source", + feed_name="Stored Feed", + feed_region="asia-pacific", + homepage_url="https://example.com", + published_at=datetime(2026, 5, 15, 4, 0, tzinfo=UTC), + ) + + async def fake_get_earth_news_freshness(_db, *, active_region): + return 12, datetime.now(UTC) + + async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None): + return [current_item] + + async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None): + return [current_item, cruise_item] + + async def fake_enqueue_target_location_job(_payload, **_kwargs): + return True + + async def fail_fetch(_sources): + raise AssertionError("fresh database items should not fetch RSS") + + monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness) + monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items) + monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items) + monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job) + monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch) + + payload = await get_earth_news_payload(lat=35.0, lon=-100.0, db=FakeDb()) + + assert [item["id"] for item in payload["items"]] == ["db:current"] + assert [item["id"] for item in payload["cruise_items"]] == ["db:current", "db:apac"] + assert payload["cruise_items"][1]["region"] == "asia-pacific" + + +@pytest.mark.asyncio +async def test_earth_news_payload_passes_region_and_category_filters_to_store(monkeypatch): + class FakeDb: + execute = object() + + captured = {} + item = ParsedNewsItem( + id="db:business", + title="Business story", + summary="Business summary", + url="https://example.com/business", + source="Stored Source", + feed_name="Stored Feed", + feed_region="europe", + homepage_url="https://example.com", + published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC), + category="business", + ) + + async def fake_get_earth_news_freshness(_db, *, active_region): + captured["freshness_region"] = active_region + return 12, datetime.now(UTC) + + async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None): + captured["items_region"] = active_region + captured["items_categories"] = categories + captured.setdefault("items_source_ids", []).append(source_ids) + return [item] if source_ids is None else [] + + async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None, source_ids=None): + captured["cruise_categories"] = categories + captured["cruise_source_ids"] = source_ids + return [item] + + async def fake_enqueue_target_location_job(_payload, **_kwargs): + return True + + monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness) + monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items) + monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items) + monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job) + monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", lambda _sources: (_ for _ in ()).throw(AssertionError("fresh database items should not fetch RSS"))) + + payload = await get_earth_news_payload( + lat=35.0, + lon=-100.0, + region="europe", + categories={"business", "ecommerce"}, + db=FakeDb(), + ) + + assert captured["freshness_region"] == "europe" + assert captured["items_region"] == "europe" + assert captured["items_categories"] == {"business", "ecommerce"} + assert captured["items_source_ids"][0] is None + assert any(source_ids for source_ids in captured["items_source_ids"][1:]) + assert captured["cruise_categories"] == {"business", "ecommerce"} + assert captured["cruise_source_ids"] is None + assert payload["filters"] == { + "region": "europe", + "categories": ["business", "ecommerce"], + "sources": [], + "limit": 12, + "locale": "zh-CN", + "has_breaking": False, + "highest_breaking_level": "none", + } + assert payload["items"][0]["category"] == "business" + + +@pytest.mark.asyncio +async def test_news_source_test_treats_type_reference_as_non_fetching(): + result = await run_news_source_config_test( + { + "id": "reference-only", + "name": "Reference Only", + "type": "reference", + "feed_url": "https://example.com", + } + ) + + assert result["ok"] is False + assert result["health"]["status"] == "reference" + assert "不参与 RSS/Atom 抓取" in result["error"] + + @pytest.mark.asyncio async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch): db = object() @@ -456,14 +1271,14 @@ async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatc async def fake_get_earth_news_freshness(_db, *, active_region): return 0, None - async def fake_fetch_rss_items_for_sources(_sources): - return [item], [] + async def fake_fetch_rss_items_for_sources(_sources, **_kwargs): + return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}} async def fake_upsert_earth_news_items(_db, items): upserted.extend(items) return len(items) - async def fake_list_earth_news_items(_db, *, active_region, limit): + async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None): return [item] async def fake_enqueue_target_location_job(payload, **_kwargs): @@ -512,14 +1327,14 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch): async def fake_get_earth_news_freshness(_db, *, active_region): return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC) - async def fake_fetch_rss_items_for_sources(_sources): + async def fake_fetch_rss_items_for_sources(_sources, **_kwargs): fetched.append(True) - return [old_item], [] + return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}} async def fake_upsert_earth_news_items(_db, items): return len(items) - async def fake_list_earth_news_items(_db, *, active_region, limit): + async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None): return [old_item] async def fake_enqueue_target_location_job(_payload, **_kwargs): @@ -574,8 +1389,8 @@ async def test_earth_news_payload_merges_cached_location_patch(monkeypatch): }, } - async def fake_fetch_source(_client, feed_source): - return feed_source, [item], None + async def fake_fetch_source(_client, feed_source, **_kwargs): + return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1} async def fake_get_cached_target_location_patch(_item_id): return cached_patch @@ -641,8 +1456,8 @@ async def test_earth_news_payload_requeues_cached_failed_localization(monkeypatc } enqueued = [] - async def fake_fetch_source(_client, feed_source): - return feed_source, [item], None + async def fake_fetch_source(_client, feed_source, **_kwargs): + return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1} async def fake_get_cached_target_location_patch(_item_id): return cached_patch diff --git a/backend/tests/test_earth_news_manual.py b/backend/tests/test_earth_news_manual.py new file mode 100644 index 00000000..3cd65e8f --- /dev/null +++ b/backend/tests/test_earth_news_manual.py @@ -0,0 +1,252 @@ +from datetime import UTC, datetime + +import pytest + +from app.core.enums import NewsSourceType +from app.models.earth_news import EarthNewsItem +from app.models.system_setting import SystemSetting +from app.services.earth_news import REGION_ANCHORS +from app.services.earth_news_manual import ( + DEFAULT_MANUAL_NEWS_GROUP_ID, + create_manual_news_group, + import_manual_news_items, + list_news_groups, + list_news_records, + parse_manual_news_import_upload, + rename_manual_news_group, + upsert_manual_news_item, +) + + +class _FakeResult: + def __init__(self, rows=None, scalar=None): + self.rows = rows or [] + self._scalar = scalar + + def scalar_one_or_none(self): + return self._scalar + + def scalar(self): + return self._scalar + + def scalars(self): + return self + + def all(self): + return self.rows + + +class _FakeNewsSession: + def __init__(self, records=None, setting=None): + self.records = dict(records or {}) + self.setting = setting + + async def get(self, _model, item_id): + return self.records.get(item_id) + + def add(self, item): + if isinstance(item, SystemSetting): + self.setting = item + else: + self.records[item.id] = item + + async def execute(self, stmt): + statement = str(stmt) + if "system_settings" in statement: + return _FakeResult(scalar=self.setting) + if "count" in statement.lower(): + return _FakeResult(scalar=len(self.records)) + return _FakeResult(rows=list(self.records.values())) + + async def flush(self): + return None + + +@pytest.fixture +def fake_news_queue(monkeypatch): + queued = [] + + async def _enqueue(payload, force=False): + queued.append({"payload": payload, "force": force}) + return True + + monkeypatch.setattr("app.services.earth_news_manual.enqueue_target_location_job", _enqueue) + return queued + + +@pytest.mark.asyncio +async def test_manual_news_upsert_uses_region_anchor_and_manual_metadata(fake_news_queue): + db = _FakeNewsSession() + + result = await upsert_manual_news_item( + db, + { + "title": "手动添加的新闻", + "summary": "一条用于测试的手动新闻。", + "source": "人工录入", + "region": "europe", + "published_at": "2026-05-15T03:00:00Z", + "tags": ["manual", "test"], + }, + ) + + anchor = REGION_ANCHORS["europe"] + assert result.created is True + assert result.queued is True + assert result.item.id.startswith("manual:") + assert result.item.feed_name == "手动添加" + assert result.item.source == "人工录入" + assert result.item.latitude == anchor.latitude + assert result.item.longitude == anchor.longitude + assert result.item.location_source == "region_anchor" + assert result.item.verified is False + assert result.item.location_meta["news_meta"]["feed_type"] == NewsSourceType.MANUAL.value + assert result.item.location_meta["news_meta"]["source_type"] == NewsSourceType.MANUAL.value + assert result.item.location_meta["news_meta"]["manual_group_id"] == DEFAULT_MANUAL_NEWS_GROUP_ID + assert len(fake_news_queue) == 1 + + +@pytest.mark.asyncio +async def test_manual_news_duplicate_import_upserts_without_duplicate_rows(fake_news_queue): + db = _FakeNewsSession() + payload = { + "title": "Same manual story", + "source": "Manual Desk", + "published_at": "2026-05-15T03:00:00Z", + "region": "global", + } + + first = await upsert_manual_news_item(db, payload) + second = await upsert_manual_news_item(db, {**payload, "summary": "Updated summary"}) + + assert first.created is True + assert second.created is False + assert len(db.records) == 1 + assert db.records[first.item.id].summary == "Updated summary" + + +@pytest.mark.asyncio +async def test_manual_news_edit_without_location_preserves_manual_coordinates(fake_news_queue): + db = _FakeNewsSession() + created = await upsert_manual_news_item( + db, + { + "title": "Taipei-1 data center update", + "summary": "Initial summary.", + "region": "asia-pacific", + "published_at": "2026-05-15T03:00:00Z", + "location": {"label": "Kaohsiung, Taiwan", "latitude": 22.6273, "longitude": 120.3014}, + }, + ) + + updated = await upsert_manual_news_item( + db, + { + "title": "Taipei-1 data center update", + "summary": "Edited summary only.", + "region": "asia-pacific", + "published_at": "2026-05-15T03:00:00Z", + }, + item_id_override=created.item.id, + ) + + assert updated.created is False + assert updated.item.latitude == pytest.approx(22.6273) + assert updated.item.longitude == pytest.approx(120.3014) + assert updated.item.location_source == "manual_location" + assert updated.item.verified is True + + +@pytest.mark.asyncio +async def test_manual_news_api_service_rejects_rss_records(fake_news_queue): + rss_record = EarthNewsItem( + id="bbc-world:example", + title="RSS story", + summary="RSS summary", + source="BBC World", + feed_name="BBC World", + region="global", + latitude=20, + longitude=0, + location_label="全球", + location_source="region_anchor", + verified=False, + location_meta={"news_meta": {"feed_type": "rss"}}, + first_seen_at=datetime.now(UTC), + last_seen_at=datetime.now(UTC), + ) + db = _FakeNewsSession({rss_record.id: rss_record}) + + with pytest.raises(PermissionError): + await upsert_manual_news_item( + db, + {"title": "Edited title", "region": "global"}, + item_id_override=rss_record.id, + ) + + +@pytest.mark.asyncio +async def test_manual_news_import_reports_per_item_errors(fake_news_queue): + db = _FakeNewsSession() + + result = await import_manual_news_items( + db, + [ + {"title": "Valid manual news", "region": "global"}, + {"summary": "missing title"}, + ], + ) + + assert result["created"] == 1 + assert result["failed"] == 1 + assert result["errors"][0]["index"] == 1 + + +@pytest.mark.asyncio +async def test_manual_news_groups_default_create_and_rename(fake_news_queue): + db = _FakeNewsSession() + + initial = await list_news_groups(db) + assert initial["manual_groups"][0]["id"] == DEFAULT_MANUAL_NEWS_GROUP_ID + assert initial["manual_groups"][0]["name"] == "新建新闻组" + + group = await create_manual_news_group(db, "专题组") + assert group["name"] == "专题组" + assert db.setting is not None + + await upsert_manual_news_item(db, {"title": "Grouped story", "region": "global"}, group_id=group["id"]) + renamed = await rename_manual_news_group(db, group["id"], "重命名专题") + + record = next(iter(db.records.values())) + assert renamed["name"] == "重命名专题" + assert record.location_meta["news_meta"]["manual_group_id"] == group["id"] + assert record.location_meta["news_meta"]["manual_group_name"] == "重命名专题" + + +@pytest.mark.asyncio +async def test_manual_news_list_filters_by_group_id(fake_news_queue): + db = _FakeNewsSession() + group = await create_manual_news_group(db, "导入组") + + await import_manual_news_items( + db, + [ + {"title": "In group", "region": "global"}, + {"title": "Also in group", "region": "global"}, + ], + group_id=group["id"], + ) + await upsert_manual_news_item(db, {"title": "Default group", "region": "global"}) + + grouped = await list_news_records(db, page=1, page_size=20, group_id=group["id"]) + default_group = await list_news_records(db, page=1, page_size=20, group_id=DEFAULT_MANUAL_NEWS_GROUP_ID) + + assert grouped["total"] == 2 + assert {item["manual_group_id"] for item in grouped["items"]} == {group["id"]} + assert default_group["total"] == 1 + + +@pytest.mark.asyncio +async def test_manual_news_import_parser_requires_json_array(): + with pytest.raises(ValueError, match="顶层必须是数组"): + await parse_manual_news_import_upload(b'{"title":"not an array"}') diff --git a/backend/tests/test_enum_contracts.py b/backend/tests/test_enum_contracts.py new file mode 100644 index 00000000..10156731 --- /dev/null +++ b/backend/tests/test_enum_contracts.py @@ -0,0 +1,74 @@ +"""Compatibility contracts for stable backend protocol enums.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from types import SimpleNamespace + +from app.core.enums import ( + BreakingLevel, + BreakingScope, + JobStatus, + NewsImportanceLevel, + NewsSourceType, + PlaygroundMessageKind, + PlaygroundMessageStatus, + UserRole, + parse_enum, +) +from app.services.earth_news_classification import ( + BREAKING_LEVEL_RANK, + BREAKING_TTL, + breaking_sort_rank, + importance_level, +) + + +def test_protocol_enum_values_remain_api_compatible() -> None: + assert [item.value for item in NewsImportanceLevel] == ["low", "medium", "high", "critical"] + assert [item.value for item in BreakingLevel] == ["none", "watch", "breaking", "critical"] + assert [item.value for item in BreakingScope] == ["regional", "global"] + assert [item.value for item in NewsSourceType] == ["rss", "atom", "aggregated", "reference", "manual"] + assert [item.value for item in UserRole] == ["viewer", "admin", "super_admin"] + assert JobStatus.RUNNING.value == "running" + assert PlaygroundMessageKind.THINKING.value == "thinking" + assert PlaygroundMessageStatus.ERROR.value == "error" + assert PlaygroundMessageStatus.STOPPED.value == "stopped" + + +def test_parse_enum_accepts_legacy_strings_and_safely_falls_back(caplog) -> None: + assert parse_enum(JobStatus, "RUNNING", JobStatus.FAILED) is JobStatus.RUNNING + assert parse_enum(JobStatus, None, JobStatus.QUEUED) is JobStatus.QUEUED + assert parse_enum(JobStatus, "legacy-unknown", JobStatus.FAILED) is JobStatus.FAILED + assert "legacy-unknown" in caplog.text + + +def test_importance_level_boundaries() -> None: + expected = { + 34: NewsImportanceLevel.LOW, + 35: NewsImportanceLevel.MEDIUM, + 59: NewsImportanceLevel.MEDIUM, + 60: NewsImportanceLevel.HIGH, + 79: NewsImportanceLevel.HIGH, + 80: NewsImportanceLevel.CRITICAL, + } + assert {score: importance_level(score) for score in expected} == expected + + +def test_breaking_rank_and_ttl_contracts() -> None: + assert BREAKING_LEVEL_RANK[BreakingLevel.CRITICAL] > BREAKING_LEVEL_RANK[BreakingLevel.BREAKING] + assert BREAKING_TTL[BreakingLevel.WATCH] == timedelta(hours=6) + assert BREAKING_TTL[BreakingLevel.BREAKING] == timedelta(hours=12) + assert BREAKING_TTL[BreakingLevel.CRITICAL] == timedelta(hours=24) + + now = datetime.now(UTC) + active = SimpleNamespace( + breaking_level=BreakingLevel.BREAKING.value, + breaking_expires_at=now + timedelta(minutes=1), + ) + expired = SimpleNamespace( + breaking_level=BreakingLevel.CRITICAL.value, + breaking_expires_at=now - timedelta(minutes=1), + ) + assert breaking_sort_rank(active) == BREAKING_LEVEL_RANK[BreakingLevel.BREAKING] + assert breaking_sort_rank(expired) == 0 diff --git a/backend/tests/test_logging.py b/backend/tests/test_logging.py index 53f790ff..9573f4e8 100644 --- a/backend/tests/test_logging.py +++ b/backend/tests/test_logging.py @@ -4,8 +4,13 @@ import logging from io import StringIO +import pytest + from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger from app.core.request_context import set_request_id +from app.services import business_logs +from app.services import persistent_logs +from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup def _capture_output(callback): @@ -76,3 +81,123 @@ def test_structured_logger_redacts_sensitive_text_and_context(): assert "hunter2" not in output assert "[REDACTED]" in output assert '"safe": "visible"' in output + + +def test_business_context_redacts_nested_sensitive_values(): + context = business_logs.build_business_context( + { + "provider": "openai", + "api_key": "sk-secret", + "nested": { + "token": "plain-token", + "safe": "visible", + }, + } + ) + + assert context["api_key"] == "[REDACTED]" + assert context["nested"]["token"] == "[REDACTED]" + assert context["nested"]["safe"] == "visible" + + +def test_observability_fingerprint_normalizes_hls_fragments(): + first = persistent_logs.build_observability_fingerprint( + source="earth-client", + service="earth", + module="tv", + category="hls-proxy", + event="hls.fragment.failed", + message="HLS 分片加载失败: index_5_9086220.ts?m=1725933270", + context={"status_code": 502}, + ) + second = persistent_logs.build_observability_fingerprint( + source="earth-client", + service="earth", + module="tv", + category="hls-proxy", + event="hls.fragment.failed", + message="HLS 分片加载失败: index_5_9086361.ts?m=1725934270", + context={"status_code": 502}, + ) + + assert first == second + + +@pytest.mark.asyncio +async def test_record_observability_event_updates_group_count(monkeypatch): + events: list[ObservabilityEvent] = [] + groups: dict[str, ObservabilityEventGroup] = {} + + class FakeSession: + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + def add(self, item): + if isinstance(item, ObservabilityEvent): + events.append(item) + elif isinstance(item, ObservabilityEventGroup): + groups[item.fingerprint] = item + + async def get(self, model, key): + if model is ObservabilityEventGroup: + return groups.get(key) + return None + + async def commit(self): + return None + + monkeypatch.setattr(persistent_logs, "async_session_factory", lambda: FakeSession()) + + await persistent_logs.record_observability_event( + source="earth-client", + level="error", + service="earth", + module="tv", + category="hls-proxy", + event="hls.fragment.failed", + message="HLS 分片加载失败: index_5_9086220.ts?m=1725933270", + context={"status_code": 502}, + occurrence_count=2, + ) + await persistent_logs.record_observability_event( + source="earth-client", + level="error", + service="earth", + module="tv", + category="hls-proxy", + event="hls.fragment.failed", + message="HLS 分片加载失败: index_5_9086361.ts?m=1725934270", + context={"status_code": 502}, + occurrence_count=1, + ) + + assert len(events) == 2 + assert len(groups) == 1 + group = next(iter(groups.values())) + assert group.count == 3 + + +@pytest.mark.asyncio +async def test_emit_business_log_persists_sanitized_system_event(monkeypatch): + events = [] + + async def fake_record_system_log(**payload): + events.append(payload) + + monkeypatch.setattr(business_logs, "record_system_log", fake_record_system_log) + + await business_logs.emit_business_log( + get_logger("tests.business"), + event="ai.provider.analyze.success", + message="AI request completed", + category="ai", + context={"model": "gpt-test", "api_key": "sk-secret"}, + ) + + assert events[0]["event"] == "ai.provider.analyze.success" + assert events[0]["category"] == "ai" + assert events[0]["context"]["model"] == "gpt-test" + assert events[0]["context"]["api_key"] == "[REDACTED]" diff --git a/backend/tests/test_motion_agent.py b/backend/tests/test_motion_agent.py index 99791d0a..223995cb 100644 --- a/backend/tests/test_motion_agent.py +++ b/backend/tests/test_motion_agent.py @@ -40,6 +40,9 @@ def test_gesture_event_serializes_stable_protocol_fields(): assert payload["seq"] == 7 assert payload["source"] == "motion-agent" assert payload["mode"] == "single" + assert payload["protocol_version"] == "motion.v2" + assert payload["input_mode"] == "single" + assert payload["camera_id"] == "unknown" assert payload["payload"] == {} @@ -88,8 +91,13 @@ def test_motion_server_status_includes_dry_run_camera_and_heartbeat(): assert status["camera_count"] == 1 assert status["active_camera_ids"] == ["dry-run:null-camera"] assert status["recognizer"] == "dry-run" + assert status["protocol_version"] == "motion.v2" + assert status["armed"] is False + assert status["paused"] is False + assert status["devices_open"] is False assert heartbeat == { "timestamp_ms": 123, + "protocol_version": "motion.v2", "source": "motion-agent", "type": "heartbeat", } @@ -109,6 +117,7 @@ def test_skeleton_event_serializes_without_raw_image_fields(): payload = json.loads(event.to_json()) assert payload["type"] == "skeleton" + assert payload["protocol_version"] == "motion.v2" assert payload["matched_gesture"] == "rotate_left" assert payload["confidence"] == 0.91 assert payload["camera_id"] == "usb:0" @@ -120,6 +129,25 @@ def test_skeleton_event_serializes_without_raw_image_fields(): assert "frame" not in payload +def test_v2_gesture_set_accepts_frontend_motion_gestures(): + state = GestureStateMachine(confidence_threshold=0.7, cooldown_ms=0) + + for gesture in [ + "rotate_up", + "rotate_down", + "focus_prev", + "focus_next", + "layer_prev", + "layer_next", + ]: + event = state.accept( + GestureObservation(gesture, confidence=0.9, intensity=0.8, timestamp_ms=1000) + ) + + assert event is not None + assert event.gesture == gesture + + def test_dry_run_recognizer_produces_debug_skeleton(): server = MotionAgentServer(MotionAgentConfig(dry_run=True)) @@ -240,3 +268,92 @@ async def test_motion_agent_cli_reports_dependency_error_without_traceback(monke assert exit_code == 2 assert "Motion agent failed: missing cv stack" in captured.err assert "Traceback" not in captured.err + + +@pytest.mark.asyncio +async def test_motion_agent_command_updates_control_state(): + server = MotionAgentServer(MotionAgentConfig(dry_run=True)) + + armed = await server.handle_command( + json.dumps( + { + "type": "command", + "command": "set_armed", + "request_id": "req-armed", + "payload": {"armed": True}, + } + ) + ) + paused = await server.handle_command( + { + "type": "command", + "command": "set_paused", + "request_id": "req-paused", + "payload": {"paused": True}, + } + ) + + assert armed.ok is True + assert armed.request_id == "req-armed" + assert armed.status["armed"] is True + assert paused.ok is True + assert paused.status["paused"] is True + + +@pytest.mark.asyncio +async def test_motion_agent_open_devices_command_accepts_dual_mode(): + server = MotionAgentServer(MotionAgentConfig(dry_run=True)) + + try: + result = await server.handle_command( + { + "type": "command", + "command": "open_devices", + "request_id": "req-open", + "payload": {"input_mode": "dual_redundant"}, + } + ) + + assert result.ok is True + assert result.status["input_mode"] == "dual_redundant" + assert result.status["active_camera_ids"] == ("dry-run:null-camera",) + assert server._recognition_subprocess is not None + assert server._recognition_subprocess.returncode is None + finally: + await server.stop_recognition_subprocess() + + +def test_motion_agent_dual_fusion_merges_matching_observations(): + server = MotionAgentServer(MotionAgentConfig(dry_run=True)) + server.state.mode = "dual_redundant" + + selected, fusion = server._fuse_observations( + [ + GestureObservation("zoom_in", confidence=0.82, intensity=0.4, camera_id="usb:0"), + GestureObservation("zoom_in", confidence=0.86, intensity=0.8, camera_id="usb:1"), + ] + ) + + assert selected.gesture == "zoom_in" + assert selected.camera_id == "fusion" + assert selected.confidence > 0.86 + assert fusion == { + "source_cameras": ["usb:1", "usb:0"], + "window_ms": 120, + "reason": "matched_observations", + } + + +def test_motion_agent_dual_fusion_suppresses_close_conflict(): + server = MotionAgentServer(MotionAgentConfig(dry_run=True, confidence_threshold=0.7)) + + selected, fusion = server._fuse_observations( + [ + GestureObservation("zoom_in", confidence=0.82, intensity=0.5, camera_id="usb:0"), + GestureObservation("zoom_out", confidence=0.78, intensity=0.5, camera_id="usb:1"), + ] + ) + + assert selected.gesture == "zoom_in" + assert selected.confidence == 0 + assert fusion["reason"] == "conflict_ignored" diff --git a/backend/tests/test_system_logs.py b/backend/tests/test_system_logs.py index f2b8d127..31fb9cb6 100644 --- a/backend/tests/test_system_logs.py +++ b/backend/tests/test_system_logs.py @@ -2,8 +2,10 @@ from __future__ import annotations import json +from datetime import UTC, datetime from pathlib import Path +from app.models.system_log import SystemLog from app.services import system_logs @@ -40,10 +42,10 @@ def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(mon { "earth-client": system_logs.LogSource( source_id="earth-client", - name="Earth 浏览器端", + name="智能星球浏览器端", kind="buffer", location="redis://planet:system_logs:earth-client", - description="Earth 浏览器端上报日志", + description="智能星球浏览器端上报日志", category="client", buffer_key=system_logs.get_buffer_log_key("earth-client"), ) @@ -156,6 +158,52 @@ def test_append_buffer_log_persists_normalized_level(monkeypatch): assert payload["message"] == "feed delayed" +def test_list_log_sources_includes_admin_client(monkeypatch): + fake_redis = FakeRedis() + monkeypatch.setattr(system_logs, "redis_client", fake_redis) + + sources = system_logs.list_log_sources() + + admin_source = next(item for item in sources if item["source_id"] == "admin-client") + assert admin_source["kind"] == "buffer" + assert admin_source["category"] == "client" + + +def test_read_log_events_returns_stable_cursors(tmp_path: Path, monkeypatch): + log_path = tmp_path / "backend.log" + log_path.write_text( + "\n".join( + [ + "2026-04-22 08:00:00 INFO service booted", + "2026-04-22 08:01:00 ERROR service failed", + ] + ), + encoding="utf-8", + ) + monkeypatch.setattr( + system_logs, + "LOG_SOURCES", + { + "backend": system_logs.LogSource( + source_id="backend", + name="后端服务", + kind="file", + location=str(log_path), + description="测试文件日志", + category="service", + ) + }, + ) + + events = system_logs.read_log_events("backend", 50, level="error") + + assert events is not None + assert len(events) == 1 + assert events[0].source_id == "backend" + assert events[0].cursor.startswith("backend:") + assert events[0].line.endswith("ERROR service failed") + + def test_infer_log_level_prefers_leading_prefix_over_query_string(): line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK' @@ -215,3 +263,22 @@ def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monk "ERROR: bind failed", "2026-04-23 23:41:32 INFO service=backend message=request served", ] + + +def test_database_system_log_search_matches_context_key_value_aliases(): + record = SystemLog( + id=2218, + occurred_at=datetime(2026, 5, 28, 9, 14, 50, tzinfo=UTC), + source="backend", + service="collector", + module="app.services.collectors.base", + event="collector.run.failed", + level="error", + message="Collector run failed", + context={"collector_name": "celestrak_tle", "datasource_id": 20, "task_id": 26906}, + ) + + event = system_logs._database_event_from_system_record(record) + + assert system_logs.event_matches_search(event, "task_id=26906") + assert system_logs.event_matches_search(event, "datasource_id=20") diff --git a/backend/tests/test_tv_proxy.py b/backend/tests/test_tv_proxy.py new file mode 100644 index 00000000..0c06de5a --- /dev/null +++ b/backend/tests/test_tv_proxy.py @@ -0,0 +1,35 @@ +from app.api.v1.tv import _rewrite_hls_uri_attributes, _should_strip_hls_metadata_line + + +def test_rewrite_hls_uri_attributes_rewrites_subtitle_manifest_url(): + line = '#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",URI="index_3_0.m3u8"' + + rewritten = _rewrite_hls_uri_attributes( + line, + base_url="https://example.com/live/master.m3u8", + ) + + assert 'URI="/api/v1/tv/proxy?url=https%3A%2F%2Fexample.com%2Flive%2Findex_3_0.m3u8"' in rewritten + + +def test_rewrite_hls_uri_attributes_rewrites_absolute_uri(): + line = '#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=1234,URI="https://cdn.example.com/live/iframe.m3u8"' + + rewritten = _rewrite_hls_uri_attributes( + line, + base_url="https://example.com/live/master.m3u8", + ) + + assert 'URI="/api/v1/tv/proxy?url=https%3A%2F%2Fcdn.example.com%2Flive%2Fiframe.m3u8"' in rewritten + + +def test_strip_hls_subtitle_media_metadata(): + line = '#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",URI="index_3_0.m3u8"' + + assert _should_strip_hls_metadata_line(line) is True + + +def test_keep_hls_audio_media_metadata(): + line = '#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="English",URI="audio.m3u8"' + + assert _should_strip_hls_metadata_line(line) is False diff --git a/backend/tests/test_vessels.py b/backend/tests/test_vessels.py index bb660e95..7476a5f7 100644 --- a/backend/tests/test_vessels.py +++ b/backend/tests/test_vessels.py @@ -8,7 +8,7 @@ from app.api.v1 import visualization from app.api.v1.visualization import convert_vessels_to_geojson from app.db.session import get_db from app.main import app -from app.models.vessel import AISRawObservation, VesselPosition, VesselStatic +from app.models.vessel import AISRawObservation, VesselCurrentState, VesselPosition, VesselStatic from app.services import barentswatch from app.services.collectors.aisstream import AISStreamCollector from app.services.collectors.vessel_ais import VesselAISCollector @@ -17,6 +17,7 @@ from app.services.vessel_ais_aggregation import ( build_field_conflict_candidates, build_observation_hash, record_vessel_ais_observation, + upsert_vessel_current_state, ) @@ -109,6 +110,58 @@ async def test_record_vessel_ais_observation_skips_existing_hash(): assert db.added == [] +@pytest.mark.asyncio +async def test_upsert_vessel_current_state_keeps_latest_position_and_static_fields(): + current = VesselCurrentState( + mmsi=257123000, + lat=59.91, + lon=10.73, + name="OSLO TRADER", + source="barentswatch_vessels", + observed_at=datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc), + field_sources={"name": "aisstream_vessels"}, + ) + + class _Session: + async def get(self, _model, _mmsi): + return current + + def add(self, _item): + raise AssertionError("existing current state should be updated") + + db = _Session() + result = await upsert_vessel_current_state( + db, + source="aisstream_vessels", + normalized_payload={"mmsi": 257123000, "lat": 59.92, "lon": 10.74, "sog": 12.4}, + observed_at=datetime(2026, 4, 30, 12, 1, tzinfo=timezone.utc), + ) + + assert result is current + assert current.lat == pytest.approx(59.92) + assert current.lon == pytest.approx(10.74) + assert current.name == "OSLO TRADER" + assert current.source == "aisstream_vessels" + + await upsert_vessel_current_state( + db, + source="barentswatch_vessels", + normalized_payload={"mmsi": 257123000, "lat": 59.93, "lon": 10.75, "name": "LOW PRIORITY"}, + observed_at=datetime(2026, 4, 30, 12, 2, tzinfo=timezone.utc), + ) + assert current.lat == pytest.approx(59.93) + assert current.name == "OSLO TRADER" + + await upsert_vessel_current_state( + db, + source="barentswatch_vessels", + normalized_payload={"mmsi": 257123000, "lat": 1, "lon": 2, "name": "OLD"}, + observed_at=datetime(2026, 4, 30, 11, 59, tzinfo=timezone.utc), + ) + assert current.lat == pytest.approx(59.93) + assert current.name == "OSLO TRADER" + + def test_build_field_conflict_candidates_from_raw_observations(): observations = [ AISRawObservation( @@ -527,7 +580,7 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch): now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc) monkeypatch.setattr( visualization, - "get_aggregated_vessels_snapshot", + "get_current_vessels_snapshot", AsyncMock( return_value=[ { @@ -573,6 +626,36 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch): app.dependency_overrides.clear() +@pytest.mark.asyncio +async def test_vessel_snapshot_accepts_fractional_zoom(monkeypatch): + monkeypatch.setattr( + visualization, + "get_current_vessels_snapshot", + AsyncMock(return_value=[]), + ) + + async def override_get_db(): + yield object() + + app.dependency_overrides[get_db] = override_get_db + transport = ASGITransport(app=app) + try: + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get( + "/api/v1/vessels/snapshot", + params={ + "bbox": "-180,-85.05112878,180,85.05112878", + "zoom": 1.6, + "limit": 3000, + }, + ) + + assert response.status_code == 200 + assert response.json()["count"] == 0 + finally: + app.dependency_overrides.clear() + + @pytest.mark.asyncio async def test_legacy_vessels_geojson_route_is_not_registered(): transport = ASGITransport(app=app) @@ -597,7 +680,7 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch): now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc) captured = {} - async def fake_get_aggregated_vessels_snapshot(db, *, bbox, limit, observed_since): + async def fake_get_current_vessels_snapshot(db, *, bbox, limit, observed_since): captured["bbox"] = bbox captured["limit"] = limit captured["observed_since"] = observed_since @@ -624,8 +707,8 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch): monkeypatch.setattr( visualization, - "get_aggregated_vessels_snapshot", - fake_get_aggregated_vessels_snapshot, + "get_current_vessels_snapshot", + fake_get_current_vessels_snapshot, ) class _Result: @@ -661,6 +744,8 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch): assert captured["bbox"] == (10.0, 59.0, 11.0, 60.0) assert captured["limit"] == 5000 assert data["diagnostics"]["bbox_applied"] is True + assert data["diagnostics"]["source"] == "vessel_current_state" + assert data["diagnostics"]["current_state_count"] == 2 assert data["diagnostics"]["legacy_feature_count"] == 0 assert data["diagnostics"]["legacy_backfilled_mmsi"] == 0 finally: @@ -668,34 +753,12 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch): @pytest.mark.asyncio -async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(monkeypatch): - now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc) +async def test_vessel_snapshot_does_not_fallback_to_history_when_current_state_is_empty(monkeypatch): monkeypatch.setattr( visualization, - "get_aggregated_vessels_snapshot", + "get_current_vessels_snapshot", AsyncMock(return_value=[]), ) - monkeypatch.setattr( - visualization, - "_load_legacy_vessel_snapshot_features", - AsyncMock( - return_value=[ - { - "type": "Feature", - "id": 257123000, - "geometry": {"type": "Point", "coordinates": [10.73, 59.91]}, - "properties": { - "mmsi": 257123000, - "name": "OSLO TRADER", - "vessel_type": 70, - "vessel_type_name": "Cargo", - "received_at": now.isoformat(), - }, - } - ] - ), - ) - result = await visualization.build_vessel_snapshot_response( object(), bbox=(10.0, 59.0, 11.0, 60.0), @@ -705,12 +768,11 @@ async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(mon since_minutes=60, ) - assert result["count"] == 1 - assert result["features"][0]["properties"]["name"] == "OSLO TRADER" + assert result["count"] == 0 assert result["diagnostics"]["raw_feature_count"] == 0 - assert result["diagnostics"]["legacy_feature_count"] == 1 - assert result["diagnostics"]["legacy_backfilled_mmsi"] == 1 - assert result["diagnostics"]["legacy_fallback_used"] is True + assert result["diagnostics"]["legacy_feature_count"] == 0 + assert result["diagnostics"]["legacy_backfilled_mmsi"] == 0 + assert result["diagnostics"]["legacy_fallback_used"] is False @pytest.mark.asyncio diff --git a/backend/tests/test_visualization_compute_centers.py b/backend/tests/test_visualization_compute_centers.py index 1c989228..3ff26449 100644 --- a/backend/tests/test_visualization_compute_centers.py +++ b/backend/tests/test_visualization_compute_centers.py @@ -816,6 +816,27 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch): app.dependency_overrides.clear() +@pytest.mark.asyncio +async def test_empty_cable_layers_return_empty_feature_collections(): + class _ScalarResult: + def scalars(self): + class _Scalars: + def all(self): + return [] + + return _Scalars() + + class _FakeSession: + async def execute(self, _query): + return _ScalarResult() + + cables = await visualization_api._build_cables_geojson(_FakeSession()) + landing_points = await visualization_api._build_landing_points_geojson(_FakeSession()) + + assert cables == {"type": "FeatureCollection", "features": []} + assert landing_points == {"type": "FeatureCollection", "features": []} + + @pytest.mark.asyncio async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch): def _fake_ror(query): diff --git a/docker-compose.local-model.yml b/docker-compose.local-model.yml index 0927245a..50f2c14b 100644 --- a/docker-compose.local-model.yml +++ b/docker-compose.local-model.yml @@ -15,12 +15,16 @@ services: retries: 10 aiprovider: + image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest} build: context: . dockerfile: aiprovider/Dockerfile args: PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim} UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest} + AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown} + secrets: + - planet_uv_config container_name: planet_aiprovider ports: - "8010:8010" @@ -43,3 +47,7 @@ services: volumes: ollama_data: + +secrets: + planet_uv_config: + file: ${PLANET_UV_CONFIG_FILE:-/dev/null} diff --git a/docker-compose.simple.yml b/docker-compose.simple.yml index c8c2e3fd..cd2e1a97 100644 --- a/docker-compose.simple.yml +++ b/docker-compose.simple.yml @@ -2,12 +2,16 @@ version: '3.8' services: aiprovider: + image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest} build: context: . dockerfile: aiprovider/Dockerfile args: PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim} UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest} + AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown} + secrets: + - planet_uv_config env_file: - ./aiprovider/.env - ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env} @@ -43,3 +47,7 @@ services: volumes: postgres_data: redis_data: + +secrets: + planet_uv_config: + file: ${PLANET_UV_CONFIG_FILE:-/dev/null} diff --git a/docker-compose.yml b/docker-compose.yml index 3ee9f9eb..f05a6415 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,12 +2,16 @@ version: '3.8' services: aiprovider: + image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest} build: context: . dockerfile: aiprovider/Dockerfile args: PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim} UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest} + AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown} + secrets: + - planet_uv_config env_file: - ./aiprovider/.env - ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env} @@ -53,3 +57,7 @@ services: volumes: postgres_data: redis_data: + +secrets: + planet_uv_config: + file: ${PLANET_UV_CONFIG_FILE:-/dev/null} diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index d92e5aa4..720d9b98 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -8,6 +8,324 @@ This project follows the repository versioning rule: - `improvement` -> `+0.0.1`（bugfix + 小功能混合） - `bugfix` -> `+0.0.1` +## [0.74.3] — 2026-09-13 + +Released: 2026-09-13 + +### Highlights +- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限，减少手工安装步骤。 +- 数据库初始化先核对容器端口与后端真实连接，连接和认证通过后才创建表和默认数据。 + +### Added / Fixed / Improved +- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足，修正未安装 Docker 时误提示启动 socket 的诊断。 +- 自动补齐缺失的 Docker 依赖并启动本地服务，以原用户身份刷新 Docker 组权限；保留参数和 PATH，不依赖 sg。 +- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置，保留端口冲突等具体错误；端口映射异常时最多保留数据卷重建一次 PostgreSQL。 +- 新增后端数据库只读连接检查，对认证、库名和网络失败给出不含密码或完整连接串的诊断。 +- 将 Docker 与数据库启动隔离回归测试接入快速检查，并同步 README、harness 和中英文运维说明。 + +--- + +## [0.74.2] — 2026-07-01 + +Released: 2026-07-01 + +### Highlights +- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`，删除重复维护的旧 Claude command 入口。 +- 强化视觉证据规则：截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置，而不是跳过后猜测。 +- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback，同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。 + +### Added / Fixed / Improved +- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容，保留最新单一入口和 harness 验证说明。 +- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate，补齐路径解析、访问失败报告和 OCR fallback 边界。 +- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口，并更新文档受众计划中的旧路径引用。 + +--- + +## [0.74.1] — 2026-06-30 + +Released: 2026-06-30 + +### Highlights +- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内，移除旧的全局“选择资产/上传”工具栏。 +- 新增字段级图片拖拽反馈，拖到对应字段时直接提示将图片复制为 Logo 或标题图。 +- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式，并同步中英文使用手册、快速开始和控制台上下文文档。 + +### Added / Fixed / Improved +- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。 +- `FieldGrid` 支持按字段注入自定义输入控件，同时复用统一草稿提交路径。 +- 品牌上传拖拽态改为低饱和 tactile 配色，上传按钮保持蓝色轻立体样式，容器内上下/右侧留白对齐为 3px。 +- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。 + +--- + +## [0.74.0] — 2026-06-30 + +Released: 2026-06-30 + +### Highlights +- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口，减少英文界面中文残留。 +- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示，避免英文态裁切或错位。 +- 将容易遗漏的 i18n 入口和视觉回归加入 harness，让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。 + +### Added / Fixed / Improved +- 新增 Earth runtime i18n 入口，统一高清材质、启动状态、错误提示和图层状态文案来源。 +- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。 +- 更新控制台与 Earth 布局规则，保留 brand 尺寸语义，同时让标题、副标题和通知胶囊按内容完整显示。 +- 扩展 frontend smoke 与 harness 文档，固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。 +- 更新 Earth 新闻本地化服务与测试，确保英文界面新闻内容不再回退中文 UI 文案。 + +--- + +## [0.73.0] — 2026-06-29 + +Released: 2026-06-29 + +### Highlights +- 新增前端统一 i18n 基础设施，让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。 +- 控制台侧边栏偏好面板接入语言与主题切换，并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。 +- 扩展 harness 与 smoke 覆盖，确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。 + +### Added / Fixed / Improved +- 新增 `frontend/src/i18n/`，用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。 +- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。 +- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译，并覆盖动态计数字符串。 +- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership，避免首屏溢出和状态词裁切。 +- 更新 i18n 计划、控制台前端上下文、harness 文档和规则，记录语言迁移边界、状态指示器布局约束和一屏验证要求。 + +--- + +## [0.72.0] — 2026-06-29 + +Released: 2026-06-29 + +### Highlights +- 将 agent 入口收敛到单一 `AGENTS.md`，并让 harness 明确阻止小写入口再次分叉。 +- 新增完整本地 harness 验证层，覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。 +- 扩展 Earth News 与控制台 smoke，确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。 + +### Added / Fixed / Improved +- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本，并将未跟踪 harness 设施纳入发布。 +- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出，改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。 +- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。 +- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档，明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。 +- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。 + +--- + +## [0.71.1] — 2026-06-26 + +Released: 2026-06-26 + +### Highlights +- 修复 Earth 欧洲、美洲、中东与非洲等区域新闻被亚太来源和旧来源过滤饿死的问题，滚动条、面板和巡航重新回到同一批区域 payload。 +- 将当前可见新闻和巡航新闻提升到目标位置/翻译优先队列，避免历史普通 Redis backlog 阻塞用户正在看的新闻精修。 +- 新增 agent harness 入口、代码地图、验证脚本与双语技术说明，让后续维护能按现有 uv/Bun/Gitea 工作流检查而不替代项目规则。 + +### Added / Fixed / Improved +- EarthFeed 在全局和巡航队列中按区域轮转候选新闻，保留区域视图的“当前区域 + global”规则，并补充回归测试。 +- `news.js` 按区域、类型、来源和数量隔离并发刷新请求，丢弃旧区域响应；跨区域时不再复用旧来源筛选。 +- 新闻展示在中文本地化未完成时回退原始标题和摘要，避免出现有内容却显示“新闻汉化中”的卡片。 +- 新闻目标位置 worker 新增优先 stream、pending reclaim、任务超时和并发处理；无效消息会确认并删除，减少队列堆积。 +- 补充 Earth 新闻源、Earth 前端结构、harness 和版本历史文档，并移除控制台 auth store 的调试日志。 + +--- + +## [0.71.0] — 2026-06-11 + +Released: 2026-06-11 + +### Highlights +- 将 Motion Agent 升级为可供 Web/UE 共用的双向控制服务，补齐真实 MediaPipe 识别 worker、设备控制、动作白名单和 WSL 摄像头开箱启动链路。 +- 新增 Earth 手动新闻内容组、条目、导入与重处理能力，并改进按 locale 和启用来源进行的新闻补充与多样化。 +- 对齐动捕模式下的 Earth 点击、详情锁定和卫星轨迹交互，同时完善启动脚本、测试 harness 与运维说明。 + +### Added / Fixed / Improved +- Motion Agent 支持命令结果、状态、骨架与手势事件，统一单路/双路输入配置，并随 `planet.sh` 默认启动；仓库内提供 usbipd-win fallback 安装包。 +- Earth 新闻服务集中处理显示就绪判断和来源多样化，避免存储层与编排层重复筛选；新增手动新闻 API 与回归测试。 +- 清理 Motion Agent 重复识别执行路径、前端不稳定随机 key 和过时计划描述，补齐 pytest 路径 harness、双语使用手册、快速开始与数据流文档。 + +--- + +## [0.70.0] — 2026-06-04 + +Released: 2026-06-04 + +### Highlights +- 新增后端枚举契约治理，将稳定协议状态集中到 `app/core/enums.py`，同时保持数据库和 API 的小写字符串兼容。 +- 改进 Earth 新闻分类、重要度与 Breaking 插队链路，并补齐中英文新闻源与枚举契约文档。 +- 将 Earth 船只展示改为 `vessel_current_state` 当前状态快照，保留 AIS 原始历史用于轨迹和态势分析。 +- 清理错误的船只视口刷新/订阅思路，恢复全球船只显示，并让性能优化集中到批量渲染、关闭动态聚类和减少 hover/rebuild 开销。 + +### Added / Fixed / Improved +- `earth_news_classification.py` 集中管理新闻类型、重要度和 Breaking 规则，避免抓取编排层重复判断。 +- `/api/v1/vessels/snapshot` 支持全球当前状态读取和小数 zoom，诊断信息明确返回 `vessel_current_state` 来源。 +- 船只前端使用全球 bbox + `limit=3000`，不再随相机视口重复请求或建立多视口 WebSocket 订阅。 +- 中英文技术文档同步更新船只、采集器、数据流、渲染层级、样式参数和历史计划状态。 + +--- + +## [0.69.0] — 2026-06-03 + +Released: 2026-06-03 + +### Highlights +- 新增 Earth 新闻源治理能力，支持多 Feed 子项、源属性标签、新闻类型过滤、重要度规则和健康测试。 +- 新增观测日志聚合视图，按 fingerprint 汇总 Earth、Admin 和服务端重复运行时事件，并保留原始发生明细。 +- 改进 TV/HLS 播放恢复和代理重写，降低字幕、分片和源站波动导致的直播不可用噪声。 + +### Added / Fixed / Improved +- Earth 新闻面板和 UE 端统一通过 `/api/v1/news/earth-feed` 使用 `categories` 与 `locale` 服务端过滤，Web 端新闻类型偏好仅保存在当前浏览器。 +- 控制台日志页新增重复统计、原始日志和审计日志模式，前端上报器会合并短窗口内的重复错误并提交 `occurrence_count`。 +- AI Provider / 服务端运行时可通过受保护的 observability ingest 入口写入结构化事件。 +- 新闻源文档新增中英文配置说明，并补齐 Earth 前端、控制台日志和公开 Docs 索引。 + +--- + +## [0.68.1] — 2026-05-28 + +Released: 2026-05-28 + +### Highlights +- 修复 CelesTrak active 未更新窗口下清库后无法恢复的问题，fallback 会优先复用本地有效 group 缓存。 +- 修复数据源任务队列“查看日志”无法按 `task_id=...` 命中数据库结构化日志的问题。 + +### Added / Fixed / Improved +- CelesTrak fallback group 列表改为公开可用分组，移除失效 group，并在没有 active 缓存时仍可从本地 group 缓存恢复采集。 +- 数据库日志搜索补充 JSON context 的 `key=value` 别名，支持 `task_id=26906`、`datasource_id=20` 这类控制台跳转查询。 +- 补充 CelesTrak 缓存边界、数据源任务日志跳转和运维恢复说明的中英文文档。 + +--- + +## [0.68.0] — 2026-05-28 + +Released: 2026-05-28 + +### Highlights +- 新增数据源任务队列的实时指标校准和批量删除进度，让大表清理、取消和完成状态在控制台中可感知。 +- 新增智能星球 interactable 可插拔聚类策略，支持稳定 3D 球面聚类、动态屏幕聚类和 250% 以上自动散开。 +- 改进新设备启动流程，`planet.sh` 会在启动前同步前端依赖，避免缺失依赖导致控制台动态导入失败。 + +### Added / Fixed / Improved +- 数据源列表改为中文记录数指标，并对 AIS 大表使用统计估算 + 单条详情精确校准，降低首次加载成本。 +- 数据删除任务改为分批删除并广播进度，清理 AIS 衍生表后自动 `ANALYZE`，同时修复取消中任务恢复和状态文案。 +- Earth BGP、算力中心和 interactable 图层默认使用 `stable-spherical` 聚类，船舶实时层保留 `dynamic-screen`。 +- 新增中英文 Earth interactable clustering 文档，并补充采集队列、后端删除语义和前端依赖同步说明。 + +--- + +## [0.67.0] — 2026-05-27 + +Released: 2026-05-27 + +### Highlights +- 新增控制台日志实时跟随与前端运行时错误上报，帮助在控制台内直接排查 Admin / Earth 客户端异常。 +- 重构智能星球 Interactable 聚合和 wheel 缩放输入，保持真实地理锚点稳定，同时让鼠标滚轮和触控板拥有各自合适的缩放手感。 + +### Added / Fixed / Improved +- 新增 `/ws` 日志 tail 通道、数据库/文件日志统一事件读取，以及 Admin `error` / `unhandledrejection` / React ErrorBoundary 上报链路。 +- 优化控制台日志页状态颜色、跟随体验和运行时错误展示，并将 Admin 本地工具模块从 `lib` 命名迁移为局部 `utils`。 +- 修复智能星球国界/高清材质壳半径对齐、Interactable cluster 圆点显示、缩放目标累积和触控板连续缩放问题。 +- 更新 `planet.sh` 与 `.gitignore`，避免新环境构建污染锁文件并移除前端 `lib` 目录特殊放行。 +- 补充智能星球前端、渲染层级、控制台状态与运行时日志文档。 + +--- + +## [0.66.3] — 2026-05-26 + +Released: 2026-05-26 + +### Highlights +- 修复控制台 Admin 动态导入时缺失 `../lib/utils` 导致 Vite 返回 500 的问题。 +- 补齐前端依赖安装状态，确保 Markdown Mermaid 渲染器可解析 `mermaid` 包。 + +### Added / Fixed / Improved +- 新增 Admin 本地 `cn` 与 `formatNumber` 工具模块，恢复布局、UI 组件和 Dashboard 的共享工具引用。 +- 放开 `.gitignore` 中 `frontend/src/admin/lib` 的源码例外，避免 utility module 再次被全局 `lib/` 规则漏提交。 +- 重启前端开发服务并验证 `/admin` 无 Vite overlay 和 console error。 + +--- + +## [0.66.2] — 2026-05-26 + +Released: 2026-05-26 + +### Highlights +- 统一中文界面和公开文档中的产品命名：`Earth` 显示为“智能星球”，`Admin` 显示为“控制台”，`Docs` 显示为“文档”。 +- 修复文档手册、Docs catalog、控制台入口和智能星球设置中残留的中英混排标题。 + +### Added / Fixed / Improved +- 更新控制台侧边栏、智能星球系统设置、移动端抽屉和登录页品牌文案。 +- 同步前后端 Docs metadata、中文技术文档标题、使用手册、快速开始、术语表和运维手册中的产品命名。 +- 补充清理智能星球图层缓存、品牌配置、展示缓存等控制台 toast / dialog 文案。 + +--- + +## [0.66.1] — 2026-05-26 + +Released: 2026-05-26 + +### Highlights +- 修复 CelesTrak active 更新窗口内清库后无法恢复的问题，新增持久原始下载缓存和完整 fallback group mode。 +- 避免数据源任务 WebSocket 与轮询同时完成时重复弹出采集失败 toast。 +- `planet.sh destroy` 保留上游原始下载缓存，数据库清空后仍可用缓存重灌。 + +### Added / Fixed / Improved +- CelesTrak 403 `GP data has not updated` 会先复用 active 缓存；无 active 缓存时完整拉取 `starlink/gps-ops/galileo/glonass/beidou/leo/geo/iridium-next`，任一 group 缺失则整体失败，不保存 partial。 +- 下载器缓存从 `/tmp` 迁到 `$PLANET_CACHE_DIR/downloads`，并保留 HTTP 错误响应正文以支持上游限频语义判断。 +- 补充 CelesTrak cache/fallback 回归测试和中英文运维/卫星策略文档。 + +--- + +## [0.66.0] — 2026-05-26 + +Released: 2026-05-26 + +### Highlights +- 将 Admin 正式化为唯一控制台入口，移除旧 AntD 后台、Admin Next 路由痕迹和相关依赖。 +- 引入 PostgreSQL 数据作业队列、Earth outbox 同步和可交互对象管线，让采集、清理和 Earth 刷新进入可追踪异步链路。 +- 强化 AI、AI 工具、设置连接测试和采集任务的结构化业务日志，关键事件可在系统日志中检索。 +- CelesTrak 轨道根数采集改为完整 `active` 目录下载、续传和重试，避免部分分组失败时保存不完整卫星数据。 + +### Added / Fixed / Improved +- Admin 数据源页修复内置源调度启停按钮，使用 `is_active` 判断 `/enable` 与 `/disable`，避免采集状态误导调度开关。 +- Earth 新增通用 interactables 层和平台数据流文档，支持对象级 delta 同步与后续小图层扩展。 +- `planet.sh init/start/destroy` 补齐 uv 镜像回退、Bun unzip 依赖、HTTPS/LAN 跳转协议和 OOBE 清理语义。 +- Markdown code block、数据源队列、演示模式、About 版本展示、TV 链接协议和文档索引同步完成。 + +--- + +## [0.65.2] — 2026-05-22 + +Released: 2026-05-22 + +### Highlights +- Stabilize AI Provider rebuild detection so unchanged source no longer rebuilds just because file timestamps or local cache state changed. +- Keep `uv.lock` on the official registry while still allowing local and Docker builds to use the user's `uv.toml`. +- Prevent local startup and bootstrap commands from rewriting `uv.lock` when a user-level mirror is configured. + +### Added / Fixed / Improved +- AI Provider fingerprints now use content hashes, base image inputs, dependency metadata, and an image label instead of file mtimes. +- `planet.sh` and `scripts/bootstrap-dev.sh` now use `uv sync --frozen`, and backend startup uses `uv run --frozen`. +- Docker Compose passes the AI Provider fingerprint into the image build so future starts can inspect the image label directly. + +--- + +## [0.65.1] — 2026-05-22 + +Released: 2026-05-22 + +### Highlights +- 修复 `planet.sh` 与 Docker Compose 对 AI Provider 镜像名不一致导致的重复构建问题。 +- 让本地 `uv` 与 Docker build 统一使用用户机器上的 `uv.toml`，避免把镜像源写入 `uv.lock`。 +- 通过 BuildKit secret 向容器构建传入 uv 配置，保留用户源选择且不把配置写入镜像层。 + +### Added / Fixed / Improved +- `planet.sh` 自动解析 `UV_CONFIG_FILE`、仓库 `uv.toml` 与用户级 `~/.config/uv/uv.toml`，并清理会覆盖配置的 `UV_INDEX_URL` 环境变量。 +- AI Provider 和后端 Dockerfile 在 `uv sync` 阶段挂载 uv 配置 secret，并复用 BuildKit uv cache。 +- Docker Compose 三套配置统一声明 `planet-aiprovider:latest` 镜像名和 `planet_uv_config` build secret。 + +--- + ## [0.65.0] — 2026-05-21 Released: 2026-05-21 diff --git a/docs/HARNESS.md b/docs/HARNESS.md new file mode 100644 index 00000000..a813e322 --- /dev/null +++ b/docs/HARNESS.md @@ -0,0 +1,325 @@ +# Agent Harness + +This harness improves discoverability, repeatability, and agent safety for the +existing Planet project. It does not replace current project rules, scripts, CI, +or release workflows. + +## Authority And Conflicts + +Existing project rules are authoritative: + +1. `rules.md` +2. `AGENTS.md` +3. Current implementation docs under `docs/technical/` +4. Existing scripts, especially `planet.sh` +5. Existing Gitea workflow files under `.gitea/workflows/` + +When harness guidance conflicts with any of the above, keep the existing rule, +do not overwrite the existing workflow, and add a compatibility note here or in +`docs/harness-audit.md`. + +For frontend or documentation audits, also read the Rules Coverage Evidence +section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current +static checks, Playwright smoke coverage, and remaining manual review areas, so +an agent can distinguish a proved harness pass from a rule that still needs +human-quality inspection. + +When the user describes work with product words rather than module names, use +the `rules.md` **Agent Discovery Index** before deciding which modules to load. +It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`, +`地球`, `模型供应商`, and `发版` to the required rule modules. + +## Starting Work + +Recommended startup flow: + +```bash +git status --short +scripts/harness/doctor.sh +``` + +Then read only the relevant implementation docs: + +- Backend/API/data work: `docs/technical/zh/backend-*.md` and matching English + docs when public docs are affected. +- Frontend/admin work: `docs/technical/zh/frontend-admin-frontend-context.md`. +- Earth work: `docs/technical/zh/earth-frontend-context.md`, + `docs/technical/zh/earth-render-layer-order.md`, and style docs when visual + semantics change. +- Operations work: `docs/technical/zh/ops-runbook.md` and + `docs/technical/zh/ops-planet-sh-startup.md`. +- AI Provider work: `docs/technical/zh/agents-aiprovider.md`. +- Documentation work: `docs/documentation-coverage-rules.md`. + +Use focused inspection commands before broad reads: + +```bash +rg -n "" +git diff --stat HEAD +git diff --name-only HEAD +git diff --unified=0 HEAD -- +``` + +## Existing Commands + +| Purpose | Command | +| --- | --- | +| First setup | `./planet.sh init` | +| Start local stack | `./planet.sh start` | +| Start with LAN access | `./planet.sh start --allow-lan` | +| Restart all services | `./planet.sh restart` | +| Restart one area | `./planet.sh restart -b`, `-f`, `-a`, or `-d` | +| Health check | `./planet.sh health` | +| Logs | `./planet.sh log`, `./planet.sh log -b`, `-f`, `-a`, or `-m` | +| Create local user | `./planet.sh createuser` | +| Destructive local reset | `./planet.sh destroy` | +| Backend smoke tests | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` | +| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` | +| Mock AIS WebSocket | `bun run mock:ais-ws` | + +## Harness Commands + +| Tier | Command | What It Does | +| --- | --- | --- | +| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. | +| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. | +| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. | +| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. | +| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. | +| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. | +| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. | + +Docker image smoke builds are expensive and are off by default: + +```bash +PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh +``` + +Frontend Playwright smoke runs by default in full validation after the frontend +build. It starts a local Vite preview and checks the `/` to Earth redirect, +public pages, unknown-route login fallback, protected admin route login +fallback, authenticated unknown-route fallback to `/admin`, Docs loading with +mocked API content, Docs detail page +language/theme/search interactions, every Docs catalog slug exposed by the +frontend/backend metadata, the Earth iframe entry point, login error handling, +register + email verification, password reset, standalone email verification, +and authenticated `super_admin` rendering for every admin route plus core +`section` deep links derived from the actual admin route and section config. +Authenticated admin +checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and +mobile passes also fail on global horizontal overflow so table/detail panels +must keep overflow ownership inside their own scroll regions. To enforce the +existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a +hard rendered check: the shell must resolve to the viewport height through the +root 100% height chain, `#root`/document/body must not gain vertical overflow, +and the desktop sidebar account/preferences area must remain inside the first +viewport while the nav owns any excess scrolling. The smoke also +derives the sidebar menu from the actual admin route manifest and clicks every +visible `super_admin` menu entry on both desktop and mobile viewports, then +exercises safe interaction paths for admin search, section tabs, the AI settings +shortcut, logs view switching, user dialog opening, and data distribution toggles. +It also exercises Earth News source testing, add/cancel source draft behavior, +and manual news group creation against mocked `/earth/news-*` APIs. +Documented AI and collector +deep links such as `/ai?section=integrations`, `/ai?section=playground`, and +`/collection-management?section=collector_credentials` are part of the rendered +smoke surface: + +```bash +PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh +PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh +``` + +### Visual Evidence And Style Consistency + +- Treat user-provided screenshots and images as primary visual evidence. If a + screenshot contradicts written text, inspect the image first and explicitly + call out the mismatch before deciding what to change. +- Path resolution is part of visual evidence handling. If a referenced + screenshot path cannot be opened, try reasonable local equivalents first: + WSL/Windows path conversion, workspace-relative paths, absolute paths, current + thread attachments, repository files, and obvious local attachment/download + locations. +- If the image still cannot be found or opened, report the exact path/access + blocker instead of guessing. Do not infer image content from the filename, alt + text, surrounding prose, logs, or memory. +- OCR is acceptable evidence for text-only questions or non-multimodal + environments; state when OCR was the fallback. Layout, color, spacing, pixel, + and rendering issues still require a real visual inspection or an explicit + "could not verify visually" note. +- Same-category UI surfaces must use one visual system per product area. Badges, + chips, pills, tags, status labels, small buttons, cards, panels, and toolbar + controls should reuse the shared component, shared token, or established CSS + owner for that area instead of introducing a page-local lookalike. +- `scripts/harness/frontend-rules-check.sh` warns when semantic + `badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the + approved React and Earth CSS owner files. A warning means the reviewer should + either move the style into the shared owner or document why this is a genuinely + new visual family. + +### Earth I18n Harness Rules + +Earth i18n work must validate rendered behavior, not only static text lookup. +Agents often miss dynamic strings that are created after initial page load, so +the smoke treats these as first-class i18n surfaces: + +- **Visible text and attributes**: translated checks must include `innerText` + plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only + buttons are user-facing copy, not implementation details. +- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards, + BGP markers, compute centers, vessels, and news must render field labels, + status values, source tags, action buttons, and disabled/tooltips in the active + language. +- **English content safety**: English mode must not fall back to Chinese news + titles, summaries, feed names, measure words, or generic status labels. If no + English localization exists, hide the item or use a neutral English fallback. +- **Brand assets**: locale switching must update both text and image assets. + The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png` + for English while keeping the same top-left layout and logo position. +- **Controls and state**: switch/segmented-control visuals must follow the real + checked/pressed state after both direct clicks and programmatic panel changes. + A control is not valid if the state changes but the thumb, active pill, or + `aria-*` state stays stale. +- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy + must enter through `earthMessage(...)` plus the centralized + `EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind + `showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`, + `showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`, + or `earth:status` events. +- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not + overflow their panel. Prefer a slightly wider owning panel for important + status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a + translated tooltip. + +Current frontend smoke explicitly covers the Earth English locale flow: brand +image swap, settings language controls, panel switch visual sync, English news +filtering, English detail-card text and tooltips, and TV default/source labels. + +## Environment Requirements + +Required for normal development: + +- `zsh` for `planet.sh` +- `uv` for Python dependency and test execution +- `bun` for frontend dependency and build execution +- Python resolved by `uv` from the root `pyproject.toml` + +Harness command lookup first checks the current non-interactive `PATH`. If a +required tool is not visible there, `scripts/harness/lib.sh` asks the user's +login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command +path. This avoids hardcoding a dotfile while still covering agent environments +that do not inherit the user's normal shell setup. + +Required for full local stack operation: + +- Docker and Docker Compose +- PostgreSQL and Redis containers started by `planet.sh` + +Optional for delivery smoke: + +- Docker daemon for image builds +- Helm for chart lint/template checks + +For routine harness validation, do not install missing system software +automatically. Report the gap and point to the explicit bootstrap entry points. +`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on +Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access. +This bootstrap behavior is intentional; do not invoke it merely to make harness +checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies. + +Docker bootstrap regression checks use isolated command stubs and never install +packages or modify the host daemon: + +```bash +uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py +uv run --frozen --project . python scripts/harness/test_database_startup.py +``` + +Database startup regressions also run in quick-check. They cover Compose +reconciliation of existing containers, visible startup errors, published-port +checks, bounded recreation that preserves volumes, and the backend connection +gate before schema initialization. Their command stubs and driver mocks do not +modify the host Docker environment. + +## What Agents Must Not Change Automatically + +- Do not replace Bun with npm, pnpm, or yarn. +- Do not migrate CI from `.gitea/workflows/` to `.github/workflows/`. +- Do not rewrite `planet.sh` lifecycle behavior as a parallel script. +- Do not run `./planet.sh destroy` unless explicitly requested. +- Do not commit `.env`, secrets, private keys, logs, or generated build output. +- Do not add external integrations, hooks, or new dependency managers just to + satisfy harness structure. +- Do not publish internal harness docs into the product Docs UI unless a + maintainer explicitly asks for it. + +## Hooks And Reminders + +No automatic hooks are installed in this phase. Manual reminders: + +- Run `scripts/harness/quick-check.sh` before handing off small changes. +- Run `scripts/harness/validate.sh` before larger cross-subsystem changes. +- Run `scripts/harness/security-check.sh` after touching config, auth, + credentials, docs examples, or generated fixtures. +- Run `scripts/harness/backend-rules-check.sh` after backend service edits to + catch direct stdout/debugger calls before they reach runtime logs. +- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose + route, package-manager, debug-output, and UI rule warnings. +- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature + route changes. +- Add focused tests before modifying backend service behavior or frontend + workflows. +- For docs changes, run the checks listed in + `docs/documentation-coverage-rules.md`. + +## Reusable Workflows + +### Feature Work + +1. Read `rules.md` modules for the touched area. +2. Check `CODEMAP.md` for entry points and ownership boundaries. +3. Inspect existing tests and docs before editing. +4. Make the smallest behavior-preserving or feature-scoped change. +5. Run `scripts/harness/quick-check.sh` or a narrower documented command. +6. Update relevant docs when behavior, workflow, or operations change. +7. For rendered frontend changes, verify the affected route with Playwright or + the full harness smoke, because `bun run build` alone does not prove page + usability. + +### Bug Fix + +1. Reproduce with a focused test or command. +2. Patch the owning module, not a caller-side workaround. +3. Run the focused regression test. +4. Run `scripts/harness/quick-check.sh` when the change is safe to validate + locally. + +### Documentation Change + +1. Read `docs/documentation-coverage-rules.md`. +2. Route docs by audience: UI users, operations, or second-party developers. +3. Keep Chinese and English technical docs paired by filename; public Docs also + need matching frontend/backend metadata when exposed in the product Docs UI. +4. Run the repository-specific docs checks that match the changed files. + +### Release Or Delivery Change + +Use the existing release skill/workflow and `.gitea/workflows/` files. Harness +validation can smoke-check Helm and Docker locally, but it must not replace the +release process. + +## Implementation Notes + +- `docs/harness-audit.md` records the discovery pass that led to this harness. +- `AGENTS.md` is the single authoritative agent guide. The older lowercase + `agents.md` entry has been merged into it and should remain absent. +- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in + `docs/technical/{zh,en}/`. +- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke + with mocked API data. It proves route shells, auth guards, and primary admin + sections render, but it is not a replacement for feature-specific browser QA + against a real backend. +- Frontend smoke prints phase-level progress by default. Use + `PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item + when diagnosing a slow or failing smoke run, or set it to `0` to suppress + progress lines. diff --git a/docs/deprecated/README.md b/docs/deprecated/README.md index 30596739..5022cb7e 100644 --- a/docs/deprecated/README.md +++ b/docs/deprecated/README.md @@ -16,7 +16,24 @@ - 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划，归档 - 仍然指导未来开发、尚未完成或仍有明确执行价值的文档，继续保留在 `docs/` +当前替代入口： + +- 业务和数据产品链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。 +- 当前后端作业、outbox 和 Earth 同步实现见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)。 +- 用户操作流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)。 +- 当前代码结构见 [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)。 + 补充说明： - 一部分归档文档来自外部或临时工作流草案，例如 sisyphus 生成的初稿 - 这类文档如果有可用内容，应先吸收到 `docs/plans/` 或 `docs/technical/`，再归档保留来源记录 + +## 近期归档 + +- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/deprecated/docs-gatekeeper-auth-plan.md)：已落地，当前实现见技术文档。 +- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/deprecated/location-resolver-shared-pipeline-plan.md)：已落地，当前实现见技术文档。 +- [Earth Surface Hover Info Plan](/home/ray/dev/linkong/planet/docs/deprecated/earth-surface-hover-info-plan.md)：已实现，保留为历史记录。 +- [Admin Next Dual Track Full Migration Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-dual-track-full-migration-plan.md)：已被当前 `frontend/src/admin/` 控制台结构替代。 +- [Admin Next Soft Glass Goal Driven Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-soft-glass-goal-driven-plan.md)：已被当前 `frontend/src/admin/` 控制台结构替代。 +- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-checklist.md)：旧 `/admin-next/*` 验收资料，保留为迁移历史。 +- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-audit-closeout-plan.md)：旧 `/admin-next/*` 审计资料，保留为迁移历史。 diff --git a/docs/plans/admin-next-dual-track-full-migration-plan.md b/docs/deprecated/admin-next-dual-track-full-migration-plan.md similarity index 100% rename from docs/plans/admin-next-dual-track-full-migration-plan.md rename to docs/deprecated/admin-next-dual-track-full-migration-plan.md diff --git a/docs/plans/admin-next-parity-audit-closeout-plan.md b/docs/deprecated/admin-next-parity-audit-closeout-plan.md similarity index 98% rename from docs/plans/admin-next-parity-audit-closeout-plan.md rename to docs/deprecated/admin-next-parity-audit-closeout-plan.md index 6dd3f3ad..0e8b58d9 100644 --- a/docs/plans/admin-next-parity-audit-closeout-plan.md +++ b/docs/deprecated/admin-next-parity-audit-closeout-plan.md @@ -104,7 +104,7 @@ The old AntD page itself treated these Earth content tabs as placeholder-level c - `models_3d` - `news_anchor_strategy` -If backend endpoints are later added, these items must be promoted into `docs/plans/admin-next-parity-checklist.md` with concrete API and UI acceptance criteria. +If backend endpoints are later added, these items must be promoted into a new active plan under `docs/plans/` with concrete API and UI acceptance criteria. ## Final Gate @@ -114,7 +114,7 @@ After the route promotion, the final gate is no longer “switch old routes.” 2. Run the static checks: - `rg "map: \\(\\) => \\[\\]|暂不支持保存|placeholder" frontend/src/admin-next` - `rg "ShadowPage|FeatureConsole|GlassPanel|InspectorDrawer" frontend/src/admin-next` -3. Manually verify every official route listed in `docs/plans/admin-next-parity-checklist.md`. +3. Manually verify every official route listed in `docs/deprecated/admin-next-parity-checklist.md`. 4. Confirm `/legacy/admin/*` still opens old AntD pages during the validation window. 5. Delete old AntD pages and remove AntD dependencies only as a separate final cleanup task after explicit confirmation. diff --git a/docs/plans/admin-next-parity-checklist.md b/docs/deprecated/admin-next-parity-checklist.md similarity index 100% rename from docs/plans/admin-next-parity-checklist.md rename to docs/deprecated/admin-next-parity-checklist.md diff --git a/docs/plans/admin-next-soft-glass-goal-driven-plan.md b/docs/deprecated/admin-next-soft-glass-goal-driven-plan.md similarity index 98% rename from docs/plans/admin-next-soft-glass-goal-driven-plan.md rename to docs/deprecated/admin-next-soft-glass-goal-driven-plan.md index 2f31d3c2..048c9e20 100644 --- a/docs/plans/admin-next-soft-glass-goal-driven-plan.md +++ b/docs/deprecated/admin-next-soft-glass-goal-driven-plan.md @@ -8,7 +8,7 @@ The redesign must cover desktop and mobile. Data display, icon semantics, table ## Criteria For Success -- This plan exists at `docs/plans/admin-next-soft-glass-goal-driven-plan.md`. +- This archived plan exists at `docs/deprecated/admin-next-soft-glass-goal-driven-plan.md`. - `/admin-next/*` has real pages for every route; route usage of `ShadowPage` is removed. - Admin Next supports `system`, `light`, and `dark` theme modes using the same persistence and system-theme idea as Docs. - The visual language reads as soft-glass / light-neumorphic instead of an AntD reskin: translucent panels, fine borders, subtle glow, cool backgrounds, restrained accent colors, crisp icons, and tactile controls. @@ -214,4 +214,3 @@ Manual viewport checks: - low height; - 125% / 150% browser zoom; - light / dark / system theme modes. - diff --git a/docs/plans/docs-gatekeeper-auth-plan.md b/docs/deprecated/docs-gatekeeper-auth-plan.md similarity index 100% rename from docs/plans/docs-gatekeeper-auth-plan.md rename to docs/deprecated/docs-gatekeeper-auth-plan.md diff --git a/docs/plans/earth-surface-hover-info-plan.md b/docs/deprecated/earth-surface-hover-info-plan.md similarity index 100% rename from docs/plans/earth-surface-hover-info-plan.md rename to docs/deprecated/earth-surface-hover-info-plan.md diff --git a/docs/plans/location-resolver-shared-pipeline-plan.md b/docs/deprecated/location-resolver-shared-pipeline-plan.md similarity index 96% rename from docs/plans/location-resolver-shared-pipeline-plan.md rename to docs/deprecated/location-resolver-shared-pipeline-plan.md index d54caa4e..9f9af54d 100644 --- a/docs/plans/location-resolver-shared-pipeline-plan.md +++ b/docs/deprecated/location-resolver-shared-pipeline-plan.md @@ -1,6 +1,6 @@ # Location Resolver Shared Pipeline Plan -**状态**：已实现，当前用户流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)，开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。 +**状态**：已实现，当前用户流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Earth 位置候选采集章节，开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。 ## Goal diff --git a/docs/documentation-coverage-rules.md b/docs/documentation-coverage-rules.md index 5ed0fca4..afdefd95 100644 --- a/docs/documentation-coverage-rules.md +++ b/docs/documentation-coverage-rules.md @@ -37,7 +37,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t - a `credential_provider` in `backend/app/core/datasource_defaults.py`; - a default credential guide in `backend/app/services/credential_guides.py`; - a supported connectivity provider in `backend/app/services/datasource_connectivity.py`; - - settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`; + - settings UI guidance or a credential form in `frontend/src/admin/pages/PlainResourcePages.tsx`; - a regression test that fails if the guide/provider is missing. ## Recommended Checks diff --git a/docs/harness-audit.md b/docs/harness-audit.md new file mode 100644 index 00000000..e1f592e9 --- /dev/null +++ b/docs/harness-audit.md @@ -0,0 +1,161 @@ +# Harness Audit + +Last audited: 2026-06-26 + +This audit records the repository state used to add the agent harness. It is a +compatibility note, not a replacement for existing rules or architecture docs. + +## Existing Commands + +| Area | Existing Command | Notes | +| --- | --- | --- | +| Bootstrap | `./planet.sh init` | Syncs uv/Bun dependencies, creates missing env files, starts data services, seeds defaults. | +| Start | `./planet.sh start` | Starts backend, frontend, AI Provider, PostgreSQL/Redis, and Motion Agent when available. | +| LAN start | `./planet.sh start --allow-lan` | Opens frontend/backend/AI Provider ports and requests Windows firewall/port cleanup when needed. | +| Restart | `./planet.sh restart` | Supports scoped restart flags for backend, frontend, AI Provider, database, and Motion Agent. | +| Health | `./planet.sh health` | Checks containers, backend `/health`, AI Provider `/health`, frontend, and Motion Agent state. | +| Logs | `./planet.sh log` | Supports backend, frontend, AI Provider, and Motion Agent log views. | +| User fallback | `./planet.sh createuser` | Interactive emergency/local account creation. | +| Destructive reset | `./planet.sh destroy` | Requires confirmation and removes Planet-owned Docker/build/runtime state. Not a validation command. | +| Backend CI smoke | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` | Mirrors `.gitea/workflows/ci.yaml`. | +| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` | Bun-only workflow. | +| Root helper | `bun run mock:ais-ws` | Runs `scripts/mock-ais-ws-server.ts` from the root package. | + +## Existing Agent Instructions + +| File | Status | Notes | +| --- | --- | --- | +| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. | +| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. | +| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. | +| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. | +| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. | + +## Existing CI Gates + +The repository uses `.gitea/workflows/`, not `.github/workflows/`. + +| Workflow | Gate | +| --- | --- | +| `.gitea/workflows/ci.yaml` | Backend smoke tests, frontend Bun build, Docker build smoke, Helm lint/template. | +| `.gitea/workflows/release.yaml` | Builds and pushes frontend, backend, and AI Provider images on main/tag/manual release events. | +| `.gitea/workflows/deploy-staging.yaml` | Deploys Helm release to staging and runs curl smoke tests inside the cluster. | + +## Existing Docs And Architecture Maps + +| Area | Docs | +| --- | --- | +| Current architecture and startup | `README.md` | +| Technical docs index | `docs/technical/zh/README.md`, `docs/technical/en/README.md` | +| Documentation rules | `docs/documentation-coverage-rules.md` | +| Operations | `docs/technical/zh/ops-runbook.md`, `docs/technical/en/ops-runbook.md` | +| Startup internals | `docs/technical/zh/ops-planet-sh-startup.md`, `docs/technical/en/ops-planet-sh-startup.md` | +| AI Provider | `docs/technical/zh/agents-aiprovider.md`, `docs/technical/en/agents-aiprovider.md` | +| Frontend admin | `docs/technical/zh/frontend-admin-frontend-context.md`, `docs/technical/en/frontend-admin-frontend-context.md` | +| Earth rendering | `docs/technical/zh/earth-frontend-context.md`, `docs/technical/zh/earth-render-layer-order.md`, `docs/technical/zh/earth-layer-style-reference.md` | +| Plans and history | `docs/plans/README.md`, `docs/deprecated/README.md` | + +## Release And Deploy Process + +- Release workflow is documented in `.codex/skills/release/SKILL.md` and + `.claude/commands/release.md`. +- Version-bearing files include `VERSION`, `frontend/package.json`, + `pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and + `docs/version-history.md`. +- Delivery automation lives in `.gitea/workflows/release.yaml` and + `.gitea/workflows/deploy-staging.yaml`. +- Helm chart entry point is `deploy/helm/planet/Chart.yaml`. + +## Missing Or Unclear Areas + +- The older lowercase `agents.md` entry has been merged into uppercase + `AGENTS.md` so coding agents and harness tools use one source of truth. +- `project_context.md` originally included older roadmap assumptions such as + Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The + harness pass updated it to separate active stack facts from future directions; + current code and technical docs still remain authoritative when details drift. +- No safe automatic hook system was already configured. This phase documents + manual reminders instead of adding hooks. +- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`. + +## Conflicts And Preserved Rules + +| Conflict Or Tension | Resolution | +| --- | --- | +| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. | +| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. | +| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. | +| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. | +| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. | +| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. | +| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. | +| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. | +| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. | +| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. | +| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. | +| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. | +| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. | +| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. | +| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. | +| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. | +| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX ` + + + + +
正在准备全球态势新闻...
@@ -640,7 +656,7 @@
- Earth Menu + 智能星球菜单 模块切换
@@ -682,7 +698,7 @@
- Layer Control + 图层控制 已启用 0 个图层
@@ -691,7 +707,7 @@
- Object Search + 对象搜索 搜索海缆、登陆点、卫星、算力中心和 BGP 事件
@@ -717,7 +733,7 @@
- Situation + 态势总览 面向移动端整合的全球态势概览
@@ -757,7 +773,7 @@
- News + 新闻 跟随当前视角聚焦全球区域新闻
@@ -768,6 +784,21 @@
0 路聚合源
+
+ + + +
+
正在准备全球态势新闻...
@@ -829,7 +860,7 @@
- Motion Capture + 动作捕捉 查看浏览器摄像头画面、骨架连线和当前匹配动作
动捕调试画面已并入设置里的动捕模式。
@@ -838,8 +869,8 @@
- Settings - 仅保留移动端仍有意义的 Earth 配置 + 设置 + 仅保留移动端仍有意义的智能星球配置
@@ -1049,6 +1080,27 @@ +
新闻类型
+
+
+ 新闻类型 + 只筛选当前浏览器的新闻面板与新闻巡航，不改变后台新闻源。 +
+
+ + + + + + + + + + + + +
+
+
+
+ 识别动作 + 关闭后不会触发对应星球控制 +
+
+ + + + + + + + + + + +
+
策划人 - 黄柳青 + 方兴东、黄柳青
产品兼开发者 @@ -1555,6 +1639,27 @@ +
新闻类型
+
+
+ 新闻类型 + 只筛选当前浏览器的新闻面板与新闻巡航，不改变后台新闻源。 +
+
+ + + + + + + + + + + + +
+
+
+
+ 识别动作 + 未勾选的动作不会触发星球端控制；Motion Agent 会同步过滤这些动作 +
+
+ + + + + + + + + + + +
+
策划人 - 黄柳青 + 方兴东、黄柳青
产品兼开发者 diff --git a/frontend/public/earth/js/about.js b/frontend/public/earth/js/about.js index f2d4f3aa..628eeb34 100644 --- a/frontend/public/earth/js/about.js +++ b/frontend/public/earth/js/about.js @@ -4,12 +4,12 @@ const DEFAULT_ABOUT = { logo_src: "./assets/brand/lim-logo.png", kicker: "About", title: "智能星球计划", - version: "v0.64.0", + version: "v0.65.2", description: "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航，提供可视化观测、事件聚合与交互式探索能力。", meta: [ { label: "出品方", value: "浙江大学临空智能媒体研究院" }, - { label: "策划人", value: "黄柳青" }, + { label: "策划人", value: "方兴东、黄柳青" }, { label: "产品兼开发者", value: "钱坤、张鸽、齐鹏" }, ], }; diff --git a/frontend/public/earth/js/bgp.js b/frontend/public/earth/js/bgp.js index 0cdd6255..3dfef3f5 100644 --- a/frontend/public/earth/js/bgp.js +++ b/frontend/public/earth/js/bgp.js @@ -333,6 +333,9 @@ const bgpEventIconLayer = createInteractableLayer({ pulseOffset: Math.random() * Math.PI * 2, }), avoidance: SURFACE_AVOIDANCE_PROFILES.city, + cluster: { + strategy: "stable-spherical", + }, }); const bgpCollectorIconLayer = createInteractableLayer({ @@ -402,6 +405,9 @@ const bgpCollectorIconLayer = createInteractableLayer({ }; }, avoidance: SURFACE_AVOIDANCE_PROFILES.city, + cluster: { + strategy: "stable-spherical", + }, }); function clamp(value, min, max) { @@ -1332,7 +1338,7 @@ function applyCollectorCounts() { async function fetchGeoJSONWithTimeout(url, timeoutMs, warningMessage, fallbackPayload) { try { - const response = await fetch(url, { signal: AbortSignal.timeout(timeoutMs) }); + const response = await fetch(url, { cache: "no-store", signal: AbortSignal.timeout(timeoutMs) }); if (!response.ok) { throw new Error(`HTTP ${response.status}`); } @@ -1368,7 +1374,7 @@ function selectBGPEventFeatures(incidentPayload, anomalyPayload) { } export async function loadBGPAnomalies(scene, earth) { - const collectorsResponse = await fetch(PATHS.bgpCollectorsApi); + const collectorsResponse = await fetch(PATHS.bgpCollectorsApi, { cache: "no-store" }); if (!collectorsResponse.ok) { throw new Error(`BGP collectors HTTP ${collectorsResponse.status}`); } diff --git a/frontend/public/earth/js/brand.js b/frontend/public/earth/js/brand.js index a955e26a..0391811a 100644 --- a/frontend/public/earth/js/brand.js +++ b/frontend/public/earth/js/brand.js @@ -17,11 +17,25 @@ const BRANDS = { logoSrc: "/earth/assets/brand/earth-logo.png", titleSrc: "/earth/assets/brand/title-en.png", titleText: "Intelligent Planet Program", - subtitle: "Physical-Universe Holography", - description: "Satellites · Cables · Compute Infra", + subtitle: "Reality Layer Situational Awareness System", + description: "Satellites · Subsea Cables · Compute Infrastructure", }, }; +const DEFAULT_BRAND_TITLE_BY_VARIANT = { + zh: BRANDS.zh.titleSrc, + en: BRANDS.en.titleSrc, +}; + +const LOCALIZED_FIELD_NAMES = { + ariaLabel: ["aria_label", "ariaLabel"], + titleAlt: ["title_alt", "titleAlt"], + titleSrc: ["title_src", "titleSrc"], + titleText: ["title_text", "titleText"], + subtitle: ["subtitle"], + description: ["description"], +}; + export function getDefaultBrandConfig(variant = DEFAULT_BRAND_LANGUAGE) { return BRANDS[variant] ?? BRANDS[DEFAULT_BRAND_LANGUAGE]; } @@ -35,18 +49,65 @@ function escapeHtml(value = "") { .replace(/'/g, "'"); } +function hasCjkText(value = "") { + return /[\u3400-\u9fff]/.test(String(value ?? "")); +} + +function readConfigValue(config, keys = []) { + for (const key of keys) { + if (config[key] !== undefined && config[key] !== null && config[key] !== "") { + return config[key]; + } + } + return undefined; +} + +function readLocalizedConfigValue(config, fieldName, variant, fallback) { + const keys = LOCALIZED_FIELD_NAMES[fieldName] || [fieldName]; + const localeSuffix = variant === "en" ? "en" : "zh"; + const localeKeys = keys.flatMap((key) => [ + `${key}_${localeSuffix}`, + `${key}${localeSuffix.charAt(0).toUpperCase()}${localeSuffix.slice(1)}`, + ]); + const localized = readConfigValue(config, localeKeys); + if (localized !== undefined) return localized; + const generic = readConfigValue(config, keys); + return generic ?? fallback; +} + function normalizeBrandConfig(config = {}, variant = DEFAULT_BRAND_LANGUAGE) { const defaults = getDefaultBrandConfig(variant); + const sourceTitleSrc = readLocalizedConfigValue(config, "titleSrc", variant, undefined); const normalized = { ...defaults, ...config, - ariaLabel: config.aria_label ?? config.ariaLabel ?? defaults.ariaLabel, - titleAlt: config.title_alt ?? config.titleAlt ?? defaults.titleAlt, + ariaLabel: readLocalizedConfigValue(config, "ariaLabel", variant, defaults.ariaLabel), + titleAlt: readLocalizedConfigValue(config, "titleAlt", variant, defaults.titleAlt), logoSrc: config.logo_src ?? config.logoSrc ?? defaults.logoSrc, - titleSrc: config.title_src ?? config.titleSrc ?? defaults.titleSrc, - titleText: config.title_text ?? config.titleText ?? defaults.titleText, + titleSrc: sourceTitleSrc ?? defaults.titleSrc, + titleText: readLocalizedConfigValue(config, "titleText", variant, defaults.titleText), + subtitle: readLocalizedConfigValue(config, "subtitle", variant, defaults.subtitle), + description: readLocalizedConfigValue(config, "description", variant, defaults.description), }; + if ( + variant === "en" && + ( + !sourceTitleSrc || + sourceTitleSrc === DEFAULT_BRAND_TITLE_BY_VARIANT.zh || + hasCjkText(normalized.titleText) || + hasCjkText(normalized.titleAlt) || + hasCjkText(normalized.ariaLabel) + ) + ) { + normalized.titleSrc = DEFAULT_BRAND_TITLE_BY_VARIANT.en; + normalized.titleAlt = defaults.titleAlt; + normalized.titleText = defaults.titleText; + normalized.ariaLabel = defaults.ariaLabel; + normalized.subtitle = defaults.subtitle; + normalized.description = defaults.description; + } + if (!normalized.titleText) normalized.titleText = defaults.titleText; if (!normalized.ariaLabel) normalized.ariaLabel = normalized.titleText; if (!normalized.titleAlt) normalized.titleAlt = normalized.titleText; diff --git a/frontend/public/earth/js/cables.js b/frontend/public/earth/js/cables.js index 83364d10..9a0154ad 100644 --- a/frontend/public/earth/js/cables.js +++ b/frontend/public/earth/js/cables.js @@ -13,10 +13,13 @@ import { getSurfaceMarkerCameraScale, latLonToVector3 } from "./utils.js"; import { setEarthStatValue, updateEarthStats, showStatusMessage } from "./ui.js"; import { showInfoCard } from "./info-card.js"; import { setLegendItems, setLegendMode } from "./legend.js"; +import { earthMessage } from "./i18n.js"; export let cableLines = []; export let landingPoints = []; export let lockedCable = null; +let cableSourceFeatureCount = 0; +let landingPointSourceFeatureCount = 0; let cableIdMap = new Map(); let cableStates = new Map(); let cablesVisible = true; @@ -302,6 +305,7 @@ function calculateGreatCirclePoints( export function clearCableLines(earthObj = null) { cableLines.forEach((line) => disposeObject(line, earthObj)); cableLines = []; + cableSourceFeatureCount = 0; cableIdMap = new Map(); cableStates.clear(); } @@ -309,6 +313,7 @@ export function clearCableLines(earthObj = null) { export function clearLandingPoints(earthObj = null) { landingPoints.forEach((point) => disposeObject(point, earthObj)); landingPoints = []; + landingPointSourceFeatureCount = 0; } export function clearCableData(earthObj = null) { @@ -319,12 +324,11 @@ export function clearCableData(earthObj = null) { export async function loadGeoJSONFromPath(scene, earthObj, options = {}) { const { silent = false } = options; - console.log("正在加载电缆数据..."); if (!silent) { - showStatusMessage("正在加载电缆数据...", "warning"); + showStatusMessage(earthMessage("loading.cableData"), "warning"); } - const response = await fetch(PATHS.cablesApi); + const response = await fetch(PATHS.cablesApi, { cache: "no-store" }); if (!response.ok) { throw new Error(`电缆接口返回 HTTP ${response.status}`); } @@ -335,6 +339,7 @@ export async function loadGeoJSONFromPath(scene, earthObj, options = {}) { } clearCableLines(earthObj); + cableSourceFeatureCount = data.features.length; for (const feature of data.features) { const geometry = feature.geometry; @@ -419,16 +424,15 @@ export async function loadGeoJSONFromPath(scene, earthObj, options = {}) { }); if (!silent) { - showStatusMessage(`成功加载 ${cableLines.length} 条电缆`, "success"); + showStatusMessage(earthMessage("status.loadedCables", { count: cableLines.length }), "success"); } return cableLines.length; } export async function loadLandingPoints(scene, earthObj, options = {}) { const { silent = false } = options; - console.log("正在加载登陆点数据..."); - const response = await fetch(PATHS.landingPointsApi); + const response = await fetch(PATHS.landingPointsApi, { cache: "no-store" }); if (!response.ok) { throw new Error(`登陆点接口返回 HTTP ${response.status}`); } @@ -439,6 +443,7 @@ export async function loadLandingPoints(scene, earthObj, options = {}) { } clearLandingPoints(earthObj); + landingPointSourceFeatureCount = data.features.length; const markerTexture = await getLandingPointTexture(); @@ -508,7 +513,7 @@ export async function loadLandingPoints(scene, earthObj, options = {}) { setEarthStatValue("landing-point-count", `${validCount}个`); if (!silent) { - showStatusMessage(`成功加载 ${validCount} 个登陆点`, "success"); + showStatusMessage(earthMessage("status.loadedLandingPoints", { count: validCount }), "success"); } return validCount; } @@ -528,7 +533,7 @@ export function handleCableClick(cable) { rfs: data.rfs, }); - showStatusMessage(`已锁定: ${data.name}`, "info"); + showStatusMessage(earthMessage("status.locked", { name: data.name }), "info"); } export function clearCableSelection() { @@ -586,6 +591,10 @@ export function getLandingPoints() { return landingPoints; } +export function getCableSourceRecordCount() { + return cableSourceFeatureCount + landingPointSourceFeatureCount; +} + export function getCableState(cableId) { return cableStates.get(cableId) || CABLE_STATE.NORMAL; } diff --git a/frontend/public/earth/js/client-logs.js b/frontend/public/earth/js/client-logs.js index d6f5a122..6babbe5e 100644 --- a/frontend/public/earth/js/client-logs.js +++ b/frontend/public/earth/js/client-logs.js @@ -2,6 +2,8 @@ import { PATHS } from "./constants.js"; const RECENT_EVENT_TTL_MS = 15_000; const recentEventMap = new Map(); +const pendingEventMap = new Map(); +let pendingFlushTimer = null; function normalizeErrorDetail(detail) { if (!detail) return ""; @@ -22,6 +24,37 @@ function dedupeKey(level, message, detail, category) { return `${level}::${category || ""}::${message}::${detail}`; } +function normalizeFingerprintText(value) { + return String(value || "") + .replace(/[?&](m|t|token|expires|signature|X-Amz-[^=]+)=[^&\s]+/gi, "") + .replace(/(index|chunk|segment)[_-]?\d+(_\d+)?\.(ts|m4s|vtt)/gi, "") + .replace(/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi, "") + .replace(/\bconn_[A-Za-z0-9:._-]+\b/g, "") + .replace(/\b\d{5,}\b/g, "") + .replace(/\s+/g, " ") + .trim(); +} + +function hashFingerprint(value) { + let hash = 5381; + for (let index = 0; index < value.length; index += 1) { + hash = ((hash << 5) + hash) ^ value.charCodeAt(index); + } + return `client-${(hash >>> 0).toString(16).padStart(8, "0")}`; +} + +function buildFingerprint(level, message, detail, category, module) { + return hashFingerprint( + [ + normalizeFingerprintText(level), + normalizeFingerprintText(category), + normalizeFingerprintText(module), + normalizeFingerprintText(message), + normalizeFingerprintText(detail), + ].join("|"), + ); +} + function shouldSkip(level, message, detail, category) { const key = dedupeKey(level, message, detail, category); const now = Date.now(); @@ -37,33 +70,14 @@ function shouldSkip(level, message, detail, category) { return lastSeenAt && now - lastSeenAt < RECENT_EVENT_TTL_MS; } -export async function reportEarthClientLog({ - level = "error", - message, - category = "runtime", - module = "earth", - detail = "", -}) { - if (!message) return; - const normalizedDetail = normalizeErrorDetail(detail); - if (shouldSkip(level, message, normalizedDetail, category)) { - return; - } - +async function sendEarthClientLog(payload) { try { await fetch(PATHS.earthClientLogsApi, { method: "POST", headers: { "Content-Type": "application/json", }, - body: JSON.stringify({ - level, - message, - category, - module, - url: window.location.href, - detail: normalizedDetail.slice(0, 4000), - }), + body: JSON.stringify(payload), keepalive: true, }); } catch { @@ -71,6 +85,61 @@ export async function reportEarthClientLog({ } } +function scheduleFlush() { + if (pendingFlushTimer) return; + pendingFlushTimer = window.setTimeout(() => { + pendingFlushTimer = null; + const pending = Array.from(pendingEventMap.values()); + pendingEventMap.clear(); + pending.forEach((entry) => { + void sendEarthClientLog({ + level: entry.level, + message: entry.message, + category: entry.category, + module: entry.module, + url: window.location.href, + detail: entry.detail.slice(0, 4000), + fingerprint: entry.fingerprint, + occurrence_count: entry.occurrenceCount, + metadata: entry.metadata, + }); + }); + }, 1000); +} + +export function reportEarthClientLog({ + level = "error", + message, + category = "runtime", + module = "earth", + detail = "", + metadata = {}, +}) { + if (!message) return; + const normalizedDetail = normalizeErrorDetail(detail); + const fingerprint = buildFingerprint(level, message, normalizedDetail, category, module); + const key = dedupeKey(level, message, normalizedDetail, category); + const existing = pendingEventMap.get(key); + if (existing) { + existing.occurrenceCount += 1; + existing.metadata = { ...existing.metadata, ...metadata }; + } else { + pendingEventMap.set(key, { + level, + message, + category, + module, + detail: normalizedDetail, + fingerprint, + occurrenceCount: 1, + metadata, + }); + } + + shouldSkip(level, message, normalizedDetail, category); + scheduleFlush(); +} + export function registerEarthClientErrorHandlers() { window.addEventListener("error", (event) => { console.error("全局错误:", event.error); diff --git a/frontend/public/earth/js/compute-centers.js b/frontend/public/earth/js/compute-centers.js index 118ebfc3..60d3c69f 100644 --- a/frontend/public/earth/js/compute-centers.js +++ b/frontend/public/earth/js/compute-centers.js @@ -27,6 +27,8 @@ let previewRingB = null; let previewRingPulseOffset = 0; const COLLECT_LOCATION_API_BASE = "/api/v1/visualization/compute-centers"; +const LOCATION_CAPABILITY_API = "/api/v1/visualization/compute-centers/location-capability"; +let computeCenterLocationCapabilityPromise = null; function getPreviewRingTexture() { if (previewRingTexture) return previewRingTexture; @@ -212,7 +214,6 @@ const computeCenterIconLayer = createInteractableLayer({ }, icon: { coordinates: "canvas", - colorable: false, fitSize: COMPUTE_CENTER_ICON_FIT_SIZE, glowBlur: 16, getSource({ marker, item }) { @@ -246,6 +247,9 @@ const computeCenterIconLayer = createInteractableLayer({ pulseOffset: Math.random() * Math.PI * 2, }), avoidance: SURFACE_AVOIDANCE_PROFILES.city, + cluster: { + strategy: "stable-spherical", + }, }); export function formatComputeCenterTypeLabel(siteType) { @@ -520,13 +524,48 @@ export async function collectLocationCandidates(endpoint, payload = {}) { }); if (!response.ok) { const text = await response.text().catch(() => ""); + let detail = ""; + try { + const payload = JSON.parse(text); + detail = payload?.detail?.reason || payload?.detail?.message || payload?.detail || ""; + } catch { + detail = text; + } throw new Error( - `Collect location failed: HTTP ${response.status} ${text}`.trim(), + `Collect location failed: HTTP ${response.status} ${detail}`.trim(), ); } return response.json(); } +export async function getComputeCenterLocationCapability({ force = false } = {}) { + if (!computeCenterLocationCapabilityPromise || force) { + computeCenterLocationCapabilityPromise = fetch(LOCATION_CAPABILITY_API, { cache: "no-store" }) + .then(async (response) => { + if (!response.ok) { + const text = await response.text().catch(() => ""); + return { + enabled: false, + provider: null, + reason: `WebSearch 状态检查失败：HTTP ${response.status} ${text}`.trim(), + }; + } + const payload = await response.json(); + return { + enabled: payload?.enabled === true, + provider: payload?.provider || "", + reason: payload?.reason || "", + }; + }) + .catch((error) => ({ + enabled: false, + provider: null, + reason: `WebSearch 状态检查失败：${error?.message || error}`, + })); + } + return computeCenterLocationCapabilityPromise; +} + export async function collectComputeCenterLocation(sourceId, context = {}) { if (!sourceId) { throw new Error("sourceId is required"); diff --git a/frontend/public/earth/js/constants.js b/frontend/public/earth/js/constants.js index 22657846..94190a34 100644 --- a/frontend/public/earth/js/constants.js +++ b/frontend/public/earth/js/constants.js @@ -64,6 +64,8 @@ export const SURFACE_HOVER_INFO_MODES = { export const DEFAULT_SURFACE_HOVER_INFO_MODE = SURFACE_HOVER_INFO_MODES.FULL; +export const EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48; + export const CRUISE_CONFIG = { dwellMs: 7_000, focusDurationMs: 1_400, @@ -233,8 +235,12 @@ export const COUNTRY_BOUNDARY_CONFIG = { tilePrefetchRing: 1, tileDebounceMs: 180, tileCacheLimit: 150, - lineAltitudeOffset: 0.115, - hoverAltitudeOffset: 0.115, + // Keep boundary/coastline strokes on the same shell as the high-res earth + // texture overlay. A lower or higher radius creates visible parallax while + // the globe rotates. + lineAltitudeOffset: EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET, + hoverAltitudeOffset: EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET, + claimLineAltitudeOffset: 0, hoverMissStickyMs: 160, lineColor: 0x7fc7ff, lineOpacity: 0.58, @@ -262,6 +268,7 @@ export const PATHS = { cablesApi: '/api/v1/visualization/geo/cables', landingPointsApi: '/api/v1/visualization/geo/landing-points', computeCentersApi: '/api/v1/visualization/geo/compute-centers', + interactablesApi: '/api/v1/interactables/geojson', vesselsApi: '/api/v1/vessels/snapshot', vesselTrackApi: (mmsi) => `/api/v1/visualization/vessels/${encodeURIComponent(mmsi)}/track`, bgpApi: '/api/v1/visualization/geo/bgp-anomalies', @@ -273,7 +280,7 @@ export const PATHS = { export const VESSEL_CONFIG = { altitudeOffset: 0.2, - maxRenderedMarkers: 0, + maxRenderedMarkers: 3000, marker: { baseScale: 7.5, baseOpacity: 0.88, @@ -539,7 +546,7 @@ export const EARTH_MATERIAL_CONFIG = { shininess: 12, emissive: 0x010609, opacity: 1, - textureOverlayAltitudeOffset: 0.48, + textureOverlayAltitudeOffset: EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET, textureOverlayOpacity: 0.88, textureOverlayRenderOrder: 0.96, textureOverlaySpecular: 0x05080d, diff --git a/frontend/public/earth/js/controls.js b/frontend/public/earth/js/controls.js index b01fb424..a54f7f90 100644 --- a/frontend/public/earth/js/controls.js +++ b/frontend/public/earth/js/controls.js @@ -109,8 +109,10 @@ import { setLayerButtonState, updateLayerButtonState, } from "./layer-button-state.js"; +import { earthMessage, translateText } from "./i18n.js"; import { DEFAULT_MOTION_PROVIDER, + MOTION_GESTURES, normalizeMotionProvider, } from "./motion-protocol.js"; @@ -125,6 +127,7 @@ let autoRotationSpeed = CONFIG.rotationSpeed; let motionDebugEnabled = false; let motionProvider = DEFAULT_MOTION_PROVIDER; let motionDebugSkeletonOnly = false; +let motionEnabledGestures = []; let activeCamera = null; let settingsApplyPromise = Promise.resolve(); let boundaryBuildPollTimer = null; @@ -159,7 +162,7 @@ const SETTINGS_SHEET_MAX_SCALE_X = 0.22; const SETTINGS_SHEET_MAX_SCALE_Y = 0.18; const EARTH_SETTINGS_STORAGE_KEY = "planet.earth.settings.v2"; const LEGACY_EARTH_SETTINGS_STORAGE_KEY = "planet.earth.settings.v1"; -const EARTH_SETTINGS_VERSION = 15; +const EARTH_SETTINGS_VERSION = 17; const GRID_LINES_DEFAULT_VERSION = 3; const SATELLITE_DISPLAY_DEFAULT_VERSION = 4; const MEDIA_PANEL_DEFAULT_VERSION = 5; @@ -172,6 +175,37 @@ const SURFACE_HOVER_INFO_DEFAULT_VERSION = 11; const KEYBOARD_SHORTCUTS_DEFAULT_VERSION = 13; const CRUISE_QUEUE_DEFAULT_VERSION = 14; const AUTO_ROTATION_SPEED_DEFAULT_VERSION = 15; +const NEWS_CATEGORY_FILTERS_DEFAULT_VERSION = 16; +const MOTION_GESTURES_DEFAULT_VERSION = 17; +const MOTION_GESTURE_DEFINITIONS = [ + { id: "rotate_left", label: "左旋" }, + { id: "rotate_right", label: "右旋" }, + { id: "rotate_up", label: "上旋" }, + { id: "rotate_down", label: "下旋" }, + { id: "zoom_in", label: "放大" }, + { id: "zoom_out", label: "缩小" }, + { id: "focus_prev", label: "上个焦点" }, + { id: "focus_next", label: "下个焦点" }, + { id: "layer_prev", label: "上一图层" }, + { id: "layer_next", label: "下一图层" }, + { id: "confirm", label: "确认" }, +]; +const DEFAULT_MOTION_ENABLED_GESTURES = MOTION_GESTURE_DEFINITIONS.map((item) => item.id); +motionEnabledGestures = [...DEFAULT_MOTION_ENABLED_GESTURES]; +const DEFAULT_NEWS_CATEGORY_FILTERS = { + politics: true, + business: true, + ecommerce: true, + finance: true, + sports: true, + technology: true, + military: true, + disaster: true, + energy: true, + society: true, + culture: true, + other: true, +}; const AUTO_ROTATION_SPEED_MIN = 0.0001; const AUTO_ROTATION_SPEED_MAX = 0.0015; const AUTO_ROTATION_SPEED_STEP = 0.00005; @@ -185,6 +219,11 @@ const KEYBOARD_ROTATION_STOP_SPEED = 0.012; const KEYBOARD_ZOOM_STEP = 0.1; const WHEEL_ZOOM_STEP = 0.1; const WHEEL_ZOOM_DURATION_MS = 180; +const WHEEL_TRACKPAD_PIXEL_THRESHOLD = 48; +const WHEEL_TRACKPAD_DEADZONE = 0.35; +const WHEEL_TRACKPAD_RESIDUAL_WINDOW_MS = 140; +const WHEEL_TRACKPAD_RESIDUAL_RATIO = 0.65; +const WHEEL_TRACKPAD_SENSITIVITY = 0.0024; const TARGET_SWITCH_ZOOM_IN_PHASE = 0.28; const TARGET_SWITCH_ROTATE_PHASE = 0.5; let settingsModalTimer = null; @@ -390,12 +429,12 @@ function getShortcutForAction(actionId) { function getAutoRotateShortcutStatusMessage(isActive) { if (rotationMode === ROTATION_MODE.CRUISE) { - return isActive ? "巡航已恢复" : "巡航已暂停"; + return earthMessage("status.runtimePaused", { label: "巡航", active: isActive }); } if (rotationMode === ROTATION_MODE.MOTION) { - return isActive ? "动捕已恢复" : "动捕已暂停"; + return earthMessage("status.runtimePaused", { label: "动捕", active: isActive }); } - return isActive ? "旋转已恢复" : "旋转已暂停"; + return earthMessage("status.runtimePaused", { label: "旋转", active: isActive }); } function getShortcutOwnerByBinding(binding, { excludeActionId = null } = {}) { @@ -655,7 +694,7 @@ function stopKeyboardRotationControl({ actionId = null, restoreAutoRotate = fals } function applyKeyboardZoom(direction) { - setZoomLevel(zoomLevel + direction * KEYBOARD_ZOOM_STEP, activeCamera); + setZoomLevel(getZoomLevelFromCamera(activeCamera) + direction * KEYBOARD_ZOOM_STEP, activeCamera); showZoomStatusCapsule({ force: true }); } @@ -693,7 +732,7 @@ function toggleLayoutExpandedFromShortcut() { const container = document.getElementById("container"); if (!(container instanceof HTMLElement)) return; const expanded = toggleLayoutExpanded(container); - showStatusMessage(expanded ? "布局已最大化" : "布局已恢复", "info"); + showStatusMessage(earthMessage("status.layoutExpanded", { expanded }), "info"); } async function toggleLayerFromShortcut(layerId) { @@ -701,11 +740,17 @@ async function toggleLayerFromShortcut(layerId) { if (!definition) return; const button = getLayerButton(layerId); if (button?.disabled || button?.classList.contains("is-disabled")) { - showStatusMessage(`${definition.label}当前不可用`, "warning"); + showStatusMessage(earthMessage("status.layerUnavailable", { layer: definition.label }), "warning"); return; } await definition.setVisible(!definition.getVisible()); - showStatusMessage(`${definition.label}${definition.getVisible() ? "已显示" : "已隐藏"}`, "info"); + showStatusMessage( + earthMessage("status.layerVisibility", { + layer: definition.label, + visible: definition.getVisible(), + }), + "info", + ); } function executeKeyboardShortcut(actionId, event = null) { @@ -1318,6 +1363,19 @@ function clampEarthZoomLevel(nextZoom) { return Math.min(CONFIG.maxZoom, Math.max(CONFIG.minZoom, parsedZoom)); } +function getZoomLevelFromCamera(camera = activeCamera) { + const cameraZ = Number(camera?.position?.z); + if (!Number.isFinite(cameraZ) || cameraZ <= 0) { + return clampEarthZoomLevel(zoomLevel); + } + return clampEarthZoomLevel(CONFIG.defaultCameraZ / cameraZ); +} + +function syncZoomLevelFromCamera(camera = activeCamera) { + zoomLevel = getZoomLevelFromCamera(camera); + return zoomLevel; +} + function formatZoomPercent(zoom) { return `${Math.round(zoom * 100)}%`; } @@ -1327,7 +1385,7 @@ function getZoomResetTooltipText(zoom) { } function getZoomResetStatusMessage(zoom) { - return `缩放已重置到${formatZoomPercent(zoom)}`; + return earthMessage("status.zoomReset", { zoom: formatZoomPercent(zoom) }); } function normalizeAutoRotationSpeed(value) { @@ -1349,6 +1407,23 @@ function canUseLocalStorage() { } } +function normalizeNewsCategoryFilters(filters) { + const normalized = { ...DEFAULT_NEWS_CATEGORY_FILTERS }; + if (!filters || typeof filters !== "object") return normalized; + Object.keys(DEFAULT_NEWS_CATEGORY_FILTERS).forEach((category) => { + if (typeof filters[category] === "boolean") { + normalized[category] = filters[category]; + } + }); + return normalized; +} + +function isNewsCategoryFilterEnabled(filters, category) { + const key = String(category || "").trim(); + if (!(key in DEFAULT_NEWS_CATEGORY_FILTERS)) return true; + return normalizeNewsCategoryFilters(filters)[key] !== false; +} + function getCurrentPanelVisibilitySnapshot() { return Object.fromEntries( HUD_PANEL_IDS.map((panelId) => { @@ -1377,11 +1452,15 @@ function getCurrentSharedSettingsSnapshot() { motionDebugEnabled, motionProvider, motionDebugSkeletonOnly, + motionEnabledGestures: getMotionEnabledGestures(), mediaPanelActiveTab: normalizeMediaPanelActiveTab(getActiveTVTab()), satelliteIdleBreathingEnabled: getSatelliteIdleBreathingEnabled(), satelliteRealAltitudeEnabled: getSatelliteRealAltitudeEnabled(), interactableCompactDotsEnabled: getInteractableCompactDotsEnabled(), surfaceHoverInfoMode: getSurfaceHoverInfoMode(), + newsCategoryFilters: normalizeNewsCategoryFilters( + earthSettingsState?.shared?.newsCategoryFilters, + ), keyboardShortcuts: normalizeKeyboardShortcuts(keyboardShortcuts), }; } @@ -1436,6 +1515,7 @@ function cloneEarthSettings(settings) { DEFAULT_MOTION_PROVIDER, ), motionDebugSkeletonOnly: Boolean(settings.shared.motionDebugSkeletonOnly), + motionEnabledGestures: normalizeMotionEnabledGestures(settings.shared.motionEnabledGestures), mediaPanelActiveTab: normalizeMediaPanelActiveTab(settings.shared.mediaPanelActiveTab), satelliteIdleBreathingEnabled: settings.shared.satelliteIdleBreathingEnabled !== false, @@ -1446,6 +1526,7 @@ function cloneEarthSettings(settings) { surfaceHoverInfoMode: normalizeSurfaceHoverInfoMode( settings.shared.surfaceHoverInfoMode, ), + newsCategoryFilters: normalizeNewsCategoryFilters(settings.shared.newsCategoryFilters), keyboardShortcuts: normalizeKeyboardShortcuts(settings.shared.keyboardShortcuts), layerVisibility: { ...(settings.shared.layerVisibility || {}) }, }, @@ -1567,6 +1648,10 @@ function normalizeEarthSettings(rawSettings, defaults) { typeof sharedSettings?.motionDebugSkeletonOnly === "boolean" ? sharedSettings.motionDebugSkeletonOnly : defaults.shared.motionDebugSkeletonOnly; + const nextMotionEnabledGestures = + (rawSettings?.version || 0) >= MOTION_GESTURES_DEFAULT_VERSION + ? normalizeMotionEnabledGestures(sharedSettings?.motionEnabledGestures) + : normalizeMotionEnabledGestures(defaults.shared.motionEnabledGestures); const nextMediaPanelActiveTab = (rawSettings?.version || 0) >= MEDIA_PANEL_ACTIVE_TAB_DEFAULT_VERSION ? normalizeMediaPanelActiveTab(sharedSettings?.mediaPanelActiveTab) @@ -1593,6 +1678,13 @@ function normalizeEarthSettings(rawSettings, defaults) { (rawSettings?.version || 0) >= KEYBOARD_SHORTCUTS_DEFAULT_VERSION ? normalizeKeyboardShortcuts(sharedSettings?.keyboardShortcuts) : defaults.shared.keyboardShortcuts; + const legacyNewsCategoryFilters = sharedSettings?.["display" + "Types"]?.news; + const nextNewsCategoryFilters = + (rawSettings?.version || 0) >= NEWS_CATEGORY_FILTERS_DEFAULT_VERSION + ? normalizeNewsCategoryFilters( + sharedSettings?.newsCategoryFilters || legacyNewsCategoryFilters, + ) + : normalizeNewsCategoryFilters(defaults.shared.newsCategoryFilters); const nextCruiseQueueMode = (rawSettings?.version || 0) >= CRUISE_QUEUE_DEFAULT_VERSION ? normalizeCruiseQueueMode(sharedSettings?.cruiseQueueMode) @@ -1628,11 +1720,13 @@ function normalizeEarthSettings(rawSettings, defaults) { motionDebugEnabled: nextMotionDebugEnabled, motionProvider: nextMotionProvider, motionDebugSkeletonOnly: nextMotionDebugSkeletonOnly, + motionEnabledGestures: nextMotionEnabledGestures, mediaPanelActiveTab: nextMediaPanelActiveTab, satelliteIdleBreathingEnabled: nextSatelliteIdleBreathingEnabled, satelliteRealAltitudeEnabled: nextSatelliteRealAltitudeEnabled, interactableCompactDotsEnabled: nextInteractableCompactDotsEnabled, surfaceHoverInfoMode: nextSurfaceHoverInfoMode, + newsCategoryFilters: nextNewsCategoryFilters, keyboardShortcuts: nextKeyboardShortcuts, }, views: { @@ -1684,6 +1778,17 @@ function syncMotionDebugSkeletonOnlyToggle(nextEnabled = motionDebugSkeletonOnly }); } +function syncMotionGestureControls() { + const enabledGestures = new Set(getMotionEnabledGestures()); + document.querySelectorAll("[data-motion-gesture-toggle]").forEach((button) => { + if (!(button instanceof HTMLButtonElement)) return; + const gesture = button.dataset.motionGestureToggle || ""; + const active = enabledGestures.has(gesture); + button.classList.toggle("is-active", active); + button.setAttribute("aria-pressed", active ? "true" : "false"); + }); +} + function dispatchMotionSettingsChange() { const effectiveDebugEnabled = rotationMode === ROTATION_MODE.MOTION && autoRotate && motionDebugEnabled; @@ -1694,6 +1799,7 @@ function dispatchMotionSettingsChange() { preferredEnabled: motionDebugEnabled, provider: motionProvider, skeletonOnly: motionDebugSkeletonOnly, + enabledGestures: getMotionEnabledGestures(), }, }), ); @@ -1804,6 +1910,18 @@ function normalizeCruiseModules(nextModules) { : [...DEFAULT_CRUISE_MODULES]; } +function normalizeMotionEnabledGestures(nextGestures) { + const sourceGestures = Array.isArray(nextGestures) + ? nextGestures + : DEFAULT_MOTION_ENABLED_GESTURES; + const normalizedGestures = Array.from( + new Set(sourceGestures.filter((gesture) => MOTION_GESTURES.has(gesture))), + ); + return normalizedGestures.length > 0 + ? normalizedGestures + : [...DEFAULT_MOTION_ENABLED_GESTURES]; +} + function normalizeCruiseQueueMode(mode) { return ALLOWED_CRUISE_QUEUE_MODES.has(mode) ? mode : DEFAULT_CRUISE_QUEUE_MODE; } @@ -1919,6 +2037,71 @@ function syncInteractableCompactDotsToggle() { }); } +function syncNewsCategoryFilterControls() { + const filters = normalizeNewsCategoryFilters( + earthSettingsState?.shared?.newsCategoryFilters, + ); + document.querySelectorAll("[data-news-category-toggle]").forEach((button) => { + if (!(button instanceof HTMLButtonElement)) return; + const category = button.dataset.newsCategoryToggle || ""; + const active = isNewsCategoryFilterEnabled(filters, category); + button.classList.toggle("is-active", active); + button.setAttribute("aria-pressed", active ? "true" : "false"); + }); +} + +function dispatchNewsCategoryFiltersChange( + filters = earthSettingsState?.shared?.newsCategoryFilters, +) { + window.dispatchEvent( + new CustomEvent("earth:news-category-filters-change", { + detail: { + categories: normalizeNewsCategoryFilters(filters), + }, + }), + ); +} + +function applyNewsCategoryFilters(filters = earthSettingsState?.shared?.newsCategoryFilters) { + const normalized = normalizeNewsCategoryFilters(filters); + syncNewsCategoryFilterControls(); + dispatchNewsCategoryFiltersChange(normalized); +} + +export function getEarthNewsCategoryFilters() { + return normalizeNewsCategoryFilters(earthSettingsState?.shared?.newsCategoryFilters); +} + +export function isEarthNewsCategoryEnabled(category) { + return isNewsCategoryFilterEnabled( + earthSettingsState?.shared?.newsCategoryFilters, + category, + ); +} + +export function setEarthNewsCategoryEnabled( + category, + enabled, + { persist = true, suppressStatus = false } = {}, +) { + const key = String(category || "").trim(); + if (!(key in DEFAULT_NEWS_CATEGORY_FILTERS)) return false; + + ensureMutableEarthSettingsState(); + const nextFilters = normalizeNewsCategoryFilters(earthSettingsState.shared.newsCategoryFilters); + nextFilters[key] = Boolean(enabled); + earthSettingsState.shared.newsCategoryFilters = nextFilters; + applyNewsCategoryFilters(nextFilters); + + if (persist) { + persistEarthSettings(); + } + if (!suppressStatus) { + showStatusMessage(earthMessage("status.layerVisibility", { layer: "新闻类型", visible: Boolean(enabled) }), "info"); + } + return true; +} + function syncSurfaceHoverInfoModeControls() { const activeMode = getSurfaceHoverInfoMode(); document.querySelectorAll("[data-surface-hover-info-mode]").forEach((button) => { @@ -1971,7 +2154,13 @@ export function setCruiseModules(nextModules, { persist = true, suppressStatus = if (!suppressStatus) { const labels = normalizedModules.map((moduleId) => CRUISE_MODULE_LABELS[moduleId] || moduleId); - showStatusMessage(`巡航模块已切换为：${labels.join(" + ")}`, "info"); + showStatusMessage( + earthMessage("status.modulesChanged", { + label: "巡航模块", + value: labels.map((label) => translateText(label)).join(" + "), + }), + "info", + ); } return normalizedModules; @@ -2002,7 +2191,7 @@ export function setCruiseQueueMode( : normalizedMode === CRUISE_QUEUE_MODES.RANDOM ? "随机" : "默认"; - showStatusMessage(`巡航队列已切换为：${label}`, "info"); + showStatusMessage(earthMessage("status.valueChanged", { label: "巡航队列", value: label }), "info"); } return normalizedMode; @@ -2027,7 +2216,7 @@ export function setCruiseRegionOrder( if (persist) persistEarthSettings(); if (!suppressStatus) { - showStatusMessage("巡航大区顺序已更新", "info"); + showStatusMessage(earthMessage("status.updated", { label: "巡航大区顺序" }), "info"); } return normalizedOrder; @@ -2061,7 +2250,7 @@ export function setSatelliteDisplayStyle( normalizedStyle === SATELLITE_DISPLAY_STYLES.GROUND_FOOTPRINT ? "真实地表覆盖" : "自身发光"; - showStatusMessage(`卫星显示风格已切换为：${nextLabel}`, "info"); + showStatusMessage(earthMessage("status.valueChanged", { label: "卫星显示风格", value: nextLabel }), "info"); } return normalizedStyle; @@ -2080,7 +2269,7 @@ export function setSatelliteIdleBreathingEnabled( persistEarthSettings(); } if (!suppressStatus) { - showStatusMessage(enabled ? "卫星呼吸闪烁已开启" : "卫星呼吸闪烁已关闭", "info"); + showStatusMessage(earthMessage("status.booleanSetting", { label: "卫星呼吸闪烁", enabled }), "info"); } return enabled; } @@ -2099,7 +2288,9 @@ export function setSatelliteRealAltitudeEnabled( } if (!suppressStatus) { showStatusMessage( - enabled ? "卫星真实高度已开启" : "卫星已切换为旧版同层高度", + enabled + ? earthMessage("status.booleanSetting", { label: "卫星真实高度", enabled }) + : earthMessage("status.valueChanged", { label: "卫星", value: "旧版同层高度" }), "info", ); } @@ -2119,7 +2310,7 @@ export function setInteractableCompactDotsEnabled( persistEarthSettings(); } if (!suppressStatus) { - showStatusMessage(enabled ? "低缩放彩色圆点已开启" : "低缩放彩色圆点已关闭", "info"); + showStatusMessage(earthMessage("status.booleanSetting", { label: "低缩放彩色圆点", enabled }), "info"); } return enabled; } @@ -2154,7 +2345,7 @@ export function setSurfaceHoverInfoMode( : normalizedMode === SURFACE_HOVER_INFO_MODES.POSITION ? "位置" : "完整"; - showStatusMessage(`悬停提示已切换为：${label}`, "info"); + showStatusMessage(earthMessage("status.valueChanged", { label: "悬停提示", value: label }), "info"); } return normalizedMode; @@ -2188,8 +2379,7 @@ function setDefaultEarthZoom(nextZoom, { persist = true, applyToCurrentView = tr syncDefaultEarthZoomUi(defaultEarthZoom); if (applyToCurrentView && activeCamera) { - zoomLevel = defaultEarthZoom; - applyZoom(activeCamera); + setZoomLevel(defaultEarthZoom, activeCamera); } if (persist) { @@ -2270,6 +2460,10 @@ async function applyEarthSettings(settings, { applyLayers = true } = {}) { persist: false, suppressStatus: true, }); + setMotionEnabledGestures(settings.shared.motionEnabledGestures, { + persist: false, + suppressStatus: true, + }); setActiveTVTab(settings.shared.mediaPanelActiveTab); keyboardShortcuts = normalizeKeyboardShortcuts(settings.shared.keyboardShortcuts); renderShortcutSettings(); @@ -2278,6 +2472,7 @@ async function applyEarthSettings(settings, { applyLayers = true } = {}) { const layerVisibility = { ...(settings.shared.layerVisibility || {}) }; applyImmediateLayerVisibilityHints(layerVisibility); deferredLayerVisibilitySettings = layerVisibility; + applyNewsCategoryFilters(settings.shared.newsCategoryFilters); return; } @@ -2286,6 +2481,7 @@ async function applyEarthSettings(settings, { applyLayers = true } = {}) { persist: false, silent: true, }); + applyNewsCategoryFilters(settings.shared.newsCategoryFilters); } export function getMotionDebugEnabled() { @@ -2300,6 +2496,10 @@ export function getMotionDebugSkeletonOnly() { return motionDebugSkeletonOnly; } +export function getMotionEnabledGestures() { + return normalizeMotionEnabledGestures(motionEnabledGestures); +} + export function setMotionDebugEnabled( nextEnabled, { persist = true, suppressStatus = false } = {}, @@ -2323,12 +2523,13 @@ export function setMotionDebugEnabled( persistEarthSettings(); } if (!suppressStatus && changed) { - const message = motionDebugEnabled - ? rotationMode === ROTATION_MODE.MOTION - ? "动捕调试模式已开启" - : "动捕调试模式将在下次进入动捕时开启" - : "动捕调试模式已关闭"; - showStatusMessage(message, "info"); + showStatusMessage( + earthMessage("status.motionDebugMode", { + enabled: motionDebugEnabled, + pending: rotationMode !== ROTATION_MODE.MOTION, + }), + "info", + ); } return motionDebugEnabled; } @@ -2352,12 +2553,7 @@ export function setMotionProvider( persistEarthSettings(); } if (!suppressStatus && changed) { - showStatusMessage( - motionProvider === "motion_agent" - ? "动捕输入源已切换为 Motion Agent" - : "动捕输入源已切换为浏览器摄像头", - "info", - ); + showStatusMessage(earthMessage("status.motionProvider", { provider: motionProvider }), "info"); } return motionProvider; } @@ -2381,14 +2577,39 @@ export function setMotionDebugSkeletonOnly( persistEarthSettings(); } if (!suppressStatus && changed) { - showStatusMessage( - motionDebugSkeletonOnly ? "动捕调试已切换为只显示骨骼" : "动捕调试已显示实时画面", - "info", - ); + showStatusMessage(earthMessage("status.motionDebugView", { skeletonOnly: motionDebugSkeletonOnly }), "info"); } return motionDebugSkeletonOnly; } +export function setMotionEnabledGestures( + nextGestures, + { persist = true, suppressStatus = false } = {}, +) { + const normalized = normalizeMotionEnabledGestures(nextGestures); + const previous = getMotionEnabledGestures(); + const changed = + normalized.length !== previous.length || + normalized.some((gesture, index) => previous[index] !== gesture); + + motionEnabledGestures = normalized; + syncMotionGestureControls(); + + ensureMutableEarthSettingsState(); + earthSettingsState.shared.motionEnabledGestures = [...motionEnabledGestures]; + + if (changed) { + dispatchMotionSettingsChange(); + } + if (persist) { + persistEarthSettings(); + } + if (!suppressStatus && changed) { + showStatusMessage(earthMessage("status.motionGesturesUpdated"), "info"); + } + return getMotionEnabledGestures(); +} + export async function applyDeferredLayerVisibilitySettings(options = {}) { const layerVisibility = deferredLayerVisibilitySettings; deferredLayerVisibilitySettings = null; @@ -2398,6 +2619,7 @@ export async function applyDeferredLayerVisibilitySettings(options = {}) { silent: true, ...options, }); + applyNewsCategoryFilters(earthSettingsState?.shared?.newsCategoryFilters); } function resetEarthSettings() { @@ -2412,7 +2634,7 @@ function resetEarthSettings() { } } void applyEarthSettings(defaults).then(() => { - showStatusMessage("Earth 设置已重置", "info"); + showStatusMessage(earthMessage("status.settingsReset"), "info"); }); } @@ -2424,7 +2646,7 @@ async function setTerrainEnabled(button, enabled, { persist = true, silent = fal syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage("地形已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "真实地形", visible: false }), "info"); } return false; } @@ -2437,7 +2659,7 @@ async function setTerrainEnabled(button, enabled, { persist = true, silent = fal statusText: "加载中", }); if (!silent) { - showStatusMessage("正在加载真实地形数据...", "info"); + showStatusMessage(earthMessage("loading.realTerrainData"), "info"); } await ensureTerrainReady(); } @@ -2447,7 +2669,7 @@ async function setTerrainEnabled(button, enabled, { persist = true, silent = fal syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage("真实地形已显示", "success"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "真实地形", visible: true }), "success"); } return true; } catch (error) { @@ -2456,7 +2678,7 @@ async function setTerrainEnabled(button, enabled, { persist = true, silent = fal syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage("真实地形暂时不可用", "error"); + showStatusMessage(earthMessage("status.terrainUnavailable"), "error"); } return false; } @@ -2474,7 +2696,7 @@ async function setSatellitesLayerEnabled(button, enabled, { persist = true, sile } await setSatellitesEnabled(enabled, { suppressStatus: silent, suppressLoadingUi: silent }); if (!enabled && !silent) { - showStatusMessage("卫星已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "卫星", visible: false }), "info"); } else if (enabled) { setEarthStatValue("satellite-count", `${getSatelliteCount()} 颗`); } @@ -2504,7 +2726,7 @@ function setGridLinesLayerEnabled(button, enabled, { persist = true, silent = fa syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage(enabled ? "经纬线已显示" : "经纬线已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "经纬线", visible: enabled }), "info"); } return enabled; } @@ -2592,7 +2814,7 @@ function setBGPLayerEnabled(button, enabled, { persist = true, silent = false } syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage(enabled ? "BGP观测已显示" : "BGP观测已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "BGP观测", visible: enabled }), "info"); } return enabled; } @@ -2608,7 +2830,7 @@ function setComputeCentersLayerEnabled(button, enabled, { persist = true, silent syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage(enabled ? "算力中心已显示" : "算力中心已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "算力中心", visible: enabled }), "info"); } return enabled; } @@ -2677,7 +2899,7 @@ function setTrailsDisplayEnabled(enabled, { persist = true, silent = false } = { syncMobileLayerCards(); if (persist) persistEarthSettings(); if (!silent) { - showStatusMessage(enabled ? "轨迹已显示" : "轨迹已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "轨迹", visible: enabled }), "info"); } return enabled; } @@ -2787,7 +3009,7 @@ function getBuiltinLayerDefinitions() { startupMode: "preload", startupAlwaysLoad: true, startupLabel: "海陆基座", - startupMessage: "正在加载海陆基座...", + startupMessage: earthMessage("startup.landOceanBase"), getVisible: () => getShowCountryBoundaries(), setVisible: (visible, options = {}) => setCountryBoundariesLayerEnabled(getLayerButton("countryBoundaries"), visible, options), @@ -2804,7 +3026,7 @@ function getBuiltinLayerDefinitions() { startupPriority: 30, startupMode: "visible", startupLabel: "高清材质", - startupMessage: "正在启用高清材质...", + startupMessage: earthMessage("startup.hdTexture"), getVisible: () => getHighResTextureEnabled(), setVisible: (visible, options = {}) => setHighResTextureLayerEnabled(getLayerButton("earthHighResTexture"), visible, options), @@ -2839,8 +3061,8 @@ function getBuiltinLayerDefinitions() { startupMode: "visible", startupLabel: "海缆", startupMessage: { - prepare: "正在加载登陆点...", - load: "正在加载海缆...", + prepare: earthMessage("startup.landingPoints"), + load: earthMessage("startup.cables"), }, getVisible: () => getShowCables(), setVisible: (visible, options = {}) => @@ -2858,7 +3080,7 @@ function getBuiltinLayerDefinitions() { startupPriority: 60, startupMode: "preload", startupLabel: "算力中心", - startupMessage: "正在加载算力中心...", + startupMessage: earthMessage("startup.computeCenters"), getVisible: () => getShowComputeCenters(), setVisible: (visible, options = {}) => setComputeCentersLayerEnabled(getLayerButton("computeCenters"), visible, options), @@ -2875,7 +3097,7 @@ function getBuiltinLayerDefinitions() { startupPriority: 70, startupMode: "preload", startupLabel: "BGP态势", - startupMessage: "正在加载BGP态势...", + startupMessage: earthMessage("startup.bgp"), getVisible: () => getShowBGP(), setVisible: (visible, options = {}) => setBGPLayerEnabled(getLayerButton("bgp"), visible, options), @@ -2892,7 +3114,7 @@ function getBuiltinLayerDefinitions() { startupPriority: 65, startupMode: "visible", startupLabel: "船只", - startupMessage: "正在加载船只...", + startupMessage: earthMessage("startup.vessels"), getVisible: () => getVesselsEnabled(), setVisible: (visible, options = {}) => setVesselsLayerEnabled(getLayerButton("vessels"), visible, options), @@ -2909,7 +3131,7 @@ function getBuiltinLayerDefinitions() { startupPriority: 80, startupMode: "visible", startupLabel: "卫星", - startupMessage: "正在加载卫星...", + startupMessage: earthMessage("startup.satellites"), getVisible: () => getSatellitesEnabled(), setVisible: (visible, options = {}) => setSatellitesLayerEnabled(getLayerButton("satellites"), visible, options), @@ -2926,7 +3148,7 @@ function getBuiltinLayerDefinitions() { startupPriority: null, startupMode: "visible", startupLabel: "地形", - startupMessage: "正在渲染地形...", + startupMessage: earthMessage("startup.terrain"), statusTarget: "terrain-status", getVisible: () => showTerrain, setVisible: (visible, options = {}) => @@ -3110,12 +3332,7 @@ export function applyImmediateView(targetEarthObj, camera, options = {}) { targetEarthObj.rotation.x = nextRotation.x; targetEarthObj.rotation.y = nextRotation.y; targetEarthObj.rotation.z = nextRotation.z; - zoomLevel = zoom; - - if (camera) { - camera.position.z = CONFIG.defaultCameraZ / zoomLevel; - updateZoomDisplay(zoomLevel, camera.position.z.toFixed(0)); - } + setZoomLevel(zoom, camera); } export function setZoomLevel(nextZoom, camera = activeCamera) { @@ -3127,13 +3344,19 @@ export function setZoomLevel(nextZoom, camera = activeCamera) { return zoomLevel; } -export function showZoomStatusCapsule({ force = false } = {}) { +export function showZoomStatusCapsule({ force = false, zoom = null } = {}) { const now = Date.now(); if (!force && now - lastZoomStatusUpdateTime < ZOOM_STATUS_UPDATE_INTERVAL_MS) { return; } lastZoomStatusUpdateTime = now; - showGestureStatusMessage(`缩放 ${Math.round(zoomLevel * 100)}%`, "info"); + const currentZoom = Number.isFinite(Number(zoom)) + ? clampEarthZoomLevel(zoom) + : syncZoomLevelFromCamera(activeCamera); + showGestureStatusMessage( + earthMessage("status.zoomPercent", { percent: Math.round(currentZoom * 100) }), + "info", + ); } function cancelSettingsSheetAnimation() { @@ -3390,10 +3613,17 @@ function syncSettingsToggle(panelId, visible) { inputs.forEach((input) => { if (input instanceof HTMLInputElement) { input.checked = visible; + syncSettingsSwitchVisual(input, visible); } }); } +function syncSettingsSwitchVisual(input, visible = input?.checked === true) { + if (!(input instanceof HTMLInputElement)) return; + const switchShell = input.closest(".earth-settings-switch, .earth-mobile-settings-switch"); + switchShell?.classList.toggle("is-checked", Boolean(visible)); +} + function syncAllHudPanelToggles() { HUD_PANEL_IDS.forEach((panelId) => { const panel = document.getElementById(panelId); @@ -3563,12 +3793,15 @@ function startBoundaryBuildPolling() { setHighPrecisionBoundariesEnabled(true); await reloadCountryBoundaries({ suppressStatus: true }); await refreshBoundaryPrecisionStatus().catch(() => {}); - showStatusMessage("高精国界已下载并应用", "info"); + showStatusMessage(earthMessage("status.boundaryDownloaded"), "info"); } } } catch (error) { stopBoundaryBuildPolling(); - showStatusMessage(`高清国界进度读取失败：${error.message || error}`, "warning"); + showStatusMessage( + earthMessage("status.failure", { label: "高清国界进度读取失败", error: error.message || error }), + "warning", + ); } }, 1000); } @@ -3579,7 +3812,7 @@ async function startBoundaryPrecisionBuild() { }); boundaryBuildAttemptedThisSession = true; await fetchBoundaryPrecisionJson("/api/v1/earth/boundaries/build", { method: "POST", body: "{}" }); - showStatusMessage("高精国界构建已启动", "info"); + showStatusMessage(earthMessage("status.boundaryBuildStarted"), "info"); startBoundaryBuildPolling(); } @@ -3594,7 +3827,10 @@ async function setupBoundaryPrecisionControls() { } } catch (error) { renderBoundaryPrecisionStatus({}); - showStatusMessage(`高清国界状态读取失败：${error.message || error}`, "warning"); + showStatusMessage( + earthMessage("status.failure", { label: "高清国界状态读取失败", error: error.message || error }), + "warning", + ); } els.buildButtons.forEach((buildButton) => { @@ -3608,14 +3844,17 @@ async function setupBoundaryPrecisionControls() { if (getHighPrecisionBoundariesEnabled()) return; setHighPrecisionBoundariesEnabled(true); await reloadCountryBoundaries({ suppressStatus: true }); - showStatusMessage("已切换到高精国界", "info"); + showStatusMessage(earthMessage("status.boundaryPrecision", { high: true }), "info"); await refreshBoundaryPrecisionStatus().catch(() => {}); return; } await startBoundaryPrecisionBuild(); } catch (error) { await refreshBoundaryPrecisionStatus().catch(() => {}); - showStatusMessage(`高精国界切换失败：${error.message || error}`, "warning"); + showStatusMessage( + earthMessage("status.failure", { label: "高精国界切换失败", error: error.message || error }), + "warning", + ); } }); }); @@ -3626,7 +3865,10 @@ async function setupBoundaryPrecisionControls() { await startBoundaryPrecisionBuild(); } catch (error) { await refreshBoundaryPrecisionStatus().catch(() => {}); - showStatusMessage(`高精国界重建启动失败：${error.message || error}`, "warning"); + showStatusMessage( + earthMessage("status.failure", { label: "高精国界重建启动失败", error: error.message || error }), + "warning", + ); } }); }); @@ -3637,10 +3879,13 @@ async function setupBoundaryPrecisionControls() { if (!getHighPrecisionBoundariesEnabled()) return; setHighPrecisionBoundariesEnabled(false); await reloadCountryBoundaries({ suppressStatus: true }); - showStatusMessage("已切换到低精国界", "info"); + showStatusMessage(earthMessage("status.boundaryPrecision", { high: false }), "info"); await refreshBoundaryPrecisionStatus().catch(() => {}); } catch (error) { - showStatusMessage(`低精国界切换失败：${error.message || error}`, "warning"); + showStatusMessage( + earthMessage("status.failure", { label: "低精国界切换失败", error: error.message || error }), + "warning", + ); } }); }); @@ -3703,10 +3948,17 @@ function markRuntimeModeSection(selector, mode) { }); } +function isRuntimeModeSectionVisible(section, mode = rotationMode) { + const sectionModes = String(section?.dataset?.runtimeModeSection || "") + .split(/\s+/) + .filter(Boolean); + return sectionModes.includes(mode); +} + function syncRuntimeModeSections() { document.querySelectorAll("[data-runtime-mode-section]").forEach((section) => { if (!(section instanceof HTMLElement)) return; - section.hidden = section.dataset.runtimeModeSection !== rotationMode; + section.hidden = !isRuntimeModeSectionVisible(section); }); } @@ -3747,11 +3999,12 @@ function integrateMotionSettingsIntoRuntime() { motionPanel.remove(); }); - markRuntimeModeSection("[data-cruise-module-toggle]", ROTATION_MODE.CRUISE); - markRuntimeModeSection("[data-cruise-queue-mode]", ROTATION_MODE.CRUISE); + markRuntimeModeSection("[data-cruise-module-toggle]", `${ROTATION_MODE.CRUISE} ${ROTATION_MODE.MOTION}`); + markRuntimeModeSection("[data-cruise-queue-mode]", `${ROTATION_MODE.CRUISE} ${ROTATION_MODE.MOTION}`); markRuntimeModeSection("[data-auto-rotation-speed-slider]", ROTATION_MODE.ROTATE); markRuntimeModeSection("[data-motion-debug-toggle]", ROTATION_MODE.MOTION); markRuntimeModeSection("[data-motion-provider]", ROTATION_MODE.MOTION); + markRuntimeModeSection("[data-motion-gesture-toggle]", ROTATION_MODE.MOTION); syncRuntimeModeSections(); } @@ -3823,7 +4076,7 @@ function setShortcutBinding(actionId, binding, { persist = true } = {}) { if (!normalizedBinding) return false; const owner = getShortcutOwnerByBinding(normalizedBinding, { excludeActionId: actionId }); if (owner) { - showStatusMessage(`快捷键已被「${owner.label}」使用`, "warning"); + showStatusMessage(earthMessage("status.shortcutConflict", { owner: owner.label }), "warning"); return false; } const nextShortcuts = normalizeKeyboardShortcuts(keyboardShortcuts); @@ -3863,7 +4116,7 @@ function setShortcutEnabled(actionId, enabled, { persist = true } = {}) { if (enabled) { const owner = getShortcutOwnerByBinding(currentShortcut.binding, { excludeActionId: actionId }); if (owner) { - showStatusMessage(`快捷键已被「${owner.label}」使用`, "warning"); + showStatusMessage(earthMessage("status.shortcutConflict", { owner: owner.label }), "warning"); renderShortcutSettings(); return false; } @@ -3890,7 +4143,7 @@ function resetAllShortcutBindings() { capturingShortcutActionId = null; renderShortcutSettings(); persistEarthSettings(); - showStatusMessage("快捷键已恢复默认", "info"); + showStatusMessage(earthMessage("status.shortcutsReset"), "info"); } function moveCruiseRegionInOrder(region, targetRegion) { @@ -4050,11 +4303,15 @@ function setupSettingsControls() { const toggleInputs = document.querySelectorAll("[data-settings-panel]"); toggleInputs.forEach((input) => { + if (input instanceof HTMLInputElement) { + syncSettingsSwitchVisual(input); + } bindListener(input, "change", (event) => { const target = event.currentTarget; if (!(target instanceof HTMLInputElement)) return; const panelId = target.dataset.settingsPanel; if (!panelId) return; + syncSettingsSwitchVisual(target, target.checked); setHudPanelVisibility(panelId, target.checked); }); }); @@ -4259,6 +4516,19 @@ function setupSettingsControls() { }); }); + document.querySelectorAll("[data-news-category-toggle]").forEach((toggle) => { + if (!(toggle instanceof HTMLButtonElement)) return; + bindListener(toggle, "click", () => { + const active = toggle.classList.contains("is-active"); + setEarthNewsCategoryEnabled(toggle.dataset.newsCategoryToggle, !active); + }); + }); + + bindListener(window, "earth:set-news-category-enabled", (event) => { + const detail = event.detail || {}; + setEarthNewsCategoryEnabled(detail.category, Boolean(detail.enabled)); + }); + document.querySelectorAll("#toggle-daynight, [data-daynight-toggle]").forEach((dayNightToggle) => { if (!(dayNightToggle instanceof HTMLInputElement)) return; bindListener(dayNightToggle, "change", () => { @@ -4287,6 +4557,21 @@ function setupSettingsControls() { }); }); + document.querySelectorAll("[data-motion-gesture-toggle]").forEach((motionGestureButton) => { + if (!(motionGestureButton instanceof HTMLButtonElement)) return; + bindListener(motionGestureButton, "click", () => { + const gesture = motionGestureButton.dataset.motionGestureToggle; + if (!gesture) return; + const nextGestures = new Set(getMotionEnabledGestures()); + if (nextGestures.has(gesture)) { + nextGestures.delete(gesture); + } else { + nextGestures.add(gesture); + } + setMotionEnabledGestures(Array.from(nextGestures)); + }); + }); + const mobileSettingsReset = document.getElementById("mobile-settings-reset"); bindListener(mobileSettingsReset, "click", () => { resetEarthSettings(); @@ -4301,11 +4586,13 @@ function setupSettingsControls() { syncSatelliteIdleBreathingToggle(); syncSatelliteRealAltitudeToggle(); syncInteractableCompactDotsToggle(); + syncNewsCategoryFilterControls(); syncSurfaceHoverInfoModeControls(); syncDayNightToggle(dayNightEnabled); syncMotionDebugToggle(motionDebugEnabled); syncMotionProviderControls(motionProvider); syncMotionDebugSkeletonOnlyToggle(motionDebugSkeletonOnly); + syncMotionGestureControls(); void setupBoundaryPrecisionControls(); } @@ -4685,27 +4972,25 @@ function setupZoomControls(camera) { const MAX_PERCENT = CONFIG.maxZoom * 100; function doZoomStep(direction) { - let currentPercent = Math.round(zoomLevel * 100); + let currentPercent = Math.round(getZoomLevelFromCamera(camera) * 100); let newPercent = direction > 0 ? currentPercent + CLICK_STEP : currentPercent - CLICK_STEP; if (newPercent > MAX_PERCENT) newPercent = MAX_PERCENT; if (newPercent < MIN_PERCENT) newPercent = MIN_PERCENT; - zoomLevel = newPercent / 100; - applyZoom(camera); + setZoomLevel(newPercent / 100, camera); showZoomStatusCapsule({ force: true }); } function doContinuousZoom(direction) { - let currentPercent = Math.round(zoomLevel * 100); + let currentPercent = Math.round(getZoomLevelFromCamera(camera) * 100); let newPercent = direction > 0 ? currentPercent + 1 : currentPercent - 1; if (newPercent > MAX_PERCENT) newPercent = MAX_PERCENT; if (newPercent < MIN_PERCENT) newPercent = MIN_PERCENT; - zoomLevel = newPercent / 100; - applyZoom(camera); + setZoomLevel(newPercent / 100, camera); showZoomStatusCapsule(); } @@ -4768,10 +5053,8 @@ function setupZoomControls(camera) { bindListener(zoomOut, "touchend", () => handleMouseUp(-1)); bindListener(zoomValue, "click", () => { - const startZoomVal = zoomLevel; + const startZoomVal = getZoomLevelFromCamera(camera); const targetZoom = getDefaultEarthZoomLevel(); - const startDistance = CONFIG.defaultCameraZ / startZoomVal; - const targetDistance = CONFIG.defaultCameraZ / targetZoom; animateValue( 0, @@ -4779,14 +5062,11 @@ function setupZoomControls(camera) { 600, (progress) => { const ease = 1 - Math.pow(1 - progress, 3); - zoomLevel = startZoomVal + (targetZoom - startZoomVal) * ease; - camera.position.z = CONFIG.defaultCameraZ / zoomLevel; - const distance = - startDistance + (targetDistance - startDistance) * ease; - updateZoomDisplay(zoomLevel, distance.toFixed(0)); + const nextZoom = startZoomVal + (targetZoom - startZoomVal) * ease; + setZoomLevel(nextZoom, camera); }, () => { - zoomLevel = targetZoom; + setZoomLevel(targetZoom, camera); showStatusMessage(getZoomResetStatusMessage(targetZoom), "info"); }, ); @@ -4795,9 +5075,51 @@ function setupZoomControls(camera) { function setupWheelZoom(camera, renderer) { let wheelZoomFrameId = null; - let wheelZoomTarget = zoomLevel; - let wheelZoomStart = zoomLevel; + let wheelZoomTarget = getZoomLevelFromCamera(camera); + let wheelZoomStart = wheelZoomTarget; let wheelZoomStartAt = 0; + let lastTrackpadDirection = 0; + let suppressedTrackpadDirection = 0; + let suppressedTrackpadUntil = 0; + let suppressedTrackpadMagnitude = 0; + + function getWheelPixelDelta(event) { + if (event.deltaMode === WheelEvent.DOM_DELTA_LINE) { + return event.deltaY * 16; + } + if (event.deltaMode === WheelEvent.DOM_DELTA_PAGE) { + return event.deltaY * window.innerHeight; + } + return event.deltaY; + } + + function isTrackpadWheel(event, pixelDelta) { + return event.deltaMode === WheelEvent.DOM_DELTA_PIXEL && + Math.abs(pixelDelta) < WHEEL_TRACKPAD_PIXEL_THRESHOLD; + } + + function shouldSuppressTrackpadResidual(pixelDelta) { + const direction = Math.sign(pixelDelta); + const magnitude = Math.abs(pixelDelta); + const now = performance.now(); + return direction !== 0 && + direction === suppressedTrackpadDirection && + now < suppressedTrackpadUntil && + magnitude < suppressedTrackpadMagnitude * WHEEL_TRACKPAD_RESIDUAL_RATIO; + } + + function recordTrackpadWheel(pixelDelta) { + const direction = Math.sign(pixelDelta); + const magnitude = Math.abs(pixelDelta); + if (direction === 0) return; + const now = performance.now(); + if (lastTrackpadDirection !== 0 && direction !== lastTrackpadDirection) { + suppressedTrackpadDirection = lastTrackpadDirection; + suppressedTrackpadUntil = now + WHEEL_TRACKPAD_RESIDUAL_WINDOW_MS; + suppressedTrackpadMagnitude = magnitude; + } + lastTrackpadDirection = direction; + } function stopWheelZoomAnimation() { if (wheelZoomFrameId !== null) { @@ -4817,28 +5139,55 @@ function setupWheelZoom(camera, renderer) { 1, ); const ease = 1 - Math.pow(1 - progress, 3); - zoomLevel = wheelZoomStart + (wheelZoomTarget - wheelZoomStart) * ease; - applyZoom(camera); + setZoomLevel( + wheelZoomStart + (wheelZoomTarget - wheelZoomStart) * ease, + camera, + ); if (progress < 1) { wheelZoomFrameId = window.requestAnimationFrame(animateWheelZoom); return; } - zoomLevel = wheelZoomTarget; - applyZoom(camera); + setZoomLevel(wheelZoomTarget, camera); wheelZoomFrameId = null; wheelZoomStartAt = 0; } function startWheelZoomAnimation() { - wheelZoomStart = zoomLevel; + wheelZoomStart = getZoomLevelFromCamera(camera); wheelZoomStartAt = 0; if (wheelZoomFrameId === null) { wheelZoomFrameId = window.requestAnimationFrame(animateWheelZoom); } } + function applyMouseWheelZoom(direction) { + suppressedTrackpadDirection = 0; + suppressedTrackpadUntil = 0; + const baseZoom = wheelZoomFrameId === null + ? getZoomLevelFromCamera(camera) + : wheelZoomTarget; + wheelZoomTarget = clampEarthZoomLevel( + baseZoom + direction * WHEEL_ZOOM_STEP, + ); + stopWheelZoomAnimation(); + startWheelZoomAnimation(); + showZoomStatusCapsule({ force: true, zoom: wheelZoomTarget }); + } + + function applyTrackpadWheelZoom(pixelDelta) { + if (Math.abs(pixelDelta) < WHEEL_TRACKPAD_DEADZONE) return; + if (shouldSuppressTrackpadResidual(pixelDelta)) return; + stopWheelZoomAnimation(); + const currentZoom = getZoomLevelFromCamera(camera); + const nextZoom = currentZoom * Math.exp(-pixelDelta * WHEEL_TRACKPAD_SENSITIVITY); + wheelZoomTarget = setZoomLevel(nextZoom, camera); + wheelZoomStart = wheelZoomTarget; + recordTrackpadWheel(pixelDelta); + showZoomStatusCapsule({ force: true, zoom: wheelZoomTarget }); + } + cleanupFns.push(stopWheelZoomAnimation); bindListener( @@ -4846,25 +5195,17 @@ function setupWheelZoom(camera, renderer) { "wheel", (e) => { e.preventDefault(); - const direction = e.deltaY < 0 ? 1 : -1; - const baseZoom = - wheelZoomFrameId === null ? zoomLevel : wheelZoomTarget; - wheelZoomTarget = clampEarthZoomLevel( - baseZoom + direction * WHEEL_ZOOM_STEP, - ); - startWheelZoomAnimation(); - showZoomStatusCapsule({ force: true }); + const pixelDelta = getWheelPixelDelta(e); + if (isTrackpadWheel(e, pixelDelta)) { + applyTrackpadWheelZoom(pixelDelta); + return; + } + applyMouseWheelZoom(pixelDelta < 0 ? 1 : -1); }, { passive: false }, ); } -function applyZoom(camera) { - camera.position.z = CONFIG.defaultCameraZ / zoomLevel; - const distance = camera.position.z.toFixed(0); - updateZoomDisplay(zoomLevel, distance); -} - function animateValue(start, end, duration, onUpdate, onComplete) { const animationToken = ++focusViewAnimationToken; const startTime = performance.now(); @@ -4943,7 +5284,10 @@ function setupRotateControls(camera) { : rotationMode === ROTATION_MODE.MOTION ? "动捕" : "自动旋转"; - showStatusMessage(isRotating ? `${label}已开启` : `${label}已暂停`, "info"); + showStatusMessage( + earthMessage("status.runtimePaused", { label, active: isRotating }), + "info", + ); }); updateRotateUI(); @@ -5214,7 +5558,7 @@ function setupTerrainControls() { bindListener(layoutBtn, "click", () => { const expanded = toggleLayoutExpanded(container); - showStatusMessage(expanded ? "布局已最大化" : "布局已恢复", "info"); + showStatusMessage(earthMessage("status.layoutExpanded", { expanded }), "info"); }); const mediaVisible = @@ -5248,7 +5592,13 @@ function setupKeyboardControls() { if (!nextBinding) return; if (setShortcutBinding(capturingShortcutActionId, nextBinding)) { const definition = KEYBOARD_SHORTCUT_DEFINITION_BY_ID.get(capturingShortcutActionId); - showStatusMessage(`${definition?.label || "快捷键"}已设置为 ${getShortcutDisplayLabel(nextBinding)}`, "info"); + showStatusMessage( + earthMessage("status.shortcutSet", { + label: definition?.label || "快捷键", + binding: getShortcutDisplayLabel(nextBinding), + }), + "info", + ); capturingShortcutActionId = null; syncShortcutCaptureUi(); } @@ -5296,6 +5646,8 @@ function setupLiquidGlassInteractions() { surface.style.setProperty("--tilt-y", "0deg"); surface.style.setProperty("--panel-tilt-x", "0deg"); surface.style.setProperty("--panel-tilt-y", "0deg"); + surface.style.setProperty("--mouse-x", "0.5"); + surface.style.setProperty("--mouse-y", "0.5"); surface.style.setProperty("--dock-scale", "1"); surface.style.setProperty("--dock-lift", "0px"); surface.style.setProperty("--dock-shift-x", "0px"); @@ -5312,13 +5664,15 @@ function setupLiquidGlassInteractions() { surfaces.forEach((surface) => { resetSurface(surface); - const isToolbarSurface = Boolean(surface.closest(".earth-toolbar-items")); + const isToolbarSurface = Boolean(surface.closest(".earth-toolbar")); const isPanelSurface = surface.classList.contains("hud-panel"); bindListener(surface, "pointermove", (event) => { const rect = surface.getBoundingClientRect(); const px = (event.clientX - rect.left) / rect.width; const py = (event.clientY - rect.top) / rect.height; + surface.style.setProperty("--mouse-x", `${px.toFixed(3)}`); + surface.style.setProperty("--mouse-y", `${py.toFixed(3)}`); if (isPanelSurface) { const panelTiltX = (0.5 - py) * 5; const panelTiltY = (px - 0.5) * 6; @@ -5704,9 +6058,9 @@ function updateRotateUI() { ? "动捕" : "自动旋转"; if (tooltip) { - tooltip.textContent = autoRotate ? `暂停${activeLabel}` : `开始${activeLabel}`; + tooltip.textContent = translateText(autoRotate ? `暂停${activeLabel}` : `开始${activeLabel}`); } - btn.title = `${getRotationModeLabel()} · ${activeLabel}`; + btn.title = translateText(`${getRotationModeLabel()} · ${activeLabel}`); } syncRotationModeButtons(); @@ -5748,7 +6102,7 @@ export function setAutoRotationSpeed(value, { persist = true, suppressStatus = f persistEarthSettings(); } if (changed && !suppressStatus) { - showStatusMessage(`旋转转速已设为 ${formatAutoRotationSpeed(normalizedSpeed)}`, "info"); + showStatusMessage(earthMessage("status.rotateSpeed", { speed: formatAutoRotationSpeed(normalizedSpeed) }), "info"); } return normalizedSpeed; } @@ -5771,7 +6125,7 @@ export function setRotationMode(nextMode, { persist = true, suppressStatus = fal persistEarthSettings(); } if (changed && !suppressStatus) { - showStatusMessage(`已切换到${getRotationModeLabel(normalizedMode)}`, "info"); + showStatusMessage(earthMessage("status.switchedTo", { label: getRotationModeLabel(normalizedMode) }), "info"); } } @@ -5793,7 +6147,7 @@ export function focusEarthView(camera, options = {}) { const startRotX = earthObj.rotation.x; const startRotY = earthObj.rotation.y; const startRotZ = earthObj.rotation.z; - const startZoom = zoomLevel; + const startZoom = getZoomLevelFromCamera(camera); const defaultZoom = getDefaultEarthZoomLevel(); const shouldRestoreZoomViaDefault = zoomTransitionMode === "restore-current-via-default" && @@ -5822,32 +6176,28 @@ export function focusEarthView(camera, options = {}) { if (progress < rotateStartProgress) { const zoomProgress = progress / rotateStartProgress; const zoomEase = 1 - Math.pow(1 - zoomProgress, 3); - zoomLevel = startZoom + (defaultZoom - startZoom) * zoomEase; + setZoomLevel(startZoom + (defaultZoom - startZoom) * zoomEase, camera); } else if (progress <= rotateEndProgress) { - zoomLevel = defaultZoom; + setZoomLevel(defaultZoom, camera); } else { const zoomProgress = (progress - rotateEndProgress) / (1 - rotateEndProgress); const zoomEase = 1 - Math.pow(1 - zoomProgress, 3); - zoomLevel = defaultZoom + (startZoom - defaultZoom) * zoomEase; + setZoomLevel(defaultZoom + (startZoom - defaultZoom) * zoomEase, camera); } } else { earthObj.rotation.x = startRotX + (nextRotation.x - startRotX) * ease; earthObj.rotation.y = startRotY + (nextRotation.y - startRotY) * ease; earthObj.rotation.z = startRotZ + (nextRotation.z - startRotZ) * ease; - zoomLevel = startZoom + (zoom - startZoom) * ease; + setZoomLevel(startZoom + (zoom - startZoom) * ease, camera); } - camera.position.z = CONFIG.defaultCameraZ / zoomLevel; - updateZoomDisplay(zoomLevel, camera.position.z.toFixed(0)); }, () => { - zoomLevel = shouldRestoreZoomViaDefault ? startZoom : zoom; + setZoomLevel(shouldRestoreZoomViaDefault ? startZoom : zoom, camera); earthObj.rotation.x = nextRotation.x; earthObj.rotation.y = nextRotation.y; earthObj.rotation.z = nextRotation.z; - camera.position.z = CONFIG.defaultCameraZ / zoomLevel; - updateZoomDisplay(zoomLevel, camera.position.z.toFixed(0)); if (!suppressStatus) { - showStatusMessage("视角已重置", "info"); + showStatusMessage(earthMessage("status.viewReset"), "info"); } resolve(); }, @@ -5856,7 +6206,7 @@ export function focusEarthView(camera, options = {}) { } export function getZoomLevel() { - return zoomLevel; + return syncZoomLevelFromCamera(activeCamera); } export function getDefaultEarthZoomLevel() { diff --git a/frontend/public/earth/js/country-boundaries.js b/frontend/public/earth/js/country-boundaries.js index e49a3097..a4105c06 100644 --- a/frontend/public/earth/js/country-boundaries.js +++ b/frontend/public/earth/js/country-boundaries.js @@ -210,7 +210,7 @@ function boundaryLineRadius({ claim = false } = {}) { return ( CONFIG.earthRadius + COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset + - (claim ? 0.018 : 0) + (claim ? COUNTRY_BOUNDARY_CONFIG.claimLineAltitudeOffset : 0) ); } diff --git a/frontend/public/earth/js/earth-interactables.js b/frontend/public/earth/js/earth-interactables.js new file mode 100644 index 00000000..6496ba72 --- /dev/null +++ b/frontend/public/earth/js/earth-interactables.js @@ -0,0 +1,185 @@ +import { PATHS } from "./constants.js"; +import { createInteractableLayer, SURFACE_AVOIDANCE_PROFILES } from "./interactable.js"; + +const interactableRevisions = new Map(); + +function featureToInteractableItem(feature) { + const props = feature?.properties || {}; + const coordinates = feature?.geometry?.coordinates || []; + const longitude = Number(props.longitude ?? coordinates[0]); + const latitude = Number(props.latitude ?? coordinates[1]); + if (!Number.isFinite(latitude) || !Number.isFinite(longitude)) return null; + return { + ...props, + id: String(props.id ?? feature.id ?? ""), + latitude, + longitude, + kind: props.kind || "default", + label: props.label || "", + revision: Number(props.revision || 0), + }; +} + +function normalizeInteractableItem(item) { + const latitude = Number(item?.latitude ?? item?.lat); + const longitude = Number(item?.longitude ?? item?.lon ?? item?.lng); + if (!Number.isFinite(latitude) || !Number.isFinite(longitude)) return null; + const id = String(item?.id || "").trim(); + if (!id) return null; + return { + ...item, + id, + latitude, + longitude, + kind: item.kind || "default", + label: item.label || "", + revision: Number(item.revision || 0), + }; +} + +function drawInteractableIcon(context, { glow = false, color = "#7dd3fc" }) { + if (glow) { + context.shadowColor = color; + context.shadowBlur = 16; + } + context.fillStyle = color; + context.strokeStyle = "rgba(8, 15, 27, 0.92)"; + context.lineWidth = 7; + context.beginPath(); + context.arc(0, 0, 27, 0, Math.PI * 2); + context.fill(); + context.stroke(); + context.fillStyle = "rgba(255,255,255,0.95)"; + context.beginPath(); + context.arc(0, 0, 9, 0, Math.PI * 2); + context.fill(); +} + +const earthInteractableLayer = createInteractableLayer({ + id: "earthInteractables", + objectType: "earth_interactable", + renderOrder: 4.55, + altitudeOffset: 0.28, + pointSize: 30, + colors: { + normal: "#7dd3fc", + default: "#7dd3fc", + note: "#facc15", + alert: "#fb7185", + place: "#86efac", + }, + opacity: { + normal: 0.9, + dimmed: 0.3, + hover: 1, + locked: 1, + }, + stateScale: { + hover: 1.18, + locked: 1.34, + dimmed: 0.82, + }, + avoidance: SURFACE_AVOIDANCE_PROFILES.city, + icon: { + draw: drawInteractableIcon, + }, + getPosition: (item) => ({ + latitude: item.latitude, + longitude: item.longitude, + }), + getKind: (item) => item.kind || "default", + getUserData: (item) => ({ + ...item, + type: "earth_interactable", + }), + cluster: { + strategy: "stable-spherical", + }, +}); + +export async function loadEarthInteractables(earth, { silent = false } = {}) { + if (!earth) return { totalCount: 0 }; + const response = await fetch(PATHS.interactablesApi, { cache: "no-store" }); + if (!response.ok) { + throw new Error(`Failed to load Earth interactables: ${response.status}`); + } + const payload = await response.json(); + const items = (payload.features || []) + .map(featureToInteractableItem) + .filter(Boolean); + interactableRevisions.clear(); + items.forEach((item) => interactableRevisions.set(item.id, Number(item.revision || 0))); + earthInteractableLayer.setData(items); + earthInteractableLayer.attach(earth); + earthInteractableLayer.setVisible(true); + if (!silent) { + console.info("Earth interactables loaded", { count: items.length }); + } + return { totalCount: earthInteractableLayer.getCount() }; +} + +export function clearEarthInteractables(earth) { + interactableRevisions.clear(); + earthInteractableLayer.setData([]); + if (earth) earthInteractableLayer.attach(earth); +} + +export function refreshEarthInteractables(earth) { + return loadEarthInteractables(earth, { silent: true }); +} + +export function applyEarthInteractableEvent(earth, payload = {}) { + if (payload.entity !== "interactable") return false; + const action = payload.action; + const item = normalizeInteractableItem(payload.item); + const ids = Array.isArray(payload.ids) ? payload.ids.map(String) : []; + const id = item?.id || ids[0]; + if (!id) return false; + + const nextRevision = Number(payload.revision ?? item?.revision ?? 0); + const currentRevision = Number(interactableRevisions.get(id) || 0); + if (nextRevision && currentRevision && nextRevision < currentRevision) { + return false; + } + + if (action === "deleted") { + const changed = earthInteractableLayer.removeItem(id); + interactableRevisions.set(id, nextRevision || currentRevision); + return changed; + } + + if (!item) { + refreshEarthInteractables(earth).catch((error) => { + console.warn("刷新 Earth interactables 失败:", error); + }); + return false; + } + + const changed = earthInteractableLayer.upsertItem(item); + interactableRevisions.set(id, nextRevision || Number(item.revision || 0)); + earthInteractableLayer.attach(earth); + earthInteractableLayer.setVisible(true); + return changed; +} + +export function getEarthInteractableMarkers() { + return earthInteractableLayer.getMarkers(); +} + +export function getEarthInteractablePointerIntersections(options = {}) { + return earthInteractableLayer.getPointerIntersections(options); +} + +export function setEarthInteractableMarkerState(marker, state = "normal") { + earthInteractableLayer.setMarkerState(marker, state); +} + +export function clearEarthInteractableSelection() { + earthInteractableLayer.getMarkers().forEach((marker) => { + earthInteractableLayer.setMarkerState(marker, "normal"); + }); +} + +export function updateEarthInteractableVisualState(focusType, focusObject, camera) { + earthInteractableLayer.updateVisualState(focusType, focusObject, camera); +} diff --git a/frontend/public/earth/js/i18n.js b/frontend/public/earth/js/i18n.js new file mode 100644 index 00000000..d022b069 --- /dev/null +++ b/frontend/public/earth/js/i18n.js @@ -0,0 +1,1641 @@ +const DEFAULT_LOCALE = "zh-CN"; +const LOCALE_STORAGE_KEY = "planet-locale"; +const LEGACY_DOCS_LANG_STORAGE_KEY = "docs-lang"; +const LOCALE_CHANGE_EVENT = "earth:locale-change"; + +const LOCALE_LABELS = { + "zh-CN": "中文", + "en-US": "English", +}; + +const EARTH_MESSAGE_MARKER = "__planetEarthMessage"; + +function localizedParam(value, locale) { + if (isEarthMessage(value)) return formatEarthMessage(value, locale); + return translateText(String(value ?? ""), locale); +} + +const EARTH_MESSAGE_TEMPLATES = { + "loading.default": { + zh: "正在加载...", + en: "Loading...", + }, + "loading.initializing": { + zh: "正在初始化...", + en: "Initializing...", + }, + "loading.cableData": { + zh: "正在加载线缆数据...", + en: "Loading cable data...", + }, + "loading.vesselData": { + zh: "正在加载船只数据...", + en: "Loading vessel data...", + }, + "loading.satelliteData": { + zh: "正在加载卫星数据...", + en: "Loading satellite data...", + }, + "loading.realTerrainData": { + zh: "正在加载真实地形数据...", + en: "Loading real terrain data...", + }, + "startup.landingPoints": { + zh: "正在加载登陆点...", + en: "Loading landing points...", + }, + "startup.cables": { + zh: "正在加载海缆...", + en: "Loading subsea cables...", + }, + "startup.satellites": { + zh: "正在加载卫星...", + en: "Loading satellites...", + }, + "startup.vessels": { + zh: "正在加载船只...", + en: "Loading vessels...", + }, + "startup.bgp": { + zh: "正在加载BGP态势...", + en: "Loading BGP situation...", + }, + "startup.clouds": { + zh: "正在加载大气云图...", + en: "Loading atmosphere and clouds...", + }, + "startup.landOceanBase": { + zh: "正在加载海陆基座...", + en: "Loading land/ocean base...", + }, + "startup.computeCenters": { + zh: "正在加载算力中心...", + en: "Loading compute centers...", + }, + "startup.hdTexture": { + zh: "正在启用高清材质...", + en: "Enabling HD texture...", + }, + "startup.terrain": { + zh: "正在渲染地形...", + en: "Rendering terrain...", + }, + "status.dataLoaded": { + zh: "数据已加载", + en: "Data loaded", + }, + "status.layoutExpanded": { + zh: ({ expanded }) => (expanded ? "布局已最大化" : "布局已恢复"), + en: ({ expanded }) => (expanded ? "Layout maximized" : "Layout restored"), + }, + "status.layerUnavailable": { + zh: ({ layer }) => `${localizedParam(layer, "zh-CN")}当前不可用`, + en: ({ layer }) => `${localizedParam(layer, "en-US")} unavailable`, + }, + "status.layerVisibility": { + zh: ({ layer, visible }) => `${localizedParam(layer, "zh-CN")}${visible ? "已显示" : "已隐藏"}`, + en: ({ layer, visible }) => `${localizedParam(layer, "en-US")} ${visible ? "shown" : "hidden"}`, + }, + "status.layerEnabled": { + zh: ({ layer, enabled }) => `${localizedParam(layer, "zh-CN")}${enabled ? "已启用" : "已隐藏"}`, + en: ({ layer, enabled }) => `${localizedParam(layer, "en-US")} ${enabled ? "enabled" : "hidden"}`, + }, + "status.loadedCables": { + zh: ({ count }) => `成功加载 ${count} 条电缆`, + en: ({ count }) => `Loaded ${count} cables`, + }, + "status.loadedLandingPoints": { + zh: ({ count }) => `成功加载 ${count} 个登陆点`, + en: ({ count }) => `Loaded ${count} landing points`, + }, + "status.locked": { + zh: ({ name }) => `已锁定: ${name}`, + en: ({ name }) => `Locked: ${name}`, + }, + "status.located": { + zh: ({ target, name }) => `已定位${localizedParam(target, "zh-CN")}：${name}`, + en: ({ target, name }) => `Located ${localizedParam(target, "en-US").toLowerCase()}: ${name}`, + }, + "status.selected": { + zh: ({ target, name }) => (target ? `已选择${localizedParam(target, "zh-CN")}: ${name}` : `已选择: ${name}`), + en: ({ target, name }) => (target ? `Selected ${localizedParam(target, "en-US")}: ${name}` : `Selected: ${name}`), + }, + "status.bgpSelected": { + zh: ({ collector, regionCount, cableCount }) => + `已选择BGP事件: ${collector} · ${regionCount}个区域 / ${cableCount}条相关海缆`, + en: ({ collector, regionCount, cableCount }) => + `Selected BGP event: ${collector} · ${regionCount} regions / ${cableCount} related cables`, + }, + "status.computeCoordinatesSaved": { + zh: "算力中心坐标已保存", + en: "Compute center coordinates saved", + }, + "status.coordinatesSavedLater": { + zh: "坐标已保存，地图稍后同步", + en: "Coordinates saved; map will sync shortly", + }, + "status.coordinatesSavedSyncing": { + zh: "坐标已保存，正在同步地图...", + en: "Coordinates saved; syncing map...", + }, + "status.newsPanel": { + zh: ({ open }) => (open ? "Live 新闻窗口已打开" : "Live 新闻窗口已关闭"), + en: ({ open }) => (open ? "Live News panel opened" : "Live News panel closed"), + }, + "status.newsRefresh": { + zh: ({ ok }) => (ok ? "态势新闻已刷新" : "态势新闻刷新失败"), + en: ({ ok }) => (ok ? "Situation news refreshed" : "Situation news refresh failed"), + }, + "status.newsSyncFailed": { + zh: "态势新闻同步失败", + en: "Situation news sync failed", + }, + "status.languageSwitched": { + zh: "语言已切换为中文", + en: "Language switched to English", + }, + "status.copyEmpty": { + zh: "无可复制内容", + en: "Nothing to copy", + }, + "status.copyValue": { + zh: ({ label, value }) => `已复制${label}：${value}`, + en: ({ label, value }) => `Copied ${label}: ${value}`, + }, + "status.copyFailed": { + zh: "复制失败", + en: "Copy failed", + }, + "status.zoomPercent": { + zh: ({ percent }) => `缩放 ${percent}%`, + en: ({ percent }) => `Zoom ${percent}%`, + }, + "status.zoomReset": { + zh: ({ zoom }) => `缩放已重置到${zoom}`, + en: ({ zoom }) => `Zoom reset to ${zoom}`, + }, + "status.runtimePaused": { + zh: ({ label, active }) => `${localizedParam(label, "zh-CN")}${active ? "已恢复" : "已暂停"}`, + en: ({ label, active }) => `${localizedParam(label, "en-US")} ${active ? "resumed" : "paused"}`, + }, + "status.valueChanged": { + zh: ({ label, value }) => `${localizedParam(label, "zh-CN")}已切换为：${localizedParam(value, "zh-CN")}`, + en: ({ label, value }) => `${localizedParam(label, "en-US")} switched to ${localizedParam(value, "en-US")}`, + }, + "status.modulesChanged": { + zh: ({ label, value }) => `${localizedParam(label, "zh-CN")}已切换为：${value}`, + en: ({ label, value }) => `${localizedParam(label, "en-US")} changed to ${value}`, + }, + "status.updated": { + zh: ({ label }) => `${localizedParam(label, "zh-CN")}已更新`, + en: ({ label }) => `${localizedParam(label, "en-US")} updated`, + }, + "status.booleanSetting": { + zh: ({ label, enabled }) => `${localizedParam(label, "zh-CN")}${enabled ? "已开启" : "已关闭"}`, + en: ({ label, enabled }) => `${localizedParam(label, "en-US")} ${enabled ? "enabled" : "disabled"}`, + }, + "status.motionDebugMode": { + zh: ({ enabled, pending }) => + enabled + ? (pending ? "动捕调试模式将在下次进入动捕时开启" : "动捕调试模式已开启") + : "动捕调试模式已关闭", + en: ({ enabled, pending }) => + enabled + ? (pending ? "Motion debug mode will enable next time Motion Mode starts" : "Motion debug mode enabled") + : "Motion debug mode disabled", + }, + "status.motionProvider": { + zh: ({ provider }) => (provider === "motion_agent" ? "Motion Agent 已接管动捕输入" : "动捕已切换到浏览器传感器"), + en: ({ provider }) => (provider === "motion_agent" ? "Motion Agent now controls motion input" : "Motion switched to browser sensors"), + }, + "status.motionDebugView": { + zh: ({ skeletonOnly }) => (skeletonOnly ? "动捕调试已切换为只显示骨骼" : "动捕调试已显示实时画面"), + en: ({ skeletonOnly }) => (skeletonOnly ? "Motion debug switched to skeleton only" : "Motion debug showing live video"), + }, + "status.motionGesturesUpdated": { + zh: "动捕识别动作已更新", + en: "Motion gestures updated", + }, + "status.settingsReset": { + zh: "Earth 设置已重置", + en: "Earth settings reset", + }, + "status.terrainUnavailable": { + zh: "真实地形暂时不可用", + en: "Real terrain is temporarily unavailable", + }, + "status.boundaryDownloaded": { + zh: "高精国界已下载并应用", + en: "High-precision boundaries downloaded and applied", + }, + "status.boundaryBuildStarted": { + zh: "高精国界构建已启动", + en: "High-precision boundary build started", + }, + "status.boundaryPrecision": { + zh: ({ high }) => (high ? "已切换到高精国界" : "已切换到低精国界"), + en: ({ high }) => (high ? "Switched to high-precision boundaries" : "Switched to low-precision boundaries"), + }, + "status.failure": { + zh: ({ label, error }) => `${localizedParam(label, "zh-CN")}：${error}`, + en: ({ label, error }) => `${localizedParam(label, "en-US")}: ${error}`, + }, + "status.layerLoadFailed": { + zh: ({ layer, error }) => `${localizedParam(layer, "zh-CN")}加载失败: ${error}`, + en: ({ layer, error }) => `Failed to load ${localizedParam(layer, "en-US")}: ${error}`, + }, + "status.loadFailedList": { + zh: ({ items }) => (Array.isArray(items) ? items : []) + .map(({ label, reason }) => + `${localizedParam(label, "zh-CN")}加载失败: ${reason?.message || String(reason)}`, + ) + .join("；"), + en: ({ items }) => (Array.isArray(items) ? items : []) + .map(({ label, reason }) => + `Failed to load ${localizedParam(label, "en-US")}: ${reason?.message || String(reason)}`, + ) + .join("; "), + }, + "status.shortcutConflict": { + zh: ({ owner }) => `快捷键已被「${owner}」使用`, + en: ({ owner }) => `Shortcut already used by "${owner}"`, + }, + "status.shortcutsReset": { + zh: "快捷键已恢复默认", + en: "Shortcuts reset to defaults", + }, + "status.shortcutSet": { + zh: ({ label, binding }) => `${localizedParam(label, "zh-CN")}已设置为 ${binding}`, + en: ({ label, binding }) => `${localizedParam(label, "en-US")} set to ${binding}`, + }, + "status.rotateSpeed": { + zh: ({ speed }) => `旋转转速已设为 ${speed}`, + en: ({ speed }) => `Rotation speed set to ${speed}`, + }, + "status.switchedTo": { + zh: ({ label }) => `已切换到${localizedParam(label, "zh-CN")}`, + en: ({ label }) => `Switched to ${localizedParam(label, "en-US")}`, + }, + "status.viewReset": { + zh: "视角已重置", + en: "View reset", + }, + "status.motionPrefix": { + zh: ({ text }) => `动捕: ${localizedParam(text, "zh-CN")}`, + en: ({ text }) => `Motion: ${localizedParam(text, "en-US")}`, + }, + "status.motionCruiseTargetSwitched": { + zh: "动捕: 已切换巡航目标", + en: "Motion: switched cruise target", + }, + "status.motionNoTarget": { + zh: ({ layer }) => (layer ? `动捕: ${localizedParam(layer, "zh-CN")}当前视野没有可选目标` : "动捕: 当前没有可用图层"), + en: ({ layer }) => (layer ? `Motion: ${localizedParam(layer, "en-US")} has no selectable targets in view` : "Motion: no available layer"), + }, + "status.motionSwitchedTo": { + zh: ({ label, layer }) => `动捕: 已切换到${localizedParam(label, "zh-CN")}${layer ? "图层" : ""}`, + en: ({ label, layer }) => `Motion: switched to ${localizedParam(label, "en-US")}${layer ? " layer" : ""}`, + }, + "status.motionNoVisibleLayer": { + zh: "动捕: 当前没有可切换的可见图层", + en: "Motion: no visible layer to switch to", + }, + "status.motionConfirmed": { + zh: ({ label }) => `动捕: 已确认${localizedParam(label, "zh-CN")}`, + en: ({ label }) => `Motion: confirmed ${localizedParam(label, "en-US")}`, + }, + "status.motionConfirmCurrent": { + zh: "动捕: 已确认当前目标", + en: "Motion: confirmed current target", + }, + "status.motionSelectTargetFirst": { + zh: "动捕: 请先选择目标", + en: "Motion: select a target first", + }, + "status.cruisePausedOpenCandidates": { + zh: "已暂停巡航，正在打开候选列表", + en: "Cruise paused; opening candidate list", + }, + "status.motionPausedOpenCandidates": { + zh: "已暂停动捕目标展示，正在打开候选列表", + en: "Motion target display paused; opening candidate list", + }, + "status.candidatesClosedCruiseResumed": { + zh: "候选列表已关闭，巡航已恢复", + en: "Candidate list closed; cruise resumed", + }, + "status.cruiseModeRequired": { + zh: "切换到巡航模式后可切换卡片", + en: "Switch to Cruise Mode before switching cards", + }, + "status.cruisePausedSpace": { + zh: "巡航已暂停，按空格恢复", + en: "Cruise paused. Press Space to resume.", + }, +}; + +const EXACT_TRANSLATIONS = { + "智能星球计划 - 现实层宇宙全息感知": "Intelligent Planet - Reality Layer Situational Awareness", + "智能星球计划": "Intelligent Planet Program", + "现实层宇宙全息感知系统": "Reality Layer Situational Awareness System", + "卫星 · 海底光缆 · 算力基础设施": "Satellites · Subsea Cables · Compute Infrastructure", + "图层": "Layers", + "折叠": "Collapse", + "关闭": "Close", + "清除": "Clear", + "开启": "On", + "已显示": "Shown", + "已隐藏": "Hidden", + "折叠图层列表": "Collapse layer list", + "关闭图层面板": "Close layer panel", + "搜索图层...": "Search layers...", + "清除搜索": "Clear search", + "正在加载...": "Loading...", + "无匹配图层": "No matching layers", + "海缆": "Subsea Cables", + "卫星": "Satellites", + "算力中心": "Compute Centers", + "船只": "Vessels", + "地形": "Terrain", + "海陆基座": "Land/Ocean Base", + "高清材质": "HD Texture", + "大气云图": "Atmosphere & Clouds", + "国界线": "Borders", + "经纬线": "Graticule", + "切换海缆显示": "Toggle subsea cables", + "切换卫星显示": "Toggle satellites", + "切换算力中心显示": "Toggle compute centers", + "切换船只显示": "Toggle vessels", + "切换BGP观测显示": "Toggle BGP observations", + "BGP观测": "BGP Observations", + "BGP态势": "BGP Situation", + "观测站": "Collector", + "切换地形显示": "Toggle terrain", + "切换高清材质显示": "Toggle HD texture", + "切换大气云图显示": "Toggle atmosphere and clouds", + "切换国界显示": "Toggle borders", + "切换经纬线显示": "Toggle graticule", + "显示卫星": "Show Satellites", + "隐藏卫星": "Hide Satellites", + "显示船只": "Show Vessels", + "隐藏船只": "Hide Vessels", + "显示线缆": "Show Cables", + "隐藏线缆": "Hide Cables", + "显示经纬线": "Show Graticule", + "隐藏经纬线": "Hide Graticule", + "显示国界": "Show Borders", + "隐藏国界": "Hide Borders", + "显示国界线": "Show Border Lines", + "隐藏国界线": "Hide Border Lines", + "显示高清材质": "Show HD Texture", + "隐藏高清材质": "Hide HD Texture", + "显示大气云图": "Show Atmosphere & Clouds", + "隐藏大气云图": "Hide Atmosphere & Clouds", + "显示BGP观测": "Show BGP Observations", + "隐藏BGP观测": "Hide BGP Observations", + "显示算力中心": "Show Compute Centers", + "隐藏算力中心": "Hide Compute Centers", + "显示轨迹": "Show Trails", + "隐藏轨迹": "Hide Trails", + "显示地形": "Show Terrain", + "隐藏地形": "Hide Terrain", + "图层控制": "Layer Controls", + "搜索": "Search", + "自动旋转": "Auto Rotate", + "重新加载数据": "Reload Data", + "缩放控制": "Zoom Controls", + "缩放:": "Zoom:", + "放大": "Zoom In", + "缩小": "Zoom Out", + "设置": "Settings", + "重置视角": "Reset View", + "最大化布局": "Maximize Layout", + "工具菜单": "Tool Menu", + "展开工具菜单": "Open tool menu", + "展开图例": "Expand legend", + "折叠图例": "Collapse legend", + "关闭图例": "Close legend", + "关闭地球信息": "Close Earth info", + "Live 新闻": "Live News", + "选择新闻直播源": "Select live news source", + "刷新直播源": "Refresh live source", + "访问官网": "Visit website", + "折叠新闻直播内容": "Collapse live news details", + "关闭 Live 新闻面板": "Close Live News panel", + "暂无可用频道": "No Channel Available", + "待加载": "Pending", + "等待加载": "Waiting", + "等待加载直播源": "Waiting for live source", + "正在同步直播源...": "Syncing live sources...", + "直播已加载": "Live stream loaded", + "当前视频流不可播放，请尝试其他频道": "This stream cannot be played. Try another channel.", + "当前频道仅支持外部打开": "This channel opens externally only", + "电视直播源加载失败": "Failed to load TV streams", + "直播加载中": "Live stream loading", + "直播流缓冲停滞，正在等待数据": "Live stream stalled; waiting for data", + "直播流缓冲写入失败": "Failed to append live stream buffer", + "HLS 播放列表格式无法解析": "Unable to parse HLS playlist", + "HLS 分片格式无法解析": "Unable to parse HLS segment", + "HLS 播放流不可用": "HLS stream unavailable", + "直播放流不可用，已回退到官网直播页": "Live stream unavailable; falling back to the official live page", + "直播流网络波动，正在重试...": "Live stream network issue. Retrying...", + "直播流正在恢复...": "Live stream recovering...", + "已加载视频流，点击播放继续": "Video stream loaded. Click play to continue.", + "当前频道仅支持跳转官网或外部播放器打开。": "This channel can only open on the official site or an external player.", + "点击查看当前频道来源、目录和补充说明": "Tap to view channel source, catalog, and notes", + "点击查看完整频道信息": "Tap to view full channel info", + "展开新闻直播内容": "Expand live news details", + "展开新闻直播信息": "Expand live news details", + "折叠新闻直播信息": "Collapse live news details", + "Live 新闻窗口已打开": "Live News panel opened", + "Live 新闻窗口已关闭": "Live News panel closed", + "内置": "Built-in", + "内置源": "Built-in", + "采集": "Collected", + "采集源": "Collector source", + "频道": "Channel", + "尚未同步": "Not synced yet", + "最近同步": "Last synced", + "配置于控制台": "Configured in console", + "当前未配置可播放新闻直播源": "No playable live news source is configured", + "频道目录待同步": "Channel catalog pending sync", + "支持后台配置默认源与采集器补充源。": "Supports default channels and collector-supplied sources from the console.", + "暂无可播放直播源，请先在系统配置中添加频道。": "No playable live source. Add a channel in system configuration first.", + "打开态势新闻": "Open situation news", + "态势聚合新闻": "Situation News", + "刷新新闻源": "Refresh news sources", + "打开源站": "Open source site", + "收起态势新闻": "Collapse situation news", + "跟随地球正面视角自动切换区域新闻": "Region news follows the front-facing globe view", + "当前关注区域": "Current Focus", + "跟随当前视角聚焦全球区域新闻": "Focus regional news from the current view", + "类型": "Type", + "来源": "Source", + "查看全部": "View all", + "返回摘要": "Back to summary", + "当前区域新闻聚合中": "Aggregating regional news", + "正在同步全球态势新闻...": "Syncing global situation news...", + "正在准备全球态势新闻...": "Preparing global situation news...", + "当前新闻类型没有可显示新闻...": "No displayable news for the selected types...", + "当前区域暂无可用新闻": "No available news in this region", + "输入关键词以搜索当前地球对象": "Enter keywords to search current Earth objects", + "支持搜索海缆、登陆点、卫星、算力中心、BGP 事件与观测站。": "Search subsea cables, landing points, satellites, compute centers, BGP events, and collectors.", + "搜索海缆、登陆点、卫星、算力中心、BGP 事件...": "Search cables, landing points, satellites, compute centers, BGP events...", + "搜索海缆、登陆点、卫星、算力中心和 BGP 事件": "Search cables, landing points, satellites, compute centers, and BGP events", + "关闭搜索": "Close search", + "设置分类": "Settings categories", + "移动端设置分类": "Mobile settings categories", + "重置设置": "Reset settings", + "关闭设置": "Close settings", + "重置": "Reset", + "运行": "Runtime", + "显示": "Display", + "面板": "Panels", + "动捕": "Motion", + "快捷键": "Shortcuts", + "系统": "System", + "关于": "About", + "旋转模式": "Rotation Mode", + "巡航模式": "Cruise Mode", + "动捕模式": "Motion Mode", + "旋转、巡航和动捕是互斥运行模式": "Rotation, cruise, and motion are mutually exclusive runtime modes", + "旋转、巡航和动捕是互斥运行模式；巡航按模块轮播，动捕消费手势控制": "Rotation, cruise, and motion are mutually exclusive. Cruise rotates through modules; motion consumes gestures.", + "选择旋转模式": "Select rotation mode", + "移动端选择旋转模式": "Select rotation mode on mobile", + "旋转转速": "Rotation Speed", + "默认速率 1x": "Default speed 1x", + "默认速率 1x，调高后自动旋转更快。": "Default speed is 1x; increase it for faster auto rotation.", + "调整旋转转速": "Adjust rotation speed", + "移动端调整旋转转速": "Adjust rotation speed on mobile", + "重置旋转转速为1x": "Reset rotation speed to 1x", + "重置为1x": "Reset to 1x", + "巡航模块": "Cruise Modules", + "选择哪些可交互图层参与巡航队列。": "Choose which interactive layers join the cruise queue.", + "选择哪些可交互图层参与巡航队列。默认 BGP，其他图层按需加入。": "Choose which interactive layers join the cruise queue. BGP is default; add others as needed.", + "选择巡航模块": "Select cruise modules", + "移动端选择巡航模块": "Select cruise modules on mobile", + "新闻": "News", + "算力": "Compute", + "线缆": "Cables", + "轨迹": "Tracks", + "真实地形": "Real terrain", + "真实地形已显示": "Real terrain shown", + "电缆": "Cables", + "新闻类型": "News Types", + "自动旋转": "Auto Rotate", + "旋转": "Rotation", + "巡航": "Cruise", + "布局": "Layout", + "视角": "View", + "当前目标": "Current target", + "国界": "Borders", + "其他": "Other", + "巡航队列": "Cruise Queue", + "巡航大区顺序": "Cruise Region Order", + "控制巡航项按原始顺序、大区轮转或随机播放。": "Control whether cruise items play in original order, by region, or randomly.", + "控制巡航项按原始顺序、大区轮转或随机洗牌播放。": "Control whether cruise items play in original order, by region, or shuffled.", + "选择巡航队列模式": "Select cruise queue mode", + "移动端选择巡航队列模式": "Select cruise queue mode on mobile", + "默认": "Default", + "按大区": "By Region", + "随机": "Random", + "地球与地表": "Globe & Surface", + "日夜模式": "Day/Night Mode", + "按真实太阳位置区分地球昼夜明暗": "Use the real sun position for day/night shading", + "按真实太阳位置区分地球昼夜明暗，关闭后全球均匀照亮": "Use the real sun position for day/night shading; disabling evenly lights the globe.", + "地球默认大小": "Default Globe Size", + "用于重置视角、缩放重置和巡航视图": "Used for reset view, zoom reset, and cruise views", + "用于重置视角、缩放重置和巡航视图的默认缩放比例": "Default zoom for reset view, zoom reset, and cruise views", + "移动端调整地球默认大小": "Adjust default globe size on mobile", + "地形透明度": "Terrain Opacity", + "调高后会呈现更明显的绿色地形覆盖效果": "Higher values make the green terrain overlay more visible", + "移动端调整地形透明度": "Adjust terrain opacity on mobile", + "国界精度": "Boundary Precision", + "正在读取高清国界状态...": "Reading high-precision boundary status...", + "选择国界精度": "Select boundary precision", + "移动端选择国界精度": "Select boundary precision on mobile", + "低精": "Low", + "高精": "High", + "重新获取并构建高清国界": "Fetch and rebuild high-precision boundaries", + "重新获取并构建": "Fetch and rebuild", + "卫星显示风格": "Satellite Style", + "选择卫星锁定态使用自身发光或真实地表覆盖范围": "Choose self glow or true ground footprint for locked satellites", + "选择卫星锁定态使用自身发光，还是强调真实地表覆盖范围。": "Choose self glow for locked satellites or emphasize their true ground footprint.", + "选择卫星显示风格": "Select satellite display style", + "移动端选择卫星显示风格": "Select satellite display style on mobile", + "自身发光": "Self Glow", + "真实地表覆盖": "Ground Footprint", + "卫星呼吸闪烁": "Satellite Breathing", + "空闲时让卫星点缓慢明暗呼吸": "Let satellite dots breathe slowly while idle", + "空闲时让卫星点缓慢明暗呼吸，拖拽或缩放时自动稳定显示": "Let satellite dots breathe while idle; stabilize them while dragging or zooming.", + "真实卫星高度": "True Satellite Altitude", + "卫星真实高度": "True Satellite Altitude", + "旧版同层高度": "legacy shell altitude", + "按真实轨道高度压缩分层显示": "Compress and layer by true orbital altitude", + "按真实轨道高度压缩分层显示，关闭后使用旧版同层球面": "Compress and layer by true orbital altitude; disabling uses the legacy single shell.", + "轨迹显示": "Track Display", + "控制卫星轨迹线显示": "Control satellite track lines", + "控制卫星轨迹线显示，卫星图层关闭时会自动隐藏": "Control satellite track lines; hidden automatically when the satellite layer is off.", + "交互提示": "Interaction Hints", + "悬停提示": "Hover Hint", + "控制鼠标悬停地表时显示国家、位置或完整信息": "Choose whether surface hover shows country, position, or full details", + "控制鼠标悬停地表时显示国家、位置或完整信息。": "Choose whether surface hover shows country, position, or full details.", + "选择悬停提示内容": "Select hover hint content", + "移动端选择悬停提示内容": "Select hover hint content on mobile", + "国家": "Country", + "位置": "Position", + "完整": "Full", + "低缩放圆点": "Compact Dots", + "低缩放彩色圆点": "Compact color dots", + "150% 以下将可交互图标简化为对应颜色圆点": "Below 150%, simplify interactable icons to colored dots", + "150% 以下将可交互图标简化为对应颜色圆点，便于巡航和总览扫视": "Below 150%, simplify interactable icons to colored dots for cruise and overview scanning.", + "新闻类型": "News Types", + "只筛选当前浏览器的新闻面板与新闻巡航，不改变后台新闻源。": "Filters only this browser's news panel and news cruise; backend sources are unchanged.", + "选择新闻类型": "Select news types", + "政治": "Politics", + "商业": "Business", + "电商": "E-commerce", + "金融": "Finance", + "体育": "Sports", + "科技": "Technology", + "军事": "Military", + "灾害": "Disaster", + "能源": "Energy", + "社会": "Society", + "文化": "Culture", + "其他": "Other", + "其他电缆": "Other Cables", + "海缆系统": "Cable Systems", + "登陆点": "Landing Points", + "8K 卫星图": "8K Satellite", + "颗": "", + "个": "", + "艘": "", + "起": "", + "在轨卫星": "Orbiting Satellites", + "AIS 船只": "AIS Vessels", + "BGP 事件": "BGP Events", + "BGP 观测站": "BGP Collectors", + "运行中": "Running", + "当前无活跃事件": "No active events", + "AISStream 未连接": "AISStream Disconnected", + "AISStream 实时已连接 · 等待首批更新": "AISStream live connected · waiting for first updates", + "AISStream 正在重连": "AISStream reconnecting", + "AISStream 正在连接": "AISStream connecting", + "暂无可播放直播源": "No playable live source", + "打开 Live 新闻": "Open Live News", + "关闭 Live 新闻": "Close Live News", + "正在加载船只...": "Loading vessels...", + "正在加载登陆点...": "Loading landing points...", + "正在加载海缆...": "Loading subsea cables...", + "正在加载卫星...": "Loading satellites...", + "正在加载BGP态势...": "Loading BGP situation...", + "正在加载地球纹理...": "Loading Earth texture...", + "正在加载大气云图...": "Loading atmosphere and clouds...", + "正在加载海陆基座...": "Loading land/ocean base...", + "正在启用高清材质...": "Enabling HD texture...", + "正在加载算力中心...": "Loading compute centers...", + "正在加载线缆数据...": "Loading cable data...", + "正在加载卫星数据...": "Loading satellite data...", + "正在加载船只数据...": "Loading vessel data...", + "正在加载电缆数据...": "Loading cable data...", + "正在加载真实地形数据...": "Loading real terrain data...", + "正在渲染地形...": "Rendering terrain...", + "地形加载中...": "Terrain loading...", + "卫星加载中...": "Satellites loading...", + "国界加载中...": "Borders loading...", + "高清材质加载中...": "HD texture loading...", + "大气云图加载中...": "Atmosphere and clouds loading...", + "船只加载中...": "Vessels loading...", + "欢迎使用智能星球计划": "Welcome to Intelligent Planet", + "登录控制台并完成首次采集后，Earth 将显示实时数据层。": "After signing in to the console and completing the first collection, Earth will display live data layers.", + "后端服务在线": "Backend service online", + "登录控制台": "Sign in to Console", + "配置或确认数据源": "Configure or confirm datasources", + "触发首次采集": "Run the first collection", + "回到 Earth 查看结果": "Return to Earth to view results", + "概要": "Summary", + "当前记录": "Current Records", + "直播源": "Live Sources", + "活跃数据源": "Active Datasources", + "登录并采集数据": "Sign in and collect data", + "先浏览 Earth": "Browse Earth first", + "控制图层控制面板显示": "Control layer panel visibility", + "控制右侧图层控制面板显示": "Control the right-side layer panel", + "图例": "Legend", + "其他电缆": "Other Cables", + "赤道轨道": "Equatorial", + "低倾角轨道": "Low Incl.", + "中倾角轨道": "Mid Incl.", + "高倾角轨道": "High Incl.", + "逆行轨道": "Retrograde", + "赤道轨道（0-30°）": "Equatorial (0-30 deg)", + "低倾角轨道（30-60°）": "Low Incl. (30-60 deg)", + "中倾角轨道（60-90°）": "Mid Incl. (60-90 deg)", + "高倾角轨道（90-120°）": "High Incl. (90-120 deg)", + "逆行轨道（120-180°）": "Retrograde (120-180 deg)", + "低轨": "LEO", + "中轨": "MEO", + "高轨": "HEO", + "近地轨道": "LEO", + "低地球轨道": "LEO", + "中地球轨道": "MEO", + "高地球轨道": "HEO", + "同步轨道": "GSO", + "地球同步轨道": "GSO", + "静止轨道": "GEO", + "地球静止轨道": "GEO", + "太阳同步轨道": "SSO", + "极轨": "Polar", + "低轨（LEO）": "LEO", + "中轨（MEO）": "MEO", + "高轨（HEO）": "HEO", + "地球同步轨道（GSO）": "GSO", + "地球静止轨道（GEO）": "GEO", + "太阳同步轨道（SSO）": "SSO", + "极轨（Polar）": "Polar", + "陆地填色": "Land Fill", + "海洋填色": "Ocean Fill", + "货轮": "Cargo Ship", + "油轮": "Tanker", + "客船": "Passenger Ship", + "渔船": "Fishing Vessel", + "军舰": "Naval Vessel", + "停泊/低速": "Anchored / Slow", + "其他船只": "Other Vessels", + "中活跃观测站": "Medium-Activity Collector", + "高活跃观测站": "High-Activity Collector", + "观测范围示意": "Observation Coverage", + "事件连线 / 枢纽": "Event Links / Hubs", + "严重事件": "Critical Events", + "高危事件": "High-Risk Events", + "中危事件": "Medium-Risk Events", + "低危事件": "Low-Risk Events", + "控制图例面板显示": "Control legend panel visibility", + "控制左下角图例面板显示": "Control the lower-left legend panel", + "全球态势": "Global Situation", + "控制全球态势统计面板显示": "Control the global situation stats panel", + "控制右下角全球态势统计面板显示": "Control the lower-right global situation stats panel", + "新闻直播": "Live News", + "控制 Live 新闻面板显示": "Control the Live News panel", + "控制右上角 Live 新闻面板显示": "Control the upper-right Live News panel", + "动捕调试模式": "Motion Debug Mode", + "显示摄像头识别到的骨架连线和匹配动作": "Show detected skeleton lines and matched gestures", + "打开骨架连线面板；未匹配为红线，匹配动作后变绿": "Open the skeleton panel; unmatched lines are red and matched gestures turn green.", + "动捕输入源": "Motion Input", + "网页摄像头无需安装；Motion Agent 用于双摄或网络摄像头": "Browser camera needs no install; Motion Agent is for dual-camera or network-camera setups.", + "浏览器摄像头适合网页/SaaS；Motion Agent 适合双摄、RTSP/HTTP 和客户端": "Browser camera fits web/SaaS; Motion Agent fits dual cameras, RTSP/HTTP, and clients.", + "选择动捕输入源": "Select motion input", + "移动端选择动捕输入源": "Select motion input on mobile", + "浏览器摄像头": "Browser Camera", + "识别动作": "Recognized Gestures", + "关闭后不会触发对应星球控制": "Disabled gestures will not trigger Earth controls", + "未勾选的动作不会触发星球端控制；Motion Agent 会同步过滤这些动作": "Unchecked gestures will not trigger Earth controls; Motion Agent filters them too.", + "选择动捕识别动作": "Select recognized gestures", + "移动端选择动捕识别动作": "Select recognized gestures on mobile", + "左旋": "Rotate Left", + "右旋": "Rotate Right", + "上旋": "Rotate Up", + "下旋": "Rotate Down", + "上个焦点": "Previous Focus", + "下个焦点": "Next Focus", + "上一图层": "Previous Layer", + "下一图层": "Next Layer", + "确认": "Confirm", + "键盘控制": "Keyboard Controls", + "点击按键后按下新的快捷键；Esc 取消录入": "Click a key, then press the new shortcut. Esc cancels capture.", + "点击按键后按下新的快捷键；Esc 取消录入，冲突快捷键不会保存": "Click a key, then press the new shortcut. Esc cancels capture; conflicts are not saved.", + "恢复默认快捷键": "Restore Default Shortcuts", + "恢复默认": "Restore Defaults", + "控制台": "Console", + "打开数据源、任务和系统运维工作台": "Open the datasource, task, and operations workspace", + "文档": "Docs", + "查看使用手册与开发文档": "Read manuals and developer docs", + "语言": "Language", + "星球语言": "Earth Language", + "同步 Docs 和控制台语言偏好": "Syncs with Docs and console language preference", + "中文": "中文", + "打开控制台": "Open Console", + "访问": "Open", + "查看": "View", + "仅保留移动端仍有意义的智能星球配置": "Keeps only Earth settings that still matter on mobile", + "动作捕捉": "Motion Capture", + "查看浏览器摄像头画面、骨架连线和当前匹配动作": "View browser camera, skeleton lines, and current matched gesture", + "动捕调试画面已并入设置里的动捕模式。": "Motion debug view has moved into Motion Mode in settings.", + "频道信息": "Channel Info", + "展开查看当前频道来源、目录和补充说明": "Expand to view current channel source, catalog, and notes", + "频道摘要标签": "Channel summary tags", + "刷新频道列表": "Refresh channel list", + "访问频道官网": "Visit channel website", + "移动端新闻直播": "Mobile live news", + "移动端新闻直播和频道切换": "Mobile live news and channel switching", + "Details": "Details", + "点击地球对象后查看统一详情": "Select an Earth object to view unified details", + "对象详情": "Object Details", + "等待选择对象": "Waiting for selection", + "点击海缆、算力中心、BGP 事件或卫星后在这里查看详情。": "Select a cable, compute center, BGP event, or satellite to view details here.", + "出品方": "Producer", + "策划人": "Planners", + "产品兼开发者": "Product & Development", + "浙江大学临空智能媒体研究院": "Zhejiang University Linkong Institute of Intelligent Media", + "方兴东、黄柳青": "Fang Xingdong, Huang Liuqing", + "钱坤、张鸽、齐鹏": "Qian Kun, Zhang Ge, Qi Peng", + "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航，提供可视化观测、事件聚合与交互式探索能力。": "For intelligent media research, global situational awareness, and multi-source open-data cruise in linkong scenarios, providing visual observation, event aggregation, and interactive exploration.", + "暂无": "None", + "未选": "None selected", + "全部": "All", + "区域监测": "Regional Monitor", + "聚合源": "Aggregated Source", + "手动添加": "Manual", + "单源 Atom": "Single-source Atom", + "配置保留": "Reserved", + "单源 RSS": "Single-source RSS", + "关注": "Watch", + "突发": "Breaking", + "严重突发": "Critical Breaking", + "区域": "Regional", + "大洲": "Continent", + "亚洲": "Asia", + "欧洲": "Europe", + "非洲": "Africa", + "北美洲": "North America", + "南美洲": "South America", + "大洋洲": "Oceania", + "南极洲": "Antarctica", + "全球": "Global", + "美洲": "Americas", + "欧洲": "Europe", + "中东与非洲": "Middle East & Africa", + "亚太": "Asia Pacific", + "全球焦点": "Global Focus", + "源类型": "Source type", + "抓取通道": "Fetch channel", + "新闻来源": "News Sources", + "按大来源筛选，不影响后台抓取。": "Filter by source group; backend fetching is unchanged.", + "按新闻内容分类筛选。": "Filter by news content category.", + "暂无可筛选来源。": "No filterable sources.", + "当前区域暂无可用新闻，已完成一次聚合尝试": "No available news in this region; one aggregation attempt completed", + "当前未拉到可用新闻，请稍后刷新或切换视角区域。": "No available news yet. Refresh later or switch view region.", + "当前新闻类型没有可显示新闻。": "No displayable news for the selected types.", + "等待聚合新闻源": "Waiting for aggregated news sources", + "正在准备全球态势新闻聚合源...": "Preparing global situation news sources...", + "跟随当前视角自动聚焦": "Follows the current view automatically", + "刚刚同步": "Just synced", + "新闻聚合请求超时，请稍后重试": "News aggregation timed out. Try again later.", + "新闻聚合暂时不可用": "News aggregation is temporarily unavailable", + "态势新闻同步失败": "Failed to sync situation news", + "态势新闻已刷新": "Situation news refreshed", + "态势新闻刷新失败": "Failed to refresh situation news", + "快捷键已恢复默认": "Shortcuts restored to defaults", + "布局已最大化": "Layout maximized", + "布局已恢复": "Layout restored", + "视角已重置": "View reset", + "数据已加载": "Data loaded", + "巡航已暂停，按空格恢复": "Cruise paused. Press Space to resume.", + "巡航大区顺序已更新": "Cruise region order updated", + "候选列表已关闭，巡航已恢复": "Candidate list closed; cruise resumed", + "已暂停巡航，正在打开候选列表": "Cruise paused; opening candidate list", + "已暂停动捕目标展示，正在打开候选列表": "Motion target display paused; opening candidate list", + "切换到巡航模式后可切换卡片": "Switch to Cruise Mode to change cards", + "动捕识别动作已更新": "Motion gestures updated", + "动捕调试模式已开启": "Motion debug mode enabled", + "动捕调试模式已关闭": "Motion debug mode disabled", + "动捕调试模式将在下次进入动捕时开启": "Motion debug mode will enable next time Motion Mode starts", + "动捕调试已切换为只显示骨骼": "Motion debug switched to skeleton only", + "动捕调试已显示实时画面": "Motion debug now shows live video", + "动捕输入源已切换为 Motion Agent": "Motion input switched to Motion Agent", + "动捕输入源已切换为浏览器摄像头": "Motion input switched to browser camera", + "动捕: 已切换巡航目标": "Motion: cruise target changed", + "动捕: 当前没有可用图层": "Motion: no available layers", + "动捕: 当前没有可切换的可见图层": "Motion: no switchable visible layers", + "动捕: 已确认当前目标": "Motion: current target confirmed", + "动捕: 请先选择目标": "Motion: select a target first", + "Earth 设置已重置": "Earth settings reset", + "按键...": "Press key...", + "启用快捷键": "Enable shortcut", + "视角控制": "View Controls", + "工具": "Tools", + "向上旋转": "Rotate Up", + "向左旋转": "Rotate Left", + "向下旋转": "Rotate Down", + "向右旋转": "Rotate Right", + "关闭当前焦点菜单": "Close Current Focus Menu", + "打开搜索": "Open Search", + "打开/关闭图层面板": "Toggle Layer Panel", + "打开/关闭新闻直播": "Toggle Live News", + "暂停/恢复运行": "Pause/Resume Runtime", + "下一张巡航卡片": "Next Cruise Card", + "切换海缆": "Toggle Cables", + "切换卫星": "Toggle Satellites", + "切换算力中心": "Toggle Compute Centers", + "切换船只": "Toggle Vessels", + "切换 BGP观测": "Toggle BGP Observations", + "切换地形": "Toggle Terrain", + "切换高清材质": "Toggle HD Texture", + "切换大气云图": "Toggle Atmosphere & Clouds", + "切换国界线": "Toggle Borders", + "切换经纬线": "Toggle Graticule", + "卫星关闭时不可用": "Unavailable while Satellites are disabled", + "高清材质关闭时不可用": "Unavailable while HD texture is disabled", + "切换到动捕模式后可开启调试面板": "Switch to Motion Mode to enable the debug panel", + "卫星已切换为旧版同层高度": "Satellites switched to legacy same-layer altitude", + "关闭动捕调试面板": "Close motion debug panel", + "重置缩放到100%": "Reset zoom to 100%", + "移动端菜单": "Mobile menu", + "输入名称、地点、NORAD、ASN...": "Search by name, location, NORAD, ASN...", + "移动端搜索结果": "Mobile search results", + "选择移动端新闻直播源": "Select mobile live news source", + "搜索结果": "Search results", + "调整地球默认大小": "Adjust default Earth size", + "调整地形透明度": "Adjust terrain opacity", + "浙江大学临空智能媒体研究院 Logo": "Zhejiang University Linkong Institute of Intelligent Media Logo", + "Earth 页面发生未捕获错误": "Earth page uncaught error", + "Earth 页面发生未处理 Promise 错误": "Earth page unhandled Promise error", + "等待动捕数据": "Waiting for motion data", + "当前浏览器不支持 WebSocket": "This browser does not support WebSocket", + "无法创建 Motion Agent 连接": "Unable to create Motion Agent connection", + "Motion Agent 连接异常": "Motion Agent connection error", + "Motion Agent 未连接": "Motion Agent disconnected", + "演示模式已开启": "Demo mode enabled", + "新闻汉化待重试": "News localization pending retry", + "已汉化": "Localized", + "未知线缆": "Unknown Cable", + "未知登陆点": "Unknown Landing Point", + "未知节点": "Unknown Node", + "未知目标": "Unknown Target", + "未知错误": "Unknown error", + "请检查更新源": "Check update source", + "构建失败": "Build failed", + "已选择:": "Selected:", + "海底光缆系统": "Subsea Cable System", + "后端没有返回 JSON 状态": "Backend did not return JSON status", + "正在准备高清国界": "Preparing high-precision boundaries", + "当前使用高精国界；可重新获取并构建。": "Using high-precision boundaries; you can fetch and rebuild.", + "高精国界已就绪，切到高精会立即应用。": "High-precision boundaries are ready and can be applied immediately.", + "当前使用低精国界；切到高精会下载并构建。": "Using low-precision boundaries; switching to high will download and build.", + "高精国界已下载并应用": "High-precision boundaries downloaded and applied", + "正在启动高精国界构建": "Starting high-precision boundary build", + "高精国界构建已启动": "High-precision boundary build started", + "已切换到高精国界": "Switched to high-precision boundaries", + "高清国界进度读取失败": "Failed to read high-precision boundary progress", + "高清国界状态读取失败": "Failed to read high-precision boundary status", + "高精国界切换失败": "Failed to switch high-precision boundaries", + "高精国界重建启动失败": "Failed to start high-precision boundary rebuild", + "低精国界切换失败": "Failed to switch low-precision boundaries", + "已切换到低精国界": "Switched to low-precision boundaries", + "详情": "Details", + "关闭详情": "Close details", + "电缆详情": "Cable Details", + "登陆点详情": "Landing Point Details", + "卫星详情": "Satellite Details", + "BGP事件详情": "BGP Event Details", + "新闻事件详情": "News Event Details", + "BGP观测站详情": "BGP Collector Details", + "超算中心详情": "Supercomputer Details", + "GPU集群详情": "GPU Cluster Details", + "船只详情": "Vessel Details", + "交互点详情": "Interactable Details", + "交互点": "Interactable", + "数据点": "Data Point", + "名称": "Name", + "所有者": "Owner", + "状态": "Status", + "长度": "Length", + "经纬度": "Coordinates", + "海拔": "Altitude", + "投入使用": "Ready for Service", + "关联海缆数": "Related Cable Count", + "关联海缆": "Related Cables", + "星座/分组": "Constellation / Group", + "覆盖能力": "Coverage Capability", + "当前显示": "Current Display", + "覆盖模型": "Coverage Model", + "倾角": "Inclination", + "周期": "Period", + "分钟": "min", + "近地点高度": "Perigee Altitude", + "远地点高度": "Apogee Altitude", + "事件类型": "Event Type", + "严重度": "Severity", + "事件特征": "Event Feature", + "前缀": "Prefix", + "传播路径": "AS Path", + "涉及 ASN": "Origin ASN", + "关联 ASN": "Related ASN", + "置信度": "Confidence", + "主观测站": "Primary Collector", + "观测范围": "Observation Scope", + "影响区域": "Impact Region", + "附近基础设施": "Nearby Infrastructure", + "附近卫星": "Nearby Satellites", + "观测位置": "Observation Location", + "事件时间": "Event Time", + "摘要": "Summary", + "媒体来源": "Media Source", + "source": "Source", + "name": "Name", + "country": "Country", + "country_code": "Country Code", + "flag": "Flag", + "type": "Type", + "vessel_type": "Vessel Type", + "width": "Width", + "gross_tonnage": "Gross Tonnage", + "deadweight": "Deadweight", + "built_year": "Built Year", + "operator": "Operator", + "owner": "Owner", + "status": "Status", + "last_updated": "Last Updated", + "发布时间": "Published", + "发生地": "Location", + "原文链接": "Source Link", + "采集器": "Collector", + "当前事件数": "Current Events", + "观测事件数": "Observed Events", + "近24h事件数": "24h Events", + "近7d事件数": "7d Events", + "观测前缀数": "Observed Prefixes", + "观测 ASN 数": "Observed ASNs", + "主要事件类型": "Top Event Types", + "日常活跃度": "Daily Activity", + "最近事件类型": "Latest Event Type", + "最近活跃时间": "Latest Activity", + "日常覆盖范围": "Baseline Scope", + "类型": "Type", + "排名": "Rank", + "实测算力": "Measured Compute", + "估算算力": "Estimated Compute", + "厂商": "Vendor", + "运营方": "Operator", + "核心数": "Cores", + "功耗": "Power", + "城市": "City", + "位置精度": "Location Precision", + "位置来源": "Location Source", + "位置置信度": "Location Confidence", + "核验状态": "Verification", + "解析依据": "Evidence", + "位置来源说明": "Location Source Note", + "匹配的位置名称": "Matched Location", + "位置核验时间": "Verified At", + "更新时间": "Updated", + "GPU 数量": "GPU Count", + "GPU 型号": "GPU Model", + "芯片/平台": "Chip / Platform", + "旗帜": "Flag", + "船型": "Vessel Type", + "当前航速": "Speed", + "航向": "Course", + "船长": "Length", + "标识": "ID", + "纬度": "Latitude", + "经度": "Longitude", + "说明": "Description", + "字段来源": "Field Source", + "新闻信号": "News Signal", + "船舶资料": "Vessel Profile", + "资料缓存中": "Profile caching", + "资料": "Profile", + "媒体": "Media", + "无可复制内容": "Nothing to copy", + "复制失败": "Copy failed", + "未知来源": "Unknown Source", + "不适用": "N/A", + "Starlink 单星地表覆盖": "Starlink single-satellite ground footprint", + "Iridium 外圈半透明覆盖": "Iridium translucent outer coverage ring", + "锚泊": "At anchor", + "停靠": "Moored", + "航行中": "Under way", + "重新自动采集坐标": "Recollect Coordinates", + "自动采集坐标候选": "Collect Coordinate Candidates", + "采集坐标候选": "Collect coordinate candidates", + "一键定位并采用最高置信候选": "Locate all and apply highest-confidence candidates", + "一键定位": "Locate All", + "采集": "Collected", + "正在检查 WebSearch 状态，稍候即可定位。": "Checking WebSearch status; location will be available shortly.", + "WebSearch 未开启，无法进行事实核查定位。": "WebSearch is disabled, so fact-checked location is unavailable.", + "精确": "Precise", + "站点": "Site", + "候选": "Candidate", + "预览": "Preview", + "保存": "Save", + "当前没有待定位算力中心": "No compute centers need location", + "缺少可用地址字段": "Missing usable address fields", + "待定位算力中心": "Compute Centers Pending Location", + "未命名算力中心": "Unnamed Compute Center", + "超级计算机": "Supercomputer", + "GPU集群": "GPU Cluster", + "BGP路由异常": "BGP Route Anomaly", + "态势新闻": "Situation News", + "超算": "Supercomputer", + "超算中心": "Supercomputer", + "GPU 集群": "GPU Cluster", + "BGP事件": "BGP Event", + "未知": "Unknown", + "未知国家": "Unknown Country", + "未知海缆": "Unknown Cable", + "未知卫星": "Unknown Satellite", + "未知船只": "Unknown Vessel", + "未知观测站": "Unknown Collector", + "未知算力中心": "Unknown Compute Center", + "精确坐标": "Precise Coordinates", + "站点级位置": "Site-Level Location", + "城市级位置": "City-Level Location", + "位置未确认": "Location Unconfirmed", + "源数据自带坐标": "Source Coordinates", + "ROR 组织注册 API": "ROR Registry API", + "Nominatim 在线搜索": "Nominatim Online Search", + "LLM factcheck 兜底": "LLM Factcheck Fallback", + "待人工核验": "Needs Review", + "估算位置": "Estimated Location", + "已确认": "Confirmed", + "严重": "Critical", + "高": "High", + "中": "Medium", + "低": "Low", + "前缀劫持": "Prefix Hijack", + "路由泄露": "Route Leak", + "大规模撤销": "Mass Withdrawal", + "更具体前缀异常": "More-Specific Prefix Anomaly", + "路径突变": "Path Change", + "路由抖动": "Route Flap", + "活跃": "Active", + "已恢复": "Resolved", + "已抑制": "Suppressed", + "在线": "Online", + "离线": "Offline", + "静态观测站": "Static Collector", + "附近登陆点": "Nearby Landing Point", + "支持 Starlink 地表覆盖": "Starlink ground footprint supported", + "支持 Iridium 外圈覆盖": "Iridium outer ring supported", + "默认不显示 footprint": "Footprint hidden by default", + "真实地表覆盖（Starlink）": "Ground Footprint (Starlink)", + "真实地表覆盖（Iridium 外圈）": "Ground Footprint (Iridium outer ring)", + "中国": "China", + "美国": "United States", + "英国": "United Kingdom", + "法国": "France", + "德国": "Germany", + "日本": "Japan", + "韩国": "South Korea", + "新加坡": "Singapore", + "印度": "India", + "俄罗斯": "Russia", + "加拿大": "Canada", + "澳大利亚": "Australia", + "荷兰": "Netherlands", + "瑞士": "Switzerland", + "意大利": "Italy", + "西班牙": "Spain", + "瑞典": "Sweden", + "挪威": "Norway", + "芬兰": "Finland", + "丹麦": "Denmark", + "巴西": "Brazil", + "墨西哥": "Mexico", + "南非": "South Africa", + "阿联酋": "United Arab Emirates", + "沙特阿拉伯": "Saudi Arabia", + "以色列": "Israel", + "中国台湾": "Taiwan", + "台湾": "Taiwan", + "中国香港": "Hong Kong", + "香港": "Hong Kong", + "正在初始化...": "Initializing...", + "算力中心坐标已保存": "Compute center coordinates saved", + "坐标已保存，地图稍后同步": "Coordinates saved; map will sync shortly", + "坐标已保存，正在同步地图...": "Coordinates saved; syncing map...", +}; + +const PATTERN_TRANSLATIONS = [ + [/^切换(.+)显示$/, (match, locale) => `Toggle ${translateText(match[1], locale)} display`], + [/^(\d+) 项$/, "$1"], + [/^(\d+) 条$/, "$1"], + [/^(\d+)个$/, "$1"], + [/^(\d+) 个$/, "$1"], + [/^(\d+) 颗$/, "$1"], + [/^(\d+)颗$/, "$1"], + [/^(\d+) 艘$/, "$1"], + [/^(\d+) 起$/, "$1"], + [/^(\d+)\/(\d+) 运行中$/, "$1/$2 running"], + [/^(\d+) 起活跃事件$/, "$1 active events"], + [/^(\d+) 起活跃事件 \/ (\d+) 条异常$/, "$1 active events / $2 anomalies"], + [/^(\d+) 条活跃异常$/, "$1 active anomalies"], + [/^(.+) \((\d+) pending location\)$/, (match, locale) => `${translateText(match[1], locale)} (${match[2]} pending location)`], + [/^(.+) · (.+)$/, (match, locale) => `${translateText(match[1], locale)} · ${translateText(match[2], locale)}`], + [/^暂停(.+)$/, (match, locale) => `Pause ${translateText(match[1], locale)}`], + [/^开始(.+)$/, (match, locale) => `Start ${translateText(match[1], locale)}`], + [/^共 (\d+) 个频道 · 尚未同步$/, "$1 channels · Not synced yet"], + [/^共 (\d+) 个频道 · 最近同步 (.+)$/, "$1 channels · Last synced $2"], + [/^采集源：(.+)$/, "Collector source: $1"], + [/^采集:(.+)$/, "Collector: $1"], + [/^成功加载 (\d+) 条电缆$/, "Loaded $1 cables"], + [/^成功加载 (\d+) 个登陆点$/, "Loaded $1 landing points"], + [/^已锁定: (.+)$/, "Locked: $1"], + [/^已启用 (\d+) 个图层$/, "$1 layers enabled"], + [/^HLS 主播放列表加载失败（HTTP (\d+)）$/, "Failed to load HLS master playlist (HTTP $1)"], + [/^HLS 清晰度播放列表加载失败（HTTP (\d+)）$/, "Failed to load HLS variant playlist (HTTP $1)"], + [/^HLS 分片加载失败（HTTP (\d+)）$/, "Failed to load HLS segment (HTTP $1)"], + [/^HLS 主播放列表加载失败$/, "Failed to load HLS master playlist"], + [/^HLS 清晰度播放列表加载失败$/, "Failed to load HLS variant playlist"], + [/^HLS 分片加载失败$/, "Failed to load HLS segment"], + [/^HLS 资源加载失败：(.+)$/, "Failed to load HLS resource: $1"], + [/^直播流连接异常，正在重试 \((\d+)\/(\d+)\)\.\.\.$/, "Live stream connection issue. Retrying ($1/$2)..."], + [/^直播流解码异常，正在恢复 \((\d+)\/(\d+)\)\.\.\.$/, "Live stream decode issue. Recovering ($1/$2)..."], + [/^失败：(.*)$/, "Failed: $1"], + [/^高清国界更新源未配置完整：(.*)$/, "High-precision boundary sources are incomplete: $1"], + [/^高清国界下载失败：(.*)$/, "High-precision boundary download failed: $1"], + [/^高清国界进度读取失败：(.*)$/, "Failed to read high-precision boundary progress: $1"], + [/^高清国界状态读取失败：(.*)$/, "Failed to read high-precision boundary status: $1"], + [/^高精国界切换失败：(.*)$/, "Failed to switch high-precision boundaries: $1"], + [/^高精国界重建启动失败：(.*)$/, "Failed to start high-precision boundary rebuild: $1"], + [/^低精国界切换失败：(.*)$/, "Failed to switch low-precision boundaries: $1"], + [/^快捷键已被「(.+)」使用$/, "Shortcut is already used by \"$1\""], + [/^(.+)已显示$/, (match, locale) => `${translateText(match[1], locale)} shown`], + [/^(.+)已隐藏$/, (match, locale) => `${translateText(match[1], locale)} hidden`], + [/^(.+)已启用$/, (match, locale) => `${translateText(match[1], locale)} enabled`], + [/^(.+)已开启$/, (match, locale) => `${translateText(match[1], locale)} enabled`], + [/^(.+)已关闭$/, (match, locale) => `${translateText(match[1], locale)} disabled`], + [/^(.+)已暂停$/, (match, locale) => `${translateText(match[1], locale)} paused`], + [/^(.+)已恢复$/, (match, locale) => `${translateText(match[1], locale)} resumed`], + [/^(.+)当前不可用$/, (match, locale) => `${translateText(match[1], locale)} unavailable`], + [/^(.+)暂时不可用$/, (match, locale) => `${translateText(match[1], locale)} temporarily unavailable`], + [/^正在加载(.+)数据\.\.\.$/, (match, locale) => `Loading ${translateText(match[1], locale).toLowerCase()} data...`], + [/^正在加载(.+)\.\.\.$/, (match, locale) => `Loading ${translateText(match[1], locale).toLowerCase()}...`], + [/^(.+)加载失败: (.+)$/, (match, locale) => `${translateText(match[1], locale)} load failed: ${match[2]}`], + [/^已定位(.+)：(.+)$/, (match, locale) => `Located ${translateText(match[1], locale).toLowerCase()}: ${match[2]}`], + [/^已选择(.+): (.+)$/, (match, locale) => `Selected ${translateText(match[1], locale).toLowerCase()}: ${match[2]}`], + [/^(.+)已切换为：(.+)$/, (match, locale) => `${translateText(match[1], locale)} switched to: ${translateText(match[2], locale)}`], + [/^(.+)已设置为 (.+)$/, "$1 set to $2"], + [/^(.+)已设为 (.+)$/, (match, locale) => `${translateText(match[1], locale)} set to ${translateText(match[2], locale)}`], + [/^重置(.+)$/, "Reset $1"], + [/^已切换到(.+)$/, (match, locale) => `Switched to ${translateText(match[1], locale)}`], + [/^最近同步 (.+)$/, "Last synced $1"], + [/^缩放 (\d+)%$/, "Zoom $1%"], + [/^AISStream 实时已连接 · (\d+) 次更新$/, "AISStream live connected · $1 updates"], + [/^AISStream 实时已连接 · (\d+) 次更新 · (.+)$/, "AISStream live connected · $1 updates · $2"], + [/^AISStream 后台已连接$/, "AISStream backend connected"], + [/^AISStream 后台已连接 · 最近 (.+)$/, "AISStream backend connected · last seen $1"], + [/^(\d+) 秒前$/, "$1s ago"], + [/^(\d+) 分钟前$/, "$1m ago"], + [/^已选择BGP事件: (.+) · (\d+)个区域 \/ (\d+)条相关海缆$/, "Selected BGP event: $1 · $2 regions / $3 related cables"], + [/^已选择观测站: (.+)$/, "Selected collector: $1"], + [/^新闻源请求失败: (\d+)$/, "News request failed: $1"], + [/^来源 (.+)$/, "Source $1"], + [/^更新 (.+)$/, "Updated $1"], + [/^置信 (.+)$/, "Confidence $1"], + [/^已复制(.+)：(.+)$/, "Copied $1: $2"], + [/^已选择: (.+)$/, "Selected: $1"], + [/^动捕: (.+)当前视野没有可选目标$/, (match, locale) => `Motion: ${translateText(match[1], locale)} has no selectable targets in view`], + [/^动捕: 已切换到(.+)图层$/, (match, locale) => `Motion: switched to ${translateText(match[1], locale)} layer`], + [/^动捕: 已切换到(.+)$/, (match, locale) => `Motion: switched to ${translateText(match[1], locale)}`], + [/^动捕: 已确认(.+)$/, (match, locale) => `Motion: confirmed ${translateText(match[1], locale)}`], + [/^共找到 (\d+) 个候选位置$/, "Found $1 candidate locations"], + [/^(\d+) 个算力中心没有可信坐标$/, "$1 compute centers lack trusted coordinates"], + [/^(\d+) 个待定位$/, "$1 pending location"], + [/^正在定位并采用最高置信候选 (\d+)\/(\d+)\.\.\.$/, "Locating and applying highest-confidence candidate $1/$2..."], + [/^一键定位进行中 (\d+)\/(\d+)\.\.\.$/, "Batch location in progress $1/$2..."], + [/^找到 (\d+) 个候选，正在保存最高置信位置\.\.\.$/, "Found $1 candidates; saving highest-confidence location..."], + [/^坐标已保存，等待图层刷新$/, "Coordinates saved; waiting for layer refresh"], + [/^坐标已保存$/, "Coordinates saved"], + [/^正在保存所选坐标\.\.\.$/, "Saving selected coordinates..."], + [/^正在采集坐标候选\.\.\.$/, "Collecting coordinate candidates..."], + [/^采集失败：(.+)$/, "Collection failed: $1"], + [/^保存失败：(.+)$/, "Save failed: $1"], + [/^未能采集到坐标：(.+)$/, "Could not collect coordinates: $1"], + [/^未找到可采用候选：(.+)$/, "No applicable candidate found: $1"], + [/^未找到包含有效经纬度的候选$/, "No candidate includes valid coordinates"], + [/^常规来源没有可用坐标候选$/, "No coordinate candidates from regular sources"], + [/^常规来源无结果；LLM 兜底未生成可用候选：(.+)$/, "Regular sources returned no result; LLM fallback produced no usable candidate: $1"], + [/^常规来源无结果；LLM 兜底已尝试但没有返回可用候选。(.+)$/, "Regular sources returned no result; LLM fallback was attempted but returned no usable candidate. $1"], + [/^(\d+)个观测站 \((.+)\)$/, "$1 collectors ($2)"], + [/^(.+) 等(\d+)地$/, "$1 + $2 regions"], + [/^(.+) 等(\d+)处$/, "$1 + $2 sites"], + [/^近24h (\d+)条事件 \/ (\d+)个前缀$/, "24h $1 events / $2 prefixes"], + [/^一键定位进行中 (\d+)\/(\d+)，已保存 (\d+) 个，失败 (\d+) 个$/, "Batch location in progress $1/$2, saved $3, failed $4"], + [/^一键定位进行中 (\d+)\/(\d+)，已保存 (\d+) 个$/, "Batch location in progress $1/$2, saved $3"], + [/^已定位并采用 (\d+) 个最高置信候选，(\d+) 个仍需手动处理$/, "Applied $1 highest-confidence candidates; $2 still need manual review"], + [/^已定位并采用 (\d+) 个最高置信候选$/, "Applied $1 highest-confidence candidates"], + [/^(\d+) 个都没有可自动采用的候选，需要手动处理$/, "$1 items need manual handling; no automatic candidates were usable"], + [/^一键定位失败：(.+)$/, "Batch location failed: $1"], + [/^一键采用失败：(.+)$/, "Batch apply failed: $1"], + [/^未找到可采用候选$/, "No applicable candidate found"], +]; + +const SKIP_SELECTOR = [ + "script", + "style", + "svg", + "canvas", + "video", + "iframe", + ".material-symbols-rounded", +].join(","); + +const TRANSLATABLE_ATTRIBUTES = ["title", "aria-label", "placeholder", "alt"]; + +let activeLocale = readStoredLocale(); +let observer = null; +let translating = false; + +function normalizeText(value) { + return String(value ?? "").replace(/\s+/g, " ").trim(); +} + +export function hasCjkText(value) { + return /[\u4e00-\u9fff]/.test(String(value ?? "")); +} + +export function normalizeLocale(value) { + const normalized = String(value || "").toLowerCase(); + if (normalized === "en" || normalized === "en-us" || normalized.startsWith("en-")) return "en-US"; + if (normalized === "zh" || normalized === "zh-cn" || normalized.startsWith("zh-")) return "zh-CN"; + return DEFAULT_LOCALE; +} + +export function readStoredLocale() { + try { + const storedLocale = window.localStorage?.getItem(LOCALE_STORAGE_KEY); + if (storedLocale) return normalizeLocale(storedLocale); + const docsLang = window.localStorage?.getItem(LEGACY_DOCS_LANG_STORAGE_KEY); + if (docsLang === "en") return "en-US"; + if (docsLang === "zh") return "zh-CN"; + } catch { + // Ignore storage failures and fall back to the default language. + } + return DEFAULT_LOCALE; +} + +export function getEarthLocale() { + return activeLocale; +} + +export function docsLangFromLocale(locale) { + return normalizeLocale(locale) === "en-US" ? "en" : "zh"; +} + +export function getEarthLocaleLabel(locale = activeLocale) { + return LOCALE_LABELS[normalizeLocale(locale)] || LOCALE_LABELS[DEFAULT_LOCALE]; +} + +function syncLocaleSegmentedSliders() { + document.querySelectorAll(".earth-settings-segmented, .earth-mobile-settings-segmented").forEach((segmented) => { + if (!(segmented instanceof HTMLElement)) return; + const buttons = Array.from(segmented.querySelectorAll("[data-earth-locale]")); + if (buttons.length === 0) return; + const activeIndex = Math.max(0, buttons.findIndex((button) => button.classList.contains("is-active"))); + segmented.style.setProperty("--item-count", String(buttons.length)); + segmented.style.setProperty("--active-index", String(activeIndex)); + }); +} + +export function translateText(value, locale = activeLocale) { + const text = normalizeText(value); + if (!text || normalizeLocale(locale) === "zh-CN") return text; + if (EXACT_TRANSLATIONS[text]) return EXACT_TRANSLATIONS[text]; + for (const [pattern, replacement] of PATTERN_TRANSLATIONS) { + const match = text.match(pattern); + if (!match) continue; + if (typeof replacement === "function") return replacement(match, locale); + return text.replace(pattern, replacement); + } + return text; +} + +export function earthMessage(key, params = {}) { + return { + [EARTH_MESSAGE_MARKER]: true, + key, + params, + }; +} + +export function isEarthMessage(value) { + return Boolean(value && typeof value === "object" && value[EARTH_MESSAGE_MARKER] === true); +} + +function interpolateEarthMessage(template, params = {}, locale = activeLocale) { + return String(template).replace(/\{([A-Za-z0-9_]+)\}/g, (_, key) => { + const value = params[key]; + return value === undefined || value === null ? "" : String(value); + }); +} + +export function formatEarthMessage(value, locale = activeLocale) { + if (!isEarthMessage(value)) { + return translateText(value, locale); + } + + const normalizedLocale = normalizeLocale(locale); + const localeKey = normalizedLocale === "en-US" ? "en" : "zh"; + const template = EARTH_MESSAGE_TEMPLATES[value.key]; + if (!template) return translateText(value.key, normalizedLocale); + const params = value.params || {}; + const renderer = template[localeKey] || template.zh || template.en; + if (typeof renderer === "function") { + return normalizeText(renderer(params, normalizedLocale)); + } + return normalizeText(interpolateEarthMessage(renderer, params, normalizedLocale)); +} + +export function localizeCountryName(value, locale = activeLocale) { + const normalizedLocale = normalizeLocale(locale); + if (!value) return ""; + if (typeof value === "object") { + const source = value || {}; + if (normalizedLocale === "en-US") { + return normalizeText( + source.nameEn || + source.name || + source.NAME_EN || + source.ADMIN || + source.nameZh || + source.NAME_ZH || + "", + ); + } + return normalizeText( + source.nameZh || + source.NAME_ZH || + source.name || + source.nameEn || + source.NAME_EN || + source.ADMIN || + "", + ); + } + return translateText(value, locale); +} + +export function formatLocaleDateTime(value, options = {}) { + const date = value instanceof Date ? value : new Date(value); + if (Number.isNaN(date.getTime())) return ""; + return date.toLocaleString(activeLocale, { hour12: false, ...options }); +} + +function rememberOriginalText(node) { + if (!node.parentElement) return ""; + const originals = node.parentElement.__planetEarthI18nOriginalText || new WeakMap(); + if (!node.parentElement.__planetEarthI18nOriginalText) { + node.parentElement.__planetEarthI18nOriginalText = originals; + } + const current = normalizeText(node.nodeValue); + if (!originals.has(node)) { + originals.set(node, current); + return current; + } + const original = originals.get(node); + const translated = translateText(original, activeLocale); + const currentLooksSource = hasCjkText(current); + if (current !== translated && current !== original && currentLooksSource) { + originals.set(node, current); + return current; + } + return original; +} + +function translateTextNode(node) { + if (!node.parentElement || node.parentElement.closest(SKIP_SELECTOR)) return; + const original = rememberOriginalText(node); + if (!original) return; + const translated = translateText(original, activeLocale); + if (node.nodeValue !== translated) { + node.nodeValue = translated; + } +} + +function rememberOriginalAttribute(element, attribute) { + const dataKey = `i18nOriginal${attribute + .replace(/(^|-)([a-z])/g, (_, __, letter) => letter.toUpperCase())}`; + const current = normalizeText(element.getAttribute(attribute)); + if (!element.dataset[dataKey]) { + element.dataset[dataKey] = current; + return current; + } + const original = element.dataset[dataKey]; + const translated = translateText(original, activeLocale); + const currentLooksSource = hasCjkText(current); + if (current && current !== translated && current !== original && currentLooksSource) { + element.dataset[dataKey] = current; + return current; + } + return original; +} + +function translateAttributes(element) { + TRANSLATABLE_ATTRIBUTES.forEach((attribute) => { + if (!element.hasAttribute(attribute)) return; + const original = rememberOriginalAttribute(element, attribute); + if (!original) return; + const translated = translateText(original, activeLocale); + if (element.getAttribute(attribute) !== translated) { + element.setAttribute(attribute, translated); + } + }); +} + +function translateElementTree(root) { + if (!(root instanceof Element)) return; + if (root.closest(SKIP_SELECTOR)) return; + translateAttributes(root); + root.querySelectorAll("*").forEach((element) => { + if (element instanceof Element && !element.closest(SKIP_SELECTOR)) { + translateAttributes(element); + } + }); + const walker = document.createTreeWalker(root, NodeFilter.SHOW_TEXT, { + acceptNode(node) { + if (!normalizeText(node.nodeValue)) return NodeFilter.FILTER_REJECT; + if (node.parentElement?.closest(SKIP_SELECTOR)) return NodeFilter.FILTER_REJECT; + return NodeFilter.FILTER_ACCEPT; + }, + }); + const nodes = []; + while (walker.nextNode()) nodes.push(walker.currentNode); + nodes.forEach(translateTextNode); +} + +function syncDocumentLocale() { + document.documentElement.lang = activeLocale; + document.title = translateText("智能星球计划 - 现实层宇宙全息感知", activeLocale); +} + +function persistLocale(locale) { + try { + window.localStorage?.setItem(LOCALE_STORAGE_KEY, locale); + window.localStorage?.setItem(LEGACY_DOCS_LANG_STORAGE_KEY, docsLangFromLocale(locale)); + } catch { + // Ignore storage failures; the current page still changes language. + } +} + +export function applyEarthI18n(root = document.body) { + if (!root) return; + if (translating) return; + translating = true; + try { + syncDocumentLocale(); + if (root === document || root === document.body || root instanceof Document) { + translateElementTree(document.body); + } else if (root instanceof Element) { + translateElementTree(root); + } + syncEarthLocaleControls(); + } finally { + translating = false; + } +} + +function observeMutations() { + if (observer || !document.body) return; + observer = new MutationObserver((mutations) => { + if (translating) return; + const roots = new Set(); + let attributeChanged = false; + mutations.forEach((mutation) => { + if (mutation.type === "childList") { + mutation.addedNodes.forEach((node) => { + if (node instanceof Element) roots.add(node); + else if (node.parentElement) roots.add(node.parentElement); + }); + } else if (mutation.type === "characterData" && mutation.target.parentElement) { + roots.add(mutation.target.parentElement); + } else if (mutation.type === "attributes" && mutation.target instanceof Element) { + roots.add(mutation.target); + attributeChanged = true; + } + }); + if (roots.size === 0 && !attributeChanged) return; + roots.forEach((root) => applyEarthI18n(root)); + }); + observer.observe(document.body, { + attributes: true, + attributeFilter: TRANSLATABLE_ATTRIBUTES, + childList: true, + characterData: true, + subtree: true, + }); +} + +export function setEarthLocale(nextLocale, { persist = true, announce = true } = {}) { + const normalized = normalizeLocale(nextLocale); + if (normalized === activeLocale) { + applyEarthI18n(); + return activeLocale; + } + activeLocale = normalized; + if (persist) persistLocale(normalized); + applyEarthI18n(); + window.dispatchEvent(new CustomEvent(LOCALE_CHANGE_EVENT, { detail: { locale: activeLocale } })); + if (announce) { + window.dispatchEvent( + new CustomEvent("earth:status", { + detail: { message: earthMessage("status.languageSwitched") }, + }), + ); + } + return activeLocale; +} + +export function syncEarthLocaleControls() { + document.querySelectorAll("[data-earth-locale]").forEach((button) => { + if (!(button instanceof HTMLButtonElement)) return; + const locale = normalizeLocale(button.dataset.earthLocale); + const active = locale === activeLocale; + button.classList.toggle("is-active", active); + button.setAttribute("aria-pressed", active ? "true" : "false"); + }); + syncLocaleSegmentedSliders(); + document.querySelectorAll("[data-earth-locale-switch]").forEach((input) => { + if (!(input instanceof HTMLInputElement)) return; + const checked = activeLocale === "en-US"; + input.checked = checked; + input.setAttribute("aria-checked", checked ? "true" : "false"); + input.closest(".earth-settings-switch, .earth-mobile-settings-switch") + ?.classList.toggle("is-checked", checked); + }); +} + +export function setupEarthLocaleControls(root = document) { + root.querySelectorAll("[data-earth-locale]").forEach((button) => { + if (!(button instanceof HTMLButtonElement) || button.dataset.localeBound === "true") return; + button.dataset.localeBound = "true"; + button.addEventListener("click", () => { + setEarthLocale(button.dataset.earthLocale || DEFAULT_LOCALE); + }); + }); + root.querySelectorAll("[data-earth-locale-switch]").forEach((input) => { + if (!(input instanceof HTMLInputElement) || input.dataset.localeBound === "true") return; + input.dataset.localeBound = "true"; + input.addEventListener("change", () => { + setEarthLocale(input.checked ? "en-US" : "zh-CN"); + }); + }); + syncEarthLocaleControls(); +} + +export function onEarthLocaleChange(callback) { + const handler = (event) => callback(event.detail?.locale || activeLocale); + window.addEventListener(LOCALE_CHANGE_EVENT, handler); + return () => window.removeEventListener(LOCALE_CHANGE_EVENT, handler); +} + +export function initEarthI18n() { + activeLocale = readStoredLocale(); + if (document.readyState === "loading") { + document.addEventListener("DOMContentLoaded", () => { + applyEarthI18n(); + setupEarthLocaleControls(); + observeMutations(); + }, { once: true }); + } else { + applyEarthI18n(); + setupEarthLocaleControls(); + observeMutations(); + } + window.addEventListener("storage", (event) => { + if (event.key !== LOCALE_STORAGE_KEY && event.key !== LEGACY_DOCS_LANG_STORAGE_KEY) return; + setEarthLocale(readStoredLocale(), { persist: false, announce: false }); + }); +} diff --git a/frontend/public/earth/js/info-card.js b/frontend/public/earth/js/info-card.js index ebcaee8f..8e8c3bb2 100644 --- a/frontend/public/earth/js/info-card.js +++ b/frontend/public/earth/js/info-card.js @@ -1,5 +1,12 @@ // info-card.js - Unified info card module import { showStatusMessage } from './ui.js'; +import { + earthMessage, + getEarthLocale, + hasCjkText, + localizeCountryName, + translateText, +} from './i18n.js'; import { getNewsDisplaySummary, getNewsDisplayTitle, @@ -14,6 +21,16 @@ let mobileDetailsListenerBound = false; let renderedMobileDetailKey = null; const locationCollectStateCache = new Map(); const locationCollectContextCache = new Map(); +let computeCenterLocationCapability = null; +let computeCenterLocationCapabilityPromise = null; +let computeCenterUnresolvedBatchState = { + running: false, + statusText: '', + total: 0, + processed: 0, + saved: 0, + missed: 0, +}; // Latest candidate list per cache-key. Populated whenever state.candidates is // updated, and read by the click handler via `data-candidate-index` so we // never have to round-trip a candidate object through an HTML attribute. @@ -74,6 +91,82 @@ function clearLocationCollectState(contextOrKey) { updateLocationCollectDomFromState(key); } +function getWebSearchDisabledTitle(capability = computeCenterLocationCapability) { + if (!capability) return infoText('正在检查 WebSearch 状态，稍候即可定位。'); + return infoText(capability.reason || 'WebSearch 未开启，无法进行事实核查定位。'); +} + +function isComputeCenterLocationBlocked() { + return computeCenterLocationCapability?.enabled !== true; +} + +async function ensureComputeCenterLocationCapability() { + if (computeCenterLocationCapability) return computeCenterLocationCapability; + if (!computeCenterLocationCapabilityPromise) { + computeCenterLocationCapabilityPromise = import('./compute-centers.js') + .then((mod) => mod.getComputeCenterLocationCapability()) + .then((capability) => { + computeCenterLocationCapability = capability; + updateComputeCenterLocationCapabilityDom(); + return capability; + }) + .catch((error) => { + computeCenterLocationCapability = { + enabled: false, + reason: `WebSearch 状态检查失败：${error?.message || error}`, + }; + updateComputeCenterLocationCapabilityDom(); + return computeCenterLocationCapability; + }); + } + return computeCenterLocationCapabilityPromise; +} + +function updateComputeCenterLocationCapabilityDom() { + const blocked = isComputeCenterLocationBlocked(); + const title = getWebSearchDisabledTitle(); + document + .querySelectorAll('[data-requires-web-search="true"]') + .forEach((button) => { + if (!(button instanceof HTMLButtonElement)) return; + button.disabled = blocked || button.classList.contains('is-loading'); + button.classList.toggle('is-websearch-blocked', blocked); + button.title = blocked ? title : button.dataset.readyTitle || button.title || ''; + button.setAttribute('aria-disabled', blocked ? 'true' : 'false'); + }); +} + +function setLocationButtonLoading(button, loading, label) { + if (!(button instanceof HTMLButtonElement)) return; + button.classList.toggle('is-loading', Boolean(loading)); + button.toggleAttribute('aria-busy', Boolean(loading)); + if (label) { + button.dataset.loadingLabel = infoText(label); + } + updateComputeCenterLocationCapabilityDom(); +} + +function updateUnresolvedBatchDom() { + document.querySelectorAll('[data-unresolved-batch-status]').forEach((statusEl) => { + statusEl.textContent = computeCenterUnresolvedBatchState.statusText || ''; + }); + document.querySelectorAll('[data-unresolved-adopt-all]').forEach((button) => { + if (!(button instanceof HTMLButtonElement)) return; + button.classList.toggle('is-loading', computeCenterUnresolvedBatchState.running); + button.toggleAttribute('aria-busy', computeCenterUnresolvedBatchState.running); + }); + updateComputeCenterLocationCapabilityDom(); +} + +function setUnresolvedBatchState(patch = {}) { + computeCenterUnresolvedBatchState = { + ...computeCenterUnresolvedBatchState, + ...patch, + }; + updateUnresolvedBatchDom(); + return computeCenterUnresolvedBatchState; +} + function getCandidateForButton(button) { if (!(button instanceof HTMLElement)) return null; const root = button.closest('[data-collect-cache-key]'); @@ -102,14 +195,27 @@ function formatInfoCardValue(field, rawValue) { if (IDENTIFIER_FIELD_KEYS.has(field.key)) { value = String(value); } else if (typeof value === 'number') { - value = value.toLocaleString(); + value = value.toLocaleString(getEarthLocale()); + } + if (field.key === 'country') { + value = localizeCountryName(value) || value; } if (field.unit && value !== '-') { - value = value + ' ' + field.unit; + value = value + ' ' + translateText(field.unit); + } else if (typeof value === 'string') { + value = translateText(value); } return value; } +function infoText(value, fallback = '') { + const translated = translateText(value); + if (getEarthLocale() === 'en-US' && hasCjkText(translated) && fallback) { + return fallback; + } + return translated; +} + function escapeInfoCardHtml(value) { return String(value ?? '').replace(/[&<>"']/g, (char) => ({ '&': '&', @@ -201,12 +307,30 @@ function renderNewsCardContent(content, data) { if (!(content instanceof HTMLElement)) return; const summary = getNewsCardSummary(data); const title = getNewsCardTitle(data); + const metaItems = [ + ['媒体来源', data?.source], + ['RSS 来源', data?.feedName], + ['源类型', data?.feedSourceTypeLabel], + ['抓取通道', data?.fetchChannelLabel], + ['新闻类型', data?.categoryLabel], + ['区域', data?.regionLabel], + ['发布时间', data?.publishedAtDisplay], + ].filter(([, value]) => String(value ?? '').trim()); + const metaHtml = metaItems.length + ? `
${metaItems.map(([label, value]) => ` +
+ ${escapeInfoCardHtml(infoText(label))} + ${escapeInfoCardHtml(value)} +
+ `).join('')}
` + : ''; content.innerHTML = `
-
新闻信号
+
${escapeInfoCardHtml(infoText('新闻信号'))}
${escapeInfoCardHtml(title)}
+ ${metaHtml}
-
概要
+
${escapeInfoCardHtml(infoText('概要'))}
@@ -219,12 +343,27 @@ function renderMobileNewsCardContent(content, data) { if (!(content instanceof HTMLElement)) return; const summary = getNewsCardSummary(data); const title = getNewsCardTitle(data); + const metaItems = [ + ['来源', data?.source], + ['RSS', data?.feedName], + ['类型', data?.categoryLabel], + ['区域', data?.regionLabel], + ].filter(([, value]) => String(value ?? '').trim()); + const metaHtml = metaItems.length + ? `
${metaItems.map(([label, value]) => ` +
+ ${escapeInfoCardHtml(infoText(label))} + ${escapeInfoCardHtml(value)} +
+ `).join('')}
` + : ''; content.innerHTML = `
-
新闻信号
+
${escapeInfoCardHtml(infoText('新闻信号'))}
${escapeInfoCardHtml(title)}
+ ${metaHtml}
-
概要
+
${escapeInfoCardHtml(infoText('概要'))}
@@ -252,7 +391,7 @@ function renderMobileDetailContent(type, config, data) { const value = formatInfoCardValue(field, data[field.key]); html += `
- ${field.label} + ${escapeInfoCardHtml(infoText(field.label))} ${value}
`; @@ -309,7 +448,7 @@ function renderDefaultCardContent(content, config, data) { const sourceLabel = getFieldSourceLabel(data, field.key); html += `
- ${field.label} + ${escapeInfoCardHtml(infoText(field.label))} ${value}${sourceLabel}
`; @@ -400,13 +539,21 @@ function buildLocationCollectContext(config, data) { function renderLocationCollectSection(context) { const buttonLabel = context.needsConfirmation - ? '重新自动采集坐标' - : '自动采集坐标候选'; + ? infoText('重新自动采集坐标') + : infoText('自动采集坐标候选'); const cacheKey = getLocationCollectCacheKey(context); const cached = getLocationCollectState(cacheKey); + ensureComputeCenterLocationCapability(); + const blocked = isComputeCenterLocationBlocked(); + const disabledTitle = getWebSearchDisabledTitle(); return `
- @@ -432,9 +579,14 @@ function hydrateLocationCollectRoot(root, state) { const statusEl = root.querySelector('[data-collect-status], [data-unresolved-status]'); const candidatesEl = root.querySelector('[data-collect-candidates], [data-unresolved-candidates]'); const button = root.querySelector('[data-collect-action="run"], [data-unresolved-collect]'); - if (statusEl) statusEl.textContent = state?.statusText || ''; + if (statusEl) statusEl.textContent = infoText(state?.statusText || ''); if (candidatesEl) candidatesEl.innerHTML = renderCachedCollectCandidates(state); - if (button instanceof HTMLButtonElement) button.disabled = state?.loading === true; + if (button instanceof HTMLButtonElement) { + button.classList.toggle('is-loading', state?.loading === true); + button.toggleAttribute('aria-busy', state?.loading === true); + } + root.classList.toggle('is-locating', state?.loading === true); + updateComputeCenterLocationCapabilityDom(); } function rememberLocationCollectContext(context) { @@ -522,15 +674,15 @@ function ensureCandidateActionBindings(rootOrChild, context) { if (typeof actionContext.save !== 'function') return; const statusEl = root.querySelector('[data-collect-status], [data-unresolved-status]'); button.disabled = true; - if (statusEl) statusEl.textContent = '正在保存所选坐标...'; + if (statusEl) statusEl.textContent = infoText('正在保存所选坐标...'); try { const saveResult = await actionContext.save(candidate); setLocationCollectState(actionContext, { loading: false, - statusText: '坐标已保存', + statusText: infoText('坐标已保存'), candidates: [], }); - if (statusEl) statusEl.textContent = '坐标已保存'; + if (statusEl) statusEl.textContent = infoText('坐标已保存'); window.dispatchEvent( new CustomEvent('earth:compute-center-location-saved', { detail: { @@ -553,7 +705,7 @@ function ensureCandidateActionBindings(rootOrChild, context) { } } catch (error) { console.error('save compute-center location failed', error); - if (statusEl) statusEl.textContent = `保存失败：${error?.message || error}`; + if (statusEl) statusEl.textContent = infoText(`保存失败：${error?.message || error}`); button.disabled = false; } }); @@ -563,14 +715,14 @@ function formatLocationCollectFailure(result) { const regularReason = result?.failure_reason || '常规来源没有可用坐标候选'; const llmReason = result?.llm_failure_reason; if (llmReason) { - return `常规来源无结果；LLM 兜底未生成可用候选：${llmReason}`; + return infoText(`常规来源无结果；LLM 兜底未生成可用候选：${llmReason}`); } const attempted = Array.isArray(result?.attempted_queries) ? result.attempted_queries : []; const attemptedLlm = attempted.some((query) => String(query || '').startsWith('llm_factcheck:')); if (attemptedLlm) { - return `常规来源无结果；LLM 兜底已尝试但没有返回可用候选。${regularReason}`; + return infoText(`常规来源无结果；LLM 兜底已尝试但没有返回可用候选。${regularReason}`); } - return regularReason; + return infoText(regularReason); } function bindLocationCollectControls(content, context) { @@ -588,10 +740,15 @@ function bindLocationCollectControls(content, context) { } button.addEventListener('click', async (event) => { event.stopPropagation(); - button.disabled = true; + const capability = await ensureComputeCenterLocationCapability(); + if (!capability?.enabled) { + button.title = getWebSearchDisabledTitle(capability); + return; + } + setLocationButtonLoading(button, true, '正在定位'); setLocationCollectState(context, { loading: true, - statusText: '正在采集坐标候选...', + statusText: infoText('正在采集坐标候选...'), candidates: [], }); try { @@ -599,7 +756,7 @@ function bindLocationCollectControls(content, context) { if (!result?.success) { setLocationCollectState(context, { loading: false, - statusText: `未能采集到坐标：${formatLocationCollectFailure(result)}`, + statusText: infoText(`未能采集到坐标：${formatLocationCollectFailure(result)}`), candidates: [], result, }); @@ -608,7 +765,7 @@ function bindLocationCollectControls(content, context) { const candidates = Array.isArray(result.candidates) ? result.candidates : []; setLocationCollectState(context, { loading: false, - statusText: `共找到 ${candidates.length} 个候选位置`, + statusText: infoText(`共找到 ${candidates.length} 个候选位置`), candidates, result, }); @@ -616,11 +773,11 @@ function bindLocationCollectControls(content, context) { console.error('collect-location failed', error); setLocationCollectState(context, { loading: false, - statusText: `采集失败：${error?.message || error}`, + statusText: infoText(`采集失败：${error?.message || error}`), candidates: [], }); } finally { - button.disabled = false; + setLocationButtonLoading(button, false); updateLocationCollectDomFromState(getLocationCollectCacheKey(context)); } }, { once: false }); @@ -636,22 +793,22 @@ function renderCollectCandidateRow(candidate, isBest, index) { ? `${Math.round(Number(candidate.confidence) * 100)}%` : '-'; const safeIndex = Number.isFinite(Number(index)) ? Number(index) : 0; - const name = escapeInfoCardHtml(candidate.matched_location_name || candidate.display_name || '候选'); + const name = escapeInfoCardHtml(candidate.matched_location_name || candidate.display_name || infoText('候选')); const sourceLabel = escapeInfoCardHtml(candidate.source || ''); return `
${name} - ${escapeInfoCardHtml(precisionLabel)} + ${escapeInfoCardHtml(infoText(precisionLabel))}
${sourceLabel} - 置信 ${escapeInfoCardHtml(confidence)} + ${escapeInfoCardHtml(infoText(`置信 ${confidence}`))}
${Number(candidate.latitude).toFixed(4)}, ${Number(candidate.longitude).toFixed(4)} - - + +
`; @@ -666,7 +823,7 @@ function getUnresolvedComputeCenterContext(item) { entityId: item?.source_id || item?.id || '', sourceId: item?.source_id || item?.id || '', recordId: item?.id || item?.record_id || '', - name: item?.name || item?.title || '未命名算力中心', + name: item?.name || item?.title || infoText('未命名算力中心'), site_type: item?.site_type || metadata.site_type || '', operator: item?.operator || item?.vendor || metadata.operator || '', site: item?.site || metadata.site || metadata.organization || '', @@ -678,10 +835,13 @@ function getUnresolvedComputeCenterContext(item) { function renderComputeCenterUnresolvedContent(content, data) { const items = Array.isArray(data?.items) ? data.items : []; + ensureComputeCenterLocationCapability(); + const blocked = isComputeCenterLocationBlocked(); + const disabledTitle = getWebSearchDisabledTitle(); if (!items.length) { content.innerHTML = `
- 当前没有待定位算力中心 + ${escapeInfoCardHtml(infoText('当前没有待定位算力中心'))}
`; return; @@ -695,7 +855,7 @@ function renderComputeCenterUnresolvedContent(content, data) { const cached = getLocationCollectState(cacheKey); const meta = [context.site || context.operator, context.city, context.country] .filter(Boolean) - .join(' · ') || '缺少可用地址字段'; + .join(' · ') || infoText('缺少可用地址字段'); return `
@@ -704,8 +864,13 @@ function renderComputeCenterUnresolvedContent(content, data) {
${escapeInfoCardHtml(context.name)}
${escapeInfoCardHtml(meta)}
- +
${escapeInfoCardHtml(cached?.statusText || '')}
@@ -718,16 +883,23 @@ function renderComputeCenterUnresolvedContent(content, data) { content.innerHTML = `
- ${items.length} 个算力中心没有可信坐标 -
-
+
${escapeInfoCardHtml(computeCenterUnresolvedBatchState.statusText || '')}
${rows}
`; + updateUnresolvedBatchDom(); + updateComputeCenterLocationCapabilityDom(); bindComputeCenterUnresolvedControls(content); } @@ -736,8 +908,8 @@ function updateUnresolvedSummary(content) { const summaryText = content.querySelector('[data-unresolved-summary-text]'); if (summaryText) { summaryText.textContent = remainingCount > 0 - ? `${remainingCount} 个算力中心没有可信坐标` - : '当前没有待定位算力中心'; + ? infoText(`${remainingCount} 个算力中心没有可信坐标`) + : infoText('当前没有待定位算力中心'); } const adoptAllButton = content.querySelector('[data-unresolved-adopt-all]'); if (adoptAllButton instanceof HTMLButtonElement) { @@ -793,6 +965,58 @@ async function collectUnresolvedComputeCenterCandidates(context, options = {}) { }; } +async function saveBestUnresolvedComputeCenterCandidate(context, progressLabel = '') { + setLocationCollectState(context, { + loading: true, + statusText: infoText(progressLabel || '正在一键定位并采用最高置信候选...'), + candidates: [], + }); + const { mod, result, candidates } = await collectUnresolvedComputeCenterCandidates( + context, + { useCached: true }, + ); + if (!result?.success) { + setLocationCollectState(context, { + loading: false, + statusText: infoText(`未找到可采用候选：${formatLocationCollectFailure(result)}`), + candidates: [], + result, + }); + return { saved: false, result, reason: 'no_result' }; + } + setLocationCollectState(context, { + loading: true, + statusText: infoText(`找到 ${candidates.length} 个候选，正在保存最高置信位置...`), + candidates, + result, + }); + const bestCandidate = getBestLocationCandidate(candidates); + if (!bestCandidate) { + setLocationCollectState(context, { + loading: false, + statusText: infoText('未找到包含有效经纬度的候选'), + candidates, + result, + }); + return { saved: false, result, reason: 'no_valid_candidate' }; + } + const saveResult = await mod.saveComputeCenterLocation(context.sourceId, bestCandidate, context); + setLocationCollectState(context, { + loading: false, + statusText: infoText('坐标已保存，等待图层刷新'), + candidates, + result, + savedCandidate: bestCandidate, + saveResult, + }); + return { + saved: true, + candidate: bestCandidate, + result, + saveResult, + }; +} + function getBestLocationCandidate(candidates) { return candidates .filter((candidate) => ( @@ -833,16 +1057,22 @@ function bindComputeCenterUnresolvedControls(content) { content.querySelectorAll('[data-unresolved-collect]').forEach((button) => { button.addEventListener('click', async (event) => { event.stopPropagation(); + const capability = await ensureComputeCenterLocationCapability(); + if (!capability?.enabled) { + button.title = getWebSearchDisabledTitle(capability); + return; + } const itemRoot = button.closest('[data-unresolved-item]'); const statusEl = itemRoot?.querySelector('[data-unresolved-status]'); const candidatesEl = itemRoot?.querySelector('[data-unresolved-candidates]'); const context = JSON.parse(button.dataset.contextJson || '{}'); if (!context.sourceId || !statusEl || !candidatesEl) return; - button.disabled = true; + itemRoot?.classList.add('is-locating'); + setLocationButtonLoading(button, true, '正在定位'); setLocationCollectState(context, { loading: true, - statusText: '正在采集坐标候选...', + statusText: infoText('正在采集坐标候选...'), candidates: [], }); try { @@ -850,7 +1080,7 @@ function bindComputeCenterUnresolvedControls(content) { if (!result?.success) { setLocationCollectState(context, { loading: false, - statusText: `未能采集到坐标：${formatLocationCollectFailure(result)}`, + statusText: infoText(`未能采集到坐标：${formatLocationCollectFailure(result)}`), candidates: [], result, }); @@ -858,7 +1088,7 @@ function bindComputeCenterUnresolvedControls(content) { } setLocationCollectState(context, { loading: false, - statusText: `共找到 ${candidates.length} 个候选位置`, + statusText: infoText(`共找到 ${candidates.length} 个候选位置`), candidates, result, }); @@ -874,11 +1104,12 @@ function bindComputeCenterUnresolvedControls(content) { console.error('collect unresolved compute-center location failed', error); setLocationCollectState(context, { loading: false, - statusText: `采集失败：${error?.message || error}`, + statusText: infoText(`采集失败：${error?.message || error}`), candidates: [], }); } finally { - button.disabled = false; + itemRoot?.classList.remove('is-locating'); + setLocationButtonLoading(button, false); updateLocationCollectDomFromState(getLocationCollectCacheKey(context)); } }); @@ -888,6 +1119,15 @@ function bindComputeCenterUnresolvedControls(content) { if (adoptAllButton instanceof HTMLButtonElement) { adoptAllButton.addEventListener('click', async (event) => { event.stopPropagation(); + const capability = await ensureComputeCenterLocationCapability(); + if (!capability?.enabled) { + adoptAllButton.title = getWebSearchDisabledTitle(capability); + return; + } + if (computeCenterUnresolvedBatchState.running) { + updateUnresolvedBatchDom(); + return; + } const statusEl = content.querySelector('[data-unresolved-batch-status]'); const buttons = Array.from(content.querySelectorAll('button')); const pendingItems = Array.from(content.querySelectorAll('[data-unresolved-item]')) @@ -900,55 +1140,72 @@ function bindComputeCenterUnresolvedControls(content) { if (!pendingItems.length) return; buttons.forEach((button) => { button.disabled = true; }); + setLocationButtonLoading(adoptAllButton, true, '正在定位'); + setUnresolvedBatchState({ + running: true, + total: pendingItems.length, + processed: 0, + saved: 0, + missed: 0, + statusText: infoText(`一键定位进行中 0/${pendingItems.length}...`), + }); let savedCount = 0; let missedCount = 0; try { for (const [index, { itemRoot, context }] of pendingItems.entries()) { const itemStatusEl = itemRoot.querySelector('[data-unresolved-status]'); - if (statusEl) { - statusEl.textContent = `正在采用最高置信候选 ${index + 1}/${pendingItems.length}...`; - } + itemRoot.classList.add('is-locating'); + const progressText = infoText(`正在定位并采用最高置信候选 ${index + 1}/${pendingItems.length}...`); + setUnresolvedBatchState({ + statusText: progressText, + processed: index, + saved: savedCount, + missed: missedCount, + }); + if (statusEl) statusEl.textContent = progressText; try { - const { mod, result, candidates, fromCache } = await collectUnresolvedComputeCenterCandidates( - context, - { useCached: true }, - ); - if (!result?.success) { - if (itemStatusEl) { - itemStatusEl.textContent = `未找到可采用候选：${formatLocationCollectFailure(result)}`; - } + const saveOutcome = await saveBestUnresolvedComputeCenterCandidate(context, progressText); + if (!saveOutcome.saved) { + if (itemStatusEl) itemStatusEl.textContent = getLocationCollectState(context)?.statusText || infoText('未找到可采用候选'); missedCount += 1; continue; } - const bestCandidate = getBestLocationCandidate(candidates); - if (!bestCandidate) { - if (itemStatusEl) { - itemStatusEl.textContent = '未找到包含有效经纬度的候选'; - } - missedCount += 1; - continue; - } - await mod.saveComputeCenterLocation(context.sourceId, bestCandidate, context); - if (fromCache) { - clearLocationCollectState(context); - } savedCount += 1; removeResolvedUnresolvedItem(content, itemRoot); } catch (error) { console.error('adopt unresolved compute-center location failed', error); + setLocationCollectState(context, { + loading: false, + statusText: infoText(`一键定位失败：${error?.message || error}`), + candidates: [], + }); if (itemStatusEl) { - itemStatusEl.textContent = `一键采用失败：${error?.message || error}`; + itemStatusEl.textContent = infoText(`一键采用失败：${error?.message || error}`); } missedCount += 1; + } finally { + itemRoot.classList.remove('is-locating'); + setUnresolvedBatchState({ + processed: index + 1, + saved: savedCount, + missed: missedCount, + statusText: infoText(`一键定位进行中 ${index + 1}/${pendingItems.length}，已保存 ${savedCount} 个${missedCount ? `，失败 ${missedCount} 个` : ''}`), + }); } } - if (statusEl) { - statusEl.textContent = savedCount > 0 - ? `已采用 ${savedCount} 个最高置信候选${missedCount ? `，${missedCount} 个仍需手动处理` : ''}` - : `${missedCount} 个都没有可自动采用的候选，需要手动处理`; - } + const finalStatus = savedCount > 0 + ? infoText(`已定位并采用 ${savedCount} 个最高置信候选${missedCount ? `，${missedCount} 个仍需手动处理` : ''}`) + : infoText(`${missedCount} 个都没有可自动采用的候选，需要手动处理`); + if (statusEl) statusEl.textContent = finalStatus; + setUnresolvedBatchState({ + running: false, + processed: pendingItems.length, + saved: savedCount, + missed: missedCount, + statusText: finalStatus, + }); if (savedCount > 0) { window.dispatchEvent( new CustomEvent('earth:compute-center-location-saved', { @@ -962,7 +1219,10 @@ function bindComputeCenterUnresolvedControls(content) { ); } } finally { + setUnresolvedBatchState({ running: false }); + setLocationButtonLoading(adoptAllButton, false); buttons.forEach((button) => { button.disabled = false; }); + updateComputeCenterLocationCapabilityDom(); } }); } @@ -973,7 +1233,7 @@ function getFieldSourceLabel(data, fieldKey) { if (!sources || typeof sources !== 'object') return ''; const source = sources[fieldKey]; if (!source) return ''; - return ` ${source}`; + return ` ${escapeInfoCardHtml(infoText(source))}`; } function renderVesselEnrichmentSection(enrichment) { @@ -983,8 +1243,8 @@ function renderVesselEnrichmentSection(enrichment) { if (!profile && !media) { return `
-
船舶资料
-
资料缓存中
+
${escapeInfoCardHtml(infoText('船舶资料'))}
+
${escapeInfoCardHtml(infoText('资料缓存中'))}
`; } @@ -1004,11 +1264,11 @@ function renderVesselEnrichmentSection(enrichment) { inner += renderEnrichmentMeta('媒体', media); } if (!inner) { - inner = '
资料缓存中
'; + inner = `
${escapeInfoCardHtml(infoText('资料缓存中'))}
`; } return `
-
船舶资料
+
${escapeInfoCardHtml(infoText('船舶资料'))}
${inner}
`; @@ -1021,8 +1281,8 @@ function renderEnrichmentPayloadRows(payload) { if (typeof value === 'object') continue; rows += `
- ${key} - ${String(value)} + ${escapeInfoCardHtml(infoText(key))} + ${escapeInfoCardHtml(infoText(String(value)))}
`; } @@ -1031,45 +1291,45 @@ function renderEnrichmentPayloadRows(payload) { function renderEnrichmentMeta(label, record) { const parts = []; - if (record.source) parts.push(`来源 ${record.source}`); - if (record.fetched_at) parts.push(`更新 ${record.fetched_at}`); + if (record.source) parts.push(infoText(`来源 ${record.source}`)); + if (record.fetched_at) parts.push(infoText(`更新 ${record.fetched_at}`)); if (record.confidence !== null && record.confidence !== undefined) { - parts.push(`置信 ${Number(record.confidence).toFixed(2)}`); + parts.push(infoText(`置信 ${Number(record.confidence).toFixed(2)}`)); } if (!parts.length) return ''; - return `
${label}：${parts.join(' · ')}
`; + return `
${escapeInfoCardHtml(infoText(label))}: ${escapeInfoCardHtml(parts.join(' · '))}
`; } // ── Mobile popup ───────────────────────────────────────────── function getMobilePopupTitle(type, data) { switch (type) { - case 'cable': return data.name || '海缆'; - case 'landing_point': return data.name || '登陆点'; - case 'satellite': return data.name || '卫星'; - case 'bgp': return data.anomaly_type || 'BGP事件'; + case 'cable': return data.name || infoText('海缆'); + case 'landing_point': return data.name || infoText('登陆点'); + case 'satellite': return data.name || infoText('卫星'); + case 'bgp': return data.anomaly_type || infoText('BGP事件'); case 'news': return getNewsCardTitle(data); - case 'bgp_collector': return data.collector || 'BGP观测站'; - case 'compute_center_unresolved': return '待定位算力中心'; - case 'supercomputer': return data.name || '超算'; - case 'gpu_cluster': return data.name || 'GPU集群'; - case 'vessel': return data.name || '船只'; - default: return '详情'; + case 'bgp_collector': return data.collector || infoText('BGP观测站'); + case 'compute_center_unresolved': return infoText('待定位算力中心'); + case 'supercomputer': return data.name || infoText('超算'); + case 'gpu_cluster': return data.name || infoText('GPU集群'); + case 'vessel': return data.name || infoText('船只'); + default: return infoText('详情'); } } function getMobilePopupSubtitle(type, data) { switch (type) { - case 'cable': return data.owner || data.status || '海缆'; - case 'landing_point': return data.country || '登陆点'; - case 'satellite': return data.norad_id ? `NORAD ${data.norad_id}` : '卫星'; - case 'bgp': return data.severity || 'BGP路由异常'; - case 'news': return getNewsCardSummaryPreview(data, 30) || '态势新闻'; - case 'bgp_collector': return data.location || 'BGP观测站'; - case 'compute_center_unresolved': return `${data?.totalCount || 0} 个待定位`; - case 'supercomputer': return data.country || '超级计算机'; - case 'gpu_cluster': return data.country || 'GPU集群'; - case 'vessel': return data.vessel_type || 'AIS 船只'; + case 'cable': return infoText(data.owner || data.status || '海缆'); + case 'landing_point': return localizeCountryName(data.country) || infoText('登陆点'); + case 'satellite': return data.norad_id ? `NORAD ${data.norad_id}` : infoText('卫星'); + case 'bgp': return infoText(data.severity || 'BGP路由异常'); + case 'news': return getNewsCardSummaryPreview(data, 30) || infoText('态势新闻'); + case 'bgp_collector': return data.location || infoText('BGP观测站'); + case 'compute_center_unresolved': return infoText(`${data?.totalCount || 0} 个待定位`); + case 'supercomputer': return localizeCountryName(data.country) || infoText('超级计算机'); + case 'gpu_cluster': return localizeCountryName(data.country) || infoText('GPU集群'); + case 'vessel': return data.vessel_type || infoText('AIS 船只'); default: return ''; } } @@ -1437,6 +1697,22 @@ const CARD_CONFIG = { { key: 'length', label: '船长', unit: 'm' }, { key: 'received_at', label: '更新时间' } ] + }, + earth_interactable: { + icon: '📍', + title: '交互点详情', + className: 'earth_interactable', + fields: [ + { key: 'label', label: '名称' }, + { key: 'kind', label: '类型' }, + { key: 'id', label: '标识' }, + { key: 'latitude', label: '纬度' }, + { key: 'longitude', label: '经度' }, + { key: 'description', label: '说明' }, + { key: 'source', label: '来源' }, + { key: 'status', label: '状态' }, + { key: 'updated_at', label: '更新时间' } + ] } }; @@ -1553,8 +1829,8 @@ function mountCard() {
🛰️ -

详情

-
@@ -1594,16 +1870,19 @@ function mountCard() { const value = valueEl?.textContent?.trim(); if (!value || value === '-') { - showStatusMessage('无可复制内容', 'warning'); + showStatusMessage(earthMessage("status.copyEmpty"), 'warning'); return; } try { await navigator.clipboard.writeText(value); - showStatusMessage(`已复制${label.textContent}：${value}`, 'success'); + showStatusMessage( + earthMessage("status.copyValue", { label: label.textContent, value }), + 'success', + ); } catch (error) { console.error('Copy failed:', error); - showStatusMessage('复制失败', 'error'); + showStatusMessage(earthMessage("status.copyFailed"), 'error'); } }); @@ -1629,7 +1908,7 @@ function positionPanel(panel, x, y, options = {}) { const scale = parseFloat( getComputedStyle(document.documentElement).getPropertyValue('--hud-scale') ) || 1; - const estW = Math.min(300 * scale, vpW - 32); + const estW = Math.min(340 * scale, vpW - 32); const estH = Math.min(420 * scale, vpH * 0.7); if (options.absolute === true) { @@ -1744,11 +2023,11 @@ export function showInfoCard(type, data, options = {}) { if (title) { title.textContent = type === 'news' ? getNewsCardTitle(data) - : config.title; + : infoText(config.title); } if (typeLabel) { typeLabel.textContent = type === 'news' - ? '新闻信号' + ? infoText('新闻信号') : type.replaceAll('_', ' '); } @@ -1785,7 +2064,7 @@ export function showInfoCard(type, data, options = {}) { icon.textContent = config.icon; title.textContent = type === 'news' ? getNewsCardTitle(data) - : config.title; + : infoText(config.title); if (type === 'news') { renderNewsCardContent(content, data); diff --git a/frontend/public/earth/js/interactable.js b/frontend/public/earth/js/interactable.js index 82f8af86..c75833a2 100644 --- a/frontend/public/earth/js/interactable.js +++ b/frontend/public/earth/js/interactable.js @@ -8,22 +8,53 @@ const assetImageLoadPromises = new Map(); const surfaceAvoidanceBuckets = new Map(); const interactableLayerControllers = new Map(); const DEFAULT_AVOIDANCE_PRECISION = 4; -const DEFAULT_AVOIDANCE_RADIUS = 1.1; -const DEFAULT_AVOIDANCE_STEP = 0.35; -const AVOIDANCE_RING_SLOT_COUNT = 8; -const SCREEN_AVOIDANCE_ZOOM_BUCKET_SIZE = 0.03; -const SCREEN_AVOIDANCE_OVERLAP_FACTOR = 0.82; -const SCREEN_AVOIDANCE_COLLAPSE_FACTOR = 0.32; -const COMPACT_DOT_ZOOM_THRESHOLD = 1.5; +const COMPACT_DOT_ZOOM_THRESHOLD = 1.7; const COMPACT_DOT_POINT_SIZE = 12; const COMPACT_DOT_RADIUS_RATIO = 0.26; +const CLUSTER_OVERLAP_FACTOR = 0.9; +const CLUSTER_MIN_COUNT = 2; +const CLUSTER_MAX_MARKERS_PER_DOT = 14; +const CLUSTER_MAX_POINT_SIZE = 32; +const CLUSTER_MAX_SCREEN_DIAMETER_PX = 96; +const CLUSTER_SEED_DISTANCE_FACTOR = 1.8; +const CLUSTER_TRANSITION_MS = 220; +const CLUSTER_BAND_HYSTERESIS = 0.08; +const CLUSTER_DISABLE_ABOVE_ZOOM = 2.5; +const CLUSTER_STRATEGY_DYNAMIC_SCREEN = "dynamic-screen"; +const CLUSTER_STRATEGY_STABLE_SPHERICAL = "stable-spherical"; +const CLUSTER_STRATEGY_NONE = "none"; +const CLUSTER_ZOOM_BANDS = Object.freeze([ + Object.freeze({ + maxZoom: 2.0, + referenceZoom: 1.0, + overlapFactor: 0.9, + maxDiameterPx: 96, + }), + Object.freeze({ + maxZoom: 3.0, + referenceZoom: 2.0, + overlapFactor: 0.45, + maxDiameterPx: 72, + }), + Object.freeze({ + maxZoom: Number.POSITIVE_INFINITY, + referenceZoom: 3.0, + overlapFactor: 0.2, + maxDiameterPx: 48, + }), +]); +const SPHERICAL_CLUSTER_BANDS = Object.freeze([ + Object.freeze({ key: "far", maxZoom: 1.7, distance: 15 }), + Object.freeze({ key: "mid", maxZoom: 2.6, distance: 8 }), + Object.freeze({ key: "near", maxZoom: 3.5, distance: 4 }), + Object.freeze({ key: "detail", maxZoom: Number.POSITIVE_INFINITY, distance: 0 }), +]); let compactDotsEnabled = true; let screenAvoidanceRevision = 0; -let screenAvoidanceSignature = ""; -// Named avoidance profiles. Layers that should mutex with each other (e.g. fan -// out when sharing the same city center) must reference the SAME profile — -// markers are bucketed by the resulting key, and only equal keys collide. +// Named overlap profiles. Layers that should be recognized as sharing a place +// must reference the SAME profile. The profile records overlap metadata only; +// it never moves the marker away from its real geographic anchor. // // city — ~1.1km grid (precision 2). Use for site/observatory/POI markers // that often share a city-center coordinate from geocoding. @@ -32,46 +63,25 @@ let screenAvoidanceSignature = ""; // // Layers that need a fully custom cluster identity (e.g. a city ID string) // can pass `getKey: (item, position) => "..."` instead of using a profile. -// Avoidance is opt-in: dynamic high-density layers such as vessels should leave -// it disabled so their markers stay at their real-time coordinates. +// Overlap tracking is opt-in: dynamic high-density layers such as vessels should +// leave it disabled so their per-frame metadata work stays minimal. export const SURFACE_AVOIDANCE_PROFILES = Object.freeze({ - city: Object.freeze({ precision: 2, radius: 1.4, step: 0.5 }), + city: Object.freeze({ precision: 2 }), precise: Object.freeze({ precision: DEFAULT_AVOIDANCE_PRECISION, - radius: DEFAULT_AVOIDANCE_RADIUS, - step: DEFAULT_AVOIDANCE_STEP, }), }); -const TANGENT_EPSILON_SQ = 1e-6; -const avoidanceNorthPole = new THREE.Vector3(0, 1, 0); -const avoidanceFallbackEast = new THREE.Vector3(1, 0, 0); -const avoidanceCenterScratch = new THREE.Vector3(); -const avoidanceEastScratch = new THREE.Vector3(); -const avoidanceNorthScratch = new THREE.Vector3(); -const avoidancePositionScratch = new THREE.Vector3(); -const screenAvoidanceStaticPositionScratch = new THREE.Vector3(); -const screenAvoidanceWorldPositionScratch = new THREE.Vector3(); -const screenAvoidanceProjectedScratch = new THREE.Vector3(); -const screenAvoidanceClusterCenterScratch = new THREE.Vector3(); - function colorToRgbArray(colorValue, fallback = "#ffffff") { const color = new THREE.Color(colorValue || fallback); return [color.r, color.g, color.b]; } -function toFiniteNumber(value, fallback) { - const numericValue = Number(value); - return Number.isFinite(numericValue) ? numericValue : fallback; -} - function normalizeAvoidanceConfig(avoidance) { if (avoidance === false || avoidance === null || avoidance === undefined) { return { enabled: false, screen: false, precision: DEFAULT_AVOIDANCE_PRECISION, - radius: DEFAULT_AVOIDANCE_RADIUS, - step: DEFAULT_AVOIDANCE_STEP, }; } @@ -79,17 +89,211 @@ function normalizeAvoidanceConfig(avoidance) { avoidance === true || typeof avoidance !== "object" ? {} : avoidance; const config = { enabled: true, - screen: true, precision: DEFAULT_AVOIDANCE_PRECISION, - radius: DEFAULT_AVOIDANCE_RADIUS, - step: DEFAULT_AVOIDANCE_STEP, + screen: false, ...customConfig, }; config.enabled = config.enabled !== false; - config.screen = config.enabled && config.screen !== false; + config.screen = false; return config; } +function normalizeClusterConfig(cluster, avoidanceConfig) { + if (cluster === false || cluster === null) { + return { + enabled: false, + strategy: CLUSTER_STRATEGY_NONE, + overlapFactor: CLUSTER_OVERLAP_FACTOR, + minCount: CLUSTER_MIN_COUNT, + maxMarkersPerDot: CLUSTER_MAX_MARKERS_PER_DOT, + bands: SPHERICAL_CLUSTER_BANDS, + transitionMs: CLUSTER_TRANSITION_MS, + bandHysteresis: CLUSTER_BAND_HYSTERESIS, + disableAboveZoom: CLUSTER_DISABLE_ABOVE_ZOOM, + }; + } + + const customConfig = typeof cluster === "object" ? cluster : {}; + const requestedStrategy = String(customConfig.strategy || "").trim(); + const enabled = customConfig.enabled ?? avoidanceConfig.enabled; + const strategy = + enabled === false + ? CLUSTER_STRATEGY_NONE + : [ + CLUSTER_STRATEGY_DYNAMIC_SCREEN, + CLUSTER_STRATEGY_STABLE_SPHERICAL, + CLUSTER_STRATEGY_NONE, + ].includes(requestedStrategy) + ? requestedStrategy + : CLUSTER_STRATEGY_DYNAMIC_SCREEN; + return { + enabled: enabled !== false && strategy !== CLUSTER_STRATEGY_NONE, + strategy, + overlapFactor: Number.isFinite(Number(customConfig.overlapFactor)) + ? Number(customConfig.overlapFactor) + : CLUSTER_OVERLAP_FACTOR, + minCount: Math.max(2, Math.round(Number(customConfig.minCount ?? CLUSTER_MIN_COUNT))), + maxMarkersPerDot: Math.max( + 2, + Math.round(Number(customConfig.maxMarkersPerDot ?? CLUSTER_MAX_MARKERS_PER_DOT)), + ), + bands: normalizeSphericalClusterBands(customConfig.bands), + transitionMs: Math.max( + 0, + Math.round(Number(customConfig.transitionMs ?? CLUSTER_TRANSITION_MS)), + ), + bandHysteresis: Math.max( + 0, + Number(customConfig.bandHysteresis ?? CLUSTER_BAND_HYSTERESIS), + ), + disableAboveZoom: + customConfig.disableAboveZoom === false || customConfig.disableAboveZoom === null + ? Number.POSITIVE_INFINITY + : Number.isFinite(Number(customConfig.disableAboveZoom)) + ? Number(customConfig.disableAboveZoom) + : CLUSTER_DISABLE_ABOVE_ZOOM, + }; +} + +function normalizeSphericalClusterBands(bands) { + if (!Array.isArray(bands) || bands.length === 0) return SPHERICAL_CLUSTER_BANDS; + const normalizedBands = bands + .map((band, index) => { + const maxZoom = Number(band?.maxZoom); + const distance = Number(band?.distance); + return { + key: String(band?.key || `band-${index}`), + maxZoom: Number.isFinite(maxZoom) ? maxZoom : Number.POSITIVE_INFINITY, + distance: Number.isFinite(distance) ? Math.max(0, distance) : 0, + }; + }) + .sort((a, b) => a.maxZoom - b.maxZoom); + return normalizedBands.length > 0 ? normalizedBands : SPHERICAL_CLUSTER_BANDS; +} + +function getClusterPointSize(count) { + const safeCount = Math.max(2, Number(count) || 2); + return Math.min( + CLUSTER_MAX_POINT_SIZE, + COMPACT_DOT_POINT_SIZE + Math.log2(safeCount) * 5.5, + ); +} + +function getClusterZoomBand(zoom) { + return ( + CLUSTER_ZOOM_BANDS.find((band) => zoom <= band.maxZoom) || + CLUSTER_ZOOM_BANDS[CLUSTER_ZOOM_BANDS.length - 1] + ); +} + +function getSphericalClusterBand(config, zoom, previousBandKey = null) { + const bands = config?.bands || SPHERICAL_CLUSTER_BANDS; + const previousIndex = previousBandKey + ? bands.findIndex((band) => band.key === previousBandKey) + : -1; + const hysteresis = Number(config?.bandHysteresis) || 0; + if (previousIndex >= 0 && hysteresis > 0) { + const previousBand = bands[previousIndex]; + const lowerBoundary = + previousIndex > 0 ? bands[previousIndex - 1].maxZoom : Number.NEGATIVE_INFINITY; + const upperBoundary = previousBand.maxZoom; + if (zoom > lowerBoundary - hysteresis && zoom <= upperBoundary + hysteresis) { + return previousBand; + } + } + return bands.find((band) => zoom <= band.maxZoom) || bands[bands.length - 1]; +} + +function getNowMs() { + return typeof performance !== "undefined" && typeof performance.now === "function" + ? performance.now() + : Date.now(); +} + +function easeOutCubic(value) { + const t = Math.max(0, Math.min(1, value)); + return 1 - Math.pow(1 - t, 3); +} + +function getTransitionProgress(startedAt, durationMs) { + if (!startedAt || !durationMs || durationMs <= 0) return 1; + return easeOutCubic((getNowMs() - startedAt) / durationMs); +} + +function getNominalGlobePixelsPerWorldUnit(camera, referenceZoom) { + if (!camera) return 1; + const viewportHeight = window.innerHeight || 1; + const fovRad = ((camera.fov || 75) * Math.PI) / 180; + const distance = Math.max( + 1, + Number.isFinite(referenceZoom) && referenceZoom > 0 + ? CONFIG.defaultCameraZ / referenceZoom + : camera.position?.length?.() || CONFIG.defaultCameraZ, + ); + return viewportHeight / (2 * Math.tan(fovRad / 2) * distance); +} + +function getAngularRadiusFromPixels(pixelRadius, worldRadius, camera, referenceZoom) { + const globeRadiusPx = Math.max( + 1, + worldRadius * getNominalGlobePixelsPerWorldUnit(camera, referenceZoom), + ); + return Math.max(0, pixelRadius) / globeRadiusPx; +} + +function getClusterOverlapFactorForBand(band, baseFactor = CLUSTER_OVERLAP_FACTOR) { + return Math.min(baseFactor, band.overlapFactor); +} + +function getSphericalBucketKey(latIndex, lonIndex) { + return `${latIndex}:${lonIndex}`; +} + +function normalizeSphericalLongitudeIndex(lonIndex, lonBucketCount) { + if (!Number.isFinite(lonBucketCount) || lonBucketCount <= 0) return lonIndex; + return ((lonIndex % lonBucketCount) + lonBucketCount) % lonBucketCount; +} + +function getNeighboringSphericalBucketKeys(latIndex, lonIndex, lonBucketCount) { + const keys = []; + for (let latOffset = -1; latOffset <= 1; latOffset += 1) { + for (let lonOffset = -1; lonOffset <= 1; lonOffset += 1) { + keys.push( + getSphericalBucketKey( + latIndex + latOffset, + normalizeSphericalLongitudeIndex(lonIndex + lonOffset, lonBucketCount), + ), + ); + } + } + return keys; +} + +function getEntrySphericalCoordinates(entry) { + const direction = entry.direction; + const lat = Math.asin(Math.max(-1, Math.min(1, direction.y))); + const lon = Math.atan2(direction.z, direction.x); + return { lat, lon }; +} + +function hashStableIds(entries) { + let hash = 2166136261; + entries.forEach((entry) => { + const value = entry.stableId; + for (let index = 0; index < value.length; index += 1) { + hash ^= value.charCodeAt(index); + hash = Math.imul(hash, 16777619); + } + }); + return (hash >>> 0).toString(36); +} + +function getClusterRecordId(entries) { + const sortedEntries = [...entries].sort((a, b) => a.stableId.localeCompare(b.stableId)); + const anchorId = sortedEntries[0]?.stableId || "unknown"; + return `cluster:${anchorId}:${sortedEntries.length}:${hashStableIds(sortedEntries)}`; +} + function disposeGroupChildren(group) { for (let index = group.children.length - 1; index >= 0; index -= 1) { const child = group.children[index]; @@ -134,7 +338,6 @@ export function setInteractableCompactDotsEnabled(enabled) { function invalidateScreenAvoidance() { screenAvoidanceRevision += 1; - screenAvoidanceSignature = ""; } function getAvoidanceKey(item, position, basePosition, config) { @@ -178,8 +381,7 @@ function recomputeAvoidanceBucket(key) { if (entries.length === 1) { const entry = entries[0]; entry.marker.position.copy(entry.marker.userData.icon_base_position); - entry.marker.userData.icon_static_avoidance_position = - entry.marker.userData.icon_base_position.clone(); + entry.marker.userData.icon_static_avoidance_position = null; entry.marker.userData.icon_avoidance_index = 0; entry.marker.userData.icon_avoidance_count = 1; entry.marker.userData.icon_avoidance_layer_count = 1; @@ -190,37 +392,8 @@ function recomputeAvoidanceBucket(key) { const count = entries.length; entries.forEach((entry, index) => { - const basePosition = - entry.marker.userData.icon_base_position || entry.marker.position; - const altitudeRadius = basePosition.length(); - const ringIndex = Math.floor(index / AVOIDANCE_RING_SLOT_COUNT); - const radius = - Math.max(0, entry.radius) + ringIndex * Math.max(0, entry.step); - const angle = -Math.PI / 2 + (Math.PI * 2 * index) / count; - - avoidanceCenterScratch.copy(basePosition).normalize(); - avoidanceEastScratch - .copy(avoidanceNorthPole) - .cross(avoidanceCenterScratch); - if (avoidanceEastScratch.lengthSq() < TANGENT_EPSILON_SQ) { - avoidanceEastScratch.copy(avoidanceFallbackEast); - } - avoidanceEastScratch.normalize(); - avoidanceNorthScratch - .copy(avoidanceCenterScratch) - .cross(avoidanceEastScratch) - .normalize(); - - avoidancePositionScratch - .copy(basePosition) - .addScaledVector(avoidanceEastScratch, Math.cos(angle) * radius) - .addScaledVector(avoidanceNorthScratch, Math.sin(angle) * radius) - .normalize() - .multiplyScalar(altitudeRadius); - - entry.marker.position.copy(avoidancePositionScratch); - entry.marker.userData.icon_static_avoidance_position = - avoidancePositionScratch.clone(); + entry.marker.position.copy(entry.marker.userData.icon_base_position); + entry.marker.userData.icon_static_avoidance_position = null; entry.marker.userData.icon_avoidance_index = index; entry.marker.userData.icon_avoidance_count = count; entry.marker.userData.icon_avoidance_layer_count = affectedLayerIds.size; @@ -259,11 +432,6 @@ function registerLayerAvoidance(layerId, markers, avoidanceConfig) { surfaceAvoidanceBuckets.get(key).push({ layerId, marker, - radius: toFiniteNumber( - avoidanceConfig.radius, - DEFAULT_AVOIDANCE_RADIUS, - ), - step: toFiniteNumber(avoidanceConfig.step, DEFAULT_AVOIDANCE_STEP), }); affectedKeys.add(key); }); @@ -273,175 +441,275 @@ function registerLayerAvoidance(layerId, markers, avoidanceConfig) { function getMarkerStaticAvoidancePosition(marker) { return ( - marker?.userData?.icon_static_avoidance_position || marker?.userData?.icon_base_position || marker?.position ); } -function collectScreenAvoidanceEntries(camera) { +function collectClusterUpdates(camera) { if (!camera) return []; - const entries = []; + const updates = []; interactableLayerControllers.forEach((controller) => { - entries.push(...(controller.collectScreenAvoidanceEntries?.(camera) || [])); + const update = controller.collectClusterEntries?.(camera); + if (!update) return; + if (Array.isArray(update)) { + updates.push({ + strategy: CLUSTER_STRATEGY_DYNAMIC_SCREEN, + entries: update, + records: [], + }); + return; + } + updates.push(update); }); - return entries; + return updates; } -function findScreenAvoidanceGroups(entries) { - const visited = new Set(); +function findScreenClusterGroups(entries) { + const unvisited = new Set(entries); const groups = []; - entries.forEach((entry) => { - if (visited.has(entry)) return; - const group = [entry]; - const queue = [entry]; - visited.add(entry); + entries.forEach((seed) => { + if (!unvisited.has(seed)) return; - while (queue.length > 0) { - const current = queue.shift(); - entries.forEach((candidate) => { - if (visited.has(candidate)) return; - const dx = current.x - candidate.x; - const dy = current.y - candidate.y; - const minDistance = - (current.radiusPx + candidate.radiusPx) * - SCREEN_AVOIDANCE_OVERLAP_FACTOR; - if (dx * dx + dy * dy > minDistance * minDistance) return; - visited.add(candidate); - queue.push(candidate); - group.push(candidate); + const groupEntries = [seed]; + unvisited.delete(seed); + + let changed = true; + while (changed) { + changed = false; + const candidates = [...unvisited].sort((a, b) => { + const distanceA = getAngularDistance(seed, a); + const distanceB = getAngularDistance(seed, b); + return distanceA - distanceB || a.stableId.localeCompare(b.stableId); }); + + for (const candidate of candidates) { + const overlapFactor = Math.min( + seed.overlapFactor || CLUSTER_OVERLAP_FACTOR, + candidate.overlapFactor || CLUSTER_OVERLAP_FACTOR, + ); + const seedDistance = getAngularDistance(seed, candidate); + const seedLimit = + (seed.angularRadius + candidate.angularRadius) * + CLUSTER_SEED_DISTANCE_FACTOR; + if (seedDistance > seedLimit) continue; + + const overlapsGroup = groupEntries.some((entry) => { + const distance = getAngularDistance(candidate, entry); + return distance <= (candidate.angularRadius + entry.angularRadius) * overlapFactor; + }); + if (!overlapsGroup) continue; + + const maxAngularDiameter = Math.max( + seed.maxAngularDiameter, + candidate.maxAngularDiameter, + ); + const staysLocal = groupEntries.every( + (entry) => getAngularDistance(candidate, entry) <= maxAngularDiameter, + ); + if (!staysLocal) { + continue; + } + + groupEntries.push(candidate); + unvisited.delete(candidate); + changed = true; + } } - groups.push(group); + if (groupEntries.length >= CLUSTER_MIN_COUNT) { + groups.push(groupEntries.sort((a, b) => a.stableId.localeCompare(b.stableId))); + return; + } + + groupEntries.forEach((entry) => unvisited.add(entry)); }); return groups; } -function shouldCollapseScreenAvoidanceGroup(group) { - if (group.length <= 1) return false; +function getScreenDistance(entryA, entryB) { + return Math.hypot(entryA.x - entryB.x, entryA.y - entryB.y); +} - for (let index = 0; index < group.length; index += 1) { - for (let nextIndex = index + 1; nextIndex < group.length; nextIndex += 1) { - const current = group[index]; - const candidate = group[nextIndex]; - const dx = current.x - candidate.x; - const dy = current.y - candidate.y; - const collapseDistance = - (current.radiusPx + candidate.radiusPx) * - SCREEN_AVOIDANCE_COLLAPSE_FACTOR; - if (dx * dx + dy * dy > collapseDistance * collapseDistance) { - return false; +function getAngularDistance(entryA, entryB) { + const dot = entryA.direction.dot(entryB.direction); + return Math.acos(Math.min(1, Math.max(-1, dot))); +} + +function getScreenBounds(entries) { + return entries.reduce( + (bounds, entry) => ({ + left: Math.min(bounds.left, entry.x - entry.radiusPx), + right: Math.max(bounds.right, entry.x + entry.radiusPx), + top: Math.min(bounds.top, entry.y - entry.radiusPx), + bottom: Math.max(bounds.bottom, entry.y + entry.radiusPx), + }), + { + left: Number.POSITIVE_INFINITY, + right: Number.NEGATIVE_INFINITY, + top: Number.POSITIVE_INFINITY, + bottom: Number.NEGATIVE_INFINITY, + }, + ); +} + +function splitLargeScreenClusterGroup(groupEntries) { + const maxMarkersPerDot = groupEntries.reduce( + (maxCount, entry) => Math.min(maxCount, entry.maxMarkersPerDot || CLUSTER_MAX_MARKERS_PER_DOT), + Number.POSITIVE_INFINITY, + ); + const safeMaxMarkersPerDot = Number.isFinite(maxMarkersPerDot) + ? maxMarkersPerDot + : CLUSTER_MAX_MARKERS_PER_DOT; + if (groupEntries.length <= safeMaxMarkersPerDot) return [groupEntries]; + + const sortedEntries = [...groupEntries].sort(compareEntriesByStableGeography); + const chunks = []; + for (let index = 0; index < sortedEntries.length; index += safeMaxMarkersPerDot) { + chunks.push(sortedEntries.slice(index, index + safeMaxMarkersPerDot)); + } + return chunks; +} + +function computeSphericalClusterRecords(entries, config, band) { + if (!entries.length || !band || band.distance <= 0) return []; + + const radius = + entries.find((entry) => Number.isFinite(entry.radiusWorld))?.radiusWorld || + CONFIG.earthRadius; + const angularThreshold = Math.max(0.00001, band.distance / Math.max(1, radius)); + const lonBucketCount = Math.max(1, Math.ceil((Math.PI * 2) / angularThreshold)); + const buckets = new Map(); + const entryState = entries.map((entry) => { + const { lat, lon } = getEntrySphericalCoordinates(entry); + const latIndex = Math.floor(lat / angularThreshold); + const lonIndex = normalizeSphericalLongitudeIndex( + Math.floor(lon / angularThreshold), + lonBucketCount, + ); + const state = { + ...entry, + latIndex, + lonIndex, + }; + const key = getSphericalBucketKey(latIndex, lonIndex); + if (!buckets.has(key)) buckets.set(key, []); + buckets.get(key).push(state); + return state; + }); + + const unvisited = new Set(entryState); + const groups = []; + entryState + .slice() + .sort((a, b) => a.stableId.localeCompare(b.stableId)) + .forEach((seed) => { + if (!unvisited.has(seed)) return; + const groupEntries = [seed]; + unvisited.delete(seed); + const neighborKeys = getNeighboringSphericalBucketKeys( + seed.latIndex, + seed.lonIndex, + lonBucketCount, + ); + const candidates = neighborKeys + .flatMap((key) => buckets.get(key) || []) + .filter((candidate) => unvisited.has(candidate)) + .sort((a, b) => getAngularDistance(seed, a) - getAngularDistance(seed, b) || a.stableId.localeCompare(b.stableId)); + + candidates.forEach((candidate) => { + if (!unvisited.has(candidate)) return; + if (getAngularDistance(seed, candidate) > angularThreshold) return; + const overlapsGroup = groupEntries.some( + (entry) => getAngularDistance(candidate, entry) <= angularThreshold, + ); + if (!overlapsGroup) return; + groupEntries.push(candidate); + unvisited.delete(candidate); + }); + + if (groupEntries.length >= config.minCount) { + groups.push(groupEntries.sort((a, b) => a.stableId.localeCompare(b.stableId))); + return; } - } - } - return true; + groupEntries.forEach((entry) => unvisited.add(entry)); + }); + + return groups + .flatMap(splitLargeScreenClusterGroup) + .map(createScreenClusterRecord) + .filter(Boolean); } -function applyCollapsedScreenAvoidanceGroup(group) { - screenAvoidanceClusterCenterScratch.set(0, 0, 0); - let count = 0; - - group.forEach((entry) => { - const basePosition = getMarkerStaticAvoidancePosition(entry.marker); - if (!(basePosition instanceof THREE.Vector3)) return; - screenAvoidanceClusterCenterScratch.add(basePosition); - count += 1; - }); - - if (count === 0) return; - screenAvoidanceClusterCenterScratch.divideScalar(count); - if (screenAvoidanceClusterCenterScratch.lengthSq() < TANGENT_EPSILON_SQ) { - const fallbackPosition = getMarkerStaticAvoidancePosition(group[0]?.marker); - if (!(fallbackPosition instanceof THREE.Vector3)) return; - screenAvoidanceClusterCenterScratch.copy(fallbackPosition); - } - screenAvoidanceClusterCenterScratch.normalize(); - - group.forEach((entry, index) => { - const basePosition = getMarkerStaticAvoidancePosition(entry.marker); - if (!(basePosition instanceof THREE.Vector3)) return; - entry.marker.position - .copy(screenAvoidanceClusterCenterScratch) - .multiplyScalar(basePosition.length()); - entry.marker.userData.icon_screen_avoidance_count = group.length; - entry.marker.userData.icon_screen_avoidance_index = index; - entry.marker.userData.icon_screen_clustered = true; - }); +function compareEntriesByStableGeography(a, b) { + const lonA = Math.atan2(a.direction.z, a.direction.x); + const lonB = Math.atan2(b.direction.z, b.direction.x); + if (Math.abs(lonA - lonB) > 1e-6) return lonA - lonB; + const latA = Math.asin(Math.max(-1, Math.min(1, a.direction.y))); + const latB = Math.asin(Math.max(-1, Math.min(1, b.direction.y))); + if (Math.abs(latA - latB) > 1e-6) return latA - latB; + return a.stableId.localeCompare(b.stableId); } -function applyScreenAvoidanceGroup(group) { - if (group.length <= 1) return; - if (shouldCollapseScreenAvoidanceGroup(group)) { - applyCollapsedScreenAvoidanceGroup(group); - return; - } - - const count = group.length; - group.forEach((entry, index) => { - const basePosition = getMarkerStaticAvoidancePosition(entry.marker); - if (!(basePosition instanceof THREE.Vector3)) return; - const altitudeRadius = basePosition.length(); - const ringIndex = Math.floor(index / AVOIDANCE_RING_SLOT_COUNT); - const radius = Math.max(0, entry.radius) + ringIndex * Math.max(0, entry.step); - const angle = -Math.PI / 2 + (Math.PI * 2 * index) / count; - - avoidanceCenterScratch.copy(basePosition).normalize(); - avoidanceEastScratch - .copy(avoidanceNorthPole) - .cross(avoidanceCenterScratch); - if (avoidanceEastScratch.lengthSq() < TANGENT_EPSILON_SQ) { - avoidanceEastScratch.copy(avoidanceFallbackEast); - } - avoidanceEastScratch.normalize(); - avoidanceNorthScratch - .copy(avoidanceCenterScratch) - .cross(avoidanceEastScratch) - .normalize(); - - avoidancePositionScratch - .copy(basePosition) - .addScaledVector(avoidanceEastScratch, Math.cos(angle) * radius) - .addScaledVector(avoidanceNorthScratch, Math.sin(angle) * radius) - .normalize() - .multiplyScalar(altitudeRadius); - - entry.marker.position.copy(avoidancePositionScratch); - entry.marker.userData.icon_screen_avoidance_count = count; - entry.marker.userData.icon_screen_avoidance_index = index; - entry.marker.userData.icon_screen_clustered = false; +function createScreenClusterRecord(groupEntries) { + const sortedEntries = [...groupEntries].sort((a, b) => a.stableId.localeCompare(b.stableId)); + const anchorEntry = sortedEntries[0]; + const clusterPosition = new THREE.Vector3(); + let positionCount = 0; + sortedEntries.forEach((entry) => { + const position = getMarkerStaticAvoidancePosition(entry.marker); + if (!(position instanceof THREE.Vector3)) return; + clusterPosition.add(position); + positionCount += 1; }); -} + if (positionCount === 0) return null; + const fallbackPosition = getMarkerStaticAvoidancePosition(anchorEntry?.marker); + const altitudeRadius = + fallbackPosition instanceof THREE.Vector3 + ? fallbackPosition.length() + : CONFIG.earthRadius; + clusterPosition.normalize().multiplyScalar(altitudeRadius); -function getScreenAvoidanceSignature(camera) { - if (!camera?.position) return `none:${screenAvoidanceRevision}`; - const zoom = CONFIG.defaultCameraZ / Math.max(1, camera.position.length()); - const zoomBucket = - Math.round(zoom / SCREEN_AVOIDANCE_ZOOM_BUCKET_SIZE) * - SCREEN_AVOIDANCE_ZOOM_BUCKET_SIZE; - const viewportBucket = [ - Math.round(window.innerWidth || 1), - Math.round(window.innerHeight || 1), - Math.round(window.devicePixelRatio || 1), - ].join("x"); - return [ - zoomBucket.toFixed(2), - compactDotsEnabled ? "dots-on" : "dots-off", - viewportBucket, - screenAvoidanceRevision, - ].join(":"); + const center = { x: 0, y: 0 }; + sortedEntries.forEach((entry) => { + center.x += entry.x; + center.y += entry.y; + }); + center.x /= sortedEntries.length; + center.y /= sortedEntries.length; + const clusterId = getClusterRecordId(sortedEntries); + return { + clusterId, + markers: sortedEntries.map((entry) => entry.marker), + owner: anchorEntry.controller, + position: clusterPosition, + pointSize: getClusterPointSize(sortedEntries.length), + anchorStableId: anchorEntry.stableId, + clusterZoomBand: anchorEntry.clusterBandKey || anchorEntry.clusterZoomBand, + screenX: center.x, + screenY: center.y, + transitionStartedAt: getNowMs(), + }; } function recomputeScreenAvoidance(camera) { - const nextSignature = getScreenAvoidanceSignature(camera); - if (nextSignature === screenAvoidanceSignature) return; - screenAvoidanceSignature = nextSignature; - - const entries = collectScreenAvoidanceEntries(camera); - const affectedControllers = new Set(); + if (!camera) return; + const controllersToRebuild = new Set(); + interactableLayerControllers.forEach((controller) => { + controller.beginClusterUpdate?.(); + }); + const updates = collectClusterUpdates(camera); + const dynamicEntries = updates + .filter((update) => update.strategy === CLUSTER_STRATEGY_DYNAMIC_SCREEN) + .flatMap((update) => update.entries || []) + .sort((a, b) => a.stableId.localeCompare(b.stableId)); + const stableRecords = updates + .filter((update) => update.strategy === CLUSTER_STRATEGY_STABLE_SPHERICAL) + .flatMap((update) => update.records || []); + const entries = dynamicEntries; entries.forEach((entry) => { const basePosition = getMarkerStaticAvoidancePosition(entry.marker); @@ -451,11 +719,27 @@ function recomputeScreenAvoidance(camera) { entry.marker.userData.icon_screen_avoidance_count = 1; entry.marker.userData.icon_screen_avoidance_index = 0; entry.marker.userData.icon_screen_clustered = false; - affectedControllers.add(entry.controller); }); - findScreenAvoidanceGroups(entries).forEach(applyScreenAvoidanceGroup); - affectedControllers.forEach((controller) => controller.refreshPositions?.()); + findScreenClusterGroups(entries) + .flatMap(splitLargeScreenClusterGroup) + .map(createScreenClusterRecord) + .filter(Boolean) + .concat(stableRecords) + .forEach((record) => { + record.owner.addOwnedCluster?.(record); + record.markers.forEach((marker) => { + const owner = interactableLayerControllers.get(marker.userData?.icon_layer_id); + owner?.markMarkerClustered?.(marker, record); + }); + }); + + interactableLayerControllers.forEach((controller) => { + if (controller.commitClusterUpdate?.()) { + controllersToRebuild.add(controller); + } + }); + controllersToRebuild.forEach((controller) => controller.rebuildPointLayers?.(camera)); } function loadAssetImage(source) { @@ -501,11 +785,13 @@ export function createInteractableLayer(options = {}) { stateScale = {}, pulse = {}, avoidance = false, + cluster = undefined, icon, getPosition, getKind = (item) => item?.type || "default", getRotationBin = () => 0, getBucketKey = (marker) => String(getRotationBin(marker)), + getItemId = (item) => item?.id ?? item?.source_id ?? item?.entity_key, getPointSizeMultiplier = () => 1, getPointOpacity = null, getUserData = (item) => item, @@ -526,13 +812,23 @@ export function createInteractableLayer(options = {}) { const markers = []; const pointObjects = []; + const clusterPointObjects = []; const textureCache = new Map(); let pointsGroup = null; + let clusterGroup = null; let hoverOverlay = null; let lockedOverlay = null; let visible = false; let lastVisualStateKey = ""; + let lastClusterSignature = ""; + let pendingClusterSignature = ""; + let clusterUpdateActive = false; let visualStateVersion = 0; + let clusterTopologyRevision = 0; + let lastStableClusterKey = ""; + let lastStableClusterBandKey = ""; + let stableClusterRecords = []; + const ownedClusterRecords = []; const scratchDirection = new THREE.Vector3(); const scratchCameraLocal = new THREE.Vector3(); const scratchWorldPosition = new THREE.Vector3(); @@ -558,12 +854,23 @@ export function createInteractableLayer(options = {}) { Math.abs(iconAnchor.x - 0.5) > 0.001 || Math.abs(iconAnchor.y - 0.5) > 0.001; const avoidanceConfig = normalizeAvoidanceConfig(avoidance); + const clusterConfig = normalizeClusterConfig(cluster, avoidanceConfig); + const clusterWorldScratch = new THREE.Vector3(); + const clusterProjectedScratch = new THREE.Vector3(); + const clusterCameraLocalScratch = new THREE.Vector3(); + const clusterDirectionScratch = new THREE.Vector3(); function invalidateVisualState() { visualStateVersion += 1; lastVisualStateKey = ""; } + function invalidateClusterTopology() { + clusterTopologyRevision += 1; + lastStableClusterKey = ""; + lastStableClusterBandKey = ""; + } + function refreshViewportSize() { const pixelRatio = window.devicePixelRatio || 1; viewportSize.set( @@ -745,6 +1052,46 @@ export function createInteractableLayer(options = {}) { return texture; } + function getClusterMajorityColor(clusterMarkers) { + const colorCounts = new Map(); + clusterMarkers.forEach((marker) => { + const color = getMarkerColor(marker); + colorCounts.set(color, (colorCounts.get(color) || 0) + 1); + }); + return [...colorCounts.entries()] + .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))[0]?.[0] || + colors.normal || + "#ffffff"; + } + + function createClusterTexture(clusterMarkers) { + const color = getClusterMajorityColor(clusterMarkers); + const textureKey = `cluster-dot:${color}`; + if (textureCache.has(textureKey)) return textureCache.get(textureKey); + + const canvas = document.createElement("canvas"); + canvas.width = atlasCellSize; + canvas.height = atlasCellSize; + const context = canvas.getContext("2d"); + const center = atlasCellSize / 2; + const radius = atlasCellSize * COMPACT_DOT_RADIUS_RATIO; + context.clearRect(0, 0, canvas.width, canvas.height); + context.fillStyle = color; + context.shadowColor = color; + context.shadowBlur = atlasCellSize * 0.08; + context.beginPath(); + context.arc(center, center, radius, 0, Math.PI * 2); + context.fill(); + + const texture = new THREE.CanvasTexture(canvas); + texture.generateMipmaps = false; + texture.minFilter = THREE.LinearFilter; + texture.magFilter = THREE.LinearFilter; + texture.needsUpdate = true; + textureCache.set(textureKey, texture); + return texture; + } + function createOverlayTexture(marker, state, compactDotMode = false) { if (compactDotMode) { return createCompactDotTexture(marker, state); @@ -786,7 +1133,223 @@ export function createInteractableLayer(options = {}) { } function shouldUseCompactDots(camera) { - return compactDotsEnabled && getCameraZoom(camera) < COMPACT_DOT_ZOOM_THRESHOLD; + return getCameraZoom(camera) <= COMPACT_DOT_ZOOM_THRESHOLD; + } + + function getMarkerStableId(marker, fallbackIndex = 0) { + const itemId = getItemId(marker?.userData || {}); + if (itemId !== undefined && itemId !== null && String(itemId).trim() !== "") { + return String(itemId); + } + return `${id}:${fallbackIndex}`; + } + + function beginClusterUpdate() { + clusterUpdateActive = true; + ownedClusterRecords.length = 0; + markers.forEach((marker) => { + marker.userData.icon_cluster_hidden = false; + marker.userData.icon_cluster_count = 1; + marker.userData.icon_cluster_id = null; + }); + } + + function collectClusterEntries(camera) { + if (!visible || !clusterConfig.enabled || !camera || markers.length === 0) { + return null; + } + const cameraScale = getCameraScale(camera); + const zoom = getCameraZoom(camera); + if (zoom > clusterConfig.disableAboveZoom) { + lastStableClusterBandKey = ""; + return { + strategy: CLUSTER_STRATEGY_DYNAMIC_SCREEN, + entries: [], + records: [], + }; + } + const clusterZoomBand = getClusterZoomBand(zoom); + const sphericalBand = getSphericalClusterBand( + clusterConfig, + zoom, + lastStableClusterBandKey, + ); + if (clusterConfig.strategy === CLUSTER_STRATEGY_STABLE_SPHERICAL && sphericalBand?.key) { + lastStableClusterBandKey = sphericalBand.key; + } + const stableSpherical = clusterConfig.strategy === CLUSTER_STRATEGY_STABLE_SPHERICAL; + const visualPointSize = shouldUseCompactDots(camera) + ? COMPACT_DOT_POINT_SIZE + : pointSize; + const width = window.innerWidth || 1; + const height = window.innerHeight || 1; + + clusterCameraLocalScratch.copy(camera.position); + group.parent?.worldToLocal?.(clusterCameraLocalScratch); + clusterCameraLocalScratch.normalize(); + group.updateMatrixWorld?.(true); + + const entries = markers + .map((marker, index) => { + const basePosition = marker.userData?.icon_base_position || marker.position; + if (!(basePosition instanceof THREE.Vector3)) return null; + const direction = clusterDirectionScratch.copy(basePosition).normalize().clone(); + if (!stableSpherical) { + if (clusterCameraLocalScratch.dot(clusterDirectionScratch) <= 0) return null; + + clusterWorldScratch.copy(basePosition); + group.localToWorld(clusterWorldScratch); + clusterProjectedScratch.copy(clusterWorldScratch).project(camera); + if (clusterProjectedScratch.z < -1 || clusterProjectedScratch.z > 1) { + return null; + } + } + + const pointSizeMultiplier = getPointSizeMultiplier(marker) * cameraScale; + const sizePx = visualPointSize * pointSizeMultiplier; + const screenX = + (stableSpherical ? 0 : (clusterProjectedScratch.x * 0.5 + 0.5) * width) + + (0.5 - iconAnchor.x) * sizePx; + const screenY = + (stableSpherical ? 0 : (-clusterProjectedScratch.y * 0.5 + 0.5) * height) + + (0.5 - iconAnchor.y) * sizePx; + const nominalRadiusPx = Math.max(8, visualPointSize * 0.5); + const angularRadius = getAngularRadiusFromPixels( + nominalRadiusPx, + basePosition.length(), + camera, + clusterZoomBand.referenceZoom, + ); + const maxAngularDiameter = getAngularRadiusFromPixels( + Math.max(clusterZoomBand.maxDiameterPx, nominalRadiusPx * 6), + basePosition.length(), + camera, + clusterZoomBand.referenceZoom, + ); + return { + marker, + stableId: `${id}:${getMarkerStableId(marker, index)}`, + controller, + x: screenX, + y: screenY, + radiusPx: Math.max(8, sizePx * 0.5), + direction, + angularRadius, + maxAngularDiameter, + overlapFactor: getClusterOverlapFactorForBand( + clusterZoomBand, + clusterConfig.overlapFactor, + ), + maxMarkersPerDot: clusterConfig.maxMarkersPerDot, + clusterZoomBand: clusterZoomBand.referenceZoom, + clusterBandKey: sphericalBand?.key || String(clusterZoomBand.referenceZoom), + radiusWorld: basePosition.length(), + }; + }) + .filter(Boolean) + .sort((a, b) => a.stableId.localeCompare(b.stableId)); + + if (clusterConfig.strategy !== CLUSTER_STRATEGY_STABLE_SPHERICAL) { + return { + strategy: CLUSTER_STRATEGY_DYNAMIC_SCREEN, + entries, + records: [], + }; + } + + const stableKey = [ + id, + sphericalBand?.key || "unknown", + sphericalBand?.distance ?? 0, + clusterTopologyRevision, + entries.length, + ].join(":"); + if (stableKey !== lastStableClusterKey) { + stableClusterRecords = computeSphericalClusterRecords( + entries, + clusterConfig, + sphericalBand, + ); + lastStableClusterKey = stableKey; + } + return { + strategy: CLUSTER_STRATEGY_STABLE_SPHERICAL, + entries: [], + records: stableClusterRecords, + }; + } + + function addOwnedCluster(record) { + ownedClusterRecords.push(record); + } + + function markMarkerClustered(marker, record) { + if (!marker || marker.userData?.icon_layer_id !== id) return; + marker.userData.icon_cluster_hidden = true; + marker.userData.icon_cluster_count = record.markers.length; + marker.userData.icon_cluster_id = record.clusterId; + } + + function getClusterSignature() { + return ownedClusterRecords + .map((record) => + `${record.clusterId}:${record.anchorStableId}:${record.clusterZoomBand}:${record.pointSize.toFixed(1)}`, + ) + .join("|"); + } + + function commitClusterUpdate() { + if (!clusterUpdateActive) return false; + clusterUpdateActive = false; + pendingClusterSignature = getClusterSignature(); + if (pendingClusterSignature === lastClusterSignature) return false; + lastClusterSignature = pendingClusterSignature; + return true; + } + + function hasActiveRenderTransitions() { + if (clusterConfig.transitionMs <= 0) return false; + const now = getNowMs(); + return pointObjects + .concat(clusterPointObjects) + .some((points) => { + const startedAt = points.userData?.transitionStartedAt; + return startedAt && now - startedAt < clusterConfig.transitionMs; + }); + } + + function disposePointsGroup() { + if (pointsGroup?.parent) { + pointsGroup.parent.remove(pointsGroup); + } + pointObjects.forEach((points) => { + points.geometry?.dispose?.(); + points.material?.dispose?.(); + }); + pointsGroup = null; + pointObjects.length = 0; + } + + function disposeClusterGroup() { + if (clusterGroup?.parent) { + clusterGroup.parent.remove(clusterGroup); + } + clusterPointObjects.forEach((points) => { + points.geometry?.dispose?.(); + points.material?.dispose?.(); + }); + clusterGroup = null; + clusterPointObjects.length = 0; + } + + function rebuildPointLayers(camera = null) { + disposePointsGroup(); + disposeClusterGroup(); + buildPoints(camera); + buildClusters(); + if (pointsGroup) pointsGroup.visible = visible; + if (clusterGroup) clusterGroup.visible = visible; + invalidateVisualState(); } function updatePointColors(points, compactDotMode) { @@ -807,8 +1370,9 @@ export function createInteractableLayer(options = {}) { colorAttribute.needsUpdate = true; } - function buildPoints() { + function buildPoints(camera = null) { refreshViewportSize(); + const compactDotMode = camera ? shouldUseCompactDots(camera) : false; pointsGroup = new THREE.Group(); pointsGroup.visible = visible; pointsGroup.renderOrder = renderOrder; @@ -816,13 +1380,15 @@ export function createInteractableLayer(options = {}) { pointObjects.length = 0; const buckets = new Map(); - markers.forEach((marker) => { - const key = getBucketKey(marker); - if (!buckets.has(key)) { - buckets.set(key, []); - } - buckets.get(key).push(marker); - }); + markers + .filter((marker) => !marker.userData?.icon_cluster_hidden) + .forEach((marker) => { + const key = getBucketKey(marker); + if (!buckets.has(key)) { + buckets.set(key, []); + } + buckets.get(key).push(marker); + }); buckets.forEach((bucketMarkers, bucketKey) => { const count = bucketMarkers.length; @@ -848,8 +1414,12 @@ export function createInteractableLayer(options = {}) { const material = applyIconAnchor( new THREE.PointsMaterial({ - map: createPointTexture(bucketKey, bucketMarkers), - size: pointSize * getPointSizeMultiplier(bucketMarkers[0]), + map: compactDotMode + ? createCompactDotTexture(bucketMarkers[0]) + : createPointTexture(bucketKey, bucketMarkers), + size: + (compactDotMode ? COMPACT_DOT_POINT_SIZE : pointSize) * + getPointSizeMultiplier(bucketMarkers[0]), sizeAttenuation: false, vertexColors: true, transparent: true, @@ -869,6 +1439,7 @@ export function createInteractableLayer(options = {}) { bucketKey, markers: bucketMarkers, pointSizeMultiplier: getPointSizeMultiplier(bucketMarkers[0]), + transitionStartedAt: getNowMs(), }; pointObjects.push(points); pointsGroup.add(points); @@ -877,6 +1448,64 @@ export function createInteractableLayer(options = {}) { group.add(pointsGroup); } + function buildClusters() { + if (ownedClusterRecords.length === 0) return; + + clusterGroup = new THREE.Group(); + clusterGroup.visible = visible; + clusterGroup.renderOrder = renderOrder + 0.05; + clusterGroup.userData = { type: `${id}_clusters`, id }; + + ownedClusterRecords.forEach((record) => { + if (!(record.position instanceof THREE.Vector3) || record.markers.length < 2) return; + const geometry = new THREE.BufferGeometry(); + geometry.setAttribute( + "position", + new THREE.BufferAttribute( + new Float32Array([ + record.position.x, + record.position.y, + record.position.z, + ]), + 3, + ), + ); + geometry.computeBoundingSphere(); + + const material = applyIconAnchor( + new THREE.PointsMaterial({ + map: createClusterTexture(record.markers), + size: record.pointSize, + sizeAttenuation: false, + transparent: true, + opacity: baseOpacity, + depthWrite, + depthTest, + alphaTest, + }), + ); + const points = new THREE.Points(geometry, material); + points.renderOrder = renderOrder + 0.05; + points.frustumCulled = false; + points.userData = { + type: `${id}_cluster`, + id, + clusterId: record.clusterId, + markers: record.markers, + clusterPointSize: record.pointSize, + transitionStartedAt: record.transitionStartedAt || getNowMs(), + }; + clusterPointObjects.push(points); + clusterGroup.add(points); + }); + + if (clusterGroup.children.length > 0) { + group.add(clusterGroup); + } else { + clusterGroup = null; + } + } + function ensureOverlay(kind) { const existing = kind === "locked" ? lockedOverlay : hoverOverlay; if (existing) return existing; @@ -937,21 +1566,14 @@ export function createInteractableLayer(options = {}) { } function clearRenderObjects() { - if (pointsGroup?.parent) { - pointsGroup.parent.remove(pointsGroup); - } - pointObjects.forEach((points) => { - points.geometry?.dispose?.(); - points.material?.dispose?.(); - }); + disposePointsGroup(); + disposeClusterGroup(); hoverOverlay?.geometry?.dispose?.(); hoverOverlay?.material?.dispose?.(); hoverOverlay?.parent?.remove?.(hoverOverlay); lockedOverlay?.geometry?.dispose?.(); lockedOverlay?.material?.dispose?.(); lockedOverlay?.parent?.remove?.(lockedOverlay); - pointsGroup = null; - pointObjects.length = 0; hoverOverlay = null; lockedOverlay = null; } @@ -978,15 +1600,16 @@ export function createInteractableLayer(options = {}) { function refreshVisuals() { invalidateScreenAvoidance(); invalidateVisualState(); - if (!pointsGroup) return; - clearRenderObjects(); - buildPoints(); + invalidateClusterTopology(); + if (!pointsGroup && !clusterGroup) return; + rebuildPointLayers(); group.visible = visible; } function setData(items = []) { invalidateScreenAvoidance(); invalidateVisualState(); + invalidateClusterTopology(); unregisterLayerAvoidance(id); markers.length = 0; clearRenderObjects(); @@ -1020,6 +1643,43 @@ export function createInteractableLayer(options = {}) { group.visible = visible; } + function rebuildFromMarkerData(items) { + const wasVisible = visible; + setData(items); + group.visible = wasVisible; + } + + function upsertItem(item) { + const itemId = getItemId(item); + if (itemId === undefined || itemId === null || String(itemId).trim() === "") { + return false; + } + let replaced = false; + const nextItems = markers.map((marker) => { + const markerItem = marker.userData || {}; + if (String(getItemId(markerItem)) !== String(itemId)) return { ...markerItem }; + replaced = true; + return { ...markerItem, ...item }; + }); + if (!replaced) { + nextItems.push(item); + } + rebuildFromMarkerData(nextItems); + return true; + } + + function removeItem(itemId) { + if (itemId === undefined || itemId === null || String(itemId).trim() === "") { + return false; + } + const nextItems = markers + .map((marker) => ({ ...(marker.userData || {}) })) + .filter((item) => String(getItemId(item)) !== String(itemId)); + if (nextItems.length === markers.length) return false; + rebuildFromMarkerData(nextItems); + return true; + } + async function preloadAssets(items = []) { if (icon.draw && !icon.source && !icon.getSource && !icon.stateSources) { return; @@ -1059,6 +1719,7 @@ export function createInteractableLayer(options = {}) { function clearData(parent) { invalidateScreenAvoidance(); invalidateVisualState(); + invalidateClusterTopology(); unregisterLayerAvoidance(id); markers.length = 0; clearRenderObjects(); @@ -1079,10 +1740,14 @@ export function createInteractableLayer(options = {}) { visible = Boolean(nextVisible); invalidateScreenAvoidance(); invalidateVisualState(); + invalidateClusterTopology(); group.visible = visible; if (pointsGroup) { pointsGroup.visible = visible; } + if (clusterGroup) { + clusterGroup.visible = visible; + } } function setMarkerState(marker, state = "normal") { @@ -1095,9 +1760,11 @@ export function createInteractableLayer(options = {}) { function updateVisualState(focusType, focusObject, camera) { refreshViewportSize(); recomputeScreenAvoidance(camera); + const hasFocus = focusType === objectType && focusObject; if (!visible || markers.length === 0 || !pointsGroup) { if (lastVisualStateKey !== "hidden") { if (pointsGroup) pointsGroup.visible = false; + if (clusterGroup) clusterGroup.visible = false; if (hoverOverlay) hoverOverlay.visible = false; if (lockedOverlay) lockedOverlay.visible = false; lastVisualStateKey = "hidden"; @@ -1106,7 +1773,7 @@ export function createInteractableLayer(options = {}) { } pointsGroup.visible = true; - const hasFocus = focusType === objectType && focusObject; + if (clusterGroup) clusterGroup.visible = true; const lockedKey = hasFocus ? focusObject?.userData?.mmsi || focusObject?.uuid || "locked" : "none"; @@ -1125,7 +1792,8 @@ export function createInteractableLayer(options = {}) { if ( nextStateKey === lastVisualStateKey && !(pulse.enabled && hasFocus) && - !dynamicVisuals + !dynamicVisuals && + !hasActiveRenderTransitions() ) return; lastVisualStateKey = nextStateKey; @@ -1140,14 +1808,35 @@ export function createInteractableLayer(options = {}) { } updatePointColors(points, compactDotMode); points.visible = visible; + const transitionProgress = getTransitionProgress( + points.userData?.transitionStartedAt, + clusterConfig.transitionMs, + ); + const transitionScale = 0.82 + transitionProgress * 0.18; points.material.opacity = getPointOpacity?.(sampleMarker) ?? - (hasFocus ? dimmedOpacity : baseOpacity); + (hasFocus ? dimmedOpacity : baseOpacity) * transitionProgress; points.material.size = (compactDotMode ? COMPACT_DOT_POINT_SIZE : pointSize) * getPointSizeMultiplier(sampleMarker) * cameraScale * - (hasFocus ? dimmedScale : 1); + (hasFocus ? dimmedScale : 1) * + transitionScale; + }); + clusterPointObjects.forEach((points) => { + points.visible = visible; + const transitionProgress = getTransitionProgress( + points.userData?.transitionStartedAt, + clusterConfig.transitionMs, + ); + const transitionScale = 0.68 + transitionProgress * 0.32; + const pulseScale = 1 + Math.sin(Date.now() / 260) * 0.018; + points.material.opacity = (hasFocus ? dimmedOpacity : baseOpacity) * transitionProgress; + points.material.size = + (points.userData?.clusterPointSize || COMPACT_DOT_POINT_SIZE) * + (hasFocus ? dimmedScale : 1) * + transitionScale * + pulseScale; }); const hoverMarker = markers.find( @@ -1175,60 +1864,6 @@ export function createInteractableLayer(options = {}) { ); } - function collectScreenAvoidanceEntries(camera) { - if ( - !visible || - !camera || - markers.length === 0 || - !avoidanceConfig.enabled || - !avoidanceConfig.screen - ) { - return []; - } - - const compactDotMode = shouldUseCompactDots(camera); - const cameraScale = getCameraScale(camera); - const visualPointSize = compactDotMode ? COMPACT_DOT_POINT_SIZE : pointSize; - group.updateMatrixWorld?.(true); - - return markers - .map((marker) => { - const staticPosition = getMarkerStaticAvoidancePosition(marker); - if (!(staticPosition instanceof THREE.Vector3)) return null; - screenAvoidanceStaticPositionScratch.copy(staticPosition); - screenAvoidanceWorldPositionScratch.copy(screenAvoidanceStaticPositionScratch); - group.localToWorld(screenAvoidanceWorldPositionScratch); - screenAvoidanceProjectedScratch - .copy(screenAvoidanceWorldPositionScratch) - .project(camera); - if ( - screenAvoidanceProjectedScratch.z < -1 || - screenAvoidanceProjectedScratch.z > 1 - ) { - return null; - } - - const pointSizeMultiplier = getPointSizeMultiplier(marker) * cameraScale; - const sizePx = visualPointSize * pointSizeMultiplier; - return { - controller, - marker, - x: - (screenAvoidanceProjectedScratch.x * 0.5 + 0.5) * - (window.innerWidth || 1) + - (0.5 - iconAnchor.x) * sizePx, - y: - (-screenAvoidanceProjectedScratch.y * 0.5 + 0.5) * - (window.innerHeight || 1) + - (0.5 - iconAnchor.y) * sizePx, - radiusPx: Math.max(8, sizePx * 0.48), - radius: toFiniteNumber(avoidanceConfig.radius, DEFAULT_AVOIDANCE_RADIUS), - step: toFiniteNumber(avoidanceConfig.step, DEFAULT_AVOIDANCE_STEP), - }; - }) - .filter(Boolean); - } - function getPointerIntersections({ earth, camera, @@ -1249,8 +1884,59 @@ export function createInteractableLayer(options = {}) { const radiusSq = radiusPx * radiusPx; const intersections = []; const cameraScale = getCameraScale(camera); + const compactDotMode = shouldUseCompactDots(camera); + const visualPointSize = compactDotMode ? COMPACT_DOT_POINT_SIZE : pointSize; + + clusterPointObjects.forEach((clusterPoints) => { + const clusterMarkers = clusterPoints.userData?.markers || []; + const positionAttribute = clusterPoints.geometry?.getAttribute("position"); + if (!positionAttribute || clusterMarkers.length === 0) return; + + scratchDirection + .set( + positionAttribute.getX(0), + positionAttribute.getY(0), + positionAttribute.getZ(0), + ) + .normalize(); + if (scratchCameraLocal.dot(scratchDirection) <= frontFacingDotThreshold) { + return; + } + + scratchWorldPosition.set( + positionAttribute.getX(0), + positionAttribute.getY(0), + positionAttribute.getZ(0), + ); + earth.localToWorld(scratchWorldPosition); + scratchScreenPosition.copy(scratchWorldPosition).project(camera); + if (scratchScreenPosition.z < -1 || scratchScreenPosition.z > 1) { + return; + } + + const screenX = (scratchScreenPosition.x * 0.5 + 0.5) * width; + const screenY = (-scratchScreenPosition.y * 0.5 + 0.5) * height; + const deltaX = screenX - pointerX; + const deltaY = screenY - pointerY; + const clusterPointSize = + clusterPoints.userData?.clusterPointSize || COMPACT_DOT_POINT_SIZE; + const clusterRadius = Math.max(radiusPx, clusterPointSize * 0.9); + const distancePxSq = deltaX * deltaX + deltaY * deltaY; + if (distancePxSq > clusterRadius * clusterRadius) return; + + intersections.push({ + cluster: true, + clusterCount: clusterMarkers.length, + clusterMarkers, + object: null, + point: scratchWorldPosition.clone(), + distance: camera.position.distanceTo(scratchWorldPosition), + distancePxSq, + }); + }); markers.forEach((marker) => { + if (marker.userData?.icon_cluster_hidden) return; scratchDirection.copy(marker.position).normalize(); if (scratchCameraLocal.dot(scratchDirection) <= frontFacingDotThreshold) { return; @@ -1267,13 +1953,17 @@ export function createInteractableLayer(options = {}) { const screenY = (-scratchScreenPosition.y * 0.5 + 0.5) * height; const pointSizeMultiplier = getPointSizeMultiplier(marker) * cameraScale; const visualCenterX = - screenX + (0.5 - iconAnchor.x) * pointSize * pointSizeMultiplier; + screenX + (0.5 - iconAnchor.x) * visualPointSize * pointSizeMultiplier; const visualCenterY = - screenY + (0.5 - iconAnchor.y) * pointSize * pointSizeMultiplier; + screenY + (0.5 - iconAnchor.y) * visualPointSize * pointSizeMultiplier; const deltaX = visualCenterX - pointerX; const deltaY = visualCenterY - pointerY; const distancePxSq = deltaX * deltaX + deltaY * deltaY; - if (distancePxSq > radiusSq) return; + const markerRadiusPx = Math.max( + radiusPx, + visualPointSize * pointSizeMultiplier * 0.5, + ); + if (distancePxSq > Math.max(radiusSq, markerRadiusPx * markerRadiusPx)) return; intersections.push({ object: marker, @@ -1289,7 +1979,12 @@ export function createInteractableLayer(options = {}) { const controller = { refreshPositions, refreshVisuals, - collectScreenAvoidanceEntries, + beginClusterUpdate, + collectClusterEntries, + addOwnedCluster, + markMarkerClustered, + commitClusterUpdate, + rebuildPointLayers, }; interactableLayerControllers.set(id, controller); @@ -1300,6 +1995,8 @@ export function createInteractableLayer(options = {}) { getCount: () => markers.length, isVisible: () => visible, setData, + upsertItem, + removeItem, preloadAssets, clearData, attach, diff --git a/frontend/public/earth/js/layer-button-state.js b/frontend/public/earth/js/layer-button-state.js index b157166f..86ac850e 100644 --- a/frontend/public/earth/js/layer-button-state.js +++ b/frontend/public/earth/js/layer-button-state.js @@ -1,10 +1,13 @@ +import { translateText } from "./i18n.js"; + export function setButtonTooltip(button, text) { + const translatedText = translateText(text); if (button instanceof HTMLElement) { - button.title = text; + button.title = translatedText; } const tooltip = button?.querySelector(".earth-toolbar-tooltip"); if (tooltip) { - tooltip.textContent = text; + tooltip.textContent = translatedText; } } diff --git a/frontend/public/earth/js/layer-startup-tasks.js b/frontend/public/earth/js/layer-startup-tasks.js index 46960039..c29bc230 100644 --- a/frontend/public/earth/js/layer-startup-tasks.js +++ b/frontend/public/earth/js/layer-startup-tasks.js @@ -32,6 +32,7 @@ import { loadCountryBoundaries, toggleCountryBoundaries, } from "./country-boundaries.js"; +import { earthMessage } from "./i18n.js"; /** * Layer startup task registry. @@ -101,7 +102,7 @@ function registerVesselStartupTask() { if (!context.getShowVessels()) return; context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载船只..."), + resolveStartupMessage(layer, "load", earthMessage("startup.vessels")), ); await context.yieldFrame(12); try { @@ -125,7 +126,7 @@ function registerCableStartupTask() { if (!context.isCablesEnabled()) return; context.setLoadingMessage( - resolveStartupMessage(layer, "prepare", "正在加载登陆点..."), + resolveStartupMessage(layer, "prepare", earthMessage("startup.landingPoints")), ); await context.yieldFrame(12); try { @@ -137,7 +138,7 @@ function registerCableStartupTask() { await context.yieldFrame(16); context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载海缆..."), + resolveStartupMessage(layer, "load", earthMessage("startup.cables")), ); await context.yieldFrame(12); try { @@ -161,7 +162,7 @@ function registerSatelliteStartupTask() { if (!context.isSatellitesEnabled()) return; context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载卫星..."), + resolveStartupMessage(layer, "load", earthMessage("startup.satellites")), ); await context.yieldFrame(12); try { @@ -200,7 +201,7 @@ function registerSatelliteStartupTask() { function registerBGPStartupTask() { registerLayerStartupTask("bgp", (context) => async (layer) => { context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载BGP态势..."), + resolveStartupMessage(layer, "load", earthMessage("startup.bgp")), ); await context.yieldFrame(12); try { @@ -223,7 +224,7 @@ function registerEarthTextureStartupTask() { if (!context.isEarthTextureVisible()) return; context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载地球纹理..."), + resolveStartupMessage(layer, "load", earthMessage("startup.hdTexture")), ); await context.yieldFrame(12); try { @@ -241,7 +242,7 @@ function registerCloudStartupTask() { if (!context.isCloudsEnabled()) return; context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载大气云图..."), + resolveStartupMessage(layer, "load", earthMessage("startup.clouds")), ); await context.yieldFrame(12); try { @@ -257,7 +258,7 @@ function registerCloudStartupTask() { function registerCountryBoundaryStartupTask() { registerLayerStartupTask("countryBoundaries", (context) => async (layer) => { context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载海陆基座..."), + resolveStartupMessage(layer, "load", earthMessage("startup.landOceanBase")), ); await context.yieldFrame(12); try { @@ -283,7 +284,7 @@ function registerCountryBoundaryStartupTask() { function registerComputeCenterStartupTask() { registerLayerStartupTask("computeCenters", (context) => async (layer) => { context.setLoadingMessage( - resolveStartupMessage(layer, "load", "正在加载算力中心..."), + resolveStartupMessage(layer, "load", earthMessage("startup.computeCenters")), ); await context.yieldFrame(12); try { diff --git a/frontend/public/earth/js/legend.js b/frontend/public/earth/js/legend.js index 6597fee4..9d0dd585 100644 --- a/frontend/public/earth/js/legend.js +++ b/frontend/public/earth/js/legend.js @@ -1,8 +1,9 @@ import { createHUDPanel } from "./hud-panels.js"; +import { getEarthLocale, translateText } from "./i18n.js"; const LEGEND_MODES = { - cables: { title: "海缆" }, - satellites: { title: "卫星" }, + cables: { title: "海缆", compactTitleEn: "Cables" }, + satellites: { title: "卫星", compactTitleEn: "Orbits" }, countryBoundaries: { title: "国界" }, computeCenters: { title: "算力" }, vessels: { title: "船只" }, @@ -11,6 +12,7 @@ const LEGEND_MODES = { let currentLegendMode = "cables"; let legendPanel = null; +let legendLocaleListenerBound = false; let legendItemsByMode = { cables: [], satellites: [], @@ -40,6 +42,14 @@ export function initLegend() { }); } + if (!legendLocaleListenerBound) { + legendLocaleListenerBound = true; + window.addEventListener("earth:locale-change", () => { + syncCurrentLabel(currentLegendMode); + renderLegend(currentLegendMode); + }); + } + syncCurrentLabel(currentLegendMode); renderLegend(currentLegendMode); } @@ -68,24 +78,38 @@ export function setLegendItems(mode, items) { } function syncCurrentLabel(mode) { - const nextLabel = LEGEND_MODES[mode]?.title || LEGEND_MODES.cables.title; + const nextLabel = getLegendModeTitle(mode); [document.getElementById("legend-current-label"), document.getElementById("mobile-situation-legend-mode")] .forEach((labelEl) => { if (labelEl) { - labelEl.textContent = nextLabel; + const translatedLabel = getEarthLocale() === "en-US" ? nextLabel : translateText(nextLabel); + labelEl.textContent = translatedLabel; + labelEl.dataset.i18nOriginalTitle = translatedLabel; + labelEl.title = translatedLabel; } }); } +function getLegendModeTitle(mode) { + const definition = LEGEND_MODES[mode] || LEGEND_MODES.cables; + if (getEarthLocale() === "en-US" && definition.compactTitleEn) { + return definition.compactTitleEn; + } + return definition.title; +} + function renderLegend(mode) { const items = legendItemsByMode[mode] || []; const html = items .map( - (item) => ` + (item) => { + const label = escapeLegendHtml(translateText(item.label)); + return `
- ${item.label} -
`, + ${label} +
`; + }, ) .join(""); @@ -99,3 +123,12 @@ function renderLegend(mode) { mobileList.innerHTML = html; } } + +function escapeLegendHtml(value) { + return String(value ?? "") + .replace(/&/g, "&") + .replace(//g, ">") + .replace(/"/g, """) + .replace(/'/g, "'"); +} diff --git a/frontend/public/earth/js/main.js b/frontend/public/earth/js/main.js index fb0fd28a..3445b3df 100644 --- a/frontend/public/earth/js/main.js +++ b/frontend/public/earth/js/main.js @@ -1,5 +1,13 @@ import * as THREE from "three"; +import { + earthMessage, + getEarthLocale, + initEarthI18n, + localizeCountryName, + onEarthLocaleChange, + translateText, +} from "./i18n.js"; import { CONFIG, CRUISE_MODULES, @@ -15,12 +23,18 @@ import { SCENE_LIGHT_CONFIG, SURFACE_HOVER_INFO_MODES, } from "./constants.js"; +import { + canAttemptEarthRealtime, + getEarthRealtimeCooldownMs, + getEarthRealtimeUrl, + recordEarthRealtimeFailure, + recordEarthRealtimeOpen, +} from "./realtime.js"; import { vector3ToLatLon, screenToEarthCoords } from "./utils.js"; import { showStatusMessage, queueStatusMessage, updateCoordinatesDisplay, - updateZoomDisplay, updateEarthStats, setEarthStatValue, setLoading, @@ -85,6 +99,7 @@ import { clearAllCableStates, applyLandingPointVisualState, resetLandingPointVisualState, + getCableSourceRecordCount, getShowCables, clearCableData, getLandingPoints, @@ -201,6 +216,16 @@ import { toggleVessels, updateVesselVisualState, } from "./vessels.js"; +import { + applyEarthInteractableEvent, + clearEarthInteractableSelection, + clearEarthInteractables, + getEarthInteractablePointerIntersections as getEarthInteractableIconPointerIntersections, + loadEarthInteractables, + refreshEarthInteractables, + setEarthInteractableMarkerState, + updateEarthInteractableVisualState, +} from "./earth-interactables.js"; import { setupControls, getAutoRotate, @@ -222,6 +247,7 @@ import { getDayNightEnabled, getMotionDebugEnabled, getMotionDebugSkeletonOnly, + getMotionEnabledGestures, getMotionProvider, getVisibleMotionLayerDefinitions, setMotionDebugEnabled, @@ -286,6 +312,12 @@ import { setMotionDebugPanelVisible, } from "./motion-debug-panel.js"; +initEarthI18n(); +window.addEventListener("earth:status", (event) => { + const message = event.detail?.message; + if (message) showStatusMessage(message, event.detail?.type || "info"); +}); + const EARTH_RADIUS_KM = 6371; const EARTH_GRAVITATIONAL_PARAMETER_KM3_S2 = 398600.4418; const SECONDS_PER_DAY = 86400; @@ -294,6 +326,8 @@ export let scene; export let camera; export let renderer; +const WEBGL_GPU_DIAGNOSTICS_EVENT = "earth:gpu-diagnostics"; + let isDragging = false; let previousMousePosition = { x: 0, y: 0 }; let targetRotation = { x: 0, y: 0 }; @@ -302,6 +336,7 @@ let hoveredCable = null; let hoveredBGP = null; let hoveredComputeCenter = null; let hoveredVessel = null; +let hoveredEarthInteractable = null; let hoveredSatellite = null; let hoveredSatelliteIndex = null; let lockedSatellite = null; @@ -322,6 +357,7 @@ let earthTexture = null; let animationFrameId = null; let initialized = false; let destroyed = false; +let runtimeBrandConfig = null; let isDataLoading = false; let currentLoadToken = 0; let cablesEnabled = true; @@ -336,14 +372,37 @@ let calloutConnector = null; let cruiseBGPAdapter = null; let cruiseNewsAdapter = null; let cruiseSequencer = null; + +function getEarthBrandLanguage() { + return getEarthLocale() === "en-US" ? "en" : (HUD_CONFIG.brandLanguage || "zh"); +} + +function getLocalizedBrandConfig(config = null) { + if (config && typeof config === "object") { + return { ...config, variant: config.variant || getEarthBrandLanguage() }; + } + return getEarthBrandLanguage(); +} + +function remountEarthBrand() { + mountBrand(document.getElementById("brand-root"), getLocalizedBrandConfig(runtimeBrandConfig)); +} let cruiseRandomQueueSignature = ""; let cruiseRandomQueueItems = []; let earthUpdatesSocket = null; let earthUpdatesReconnectTimer = null; +let earthDataReconcileTimer = null; +let earthDataReconcileInFlight = false; +let earthUpdateFlushTimer = null; +let earthUpdateFlushPromise = null; +const pendingEarthUpdateLayers = new Set(); +const pendingEarthUpdateStrategies = new Map(); +let pendingEarthUpdateIsDatabaseChange = false; let presentationController = null; let motionControlAdapter = null; let motionCruiseAdapter = null; let motionCruiseSequencer = null; +let motionSharedCruiseSequencer = null; let motionFocusCandidates = []; let motionFocusIndex = 0; let motionFocusedCandidate = null; @@ -362,6 +421,8 @@ let shouldRefreshSatellitesAfterVisibilityResume = false; const EARTH_UPDATES_CHANNEL = "earth_updates"; const EARTH_UPDATES_RECONNECT_DELAY_MS = 5000; +const EARTH_DATA_RECONCILE_INTERVAL_MS = 60000; +const EARTH_UPDATE_CLIENT_DEBOUNCE_MS = 250; const clock = new THREE.Clock(); const interactionRaycaster = new THREE.Raycaster(); @@ -388,6 +449,21 @@ const VESSEL_POINTER_RADIUS_PX = 22; const INTERACTABLE_POINTER_RADIUS_PX = 24; const MOTION_FOCUS_RADIUS_PX = 180; const MOTION_FOCUS_REFRESH_MS = 350; + +function isEnglishEarthLocale() { + return getEarthLocale() === "en-US"; +} + +function formatUnresolvedComputeTitle(count) { + return isEnglishEarthLocale() + ? `${count} compute centers pending location` + : `${count} 个算力中心待定位`; +} + +function formatUnresolvedComputeSuffix(count) { + if (count <= 0) return ""; + return isEnglishEarthLocale() ? ` (${count} pending location)` : `（${count} 个待定位）`; +} const MOTION_MARKER_ANCHOR_SIZE_PX = 24; const MOTION_SATELLITE_ANCHOR_SIZE_PX = 18; const MOTION_CABLE_ANCHOR_SIZE_PX = 14; @@ -434,6 +510,29 @@ function getViewportAspect() { return window.innerWidth / window.innerHeight; } +function getWebGLGpuDiagnostics(activeRenderer) { + const gl = activeRenderer?.getContext?.(); + if (!gl) return null; + const debugInfo = gl.getExtension?.("WEBGL_debug_renderer_info"); + const attrs = gl.getContextAttributes?.() || {}; + return { + requestedPowerPreference: "high-performance", + contextPowerPreference: attrs.powerPreference || null, + vendor: debugInfo ? gl.getParameter(debugInfo.UNMASKED_VENDOR_WEBGL) : gl.getParameter(gl.VENDOR), + renderer: debugInfo ? gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL) : gl.getParameter(gl.RENDERER), + antialias: Boolean(attrs.antialias), + alpha: Boolean(attrs.alpha), + }; +} + +function publishWebGLGpuDiagnostics(activeRenderer) { + const diagnostics = getWebGLGpuDiagnostics(activeRenderer); + if (!diagnostics) return; + window.__planetEarthGpu = diagnostics; + console.info("[智能星球] WebGL GPU diagnostics", diagnostics); + window.dispatchEvent(new CustomEvent(WEBGL_GPU_DIAGNOSTICS_EVENT, { detail: diagnostics })); +} + function syncRendererViewport() { if (!camera || !renderer) return; camera.aspect = getViewportAspect(); @@ -570,6 +669,7 @@ export function clearLockedObject() { clearBGPSelection(); clearComputeCenterSelection(); clearVesselSelection(); + clearEarthInteractableSelection(); clearRelatedSatelliteHighlights(); setSatelliteRingState(null, "none", null); clearRuntimeSelection(); @@ -644,9 +744,11 @@ function clearTransientHoverState() { resetTransientBGPStates(); resetTransientComputeCenterStates(); resetTransientVesselStates(); + resetTransientEarthInteractableStates(); hoveredBGP = null; hoveredComputeCenter = null; hoveredVessel = null; + hoveredEarthInteractable = null; if (hoveredCable && !isSameCable(hoveredCable, lockedObject)) { setCableState(hoveredCable.userData.cableId, CABLE_STATE.NORMAL); @@ -672,6 +774,17 @@ function getVesselPointerIntersections() { }); } +function getEarthInteractablePointerIntersections() { + const earth = getEarth(); + return getEarthInteractableIconPointerIntersections({ + earth, + camera, + pointer: interactionMouse, + radiusPx: INTERACTABLE_POINTER_RADIUS_PX, + frontFacingDotThreshold: SATELLITE_CONFIG.frontFacingDotThreshold, + }); +} + function getBGPEventPointerIntersections() { const earth = getEarth(); return getBGPEventIconPointerIntersections({ @@ -776,12 +889,56 @@ function isSameVessel(marker1, marker2) { return Boolean(marker1 && marker2 && marker1.userData?.mmsi === marker2.userData?.mmsi); } -function getPrimaryBGPHoverTarget(bgpAnomalyIntersects, bgpCollectorIntersects) { - if (bgpAnomalyIntersects.length > 0) { - return bgpAnomalyIntersects[0].object; +function isSameEarthInteractable(marker1, marker2) { + return Boolean(marker1 && marker2 && marker1.userData?.id === marker2.userData?.id); +} + +function resetTransientEarthInteractableStates() { + if (hoveredEarthInteractable && hoveredEarthInteractable !== lockedObject) { + setEarthInteractableMarkerState(hoveredEarthInteractable, "normal"); } - if (bgpCollectorIntersects.length > 0) { - return bgpCollectorIntersects[0].object; + hoveredEarthInteractable = null; +} + +function applyEarthInteractableHoverState(marker) { + if (isSameEarthInteractable(hoveredEarthInteractable, marker)) return; + resetTransientEarthInteractableStates(); + if (!marker) { + hoveredEarthInteractable = null; + return; + } + hoveredEarthInteractable = marker; + if (marker !== lockedObject) { + setEarthInteractableMarkerState(marker, "hover"); + } +} + +function getFirstObjectIntersection(intersections) { + return intersections.find((hit) => !hit?.cluster && hit?.object)?.object || null; +} + +function getPrimaryClusterHit(...intersectionGroups) { + return intersectionGroups + .flat() + .filter((hit) => hit?.cluster && Number(hit.clusterCount) > 1) + .sort((a, b) => a.distancePxSq - b.distancePxSq)[0] || null; +} + +function getClusterBriefHtml(hit) { + const count = Number(hit?.clusterCount || hit?.clusterMarkers?.length || 0); + const title = isEnglishEarthLocale() ? `${count} objects` : `共 ${count} 个对象`; + const hint = isEnglishEarthLocale() ? "Zoom in to inspect individual markers" : "放大后可查看单个图标"; + return `${title}
${hint}`; +} + +function getPrimaryBGPHoverTarget(bgpAnomalyIntersects, bgpCollectorIntersects) { + const anomalyMarker = getFirstObjectIntersection(bgpAnomalyIntersects); + if (anomalyMarker) { + return anomalyMarker; + } + const collectorMarker = getFirstObjectIntersection(bgpCollectorIntersects); + if (collectorMarker) { + return collectorMarker; } return null; } @@ -791,8 +948,8 @@ function getPrimaryBGPClickTarget( bgpAnomalyIntersects, bgpCollectorIntersects, ) { - const anomalyMarker = bgpAnomalyIntersects[0]?.object || null; - const collectorMarker = bgpCollectorIntersects[0]?.object || null; + const anomalyMarker = getFirstObjectIntersection(bgpAnomalyIntersects); + const collectorMarker = getFirstObjectIntersection(bgpCollectorIntersects); if (!anomalyMarker && !collectorMarker) return null; if (!anomalyMarker) return collectorMarker; if (!collectorMarker) return anomalyMarker; @@ -821,7 +978,10 @@ function isMotionControlActive() { } function isMotionPresentationActive() { - return presentationController?.isActive?.("motion") === true; + return ( + presentationController?.isActive?.("motion") === true || + motionSharedCruiseSequencer?.isPresentationPinned?.() === true + ); } function getMotionCenterPointer() { @@ -874,7 +1034,8 @@ function getMotionCandidateScreenCoords(candidate) { candidate.type === "bgp" || candidate.type === "bgp_collector" || candidate.type === "compute_center" || - candidate.type === "vessel" + candidate.type === "vessel" || + candidate.type === "earth_interactable" ) { return getMarkerMotionScreenPoint(candidate.object) || candidate.screen || getMotionCenterScreenPoint(); } @@ -902,7 +1063,8 @@ function getMotionCandidateAnchor(candidate) { candidate.type === "bgp" || candidate.type === "bgp_collector" || candidate.type === "compute_center" || - candidate.type === "vessel" + candidate.type === "vessel" || + candidate.type === "earth_interactable" ) { return getMotionAnchorRectFromCenter( getMarkerMotionScreenPoint(candidate.object) || candidate.screen, @@ -932,22 +1094,30 @@ function getMotionIconCandidates() { const candidates = []; if (getShowBGP()) { getBGPEventIconPointerIntersections(sharedOptions).forEach((hit) => { + if (hit.cluster || !hit.object) return; candidates.push({ type: "bgp", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq }); }); getBGPCollectorIconPointerIntersections(sharedOptions).forEach((hit) => { + if (hit.cluster || !hit.object) return; candidates.push({ type: "bgp_collector", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq }); }); } if (getShowComputeCenters()) { getComputeCenterIconPointerIntersections(sharedOptions).forEach((hit) => { + if (hit.cluster || !hit.object) return; candidates.push({ type: "compute_center", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq }); }); } if (getShowVessels()) { getVesselIconPointerIntersections(sharedOptions).forEach((hit) => { + if (hit.cluster || !hit.object) return; candidates.push({ type: "vessel", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq }); }); } + getEarthInteractableIconPointerIntersections(sharedOptions).forEach((hit) => { + if (hit.cluster || !hit.object) return; + candidates.push({ type: "earth_interactable", object: hit.object, screen: getMotionScreenPointFromWorld(hit.point), distancePxSq: hit.distancePxSq }); + }); return candidates; } @@ -1023,7 +1193,7 @@ function collectMotionFocusCandidates() { ...getMotionSatelliteCandidates(), ...getMotionCableCandidates(), ] - .filter((candidate) => !layerId || getMotionCandidateLayerId(candidate) === layerId) + .filter((candidate) => candidate.type === "earth_interactable" || !layerId || getMotionCandidateLayerId(candidate) === layerId) .sort((left, right) => left.distancePxSq - right.distancePxSq); } @@ -1034,6 +1204,7 @@ function getMotionCandidateLayerId(candidate) { if (candidate.type === "bgp" || candidate.type === "bgp_collector") return "bgp"; if (candidate.type === "compute_center") return "computeCenters"; if (candidate.type === "vessel") return "vessels"; + if (candidate.type === "earth_interactable") return "interactables"; return null; } @@ -1100,10 +1271,20 @@ function ensureMotionCruiseAdapter() { } function repositionMotionConnector() { - if (!motionCruiseSequencer?.isPresentationPinned?.()) return; - const item = motionCruiseSequencer.getCurrentItem?.(); + if (motionCruiseSequencer?.isPresentationPinned?.()) { + const item = motionCruiseSequencer.getCurrentItem?.(); + if (item) { + ensureMotionCruiseAdapter().repositionConnector(); + } + } + repositionMotionSharedCruiseConnector(); +} + +function repositionMotionSharedCruiseConnector() { + if (!motionSharedCruiseSequencer?.isPresentationPinned?.()) return; + const item = motionSharedCruiseSequencer.getCurrentItem?.(); if (!item) return; - ensureMotionCruiseAdapter().repositionConnector(); + getCruiseModuleForItem(item)?.repositionPresentation?.(item); } function ensureMotionCruiseSequencer() { @@ -1132,6 +1313,81 @@ function ensureMotionCruiseSequencer() { return motionCruiseSequencer; } +function ensureMotionSharedCruiseSequencer() { + if (motionSharedCruiseSequencer) return motionSharedCruiseSequencer; + + motionSharedCruiseSequencer = new CruiseSequencer({ + presentationMode: "pinned", + isActive: isMotionControlActive, + getItems: () => getCruiseQueueItemsSorted(), + getItemId: (item) => item?.id || null, + clearCurrent: () => { + clearCruiseHighlights(); + clearLockedObject(); + if (!shouldPreserveInfoCardOnCruiseInterrupt()) { + hideInfoCard(); + } + setCruisePresentationVisible(false); + }, + onStop: ({ preservePresentation }) => { + clearBGPSelection(); + if (!preservePresentation && !lockedObject) { + hideInfoCard(); + } + }, + focusItem: async (item, { interrupt }) => + getCruiseModuleForItem(item)?.focusQueueItem?.(item, { interrupt }), + presentItem: async (item, { context }) => { + setCruisePresentationVisible(true); + const presented = await getCruiseModuleForItem(item)?.presentQueueItem?.(item, { + context, + }); + if (!presented) { + setCruisePresentationVisible(false); + } + return presented; + }, + hideItem: async (item, { context }) => { + await getCruiseModuleForItem(item)?.hidePresentation?.({ context }); + setCruisePresentationVisible(false); + }, + }); + + return motionSharedCruiseSequencer; +} + +async function applyMotionSharedCruiseFocus(direction = "next") { + await syncCruiseModuleKnownEventIds(); + const items = getCruiseQueueItemsSorted(); + if (!Array.isArray(items) || items.length === 0) return false; + + const sequencer = ensureMotionSharedCruiseSequencer(); + const currentId = + sequencer.getCurrentItemId?.() || + cruiseSequencer?.getCurrentItemId?.() || + ""; + const currentIndex = items.findIndex((item) => item?.id === currentId); + const delta = direction === "prev" ? -1 : 1; + const nextIndex = + currentIndex >= 0 + ? (currentIndex + delta + items.length) % items.length + : direction === "prev" + ? items.length - 1 + : 0; + const targetItem = items[nextIndex] || null; + if (!targetItem) return false; + + motionCruiseSequencer?.stop?.(); + motionFocusedCandidate = null; + clearMotionFocusVisual(); + + const presented = await sequencer.presentSpecificItem(targetItem, { interrupt: true }); + if (presented) { + showStatusMessage(earthMessage("status.motionCruiseTargetSwitched"), "info"); + } + return Boolean(presented); +} + async function presentMotionCandidate(candidate, { interrupt = true } = {}) { const item = createMotionCruiseItem(candidate); if (!item) return false; @@ -1150,6 +1406,7 @@ function clearMotionFocusVisual() { if (candidate.type === "bgp" || candidate.type === "bgp_collector") resetTransientBGPStates(); if (candidate.type === "compute_center") resetTransientComputeCenterStates(); if (candidate.type === "vessel") resetTransientVesselStates(); + if (candidate.type === "earth_interactable") resetTransientEarthInteractableStates(); if (candidate.type === "satellite" && candidate.index !== lockedSatelliteIndex) { setSatelliteRingState(candidate.index, "none", null); setHoveredSatelliteIndex(null); @@ -1173,6 +1430,8 @@ function applyMotionFocusVisual(candidate) { applyComputeCenterHoverState(candidate.object); } else if (candidate.type === "vessel") { applyVesselHoverState(candidate.object); + } else if (candidate.type === "earth_interactable") { + applyEarthInteractableHoverState(candidate.object); } else if (candidate.type === "satellite") { hoveredSatelliteIndex = candidate.index; hoveredSatellite = { properties: candidate.object }; @@ -1246,10 +1505,17 @@ export async function applyMotionFocus(direction = "next") { const releaseGate = beginMotionActionGate("focus"); if (!releaseGate) return false; try { + if (await applyMotionSharedCruiseFocus(direction)) { + return true; + } + refreshMotionFocusCandidates({ force: true }); if (motionFocusCandidates.length === 0) { const layer = getCurrentMotionFocusLayer(); - showStatusMessage(layer ? `动捕: ${layer.label}当前视野没有可选目标` : "动捕: 当前没有可用图层", "info"); + showStatusMessage( + earthMessage("status.motionNoTarget", { layer: layer?.label || "" }), + "info", + ); return false; } const delta = direction === "prev" ? -1 : 1; @@ -1257,7 +1523,10 @@ export async function applyMotionFocus(direction = "next") { const candidate = motionFocusCandidates[motionFocusIndex]; applyMotionFocusVisual(candidate); await presentMotionCandidate(candidate); - showStatusMessage(`动捕: 已切换到${getMotionCandidateLabel(candidate)}`, "info"); + showStatusMessage( + earthMessage("status.motionSwitchedTo", { label: getMotionCandidateLabel(candidate) }), + "info", + ); return true; } finally { releaseGate(); @@ -1270,7 +1539,7 @@ export async function applyMotionLayerSwitch(direction = "next") { try { const layers = getVisibleMotionLayerDefinitions(); if (layers.length === 0) { - showStatusMessage("动捕: 当前没有可切换的可见图层", "info"); + showStatusMessage(earthMessage("status.motionNoVisibleLayer"), "info"); return false; } const currentIndex = layers.findIndex((layer) => layer.id === motionFocusLayerId); @@ -1289,7 +1558,10 @@ export async function applyMotionLayerSwitch(direction = "next") { if (candidate) { await presentMotionCandidate(candidate); } - showStatusMessage(`动捕: 已切换到${nextLayer.label}图层`, "info"); + showStatusMessage( + earthMessage("status.motionSwitchedTo", { label: nextLayer.label, layer: true }), + "info", + ); return true; } finally { releaseGate(); @@ -1305,6 +1577,8 @@ function showMotionCandidateInfo(candidate, options = {}) { showComputeCenterInfo(candidate.object, options); } else if (candidate.type === "vessel") { showVesselInfo(candidate.object, options); + } else if (candidate.type === "earth_interactable") { + showEarthInteractableInfo(candidate.object, options); } else if (candidate.type === "cable") { showCableInfo(candidate.object, options); } else if (candidate.type === "satellite") { @@ -1319,6 +1593,7 @@ function getMotionCandidateFocusCoords(candidate) { if (candidate.type === "bgp" || candidate.type === "bgp_collector") return getBGPFocusCoords(candidate.object); if (candidate.type === "compute_center") return getComputeCenterFocusCoords(candidate.object); if (candidate.type === "vessel") return getVesselFocusCoords(candidate.object); + if (candidate.type === "earth_interactable") return getEarthInteractableFocusCoords(candidate.object); return null; } @@ -1329,17 +1604,58 @@ function getMotionCandidateLabel(candidate) { candidate.object?.userData?.name || candidate.object?.userData?.collector || candidate.object?.userData?.mmsi || + candidate.object?.userData?.label || + candidate.object?.userData?.id || "目标" ); } -function confirmMotionCandidate(candidate) { +function getMotionCandidateFromCruiseItem(item) { + const payload = item?.payload || null; + if (!payload) return null; + if ( + payload.type === "bgp" || + payload.type === "bgp_collector" || + payload.type === "compute_center" || + payload.type === "vessel" || + payload.type === "earth_interactable" || + payload.type === "cable" || + payload.type === "satellite" + ) { + return payload; + } + + const userDataType = payload?.userData?.type; + if (userDataType === "bgp" || userDataType === "bgp_collector" || userDataType === "earth_interactable") { + return { type: userDataType, object: payload }; + } + return null; +} + +function getCurrentMotionConfirmationCandidate() { + return ( + motionFocusedCandidate || + getMotionCandidateFromCruiseItem(motionCruiseSequencer?.getCurrentItem?.()) || + getMotionCandidateFromCruiseItem(motionSharedCruiseSequencer?.getCurrentItem?.()) || + motionFocusCandidates[motionFocusIndex] || + null + ); +} + +function dismissMotionPresentationsForConfirmation() { + motionCruiseSequencer?.stop?.({ preservePresentation: false }); + motionSharedCruiseSequencer?.stop?.({ preservePresentation: false }); + presentationController?.dismiss?.("motion_confirm"); +} + +function confirmMotionCandidate(candidate, { showMotionStatus = true } = {}) { const earth = getEarth(); if (!candidate || !earth) return false; interruptCruisePresentation(); + dismissMotionPresentationsForConfirmation(); clearLockedObject(); setAutoRotate(false); - const sourceCoords = candidate.screen || getMotionCenterScreenPoint(); + const sourceCoords = candidate.screen || getMotionCandidateScreenCoords(candidate) || getMotionCenterScreenPoint(); if (candidate.type === "bgp") { const marker = candidate.object; @@ -1376,6 +1692,11 @@ function confirmMotionCandidate(candidate) { showVesselTrack(marker, earth).catch((error) => { console.warn("船只轨迹加载失败:", error); }); + } else if (candidate.type === "earth_interactable") { + const marker = candidate.object; + setEarthInteractableMarkerState(marker, "locked"); + lockedObject = marker; + lockedObjectType = "earth_interactable"; } else if (candidate.type === "cable") { const cable = candidate.object; setCableState(cable.userData.cableId, CABLE_STATE.LOCKED); @@ -1400,8 +1721,19 @@ function confirmMotionCandidate(candidate) { } motionFocusedCandidate = candidate; + showMotionCandidateInfo(candidate, { + x: sourceCoords.x, + y: sourceCoords.y, + absolute: true, + anchorStable: true, + }); window.dispatchEvent(new CustomEvent("earth:open-details-tab")); - showStatusMessage(`动捕: 已确认${getMotionCandidateLabel(candidate)}`, "info"); + if (showMotionStatus) { + showStatusMessage( + earthMessage("status.motionConfirmed", { label: getMotionCandidateLabel(candidate) }), + "info", + ); + } return true; } @@ -1420,7 +1752,7 @@ function showCableInfo(cable, coords) { function getCableBriefHtml(cable) { const name = cable.userData.name || "未知海缆"; const status = cable.userData.status || ""; - return `${name}${status ? `
${status}` : ""}`; + return `${translateText(name)}${status ? `
${translateText(status)}` : ""}`; } function showSatelliteInfo(props, coords) { @@ -1549,33 +1881,55 @@ function showVesselInfo(marker, coords) { status: formatVesselStatus(marker.userData?.nav_status), length: marker.userData?.length ?? "-", received_at: marker.userData?.received_at - ? new Date(marker.userData.received_at).toLocaleString("zh-CN", { hour12: false }) + ? new Date(marker.userData.received_at).toLocaleString(getEarthLocale(), { hour12: false }) : "-", }, coords); } +function showEarthInteractableInfo(marker, coords) { + const ud = marker?.userData || {}; + showInfoCard("earth_interactable", { + id: ud.id || "-", + label: ud.label || ud.name || "-", + kind: ud.kind || "-", + latitude: Number.isFinite(Number(ud.latitude)) ? Number(ud.latitude).toFixed(4) : "-", + longitude: Number.isFinite(Number(ud.longitude)) ? Number(ud.longitude).toFixed(4) : "-", + description: ud.description || ud.summary || "-", + source: ud.source || "-", + status: ud.status || "-", + updated_at: ud.updated_at || ud.updatedAt || "-", + }, coords); +} + +function getEarthInteractableBriefHtml(marker) { + const ud = marker?.userData || {}; + const name = ud.label || ud.name || "交互点"; + const kind = ud.kind || "数据点"; + return `${translateText(name)}
${translateText(kind)}`; +} + function getVesselBriefHtml(marker) { const name = marker.userData?.name || `MMSI ${marker.userData?.mmsi}`; const speed = marker.userData?.sog ?? "-"; const vesselType = marker.userData?.vessel_type_display || marker.userData?.vessel_type_name || "Vessel"; - return `${name}
${vesselType} · ${speed} kn`; + return `${translateText(name)}
${translateText(vesselType)} · ${speed} kn`; } function getComputeCenterBriefHtml(marker) { const name = marker.userData?.name || "算力中心"; const type = formatComputeCenterTypeLabel(marker.userData?.site_type); - const location = [marker.userData?.city, marker.userData?.country] + const location = [marker.userData?.city, localizeCountryName(marker.userData?.country)] .filter(Boolean) .join(", "); - const precision = marker.userData?.is_estimated ? " · 估算位置" : ""; - return `${name}
${type}${location ? ` · ${location}` : ""}${precision}`; + const precision = marker.userData?.is_estimated ? ` · ${translateText("估算位置")}` : ""; + return `${translateText(name)}
${translateText(type)}${location ? ` · ${location}` : ""}${precision}`; } function getCountryBoundaryBriefHtml(country) { - const name = country?.nameZh || country?.name || "未知国家"; + const name = localizeCountryName(country) || translateText("未知国家"); const code = country?.isoA3 || country?.isoA2 || "-"; - const continent = country?.continent || "-"; - return `${name}
ISO: ${code}
大洲: ${continent}`; + const continent = translateText(country?.continent || "-"); + return `${name}
ISO: ${code}
${translateText("大洲")}: ${continent}`; } function getSurfacePositionBriefHtml(coords) { @@ -1585,7 +1939,7 @@ function getSurfacePositionBriefHtml(coords) { ? `${(elevMeters / 1000).toFixed(2)} km` : `${Math.round(elevMeters)} m` : "—"; - return `纬度: ${coords.lat}°
经度: ${coords.lon}°
海拔: ${elevText}`; + return `${translateText("纬度")}: ${coords.lat}°
${translateText("经度")}: ${coords.lon}°
${translateText("海拔")}: ${elevText}`; } function showBGPInfo(marker, coords) { @@ -1625,7 +1979,9 @@ function showBGPInfo(marker, coords) { ), prefix: Array.isArray(marker.userData.prefixes) && marker.userData.prefixes.length > 1 - ? `${marker.userData.prefixes[0]} 等${marker.userData.prefixes.length}个` + ? isEnglishEarthLocale() + ? `${marker.userData.prefixes[0]} + ${marker.userData.prefixes.length - 1}` + : `${marker.userData.prefixes[0]} 等${marker.userData.prefixes.length}个` : marker.userData.prefix, as_path_display: Array.isArray(marker.userData.as_path) && marker.userData.as_path.length > 0 @@ -1637,7 +1993,9 @@ function showBGPInfo(marker, coords) { : marker.userData.origin_asn, new_origin_asn: Array.isArray(marker.userData.affected_asns) && marker.userData.affected_asns.length > 3 - ? `共${marker.userData.affected_asns.length}个ASN` + ? isEnglishEarthLocale() + ? `${marker.userData.affected_asns.length} ASNs` + : `共${marker.userData.affected_asns.length}个ASN` : marker.userData.new_origin_asn, confidence: formatBGPConfidence(marker.userData.confidence), collector: marker.userData.collector, @@ -1646,7 +2004,9 @@ function showBGPInfo(marker, coords) { related_cables: relatedCables, related_satellites: marker.userData.related_satellite_count > 0 - ? `${marker.userData.related_satellite_count}颗事件附近卫星` + ? isEnglishEarthLocale() + ? `${marker.userData.related_satellite_count} nearby event satellites` + : `${marker.userData.related_satellite_count}颗事件附近卫星` : "-", location: marker.userData.location || @@ -1688,7 +2048,8 @@ function showBGPCollectorInfo(marker, coords) { function getBGPCollectorBriefHtml(marker) { const name = marker.userData.collector || "观测站"; const count = marker.userData.anomaly_count ?? 0; - return `${name}
${count} 条事件`; + const eventText = isEnglishEarthLocale() ? `${count} events` : `${count} 条事件`; + return `${translateText(name)}
${eventText}`; } function getSearchCardCoords() { @@ -1777,6 +2138,13 @@ function getVesselFocusCoords(marker) { return { lat, lon }; } +function getEarthInteractableFocusCoords(marker) { + const lat = Number(marker?.userData?.latitude); + const lon = Number(marker?.userData?.longitude); + if (!Number.isFinite(lat) || !Number.isFinite(lon)) return null; + return { lat, lon }; +} + async function focusSearchTarget(coords, zoom = Math.max(getZoomLevel(), 1.12)) { if (!coords || !camera) return; await focusEarthView(camera, { @@ -1855,7 +2223,10 @@ async function focusSearchLandingPoint(point) { }); applyLandingPointVisualState(relatedCableNames, relatedCableNames.length === 0, camera); showLandingPointInfo(point, getSearchCardCoords()); - showStatusMessage(`已定位登陆点：${point.userData?.name || "未知登陆点"}`, "info"); + showStatusMessage( + earthMessage("status.located", { target: "登陆点", name: point.userData?.name || "未知登陆点" }), + "info", + ); } async function focusSearchSatellite(index) { @@ -1889,7 +2260,13 @@ async function focusSearchSatellite(index) { } } showSatelliteInfo(sat.properties, getSearchCardCoords()); - showStatusMessage(`已定位卫星：${sat.properties.name || sat.properties.norad_cat_id || "未知卫星"}`, "info"); + showStatusMessage( + earthMessage("status.located", { + target: "卫星", + name: sat.properties.name || sat.properties.norad_cat_id || "未知卫星", + }), + "info", + ); } async function focusSearchBGPMarker(marker) { @@ -1913,7 +2290,13 @@ async function focusSearchBGPMarker(marker) { showBGPEventOverlay(marker, earth); applyBGPEventSatelliteHighlights(marker); showBGPInfo(marker, getSearchCardCoords()); - showStatusMessage(`已定位 BGP 事件：${marker.userData?.collector || "未知观测站"}`, "info"); + showStatusMessage( + earthMessage("status.located", { + target: "BGP 事件", + name: marker.userData?.collector || "未知观测站", + }), + "info", + ); return; } @@ -1923,7 +2306,13 @@ async function focusSearchBGPMarker(marker) { lockedObjectType = "bgp_collector"; showBGPCollectorCoverageOverlay(marker, earth); showBGPCollectorInfo(marker, getSearchCardCoords()); - showStatusMessage(`已定位观测站：${marker.userData?.collector || "未知观测站"}`, "info"); + showStatusMessage( + earthMessage("status.located", { + target: "观测站", + name: marker.userData?.collector || "未知观测站", + }), + "info", + ); } } @@ -1945,7 +2334,10 @@ async function focusSearchComputeCenter(marker) { lockedObjectType = "compute_center"; showComputeCenterInfo(marker, getSearchCardCoords()); showStatusMessage( - `已定位算力中心：${marker.userData?.name || "未知节点"}`, + earthMessage("status.located", { + target: "算力中心", + name: marker.userData?.name || "未知节点", + }), "info", ); } @@ -2008,7 +2400,7 @@ async function spawnComputeCenterAfterLocationSave(detail = {}) { lockedObject = result.marker; lockedObjectType = "compute_center"; } - showStatusMessage("算力中心坐标已保存", "success"); + showStatusMessage(earthMessage("status.computeCoordinatesSaved"), "success"); return result; } @@ -2034,7 +2426,10 @@ async function focusSearchVessel(marker) { console.warn("船只轨迹加载失败:", error); }); showStatusMessage( - `已定位船只：${marker.userData?.name || marker.userData?.mmsi || "未知船只"}`, + earthMessage("status.located", { + target: "船只", + name: marker.userData?.name || marker.userData?.mmsi || "未知船只", + }), "info", ); } @@ -2297,9 +2692,18 @@ function formatBGPStatusFromSummary(summary) { return "当前无活跃事件"; } -async function loadEarthStatsSummary({ shouldApply = () => true } = {}) { +async function loadEarthStatsSummary({ + shouldApply = () => true, + cacheBust = false, +} = {}) { try { - const response = await fetch(PATHS.earthSummaryApi); + const url = new URL(PATHS.earthSummaryApi, window.location.origin); + if (cacheBust) { + url.searchParams.set("_", String(Date.now())); + } + const response = await fetch(url.toString(), { + cache: cacheBust ? "no-store" : "default", + }); if (!response.ok) { throw new Error(`Earth summary HTTP ${response.status}`); } @@ -2334,16 +2738,16 @@ async function loadEarthStatsSummary({ shouldApply = () => true } = {}) { function updateComputeCenterHud(computeCenterResult) { const computeBtn = document.getElementById("toggle-compute-centers"); const unresolvedCount = Number(computeCenterResult?.unresolvedCount) || 0; - const unresolvedTooltip = - unresolvedCount > 0 ? `（${unresolvedCount} 个待定位）` : ""; + const unresolvedTooltip = formatUnresolvedComputeSuffix(unresolvedCount); if (computeBtn) { + const tooltip = getShowComputeCenters() + ? `隐藏算力中心${unresolvedTooltip}` + : `显示算力中心${unresolvedTooltip}`; setLayerButtonState(computeBtn, { active: getShowComputeCenters(), loading: false, - tooltip: getShowComputeCenters() - ? `隐藏算力中心${unresolvedTooltip}` - : `显示算力中心${unresolvedTooltip}`, + tooltip: translateText(tooltip), }); updateComputeCenterUnresolvedBadge(computeBtn, unresolvedCount); } @@ -2379,8 +2783,12 @@ function updateComputeCenterUnresolvedBadge(computeBtn, unresolvedCount) { computeBtn.dataset.unresolvedCount = String(count); badge.textContent = count > 99 ? "99+" : String(count); - badge.title = `${count} 个算力中心待定位`; - badge.setAttribute("aria-label", `${count} 个算力中心待定位，点击查看`); + const title = formatUnresolvedComputeTitle(count); + badge.title = title; + badge.setAttribute( + "aria-label", + isEnglishEarthLocale() ? `${title}, click to view` : `${title}，点击查看`, + ); } function syncComputeCenterUnresolvedCount(unresolvedCount) { @@ -2462,11 +2870,12 @@ function prepareToolInfoCard(type) { if (hadCruisePresentation || restoreCruise) { stopCruiseMode({ preserveCard: false }); - showStatusMessage("已暂停巡航，正在打开候选列表", "info"); + showStatusMessage(earthMessage("status.cruisePausedOpenCandidates"), "info"); } else if (hadMotionPresentation) { presentationController?.dismiss?.("tool_card_open"); motionCruiseSequencer?.stop?.(); - showStatusMessage("已暂停动捕目标展示，正在打开候选列表", "info"); + motionSharedCruiseSequencer?.stop?.(); + showStatusMessage(earthMessage("status.motionPausedOpenCandidates"), "info"); } activeToolInfoCard = { @@ -2497,7 +2906,7 @@ function resumeCruiseAfterToolInfoCard(context) { return; } - showStatusMessage("候选列表已关闭，巡航已恢复", "info"); + showStatusMessage(earthMessage("status.candidatesClosedCruiseResumed"), "info"); ensureCruisePolling(); syncCruiseModuleKnownEventIds() .catch((error) => { @@ -2963,6 +3372,11 @@ function getCurrentCruiseBGPMarker() { return item?.moduleId === CRUISE_MODULES.BGP ? item.payload || null : null; } +function getCurrentMotionSharedCruiseBGPMarker() { + const item = motionSharedCruiseSequencer?.getCurrentItem?.() ?? null; + return item?.moduleId === CRUISE_MODULES.BGP ? item.payload || null : null; +} + async function syncCruiseModuleKnownEventIds() { const modules = getEnabledCruiseModuleDefinitions(); await Promise.all( @@ -3138,6 +3552,7 @@ function handleRotationModeChange(event) { setupMotionControl(); if (detailMode !== ROTATION_MODE.MOTION) { motionCruiseSequencer?.stop?.(); + motionSharedCruiseSequencer?.stop?.(); } return; } @@ -3162,6 +3577,7 @@ function handleRotationModeChange(event) { function handleCruiseModulesChange() { interruptCruisePresentation({ resetLoop: true }); + motionSharedCruiseSequencer?.interruptPresentation?.({ resetLoop: true }); clearBGPSelection(); cruiseRandomQueueSignature = ""; cruiseRandomQueueItems = []; @@ -3188,6 +3604,7 @@ function handleCruiseModulesChange() { function handleCruiseQueueSettingsChange() { interruptCruisePresentation({ resetLoop: true }); + motionSharedCruiseSequencer?.interruptPresentation?.({ resetLoop: true }); cruiseRandomQueueSignature = ""; cruiseRandomQueueItems = []; @@ -3202,12 +3619,12 @@ function handleCruiseQueueSettingsChange() { function handleCruiseNextCardShortcut() { if (!isCruiseModeActive()) { - showStatusMessage("切换到巡航模式后可切换卡片", "info"); + showStatusMessage(earthMessage("status.cruiseModeRequired"), "info"); return; } if (!getAutoRotate()) { - showStatusMessage("巡航已暂停，按空格恢复", "info"); + showStatusMessage(earthMessage("status.cruisePausedSpace"), "info"); return; } @@ -3450,12 +3867,7 @@ function getSatellitePointerIntersections(event) { function buildLoadErrorMessage(errors) { if (errors.length === 0) return ""; - return errors - .map( - ({ label, reason }) => - `${label}加载失败: ${reason?.message || String(reason)}`, - ) - .join("；"); + return earthMessage("status.loadFailedList", { items: errors }); } function updateSatelliteToggleUi(enabled, satelliteCount = getSatelliteCount()) { @@ -3468,10 +3880,79 @@ function updateSatelliteToggleUi(enabled, satelliteCount = getSatelliteCount()) }); } - const resolvedCount = satelliteCount || earthStatsSummary?.satelliteCount || 0; + const resolvedCount = getAuthoritativeSatelliteCount(satelliteCount); setEarthStatValue("satellite-count", `${resolvedCount} 颗`); } +function getAuthoritativeSatelliteCount(fallback = getSatelliteCount()) { + return getAuthoritativeCount("satelliteCount", fallback); +} + +function getAuthoritativeCount(summaryKey, fallback = 0) { + const summaryCount = Number(earthStatsSummary?.[summaryKey]); + if (Number.isFinite(summaryCount)) { + return summaryCount; + } + const fallbackCount = Number(fallback); + return Number.isFinite(fallbackCount) ? fallbackCount : 0; +} + +function hasAuthoritativeZeroSummary(...summaryKeys) { + if (!earthStatsSummary || summaryKeys.length === 0) return false; + return summaryKeys.every((summaryKey) => { + const summaryCount = Number(earthStatsSummary?.[summaryKey]); + return Number.isFinite(summaryCount) && summaryCount <= 0; + }); +} + +function applySatelliteEmptyState() { + satelliteHydrationToken += 1; + clearSatelliteData(); + updateSatelliteToggleUi(satellitesEnabled || getShowSatellites(), 0); + setLegendItems("satellites", getSatelliteLegendItems()); + refreshLegend(); +} + +function applyCableEmptyState() { + const earth = getEarth(); + clearCableData(earth); + updateCableToggleUi(cablesEnabled || getShowCables()); + setLegendItems("cables", getCableLegendItems()); + refreshLegend(); +} + +function applyComputeCenterEmptyState() { + const earth = getEarth(); + clearComputeCenterData(earth); + updateComputeCenterHud({ + totalCount: 0, + unresolvedCount: 0, + }); + setLegendItems("computeCenters", getComputeCenterLegendItems()); + refreshLegend(); +} + +function applyVesselEmptyState() { + const earth = getEarth(); + clearVesselData(earth); + updateVesselToggleUi(vesselsEnabled || getShowVessels(), 0); + setLegendItems("vessels", getVesselLegendItems()); + refreshLegend(); +} + +function applyBGPEmptyState() { + const earth = getEarth(); + clearBGPData(earth); + updateBGPHud({ + totalCount: 0, + anomalyCount: 0, + collectorCount: 0, + }); + setLegendItems("bgp", getBGPLegendItems()); + refreshLegend(); + ensureBGPCruiseAdapter().syncKnownEventIds(); +} + function updateVesselHud(result = {}) { const count = Number(result.totalCount ?? getVesselCount() ?? 0); setEarthStatValue("vessel-count", `${count} 艘`); @@ -3516,7 +3997,7 @@ function updateVesselToggleUi(enabled, vesselCount = getVesselCount()) { tooltip: enabled ? "隐藏船只" : "显示船只", }); } - setEarthStatValue("vessel-count", `${vesselCount || 0} 艘`); + setEarthStatValue("vessel-count", `${getAuthoritativeCount("vesselCount", vesselCount)} 艘`); setEarthStatValue("vessel-live-summary", formatVesselLiveSummary()); } @@ -3530,9 +4011,11 @@ function updateCableToggleUi(enabled) { }); } - const cableCount = getCableLines().length || earthStatsSummary?.cableCount || 0; - const landingPointCount = - getLandingPoints().length || earthStatsSummary?.landingPointCount || 0; + const cableCount = getAuthoritativeCount("cableCount", getCableLines().length); + const landingPointCount = getAuthoritativeCount( + "landingPointCount", + getLandingPoints().length, + ); setEarthStatValue("cable-count", `${cableCount}个`); setEarthStatValue("landing-point-count", `${landingPointCount}个`); } @@ -3600,6 +4083,7 @@ async function ensureSatellitesEnabled() { clearSatelliteData(); const loadResult = await loadSatellites({ limit: getInitialSatelliteLoadLimit(), + cacheBust: true, }); if ( @@ -3644,7 +4128,8 @@ async function ensureVesselsEnabled() { if (!earth) return 0; vesselsEnabled = true; - const result = await loadVessels(scene, earth); + const zoom = getZoomLevel(); + const result = await loadVessels(scene, earth, { zoom }); toggleVessels(true); startVesselRealtime(earth, { onUpdate: ({ totalCount }) => { @@ -3680,16 +4165,22 @@ function disableSatellites() { } function updateStatsSummary() { - const cableCount = getCableLines().length || earthStatsSummary?.cableCount || 0; - const landingPointCount = - getLandingPoints().length || earthStatsSummary?.landingPointCount || 0; - const satelliteCount = getSatelliteCount() || earthStatsSummary?.satelliteCount || 0; - const vesselCount = getVesselCount() || earthStatsSummary?.vesselCount || 0; - const computeCenterCount = - getComputeCenterCount() || earthStatsSummary?.computeCenterCount || 0; - const bgpEventCount = getBGPCount() || earthStatsSummary?.bgpEventCount || 0; - const bgpCollectorCount = - getBGPCollectorCount() || earthStatsSummary?.bgpCollectorCount || 0; + const cableCount = getAuthoritativeCount("cableCount", getCableLines().length); + const landingPointCount = getAuthoritativeCount( + "landingPointCount", + getLandingPoints().length, + ); + const satelliteCount = getAuthoritativeSatelliteCount(); + const vesselCount = getAuthoritativeCount("vesselCount", getVesselCount()); + const computeCenterCount = getAuthoritativeCount( + "computeCenterCount", + getComputeCenterCount(), + ); + const bgpEventCount = getAuthoritativeCount("bgpEventCount", getBGPCount()); + const bgpCollectorCount = getAuthoritativeCount( + "bgpCollectorCount", + getBGPCollectorCount(), + ); updateEarthStats({ cableCount: `${cableCount}个`, landingPointCount: `${landingPointCount}个`, @@ -3729,20 +4220,36 @@ export function init() { destroyed = false; initialized = true; updateHudScale(); - const brandRoot = document.getElementById("brand-root"); - mountBrand(brandRoot, HUD_CONFIG.brandLanguage); + remountEarthBrand(); fetchEarthBrandConfig() .then((brandConfig) => { - mountBrand(brandRoot, brandConfig); + runtimeBrandConfig = brandConfig; + remountEarthBrand(); }) .catch((error) => { console.warn("Earth brand config unavailable, using defaults.", error); }); + cleanupFns.push(onEarthLocaleChange(() => { + remountEarthBrand(); + updateCableToggleUi(getShowCables()); + updateSatelliteToggleUi(getShowSatellites()); + updateComputeCenterHud({ + totalCount: getComputeCenterCount(), + unresolvedCount: getUnresolvedComputeCenters().length, + }); + updateVesselToggleUi(getShowVessels()); + updateBGPHud({ + totalCount: getBGPCount(), + collectorCount: getBGPCollectorCount(), + }); + updateStatsSummary(); + })); initTVPanel(); initEarthAbout(); initEarthOobe(); initNewsPanel(); connectEarthUpdatesRealtime(); + startEarthDataReconciliation(); initSearchPanel({ resolveResults: resolveEarthSearchResults, onSelectResult: handleSearchSelection, @@ -3755,7 +4262,7 @@ export function init() { 0.1, 5000, ); - camera.position.z = CONFIG.defaultCameraZ; + setZoomLevel(getDefaultEarthZoomLevel(), camera); setSatelliteCamera(camera); renderer = new THREE.WebGLRenderer({ @@ -3766,6 +4273,7 @@ export function init() { syncRendererViewport(); renderer.setClearColor(0x02040a, 1); renderer.setPixelRatio(window.devicePixelRatio); + publishWebGLGpuDiagnostics(renderer); const container = document.getElementById("container"); if (container) { @@ -3836,6 +4344,7 @@ function setupMotionControl() { setMotionDebugPanelSkeletonOnly(getMotionDebugSkeletonOnly()); if (!motionEnabled) { motionCruiseSequencer?.stop?.(); + motionSharedCruiseSequencer?.stop?.(); } motionControlAdapter = createMotionControlAdapter({ enabled: motionEnabled, @@ -3846,6 +4355,8 @@ function setupMotionControl() { onFocus: applyMotionFocus, onLayer: applyMotionLayerSwitch, onStatus: (message, type) => showStatusMessage(message, type), + debugSkeleton: motionDebugEnabled, + enabledGestures: getMotionEnabledGestures(), }); motionControlAdapter.start(); } @@ -3876,7 +4387,7 @@ export function applyMotionRotate(axisOrDirection, directionOrIntensity = 1, may up: "向上旋转", down: "向下旋转", }[direction] || "旋转"; - showStatusMessage(`动捕: ${label}`, "info"); + showStatusMessage(earthMessage("status.motionPrefix", { text: label }), "info"); return true; } @@ -3895,20 +4406,20 @@ export function applyMotionConfirm() { const releaseGate = beginMotionActionGate("confirm"); if (!releaseGate) return false; try { - let candidate = motionFocusedCandidate || motionFocusCandidates[motionFocusIndex] || null; + let candidate = getCurrentMotionConfirmationCandidate(); if (!candidate) { refreshMotionFocusCandidates({ force: true }); - candidate = motionFocusedCandidate || motionFocusCandidates[motionFocusIndex] || null; + candidate = getCurrentMotionConfirmationCandidate(); } if (candidate && confirmMotionCandidate(candidate)) { return true; } if (lockedObject || lockedSatellite) { window.dispatchEvent(new CustomEvent("earth:open-details-tab")); - showStatusMessage("动捕: 已确认当前目标", "info"); + showStatusMessage(earthMessage("status.motionConfirmCurrent"), "info"); return true; } - showStatusMessage("动捕: 请先选择目标", "info"); + showStatusMessage(earthMessage("status.motionSelectTargetFirst"), "info"); return false; } finally { releaseGate(); @@ -4022,7 +4533,7 @@ function shouldHydrateFullSatelliteSet(loadResult) { async function hydrateAllSatellitesInBackground(guardFn) { try { - const loadResult = await loadSatellites({ limit: null }); + const loadResult = await loadSatellites({ limit: null, cacheBust: true }); if (!guardFn()) return; updateSatelliteToggleUi(true, loadResult.count); setLegendItems("satellites", getSatelliteLegendItems()); @@ -4033,17 +4544,13 @@ async function hydrateAllSatellitesInBackground(guardFn) { } } -function getEarthUpdatesRealtimeUrl() { - const protocol = window.location.protocol === "https:" ? "wss:" : "ws:"; - return `${protocol}//${window.location.host}/ws`; -} - function scheduleEarthUpdatesReconnect() { if (earthUpdatesReconnectTimer || destroyed) return; + const delay = Math.max(EARTH_UPDATES_RECONNECT_DELAY_MS, getEarthRealtimeCooldownMs()); earthUpdatesReconnectTimer = window.setTimeout(() => { earthUpdatesReconnectTimer = null; connectEarthUpdatesRealtime(); - }, EARTH_UPDATES_RECONNECT_DELAY_MS); + }, delay); } async function refreshBGPRealtimeLayer() { @@ -4059,8 +4566,8 @@ async function refreshCablesRealtimeLayer() { const earth = getEarth(); if (!scene || !earth) return; const wasEnabled = cablesEnabled || getShowCables(); - clearCableData(earth); if (!wasEnabled) { + clearCableData(earth); updateCableToggleUi(false); setLegendItems("cables", getCableLegendItems()); refreshLegend(); @@ -4075,55 +4582,253 @@ async function refreshCablesRealtimeLayer() { refreshLegend(); } -async function refreshSatellitesRealtimeLayer() { - if (!satellitesEnabled && !getShowSatellites()) return; - await ensureSatellitesEnabled(); +function shouldRefreshLayerFromServerState(handler) { + return ( + handler.isVisible?.() || + handler.localCount?.() > 0 || + handler.summaryKeys?.some((summaryKey) => getAuthoritativeCount(summaryKey, 0) > 0) + ); } -async function refreshVesselsRealtimeLayer() { - if (!vesselsEnabled && !getShowVessels()) return; - await ensureVesselsEnabled(); +function shouldClearLayerFromServerState(handler) { + return hasAuthoritativeZeroSummary(...(handler.summaryKeys || [])); } -async function refreshEarthLayerFromUpdate(layer) { - if (layer === "bgp") { - await refreshBGPRealtimeLayer(); - } else if (layer === "computeCenters") { - await refreshComputeCentersAfterLocationSave(); - } else if (layer === "cables") { - await refreshCablesRealtimeLayer(); - } else if (layer === "satellites") { - await refreshSatellitesRealtimeLayer(); - } else if (layer === "vessels") { - await refreshVesselsRealtimeLayer(); +function getLayerSummaryTotal(handler) { + if (!earthStatsSummary || !Array.isArray(handler.summaryKeys)) return null; + let total = 0; + for (const summaryKey of handler.summaryKeys) { + const count = Number(earthStatsSummary?.[summaryKey]); + if (!Number.isFinite(count)) return null; + total += count; + } + return total; +} + +function shouldRefreshLayerForCountMismatch(handler) { + if (!handler.refreshOnCountMismatch || !handler.isVisible?.()) return false; + const summaryTotal = getLayerSummaryTotal(handler); + if (summaryTotal === null) return false; + return summaryTotal !== (handler.localCount?.() || 0); +} + +const EARTH_DATA_LAYER_HANDLERS = { + bgp: { + summaryKeys: ["bgpEventCount", "bgpCollectorCount"], + isVisible: () => getShowBGP(), + localCount: () => getBGPCount() + getBGPCollectorCount(), + clear: applyBGPEmptyState, + refresh: refreshBGPRealtimeLayer, + }, + cables: { + summaryKeys: ["cableCount", "landingPointCount"], + isVisible: () => cablesEnabled || getShowCables(), + localCount: () => getCableSourceRecordCount(), + clear: applyCableEmptyState, + refresh: refreshCablesRealtimeLayer, + refreshOnCountMismatch: true, + }, + computeCenters: { + summaryKeys: ["computeCenterCount"], + isVisible: () => getShowComputeCenters(), + localCount: () => getComputeCenterCount() + getUnresolvedComputeCenters().length, + clear: applyComputeCenterEmptyState, + refresh: refreshComputeCentersAfterLocationSave, + refreshOnCountMismatch: true, + }, + satellites: { + summaryKeys: ["satelliteCount"], + isVisible: () => satellitesEnabled || getShowSatellites(), + localCount: () => getSatelliteCount(), + clear: applySatelliteEmptyState, + refresh: ensureSatellitesEnabled, + refreshOnCountMismatch: true, + }, + vessels: { + summaryKeys: ["vesselCount"], + isVisible: () => vesselsEnabled || getShowVessels(), + localCount: () => getVesselCount(), + clear: applyVesselEmptyState, + refresh: ensureVesselsEnabled, + }, +}; + +async function refreshDataBackedEarthLayer(layer, options = {}) { + const handler = EARTH_DATA_LAYER_HANDLERS[layer]; + if (!handler) return false; + const shouldClearBeforeReload = options.strategy === "clear_then_reload"; + if (shouldClearBeforeReload) { + handler.clear?.(); + } + if (!shouldClearBeforeReload && shouldClearLayerFromServerState(handler)) { + handler.clear?.(); + return true; + } + if (!options.force && !shouldRefreshLayerFromServerState(handler)) { + return false; + } + await handler.refresh?.(); + return true; +} + +async function refreshInteractablesRealtimeLayer() { + const earth = getEarth(); + if (!earth) return; + await refreshEarthInteractables(earth); +} + +async function refreshEarthLayerFromUpdate(layer, options = {}) { + if (EARTH_DATA_LAYER_HANDLERS[layer]) { + await refreshDataBackedEarthLayer(layer, options); } else if (layer === "news") { await refreshEarthNews({ silent: true }); + } else if (layer === "interactables") { + await refreshInteractablesRealtimeLayer(); } } -function handleEarthUpdateFrame(payload = {}) { - const layers = Array.isArray(payload.layers) ? payload.layers : []; - if (layers.length === 0) return; - Promise.allSettled(layers.map((layer) => refreshEarthLayerFromUpdate(layer))) - .then((results) => { - const failed = results.some((result) => result.status === "rejected"); - if (failed) { - console.warn("部分 Earth 图层实时刷新失败:", { payload, results }); - } - updateStatsSummary(); - if (isCruiseModeActive()) { - syncCruiseModuleKnownEventIds().catch((error) => { - console.warn("实时刷新后同步巡航模块失败:", error); - }); +async function flushPendingEarthUpdates() { + if (earthUpdateFlushPromise) return earthUpdateFlushPromise; + const layers = Array.from(pendingEarthUpdateLayers); + const strategies = new Map(pendingEarthUpdateStrategies); + const isDatabaseChange = pendingEarthUpdateIsDatabaseChange; + pendingEarthUpdateLayers.clear(); + pendingEarthUpdateStrategies.clear(); + pendingEarthUpdateIsDatabaseChange = false; + if (layers.length === 0) return Promise.resolve(); + + earthUpdateFlushPromise = (async () => { + if (isDatabaseChange) { + await loadEarthStatsSummary({ + shouldApply: () => !destroyed, + cacheBust: true, + }); + } + const results = await Promise.allSettled( + layers.map((layer) => refreshEarthLayerFromUpdate(layer, { + force: isDatabaseChange, + strategy: strategies.get(layer), + })), + ); + const failed = results.some((result) => result.status === "rejected"); + if (failed) { + console.warn("部分 Earth 图层实时刷新失败:", { layers, results }); + } + updateStatsSummary(); + if (isCruiseModeActive()) { + syncCruiseModuleKnownEventIds().catch((error) => { + console.warn("实时刷新后同步巡航模块失败:", error); + }); + } + })() + .catch((error) => { + console.warn("Earth 实时刷新失败:", { layers, error }); + }) + .finally(() => { + earthUpdateFlushPromise = null; + if (pendingEarthUpdateLayers.size > 0 && !earthUpdateFlushTimer) { + earthUpdateFlushTimer = window.setTimeout(() => { + earthUpdateFlushTimer = null; + flushPendingEarthUpdates(); + }, EARTH_UPDATE_CLIENT_DEBOUNCE_MS); } }); + + return earthUpdateFlushPromise; +} + +function scheduleEarthUpdateFlush() { + if (earthUpdateFlushTimer || earthUpdateFlushPromise) return; + earthUpdateFlushTimer = window.setTimeout(() => { + earthUpdateFlushTimer = null; + flushPendingEarthUpdates(); + }, EARTH_UPDATE_CLIENT_DEBOUNCE_MS); +} + +function handleEarthUpdateFrame(payload = {}) { + if (payload.entity === "interactable") { + const earth = getEarth(); + const applied = applyEarthInteractableEvent(earth, payload); + if (applied) { + updateStatsSummary(); + } + return; + } + const layers = Array.isArray(payload.layers) ? payload.layers : []; + if (layers.length === 0) return; + const isDatabaseChange = payload.action === "database_changed"; + const strategy = typeof payload.refresh_strategy === "string" ? payload.refresh_strategy : ""; + layers.forEach((layer) => { + pendingEarthUpdateLayers.add(layer); + if (strategy === "clear_then_reload" || !pendingEarthUpdateStrategies.has(layer)) { + pendingEarthUpdateStrategies.set(layer, strategy || "reload"); + } + }); + pendingEarthUpdateIsDatabaseChange = pendingEarthUpdateIsDatabaseChange || isDatabaseChange; + scheduleEarthUpdateFlush(); +} + +async function reconcileEarthDataFromServer() { + if (destroyed || earthDataReconcileInFlight) return; + if (document.visibilityState === "hidden") return; + earthDataReconcileInFlight = true; + try { + await loadEarthStatsSummary({ + shouldApply: () => !destroyed, + cacheBust: true, + }); + if (destroyed) return; + + let changed = false; + for (const [layer, handler] of Object.entries(EARTH_DATA_LAYER_HANDLERS)) { + if (shouldClearLayerFromServerState(handler) && handler.localCount?.() > 0) { + handler.clear?.(); + changed = true; + continue; + } + const serverHasData = handler.summaryKeys?.some( + (summaryKey) => getAuthoritativeCount(summaryKey, 0) > 0, + ); + if ( + (handler.isVisible?.() && serverHasData && handler.localCount?.() === 0) || + shouldRefreshLayerForCountMismatch(handler) + ) { + await refreshDataBackedEarthLayer(layer, { force: true }); + changed = true; + } + } + + if (changed) { + updateStatsSummary(); + } + } catch (error) { + console.warn("Earth 数据一致性巡检失败:", error); + } finally { + earthDataReconcileInFlight = false; + } +} + +function startEarthDataReconciliation() { + if (earthDataReconcileTimer || destroyed) return; + earthDataReconcileTimer = window.setInterval( + () => { + reconcileEarthDataFromServer(); + }, + EARTH_DATA_RECONCILE_INTERVAL_MS, + ); } function connectEarthUpdatesRealtime() { - if (earthUpdatesSocket || typeof WebSocket === "undefined" || destroyed) return; - const socket = new WebSocket(getEarthUpdatesRealtimeUrl()); + if (earthUpdatesSocket || destroyed) return; + if (!canAttemptEarthRealtime()) { + scheduleEarthUpdatesReconnect(); + return; + } + const socket = new WebSocket(getEarthRealtimeUrl()); earthUpdatesSocket = socket; socket.onopen = () => { + socket.__planetOpened = true; + recordEarthRealtimeOpen(); if (earthUpdatesReconnectTimer) { window.clearTimeout(earthUpdatesReconnectTimer); earthUpdatesReconnectTimer = null; @@ -4153,6 +4858,9 @@ function connectEarthUpdatesRealtime() { if (earthUpdatesSocket === socket) { earthUpdatesSocket = null; } + if (!socket.__planetOpened) { + recordEarthRealtimeFailure(); + } scheduleEarthUpdatesReconnect(); }; socket.onerror = () => { @@ -4178,10 +4886,11 @@ async function loadData() { clearCableData(earth); clearComputeCenterData(earth); clearVesselData(earth); + clearEarthInteractables(earth); clearSatelliteData(); clearCountryBoundaryHover(); - setLoadingMessage("正在初始化..."); + setLoadingMessage(earthMessage("loading.initializing")); setLoading(true); await yieldFrame(18); if (loadToken !== currentLoadToken) { isDataLoading = false; return; } @@ -4248,13 +4957,17 @@ async function loadData() { if (loadToken !== currentLoadToken) { isDataLoading = false; return; } } + loadEarthInteractables(earth, { silent: true }).catch((error) => { + console.warn("Earth interactables 加载失败:", error); + }); + // Step 6 — Terrain (if enabled) if (getShowTerrain()) { const terrainLayer = startupLayers.find((layer) => layer.id === "terrain"); const terrainMessage = resolveStartupMessage( terrainLayer, "load", - "正在渲染地形...", + earthMessage("startup.terrain"), ); setLoadingMessage(terrainMessage); await yieldFrame(24); @@ -4285,7 +4998,7 @@ async function loadData() { queueStatusMessage(errorMessage, "error"); } else { hideError(); - queueStatusMessage("数据已加载", "success"); + queueStatusMessage(earthMessage("status.dataLoaded"), "success"); } applyDeferredLayerVisibilitySettings() @@ -4329,13 +5042,13 @@ export async function setCablesEnabled( clearSelectionAndInfo(); disableCables(); if (!suppressStatus) { - showStatusMessage("线缆已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "线缆", visible: false }), "info"); } return 0; } if (!suppressLoadingUi) { - setLoadingMessage("正在加载线缆数据..."); + setLoadingMessage(earthMessage("loading.cableData")); setLoading(true); hideError(); } @@ -4343,19 +5056,20 @@ export async function setCablesEnabled( try { const cableCount = await ensureCablesEnabled(); if (!suppressStatus) { - showStatusMessage("线缆已显示", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "线缆", visible: true }), "info"); } return cableCount; } catch (error) { cablesEnabled = false; clearCableData(getEarth()); updateCableToggleUi(false); - const message = `线缆加载失败: ${error?.message || String(error)}`; + const reason = error?.message || String(error); + const message = earthMessage("status.layerLoadFailed", { layer: "线缆", error: reason }); void reportEarthClientLog({ level: "error", category: "layer-toggle", module: "cables", - message, + message: `线缆加载失败: ${reason}`, detail: error, }); if (!suppressLoadingUi) { @@ -4380,7 +5094,7 @@ export async function setCountryBoundariesEnabled( toggleCountryBoundaries(false); clearCountryBoundaryHover(); if (!suppressStatus) { - showStatusMessage("国界已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "国界", visible: false }), "info"); } return 0; } @@ -4396,18 +5110,19 @@ export async function setCountryBoundariesEnabled( setLegendItems("countryBoundaries", getCountryBoundaryLegendItems()); refreshLegend(); if (!suppressStatus) { - showStatusMessage("国界已显示", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "国界", visible: true }), "info"); } return countryCount; } catch (error) { toggleCountryBoundaries(false); clearCountryBoundaryHover(); - const message = `国界加载失败: ${error?.message || String(error)}`; + const reason = error?.message || String(error); + const message = earthMessage("status.layerLoadFailed", { layer: "国界", error: reason }); void reportEarthClientLog({ level: "error", category: "layer-toggle", module: "country-boundaries", - message, + message: `国界加载失败: ${reason}`, detail: error, }); if (!suppressStatus) { @@ -4476,7 +5191,7 @@ export async function setHighResTextureEnabled(enabled, { suppressStatus = false } } if (!suppressStatus) { - showStatusMessage(enabled ? "高清材质已启用" : "高清材质已隐藏", "info"); + showStatusMessage(earthMessage("status.layerEnabled", { layer: "高清材质", enabled }), "info"); } return enabled; } @@ -4497,7 +5212,7 @@ export async function setAtmosphereCloudsEnabled(enabled, { suppressStatus = fal } } if (!suppressStatus) { - showStatusMessage(enabled ? "大气云图已显示" : "大气云图已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "大气云图", visible: enabled }), "info"); } return enabled; } @@ -4522,7 +5237,7 @@ export async function setSatellitesEnabled( } if (!suppressLoadingUi) { - setLoadingMessage("正在加载卫星数据..."); + setLoadingMessage(earthMessage("loading.satelliteData")); setLoading(true); hideError(); } @@ -4530,19 +5245,20 @@ export async function setSatellitesEnabled( try { const satelliteCount = await ensureSatellitesEnabled(); if (!suppressStatus) { - showStatusMessage("卫星已显示", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "卫星", visible: true }), "info"); } return satelliteCount; } catch (error) { satellitesEnabled = false; resetSatelliteState(); updateSatelliteToggleUi(false, 0); - const message = `卫星加载失败: ${error?.message || String(error)}`; + const reason = error?.message || String(error); + const message = earthMessage("status.layerLoadFailed", { layer: "卫星", error: reason }); void reportEarthClientLog({ level: "error", category: "layer-toggle", module: "satellites", - message, + message: `卫星加载失败: ${reason}`, detail: error, }); if (!suppressLoadingUi) { @@ -4572,13 +5288,13 @@ export async function setVesselsEnabled( clearSelectionAndInfo(); disableVessels(); if (!suppressStatus) { - showStatusMessage("船只已隐藏", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "船只", visible: false }), "info"); } return 0; } if (!suppressLoadingUi) { - setLoadingMessage("正在加载船只数据..."); + setLoadingMessage(earthMessage("loading.vesselData")); setLoading(true); hideError(); } @@ -4586,19 +5302,20 @@ export async function setVesselsEnabled( try { const vesselCount = await ensureVesselsEnabled(); if (!suppressStatus) { - showStatusMessage("船只已显示", "info"); + showStatusMessage(earthMessage("status.layerVisibility", { layer: "船只", visible: true }), "info"); } return vesselCount; } catch (error) { vesselsEnabled = false; clearVesselData(getEarth()); updateVesselToggleUi(false, 0); - const message = `船只加载失败: ${error?.message || String(error)}`; + const reason = error?.message || String(error); + const message = earthMessage("status.layerLoadFailed", { layer: "船只", error: reason }); void reportEarthClientLog({ level: "error", category: "layer-toggle", module: "vessels", - message, + message: `船只加载失败: ${reason}`, detail: error, }); if (!suppressLoadingUi) { @@ -4652,13 +5369,13 @@ function setupEventListeners() { // for the user-visible confirmation. return refreshComputeCentersAfterLocationSave() .then(() => { - showStatusMessage("算力中心坐标已保存", "success"); + showStatusMessage(earthMessage("status.computeCoordinatesSaved"), "success"); }) .catch((error) => { // Swallow refresh failure: the save itself succeeded, so we // must not surface this as a save failure. console.warn("后台校准算力中心图层失败:", error); - showStatusMessage("坐标已保存，地图稍后同步", "info"); + showStatusMessage(earthMessage("status.coordinatesSavedLater"), "info"); }); } // Optimistic marker is on screen; reconcile in the background. @@ -4669,10 +5386,10 @@ function setupEventListeners() { }) .catch((error) => { console.warn("即时生成算力中心交互物件失败，改用后台刷新:", error); - showStatusMessage("坐标已保存，正在同步地图...", "info"); + showStatusMessage(earthMessage("status.coordinatesSavedSyncing"), "info"); refreshComputeCentersAfterLocationSave() .then(() => { - showStatusMessage("算力中心坐标已保存", "success"); + showStatusMessage(earthMessage("status.computeCoordinatesSaved"), "success"); }) .catch((refreshError) => { console.warn("后台校准算力中心图层失败:", refreshError); @@ -4682,8 +5399,21 @@ function setupEventListeners() { const handleComputeCenterUnresolvedCountChange = (event) => { syncComputeCenterUnresolvedCount(event?.detail?.unresolvedCount ?? event?.detail); }; - const handleMotionDebugModeChange = () => { - setupMotionControl(); + const handleMotionDebugModeChange = (event) => { + const motionModeActive = getRotationMode() === ROTATION_MODE.MOTION && getAutoRotate(); + const nextEnabled = motionModeActive && Boolean(event?.detail?.enabled); + setMotionDebugPanelVisible(nextEnabled); + setMotionDebugPanelSkeletonOnly(Boolean(event?.detail?.skeletonOnly)); + motionControlAdapter?.sendCommand?.( + "set_debug_options", + { skeleton: nextEnabled }, + "earth-motion-debug-toggle", + ); + motionControlAdapter?.setEnabledGestures?.( + Array.isArray(event?.detail?.enabledGestures) + ? event.detail.enabledGestures + : getMotionEnabledGestures(), + ); }; const handleMotionDebugClose = () => { setMotionDebugEnabled(false); @@ -4869,6 +5599,7 @@ function onMouseMove(event) { : []; const vesselPick = getVesselHoverIntersections(); const vesselIntersects = vesselPick.intersects; + const earthInteractableIntersects = getEarthInteractablePointerIntersections(); let hoveredSat = null; let hoveredSatIndexFromIntersect = null; @@ -4883,15 +5614,21 @@ function onMouseMove(event) { bgpAnomalyIntersects, bgpCollectorIntersects, ); + const hoveredClusterHit = getPrimaryClusterHit( + bgpAnomalyIntersects, + bgpCollectorIntersects, + computeCenterIntersects, + earthInteractableIntersects, + ); if (hoveredBGP && !isSameBGPMarker(hoveredBGP, hoveredBGPMarker)) { clearTransientHoverState(); } - const hoveredComputeCenterMarker = - computeCenterIntersects.length > 0 ? computeCenterIntersects[0].object : null; + const hoveredComputeCenterMarker = getFirstObjectIntersection(computeCenterIntersects); const hoveredVesselMarker = vesselPick.checked && vesselIntersects.length > 0 ? vesselIntersects[0].object : null; + const hoveredEarthInteractableMarker = getFirstObjectIntersection(earthInteractableIntersects); const earthPoint = screenToEarthCoords( event.clientX, event.clientY, @@ -4934,6 +5671,12 @@ function onMouseMove(event) { ) { clearTransientHoverState(); } + if ( + hoveredEarthInteractable && + !isSameEarthInteractable(hoveredEarthInteractable, hoveredEarthInteractableMarker) + ) { + clearTransientHoverState(); + } if ( hoveredCable && @@ -4953,6 +5696,20 @@ function onMouseMove(event) { let objectTooltipShown = false; if ( + hoveredClusterHit && + !hoveredBGPMarker && + !hoveredComputeCenterMarker && + lockedObjectType !== "bgp" && + lockedObjectType !== "bgp_collector" && + lockedObjectType !== "compute_center" + ) { + showTooltip( + event.clientX + TOOLTIP_CURSOR_OFFSET, + event.clientY + TOOLTIP_CURSOR_OFFSET, + getClusterBriefHtml(hoveredClusterHit), + ); + objectTooltipShown = true; + } else if ( hoveredBGPMarker && getShowBGP() && lockedObjectType !== "bgp" && @@ -4990,6 +5747,17 @@ function onMouseMove(event) { getVesselBriefHtml(hoveredVesselMarker), ); objectTooltipShown = true; + } else if ( + hoveredEarthInteractableMarker && + lockedObjectType !== "earth_interactable" + ) { + applyEarthInteractableHoverState(hoveredEarthInteractableMarker); + showTooltip( + event.clientX + TOOLTIP_CURSOR_OFFSET, + event.clientY + TOOLTIP_CURSOR_OFFSET, + getEarthInteractableBriefHtml(hoveredEarthInteractableMarker), + ); + objectTooltipShown = true; } else if (cableIntersects.length > 0 && getShowCables()) { const cable = cableIntersects[0].object; hoveredCable = cable; @@ -5022,6 +5790,8 @@ function onMouseMove(event) { applyComputeCenterHoverState(lockedObject); } else if (lockedObjectType === "vessel" && lockedObject) { applyVesselHoverState(lockedObject); + } else if (lockedObjectType === "earth_interactable" && lockedObject) { + applyEarthInteractableHoverState(lockedObject); } else if ( !lockedObjectType && !isCruisePresentationPinned() && @@ -5033,6 +5803,7 @@ function onMouseMove(event) { if (vesselPick.checked) { resetTransientVesselStates(); } + resetTransientEarthInteractableStates(); hideInfoCard(); } @@ -5254,17 +6025,17 @@ function onClick(event) { const vesselIntersects = getShowVessels() ? getVesselPointerIntersections() : []; + const earthInteractableIntersects = getEarthInteractablePointerIntersections(); const satIntersects = getSatellitePointerIntersections(event); const clickedBGPMarker = getShowBGP() ? getPrimaryBGPClickTarget(event, bgpAnomalyIntersects, bgpCollectorIntersects) : null; - const clickedComputeCenterMarker = computeCenterIntersects.length > 0 - ? computeCenterIntersects[0].object - : null; + const clickedComputeCenterMarker = getFirstObjectIntersection(computeCenterIntersects); const clickedVesselMarker = vesselIntersects.length > 0 ? vesselIntersects[0].object : null; + const clickedEarthInteractableMarker = getFirstObjectIntersection(earthInteractableIntersects); if (clickedBGPMarker?.userData?.type === "bgp") { interruptCruisePresentation(); @@ -5285,7 +6056,11 @@ function onClick(event) { const incidentSummary = getBGPInfrastructureSummary(clickedMarker); showBGPInfo(clickedMarker, { x: event.clientX, y: event.clientY }); showStatusMessage( - `已选择BGP事件: ${clickedMarker.userData.collector} · ${incidentSummary.regionCount}个区域 / ${incidentSummary.cableCount}条相关海缆`, + earthMessage("status.bgpSelected", { + collector: clickedMarker.userData.collector, + regionCount: incidentSummary.regionCount, + cableCount: incidentSummary.cableCount, + }), "info", ); return; @@ -5309,7 +6084,7 @@ function onClick(event) { clickedMarker.userData.related_satellite_count = 0; showBGPCollectorInfo(clickedMarker, { x: event.clientX, y: event.clientY }); showStatusMessage( - `已选择观测站: ${clickedMarker.userData.collector}`, + earthMessage("status.selected", { target: "观测站", name: clickedMarker.userData.collector }), "info", ); return; @@ -5326,7 +6101,10 @@ function onClick(event) { setAutoRotate(false); showComputeCenterInfo(clickedMarker, { x: event.clientX, y: event.clientY }); showStatusMessage( - `已选择算力中心: ${clickedMarker.userData?.name || "未知节点"}`, + earthMessage("status.selected", { + target: "算力中心", + name: clickedMarker.userData?.name || "未知节点", + }), "info", ); return; @@ -5346,12 +6124,34 @@ function onClick(event) { console.warn("船只轨迹加载失败:", error); }); showStatusMessage( - `已选择船只: ${clickedMarker.userData?.name || clickedMarker.userData?.mmsi}`, + earthMessage("status.selected", { + target: "船只", + name: clickedMarker.userData?.name || clickedMarker.userData?.mmsi, + }), "info", ); return; } + if (clickedEarthInteractableMarker?.userData?.type === "earth_interactable") { + const clickedMarker = clickedEarthInteractableMarker; + if (confirmMotionCandidate({ + type: "earth_interactable", + object: clickedMarker, + screen: { x: event.clientX, y: event.clientY }, + distancePxSq: 0, + }, { showMotionStatus: false })) { + showStatusMessage( + earthMessage("status.selected", { + target: "交互点", + name: clickedMarker.userData?.label || clickedMarker.userData?.name || clickedMarker.userData?.id || "未知目标", + }), + "info", + ); + } + return; + } + if (cableIntersects.length > 0 && getShowCables()) { interruptCruisePresentation(); clearLockedObject(); @@ -5396,32 +6196,14 @@ function onClick(event) { const sat = selectSatellite(selectedIndex); if (!sat?.properties) return; - - interruptCruisePresentation(); - clearLockedObject(); - - lockedObject = sat; - lockedObjectType = "satellite"; - lockedSatellite = sat; - lockedSatelliteIndex = selectedIndex; - setLockedSatelliteIndex(selectedIndex); - showPredictedOrbit(sat); - setAutoRotate(false); - - const satPositions = getSatellitePositions(); - if (satPositions?.[selectedIndex]) { - setSatelliteRingState( - selectedIndex, - "locked", - satPositions[selectedIndex].current, - ); - if (hoveredSatelliteIndex === selectedIndex) { - setHoveredSatelliteIndex(selectedIndex); - } - } - - showSatelliteInfo(sat.properties, { x: event.clientX, y: event.clientY }); - showStatusMessage("已选择: " + sat.properties.name, "info"); + confirmMotionCandidate({ + type: "satellite", + index: selectedIndex, + object: sat, + screen: { x: event.clientX, y: event.clientY }, + distancePxSq: 0, + }, { showMotionStatus: false }); + showStatusMessage(earthMessage("status.selected", { name: sat.properties.name }), "info"); return; } @@ -5500,10 +6282,13 @@ function animate() { const activeCruiseMarker = isCruiseModeActive() && isCruisePresentationPinned() ? getCurrentCruiseBGPMarker() + : getRotationMode() === ROTATION_MODE.MOTION && motionSharedCruiseSequencer?.isPresentationPinned?.() + ? getCurrentMotionSharedCruiseBGPMarker() : null; updateBGPVisualState(lockedObjectType, lockedObject, camera, activeCruiseMarker); updateComputeCenterVisualState(lockedObjectType, lockedObject, camera); updateVesselVisualState(lockedObjectType, lockedObject, camera); + updateEarthInteractableVisualState(lockedObjectType, lockedObject, camera); if (lockedObjectType === "cable" && lockedObject) { applyLandingPointVisualState(lockedObject.userData.name, false, camera); @@ -5586,12 +6371,25 @@ export function destroy() { window.clearTimeout(earthUpdatesReconnectTimer); earthUpdatesReconnectTimer = null; } + if (earthDataReconcileTimer) { + window.clearInterval(earthDataReconcileTimer); + earthDataReconcileTimer = null; + } + if (earthUpdateFlushTimer) { + window.clearTimeout(earthUpdateFlushTimer); + earthUpdateFlushTimer = null; + } + pendingEarthUpdateLayers.clear(); + pendingEarthUpdateStrategies.clear(); + pendingEarthUpdateIsDatabaseChange = false; if (earthUpdatesSocket) { const socket = earthUpdatesSocket; earthUpdatesSocket = null; socket.close(); } motionControlAdapter?.stop?.(); + motionCruiseSequencer?.stop?.(); + motionSharedCruiseSequencer?.stop?.(); motionControlAdapter = null; presentationController?.dismiss?.("destroy"); while (cleanupFns.length) { diff --git a/frontend/public/earth/js/motion-agent-provider.js b/frontend/public/earth/js/motion-agent-provider.js index 9ca881db..4e8711bd 100644 --- a/frontend/public/earth/js/motion-agent-provider.js +++ b/frontend/public/earth/js/motion-agent-provider.js @@ -10,12 +10,15 @@ export function createMotionAgentProvider(options = {}) { onMessage = () => {}, onState = () => {}, onStatus = () => {}, + debugSkeleton = false, + enabledGestures = [], } = options; let socket = null; let reconnectTimer = null; let disposed = false; let connected = false; + let requestSeq = 0; function emitState(detail = {}) { onState({ @@ -55,6 +58,24 @@ export function createMotionAgentProvider(options = {}) { } } + function sendCommand(command, payload = {}, requestId = null) { + if (!socket || !connected || socket.readyState !== 1) { + return { + ok: false, + error: "motion_agent_not_connected", + requestId: requestId || null, + }; + } + const nextRequestId = requestId || `earth-motion-${Date.now()}-${++requestSeq}`; + socket.send(JSON.stringify({ + type: "command", + command, + request_id: nextRequestId, + payload, + })); + return { ok: true, requestId: nextRequestId }; + } + function connect() { if (disposed || socket) return; if (!WebSocketCtor) { @@ -73,6 +94,15 @@ export function createMotionAgentProvider(options = {}) { socket.onopen = () => { connected = true; emitState({ connected: true }); + sendCommand("set_debug_options", { skeleton: Boolean(debugSkeleton) }, "earth-motion-debug-on-connect"); + if (Array.isArray(enabledGestures) && enabledGestures.length > 0) { + sendCommand( + "set_enabled_gestures", + { gestures: enabledGestures }, + "earth-motion-enabled-gestures-on-connect", + ); + } + sendCommand("set_armed", { armed: true }, "earth-motion-armed-on-connect"); onStatus("动捕 Agent 已连接", "info"); }; socket.onmessage = (rawMessage) => onMessage(rawMessage?.data ?? rawMessage); @@ -104,5 +134,8 @@ export function createMotionAgentProvider(options = {}) { isConnected() { return connected; }, + sendCommand(command, payload = {}, requestId = null) { + return sendCommand(command, payload, requestId); + }, }; } diff --git a/frontend/public/earth/js/motion-control.js b/frontend/public/earth/js/motion-control.js index 67eedd6a..1d76e1de 100644 --- a/frontend/public/earth/js/motion-control.js +++ b/frontend/public/earth/js/motion-control.js @@ -5,6 +5,7 @@ import { import { createBrowserCameraProvider } from "./motion-browser-provider.js"; import { DEFAULT_MOTION_PROVIDER, + MOTION_GESTURES, MOTION_PROVIDER_AGENT, normalizeGestureMessage, normalizeMotionProvider, @@ -24,6 +25,7 @@ const DEFAULT_LAYER_COOLDOWN_MS = 1400; const DEFAULT_CONFIRM_COOLDOWN_MS = 1200; const ENABLED_STORAGE_KEY = "planet-earth-motion-control-enabled"; const URL_STORAGE_KEY = "planet-earth-motion-control-url"; +const DEFAULT_ENABLED_GESTURES = Array.from(MOTION_GESTURES); const GESTURE_POLICIES = { rotate_left: { group: "rotate_left", cooldownMs: DEFAULT_COOLDOWN_MS }, @@ -85,6 +87,8 @@ export function createMotionControlAdapter(options = {}) { minConfidence = DEFAULT_MIN_CONFIDENCE, cooldownMs = DEFAULT_COOLDOWN_MS, providerFactories = {}, + debugSkeleton = false, + enabledGestures = DEFAULT_ENABLED_GESTURES, WebSocketCtor = typeof WebSocket !== "undefined" ? WebSocket : null, onRotate = () => false, onZoom = () => false, @@ -102,6 +106,7 @@ export function createMotionControlAdapter(options = {}) { let activeProvider = null; let connected = false; let recognitionPaused = false; + let enabledGestureSet = normalizeEnabledGestureSet(enabledGestures); const lastHandledByGestureGroup = new Map(); function emitState(detail) { @@ -120,6 +125,7 @@ export function createMotionControlAdapter(options = {}) { } function shouldHandleGesture(event) { + if (!enabledGestureSet.has(event?.gesture)) return false; if (!event || event.confidence < minConfidence) return false; const policy = GESTURE_POLICIES[event.gesture] || { group: event.gesture, @@ -208,6 +214,8 @@ export function createMotionControlAdapter(options = {}) { ...sharedOptions, url, WebSocketCtor, + debugSkeleton, + enabledGestures: Array.from(enabledGestureSet), }); } return createBrowserCameraProvider(sharedOptions); @@ -242,6 +250,21 @@ export function createMotionControlAdapter(options = {}) { isConnected() { return Boolean(activeProvider?.isConnected?.()); }, + sendCommand(command, payload = {}, requestId = null) { + return activeProvider?.sendCommand?.(command, payload, requestId) || { + ok: false, + error: "motion_provider_commands_unavailable", + requestId, + }; + }, + setEnabledGestures(nextGestures) { + enabledGestureSet = normalizeEnabledGestureSet(nextGestures); + lastHandledByGestureGroup.clear(); + activeProvider?.sendCommand?.("set_enabled_gestures", { + gestures: Array.from(enabledGestureSet), + }, "earth-motion-enabled-gestures"); + return Array.from(enabledGestureSet); + }, getProvider() { return selectedProvider; }, @@ -253,6 +276,14 @@ export function createMotionControlAdapter(options = {}) { } } +function normalizeEnabledGestureSet(gestures) { + const values = Array.isArray(gestures) ? gestures : DEFAULT_ENABLED_GESTURES; + const normalized = values + .map((gesture) => String(gesture || "").trim()) + .filter((gesture) => MOTION_GESTURES.has(gesture)); + return new Set(normalized.length > 0 ? normalized : DEFAULT_ENABLED_GESTURES); +} + export { DEFAULT_AGENT_URL, DEFAULT_MOTION_PROVIDER, diff --git a/frontend/public/earth/js/motion-control.test.js b/frontend/public/earth/js/motion-control.test.js index 8714950c..658ca54b 100644 --- a/frontend/public/earth/js/motion-control.test.js +++ b/frontend/public/earth/js/motion-control.test.js @@ -201,6 +201,44 @@ describe("motion-control provider manager", () => { expect(debugEvent?.detail.confidence).toBe(0); }); + test("disabled gestures are ignored and can be re-enabled at runtime", () => { + installWindow(); + const rotations = []; + const sentCommands = []; + const adapter = createMotionControlAdapter({ + enabled: true, + cooldownMs: 0, + enabledGestures: ["zoom_in"], + onRotate: (...args) => rotations.push(args), + providerFactories: { + browser_camera: () => ({ + start() { + return true; + }, + stop() {}, + isConnected: () => true, + sendCommand(command, payload) { + sentCommands.push({ command, payload }); + return { ok: true }; + }, + }), + }, + }); + + adapter.start(); + adapter.handleMessage({ type: "gesture", gesture: "rotate_left", confidence: 0.91 }); + expect(rotations).toHaveLength(0); + + adapter.setEnabledGestures(["rotate_left"]); + adapter.handleMessage({ type: "gesture", gesture: "rotate_left", confidence: 0.91 }); + + expect(rotations).toHaveLength(1); + expect(sentCommands.at(-1)).toEqual({ + command: "set_enabled_gestures", + payload: { gestures: ["rotate_left"] }, + }); + }); + test("mock vertical gesture and focus gesture use dedicated callbacks", () => { installWindow(); const rotations = []; diff --git a/frontend/public/earth/js/motion-protocol.js b/frontend/public/earth/js/motion-protocol.js index 918005dd..698e49d9 100644 --- a/frontend/public/earth/js/motion-protocol.js +++ b/frontend/public/earth/js/motion-protocol.js @@ -47,6 +47,10 @@ export function normalizeGestureMessage(raw, fallbackSource = "motion-provider") seq: Number(raw.seq || 0), source: raw.source || fallbackSource, mode: raw.mode || "single", + protocolVersion: raw.protocol_version || raw.protocolVersion || "motion.v1", + cameraId: raw.camera_id || raw.cameraId || "unknown", + inputMode: raw.input_mode || raw.inputMode || raw.mode || "single", + fusion: raw.fusion && typeof raw.fusion === "object" ? raw.fusion : null, payload: raw.payload && typeof raw.payload === "object" ? raw.payload : {}, }; } diff --git a/frontend/public/earth/js/news-cruise-adapter.js b/frontend/public/earth/js/news-cruise-adapter.js index 2f39a8c7..6a63e926 100644 --- a/frontend/public/earth/js/news-cruise-adapter.js +++ b/frontend/public/earth/js/news-cruise-adapter.js @@ -3,22 +3,27 @@ import * as THREE from "three"; import { CONFIG, CONNECTOR_CONFIG, CRUISE_CONFIG } from "./constants.js"; import { showInfoCard, hideInfoCard } from "./info-card.js"; import { latLonToVector3 } from "./utils.js"; +import { formatLocaleDateTime, getEarthLocale, hasCjkText } from "./i18n.js"; import { createConnectorPath, resolveConnectorAnchor, } from "./callout-connector.js"; import { ensureNewsPanelReady, - getVisibleNewsItems, + getCruiseNewsItems, selectNewsItem, clearSelectedNewsItem, } from "./news.js"; import { getNewsDisplaySummary, getNewsDisplayTitle, + getNewsCategoryLabel, + getNewsFetchChannelLabel, getNewsFeedLabel, getNewsLocationSourceLabel, - getNewsRegionLabel, + getNewsRegionDisplayLabel, + getNewsSourceTypeLabel, + getNewsSourceNameLabel, } from "./news-locale.js"; const CRUISE_PRESENTATION_HIDE_MS = 220; @@ -35,10 +40,10 @@ function getItemTimestamp(item) { } function formatPublishedAt(rawValue) { - if (!rawValue) return "刚刚同步"; + if (!rawValue) return getEarthLocale() === "en-US" ? "Just synced" : "刚刚同步"; const parsed = new Date(rawValue); - if (Number.isNaN(parsed.getTime())) return "刚刚同步"; - return parsed.toLocaleString("zh-CN", { + if (Number.isNaN(parsed.getTime())) return getEarthLocale() === "en-US" ? "Just synced" : "刚刚同步"; + return formatLocaleDateTime(parsed, { hour12: false, month: "2-digit", day: "2-digit", @@ -89,23 +94,34 @@ function mapNewsItemToCruiseEvent(item) { return null; } + const rawFeedName = item.feed_name || ""; + const sourceType = item.source_type || (String(rawFeedName).startsWith("Global Monitor /") ? "aggregated" : "rss"); + const regionLabel = getNewsRegionDisplayLabel(item.region, item.display_region); + const locationLabel = getEarthLocale() === "en-US" && hasCjkText(item.location_label) + ? regionLabel + : item.location_label || regionLabel; + return { id: `news:${item.id}`, sourceId: item.id, type: "news", title: getNewsDisplayTitle(item), summary: getNewsDisplaySummary(item), - source: item.source || "", - feedName: getNewsFeedLabel(item.feed_name), + source: getNewsSourceNameLabel({ id: item.source_id, name: item.source || "" }), + feedName: getNewsFeedLabel(rawFeedName), + rawFeedName, + feedSourceTypeLabel: getNewsSourceTypeLabel(sourceType), + fetchChannelLabel: getNewsFetchChannelLabel(rawFeedName, sourceType), + categoryLabel: getNewsCategoryLabel(item.category), region: item.region || "global", - regionLabel: item.display_region || getNewsRegionLabel(item.region), + regionLabel, url: item.url || "", publishedAt: item.published_at || null, publishedAtDisplay: formatPublishedAt(item.published_at), latitude, longitude, - locationLabel: item.location_label || item.display_region || getNewsRegionLabel(item.region), - sourceLocationLabel: item.location_label || item.display_region || getNewsRegionLabel(item.region), + locationLabel, + sourceLocationLabel: locationLabel, targetLocationConfidence: item.location_meta?.target?.confidence ?? null, targetLocationSource: item.location_source || "", targetLocationSourceLabel: getNewsLocationSourceLabel(item.location_source), @@ -237,7 +253,7 @@ export function createNewsCruiseAdapter({ camera, earth, connector, focusView }) } function getSortedItems() { - return getVisibleNewsItems() + return getCruiseNewsItems() .map(mapNewsItemToCruiseEvent) .filter(Boolean); } @@ -275,6 +291,13 @@ export function createNewsCruiseAdapter({ camera, earth, connector, focusView }) showInfoCard("news", { title: item.title, summary: item.summary || item.title || "", + source: item.source, + feedName: item.rawFeedName || item.feedName, + feedSourceTypeLabel: item.feedSourceTypeLabel, + fetchChannelLabel: item.fetchChannelLabel, + categoryLabel: item.categoryLabel, + regionLabel: item.regionLabel, + publishedAtDisplay: item.publishedAtDisplay, }, { x: placement.x, y: placement.y, @@ -320,6 +343,13 @@ export function createNewsCruiseAdapter({ camera, earth, connector, focusView }) showInfoCard("news", { title: item.title, summary: item.summary || item.title || "", + source: item.source, + feedName: item.rawFeedName || item.feedName, + feedSourceTypeLabel: item.feedSourceTypeLabel, + fetchChannelLabel: item.fetchChannelLabel, + categoryLabel: item.categoryLabel, + regionLabel: item.regionLabel, + publishedAtDisplay: item.publishedAtDisplay, }, { x: placement.x, y: placement.y, diff --git a/frontend/public/earth/js/news-locale.js b/frontend/public/earth/js/news-locale.js index 33dabbcb..28a3edbb 100644 --- a/frontend/public/earth/js/news-locale.js +++ b/frontend/public/earth/js/news-locale.js @@ -1,117 +1,426 @@ -const DEFAULT_LOCALE = "zh-CN"; +import { getEarthLocale, hasCjkText, normalizeLocale } from "./i18n.js"; const REGION_LABELS = { - americas: "美洲", - europe: "欧洲", - "middle-east-africa": "中东与非洲", - "asia-pacific": "亚太", - global: "全球", + "zh-CN": { + americas: "美洲", + europe: "欧洲", + "middle-east-africa": "中东与非洲", + "asia-pacific": "亚太", + global: "全球", + }, + "en-US": { + americas: "Americas", + europe: "Europe", + "middle-east-africa": "Middle East & Africa", + "asia-pacific": "Asia Pacific", + global: "Global", + }, }; const FEED_LABELS = { - "Global Monitor / World": "全球监测", - "Global Monitor / Americas": "美洲监测", - "Global Monitor / Europe": "欧洲监测", - "Global Monitor / MEA": "中东与非洲监测", - "Global Monitor / APAC": "亚太监测", + "zh-CN": { + "Global Monitor / World": "区域监测", + "Global Monitor / Americas": "区域监测", + "Global Monitor / Europe": "区域监测", + "Global Monitor / MEA": "区域监测", + "Global Monitor / APAC": "区域监测", + }, + "en-US": { + "Global Monitor / World": "Regional Monitor", + "Global Monitor / Americas": "Regional Monitor", + "Global Monitor / Europe": "Regional Monitor", + "Global Monitor / MEA": "Regional Monitor", + "Global Monitor / APAC": "Regional Monitor", + }, +}; + +const CATEGORY_LABELS = { + "zh-CN": { + politics: "政治", + business: "商业", + ecommerce: "电商", + finance: "金融", + sports: "体育", + technology: "科技", + military: "军事", + disaster: "灾害", + energy: "能源", + society: "社会", + culture: "文化", + other: "其他", + }, + "en-US": { + politics: "Politics", + business: "Business", + ecommerce: "E-commerce", + finance: "Finance", + sports: "Sports", + technology: "Technology", + military: "Military", + disaster: "Disaster", + energy: "Energy", + society: "Society", + culture: "Culture", + other: "Other", + }, +}; + +const BREAKING_LEVEL_LABELS = { + "zh-CN": { + watch: "关注", + breaking: "突发", + critical: "严重突发", + }, + "en-US": { + watch: "Watch", + breaking: "Breaking", + critical: "Critical", + }, +}; + +const BREAKING_SCOPE_LABELS = { + "zh-CN": { + regional: "区域", + global: "全球", + }, + "en-US": { + regional: "Regional", + global: "Global", + }, +}; + +const SOURCE_TYPE_LABELS = { + "zh-CN": { + rss: "RSS", + atom: "Atom", + aggregated: "Aggregated", + manual: "手动添加", + reference: "Reference", + }, + "en-US": { + rss: "RSS", + atom: "Atom", + aggregated: "Aggregated", + manual: "Manual", + reference: "Reference", + }, }; const LOCATION_SOURCE_LABELS = { - region_anchor: "区域锚点", - ai_inferred_target: "AI 推断位置", - headline_location_hint: "标题位置线索", - headline_country_hint: "标题国家线索", + "zh-CN": { + region_anchor: "区域锚点", + ai_inferred_target: "AI 推断位置", + headline_location_hint: "标题位置线索", + headline_country_hint: "标题国家线索", + }, + "en-US": { + region_anchor: "Region Anchor", + ai_inferred_target: "AI-inferred Location", + headline_location_hint: "Headline Location Hint", + headline_country_hint: "Headline Country Hint", + }, }; const ENRICHMENT_STATUS_LABELS = { - pending: "待增强", - queued: "增强排队中", - attempted: "增强中", - success: "已汉化", - content_only: "已汉化", - location_only: "位置已增强", - unavailable: "AI 未配置", - provider_error: "增强失败", - parse_error: "增强解析失败", - no_result: "暂无增强结果", + "zh-CN": { + pending: "待增强", + queued: "增强排队中", + attempted: "增强中", + success: "已汉化", + content_only: "已汉化", + location_only: "位置已增强", + unavailable: "AI 未配置", + provider_error: "增强失败", + parse_error: "增强解析失败", + no_result: "暂无增强结果", + }, + "en-US": { + pending: "Pending", + queued: "Queued", + attempted: "Enhancing", + success: "Localized", + content_only: "Localized", + location_only: "Location Enhanced", + unavailable: "AI Unconfigured", + provider_error: "Enhancement Failed", + parse_error: "Parse Failed", + no_result: "No Enhancement", + }, +}; + +const SOURCE_NAME_LABELS = { + "en-US": { + "36氪": "36Kr", + "亿邦动力": "Ebrun", + "商务数据中心": "MOFCOM Data Center", + "商务部电商动态": "MOFCOM E-Commerce", + "国家统计局数据发布": "National Bureau of Statistics", + "电商物流指数": "China E-Commerce Logistics Index", + }, +}; + +const SOURCE_ID_LABELS = { + "en-US": { + "36kr": "36Kr", + ebrun: "Ebrun", + "mofcom-data": "MOFCOM Data Center", + "mofcom-ecommerce": "MOFCOM E-Commerce", + "stats-china-online-retail": "National Bureau of Statistics", + "china-ecommerce-logistics-index": "China E-Commerce Logistics Index", + }, +}; + +const FEED_NAME_LABELS = { + "en-US": { + "综合资讯": "General", + "文章资讯": "Articles", + "最新快讯": "Newsflash", + "动态内容": "Updates", + "零售": "Retail", + "服务": "Services", + "数据": "Data", + "政策": "Policy", + "数据发布": "Data Releases", + }, }; const TITLE_PLACEHOLDERS = { - queued: "新闻汉化排队中", - attempted: "新闻汉化中", - provider_error: "新闻汉化失败，正在重试", - parse_error: "新闻解析失败，正在重试", - unavailable: "等待 AI 配置", - no_result: "新闻汉化待重试", - location_only: "新闻汉化待重试", + "zh-CN": { + queued: "新闻汉化排队中", + attempted: "新闻汉化中", + provider_error: "新闻汉化失败，正在重试", + parse_error: "新闻解析失败，正在重试", + unavailable: "等待 AI 配置", + no_result: "新闻汉化待重试", + location_only: "新闻汉化待重试", + }, + "en-US": { + queued: "English translation queued", + attempted: "English translation in progress", + provider_error: "English translation retrying", + parse_error: "English translation retrying", + unavailable: "Waiting for AI configuration", + no_result: "English translation pending", + location_only: "English translation pending", + }, }; const SUMMARY_PLACEHOLDERS = { - queued: "中文概要正在生成，请稍后刷新。", - attempted: "中文概要正在生成，请稍后刷新。", - provider_error: "中文概要生成失败，系统会重新提交增强任务。", - parse_error: "中文概要解析失败，系统会重新提交增强任务。", - unavailable: "AI 服务配置完成后将生成中文概要。", - no_result: "中文概要暂未生成，系统会继续重试。", - location_only: "已完成位置增强，中文概要将继续重试。", + "zh-CN": { + queued: "中文概要正在生成，请稍后刷新。", + attempted: "中文概要正在生成，请稍后刷新。", + provider_error: "中文概要生成失败，系统会重新提交增强任务。", + parse_error: "中文概要解析失败，系统会重新提交增强任务。", + unavailable: "AI 服务配置完成后将生成中文概要。", + no_result: "中文概要暂未生成，系统会继续重试。", + location_only: "已完成位置增强，中文概要将继续重试。", + }, + "en-US": { + queued: "English summary is being generated. Refresh shortly.", + attempted: "English summary is being generated. Refresh shortly.", + provider_error: "English summary generation failed and will be retried.", + parse_error: "English summary parsing failed and will be retried.", + unavailable: "English summary will be generated after AI is configured.", + no_result: "English summary is pending and will be retried.", + location_only: "Location is ready; English summary is still pending.", + }, }; function normalizeText(value) { return String(value ?? "").replace(/\s+/g, " ").trim(); } -function getLocalization(item, locale = DEFAULT_LOCALE) { +function getLocale(locale = getEarthLocale()) { + return normalizeLocale(locale); +} + +function getLabels(table, locale = getEarthLocale()) { + const normalizedLocale = getLocale(locale); + return table[normalizedLocale] || table["zh-CN"] || {}; +} + +function isEnglishLocale(locale = getEarthLocale()) { + return getLocale(locale) === "en-US"; +} + +function isEnglishContent(item) { + const language = String(item?.content_language || "").toLowerCase(); + return language === "en" || language.startsWith("en-") || language.startsWith("en_"); +} + +function getPlaceholder(table, status, locale = getEarthLocale(), fallbackKey = "no_result") { + const labels = getLabels(table, locale); + return labels[status] || labels[fallbackKey] || ""; +} + +function safeEnglishText(value) { + const text = normalizeText(value); + return text && !hasCjkText(text) ? text : ""; +} + +function sourceIdFallback(id) { + const value = normalizeText(id); + if (!value) return ""; + return value + .replace(/[-_]+/g, " ") + .replace(/\b\w/g, (letter) => letter.toUpperCase()); +} + +function getLocalization(item, locale = getEarthLocale()) { const localizations = item?.localizations; const localized = localizations && typeof localizations === "object" - ? localizations[locale] + ? localizations[getLocale(locale)] : null; return localized && typeof localized === "object" ? localized : {}; } -export function getNewsDisplayTitle(item, locale = DEFAULT_LOCALE) { +export function getNewsDisplayTitle(item, locale = getEarthLocale()) { + const normalizedLocale = getLocale(locale); const localized = getLocalization(item, locale).title; - if (localized || item?.display_title) { + if (normalizedLocale === "zh-CN" && (item?.display_title || localized)) { return normalizeText(item?.display_title || localized); } - return normalizeText( - TITLE_PLACEHOLDERS[item?.enrichment_status] - || "新闻汉化中", - ); + if (normalizedLocale === "en-US") { + return safeEnglishText(localized) + || safeEnglishText(item?.display_title) + || (isEnglishContent(item) ? safeEnglishText(item?.title) : "") + || getPlaceholder(TITLE_PLACEHOLDERS, item?.enrichment_status, locale); + } + if (localized) { + return normalizeText(localized); + } + if (item?.title) { + return normalizeText(item.title); + } + return normalizeText(getPlaceholder(TITLE_PLACEHOLDERS, item?.enrichment_status, locale, "attempted")); } -export function getNewsDisplaySummary(item, locale = DEFAULT_LOCALE) { +export function getNewsDisplaySummary(item, locale = getEarthLocale()) { + const normalizedLocale = getLocale(locale); const localized = getLocalization(item, locale).summary; - if (localized || item?.display_summary) { + if (normalizedLocale === "zh-CN" && (item?.display_summary || localized)) { return normalizeText(item?.display_summary || localized); } - return normalizeText( - SUMMARY_PLACEHOLDERS[item?.enrichment_status] - || "中文概要生成中，请稍后刷新。", - ); + if (normalizedLocale === "en-US") { + return safeEnglishText(localized) + || safeEnglishText(item?.display_summary) + || (isEnglishContent(item) ? safeEnglishText(item?.summary) : "") + || getPlaceholder(SUMMARY_PLACEHOLDERS, item?.enrichment_status, locale); + } + if (localized) { + return normalizeText(localized); + } + if (item?.summary) { + return normalizeText(item.summary); + } + return normalizeText(getPlaceholder(SUMMARY_PLACEHOLDERS, item?.enrichment_status, locale, "attempted")); } -export function isNewsContentReady(item, locale = DEFAULT_LOCALE) { +export function isNewsContentReady(item, locale = getEarthLocale()) { const localized = getLocalization(item, locale); - const title = normalizeText(item?.display_title || localized.title); - const summary = normalizeText(item?.display_summary || localized.summary); - return ( - (item?.enrichment_status === "success" || item?.enrichment_status === "content_only") - && Boolean(title && summary) + const normalizedLocale = getLocale(locale); + if (normalizedLocale === "en-US") { + const title = safeEnglishText(localized.title) + || safeEnglishText(item?.display_title) + || (isEnglishContent(item) ? safeEnglishText(item?.title) : ""); + const summary = safeEnglishText(localized.summary) + || safeEnglishText(item?.display_summary) + || (isEnglishContent(item) ? safeEnglishText(item?.summary) : ""); + return Boolean(title && summary); + } + const title = normalizeText( + item?.display_title || localized.title || item?.title, + ); + const summary = normalizeText( + item?.display_summary || localized.summary || item?.summary, + ); + return Boolean(title && summary); +} + +export function getNewsRegionLabel(region, fallback = "", locale = getEarthLocale()) { + const labels = getLabels(REGION_LABELS, locale); + return labels[region] || fallback || region || labels.global; +} + +export function getNewsFeedLabel(feedName, locale = getEarthLocale()) { + const normalizedLocale = getLocale(locale); + const mappedFeed = getLabels(FEED_NAME_LABELS, locale)[feedName]; + if (mappedFeed) return mappedFeed; + if (normalizedLocale === "en-US" && hasCjkText(feedName)) return "News Feed"; + return getLabels(FEED_LABELS, locale)[feedName] + || feedName + || (normalizedLocale === "en-US" ? "Aggregated Source" : "聚合源"); +} + +export function getNewsCategoryLabel(category, locale = getEarthLocale()) { + const labels = getLabels(CATEGORY_LABELS, locale); + return labels[category] || category || labels.other; +} + +export function getNewsBreakingLabel(level, scope = "regional", locale = getEarthLocale()) { + const normalizedLevel = normalizeText(level).toLowerCase(); + if (!normalizedLevel || normalizedLevel === "none") return ""; + const levelLabels = getLabels(BREAKING_LEVEL_LABELS, locale); + const scopeLabels = getLabels(BREAKING_SCOPE_LABELS, locale); + const levelLabel = levelLabels[normalizedLevel] || level; + const scopeLabel = scopeLabels[normalizeText(scope).toLowerCase()] || scopeLabels.regional; + return getLocale(locale) === "en-US" ? `${scopeLabel} ${levelLabel}` : `${scopeLabel}${levelLabel}`; +} + +export function getNewsSourceTypeLabel(sourceType, locale = getEarthLocale()) { + const normalized = normalizeText(sourceType).toLowerCase(); + return getLabels(SOURCE_TYPE_LABELS, locale)[normalized] || sourceType || "RSS"; +} + +export function getNewsSourceNameLabel(sourceOrName, locale = getEarthLocale()) { + const normalizedLocale = getLocale(locale); + const source = sourceOrName && typeof sourceOrName === "object" ? sourceOrName : null; + const name = normalizeText(source ? source.name : sourceOrName); + if (normalizedLocale !== "en-US") return name; + const id = normalizeText(source?.id || source?.source_id); + return getLabels(SOURCE_ID_LABELS, locale)[id] + || getLabels(SOURCE_NAME_LABELS, locale)[name] + || safeEnglishText(name) + || sourceIdFallback(id) + || "News Source"; +} + +export function getNewsRegionDisplayLabel(region, fallback = "", locale = getEarthLocale()) { + return getNewsRegionLabel(region, isEnglishLocale(locale) ? "" : fallback, locale); +} + +export function isRegionalMonitorFeed(feedName) { + return Object.values(FEED_LABELS).some((labels) => + Object.prototype.hasOwnProperty.call(labels, feedName), ); } -export function getNewsRegionLabel(region, fallback = "") { - return REGION_LABELS[region] || fallback || region || REGION_LABELS.global; +export function getNewsFetchChannelLabel(feedName, sourceType = "") { + const normalized = normalizeText(sourceType).toLowerCase(); + const english = getEarthLocale() === "en-US"; + if (isRegionalMonitorFeed(feedName) || normalized === "aggregated") return english ? "Regional Monitor" : "区域监测"; + if (normalized === "manual") return english ? "Manual" : "手动添加"; + if (normalized === "atom") return english ? "Single-source Atom" : "单源 Atom"; + if (normalized === "reference") return english ? "Reserved" : "配置保留"; + return english ? "Single-source RSS" : "单源 RSS"; } -export function getNewsFeedLabel(feedName) { - return FEED_LABELS[feedName] || feedName || "聚合源"; +export function getNewsLocationSourceLabel(source, locale = getEarthLocale()) { + return getLabels(LOCATION_SOURCE_LABELS, locale)[source] + || source + || (getLocale(locale) === "en-US" ? "Location Source" : "位置来源"); } -export function getNewsLocationSourceLabel(source) { - return LOCATION_SOURCE_LABELS[source] || source || "位置来源"; -} - -export function getNewsEnrichmentStatusLabel(status) { - return ENRICHMENT_STATUS_LABELS[status] || status || "增强状态"; +export function getNewsEnrichmentStatusLabel(statusOrItem) { + const item = statusOrItem && typeof statusOrItem === "object" ? statusOrItem : null; + const status = item ? item.enrichment_status : statusOrItem; + const language = String(item?.content_language || "").toLowerCase(); + if (language.startsWith("zh") && status !== "success" && status !== "content_only") { + if (status === "queued" || status === "attempted") return getEarthLocale() === "en-US" ? "English Translation Pending" : "英文补译中"; + if (status === "provider_error" || status === "parse_error" || status === "no_result") return getEarthLocale() === "en-US" ? "Chinese Original" : "中文原文"; + return getEarthLocale() === "en-US" ? "Chinese Original" : "中文原文"; + } + return getLabels(ENRICHMENT_STATUS_LABELS)[status] + || status + || (getEarthLocale() === "en-US" ? "Enhancement Status" : "增强状态"); } diff --git a/frontend/public/earth/js/news.js b/frontend/public/earth/js/news.js index d7ffa88a..0030017e 100644 --- a/frontend/public/earth/js/news.js +++ b/frontend/public/earth/js/news.js @@ -1,12 +1,32 @@ import { showStatusMessage } from "./ui.js"; +import { + applyEarthI18n, + earthMessage, + getEarthLocale, + onEarthLocaleChange, + translateText, +} from "./i18n.js"; import { getNewsDisplaySummary, getNewsDisplayTitle, + getNewsCategoryLabel, + getNewsBreakingLabel, getNewsEnrichmentStatusLabel, + getNewsFetchChannelLabel, getNewsFeedLabel, getNewsRegionLabel, + getNewsRegionDisplayLabel, + getNewsSourceTypeLabel, + getNewsSourceNameLabel, isNewsContentReady, } from "./news-locale.js"; +import { + canAttemptEarthRealtime, + getEarthRealtimeCooldownMs, + getEarthRealtimeUrl, + recordEarthRealtimeFailure, + recordEarthRealtimeOpen, +} from "./realtime.js"; // Desktop news has two surfaces: // - a persistent top ticker @@ -23,9 +43,14 @@ const NEWS_HUD_MIN_WIDTH_PX = 420; const NEWS_HUD_MIN_HEIGHT_PX = 360; const NEWS_HUD_RESIZE_MARGIN_PX = 12; const NEWS_REALTIME_RECONNECT_MS = 5000; +const NEWS_SUMMARY_LIMIT = 12; +const NEWS_FULL_LIMIT = 50; +const NEWS_SOURCE_FILTER_STORAGE_KEY = "planet.earth.newsSourceFilters.v1"; let initialized = false; let refreshPromise = null; +let refreshRequestKey = ""; +let activeRefreshToken = 0; let payload = null; let lastFocus = null; let lastFetchAt = 0; @@ -34,6 +59,55 @@ let selectedCruiseStoryId = null; let morphTimer = null; let newsRealtimeSocket = null; let newsRealtimeReconnectTimer = null; +let activeNewsCategoryFilters = null; +let lastCategorySignature = ""; +let activeNewsSourceFilters = loadNewsSourceFilters(); +let lastSourceSignature = ""; +let newsFullListMode = false; +let activeFilterPopover = null; + +const NEWS_CATEGORY_ALIASES = { + politics: ["politics", "political", "policy", "政府", "政治", "政策", "政务"], + business: ["business", "economy", "economic", "commerce", "商业", "经济", "产业", "企业"], + ecommerce: ["ecommerce", "e-commerce", "online_retail", "retail_online", "电商", "电子商务", "网上零售", "跨境电商"], + finance: ["finance", "financial", "market", "stock", "金融", "财经", "市场", "证券"], + sports: ["sports", "sport", "体育"], + technology: ["technology", "tech", "science", "科技", "科学", "技术", "ai", "人工智能"], + military: ["military", "defense", "war", "军事", "防务", "战争"], + disaster: ["disaster", "emergency", "earthquake", "flood", "storm", "灾害", "灾难", "应急", "地震", "洪水"], + energy: ["energy", "oil", "gas", "power", "能源", "石油", "天然气", "电力"], + society: ["society", "social", "社会", "民生"], + culture: ["culture", "arts", "entertainment", "文化", "艺术", "娱乐"], + other: ["other", "general", "misc", "其他", "综合"], +}; +const NEWS_CATEGORY_KEYS = Object.keys(NEWS_CATEGORY_ALIASES); + +function loadNewsSourceFilters() { + try { + const raw = window.localStorage?.getItem(NEWS_SOURCE_FILTER_STORAGE_KEY); + if (!raw) return null; + const parsed = JSON.parse(raw); + if (!Array.isArray(parsed?.enabled)) return null; + return parsed.enabled.map((id) => String(id || "").trim()).filter(Boolean); + } catch { + return null; + } +} + +function persistNewsSourceFilters(enabledIds) { + try { + if (!Array.isArray(enabledIds)) { + window.localStorage?.removeItem(NEWS_SOURCE_FILTER_STORAGE_KEY); + return; + } + window.localStorage?.setItem( + NEWS_SOURCE_FILTER_STORAGE_KEY, + JSON.stringify({ enabled: enabledIds }), + ); + } catch { + // Ignore localStorage failures; source filters are display-only preferences. + } +} function getElements() { const isMobile = document.body.classList.contains("layout-mode-mobile"); @@ -53,6 +127,9 @@ function getElements() { tickerTrack: document.getElementById("news-ticker-track"), hud: document.getElementById("news-hud-panel"), hudCloseBtn: document.getElementById("news-hud-close"), + filterPopovers: document.querySelectorAll("[data-news-filter-popover]"), + filterToggles: document.querySelectorAll("[data-news-filter-toggle]"), + viewAllToggles: document.querySelectorAll("#news-view-all-toggle, #mobile-news-view-all-toggle"), }; } @@ -62,17 +139,18 @@ function formatCoord(value, positiveLabel, negativeLabel) { } function formatRelativeTime(raw) { - if (!raw) return "刚刚同步"; + const english = getEarthLocale() === "en-US"; + if (!raw) return english ? "Just synced" : "刚刚同步"; const date = new Date(raw); - if (Number.isNaN(date.getTime())) return "刚刚同步"; + if (Number.isNaN(date.getTime())) return english ? "Just synced" : "刚刚同步"; const diff = Date.now() - date.getTime(); const minutes = Math.max(1, Math.round(diff / 60000)); - if (minutes < 60) return `${minutes} 分钟前`; + if (minutes < 60) return english ? `${minutes}m ago` : `${minutes} 分钟前`; const hours = Math.round(minutes / 60); - if (hours < 24) return `${hours} 小时前`; + if (hours < 24) return english ? `${hours}h ago` : `${hours} 小时前`; const days = Math.round(hours / 24); - return `${days} 天前`; + return english ? `${days}d ago` : `${days} 天前`; } export function updateNewsToggleUI(visible) { @@ -196,6 +274,11 @@ function closeNewsHud() { setNewsHudOpen(false); } +function isNewsHudOpen() { + const { hud } = getElements(); + return hud instanceof HTMLElement && !hud.classList.contains("hud-panel-hidden"); +} + function setupNewsHudResize() { const { hud } = getElements(); const container = document.getElementById("container"); @@ -298,7 +381,348 @@ function escapeNewsHtml(value) { } function getDisplayableNewsItems(items) { - return Array.isArray(items) ? items.filter(isNewsContentReady) : []; + return Array.isArray(items) + ? items.filter((item) => isNewsContentReady(item)) + : []; +} + +function normalizeBreakingLevel(level) { + const normalized = String(level ?? "").trim().toLowerCase(); + return ["watch", "breaking", "critical"].includes(normalized) ? normalized : "none"; +} + +function normalizeBreakingScope(scope) { + const normalized = String(scope ?? "").trim().toLowerCase(); + return normalized === "global" ? "global" : "regional"; +} + +function isBreakingActive(item) { + const level = normalizeBreakingLevel(item?.breaking_level); + if (level === "none") return false; + const expiresAt = item?.breaking_expires_at ? new Date(item.breaking_expires_at) : null; + return !expiresAt || Number.isNaN(expiresAt.getTime()) || expiresAt.getTime() > Date.now(); +} + +function getHighestBreakingLevel(items) { + const ranks = { none: 0, watch: 1, breaking: 2, critical: 3 }; + return (Array.isArray(items) ? items : []).reduce((highest, item) => { + if (!isBreakingActive(item)) return highest; + const level = normalizeBreakingLevel(item?.breaking_level); + return ranks[level] > ranks[highest] ? level : highest; + }, "none"); +} + +function applyNewsBreakingShellState(nextPayload) { + const filtersLevel = normalizeBreakingLevel(nextPayload?.filters?.highest_breaking_level); + const computedLevel = getHighestBreakingLevel([ + ...(Array.isArray(nextPayload?.items) ? nextPayload.items : []), + ...(Array.isArray(nextPayload?.cruise_items) ? nextPayload.cruise_items : []), + ]); + const level = filtersLevel !== "none" ? filtersLevel : computedLevel; + document + .querySelectorAll(".earth-news-hud, .earth-news-ticker, .earth-mobile-news-board") + .forEach((element) => { + if (!(element instanceof HTMLElement)) return; + element.classList.remove( + "has-breaking-watch", + "has-breaking-breaking", + "has-breaking-critical", + ); + if (level !== "none") { + element.classList.add(`has-breaking-${level}`); + } + }); +} + +function normalizeNewsSourceType(value) { + const normalized = String(value ?? "").trim().toLowerCase(); + return normalized || ""; +} + +function getNewsSourceDescriptor(item, sourcesByName, sourcesById) { + const feedName = String(item?.feed_name || "").trim(); + const rawSourceName = String(item?.source || feedName || "NEWS").trim(); + const sourceId = String(item?.source_id || "").trim(); + const sourceConfig = sourcesById.get(sourceId) || sourcesByName.get(feedName) || null; + const sourceType = normalizeNewsSourceType(item?.source_type || sourceConfig?.source_type) + || (feedName.startsWith("Global Monitor /") ? "aggregated" : "rss"); + const sourceTypeLabel = getNewsSourceTypeLabel(sourceType); + const channelLabel = getNewsFetchChannelLabel(feedName, sourceType); + const sourceName = getNewsSourceNameLabel( + sourceConfig || { id: sourceId, name: rawSourceName }, + ); + const feedLabel = getNewsFetchChannelLabel(feedName, sourceType); + const sourceGroupName = sourceConfig ? getNewsSourceNameLabel(sourceConfig) : ""; + const originLabel = sourceGroupName + ? `${sourceGroupName} · ${channelLabel}` + : feedName && feedName !== sourceName + ? `${feedLabel} · ${sourceTypeLabel}` + : `${channelLabel} · ${sourceTypeLabel}`; + const english = getEarthLocale() === "en-US"; + const tooltip = [ + `${english ? "Media source" : "媒体来源"}: ${sourceName}`, + sourceGroupName ? `${english ? "Source group" : "来源组"}: ${sourceGroupName}` : "", + feedName ? `${english ? "RSS feed" : "RSS 来源"}: ${getNewsFeedNameForTooltip(feedName)}` : "", + `${english ? "Source type" : "源类型"}: ${sourceTypeLabel}`, + `${english ? "Fetch channel" : "抓取通道"}: ${channelLabel}`, + ].filter(Boolean).join("\n"); + return { + sourceName, + feedName, + sourceType, + sourceTypeLabel, + channelLabel, + originLabel, + tooltip, + }; +} + +function getNewsFeedNameForTooltip(feedName) { + return getEarthLocale() === "en-US" + ? getNewsFeedLabel(feedName) + : feedName; +} + +function getFocusRegionText(focus = {}) { + return getNewsRegionDisplayLabel(focus.region, focus.display_region); +} + +function getFocusLabelText(focus = {}) { + if (getEarthLocale() === "en-US") { + return getNewsRegionDisplayLabel(focus.region, focus.label); + } + return focus.label || "全球焦点"; +} + +function formatNewsSourceCount(enabledCount, totalCount) { + if (getEarthLocale() === "en-US") return `${enabledCount || totalCount} / ${totalCount} sources`; + return `${enabledCount || totalCount} / ${totalCount} 路来源`; +} + +function formatNewsStatus({ displayCount, totalCount, stale, hasErrors }) { + if (getEarthLocale() === "en-US") { + if (displayCount !== totalCount) return `Showing ${displayCount} / ${totalCount} stories`; + if (stale) return `Showing the latest available news cache, ${totalCount} stories`; + return hasErrors + ? `Aggregated ${totalCount} stories; some sources are unavailable` + : `Aggregated ${totalCount} situation stories`; + } + if (displayCount !== totalCount) return `展示 ${displayCount} / ${totalCount} 条态势新闻`; + if (stale) return `当前显示最近一次可用新闻缓存，共 ${totalCount} 条`; + return hasErrors + ? `已聚合 ${totalCount} 条，部分源不可用` + : `已聚合 ${totalCount} 条态势新闻`; +} + +function getEnabledNewsCategoryKeys(filters = activeNewsCategoryFilters) { + if (!filters || typeof filters !== "object") return [...NEWS_CATEGORY_KEYS].sort(); + return Object.entries(filters) + .filter(([, enabled]) => enabled !== false) + .map(([key]) => key) + .filter((key) => Object.prototype.hasOwnProperty.call(NEWS_CATEGORY_ALIASES, key)) + .sort(); +} + +function getNewsCategorySignature(filters = activeNewsCategoryFilters) { + const enabled = getEnabledNewsCategoryKeys(filters); + const total = NEWS_CATEGORY_KEYS.length; + if (enabled.length === 0) return "__none__"; + if (enabled.length === total) return ""; + return enabled.join(","); +} + +function getAvailableSourceIds(nextPayload = payload) { + return (Array.isArray(nextPayload?.sources) ? nextPayload.sources : []) + .map((source) => String(source?.id || "").trim()) + .filter(Boolean) + .sort(); +} + +function getEnabledNewsSourceIds(nextPayload = payload) { + const available = getAvailableSourceIds(nextPayload); + if (!available.length) return []; + if (!Array.isArray(activeNewsSourceFilters)) return available; + const allowed = new Set(activeNewsSourceFilters); + const enabled = available.filter((id) => allowed.has(id)); + return enabled.length > 0 ? enabled : available; +} + +function reconcileNewsSourceFilters(nextPayload = payload) { + const available = getAvailableSourceIds(nextPayload); + if (!available.length || !Array.isArray(activeNewsSourceFilters)) return; + + const valid = activeNewsSourceFilters.filter((id) => available.includes(id)); + const changed = valid.length !== activeNewsSourceFilters.length; + if (activeNewsSourceFilters.length > 0 && valid.length === 0) { + activeNewsSourceFilters = null; + persistNewsSourceFilters(null); + return; + } + if (valid.length === available.length) { + activeNewsSourceFilters = null; + persistNewsSourceFilters(null); + return; + } + if (changed) { + activeNewsSourceFilters = valid; + persistNewsSourceFilters(valid); + } +} + +function getNewsSourceSignature(nextPayload = payload) { + reconcileNewsSourceFilters(nextPayload); + const available = getAvailableSourceIds(nextPayload); + const enabled = getEnabledNewsSourceIds(nextPayload); + if (available.length > 0 && enabled.length === 0) return "__none__"; + if (enabled.length === available.length) return ""; + return enabled.join(","); +} + +function getNewsSourceSignatureForFetch(lat, lon) { + const currentRegion = payload?.focus?.region || null; + const nextRegion = inferRegion(lat, lon); + if (currentRegion && nextRegion !== currentRegion) { + return ""; + } + return getNewsSourceSignature(); +} + +function getNewsLimit() { + return newsFullListMode ? NEWS_FULL_LIMIT : NEWS_SUMMARY_LIMIT; +} + +function setNewsSourceFilters(enabledIds, { persist = true } = {}) { + const available = getAvailableSourceIds(); + const next = Array.isArray(enabledIds) + ? enabledIds.map((id) => String(id || "").trim()).filter((id) => available.includes(id)) + : null; + activeNewsSourceFilters = next && next.length === available.length ? null : next; + if (persist) persistNewsSourceFilters(activeNewsSourceFilters); +} + +function summarizeSelection(enabledCount, totalCount) { + if (totalCount <= 0) return translateText("暂无"); + if (enabledCount <= 0) return translateText("未选"); + if (enabledCount === totalCount) return translateText("全部"); + return translateText(`${enabledCount} 项`); +} + +function syncFilterSummaries(nextPayload = payload) { + const categories = getEnabledNewsCategoryKeys(); + const totalCategories = NEWS_CATEGORY_KEYS.length; + const sources = getAvailableSourceIds(nextPayload); + const enabledSources = getEnabledNewsSourceIds(nextPayload); + document.querySelectorAll('[data-news-filter-summary="category"]').forEach((el) => { + el.textContent = summarizeSelection(categories.length, totalCategories); + }); + document.querySelectorAll('[data-news-filter-summary="source"]').forEach((el) => { + el.textContent = summarizeSelection(enabledSources.length, sources.length); + }); + document.querySelectorAll('[data-news-filter-summary="limit"]').forEach((el) => { + const total = Array.isArray(nextPayload?.items) ? nextPayload.items.length : 0; + el.textContent = newsFullListMode + ? translateText("全部") + : translateText(`${Math.min(NEWS_SUMMARY_LIMIT, total || NEWS_SUMMARY_LIMIT)} 条`); + }); + document.querySelectorAll("[data-news-view-mode-label]").forEach((el) => { + el.textContent = translateText(newsFullListMode ? "返回摘要" : "查看全部"); + }); +} + +function closeNewsFilterPopover() { + activeFilterPopover = null; + document.querySelectorAll("[data-news-filter-popover]").forEach((popover) => { + if (popover instanceof HTMLElement) popover.hidden = true; + }); + document.querySelectorAll("[data-news-filter-toggle]").forEach((toggle) => { + if (toggle instanceof HTMLElement) toggle.setAttribute("aria-expanded", "false"); + }); +} + +function renderCategoryFilterChips() { + const enabled = new Set(getEnabledNewsCategoryKeys()); + return NEWS_CATEGORY_KEYS + .map((key) => ` + + `) + .join(""); +} + +function renderSourceFilterChips() { + const sources = Array.isArray(payload?.sources) ? payload.sources : []; + const enabled = new Set(getEnabledNewsSourceIds()); + if (!sources.length) { + return `${escapeNewsHtml(translateText("暂无可筛选来源。"))}`; + } + return sources + .map((source) => { + const id = String(source?.id || "").trim(); + if (!id) return ""; + const active = enabled.has(id); + return ` + + `; + }) + .join(""); +} + +function renderFilterPopover(kind) { + const title = translateText(kind === "source" ? "新闻来源" : "新闻类型"); + const hint = translateText(kind === "source" ? "按大来源筛选，不影响后台抓取。" : "按新闻内容分类筛选。"); + const content = kind === "source" ? renderSourceFilterChips() : renderCategoryFilterChips(); + + activeFilterPopover = kind; + document.querySelectorAll("[data-news-filter-popover]").forEach((popover) => { + if (!(popover instanceof HTMLElement)) return; + popover.hidden = false; + popover.innerHTML = ` +
+
${escapeNewsHtml(title)}
+
${escapeNewsHtml(hint)}
+
+
${content}
+ `; + }); + document.querySelectorAll("[data-news-filter-toggle]").forEach((toggle) => { + if (!(toggle instanceof HTMLElement)) return; + toggle.setAttribute("aria-expanded", toggle.dataset.newsFilterToggle === kind ? "true" : "false"); + }); +} + +function toggleNewsSource(sourceId) { + const available = getAvailableSourceIds(); + if (!available.includes(sourceId)) return; + const current = new Set(getEnabledNewsSourceIds()); + if (current.has(sourceId)) current.delete(sourceId); + else current.add(sourceId); + setNewsSourceFilters([...current]); + syncFilterSummaries(); + if (activeFilterPopover === "source") renderFilterPopover("source"); + lastFetchAt = 0; + refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {}); +} + +function toggleNewsCategory(category, enabled) { + window.dispatchEvent(new CustomEvent("earth:set-news-category-enabled", { + detail: { category, enabled }, + })); +} + +function toggleNewsListMode() { + newsFullListMode = !newsFullListMode; + syncFilterSummaries(); + lastFetchAt = 0; + refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {}); } function renderTicker(nextPayload) { @@ -308,19 +732,19 @@ function renderTicker(nextPayload) { const focus = nextPayload?.focus || {}; if (tickerRegion instanceof HTMLElement) { - tickerRegion.textContent = focus.display_region || getNewsRegionLabel(focus.region); + tickerRegion.textContent = getFocusRegionText(focus); tickerRegion.style.color = focus.accent || ""; } if (items.length === 0) { - tickerTrack.textContent = "正在准备全球态势新闻..."; + tickerTrack.textContent = translateText("正在准备全球态势新闻..."); tickerTrack.style.removeProperty("--news-ticker-duration"); return; } const visibleItems = getDisplayableNewsItems(items).slice(0, 6); if (visibleItems.length === 0) { - tickerTrack.textContent = "正在等待中文新闻..."; + tickerTrack.textContent = translateText("当前新闻类型没有可显示新闻..."); tickerTrack.style.removeProperty("--news-ticker-duration"); return; } @@ -328,7 +752,10 @@ function renderTicker(nextPayload) { tickerTrack.innerHTML = tickerItems .map((item) => ` - ${escapeTickerText(item.source || item.feed_name || "NEWS")} + ${escapeTickerText(getNewsSourceNameLabel({ + id: item.source_id, + name: item.source || item.feed_name || "NEWS", + }))} ${escapeTickerText(getNewsDisplaySummary(item))} `) @@ -344,7 +771,7 @@ function renderEmptyState(message) { empty.textContent = message; } if (status) { - status.textContent = "等待聚合新闻源"; + status.textContent = translateText("等待聚合新闻源"); } if (openBtn) openBtn.disabled = true; renderTicker({ items: [], focus: payload?.focus || { region: "global" } }); @@ -352,6 +779,7 @@ function renderEmptyState(message) { function renderPayload(nextPayload) { payload = nextPayload; + reconcileNewsSourceFilters(nextPayload); const { board, empty, @@ -366,9 +794,21 @@ function renderPayload(nextPayload) { const items = Array.isArray(nextPayload?.items) ? nextPayload.items : []; const displayItems = getDisplayableNewsItems(items); const sources = Array.isArray(nextPayload?.sources) ? nextPayload.sources : []; + const sourcesByName = new Map( + sources + .filter((source) => source && typeof source === "object" && source.name) + .map((source) => [String(source.name), source]), + ); + const sourcesById = new Map( + sources + .filter((source) => source && typeof source === "object" && source.id) + .map((source) => [String(source.id), source]), + ); const focus = nextPayload?.focus || {}; renderTicker(nextPayload); + syncFilterSummaries(nextPayload); + applyNewsBreakingShellState(nextPayload); if (!board || !status || !focusLabel || !focusCoords || !sourceCount || !regionChip) { if (document.body.classList.contains("layout-mode-mobile")) { @@ -382,7 +822,7 @@ function renderPayload(nextPayload) { } if (regionChip) { - regionChip.textContent = focus.display_region || getNewsRegionLabel(focus.region); + regionChip.textContent = getFocusRegionText(focus); regionChip.style.setProperty("--news-accent", focus.accent || "#d6e6ff"); } @@ -392,22 +832,22 @@ function renderPayload(nextPayload) { return; } - focusLabel.textContent = focus.label || "全球焦点"; + focusLabel.textContent = getFocusLabelText(focus); if (typeof focus.lat === "number" && typeof focus.lon === "number") { focusCoords.textContent = `${formatCoord(focus.lat, "N", "S")} · ${formatCoord(focus.lon, "E", "W")}`; } else { - focusCoords.textContent = "跟随当前视角自动聚焦"; + focusCoords.textContent = translateText("跟随当前视角自动聚焦"); } - sourceCount.textContent = `${sources.length} 路聚合源`; - if (nextPayload?.stale) { - status.textContent = `当前显示最近一次可用新闻缓存，共 ${items.length} 条`; - } else { - status.textContent = nextPayload?.errors?.length - ? `已聚合 ${items.length} 条，部分源不可用` - : `已聚合 ${items.length} 条态势新闻`; - } + const enabledSourceCount = getEnabledNewsSourceIds(nextPayload).length; + sourceCount.textContent = formatNewsSourceCount(enabledSourceCount, sources.length); + status.textContent = formatNewsStatus({ + displayCount: displayItems.length, + totalCount: items.length, + stale: Boolean(nextPayload?.stale), + hasErrors: Boolean(nextPayload?.errors?.length), + }); if (feedAnchor) { const matchedSource = sources.find((source) => source.region === focus.region) || sources[0]; @@ -423,8 +863,8 @@ function renderPayload(nextPayload) { if (empty) { empty.hidden = false; empty.textContent = items.length === 0 - ? "当前未拉到可用新闻，请稍后刷新或切换视角区域。" - : "正在等待中文新闻内容完成处理。"; + ? translateText("当前未拉到可用新闻，请稍后刷新或切换视角区域。") + : translateText("当前新闻类型没有可显示新闻。"); } return; } @@ -433,30 +873,47 @@ function renderPayload(nextPayload) { board.innerHTML = displayItems .map((item) => { - const cardClass = item.is_focus_match - ? "news-story-card news-story-card--focus" - : "news-story-card"; + const breakingLevel = isBreakingActive(item) ? normalizeBreakingLevel(item.breaking_level) : "none"; + const breakingScope = normalizeBreakingScope(item.breaking_scope); + const cardClass = [ + "news-story-card", + item.is_focus_match ? "news-story-card--focus" : "", + breakingLevel !== "none" ? `news-story-card--breaking-${breakingLevel}` : "", + breakingLevel !== "none" && breakingScope === "global" ? "news-story-card--breaking-global" : "", + ].filter(Boolean).join(" "); const title = getNewsDisplayTitle(item); const summaryText = getNewsDisplaySummary(item); const leadText = summaryText || title; - const regionLabel = item.display_region || getNewsRegionLabel(item.region); - const feedLabel = getNewsFeedLabel(item.feed_name); - const statusLabel = getNewsEnrichmentStatusLabel(item.enrichment_status); + const regionLabel = getNewsRegionDisplayLabel(item.region, item.display_region); + const categoryLabel = getNewsCategoryLabel(item.category); + const statusLabel = getNewsEnrichmentStatusLabel(item); + const breakingLabel = getNewsBreakingLabel(breakingLevel, breakingScope); + const sourceDescriptor = getNewsSourceDescriptor(item, sourcesByName, sourcesById); const summary = title && title !== leadText ? `
${escapeNewsHtml(title)}
` : ""; + const tagHtml = breakingLevel !== "none" + ? ` + ${escapeNewsHtml(breakingLabel)} + ${escapeNewsHtml(categoryLabel)} + ${escapeNewsHtml(regionLabel)} + ` + : ` + ${escapeNewsHtml(categoryLabel)} + ${escapeNewsHtml(regionLabel)} + ${escapeNewsHtml(statusLabel)} + `; return ` - +
- ${escapeNewsHtml(item.source || "NEWS")} + ${escapeNewsHtml(sourceDescriptor.sourceName)} ${formatRelativeTime(item.published_at)}
+
${escapeNewsHtml(sourceDescriptor.originLabel)}
${escapeNewsHtml(leadText)}
${summary}
- ${escapeNewsHtml(regionLabel)} - ${escapeNewsHtml(feedLabel)} - ${escapeNewsHtml(statusLabel)} + ${tagHtml}
`; @@ -472,11 +929,6 @@ function renderPayload(nextPayload) { })); } -function getNewsRealtimeUrl() { - const protocol = window.location.protocol === "https:" ? "wss:" : "ws:"; - return `${protocol}//${window.location.host}/ws`; -} - function clearNewsRealtimeReconnectTimer() { if (!newsRealtimeReconnectTimer) return; window.clearTimeout(newsRealtimeReconnectTimer); @@ -485,10 +937,11 @@ function clearNewsRealtimeReconnectTimer() { function scheduleNewsRealtimeReconnect() { if (newsRealtimeReconnectTimer) return; + const delay = Math.max(NEWS_REALTIME_RECONNECT_MS, getEarthRealtimeCooldownMs()); newsRealtimeReconnectTimer = window.setTimeout(() => { newsRealtimeReconnectTimer = null; connectNewsRealtime(); - }, NEWS_REALTIME_RECONNECT_MS); + }, delay); } function applyNewsRealtimePatch(updatePayload) { @@ -496,27 +949,36 @@ function applyNewsRealtimePatch(updatePayload) { const patch = updatePayload?.patch; if (!payload || !itemId || !patch || typeof patch !== "object") return; const items = Array.isArray(payload.items) ? payload.items : []; + const cruiseItems = Array.isArray(payload.cruise_items) ? payload.cruise_items : []; let changed = false; - const nextItems = items.map((item) => { + const patchItem = (item) => { if (item?.id !== itemId) return item; changed = true; return { ...item, ...patch, }; - }); + }; + const nextItems = items.map(patchItem); + const nextCruiseItems = cruiseItems.map(patchItem); if (!changed) return; renderPayload({ ...payload, items: nextItems, + cruise_items: Array.isArray(payload.cruise_items) ? nextCruiseItems : payload.cruise_items, }); } function connectNewsRealtime() { - if (newsRealtimeSocket || typeof WebSocket === "undefined") return; - const socket = new WebSocket(getNewsRealtimeUrl()); + if (newsRealtimeSocket || !canAttemptEarthRealtime()) { + scheduleNewsRealtimeReconnect(); + return; + } + const socket = new WebSocket(getEarthRealtimeUrl()); newsRealtimeSocket = socket; socket.onopen = () => { + socket.__planetOpened = true; + recordEarthRealtimeOpen(); clearNewsRealtimeReconnectTimer(); socket.send(JSON.stringify({ type: "subscribe", @@ -543,6 +1005,9 @@ function connectNewsRealtime() { if (newsRealtimeSocket === socket) { newsRealtimeSocket = null; } + if (!socket.__planetOpened) { + recordEarthRealtimeFailure(); + } scheduleNewsRealtimeReconnect(); }; socket.onerror = () => { @@ -550,10 +1015,42 @@ function connectNewsRealtime() { }; } -async function fetchNews(lat, lon) { +async function fetchNews(lat, lon, context = {}) { + const categorySignature = context.categorySignature ?? getNewsCategorySignature(); + const sourceSignature = context.sourceSignature ?? getNewsSourceSignatureForFetch(lat, lon); + if (categorySignature === "__none__" || sourceSignature === "__none__") { + const locale = getEarthLocale(); + return { + ...(payload || {}), + generated_at: new Date().toISOString(), + focus: payload?.focus || { + lat, + lon, + region: "global", + label: translateText("全球焦点"), + display_region: getNewsRegionLabel("global"), + }, + sources: payload?.sources || [], + filters: { + region: payload?.focus?.region || "global", + categories: categorySignature === "__none__" ? [] : getEnabledNewsCategoryKeys(), + sources: sourceSignature === "__none__" ? [] : getEnabledNewsSourceIds(), + limit: getNewsLimit(), + locale, + }, + items: [], + cruise_items: [], + errors: [], + stale: false, + }; + } const url = new URL(EARTH_NEWS_API, window.location.origin); if (typeof lat === "number") url.searchParams.set("lat", lat.toFixed(4)); if (typeof lon === "number") url.searchParams.set("lon", lon.toFixed(4)); + if (categorySignature) url.searchParams.set("categories", categorySignature); + if (sourceSignature) url.searchParams.set("sources", sourceSignature); + url.searchParams.set("limit", String(getNewsLimit())); + url.searchParams.set("locale", getEarthLocale()); const controller = new AbortController(); const timeoutId = window.setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS); @@ -570,39 +1067,64 @@ async function fetchNews(lat, lon) { } async function refreshNews(lat, lon, { silent = false } = {}) { - if (refreshPromise) return refreshPromise; + const targetRegion = inferRegion(lat, lon); + const categorySignature = getNewsCategorySignature(); + const sourceSignature = getNewsSourceSignatureForFetch(lat, lon); + const requestKey = [ + targetRegion, + categorySignature, + sourceSignature, + getNewsLimit(), + getEarthLocale(), + ].join("|"); + + if (refreshPromise && refreshRequestKey === requestKey) return refreshPromise; + const requestToken = ++activeRefreshToken; + refreshRequestKey = requestKey; const { status } = getElements(); if (status) { status.textContent = "正在同步全球态势新闻..."; } - refreshPromise = fetchNews(lat, lon) + refreshPromise = fetchNews(lat, lon, { categorySignature, sourceSignature }) .then((nextPayload) => { + if (requestToken !== activeRefreshToken) { + return nextPayload; + } renderPayload(nextPayload); lastFetchAt = Date.now(); + lastCategorySignature = getNewsCategorySignature(); + lastSourceSignature = getNewsSourceSignature(nextPayload); if (Array.isArray(nextPayload?.items) && nextPayload.items.length === 0) { const { status } = getElements(); if (status) { - status.textContent = "当前区域暂无可用新闻，已完成一次聚合尝试"; + status.textContent = translateText("当前区域暂无可用新闻，已完成一次聚合尝试"); } } + applyEarthI18n(); return nextPayload; }) .catch((error) => { + if (requestToken !== activeRefreshToken) { + return null; + } console.error("加载 Earth RSS 新闻失败:", error); const message = error?.name === "AbortError" - ? "新闻聚合请求超时，请稍后重试" - : `新闻聚合暂时不可用: ${error?.message || "未知错误"}`; + ? translateText("新闻聚合请求超时，请稍后重试") + : `${translateText("新闻聚合暂时不可用")}: ${error?.message || (getEarthLocale() === "en-US" ? "Unknown error" : "未知错误")}`; if (!payload) { renderEmptyState(message); } else if (!silent) { - showStatusMessage("态势新闻同步失败", "error"); + showStatusMessage(earthMessage("status.newsSyncFailed"), "error"); } throw error; }) .finally(() => { - refreshPromise = null; + if (requestToken === activeRefreshToken) { + refreshPromise = null; + refreshRequestKey = ""; + } }); return refreshPromise; @@ -614,6 +1136,8 @@ export async function refreshEarthNews({ silent = true } = {}) { function shouldRefreshForFocus(lat, lon, region) { const now = Date.now(); + if (getNewsCategorySignature() !== lastCategorySignature) return true; + if (getNewsSourceSignature() !== lastSourceSignature) return true; if (!lastFocus) return true; if (region !== lastFocus.region && now - lastRegionSwitchAt > MIN_REGION_SWITCH_INTERVAL_MS) { lastRegionSwitchAt = now; @@ -646,6 +1170,7 @@ export function updateNewsViewFocus(coords) { if (!coords || typeof coords.lat !== "number" || typeof coords.lon !== "number") return; const region = inferRegion(coords.lat, coords.lon); + const previousRegion = lastFocus?.region || null; const nextFocus = { lat: coords.lat, lon: coords.lon, @@ -654,6 +1179,9 @@ export function updateNewsViewFocus(coords) { }; const shouldRefresh = shouldRefreshForFocus(coords.lat, coords.lon, region); + if (!shouldRefresh && previousRegion && region !== previousRegion) { + return; + } lastFocus = nextFocus; if (shouldRefresh) { refreshNews(coords.lat, coords.lon, { silent: true }).catch(() => {}); @@ -678,6 +1206,12 @@ export function getVisibleNewsItems() { return getDisplayableNewsItems(payload?.items); } +export function getCruiseNewsItems() { + return getDisplayableNewsItems( + Array.isArray(payload?.cruise_items) ? payload.cruise_items : payload?.items, + ); +} + function updateBoardSelection(board, { scrollIntoView = false } = {}) { if (!(board instanceof HTMLElement)) return; const cards = board.querySelectorAll("[data-news-id]"); @@ -717,7 +1251,7 @@ export function initNewsPanel() { initialized = true; updateNewsToggleUI(true); - renderEmptyState("正在准备全球态势新闻聚合源..."); + renderEmptyState(translateText("正在准备全球态势新闻聚合源...")); const { ticker, hudCloseBtn } = getElements(); ticker?.addEventListener("click", (event) => { @@ -733,6 +1267,64 @@ export function initNewsPanel() { openNewsHud(); }); hudCloseBtn?.addEventListener("click", closeNewsHud); + document.addEventListener("keydown", (event) => { + if (event.key !== "Escape" || !isNewsHudOpen()) return; + event.preventDefault(); + if (activeFilterPopover) { + closeNewsFilterPopover(); + return; + } + closeNewsHud(); + }); + document.addEventListener("click", (event) => { + const target = event.target instanceof Element ? event.target : null; + if (!target) return; + const filterToggle = target.closest("[data-news-filter-toggle]"); + if (filterToggle instanceof HTMLElement) { + const kind = filterToggle.dataset.newsFilterToggle || ""; + if (activeFilterPopover === kind) closeNewsFilterPopover(); + else renderFilterPopover(kind); + return; + } + + const categoryToggle = target.closest("[data-news-category-toggle]"); + if (categoryToggle instanceof HTMLElement && categoryToggle.closest("[data-news-filter-popover]")) { + const category = categoryToggle.dataset.newsCategoryToggle || ""; + const active = categoryToggle.classList.contains("is-active"); + toggleNewsCategory(category, !active); + return; + } + + const sourceToggle = target.closest("[data-news-source-toggle]"); + if (sourceToggle instanceof HTMLElement) { + toggleNewsSource(sourceToggle.dataset.newsSourceToggle || ""); + return; + } + + const viewAllToggle = target.closest("#news-view-all-toggle, #mobile-news-view-all-toggle"); + if (viewAllToggle instanceof HTMLElement) { + toggleNewsListMode(); + return; + } + + if (activeFilterPopover && !target.closest("[data-news-filter-popover]")) { + closeNewsFilterPopover(); + } + }); + window.addEventListener("earth:news-category-filters-change", (event) => { + activeNewsCategoryFilters = event.detail?.categories || null; + syncFilterSummaries(); + if (activeFilterPopover === "category") renderFilterPopover("category"); + lastFetchAt = 0; + refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {}); + }); + onEarthLocaleChange(() => { + syncFilterSummaries(); + if (activeFilterPopover) renderFilterPopover(activeFilterPopover); + lastFetchAt = 0; + refreshRequestKey = ""; + refreshNews(lastFocus?.lat, lastFocus?.lon, { silent: true }).catch(() => {}); + }); setupNewsHudResize(); connectNewsRealtime(); @@ -741,9 +1333,9 @@ export function initNewsPanel() { refreshBtn?.addEventListener("click", async () => { try { await refreshNews(lastFocus?.lat, lastFocus?.lon); - showStatusMessage("态势新闻已刷新", "info"); + showStatusMessage(earthMessage("status.newsRefresh", { ok: true }), "info"); } catch { - showStatusMessage("态势新闻刷新失败", "error"); + showStatusMessage(earthMessage("status.newsRefresh", { ok: false }), "error"); } }); }); diff --git a/frontend/public/earth/js/oobe.js b/frontend/public/earth/js/oobe.js index 8c120bfb..122ed5b7 100644 --- a/frontend/public/earth/js/oobe.js +++ b/frontend/public/earth/js/oobe.js @@ -48,21 +48,29 @@ async function fetchOobeStatus() { } function statusSteps(status) { - return [ + const steps = [ { label: "后端服务在线", done: true }, { label: "登录控制台", done: Boolean(status.authenticated), warn: !status.authenticated }, { label: "配置或确认数据源", done: Number(status.datasource_count || 0) > 0 || Number(status.custom_config_count || 0) > 0 }, { label: "触发首次采集", done: Boolean(status.has_collected_data), warn: !status.has_collected_data }, { label: "回到 Earth 查看结果", done: Boolean(status.ready) }, ]; + if (status.demo_mode) { + steps.unshift({ label: "演示模式已开启", done: true }); + } + return steps; } function renderOobe(status) { const authenticated = Boolean(status.authenticated); + const demoMode = Boolean(status.demo_mode); const primaryHref = authenticated ? status.datasources_url || "/datasources" : status.login_url || "/login?next=/datasources"; const primaryText = authenticated ? "去采集数据" : "登录并采集数据"; const secondaryHref = status.collection_url || "/collection-management"; - const subtitle = authenticated + const docsHref = status.docs_url || "/docs/manual"; + const subtitle = demoMode + ? "演示模式已开启，本次访问将直接展示初始化引导，不受现有采集数据影响。" + : authenticated ? "当前还没有检测到可展示的数据，可以直接进入后台触发首次采集。" : "登录控制台并完成首次采集后，Earth 将显示实时数据层。"; const steps = statusSteps(status) @@ -99,6 +107,7 @@ function renderOobe(status) {
${primaryText} ${authenticated ? `进入采集管理` : ""} + ${authenticated ? `查看文档` : ""}
@@ -118,7 +127,8 @@ function renderOobe(status) { export async function initEarthOobe() { try { const status = await fetchOobeStatus(); - if (status?.ready || isTemporarilySkipped()) return; + const demoMode = Boolean(status?.demo_mode); + if ((status?.ready && !demoMode) || (isTemporarilySkipped() && !demoMode)) return; document.body.appendChild(renderOobe(status)); } catch (error) { console.warn("Earth OOBE status unavailable.", error); diff --git a/frontend/public/earth/js/realtime.js b/frontend/public/earth/js/realtime.js new file mode 100644 index 00000000..3aa97bbe --- /dev/null +++ b/frontend/public/earth/js/realtime.js @@ -0,0 +1,40 @@ +const WS_BACKOFF_BASE_MS = 5_000; +const WS_BACKOFF_MAX_MS = 60_000; + +const wsBackoffState = { + failures: 0, + nextAttemptAt: 0, +}; + +export function getEarthRealtimeUrl() { + const protocol = window.location.protocol === "https:" ? "wss:" : "ws:"; + const host = window.location.hostname; + const port = window.location.port; + if ((host === "localhost" || host === "127.0.0.1") && port === "3000") { + return `${protocol}//${host}:8000/ws`; + } + return `${protocol}//${window.location.host}/ws`; +} + +export function getEarthRealtimeCooldownMs() { + return Math.max(0, wsBackoffState.nextAttemptAt - Date.now()); +} + +export function canAttemptEarthRealtime() { + return typeof WebSocket !== "undefined" && getEarthRealtimeCooldownMs() <= 0; +} + +export function recordEarthRealtimeOpen() { + wsBackoffState.failures = 0; + wsBackoffState.nextAttemptAt = 0; +} + +export function recordEarthRealtimeFailure() { + wsBackoffState.failures += 1; + const delay = Math.min( + WS_BACKOFF_MAX_MS, + WS_BACKOFF_BASE_MS * 2 ** Math.min(wsBackoffState.failures - 1, 5), + ); + wsBackoffState.nextAttemptAt = Date.now() + delay; + return delay; +} diff --git a/frontend/public/earth/js/satellites.js b/frontend/public/earth/js/satellites.js index f7a0f245..1169eb53 100644 --- a/frontend/public/earth/js/satellites.js +++ b/frontend/public/earth/js/satellites.js @@ -1234,8 +1234,13 @@ export async function loadSatellites(options = {}) { if (limit !== null) { url.searchParams.set("limit", String(limit)); } + if (options.cacheBust) { + url.searchParams.set("_", String(Date.now())); + } - const response = await fetch(url.toString()); + const response = await fetch(url.toString(), { + cache: options.cache === "default" ? "default" : "no-store", + }); if (!response.ok) { throw new Error(`卫星接口返回 HTTP ${response.status}`); } diff --git a/frontend/public/earth/js/tv.js b/frontend/public/earth/js/tv.js index aa8192da..be7d3f9b 100644 --- a/frontend/public/earth/js/tv.js +++ b/frontend/public/earth/js/tv.js @@ -1,6 +1,14 @@ import Hls from "hls.js"; import { showStatusMessage } from "./ui.js"; import { createHUDPanel } from "./hud-panels.js"; +import { + earthMessage, + formatLocaleDateTime, + getEarthLocale, + hasCjkText, + onEarthLocaleChange, + translateText, +} from "./i18n.js"; // Naming convention: // - #media-panel is the outer HUD shell, responsible for drag/resize/show-hide @@ -39,12 +47,138 @@ const TV_PANEL_MIN_HEIGHT_PX = 340; const HLS_MAX_RECOVERY_ATTEMPTS = 3; const HLS_RETRY_CONFIG = { - maxNumRetry: 4, - retryDelayMs: 1500, - maxRetryDelayMs: 8000, + maxNumRetry: 1, + retryDelayMs: 1200, + maxRetryDelayMs: 3000, backoff: "exponential", }; +const HLS_NON_PLAYBACK_ERROR_DETAILS = new Set([ + "subtitleTrackLoadError", + "subtitleTrackParsingError", + "subtitleTrackSwitchError", + "audioTrackLoadError", + "audioTrackSwitchError", + "keyLoadError", +]); + +const HLS_SOURCE_FAILURE_DETAILS = new Set([ + "manifestLoadError", + "manifestLoadTimeOut", + "levelLoadError", + "levelLoadTimeOut", + "fragLoadError", + "fragLoadTimeOut", +]); + +const TV_SOURCE_TYPE_LABELS = { + hls: "HLS", + video: "Video", + iframe: "Web", + external: "External", + youtube: "YouTube", +}; + +function isEnglishLocale() { + return getEarthLocale() === "en-US"; +} + +function titleCaseIdentifier(value) { + return String(value || "") + .replace(/[-_]+/g, " ") + .replace(/\b\w/g, (letter) => letter.toUpperCase()) + .trim(); +} + +function safeTVText(value, fallback = "") { + const text = String(value ?? "").trim(); + if (!text) return fallback; + if (isEnglishLocale() && hasCjkText(text)) return fallback; + return text; +} + +function getTVSourceName(source) { + if (!source) return translateText("暂无可用频道"); + return safeTVText( + source.name, + titleCaseIdentifier(source.id) || "Live Channel", + ); +} + +function getTVSourceTypeLabel(sourceType) { + const normalized = String(sourceType || "").trim().toLowerCase(); + return TV_SOURCE_TYPE_LABELS[normalized] || safeTVText(sourceType, translateText("频道")); +} + +function getTVSourceField(value, fallback = "") { + return safeTVText(value, fallback); +} + +function getTVCollectorLabel(value) { + const collector = getTVSourceField(value, "Collector"); + return `${translateText("采集")}: ${collector}`; +} + +function getTVNotes(source) { + if (!source?.notes) return translateText("支持后台配置默认源与采集器补充源。"); + return safeTVText(source.notes, translateText("配置于控制台")); +} + +function isVideoActuallyPlaying(video) { + return ( + video instanceof HTMLVideoElement + && !video.paused + && !video.ended + && video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA + ); +} + +function getHlsErrorMessage(data) { + const details = data?.details || ""; + const response = data?.response || {}; + const status = response?.code || response?.status; + const url = data?.url || response?.url || ""; + + if (details === "manifestLoadError" || details === "manifestLoadTimeOut") { + return status + ? `HLS 主播放列表加载失败（HTTP ${status}）` + : "HLS 主播放列表加载失败"; + } + if (details === "levelLoadError" || details === "levelLoadTimeOut") { + return status + ? `HLS 清晰度播放列表加载失败（HTTP ${status}）` + : "HLS 清晰度播放列表加载失败"; + } + if (details === "fragLoadError" || details === "fragLoadTimeOut") { + return status + ? `HLS 分片加载失败（HTTP ${status}）` + : "HLS 分片加载失败"; + } + if (details === "bufferStalledError") return "直播流缓冲停滞，正在等待数据"; + if (details === "bufferAppendError") return "直播流缓冲写入失败"; + if (details === "manifestParsingError") return "HLS 播放列表格式无法解析"; + if (details === "fragParsingError") return "HLS 分片格式无法解析"; + if (url) return `HLS 资源加载失败：${url}`; + return "HLS 播放流不可用"; +} + +function logHlsDiagnostic(data, source) { + const payload = { + source_id: source?.id, + source_name: source?.name, + type: data?.type, + details: data?.details, + fatal: Boolean(data?.fatal), + url: data?.url || data?.response?.url, + status: data?.response?.code || data?.response?.status, + }; + if (data?.fatal) { + console.error("HLS 播放失败:", payload, data); + } else { + console.debug("HLS 非致命事件:", payload, data); + } +} + function getElements() { const isMobile = document.body.classList.contains("layout-mode-mobile"); return { @@ -84,9 +218,9 @@ function syncMetaToggleState(collapsed) { desktopToggle.setAttribute("aria-expanded", collapsed ? "false" : "true"); desktopToggle.setAttribute( "aria-label", - collapsed ? "展开新闻直播内容" : "折叠新闻直播内容", + translateText(collapsed ? "展开新闻直播内容" : "折叠新闻直播内容"), ); - desktopToggle.title = collapsed ? "展开新闻直播内容" : "折叠新闻直播内容"; + desktopToggle.title = translateText(collapsed ? "展开新闻直播内容" : "折叠新闻直播内容"); } } @@ -255,11 +389,11 @@ function updateToggleButton(visible) { icon.textContent = "live_tv"; } const title = visible ? "关闭 Live 新闻" : "打开 Live 新闻"; - toggleBtn.title = title; - toggleBtn.setAttribute("aria-label", title); + toggleBtn.title = translateText(title); + toggleBtn.setAttribute("aria-label", translateText(title)); const tooltip = toggleBtn.querySelector(".earth-toolbar-tooltip"); if (tooltip) { - tooltip.textContent = title; + tooltip.textContent = translateText(title); } } @@ -398,6 +532,8 @@ function tryStartPlayback(video) { function attachVideoSource(video, source) { const sourceUrl = getVideoUrl(source); if (!(video instanceof HTMLVideoElement) || !sourceUrl) return; + let hlsNetworkErrorCount = 0; + let hlsHardFailureHandled = false; destroyHlsPlayer(); video.autoplay = true; @@ -414,16 +550,18 @@ function attachVideoSource(video, source) { if (Hls.isSupported()) { hlsPlayer = new Hls({ enableWorker: true, + enableWebVTT: false, + subtitleDisplay: false, lowLatencyMode: false, manifestLoadingTimeOut: 20000, levelLoadingTimeOut: 20000, fragLoadingTimeOut: 25000, - fragLoadingMaxRetry: 3, - fragLoadingRetryDelay: 1500, - levelLoadingMaxRetry: 3, - levelLoadingRetryDelay: 1500, - manifestLoadingMaxRetry: 2, - manifestLoadingRetryDelay: 1500, + fragLoadingMaxRetry: 1, + fragLoadingRetryDelay: 1200, + levelLoadingMaxRetry: 1, + levelLoadingRetryDelay: 1200, + manifestLoadingMaxRetry: 1, + manifestLoadingRetryDelay: 1200, liveSyncDurationCount: 4, liveMaxLatencyDurationCount: 10, manifestLoadPolicy: { @@ -432,11 +570,11 @@ function attachVideoSource(video, source) { maxLoadTimeMs: 20000, timeoutRetry: { ...HLS_RETRY_CONFIG, - maxNumRetry: 2, + maxNumRetry: 1, }, errorRetry: { ...HLS_RETRY_CONFIG, - maxNumRetry: 2, + maxNumRetry: 1, }, }, }, @@ -465,14 +603,35 @@ function attachVideoSource(video, source) { tryStartPlayback(video); }); hlsPlayer.on(Hls.Events.ERROR, (_event, data) => { - console.error("HLS 播放失败:", data); + logHlsDiagnostic(data, source); + const status = Number(data?.response?.code || data?.response?.status || 0); + const sourceFailure = HLS_SOURCE_FAILURE_DETAILS.has(data?.details); + if (sourceFailure && (status >= 500 || data?.details === "manifestLoadError")) { + hlsNetworkErrorCount += 1; + } + if (!hlsHardFailureHandled && !isVideoActuallyPlaying(video) && sourceFailure && hlsNetworkErrorCount >= 2) { + hlsHardFailureHandled = true; + const reasonMessage = getHlsErrorMessage(data); + hlsPlayer?.stopLoad(); + if (!showEmbeddedFallback(source, reasonMessage) && !tryFallbackSource()) { + setPanelMessage(reasonMessage); + } + return; + } + if ( + !data?.fatal + && HLS_NON_PLAYBACK_ERROR_DETAILS.has(data?.details) + && isVideoActuallyPlaying(video) + ) { + return; + } if (!data?.fatal) { if (data?.type === Hls.ErrorTypes.NETWORK_ERROR) { - setPanelMessage("直播流网络波动，正在重试..."); + setPanelMessage(isVideoActuallyPlaying(video) ? TV_STATUS_MESSAGE.videoReady : "直播流网络波动，正在重试..."); return; } if (data?.type === Hls.ErrorTypes.MEDIA_ERROR) { - setPanelMessage("直播流正在恢复..."); + setPanelMessage(isVideoActuallyPlaying(video) ? TV_STATUS_MESSAGE.videoReady : "直播流正在恢复..."); return; } } @@ -491,8 +650,9 @@ function attachVideoSource(video, source) { } } - if (!showEmbeddedFallback(source) && !tryFallbackSource()) { - setPanelMessage(TV_STATUS_MESSAGE.videoError); + const reasonMessage = getHlsErrorMessage(data); + if (!showEmbeddedFallback(source, reasonMessage) && !tryFallbackSource()) { + setPanelMessage(reasonMessage); } }); return; @@ -529,30 +689,30 @@ function syncMobileOverviewSummary(source) { const summary = document.getElementById("mobile-tv-overview-summary"); const tags = document.getElementById("mobile-tv-overview-tags"); if (headline instanceof HTMLElement) { - headline.textContent = source?.name || "暂无可用频道"; + headline.textContent = getTVSourceName(source); } if (summary instanceof HTMLElement) { if (!source) { - summary.textContent = "点击查看当前频道来源、目录和补充说明"; + summary.textContent = translateText("点击查看当前频道来源、目录和补充说明"); } else { const parts = [ - source.provider, - source.region, - source.language, + getTVSourceField(source.provider), + getTVSourceField(source.region), + getTVSourceField(source.language), ].filter(Boolean); summary.textContent = parts.length ? parts.join(" · ") - : "点击查看完整频道信息"; + : translateText("点击查看完整频道信息"); } } if (tags instanceof HTMLElement) { const tagValues = source ? [ - { label: source.source_type || "频道", kind: "status" }, - source.collector_source ? { label: `采集:${source.collector_source}`, kind: "" } : { label: "内置源", kind: "" }, - source.region ? { label: source.region, kind: "" } : null, + { label: getTVSourceTypeLabel(source.source_type), kind: "status" }, + source.collector_source ? { label: getTVCollectorLabel(source.collector_source), kind: "" } : { label: translateText("内置源"), kind: "" }, + source.region ? { label: getTVSourceField(source.region, "Global"), kind: "" } : null, ].filter(Boolean).slice(0, 3) - : [{ label: "待加载", kind: "status" }]; + : [{ label: translateText("待加载"), kind: "status" }]; tags.replaceChildren( ...tagValues.map(({ label, kind }) => { const chip = document.createElement("span"); @@ -628,7 +788,7 @@ function getCurrentSource() { function setPanelMessage(message) { const { status } = getElements(); if (status) { - status.textContent = message || TV_STATUS_MESSAGE.idle; + status.textContent = translateText(message || TV_STATUS_MESSAGE.idle); if (status.id === "tv-source-status") { const normalized = message || TV_STATUS_MESSAGE.idle; status.classList.toggle( @@ -663,7 +823,7 @@ function resetVideo(video) { function showEmptyState(empty, message) { if (!(empty instanceof HTMLElement)) return; empty.hidden = false; - empty.textContent = message; + empty.textContent = translateText(message); } function hideEmptyState(empty) { @@ -698,11 +858,11 @@ function renderSourceOptions() { const fragment = document.createDocumentFragment(); sources.forEach((source) => { - const defaultMark = source.id === tvPayload?.default_source_id ? " · 默认" : ""; + const defaultMark = source.id === tvPayload?.default_source_id ? ` · ${translateText("默认")}` : ""; const failMark = failedSourceIds.has(source.id) ? " ⚠" : ""; const option = document.createElement("option"); option.value = source.id; - option.textContent = `${source.name}${defaultMark}${failMark}`; + option.textContent = `${getTVSourceName(source)}${defaultMark}${failMark}`; fragment.appendChild(option); }); @@ -721,27 +881,37 @@ function renderSource(source) { const isExternalOnly = Boolean(source) && !embeddedUrl && !videoUrl && Boolean(externalUrl); if (title) { - title.textContent = source?.name || "暂无可用频道"; + title.textContent = getTVSourceName(source); } if (origin instanceof HTMLElement) { - origin.textContent = source?.collector_source ? "采集" : source ? "内置" : "待加载"; - origin.title = source?.collector_source ? `采集源：${source.collector_source}` : source ? "内置源" : "待加载"; + origin.textContent = source?.collector_source ? translateText("采集") : source ? translateText("内置") : translateText("待加载"); + origin.title = source?.collector_source ? `${translateText("采集源")}: ${getTVSourceField(source.collector_source, "Collector")}` : source ? translateText("内置源") : translateText("待加载"); } if (meta) { - meta.textContent = source - ? `${source.provider} · ${source.region} · ${source.language} · ${source.source_type}` - : "当前未配置可播放新闻直播源"; + const metaParts = source + ? [ + getTVSourceField(source.provider), + getTVSourceField(source.region), + getTVSourceField(source.language), + getTVSourceTypeLabel(source.source_type), + ].filter(Boolean) + : []; + meta.textContent = metaParts.length + ? metaParts.join(" · ") + : translateText("当前未配置可播放新闻直播源"); } if (catalog) { const sourceCount = tvPayload?.source_count || tvPayload?.sources?.length || 0; const latestUpdatedAt = tvPayload?.latest_updated_at || tvPayload?.generated_at || ""; const latestLabel = latestUpdatedAt - ? `最近同步 ${new Date(latestUpdatedAt).toLocaleString("zh-CN", { hour12: false })}` - : "尚未同步"; - catalog.textContent = `共 ${sourceCount} 个频道 · ${latestLabel}`; + ? `${translateText("最近同步")} ${formatLocaleDateTime(latestUpdatedAt)}` + : translateText("尚未同步"); + catalog.textContent = isEnglishLocale() + ? `${sourceCount} channels · ${latestLabel}` + : `共 ${sourceCount} 个频道 · ${latestLabel}`; } if (notes) { - notes.textContent = source?.notes || "支持后台配置默认源与采集器补充源。"; + notes.textContent = getTVNotes(source); } syncMobileOverviewSummary(source); @@ -855,8 +1025,8 @@ export function initTVPanel() { collapseBtn: metaToggle, bodyCollapsedClass: "is-collapsed", preferredDirection: "up", - expandLabel: "展开新闻直播信息", - collapseLabel: "折叠新闻直播信息", + expandLabel: translateText("展开新闻直播信息"), + collapseLabel: translateText("折叠新闻直播信息"), }); } @@ -866,6 +1036,12 @@ export function initTVPanel() { updateToggleButton(mediaPanel?.isVisible() ?? !panel?.classList.contains("hud-panel-hidden")); syncSettingsToggle(mediaPanel?.isVisible() ?? !panel?.classList.contains("hud-panel-hidden")); + onEarthLocaleChange(() => { + syncMetaToggleState(isMetaCollapsed()); + updateToggleButton(mediaPanel?.isVisible() ?? !panel?.classList.contains("hud-panel-hidden")); + if (tvPayload) renderPanel(); + else renderSource(null); + }); toggleBtn?.addEventListener("click", async (event) => { event.preventDefault(); @@ -874,12 +1050,12 @@ export function initTVPanel() { if (!currentlyVisible) { setPanelVisible(true); await ensureTVPanelReady(); - showStatusMessage("Live 新闻窗口已打开", "info"); + showStatusMessage(earthMessage("status.newsPanel", { open: true }), "info"); return; } setPanelVisible(false); - showStatusMessage("Live 新闻窗口已关闭", "info"); + showStatusMessage(earthMessage("status.newsPanel", { open: false }), "info"); }); [select, document.getElementById("mobile-tv-source-select"), document.getElementById("tv-source-select")] diff --git a/frontend/public/earth/js/ui.js b/frontend/public/earth/js/ui.js index f70829ab..6718c176 100644 --- a/frontend/public/earth/js/ui.js +++ b/frontend/public/earth/js/ui.js @@ -1,5 +1,14 @@ // ui.js - UI update functions +import { + applyEarthI18n, + earthMessage, + formatEarthMessage, + getEarthLocale, + onEarthLocaleChange, + translateText, +} from "./i18n.js"; + let statusTimeoutId = null; let statusHideTimeoutId = null; const STATUS_BASE_CLASS = "earth-status-message"; @@ -27,14 +36,30 @@ function getEarthStatTargets(statKey) { ); } +function formatEarthStatValue(value) { + return translateText(value); +} + +function syncEarthStatElement(element) { + if (!(element instanceof HTMLElement)) return; + const sourceValue = element.dataset.earthStatSourceValue; + if (sourceValue === undefined) return; + element.textContent = formatEarthStatValue(sourceValue); +} + export function setEarthStatValue(statKey, value) { getEarthStatTargets(statKey).forEach((element) => { if (element instanceof HTMLElement) { - element.textContent = value; + element.dataset.earthStatSourceValue = String(value ?? ""); + syncEarthStatElement(element); } }); } +onEarthLocaleChange(() => { + document.querySelectorAll("[data-earth-stat-source-value]").forEach(syncEarthStatElement); +}); + function setElementDisplay(element, visible, displayValue = "block") { if (!element) return; element.style.display = visible ? displayValue : "none"; @@ -82,7 +107,7 @@ function buildStatusContent(statusEl, message, type) { const text = document.createElement("span"); text.className = "earth-status-text"; - text.textContent = message; + text.textContent = formatEarthMessage(message); statusEl.appendChild(indicator); statusEl.appendChild(text); @@ -92,23 +117,20 @@ function getStatusSidePlacement(statusEl) { if (!(statusEl instanceof HTMLElement)) return false; if (document.querySelector(".layout-mode-mobile")) return false; - const ticker = document.getElementById("desktop-news-ticker"); const brand = document.getElementById("brand-panel"); - if (!(ticker instanceof HTMLElement) || !(brand instanceof HTMLElement)) return false; - if (ticker.classList.contains("is-hidden") || ticker.offsetParent === null) return false; + if (!(brand instanceof HTMLElement)) return false; - const tickerRect = ticker.getBoundingClientRect(); const brandRect = brand.getBoundingClientRect(); const statusWidth = Math.ceil(statusEl.getBoundingClientRect().width || statusEl.scrollWidth || 0); - if (!statusWidth || !tickerRect.width || !brandRect.width) return false; + if (!statusWidth || !brandRect.width) return false; const rootStyle = getComputedStyle(document.documentElement); const hudScale = Number.parseFloat(rootStyle.getPropertyValue("--hud-scale")) || 1; const requiredGap = Math.max(10, Math.round(12 * hudScale)); - const availableWidth = tickerRect.left - brandRect.right - requiredGap * 2; + const left = Math.round(brandRect.right + requiredGap); + const availableWidth = window.innerWidth - left - requiredGap; return { - shouldStack: availableWidth < statusWidth, - left: Math.round(brandRect.right + requiredGap), + left, maxWidth: Math.max(160, Math.floor(availableWidth)), }; } @@ -122,9 +144,8 @@ function syncStatusPlacement(statusEl) { return; } const placement = getStatusSidePlacement(statusEl); - const shouldStack = !placement || placement.shouldStack; - statusEl.classList.toggle(STATUS_TICKER_STACK_CLASS, shouldStack); - if (!placement || shouldStack) { + statusEl.classList.remove(STATUS_TICKER_STACK_CLASS); + if (!placement) { statusEl.style.left = ""; statusEl.style.maxWidth = ""; return; @@ -134,10 +155,12 @@ function syncStatusPlacement(statusEl) { } function syncVisibleStatusPlacement() { - const statusEl = getElement("status-message"); - if (statusEl?.classList.contains("visible")) { - syncStatusPlacement(statusEl); - } + ["status-message", "error-message"].forEach((id) => { + const el = getElement(id); + if (el?.classList.contains("visible")) { + syncStatusPlacement(el); + } + }); } function buildPersistentErrorContent(errorEl, message) { @@ -237,7 +260,9 @@ export function updateCoordinatesDisplay(lat, lon, alt = 0) { if (longitudeEl) longitudeEl.textContent = lon.toFixed(2) + "°"; if (latitudeEl) latitudeEl.textContent = lat.toFixed(2) + "°"; if (mouseCoordsEl) { - mouseCoordsEl.textContent = `鼠标: ${lat.toFixed(2)}°, ${lon.toFixed(2)}°`; + mouseCoordsEl.textContent = getEarthLocale() === "en-US" + ? `Mouse: ${lat.toFixed(2)}°, ${lon.toFixed(2)}°` + : `鼠标: ${lat.toFixed(2)}°, ${lon.toFixed(2)}°`; } } @@ -253,7 +278,11 @@ export function updateZoomDisplay(zoomLevel, distance) { const label = `${percent}%`; zoomValueEl.textContent = label; } - if (zoomLevelEl) zoomLevelEl.textContent = "缩放: " + percent + "%"; + if (zoomLevelEl) { + zoomLevelEl.textContent = getEarthLocale() === "en-US" + ? `Zoom: ${percent}%` + : `缩放: ${percent}%`; + } if (slider) slider.value = zoomLevel; if (cameraDistanceEl) cameraDistanceEl.textContent = distance + " km"; } @@ -276,7 +305,14 @@ export function updateEarthStats(stats) { setEarthStatValue("bgp-collector-count", String(stats.bgpCollectorCount || 0)); } if (has("bgpStatusSummary")) setEarthStatValue("bgp-status-summary", stats.bgpStatusSummary || "-"); - if (has("terrainOn")) setEarthStatValue("terrain-status", stats.terrainOn ? "开启" : "关闭"); + if (has("terrainOn")) { + setEarthStatValue( + "terrain-status", + getEarthLocale() === "en-US" + ? (stats.terrainOn ? "On" : "Off") + : (stats.terrainOn ? "开启" : "关闭"), + ); + } if (has("textureQuality")) setEarthStatValue("texture-quality", stats.textureQuality || "8K 卫星图"); } @@ -292,7 +328,7 @@ export function setLoading(loading) { clearLoadingWidthLock(statusEl); buildStatusContent( statusEl, - pendingLoadingMessage || "正在加载...", + pendingLoadingMessage || earthMessage("loading.default"), "loading", ); pendingLoadingMessage = ""; @@ -331,7 +367,7 @@ export function setLoadingMessage(title) { } const textEl = statusEl.querySelector(".earth-status-text"); if (textEl) { - textEl.textContent = title; + textEl.textContent = formatEarthMessage(title); requestAnimationFrame(() => { updateLoadingWidthLock(statusEl); syncStatusPlacement(statusEl); @@ -344,6 +380,7 @@ export function showTooltip(x, y, content) { const tooltip = getElement("tooltip"); if (!tooltip) return; tooltip.innerHTML = content; + applyEarthI18n(tooltip); tooltip.style.left = x + "px"; tooltip.style.top = y + "px"; setElementDisplay(tooltip, true); @@ -364,6 +401,7 @@ export function showError(message) { buildPersistentErrorContent(errorEl, message); errorEl.className = `${STATUS_BASE_CLASS} earth-error-message error`; setElementDisplay(errorEl, true, "inline-flex"); + syncStatusPlacement(errorEl); errorEl.offsetHeight; errorEl.classList.add("visible"); } diff --git a/frontend/public/earth/js/vessels.js b/frontend/public/earth/js/vessels.js index bb2cbbb2..ec6c738c 100644 --- a/frontend/public/earth/js/vessels.js +++ b/frontend/public/earth/js/vessels.js @@ -6,9 +6,8 @@ import { latLonToVector3 } from "./utils.js"; let showVessels = false; let activeTrackLine = null; -let vesselStreamSocket = null; -let vesselStreamReconnectTimer = null; let vesselDataByKey = new Map(); +let vesselSnapshotGeneration = 0; let vesselRealtimeStats = { connected: false, updates: 0, @@ -63,54 +62,6 @@ function markerDataToDedupeKey(item) { ].join(":"); } -function buildVesselFeatureFromDelta(item) { - const lat = Number(item?.lat ?? item?.latitude); - const lon = Number(item?.lon ?? item?.lng ?? item?.longitude); - if (!Number.isFinite(lat) || !Number.isFinite(lon)) return null; - return { - type: "Feature", - id: item.mmsi, - geometry: { - type: "Point", - coordinates: [lon, lat], - }, - properties: { - ...item, - mmsi: item.mmsi, - mmsi_display: item.mmsi_display || (item.mmsi !== undefined && item.mmsi !== null ? String(item.mmsi) : undefined), - }, - }; -} - -function rebuildVesselLayerFromCache(earth) { - if (!earth) return; - vesselIconLayer.setData(Array.from(vesselDataByKey.values())); - vesselIconLayer.attach(earth); - vesselIconLayer.setVisible(showVessels); -} - -function applyVesselDeltas(earth, vessels = []) { - let changed = false; - vessels.forEach((item) => { - const feature = buildVesselFeatureFromDelta(item); - if (!feature) return; - const marker = buildVesselMarkerData(feature); - if (!marker) return; - vesselDataByKey.set(markerDataToDedupeKey(marker), marker); - changed = true; - }); - if (changed) { - rebuildVesselLayerFromCache(earth); - } - return changed; -} - -function getVesselStreamUrl() { - if (typeof window === "undefined") return "ws://localhost:8000/ws"; - const protocol = window.location.protocol === "https:" ? "wss:" : "ws:"; - return `${protocol}//${window.location.host}/ws`; -} - function normalizeVesselType(value, code) { const type = String(value || "").trim().toLowerCase(); const numericCode = Number(code); @@ -257,13 +208,13 @@ const vesselIconLayer = createInteractableLayer({ vessel_kind: item.type, baseScale: VESSEL_CONFIG.marker.baseScale, }), + cluster: false, }); -const DEFAULT_VESSEL_VIEWPORT = { - bbox: [-10, 50, 35, 75], +const GLOBAL_VESSEL_VIEWPORT = { + bbox: [-180, -85.05112878, 180, 85.05112878], zoom: 4, }; -const MAX_VESSEL_SUBSCRIPTION_BBOX_AREA = 2500; export function getVesselMarkers() { return vesselIconLayer.getMarkers(); @@ -318,25 +269,25 @@ export function clearVesselData(earth) { } export async function loadVessels(_scene, earth, options = {}) { - const params = new URLSearchParams(); + const requestGeneration = ++vesselSnapshotGeneration; const requestedLimit = Number(options.limit ?? VESSEL_CONFIG.maxRenderedMarkers); const bbox = Array.isArray(options.bbox) && options.bbox.length === 4 ? options.bbox - : DEFAULT_VESSEL_VIEWPORT.bbox; + : GLOBAL_VESSEL_VIEWPORT.bbox; const zoom = Number.isFinite(Number(options.zoom)) ? Number(options.zoom) - : DEFAULT_VESSEL_VIEWPORT.zoom; - params.set("bbox", bbox.join(",")); - params.set("zoom", String(zoom)); + : GLOBAL_VESSEL_VIEWPORT.zoom; + const params = new URLSearchParams({ bbox: bbox.join(","), zoom: String(zoom) }); if (Number.isFinite(requestedLimit) && requestedLimit > 0) { params.set("limit", String(requestedLimit)); } - const response = await fetch(`${PATHS.vesselsApi}?${params.toString()}`); - if (!response.ok) { - throw new Error(`Vessels HTTP ${response.status}`); - } + const response = await fetch(`${PATHS.vesselsApi}?${params.toString()}`, { cache: "no-store" }); + if (!response.ok) throw new Error(`Vessels HTTP ${response.status}`); const payload = await response.json(); const features = Array.isArray(payload?.features) ? payload.features : []; + if (requestGeneration !== vesselSnapshotGeneration) { + return { totalCount: getVesselCount(), stats: {} }; + } clearVesselData(earth); let markerData = dedupeVesselFeatures(features); @@ -355,123 +306,12 @@ export async function loadVessels(_scene, earth, options = {}) { }; } -function normalizeVesselViewportOptions(options = {}) { - const bbox = Array.isArray(options.bbox) && options.bbox.length === 4 - ? options.bbox.map(Number) - : DEFAULT_VESSEL_VIEWPORT.bbox; - const [lonA, latA, lonB, latB] = bbox; - const normalizedBbox = [ - Math.max(-180, Math.min(lonA, lonB)), - Math.max(-90, Math.min(latA, latB)), - Math.min(180, Math.max(lonA, lonB)), - Math.min(90, Math.max(latA, latB)), - ]; - const area = (normalizedBbox[2] - normalizedBbox[0]) * (normalizedBbox[3] - normalizedBbox[1]); - const safeBbox = Number.isFinite(area) && area <= MAX_VESSEL_SUBSCRIPTION_BBOX_AREA - ? normalizedBbox - : DEFAULT_VESSEL_VIEWPORT.bbox; - const zoom = Number.isFinite(Number(options.zoom)) - ? Number(options.zoom) - : DEFAULT_VESSEL_VIEWPORT.zoom; - return { - bbox: safeBbox, - zoom: Math.max(1, Math.min(20, Math.round(zoom))), - limit: options.limit ?? VESSEL_CONFIG.maxRenderedMarkers, - }; -} - -export function startVesselRealtime(earth, { onUpdate, bbox, zoom, limit } = {}) { - if (vesselStreamSocket || typeof WebSocket === "undefined") return; - const subscriptionOptions = normalizeVesselViewportOptions({ bbox, zoom, limit }); - const connect = () => { - if (!showVessels || vesselStreamSocket) return; - const socket = new WebSocket(getVesselStreamUrl()); - vesselStreamSocket = socket; - socket.onopen = () => { - vesselRealtimeStats = { - ...vesselRealtimeStats, - connected: true, - }; - onUpdate?.({ totalCount: getVesselCount(), stream: getVesselRealtimeStats() }); - socket.send(JSON.stringify({ - type: "subscribe", - data: { - channel: "vessels", - bbox: subscriptionOptions.bbox, - zoom: subscriptionOptions.zoom, - limit: subscriptionOptions.limit, - }, - })); - }; - socket.onmessage = (event) => { - let message; - try { - message = JSON.parse(event.data); - } catch { - return; - } - if (message.type === "heartbeat" && message.data?.action === "ping") { - socket.send(JSON.stringify({ type: "heartbeat" })); - return; - } - if (message.type !== "data_frame" || message.channel !== "vessels") return; - const payload = message.payload || {}; - if (payload.action === "reload") { - loadVessels(null, earth) - .then((result) => { - vesselRealtimeStats = { - ...vesselRealtimeStats, - connected: true, - updates: vesselRealtimeStats.updates + 1, - lastUpdateAt: new Date(), - lastBatchSize: 0, - }; - onUpdate?.({ totalCount: result?.totalCount ?? getVesselCount(), payload, stream: getVesselRealtimeStats() }); - }) - .catch(() => {}); - return; - } - if (payload.action !== "upsert" || !Array.isArray(payload.vessels)) return; - if (applyVesselDeltas(earth, payload.vessels)) { - vesselRealtimeStats = { - connected: true, - updates: vesselRealtimeStats.updates + 1, - lastUpdateAt: new Date(), - lastBatchSize: payload.vessels.length, - }; - onUpdate?.({ totalCount: getVesselCount(), payload, stream: getVesselRealtimeStats() }); - } - }; - socket.onclose = () => { - if (vesselStreamSocket === socket) { - vesselStreamSocket = null; - } - vesselRealtimeStats = { - ...vesselRealtimeStats, - connected: false, - }; - onUpdate?.({ totalCount: getVesselCount(), stream: getVesselRealtimeStats() }); - if (showVessels) { - vesselStreamReconnectTimer = window.setTimeout(connect, 3000); - } - }; - socket.onerror = () => { - socket.close(); - }; - }; - connect(); +export function startVesselRealtime(_earth, { onUpdate } = {}) { + onUpdate?.({ totalCount: getVesselCount(), stream: getVesselRealtimeStats() }); } export function stopVesselRealtime() { - if (vesselStreamReconnectTimer) { - window.clearTimeout(vesselStreamReconnectTimer); - vesselStreamReconnectTimer = null; - } - if (vesselStreamSocket) { - const socket = vesselStreamSocket; - vesselStreamSocket = null; - socket.close(); - } + vesselSnapshotGeneration += 1; vesselRealtimeStats = { connected: false, updates: 0, diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 7ac080ce..c6ec451f 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -1,28 +1,19 @@ -import { Suspense, lazy } from 'react' +import { Suspense, lazy, useEffect } from 'react' -import { Spin } from 'antd' +import { useTranslation } from 'react-i18next' import { Routes, Route, Navigate, useLocation } from 'react-router-dom' import { useAuthStore } from './stores/auth' import Login from './pages/Login/Login' +import { AdminErrorBoundary } from './admin/components/AdminErrorBoundary' +import LegacyI18nBridge from './i18n/LegacyI18nBridge' const Register = lazy(() => import('./pages/Register/Register')) const VerifyEmail = lazy(() => import('./pages/VerifyEmail/VerifyEmail')) const ForgotPassword = lazy(() => import('./pages/ForgotPassword/ForgotPassword')) -const SystemAlerts = lazy(() => import('./pages/Alerts/SystemAlerts')) -const BGPAlerts = lazy(() => import('./pages/Alerts/BGPAlerts')) -const SituationalAlerts = lazy(() => import('./pages/Alerts/SituationalAlerts')) -const Dashboard = lazy(() => import('./pages/Dashboard/Dashboard')) -const Users = lazy(() => import('./pages/Users/Users')) -const DataSources = lazy(() => import('./pages/DataSources/DataSources')) -const DataList = lazy(() => import('./pages/DataList/DataList')) const Earth = lazy(() => import('./pages/Earth/Earth')) -const Settings = lazy(() => import('./pages/Settings/Settings')) -const AISettings = lazy(() => import('./pages/AISettings/AISettings')) -const BGP = lazy(() => import('./pages/BGP/BGP')) -const Logs = lazy(() => import('./pages/Logs/Logs')) const Docs = lazy(() => import('./pages/Docs/Docs')) -const AdminNextRoutes = lazy(() => import('./admin-next/AdminNextRoutes')) +const AdminRoutes = lazy(() => import('./admin/AdminRoutes')) const ROOT_ROUTE = '/' const EARTH_ROUTE = '/earth' @@ -36,59 +27,44 @@ function isPublicPath(pathname: string) { return PUBLIC_EXACT_ROUTES.has(pathname) || pathname.startsWith(DOCS_ROUTE_PREFIX) } -function AdminNextCompatRedirect() { - const { pathname, search, hash } = useLocation() - const nextPath = pathname.replace(/^\/admin-next/, '') || '' - return -} - function App() { + const { t } = useTranslation() const { token } = useAuthStore() const { pathname } = useLocation() const isPublicRoute = isPublicPath(pathname) + useEffect(() => { + document.title = t('app.title') + }, [t]) + if (!token && !isPublicRoute) { return } return ( - - -
- )} - > - - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - - + <> + + +
+
+ )} + > + + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + + + ) } diff --git a/frontend/src/admin-next/AdminNextRoutes.tsx b/frontend/src/admin-next/AdminNextRoutes.tsx deleted file mode 100644 index 6132d7d7..00000000 --- a/frontend/src/admin-next/AdminNextRoutes.tsx +++ /dev/null @@ -1,48 +0,0 @@ -import { Navigate, Route, Routes } from 'react-router-dom' -import { AdminThemeProvider } from './design/theme' -import DashboardNext from './pages/DashboardNext' -import DataListNext from './pages/DataListNext' -import { - AINext, - BGPAlertsNext, - BGPNext, - CollectionManagementNext, - DataSourcesNext, - EarthContentNext, - SettingsNext, - SituationalAlertsNext, - SystemAlertsNext, -} from './pages/PlainResourcePages' -import UsersNext from './pages/UsersNext' -import LogsNext from './pages/LogsNext' -import { ToastProvider } from './components/ui/toast' -import { AdminSearchProvider } from './search/AdminSearchContext' -import './styles.css' - -export default function AdminNextRoutes() { - return ( - - - - - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - - - - - ) -} diff --git a/frontend/src/admin-next/pages/LogsNext.tsx b/frontend/src/admin-next/pages/LogsNext.tsx deleted file mode 100644 index eba02776..00000000 --- a/frontend/src/admin-next/pages/LogsNext.tsx +++ /dev/null @@ -1,305 +0,0 @@ -import axios from 'axios' -import { Copy, RefreshCw, Search, X } from 'lucide-react' -import { useEffect, useMemo, useState } from 'react' -import Scrollbar from '../../components/Scrollbar/Scrollbar' -import { useAuthStore } from '../../stores/auth' -import { AdminNextLayout } from '../components/layout/AdminNextLayout' -import { Badge } from '../components/ui/badge' -import { Button } from '../components/ui/button' -import { Input } from '../components/ui/input' -import { Select } from '../components/ui/select' -import { useToast } from '../components/ui/toast' -import { EmptyState, PageFrame, Panel, StatusText } from '../patterns/patterns' - -const LOG_FILTER_STORAGE_KEY = 'planet.admin-next.logs.filters' -const LOG_LIMIT_OPTIONS = [100, 200, 400, 800].map((value) => ({ value: String(value), label: `${value} 行` })) -const LOG_LEVEL_OPTIONS = [ - { value: 'all', label: '全部级别' }, - { value: 'error', label: '错误' }, - { value: 'warning', label: '警告' }, - { value: 'info', label: '信息' }, - { value: 'debug', label: '调试' }, -] - -interface LogSourceSummary { - source_id: string - name: string - kind: string - location: string - description: string - category: string - status: string -} - -interface LogSnapshot { - source_id: string - name: string - kind: string - location: string - description: string - category: string - status: string - level: string - selected_levels: string[] - search_query: string - available_levels: string[] - line_limit: number - line_count: number - lines: string[] -} - -function readStoredFilters() { - if (typeof window === 'undefined') return null - try { - const rawValue = window.localStorage.getItem(LOG_FILTER_STORAGE_KEY) - return rawValue ? JSON.parse(rawValue) as { - selectedSource?: string - lineLimit?: number - level?: string - startDate?: string - endDate?: string - searchQuery?: string - } : null - } catch { - return null - } -} - -function statusTone(status: string) { - if (status === 'ok') return 'success' - if (status === 'missing' || status === 'empty') return 'warning' - if (status.includes('unavailable')) return 'danger' - return 'neutral' -} - -function statusLabel(status: string) { - if (status === 'ok') return '可用' - if (status === 'missing') return '暂无日志' - if (status === 'empty') return '暂无上报' - if (status === 'docker_unavailable') return 'Docker 不可用' - if (status === 'source_unavailable') return '日志源不可用' - return status || '-' -} - -function getErrorMessage(error: unknown, fallback: string) { - if (!axios.isAxiosError(error)) return fallback - const detail = error.response?.data?.detail - return typeof detail === 'string' ? detail : fallback -} - -export default function LogsNext() { - const storedFilters = readStoredFilters() - const { user } = useAuthStore() - const { toast } = useToast() - const isSuperAdmin = user?.role === 'super_admin' - const [sources, setSources] = useState([]) - const [selectedSource, setSelectedSource] = useState(storedFilters?.selectedSource || 'backend') - const [lineLimit, setLineLimit] = useState(storedFilters?.lineLimit || 200) - const [level, setLevel] = useState(storedFilters?.level || 'all') - const [startDate, setStartDate] = useState(storedFilters?.startDate || '') - const [endDate, setEndDate] = useState(storedFilters?.endDate || '') - const [searchQuery, setSearchQuery] = useState(storedFilters?.searchQuery || '') - const [submittedSearch, setSubmittedSearch] = useState(storedFilters?.searchQuery || '') - const [snapshot, setSnapshot] = useState(null) - const [sourcesLoading, setSourcesLoading] = useState(false) - const [logLoading, setLogLoading] = useState(false) - const [errorMessage, setErrorMessage] = useState(null) - - const selectedSourceInfo = useMemo( - () => sources.find((source) => source.source_id === selectedSource) || null, - [selectedSource, sources], - ) - const hasActiveFilters = lineLimit !== 200 || level !== 'all' || Boolean(startDate || endDate || submittedSearch.trim() || searchQuery.trim()) - - const fetchSources = async () => { - if (!isSuperAdmin) return - setSourcesLoading(true) - try { - const response = await axios.get<{ items: LogSourceSummary[] }>('/api/v1/system/logs/sources') - const items = response.data.items || [] - setSources(items) - setErrorMessage(null) - const current = items.find((item) => item.source_id === selectedSource) - if (items.length > 0 && !current) { - setSelectedSource((items.find((item) => item.status === 'ok') || items[0]).source_id) - } - } catch (error) { - setErrorMessage(getErrorMessage(error, '加载日志源失败')) - } finally { - setSourcesLoading(false) - } - } - - const fetchSnapshot = async () => { - if (!isSuperAdmin || !selectedSource) return - setLogLoading(true) - try { - const response = await axios.get(`/api/v1/system/logs/${encodeURIComponent(selectedSource)}`, { - params: { - limit: lineLimit, - level, - levels: level === 'all' ? undefined : level, - start_date: startDate || undefined, - end_date: endDate || undefined, - search: submittedSearch.trim() || undefined, - }, - }) - setSnapshot(response.data) - setErrorMessage(null) - } catch (error) { - setSnapshot(null) - setErrorMessage(getErrorMessage(error, '加载日志内容失败')) - } finally { - setLogLoading(false) - } - } - - useEffect(() => { - void fetchSources() - }, [isSuperAdmin]) - - useEffect(() => { - void fetchSnapshot() - }, [isSuperAdmin, selectedSource, lineLimit, level, startDate, endDate, submittedSearch]) - - useEffect(() => { - const timer = window.setTimeout(() => { - setSubmittedSearch(searchQuery) - }, 250) - return () => window.clearTimeout(timer) - }, [searchQuery]) - - useEffect(() => { - if (typeof window === 'undefined') return - window.localStorage.setItem(LOG_FILTER_STORAGE_KEY, JSON.stringify({ - selectedSource, - lineLimit, - level, - startDate, - endDate, - searchQuery: submittedSearch, - })) - }, [endDate, level, lineLimit, selectedSource, startDate, submittedSearch]) - - const resetFilters = () => { - setLevel('all') - setStartDate('') - setEndDate('') - setSearchQuery('') - setSubmittedSearch('') - setLineLimit(200) - } - - const copyLogs = async () => { - await navigator.clipboard.writeText(snapshot?.lines?.join('\n') || '') - toast({ tone: 'success', title: '日志已复制' }) - } - - if (!isSuperAdmin) { - return ( - - - - - - - - ) - } - - return ( - - - - - - )} - > -
- -
-
-

日志源

-

{sources.length} 个来源

-
-
- - {sources.map((source) => ( - - ))} - {!sources.length && !sourcesLoading ? : null} - - - - -
-
-

{selectedSourceInfo?.name || selectedSource}

-

{selectedSourceInfo?.description || '选择日志源后读取快照。'}

-
-
- {snapshot ? {snapshot.line_count} 行 : null} - -
-
- -
-
- - setStartDate(event.target.value)} aria-label="开始日期" /> - setEndDate(event.target.value)} aria-label="结束日期" /> -
- - setSearchQuery(event.target.value)} - placeholder="搜索日志正文" - /> -
- {hasActiveFilters ? ( - - ) : null} -
- - {errorMessage ?
{errorMessage}
: null} - -
- {logLoading ? ( -
加载中
- ) : snapshot?.lines?.length ? ( - - 
{snapshot.lines.join('\n')}
- - ) : ( - - )} -
-
- -
- - - ) -} diff --git a/frontend/src/admin-next/routes/manifest.tsx b/frontend/src/admin-next/routes/manifest.tsx deleted file mode 100644 index 474d1b9b..00000000 --- a/frontend/src/admin-next/routes/manifest.tsx +++ /dev/null @@ -1,59 +0,0 @@ -import { - AlertTriangle, - AppWindow, - Bot, - CircleGauge, - Database, - FileText, - Globe2, - HardDrive, - Network, - Settings, - ShieldAlert, - Users, - type LucideIcon, -} from 'lucide-react' - -export interface AdminRouteItem { - path: string - label: string - group: string - icon: LucideIcon - keywords: string[] - superAdminOnly?: boolean -} - -export interface AdminRouteGroup { - key: string - label: string - icon: LucideIcon -} - -export const adminRouteGroups: AdminRouteGroup[] = [ - { key: 'overview', label: '总览', icon: CircleGauge }, - { key: 'collection', label: '采集与数据', icon: HardDrive }, - { key: 'observability', label: '专题观测', icon: AppWindow }, - { key: 'alerts', label: '告警与研判', icon: ShieldAlert }, - { key: 'ops', label: '运维与配置', icon: Settings }, -] - -export const adminRoutes: AdminRouteItem[] = [ - { path: '/admin', label: '仪表盘', group: 'overview', icon: CircleGauge, keywords: ['dashboard', '总览', '驾驶舱'] }, - { path: '/earth', label: 'Earth', group: 'overview', icon: Globe2, keywords: ['earth', '地球'] }, - { path: '/datasources', label: '数据源', group: 'collection', icon: Database, keywords: ['datasource', '采集', '目录'] }, - { path: '/data', label: '采集数据', group: 'collection', icon: AppWindow, keywords: ['data', 'records', '采集数据'] }, - { path: '/bgp', label: 'BGP观测', group: 'observability', icon: Network, keywords: ['bgp', '观测', '网络'] }, - { path: '/alerts/system', label: '系统告警', group: 'alerts', icon: AlertTriangle, keywords: ['alert', 'system', '告警'] }, - { path: '/alerts/bgp', label: 'BGP 告警', group: 'alerts', icon: Network, keywords: ['alert', 'bgp', '风险'] }, - { path: '/alerts/situational', label: '态势告警', group: 'alerts', icon: Globe2, keywords: ['situational', '态势', '研判'] }, - { path: '/ai', label: 'AI', group: 'ops', icon: Bot, keywords: ['ai', 'provider', 'playground', 'prompt'] }, - { path: '/earth-content', label: 'Earth 内容', group: 'ops', icon: Globe2, keywords: ['earth', 'tv', 'boundary', 'brand'] }, - { path: '/collection-management', label: '采集管理', group: 'ops', icon: Database, keywords: ['collector', 'mapping', 'custom source'] }, - { path: '/logs', label: '系统日志', group: 'ops', icon: FileText, keywords: ['log', '日志', 'tail'], superAdminOnly: true }, - { path: '/users', label: '用户管理', group: 'ops', icon: Users, keywords: ['users', 'role', 'gatekeeper'] }, - { path: '/settings', label: '系统设置', group: 'ops', icon: Settings, keywords: ['settings', 'smtp', 'security'] }, -] - -export function getVisibleAdminRoutes(isSuperAdmin: boolean) { - return adminRoutes.filter((route) => !route.superAdminOnly || isSuperAdmin) -} diff --git a/frontend/src/admin/AdminRoutes.tsx b/frontend/src/admin/AdminRoutes.tsx new file mode 100644 index 00000000..67b8abe6 --- /dev/null +++ b/frontend/src/admin/AdminRoutes.tsx @@ -0,0 +1,48 @@ +import { Navigate, Route, Routes } from 'react-router-dom' +import { AdminThemeProvider } from './design/theme' +import Dashboard from './pages/Dashboard' +import DataList from './pages/DataList' +import { + AI, + BGP, + BGPAlerts, + CollectionManagement, + DataSources, + EarthContent, + Settings, + SituationalAlerts, + SystemAlerts, +} from './pages/PlainResourcePages' +import Users from './pages/Users' +import Logs from './pages/Logs' +import { ToastProvider } from './components/ui/toast' +import { AdminSearchProvider } from './search/AdminSearchContext' +import './styles.css' + +export default function AdminRoutes() { + return ( + + + + + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + + + + + ) +} diff --git a/frontend/src/admin/components/AdminErrorBoundary.tsx b/frontend/src/admin/components/AdminErrorBoundary.tsx new file mode 100644 index 00000000..fe6909ba --- /dev/null +++ b/frontend/src/admin/components/AdminErrorBoundary.tsx @@ -0,0 +1,39 @@ +import { Component, type ErrorInfo, type ReactNode } from 'react' +import { reportAdminRuntimeLog } from '../runtimeLogs' + +type AdminErrorBoundaryProps = { + children: ReactNode +} + +type AdminErrorBoundaryState = { + hasError: boolean +} + +export class AdminErrorBoundary extends Component { + state: AdminErrorBoundaryState = { hasError: false } + + static getDerivedStateFromError() { + return { hasError: true } + } + + componentDidCatch(error: Error, errorInfo: ErrorInfo) { + void reportAdminRuntimeLog({ + level: 'error', + category: 'react-error-boundary', + module: 'admin', + message: error.message || '控制台渲染错误', + detail: `${error.stack || error.message}\n${errorInfo.componentStack || ''}`, + }) + } + + render() { + if (this.state.hasError) { + return ( +
+
控制台发生错误，请刷新页面重试。
+
+ ) + } + return this.props.children + } +} diff --git a/frontend/src/admin-next/components/data-table/DataTable.tsx b/frontend/src/admin/components/data-table/DataTable.tsx similarity index 93% rename from frontend/src/admin-next/components/data-table/DataTable.tsx rename to frontend/src/admin/components/data-table/DataTable.tsx index bafbb036..12f374bb 100644 --- a/frontend/src/admin-next/components/data-table/DataTable.tsx +++ b/frontend/src/admin/components/data-table/DataTable.tsx @@ -8,6 +8,7 @@ import { } from '@tanstack/react-table' import { ArrowDown, ArrowUp, ChevronsUpDown } from 'lucide-react' import { useMemo, useState } from 'react' +import { useTranslation } from 'react-i18next' import TableScrollRegion from '../../../components/Scrollbar/TableScrollRegion' import { Button } from '../ui/button' @@ -37,11 +38,12 @@ export function DataTable({ getRowClassName, selection, loading = false, - emptyText = '暂无数据', + emptyText, className = '', footer, onRowClick, }: DataTableProps) { + const { t } = useTranslation() const [sorting, setSorting] = useState([]) const memoizedColumns = useMemo(() => columns, [columns]) @@ -74,7 +76,7 @@ export function DataTable({ { @@ -114,7 +116,7 @@ export function DataTable({
- 加载中 + {t('common.loading')}
@@ -125,12 +127,13 @@ export function DataTable({ className={getRowClassName?.(row.original)} onClick={onRowClick ? () => onRowClick(row.original) : undefined} data-clickable={onRowClick ? 'true' : undefined} + data-row-id={row.id} > {selection ? ( event.stopPropagation()} @@ -148,7 +151,7 @@ export function DataTable({ ) : ( -
{emptyText}
+
{emptyText || t('common.noData')}
)} @@ -172,18 +175,19 @@ export function DataTablePager({ total: number onPageChange: (page: number) => void }) { + const { t } = useTranslation() const totalPages = Math.max(1, Math.ceil(total / pageSize)) return (
- 第 {page} / {totalPages} 页，共 {total.toLocaleString()} 条 + {t('common.page', { page, totalPages, total: total.toLocaleString() })}
diff --git a/frontend/src/admin-next/components/layout/AdminNextLayout.tsx b/frontend/src/admin/components/layout/AdminLayout.tsx similarity index 57% rename from frontend/src/admin-next/components/layout/AdminNextLayout.tsx rename to frontend/src/admin/components/layout/AdminLayout.tsx index 610aa698..d8aada2b 100644 --- a/frontend/src/admin-next/components/layout/AdminNextLayout.tsx +++ b/frontend/src/admin/components/layout/AdminLayout.tsx @@ -1,21 +1,25 @@ import { ChevronDown, + Languages, LogOut, Menu, Moon, Monitor, Search, + Settings, Sun, X, } from 'lucide-react' import { type FocusEvent, type KeyboardEvent, type ReactNode, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react' +import { useTranslation } from 'react-i18next' import { Link, useLocation, useNavigate } from 'react-router-dom' import packageJson from '../../../../package.json' import Scrollbar from '../../../components/Scrollbar/Scrollbar' import SegmentedControl from '../../../components/SegmentedControl/SegmentedControl' +import { localeOptions, useLocale, type SupportedLocale } from '../../../i18n/locale' import { useAuthStore } from '../../../stores/auth' import { useAdminTheme, type AdminThemeMode } from '../../design/theme' -import { cn } from '../../lib/utils' +import { cn } from '../../utils' import { adminRouteGroups, getVisibleAdminRoutes } from '../../routes/manifest' import { useAdminSearch } from '../../search/AdminSearchContext' import { Button } from '../ui/button' @@ -24,10 +28,12 @@ const DEFAULT_OPEN_MENU_KEY = 'collection' let cachedOpenKeys: string[] = [DEFAULT_OPEN_MENU_KEY] let cachedMenuScrollTop = 0 -export function AdminNextLayout({ children }: { children: ReactNode }) { +export function AdminLayout({ children }: { children: ReactNode }) { const location = useLocation() const navigate = useNavigate() const adminSearch = useAdminSearch() + const { t } = useTranslation() + const { locale, setLocale } = useLocale() const { user, logout } = useAuthStore() const { mode, setMode } = useAdminTheme() const [collapsed, setCollapsed] = useState(false) @@ -35,25 +41,39 @@ export function AdminNextLayout({ children }: { children: ReactNode }) { const [openKeys, setOpenKeys] = useState(cachedOpenKeys) const [searchQuery, setSearchQuery] = useState('') const [searchOpen, setSearchOpen] = useState(false) + const [preferencesOpen, setPreferencesOpen] = useState(false) const [highlightedSearchIndex, setHighlightedSearchIndex] = useState(0) const menuViewportRef = useRef(null) const searchInputRef = useRef(null) const isSuperAdmin = user?.role === 'super_admin' + const username = user?.username || '-' + const userInitial = username.trim().charAt(0).toUpperCase() || '?' + const preferencesLabel = preferencesOpen ? t('admin.collapsePreferences') : t('admin.expandPreferences') const visibleRoutes = useMemo(() => getVisibleAdminRoutes(isSuperAdmin), [isSuperAdmin]) const navGroups = useMemo(() => { return adminRouteGroups.map((group) => ({ ...group, - children: visibleRoutes.filter((route) => route.group === group.key), + label: t(group.labelKey), + children: visibleRoutes + .filter((route) => route.group === group.key) + .map((route) => ({ ...route, label: t(route.labelKey) })), })).filter((group) => group.children.length > 0) - }, [visibleRoutes]) + }, [t, visibleRoutes]) const selectedKey = location.pathname === '/admin/' ? '/admin' : location.pathname.replace(/\/$/, '') const activeRoute = visibleRoutes.find((route) => route.path === selectedKey) + const activeRouteLabel = activeRoute ? t(activeRoute.labelKey) : '' const searchResults = useMemo(() => adminSearch.search(searchQuery), [adminSearch, searchQuery]) const themeOptions = useMemo(() => [ - { value: 'light' as const, label: '浅色', title: '浅色', icon: }, - { value: 'system' as const, label: '系统', title: '跟随系统', icon: }, - { value: 'dark' as const, label: '深色', title: '深色', icon: }, - ], []) + { value: 'light' as const, label: t('common.themeLight'), title: t('common.themeLight'), icon: }, + { value: 'system' as const, label: t('common.themeSystem'), title: t('common.themeFollowSystem'), icon: }, + { value: 'dark' as const, label: t('common.themeDark'), title: t('common.themeDark'), icon: }, + ], [t]) + const languageOptions = useMemo(() => localeOptions.map((option) => ({ + value: option.value, + label: t(option.labelKey), + title: t(option.titleKey), + icon: , + })), [t]) const updateOpenKeys = (nextKeys: string[]) => { cachedOpenKeys = nextKeys @@ -107,37 +127,38 @@ export function AdminNextLayout({ children }: { children: ReactNode }) { const nav = ( <> -
setCollapsed(false) : undefined}> +
setCollapsed(false) : undefined}> {!collapsed ? ( -
- Planet - Admin Next +
+ {t('admin.brandTitle')} + {t('admin.brandSubtitle')}
) : null}
- -