forked from linkong/planet
Compare commits
29 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c1d651589a | ||
|
|
36be9e3da6 | ||
|
|
7d8822d34d | ||
|
|
ae1a4b3942 | ||
|
|
35e478642b | ||
|
|
33734b08b8 | ||
|
|
bbe47c5759 | ||
|
|
361bf258e4 | ||
|
|
83c552f9b2 | ||
|
|
15fed1e121 | ||
|
|
f4514b588e | ||
|
|
9523f2bb83 | ||
|
|
9dc3b61fa5 | ||
|
|
698863da68 | ||
|
|
4c0b49962c | ||
|
|
ffb3134d93 | ||
|
|
e8742b5d3c | ||
|
|
28a7132eec | ||
|
|
dd41588527 | ||
|
|
45604a1ebb | ||
|
|
06c122bade | ||
|
|
96e9b15fda | ||
|
|
a8713e58fd | ||
|
|
05377bf5c6 | ||
|
|
67548d8278 | ||
|
|
4b8feeb87e | ||
|
|
a2025dbfe0 | ||
|
|
003b316d42 | ||
|
|
0c40764c58 |
@@ -19,9 +19,9 @@ Do not refactor architecture, add features, or change behavior.
|
||||
|
||||
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
|
||||
|
||||
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
|
||||
Only report and fix issues present in **newly added or modified** lines of this diff — do not audit unchanged code, and do not rewrite unrelated files during cleanup.
|
||||
|
||||
## Token-Saving Rule
|
||||
## Token-Saving Rule & Harness Checks
|
||||
|
||||
Prefer deterministic CLI checks before reading files into model context:
|
||||
|
||||
@@ -29,27 +29,36 @@ Prefer deterministic CLI checks before reading files into model context:
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
Use the repository's focused harness checks for the touched surfaces:
|
||||
|
||||
- Backend Python changes: `scripts/harness/backend-rules-check.sh` (catches direct `print()`, `breakpoint()`, and debugger calls)
|
||||
- Frontend changes: `scripts/harness/frontend-rules-check.sh` (catches console logs, missing button types, Card nesting, layout height-chain issues)
|
||||
- Secrets / credentials: `scripts/harness/security-check.sh`
|
||||
|
||||
Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Duplicate Logic
|
||||
- Identical or near-identical code blocks appearing in multiple places
|
||||
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
|
||||
- Repeated DOM queries, regex literals, or template strings within the same file
|
||||
- Identical or near-identical code blocks appearing in multiple places within the modified scope
|
||||
- A helper that already exists but is re-implemented elsewhere instead of being reused
|
||||
- Repeated complex queries or template strings that cause actual maintenance hazard
|
||||
- **Anti-Over-Defensiveness**: Keep one-off logic at its call site. Do not prematurely abstract single-use or 2-use trivial helpers/templates that are clearer inline.
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
|
||||
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
|
||||
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
|
||||
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) where the intent is completely opaque and naming adds clear value
|
||||
- Hardcoded domain strings (IDs, status values, protocol fragments) scattered through logic
|
||||
- **Anti-Over-Defensiveness (AGENTS.md Quality Bar)**:
|
||||
- Idiomatically clear values such as `0`, `1`, `-1`, `100`, `""` are fine inline.
|
||||
- **Do NOT over-extract local single-use layout offsets, Three.js coordinates, padding, or math coefficients into module-level globals.**
|
||||
- Keep one-off logic at its call site unless extraction clarifies a complex operation or lets callers share the same behavior.
|
||||
|
||||
### 3. Naming Issues
|
||||
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
|
||||
- Names that do not match actual behavior
|
||||
- The same concept referred to by different names in different places
|
||||
- The same concept referred to by different names in different places within the changed scope
|
||||
|
||||
### 4. Dead Code
|
||||
- Commented-out code blocks (3+ lines)
|
||||
@@ -58,12 +67,12 @@ Read full files only when the focused diff does not provide enough surrounding c
|
||||
|
||||
### 5. Style Inconsistencies
|
||||
- Trailing whitespace
|
||||
- Mixed quote styles or indentation within the same file
|
||||
- Mixed quote styles or indentation within the same file (Python: 4 spaces Black style; TypeScript: 2 spaces Prettier style)
|
||||
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
|
||||
|
||||
### 6. Other
|
||||
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
|
||||
- Overly verbose conditions that can be simplified without changing logic
|
||||
- Keep fixes strictly scoped to the modified files; do not rewrite external callers or expand cleanup beyond the changed scope.
|
||||
|
||||
## Steps
|
||||
|
||||
@@ -107,30 +116,39 @@ If no issues are found, output "No code smells detected. Code quality looks good
|
||||
|
||||
Use the Edit tool for **minimal, targeted changes**:
|
||||
|
||||
- **Duplicate logic**: extract to a shared constant or function; update all call sites
|
||||
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
|
||||
- **Naming**: rename the variable/function; update all references
|
||||
- **Duplicate logic**: extract to a shared constant or helper within the owned file or component scope; update call sites strictly within scope
|
||||
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope only when the value is reused across places or its meaning is genuinely opaque; replace usages. Keep one-off UI/math offsets inline per `AGENTS.md`
|
||||
- **Naming**: rename the variable/function; update all references within the modified scope
|
||||
- **Dead code**: delete it
|
||||
- **Trailing whitespace / style**: fix in place
|
||||
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
|
||||
- **Unexported helper**: add `export` only when requested; do not modify unrelated callers
|
||||
|
||||
Principles:
|
||||
- Only fix issues identified in the checklist — no extra improvements
|
||||
- Only fix issues identified in the checklist — no extra improvements or speculative refactorings
|
||||
- Keep each Edit as small as possible
|
||||
- After fixing, verify the old bad pattern is gone with grep
|
||||
- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
|
||||
- After fixing, verify the old bad pattern is gone with grep or deterministic harness checks
|
||||
- Prefer `apply_patch` or targeted edit tools; use formatters only when the repository already uses them for the touched file type
|
||||
|
||||
### Step 5 — Summary
|
||||
### Step 5 — Verify and summarize
|
||||
|
||||
Run the smallest relevant harness checks for the modified surfaces:
|
||||
- Python files: `uv run python -m py_compile <changed_py_files>` and `scripts/harness/backend-rules-check.sh`
|
||||
- Frontend files: `scripts/harness/frontend-rules-check.sh` (and `bun run --cwd frontend build` if imports/types were adjusted)
|
||||
- Formatting / whitespace: `git diff --check`
|
||||
|
||||
```
|
||||
Cleanup complete:
|
||||
|
||||
Fixed N issues:
|
||||
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
|
||||
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
|
||||
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
|
||||
✓ main.js — clarified variable naming in news event payload handler
|
||||
✓ controls.js — removed unused dead import
|
||||
...
|
||||
|
||||
Validation:
|
||||
✓ git diff --check passed
|
||||
✓ scripts/harness/frontend-rules-check.sh passed
|
||||
|
||||
Skipped (needs manual review):
|
||||
! foo.js L45 — large commented-out block; confirm it is safe to delete
|
||||
```
|
||||
@@ -141,4 +159,6 @@ Skipped (needs manual review):
|
||||
- **Do not** add new features, abstractions, or parameters
|
||||
- **Do not** rewrite comments (only delete commented-out dead code)
|
||||
- **Do not** touch test file logic
|
||||
- **Do not** rewrite unrelated files during cleanup (per `AGENTS.md` Prohibited)
|
||||
- **Do not** over-extract local single-use coordinates, layout numbers, or inline math coefficients into global constants (preventing defensive over-engineering)
|
||||
- If a magic number's intent is uncertain, skip it and flag it in the summary
|
||||
|
||||
@@ -48,18 +48,22 @@ rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
|
||||
3. Write the doc:
|
||||
|
||||
- Explain background/problem, design decisions, constraints, and operational impact.
|
||||
- Include the context, rationale, or operational impact the reader needs for this change.
|
||||
- Keep code snippets short and directly relevant.
|
||||
- List related files only when they help future maintainers navigate.
|
||||
- Use the repository’s existing language, heading style, and naming conventions.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. Only update the glossary when introducing new, ambiguous, or cross-language public concepts; avoid unnecessary glossary churn for ordinary bug fixes or internal refactors.
|
||||
|
||||
4. Verify:
|
||||
|
||||
- Read the completed doc once for clarity and stale statements.
|
||||
- Verify important referenced paths exist with `test -e` or `rg --files`.
|
||||
- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
|
||||
- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
|
||||
- Run the repository's centralized documentation harness gate:
|
||||
```bash
|
||||
scripts/harness/docs-consistency-check.sh
|
||||
```
|
||||
This automatically verifies bilingual pairs, public registry whitelist, link anchors, language-scoped links, section deep-links, and stale admin route terms in a single pass.
|
||||
- Run any additional domain-specific checks from `docs/documentation-coverage-rules.md` when applicable.
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
|
||||
@@ -19,9 +19,11 @@ Do not use this skill for ordinary commits that are not being released.
|
||||
## Versioning Rules
|
||||
|
||||
- `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
|
||||
- `bugfix` -> bump `+0.0.1`
|
||||
- `improvement` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
|
||||
- `bugfix` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
|
||||
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
|
||||
|
||||
Mixed bugfix and small feature/UI work defaults to `improvement` unless the user explicitly chooses another release type.
|
||||
When intent is mixed, prefer the user's stated release intent.
|
||||
|
||||
## Required Files
|
||||
@@ -53,28 +55,32 @@ Do not inspect full diffs unless deciding whether changed code belongs in the re
|
||||
### Step 1 — Environment check
|
||||
|
||||
```bash
|
||||
git branch --show-current # must be on dev
|
||||
git branch --show-current # check current branch
|
||||
git status --short # check for unrelated uncommitted changes
|
||||
cat VERSION # read current version
|
||||
```
|
||||
|
||||
If not on `dev`, stop and tell the user. Do not proceed.
|
||||
Default release branch is `dev`. If not on `dev` (e.g. on a dedicated maintenance or release branch), report the active branch and confirm before proceeding, rather than hard-failing.
|
||||
|
||||
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
|
||||
|
||||
### Step 2 — Determine release type and next version
|
||||
|
||||
- If the user provided an explicit type (`feature` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
|
||||
- If the user provided an explicit type (`feature` / `improvement` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`:
|
||||
- New features or major capabilities -> `feature`
|
||||
- Mixed bugfix and small feature/UI work -> default to `improvement` (per `rules.md`)
|
||||
- Bugfixes only -> `bugfix`
|
||||
- Compute the next version:
|
||||
- `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
|
||||
- `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
|
||||
- `improvement`: increment patch only (e.g. `0.76.18` → `0.76.19`)
|
||||
- `bugfix`: increment patch only (e.g. `0.76.18` → `0.76.19`)
|
||||
- **Show the release plan before making any changes:**
|
||||
|
||||
```
|
||||
Release plan:
|
||||
Type: bugfix
|
||||
Version: 0.26.2 → 0.26.3
|
||||
Type: improvement
|
||||
Version: 0.76.18 → 0.76.19
|
||||
Branch: dev
|
||||
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
@@ -117,17 +123,25 @@ Get today's date with `date +%Y-%m-%d`.
|
||||
|
||||
### Step 6 — Validate
|
||||
|
||||
Run the smallest relevant validation for the changes in scope:
|
||||
Run the smallest relevant harness validation for the changes in scope:
|
||||
|
||||
- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
|
||||
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
|
||||
- **Version consistency**: confirm `VERSION`, `frontend/package.json`, `pyproject.toml`, and `uv.lock` all show the identical target version:
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
- **Security scan**: run `scripts/harness/security-check.sh`
|
||||
- **Python files changed**:
|
||||
- Compile: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `uv run python -m py_compile <changed_files>`
|
||||
- Service rules: run `scripts/harness/backend-rules-check.sh`
|
||||
- **Frontend files changed**:
|
||||
- Rules: run `scripts/harness/frontend-rules-check.sh`
|
||||
- Build: run `bun run --cwd frontend build` (Bun only; never skip)
|
||||
- **Docs changed**: run `scripts/harness/docs-consistency-check.sh`
|
||||
- **Broad / cross-subsystem changes**: run `scripts/harness/quick-check.sh`
|
||||
|
||||
### Step 7 — Pre-commit preview
|
||||
|
||||
Show what will be committed:
|
||||
@@ -141,29 +155,36 @@ Confirm all required files are present and no unexpected files (debug files, `.e
|
||||
### Step 8 — Commit, tag, and push
|
||||
|
||||
```bash
|
||||
branch=$(git branch --show-current)
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# also stage any code changes included in this release
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
Commit message format is fixed: `release: bump version to x.x.x`
|
||||
|
||||
Push to remote:
|
||||
- When remote publication is requested:
|
||||
```bash
|
||||
git push origin "$branch"
|
||||
git push origin vx.x.x
|
||||
```
|
||||
- If remote publication was not requested or remote access is restricted, report the commit and tag as complete and output the push commands for manual execution.
|
||||
|
||||
### Step 9 — Completion summary
|
||||
|
||||
```
|
||||
✓ Version bumped: 0.26.2 → 0.26.3
|
||||
✓ Version bumped: 0.76.18 → 0.76.19
|
||||
✓ CHANGELOG updated
|
||||
✓ version-history updated
|
||||
✓ uv.lock regenerated
|
||||
✓ Validation passed
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ Pushed to origin/dev
|
||||
✓ commit: release: bump version to 0.76.19
|
||||
✓ tag: v0.76.19
|
||||
✓ Pushed to origin/dev (or reported pending manual push)
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -3,6 +3,12 @@
|
||||
!pyproject.toml
|
||||
!uv.lock
|
||||
!VERSION
|
||||
!frontend/
|
||||
!frontend/public/
|
||||
!frontend/public/earth/
|
||||
!frontend/public/earth/assets/
|
||||
!frontend/public/earth/assets/cards/
|
||||
!frontend/public/earth/assets/cards/manifest.json
|
||||
!backend/
|
||||
!backend/**
|
||||
!config/
|
||||
|
||||
30
AGENTS.md
30
AGENTS.md
@@ -55,13 +55,19 @@ Preserve user changes already present in the worktree.
|
||||
|
||||
### Validation
|
||||
|
||||
Fast local harness validation:
|
||||
For an isolated change, run the smallest check that exercises the affected
|
||||
behavior, along with any gates required for that area. For example, run the
|
||||
docs consistency check for a link change, or reproduce a bug at the entry
|
||||
point where it occurred.
|
||||
|
||||
Use the quick suite when a change affects shared harness wiring or needs
|
||||
regression checks across areas:
|
||||
|
||||
```bash
|
||||
scripts/harness/quick-check.sh
|
||||
```
|
||||
|
||||
Full local validation:
|
||||
Use full validation for larger cross-subsystem changes:
|
||||
|
||||
```bash
|
||||
scripts/harness/validate.sh
|
||||
@@ -113,6 +119,9 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
## Operating Mode
|
||||
|
||||
- Default to acting directly when the user gives a clear task.
|
||||
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
|
||||
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
|
||||
and Astra for complex reasoning; the main agent owns final verification.
|
||||
- Ask before acting only when the missing decision is risky, cannot be
|
||||
discovered from repository context, and no conservative assumption is safe.
|
||||
- Read relevant files before editing.
|
||||
@@ -120,6 +129,8 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
focused file reads, tests, builds, linters, and harness scripts.
|
||||
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
|
||||
release work, and documentation unless the task requires it.
|
||||
- When the user corrects a detail, apply it to the original request. Carry the
|
||||
correction into the result without turning discarded ideas into new requirements.
|
||||
|
||||
---
|
||||
|
||||
@@ -141,6 +152,12 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
visual-dependent part of the work.
|
||||
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
|
||||
diff, direct file check, or harness result.
|
||||
- Verify the outcome through the entry point the user relies on. A source check
|
||||
can confirm a code convention. A UI test with a mocked API can confirm what
|
||||
the user sees for that response.
|
||||
- When a test fails, identify the behavior or invariant it protects. Fix the
|
||||
implementation if that regressed. If the behavior is correct and the assertion
|
||||
pins an obsolete detail, update the test to exercise the behavior.
|
||||
- For UI and rendering changes, verify the rendered result when local tooling
|
||||
allows it.
|
||||
|
||||
@@ -159,7 +176,14 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Prefer existing project patterns over new abstractions.
|
||||
- Prefer existing project patterns and the smallest change that satisfies the
|
||||
request. Keep one-off logic at its call site unless extraction clarifies a
|
||||
complex operation or lets callers share the same behavior.
|
||||
- Validate and normalize external data at the boundary that owns it. Recheck a
|
||||
guarantee when data crosses a new trust boundary or an operation can invalidate it.
|
||||
- Before adding a retry, lock, or fallback, identify the failure it addresses
|
||||
from a requirement, an existing contract, or observed behavior. Catch errors
|
||||
where the caller can recover, translate them, or add context and propagate them.
|
||||
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
|
||||
a stable path exists.
|
||||
- Centralize prompts, constants, defaults, and shared request/response handling.
|
||||
|
||||
@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
|
||||
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
|
||||
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
|
||||
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
|
||||
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
|
||||
|
||||
## Runtime Entry Points
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_
|
||||
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
|
||||
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
|
||||
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
|
||||
COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
|
||||
COPY VERSION /app/VERSION
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
from app.api.v1 import card_images
|
||||
from fastapi import APIRouter
|
||||
from app.api.v1 import (
|
||||
ai,
|
||||
@@ -58,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
|
||||
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
|
||||
|
||||
api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])
|
||||
|
||||
108
backend/app/api/v1/card_images.py
Normal file
108
backend/app/api/v1/card_images.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""Card-image configuration, upload, catalog selection and public resolution."""
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.card_images import (
|
||||
CardImageConfiguration,
|
||||
CardImagePreview,
|
||||
CardImagePublish,
|
||||
EntityType,
|
||||
MatchScope,
|
||||
)
|
||||
from app.services.card_image_configuration import (
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
resolve_image,
|
||||
)
|
||||
from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
|
||||
from app.services.card_image_targets import get_target, search_targets
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
async def image_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return await read_configuration(db)
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def save_configuration(
|
||||
payload: CardImagePublish,
|
||||
actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await publish_configuration(db, payload.revision, payload.config, actor)
|
||||
|
||||
|
||||
@router.post("/upload")
|
||||
async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
|
||||
content = await file.read(MAX_UPLOAD_BYTES + 1)
|
||||
await file.close()
|
||||
return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
|
||||
|
||||
|
||||
@router.get("/files/{filename}")
|
||||
async def image_file(filename: str):
|
||||
path = uploaded_path(filename)
|
||||
if not path.is_file():
|
||||
raise HTTPException(404, "Image not found")
|
||||
return FileResponse(
|
||||
path,
|
||||
headers={
|
||||
"Cache-Control": "public, max-age=31536000, immutable",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/targets")
|
||||
async def targets(
|
||||
entity_type: EntityType,
|
||||
scope: MatchScope = "object",
|
||||
q: str = Query(default="", max_length=100),
|
||||
limit: int = Query(default=25, ge=1, le=50),
|
||||
_actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
|
||||
|
||||
|
||||
@router.get("/resolve")
|
||||
async def resolve(
|
||||
response: Response,
|
||||
entity_type: EntityType,
|
||||
entity_id: str = Query(min_length=1, max_length=300),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
envelope = await read_configuration(db)
|
||||
config = CardImageConfiguration.model_validate(envelope["config"])
|
||||
context = await get_target(db, entity_type, entity_id)
|
||||
result = resolve_image(config, context)
|
||||
return {**result, "revision": envelope["revision"], "entity": context}
|
||||
|
||||
|
||||
@router.post("/preview")
|
||||
async def preview(
|
||||
payload: CardImagePreview,
|
||||
_actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
context = await get_target(db, payload.entity_type, payload.entity_id)
|
||||
return {**resolve_image(payload.config, context), "entity": context}
|
||||
@@ -1,28 +1,41 @@
|
||||
"""Read-only compute topic: independent macro observations and facility samples."""
|
||||
"""Compute observations, administrator configuration and independent facility samples."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.core.earth_geography_policy import display_country
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.services.compute_center_locations import refresh_compute_center_location_cache
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import (
|
||||
ComputeConfiguration,
|
||||
ComputeConfigurationReset,
|
||||
ComputeConfigurationWrite,
|
||||
)
|
||||
from app.services.compute_catalog import (
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
from app.services.compute_center_locations import refresh_compute_center_location_cache
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
configuration_response,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
router = APIRouter()
|
||||
SOURCES = ("top500", "epoch_ai_gpu")
|
||||
|
||||
|
||||
def selected_countries(countries: str) -> list[str]:
|
||||
@@ -38,10 +51,60 @@ def selected_countries(countries: str) -> list[str]:
|
||||
|
||||
@router.get("/series")
|
||||
async def get_series(
|
||||
response: Response,
|
||||
countries: str = "CN,US",
|
||||
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return compute_history(selected_countries(countries), mode)
|
||||
codes = selected_countries(countries)
|
||||
envelope = await read_configuration(db)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
|
||||
result.update(
|
||||
version=envelope["revision"],
|
||||
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response,
|
||||
_actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return configuration_response(await read_configuration(db))
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def update_configuration(
|
||||
payload: ComputeConfigurationWrite,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=payload.config, actor=actor,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
@router.post("/configuration/reset")
|
||||
async def reset_configuration(
|
||||
payload: ComputeConfigurationReset,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
|
||||
@@ -58,20 +121,24 @@ async def sample_query(db: AsyncSession, countries: list[str], year: int | None)
|
||||
.label("rank"),
|
||||
)
|
||||
.where(
|
||||
DataSnapshot.source.in_(SOURCES),
|
||||
DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
|
||||
DataSnapshot.status == "success",
|
||||
DataSnapshot.completed_at < cutoff,
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
|
||||
infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
|
||||
# Old collector rows have ambiguous units/identity and must not become
|
||||
# national evidence merely because a new UI can display them.
|
||||
query = select(CollectedData).where(
|
||||
compute_record_filter(CollectedData),
|
||||
CollectedData.deleted_at.is_(None),
|
||||
CollectedData.snapshot_id.in_(snapshots),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.extra_data["compute_contract"].as_string() == "v1",
|
||||
or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
|
||||
or_(
|
||||
infrastructure,
|
||||
CollectedData.source == "top500",
|
||||
and_(
|
||||
CollectedData.extra_data["status"].as_string() == "Existing",
|
||||
|
||||
@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
|
||||
"name": "算力",
|
||||
"sources": ["top500", "epoch_ai_gpu"],
|
||||
"primary_stat_key": "compute_center_count",
|
||||
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
|
||||
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from sqlalchemy import delete, func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings as app_settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.security import decode_token, get_current_user, redis_client
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
@@ -51,6 +52,7 @@ from app.services.earth_boundaries import (
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__)
|
||||
optional_bearer = HTTPBearer(auto_error=False)
|
||||
REPO_ROOT = Path(__file__).resolve().parents[4]
|
||||
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
|
||||
@@ -382,13 +384,28 @@ async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
|
||||
return await get_earth_news_sources_payload(db)
|
||||
|
||||
|
||||
def _schedule_earth_news_source_refresh() -> None:
|
||||
try:
|
||||
from app.services.scheduler import schedule_earth_news_refresh_for_all_regions
|
||||
|
||||
schedule_earth_news_refresh_for_all_regions(force=True)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Failed to schedule Earth news refresh after source configuration change",
|
||||
event="earth_news.rss.config_refresh_schedule_failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
@router.put("/news-sources")
|
||||
async def update_earth_news_sources(
|
||||
payload: EarthNewsSourcesPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await save_earth_news_sources_payload(db, payload.model_dump())
|
||||
result = await save_earth_news_sources_payload(db, payload.model_dump())
|
||||
_schedule_earth_news_source_refresh()
|
||||
return result
|
||||
|
||||
|
||||
@router.delete("/news-sources")
|
||||
@@ -397,7 +414,9 @@ async def reset_earth_news_sources(
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await reset_earth_news_sources_payload(db)
|
||||
result = await reset_earth_news_sources_payload(db)
|
||||
_schedule_earth_news_source_refresh()
|
||||
return result
|
||||
|
||||
|
||||
@router.post("/news-sources/test")
|
||||
|
||||
@@ -1,17 +1,26 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from uuid import UUID
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.earth_news import (
|
||||
ALLOWED_NEWS_CATEGORY_KEYS,
|
||||
SUPPORTED_NEWS_LOCALES,
|
||||
REGION_ANCHORS,
|
||||
determine_focus_region,
|
||||
get_earth_news_payload,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class EarthNewsRefreshRequest(BaseModel):
|
||||
lat: float | None = Field(default=None, ge=-90, le=90)
|
||||
lon: float | None = Field(default=None, ge=-180, le=180)
|
||||
region: str | None = Field(default=None, max_length=64)
|
||||
|
||||
|
||||
def _parse_categories(raw: str | None) -> set[str] | None:
|
||||
if raw is None or not raw.strip():
|
||||
return None
|
||||
@@ -39,7 +48,9 @@ def _parse_limit(raw: int | None) -> int:
|
||||
if raw is None:
|
||||
return 12
|
||||
if raw < 1:
|
||||
raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
|
||||
raise HTTPException(
|
||||
status_code=422, detail={"message": "News limit must be greater than 0."}
|
||||
)
|
||||
return min(raw, 100)
|
||||
|
||||
|
||||
@@ -59,18 +70,76 @@ def _parse_locale(raw: str | None) -> str:
|
||||
return requested
|
||||
|
||||
|
||||
@router.post("/refresh", status_code=status.HTTP_202_ACCEPTED)
|
||||
async def request_earth_news_refresh(payload: EarthNewsRefreshRequest):
|
||||
requested_region = payload.region.strip().lower() if payload.region else None
|
||||
if requested_region and requested_region not in REGION_ANCHORS:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={
|
||||
"message": "Unsupported news region.",
|
||||
"invalid_region": requested_region,
|
||||
"allowed_regions": list(REGION_ANCHORS.keys()),
|
||||
},
|
||||
)
|
||||
region = requested_region or determine_focus_region(payload.lat, payload.lon)
|
||||
from app.services.scheduler import request_earth_news_region_refresh
|
||||
|
||||
try:
|
||||
return request_earth_news_region_refresh(region)
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail={
|
||||
"code": "refresh_failed",
|
||||
"message": "Could not start a news refresh. Please try again.",
|
||||
},
|
||||
) from exc
|
||||
|
||||
|
||||
@router.get("/refresh/{refresh_id}")
|
||||
async def get_earth_news_refresh(refresh_id: UUID):
|
||||
from app.services.earth_news_refresh import get_earth_news_refresh_status
|
||||
|
||||
try:
|
||||
result = get_earth_news_refresh_status(refresh_id.hex)
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail={
|
||||
"code": "refresh_failed",
|
||||
"message": "Refresh status is temporarily unavailable. Please retry.",
|
||||
},
|
||||
) from exc
|
||||
if result is not None:
|
||||
return result
|
||||
return {
|
||||
"refresh_id": refresh_id.hex,
|
||||
"region": None,
|
||||
"status": "failed",
|
||||
"error": {
|
||||
"code": "expired",
|
||||
"message": "Refresh status expired. Submit a new refresh.",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
|
||||
region: str | None = Query(
|
||||
None, description="Explicit Earth news region for UE/client integrations"
|
||||
),
|
||||
categories: str | None = Query(None, description="Comma-separated news category keys"),
|
||||
sources: str | None = Query(None, description="Comma-separated news source ids"),
|
||||
limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
|
||||
locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
|
||||
normalized_region = (
|
||||
region.strip().lower() if isinstance(region, str) and region.strip() else None
|
||||
)
|
||||
if normalized_region is not None and normalized_region not in REGION_ANCHORS:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
|
||||
@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.enums import BGPStatus
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
@@ -337,22 +339,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _normalize_satellite_constellation_group(
|
||||
raw_group: Any,
|
||||
name: Optional[str],
|
||||
) -> Optional[str]:
|
||||
normalized_group = str(raw_group or "").strip().lower()
|
||||
if normalized_group:
|
||||
return normalized_group
|
||||
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
|
||||
if constellation_group == "starlink":
|
||||
@@ -505,16 +491,19 @@ async def _count_current_or_latest_task_data(
|
||||
async def _load_current_collected_data_by_sources(
|
||||
db: AsyncSession,
|
||||
sources: List[str],
|
||||
*, include_compute_types: bool = False,
|
||||
) -> Dict[str, List[CollectedData]]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
if include_compute_types:
|
||||
stmt = stmt.where(CollectedData.deleted_at.is_(None))
|
||||
result = await db.execute(stmt)
|
||||
|
||||
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
|
||||
@@ -682,11 +671,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
withheld_count += 1
|
||||
continue
|
||||
result = resolve_compute_center_location_full(record, metadata)
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
site_type = compute_site_type(record)
|
||||
|
||||
if not result.is_resolved:
|
||||
diagnostic = result.diagnostic or ResolutionDiagnostic(
|
||||
@@ -1964,9 +1949,10 @@ async def _build_compute_centers_geojson(
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
include_compute_types=True,
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
[record for group in records_by_source.values() for record in group],
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
@@ -1981,6 +1967,7 @@ async def _build_compute_centers_geojson(
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
"by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
|
||||
"unresolved": 0,
|
||||
},
|
||||
}
|
||||
@@ -2023,6 +2010,7 @@ async def _build_compute_centers_geojson(
|
||||
"count": len(sanitized_features),
|
||||
"stats": {
|
||||
"total": len(sanitized_features),
|
||||
"by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
|
||||
"supercomputers": sum(
|
||||
1 for feature in sanitized_features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
@@ -2339,7 +2327,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source_id == source_id)
|
||||
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
|
||||
.where(compute_record_filter(CollectedData))
|
||||
.where(CollectedData.deleted_at.is_(None))
|
||||
.order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
@@ -2738,6 +2727,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
|
||||
)
|
||||
|
||||
|
||||
async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
|
||||
kind = compute_site_type_expression(CollectedData)
|
||||
rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
|
||||
.where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
|
||||
CollectedData.deleted_at.is_(None))
|
||||
.group_by(CollectedData.source, kind))).all()
|
||||
counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
|
||||
present_sources = set()
|
||||
for source, site_type, count in rows:
|
||||
present_sources.add(source)
|
||||
counts[site_type] += int(count or 0)
|
||||
# Retain the existing last-task fallback for the two collected catalogs.
|
||||
for source in COMPUTE_SOURCES:
|
||||
if source not in present_sources:
|
||||
key = "supercomputer" if source == "top500" else "gpu_cluster"
|
||||
counts[key] += await _count_current_or_latest_task_data(db, source)
|
||||
return counts
|
||||
|
||||
|
||||
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
|
||||
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
|
||||
@@ -2746,9 +2754,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"celestrak_tle",
|
||||
exclude_unknown_name=True,
|
||||
)
|
||||
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
|
||||
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
|
||||
compute_center_count = supercomputer_count + gpu_cluster_count
|
||||
compute_counts = await _compute_facility_counts(db)
|
||||
supercomputer_count = compute_counts["supercomputer"]
|
||||
gpu_cluster_count = compute_counts["gpu_cluster"]
|
||||
compute_center_count = sum(compute_counts.values())
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
|
||||
@@ -2803,6 +2812,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
|
||||
"supercomputer_count": supercomputer_count,
|
||||
"gpu_cluster_count": gpu_cluster_count,
|
||||
"data_center_count": compute_counts["data_center"],
|
||||
"cloud_center_count": compute_counts["cloud_center"],
|
||||
"edge_node_count": compute_counts["edge_node"],
|
||||
"bgp_event_count": active_incident_count or active_anomaly_count,
|
||||
"bgp_incident_count": active_incident_count,
|
||||
"bgp_anomaly_count": active_anomaly_count,
|
||||
|
||||
35
backend/app/core/compute_types.py
Normal file
35
backend/app/core/compute_types.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""Compute facility classification shared by the map and image resolver."""
|
||||
|
||||
from sqlalchemy import case, or_
|
||||
|
||||
COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
|
||||
COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
|
||||
|
||||
|
||||
def compute_site_type(record) -> str:
|
||||
metadata = record.extra_data or {}
|
||||
for value in (metadata.get("site_type"), record.data_type):
|
||||
if value in COMPUTE_SITE_TYPES:
|
||||
return value
|
||||
return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
|
||||
record.source, "data_center"
|
||||
)
|
||||
|
||||
|
||||
def compute_site_type_expression(record):
|
||||
declared = record.extra_data["site_type"].as_string()
|
||||
return case(
|
||||
(declared.in_(COMPUTE_SITE_TYPES), declared),
|
||||
(record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
|
||||
(record.source == "top500", "supercomputer"),
|
||||
(record.source == "epoch_ai_gpu", "gpu_cluster"),
|
||||
else_="data_center",
|
||||
)
|
||||
|
||||
|
||||
def compute_record_filter(record):
|
||||
return or_(
|
||||
record.source.in_(COMPUTE_SOURCES),
|
||||
record.data_type.in_(COMPUTE_SITE_TYPES),
|
||||
record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
|
||||
)
|
||||
13
backend/app/core/satellite_presentation.py
Normal file
13
backend/app/core/satellite_presentation.py
Normal file
@@ -0,0 +1,13 @@
|
||||
"""Shared source-group normalization; never infer hardware generations."""
|
||||
|
||||
|
||||
def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
|
||||
group = str(raw_group or "").strip().lower()
|
||||
if group:
|
||||
return group
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
return None
|
||||
51
backend/app/db/bgp_prefix_geo.py
Normal file
51
backend/app/db/bgp_prefix_geo.py
Normal file
@@ -0,0 +1,51 @@
|
||||
"""SQL objects supporting indexed BGP prefix geography lookup."""
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncConnection
|
||||
|
||||
BGP_PREFIX_GEO_FUNCTION_SQL = (
|
||||
"""
|
||||
CREATE OR REPLACE FUNCTION planet_bgp_try_inet(value text)
|
||||
RETURNS inet
|
||||
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
|
||||
AS $$
|
||||
BEGIN
|
||||
RETURN value::inet;
|
||||
EXCEPTION WHEN data_exception THEN
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$
|
||||
""",
|
||||
"""
|
||||
CREATE OR REPLACE FUNCTION planet_bgp_try_masklen(value text)
|
||||
RETURNS integer
|
||||
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
|
||||
AS $$
|
||||
BEGIN
|
||||
RETURN masklen(value::cidr);
|
||||
EXCEPTION WHEN data_exception THEN
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$
|
||||
""",
|
||||
)
|
||||
|
||||
BGP_PREFIX_GEO_INDEX_NAME = "idx_bgp_prefix_geo_range_current"
|
||||
BGP_PREFIX_GEO_INDEX_SQL = f"""
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS {BGP_PREFIX_GEO_INDEX_NAME}
|
||||
ON collected_data USING gist (
|
||||
planet_bgp_try_inet(metadata->>'range_start') inet_ops,
|
||||
planet_bgp_try_inet(metadata->>'range_end') inet_ops
|
||||
)
|
||||
WHERE source IN (
|
||||
'opengeofeed_prefix_geo',
|
||||
'iptoasn_prefix_geo',
|
||||
'nro_delegated_prefix_geo'
|
||||
)
|
||||
AND COALESCE(is_current, TRUE) = TRUE
|
||||
"""
|
||||
|
||||
|
||||
async def install_bgp_prefix_geo_functions(connection: AsyncConnection) -> None:
|
||||
for statement in BGP_PREFIX_GEO_FUNCTION_SQL:
|
||||
await connection.execute(text(statement))
|
||||
@@ -6,6 +6,7 @@ from sqlalchemy.orm import declarative_base
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.db.bgp_prefix_geo import install_bgp_prefix_geo_functions
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -219,6 +220,7 @@ async def init_db():
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
await install_bgp_prefix_geo_functions(conn)
|
||||
users_email_verified_existed = (
|
||||
await conn.execute(
|
||||
text(
|
||||
@@ -299,8 +301,12 @@ async def init_db():
|
||||
change_event_id BIGINT;
|
||||
change_payload JSONB;
|
||||
BEGIN
|
||||
change_event_id := nextval(
|
||||
pg_get_serial_sequence('earth_data_change_events', 'id')::regclass
|
||||
);
|
||||
change_payload := jsonb_build_object(
|
||||
'event', 'earth.layer.changed',
|
||||
'event_id', change_event_id,
|
||||
'table', change_table,
|
||||
'operation', change_operation,
|
||||
'source', change_source,
|
||||
@@ -311,6 +317,7 @@ async def init_db():
|
||||
);
|
||||
|
||||
INSERT INTO earth_data_change_events (
|
||||
id,
|
||||
table_name,
|
||||
operation,
|
||||
source,
|
||||
@@ -318,23 +325,15 @@ async def init_db():
|
||||
payload,
|
||||
occurred_at
|
||||
) VALUES (
|
||||
change_event_id,
|
||||
change_table,
|
||||
change_operation,
|
||||
change_source,
|
||||
NULL,
|
||||
change_payload,
|
||||
NOW()
|
||||
)
|
||||
RETURNING id INTO change_event_id;
|
||||
|
||||
change_payload := change_payload || jsonb_build_object(
|
||||
'event_id', change_event_id
|
||||
);
|
||||
|
||||
UPDATE earth_data_change_events
|
||||
SET payload = change_payload
|
||||
WHERE id = change_event_id;
|
||||
|
||||
PERFORM pg_notify(
|
||||
'planet_earth_data_changes',
|
||||
change_payload::text
|
||||
@@ -440,6 +439,16 @@ async def init_db():
|
||||
);
|
||||
END LOOP;
|
||||
ELSIF TG_OP = 'UPDATE' THEN
|
||||
IF TG_TABLE_NAME = 'earth_news_items' AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM new_rows AS current_row
|
||||
FULL JOIN old_rows AS previous_row USING (id)
|
||||
WHERE (to_jsonb(current_row) - 'last_seen_at' - 'updated_at')
|
||||
IS DISTINCT FROM
|
||||
(to_jsonb(previous_row) - 'last_seen_at' - 'updated_at')
|
||||
) THEN
|
||||
RETURN NULL;
|
||||
END IF;
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
|
||||
FROM (
|
||||
|
||||
128
backend/app/schemas/card_images.py
Normal file
128
backend/app/schemas/card_images.py
Normal file
@@ -0,0 +1,128 @@
|
||||
"""Typed, validated configuration for optional object-card images."""
|
||||
|
||||
from typing import Literal, Self
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
|
||||
MatchScope = Literal["object", "model", "group", "category"]
|
||||
|
||||
|
||||
def safe_url(value: str, *, local: bool = False) -> str:
|
||||
if not value:
|
||||
return value
|
||||
if any(ord(char) < 32 for char in value) or "\\" in value:
|
||||
raise ValueError("Invalid URL")
|
||||
parts = urlsplit(value)
|
||||
if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
|
||||
if (
|
||||
".." not in parts.path
|
||||
and "%" not in parts.path
|
||||
and not parts.netloc
|
||||
and not parts.query
|
||||
and not parts.fragment
|
||||
):
|
||||
return value
|
||||
if (
|
||||
parts.scheme in {"http", "https"}
|
||||
and parts.hostname
|
||||
and not parts.username
|
||||
and not parts.password
|
||||
):
|
||||
return value
|
||||
raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
|
||||
|
||||
|
||||
class CardImageAsset(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
|
||||
title: str = Field(min_length=1, max_length=200)
|
||||
url: str = Field(min_length=1, max_length=2000)
|
||||
variants: dict[str, str] = Field(default_factory=dict, max_length=4)
|
||||
width: int = Field(default=0, ge=0, le=20000)
|
||||
height: int = Field(default=0, ge=0, le=20000)
|
||||
credit: str = Field(default="", max_length=500)
|
||||
source_url: str = Field(default="", max_length=2000)
|
||||
license: str = Field(default="", max_length=300)
|
||||
license_url: str = Field(default="", max_length=2000)
|
||||
kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
|
||||
rights_status: Literal["available", "permission_pending"] = "available"
|
||||
enabled: bool = True
|
||||
focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
|
||||
focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def validate_image_url(cls, value: str) -> str:
|
||||
return safe_url(value, local=True)
|
||||
|
||||
@field_validator("source_url", "license_url")
|
||||
@classmethod
|
||||
def validate_source_url(cls, value: str) -> str:
|
||||
return safe_url(value)
|
||||
|
||||
@field_validator("variants")
|
||||
@classmethod
|
||||
def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
|
||||
for width, url in value.items():
|
||||
if width not in {"640", "1280"}:
|
||||
raise ValueError("Unsupported image variant")
|
||||
safe_url(url, local=True)
|
||||
return value
|
||||
|
||||
|
||||
class CardImageRule(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
|
||||
entity_type: EntityType
|
||||
scope: MatchScope
|
||||
value: str = Field(min_length=1, max_length=300)
|
||||
label: str = Field(default="", max_length=500)
|
||||
asset_id: str | None = Field(default=None, max_length=100)
|
||||
suppress: bool = False
|
||||
enabled: bool = True
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_action(self) -> Self:
|
||||
if self.scope != "object":
|
||||
self.value = self.value.casefold()
|
||||
if self.suppress and (self.scope != "object" or self.asset_id):
|
||||
raise ValueError("Only an object rule may suppress its image, without an asset")
|
||||
if not self.suppress and not self.asset_id:
|
||||
raise ValueError("Choose an image or suppress this object")
|
||||
return self
|
||||
|
||||
def target(self) -> tuple[str, str, str]:
|
||||
return self.entity_type, self.scope, self.value
|
||||
|
||||
|
||||
class CardImageConfiguration(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
|
||||
rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_relations(self) -> Self:
|
||||
assets = {asset.id for asset in self.assets}
|
||||
if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
|
||||
self.rules
|
||||
):
|
||||
raise ValueError("Image and rule IDs must be unique")
|
||||
targets = [rule.target() for rule in self.rules]
|
||||
if len(set(targets)) != len(targets):
|
||||
raise ValueError("Each target may have only one rule")
|
||||
if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
|
||||
raise ValueError("Remove image associations before deleting the image")
|
||||
return self
|
||||
|
||||
|
||||
class CardImagePublish(BaseModel):
|
||||
revision: str = Field(max_length=200)
|
||||
config: CardImageConfiguration
|
||||
|
||||
|
||||
class CardImagePreview(BaseModel):
|
||||
entity_type: EntityType
|
||||
entity_id: str = Field(min_length=1, max_length=300)
|
||||
config: CardImageConfiguration
|
||||
94
backend/app/schemas/compute_configuration.py
Normal file
94
backend/app/schemas/compute_configuration.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""Validated, source-backed observations for the compute comparison cards."""
|
||||
|
||||
from datetime import date
|
||||
from typing import Annotated, Literal, Self
|
||||
|
||||
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
|
||||
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
|
||||
|
||||
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
|
||||
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
|
||||
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
|
||||
Text = Annotated[str, Field(min_length=1, max_length=500)]
|
||||
|
||||
|
||||
class ConfigurationModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
|
||||
|
||||
class ComputeSource(ConfigurationModel):
|
||||
id: Identifier
|
||||
title: Text
|
||||
url: HttpUrl
|
||||
publisher: Text
|
||||
report_year: Year
|
||||
published_at: date | None = None
|
||||
locator: Text
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def public_reference(cls, value: HttpUrl) -> HttpUrl:
|
||||
if value.username or value.password or len(str(value)) > 2000:
|
||||
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
|
||||
return value
|
||||
|
||||
|
||||
class ComputeObservation(ConfigurationModel):
|
||||
year: Year
|
||||
metric: Literal["global_share", "absolute"]
|
||||
source_id: Identifier
|
||||
values: dict[str, Value] = Field(min_length=1, max_length=250)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def comparable_values(self) -> Self:
|
||||
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
|
||||
raise ValueError(
|
||||
"Use countries from the canonical compute catalog, not regional totals"
|
||||
)
|
||||
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
|
||||
raise ValueError("Global shares for one year must total at most 100 percent")
|
||||
return self
|
||||
|
||||
|
||||
class InnovationObservation(ConfigurationModel):
|
||||
year: Year
|
||||
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
|
||||
source_id: Identifier
|
||||
|
||||
|
||||
class ComputeConfiguration(ConfigurationModel):
|
||||
reviewed_at: date
|
||||
sources: list[ComputeSource] = Field(max_length=200)
|
||||
observations: list[ComputeObservation] = Field(max_length=202)
|
||||
innovation: list[InnovationObservation] = Field(max_length=101)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def consistent_evidence(self) -> Self:
|
||||
ids = [source.id for source in self.sources]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("Report source IDs must be unique")
|
||||
keys = [(row.metric, row.year) for row in self.observations]
|
||||
if len(keys) != len(set(keys)):
|
||||
raise ValueError("Each metric/year may have only one observation")
|
||||
years = [row.year for row in self.innovation]
|
||||
if len(years) != len(set(years)):
|
||||
raise ValueError("Innovation observation years must be unique")
|
||||
sources = {source.id: source for source in self.sources}
|
||||
for row in [*self.observations, *self.innovation]:
|
||||
source = sources.get(row.source_id)
|
||||
if source is None:
|
||||
raise ValueError(f"Missing report source: {row.source_id}")
|
||||
if row.year > source.report_year:
|
||||
raise ValueError("Observation year must not be after its report year")
|
||||
if self.reviewed_at > date.today():
|
||||
raise ValueError("Review date must not be in the future")
|
||||
return self
|
||||
|
||||
|
||||
class ComputeConfigurationWrite(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
config: ComputeConfiguration
|
||||
|
||||
|
||||
class ComputeConfigurationReset(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
@@ -111,15 +111,24 @@ async def _lookup_prefix_geography(
|
||||
result = await db.execute(
|
||||
text(
|
||||
"""
|
||||
WITH candidates AS MATERIALIZED (
|
||||
SELECT source, id, metadata
|
||||
FROM collected_data
|
||||
WHERE source IN (
|
||||
'opengeofeed_prefix_geo',
|
||||
'iptoasn_prefix_geo',
|
||||
'nro_delegated_prefix_geo'
|
||||
)
|
||||
AND COALESCE(is_current, TRUE) = TRUE
|
||||
AND planet_bgp_try_inet(metadata->>'range_start') <= CAST(:range_start AS inet)
|
||||
AND planet_bgp_try_inet(metadata->>'range_end') >= CAST(:range_end AS inet)
|
||||
)
|
||||
SELECT metadata
|
||||
FROM collected_data
|
||||
FROM candidates
|
||||
WHERE source = :source
|
||||
AND COALESCE(is_current, TRUE) = TRUE
|
||||
AND metadata->>'family' = :family
|
||||
AND CAST(metadata->>'range_start' AS inet) <= CAST(:range_start AS inet)
|
||||
AND CAST(metadata->>'range_end' AS inet) >= CAST(:range_end AS inet)
|
||||
ORDER BY
|
||||
masklen(CAST(metadata->>'prefix' AS cidr)) DESC NULLS LAST,
|
||||
planet_bgp_try_masklen(metadata->>'prefix') DESC NULLS LAST,
|
||||
id DESC
|
||||
LIMIT 1
|
||||
"""
|
||||
@@ -146,6 +155,10 @@ async def _lookup_prefix_geography(
|
||||
|
||||
results: dict[str, dict[str, Any]] = {}
|
||||
|
||||
# The GiST range predicate is selective, but PostgreSQL 15 greatly overestimates its cost.
|
||||
# Avoid spending more time compiling JIT code than fetching the small candidate set.
|
||||
await db.execute(text("SET LOCAL jit = off"))
|
||||
|
||||
for prefix in prefix_values:
|
||||
try:
|
||||
network = ipaddress.ip_network(prefix, strict=False)
|
||||
|
||||
174
backend/app/services/card_image_configuration.py
Normal file
174
backend/app/services/card_image_configuration.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""Revision-checked image settings; new bundled assets never replace user edits."""
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.card_images import CardImageConfiguration
|
||||
from app.services.card_image_defaults import (
|
||||
bundled_configuration,
|
||||
previous_bundled_assets,
|
||||
previous_bundled_rules,
|
||||
)
|
||||
|
||||
CATEGORY = "earth_card_images"
|
||||
|
||||
|
||||
def effective_envelope(payload: dict | None) -> dict:
|
||||
defaults, version = bundled_configuration()
|
||||
config = (
|
||||
CardImageConfiguration.model_validate(payload["config"])
|
||||
if payload
|
||||
else defaults.model_copy(deep=True)
|
||||
)
|
||||
if payload:
|
||||
previous = previous_bundled_assets()
|
||||
latest = {asset.id: asset for asset in defaults.assets}
|
||||
for index, asset in enumerate(config.assets):
|
||||
old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
|
||||
# Replace the old bundled source while preserving fields the user
|
||||
# changed (crop, title, enabled state, etc.). Custom URLs are untouched.
|
||||
if old is not None and asset.url == old.url and asset.id in latest:
|
||||
overrides = {
|
||||
key: value
|
||||
for key, value in asset.model_dump().items()
|
||||
if value != getattr(old, key)
|
||||
}
|
||||
config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
|
||||
known_assets = set(payload.get("known_assets", []))
|
||||
asset_ids = {asset.id for asset in config.assets}
|
||||
config.assets.extend(
|
||||
asset.model_copy(deep=True)
|
||||
for asset in defaults.assets
|
||||
if asset.id not in known_assets and asset.id not in asset_ids
|
||||
)
|
||||
prior_rules = previous_bundled_rules()
|
||||
latest_rules = {rule.id: rule for rule in defaults.rules}
|
||||
for index, rule in enumerate(config.rules):
|
||||
old_rule = prior_rules.get(rule.id)
|
||||
if (
|
||||
old_rule
|
||||
and rule.id in latest_rules
|
||||
and rule.target() == old_rule.target()
|
||||
and rule.asset_id == old_rule.asset_id
|
||||
):
|
||||
overrides = {
|
||||
key: value
|
||||
for key, value in rule.model_dump().items()
|
||||
if value != getattr(old_rule, key)
|
||||
}
|
||||
config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
|
||||
known_rules = set(payload.get("known_rules", []))
|
||||
rule_ids = {rule.id for rule in config.rules}
|
||||
targets = {rule.target() for rule in config.rules}
|
||||
available = {asset.id for asset in config.assets}
|
||||
config.rules.extend(
|
||||
rule.model_copy(deep=True)
|
||||
for rule in defaults.rules
|
||||
if rule.id not in known_rules
|
||||
and rule.id not in rule_ids
|
||||
and rule.target() not in targets
|
||||
and rule.asset_id in available
|
||||
)
|
||||
return {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": f"{version}:{payload['revision'] if payload else 'default'}",
|
||||
"is_default": payload is None,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return effective_envelope(record.payload if record else None)
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
|
||||
) -> dict:
|
||||
defaults, _ = bundled_configuration()
|
||||
pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
|
||||
if any(
|
||||
asset.id in pending and asset.rights_status != "permission_pending"
|
||||
for asset in config.assets
|
||||
):
|
||||
raise HTTPException(422, "Bundled image permission is still unverified")
|
||||
initial = {
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"revision": "default",
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=initial)
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if effective_envelope(record.payload)["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
before = record.payload
|
||||
record.payload = {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"known_assets": [asset.id for asset in defaults.assets],
|
||||
"known_rules": [rule.id for rule in defaults.rules],
|
||||
}
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="card_images.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": record.payload},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return effective_envelope(record.payload)
|
||||
|
||||
|
||||
def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
|
||||
assets = {asset.id: asset for asset in config.assets}
|
||||
for scope in ("object", "model", "group", "category"):
|
||||
value = context.get("entity_id" if scope == "object" else scope)
|
||||
values = [value] if value else []
|
||||
if scope == "category" and value != "*":
|
||||
values.append("*")
|
||||
for value in values:
|
||||
value = value if scope == "object" else value.casefold()
|
||||
rule = next(
|
||||
(
|
||||
rule
|
||||
for rule in config.rules
|
||||
if rule.enabled
|
||||
and rule.entity_type == context["entity_type"]
|
||||
and rule.scope == scope
|
||||
and rule.value == value
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not rule:
|
||||
continue
|
||||
match = {"scope": scope, "value": value, "rule_id": rule.id}
|
||||
if rule.suppress:
|
||||
return {"image": None, "match": match, "suppressed": True}
|
||||
asset = assets.get(rule.asset_id)
|
||||
if asset and asset.enabled:
|
||||
return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
|
||||
return {"image": None, "match": None, "suppressed": False}
|
||||
108
backend/app/services/card_image_defaults.py
Normal file
108
backend/app/services/card_image_defaults.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""Read immutable bundled assets without inserting machine-specific database rows."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
|
||||
|
||||
|
||||
def _asset_from_manifest(item: dict) -> CardImageAsset:
|
||||
pending = item.get("redistribution_status") == "permission_not_verified"
|
||||
return CardImageAsset(
|
||||
id=item["id"],
|
||||
title=item["label"],
|
||||
url=item["url"],
|
||||
width=item["width"],
|
||||
height=item["height"],
|
||||
credit=item["credit"],
|
||||
source_url=item.get("source_page") or "",
|
||||
license=item["license"],
|
||||
license_url=item.get("license_url") or "",
|
||||
kind=item["kind"],
|
||||
rights_status="permission_pending" if pending else "available",
|
||||
enabled=item.get("enabled", not pending),
|
||||
focus_x=item.get("focus_x", 50),
|
||||
focus_y=item.get("focus_y", 50),
|
||||
variants=item.get("variants", {}),
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_bytes())
|
||||
return {
|
||||
key: [CardImageAsset.model_validate(item) for item in value]
|
||||
for key, value in manifest.get("previous_defaults", {}).items()
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def previous_bundled_rules() -> dict[str, CardImageRule]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_bytes())
|
||||
return {
|
||||
key: CardImageRule.model_validate(value)
|
||||
for key, value in manifest.get("previous_rules", {}).items()
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def bundled_configuration() -> tuple[CardImageConfiguration, str]:
|
||||
content = MANIFEST_PATH.read_bytes()
|
||||
manifest = json.loads(content)
|
||||
assets, rules = [], []
|
||||
for item in manifest["assets"]:
|
||||
asset = _asset_from_manifest(item)
|
||||
assets.append(asset)
|
||||
if not asset.enabled:
|
||||
continue
|
||||
hints = item.get("binding_hints", {})
|
||||
for index, target in enumerate(hints.get("defaults", [])):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=target.get("id", f"builtin-{asset.id}-default-{index}"),
|
||||
entity_type=target["entity_type"],
|
||||
scope=target["scope"],
|
||||
value=target["value"],
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
for index, entity_id in enumerate(hints.get("entity_ids", [])):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-entity-{index}",
|
||||
entity_type=item["entity_type"],
|
||||
scope="object",
|
||||
value=entity_id,
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
for norad_id in hints.get("norad_ids", []):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-{norad_id}",
|
||||
entity_type="satellite",
|
||||
scope="object",
|
||||
value=str(norad_id),
|
||||
label=f"{asset.title} · NORAD {norad_id}",
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
if group := hints.get("constellation_group"):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-group",
|
||||
entity_type="satellite",
|
||||
scope="group",
|
||||
value=group,
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()
|
||||
95
backend/app/services/card_image_files.py
Normal file
95
backend/app/services/card_image_files.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import re
|
||||
import warnings
|
||||
from uuid import uuid4
|
||||
from pathlib import Path
|
||||
import os
|
||||
|
||||
from fastapi import HTTPException
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
from app.schemas.card_images import CardImageAsset
|
||||
from app.services.card_image_defaults import REPO_ROOT
|
||||
|
||||
ASSET_DIR = Path(
|
||||
os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
|
||||
).resolve()
|
||||
MAX_UPLOAD_BYTES = 12 * 1024 * 1024
|
||||
MAX_PIXELS = 32_000_000
|
||||
FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
|
||||
FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
|
||||
|
||||
|
||||
def uploaded_path(filename: str) -> Path:
|
||||
if not FILE_PATTERN.fullmatch(filename):
|
||||
raise HTTPException(404, "Image not found")
|
||||
return ASSET_DIR / filename
|
||||
|
||||
|
||||
def store_image(content: bytes, title: str) -> CardImageAsset:
|
||||
if len(content) > MAX_UPLOAD_BYTES:
|
||||
raise HTTPException(413, "Image must be 12 MiB or smaller")
|
||||
try:
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
||||
image = Image.open(io.BytesIO(content))
|
||||
extension = FORMATS.get(image.format)
|
||||
if (
|
||||
not extension
|
||||
or image.width * image.height > MAX_PIXELS
|
||||
or getattr(image, "is_animated", False)
|
||||
):
|
||||
raise ValueError("Unsupported or oversized raster")
|
||||
image.load()
|
||||
image = ImageOps.exif_transpose(image).convert(
|
||||
"RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
|
||||
)
|
||||
except (
|
||||
UnidentifiedImageError,
|
||||
OSError,
|
||||
ValueError,
|
||||
Image.DecompressionBombError,
|
||||
Image.DecompressionBombWarning,
|
||||
) as error:
|
||||
raise HTTPException(
|
||||
422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
|
||||
) from error
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
ASSET_DIR.mkdir(parents=True, exist_ok=True)
|
||||
original = f"{digest}-original.{extension}"
|
||||
original_path = uploaded_path(original)
|
||||
if not original_path.exists():
|
||||
temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
|
||||
temporary.write_bytes(content)
|
||||
temporary.replace(original_path)
|
||||
variants = {}
|
||||
for size, content in render_variants(image).items():
|
||||
filename = f"{digest}-{size}.webp"
|
||||
path = uploaded_path(filename)
|
||||
if not path.exists():
|
||||
temporary = path.with_suffix(f".{uuid4().hex}.tmp")
|
||||
temporary.write_bytes(content)
|
||||
temporary.replace(path)
|
||||
variants[size] = f"/api/v1/earth/card-images/files/{filename}"
|
||||
return CardImageAsset(
|
||||
id=f"upload-{digest[:24]}",
|
||||
title=title or "Image",
|
||||
url=f"/api/v1/earth/card-images/files/{original}",
|
||||
variants=variants,
|
||||
width=image.width,
|
||||
height=image.height,
|
||||
)
|
||||
|
||||
|
||||
def render_variants(image: Image.Image) -> dict[str, bytes]:
|
||||
result = {}
|
||||
for size in (640, 1280):
|
||||
resized = image.copy()
|
||||
resized.thumbnail((size, size), Image.Resampling.LANCZOS)
|
||||
buffer = io.BytesIO()
|
||||
resized.save(buffer, format="WEBP", quality=93, method=4)
|
||||
result[str(size)] = buffer.getvalue()
|
||||
return result
|
||||
180
backend/app/services/card_image_targets.py
Normal file
180
backend/app/services/card_image_targets.py
Normal file
@@ -0,0 +1,180 @@
|
||||
"""Bounded catalog queries and type defaults for every Earth object card."""
|
||||
|
||||
from sqlalchemy import String, case, cast, func, literal, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import (
|
||||
COMPUTE_SITE_TYPES,
|
||||
compute_site_type_expression,
|
||||
compute_record_filter,
|
||||
)
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_collector_location import BGPCollectorLocation
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
from app.models.vessel import VesselCurrentState
|
||||
from app.schemas.card_images import EntityType, MatchScope
|
||||
|
||||
SOURCES = {
|
||||
"satellite": ("celestrak_tle", "spacetrack_tle"),
|
||||
"compute_center": ("top500", "epoch_ai_gpu"),
|
||||
"cable": ("arcgis_cables",),
|
||||
"landing_point": ("arcgis_landing_points",),
|
||||
}
|
||||
# Named constellation families, not inferred hardware models or generations.
|
||||
SATELLITE_GROUP_PATTERNS = (
|
||||
("STARLINK%", "starlink"),
|
||||
("IRIDIUM%", "iridium-next"),
|
||||
("BEIDOU-3%", "beidou-3"),
|
||||
("BEIDOU%", "beidou"),
|
||||
("QIANFAN%", "qianfan"),
|
||||
("ONEWEB%", "oneweb"),
|
||||
("KUIPER%", "kuiper"),
|
||||
)
|
||||
|
||||
|
||||
def collected_query(entity_type: EntityType):
|
||||
record, metadata = CollectedData, CollectedData.extra_data
|
||||
model = func.coalesce(
|
||||
metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
|
||||
)
|
||||
category = (
|
||||
compute_site_type_expression(record)
|
||||
if entity_type == "compute_center"
|
||||
else literal(entity_type)
|
||||
)
|
||||
if entity_type == "satellite":
|
||||
group = func.coalesce(
|
||||
func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
|
||||
case(
|
||||
*[
|
||||
(func.upper(record.name).like(pattern), value)
|
||||
for pattern, value in SATELLITE_GROUP_PATTERNS
|
||||
],
|
||||
else_="",
|
||||
),
|
||||
)
|
||||
entity_id = metadata["norad_cat_id"].as_string()
|
||||
else:
|
||||
group = func.coalesce(
|
||||
metadata["operator"].as_string(), metadata["organization"].as_string(), ""
|
||||
)
|
||||
entity_id = record.source + ":" + record.source_id
|
||||
return select(
|
||||
record.name.label("name"),
|
||||
cast(entity_id, String).label("entity_id"),
|
||||
model.label("model"),
|
||||
group.label("group"),
|
||||
category.label("category"),
|
||||
).where(
|
||||
(
|
||||
compute_record_filter(record)
|
||||
if entity_type == "compute_center"
|
||||
else record.source.in_(SOURCES[entity_type])
|
||||
),
|
||||
record.is_current.is_(True),
|
||||
record.deleted_at.is_(None),
|
||||
metadata["anonymous"].as_boolean().is_not(True),
|
||||
)
|
||||
|
||||
|
||||
def target_query(entity_type: EntityType):
|
||||
if entity_type in SOURCES:
|
||||
return collected_query(entity_type)
|
||||
if entity_type == "vessel":
|
||||
model = VesselCurrentState
|
||||
identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
|
||||
conditions = []
|
||||
elif entity_type == "bgp":
|
||||
model = BGPIncident
|
||||
identity, name, group, category = (
|
||||
model.incident_key,
|
||||
model.title,
|
||||
model.source,
|
||||
model.incident_type,
|
||||
)
|
||||
conditions = []
|
||||
elif entity_type == "bgp_collector":
|
||||
model = BGPCollectorLocation
|
||||
identity, name, group, category = (
|
||||
model.collector_id,
|
||||
model.collector_id,
|
||||
model.operator,
|
||||
literal("bgp_collector"),
|
||||
)
|
||||
conditions = []
|
||||
else:
|
||||
model = EarthInteractable
|
||||
identity, name, group, category = model.id, model.label, model.layer, model.kind
|
||||
conditions = [model.is_deleted.is_(False)]
|
||||
return select(
|
||||
name.label("name"),
|
||||
cast(identity, String).label("entity_id"),
|
||||
literal("").label("model"),
|
||||
func.coalesce(group, "").label("group"),
|
||||
func.coalesce(category, entity_type).label("category"),
|
||||
).where(*conditions)
|
||||
|
||||
|
||||
def category_context(entity_type: EntityType) -> dict[str, str]:
|
||||
return {
|
||||
"entity_type": entity_type,
|
||||
"entity_id": "",
|
||||
"name": "",
|
||||
"model": "",
|
||||
"group": "",
|
||||
"category": entity_type,
|
||||
}
|
||||
|
||||
|
||||
async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
|
||||
if entity_type == "compute_center" and entity_id.startswith("__category__:"):
|
||||
category = entity_id.partition(":")[2]
|
||||
if category in COMPUTE_SITE_TYPES:
|
||||
return {**category_context(entity_type), "category": category}
|
||||
if entity_id == "__category__":
|
||||
return category_context(entity_type)
|
||||
if entity_type in SOURCES and entity_type != "satellite":
|
||||
source, separator, source_id = entity_id.partition(":")
|
||||
if not separator or not source or not source_id:
|
||||
return category_context(entity_type)
|
||||
# Match the indexed identity columns before projecting JSON metadata.
|
||||
statement = collected_query(entity_type).where(
|
||||
CollectedData.source == source, CollectedData.source_id == source_id
|
||||
)
|
||||
else:
|
||||
catalog = target_query(entity_type).subquery()
|
||||
statement = select(catalog).where(catalog.c.entity_id == entity_id)
|
||||
row = (await db.execute(statement.limit(1))).mappings().first()
|
||||
if not row:
|
||||
# Old map snapshots can outlive a catalog refresh. They may use the type
|
||||
# illustration, but must not claim an unverified exact-object identity.
|
||||
return category_context(entity_type)
|
||||
context = {key: str(value or "") for key, value in row.items()}
|
||||
return {**context, "entity_type": entity_type}
|
||||
|
||||
|
||||
async def search_targets(
|
||||
db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
|
||||
) -> list[dict[str, str]]:
|
||||
catalog = target_query(entity_type).subquery()
|
||||
value = catalog.c["entity_id" if scope == "object" else scope]
|
||||
label = catalog.c.name if scope == "object" else value
|
||||
statement = select(value.label("value"), label.label("label")).where(
|
||||
value.is_not(None), value != ""
|
||||
)
|
||||
if query:
|
||||
escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
statement = statement.where(
|
||||
or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
|
||||
)
|
||||
rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
|
||||
items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
|
||||
if scope == "category" and not query:
|
||||
known = {item["value"] for item in items}
|
||||
if entity_type == "compute_center":
|
||||
items.extend(
|
||||
{"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
|
||||
)
|
||||
items.insert(0, {"value": "*", "label": "*"})
|
||||
return items
|
||||
@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
ACTIVE_GROUP = "active"
|
||||
FALLBACK_GROUPS = (
|
||||
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
|
||||
)
|
||||
FETCH_RETRY_ATTEMPTS = 3
|
||||
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
|
||||
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
|
||||
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
|
||||
|
||||
|
||||
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
_downloader = ResumableFileDownloader(
|
||||
cache_namespace="celestrak",
|
||||
default_accept="application/json",
|
||||
preflight_head=False,
|
||||
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
if not is_final_attempt:
|
||||
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
|
||||
|
||||
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active satellite download failed after retries: {last_error}"
|
||||
)
|
||||
|
||||
async def _fetch_fallback_groups(
|
||||
self,
|
||||
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
try:
|
||||
for group in FALLBACK_GROUPS:
|
||||
group_url = self._group_url(group)
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is not None:
|
||||
body_path = cached_path
|
||||
else:
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' download failed: {exc}"
|
||||
) from exc
|
||||
body_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if body_path is None:
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
|
||||
) from exc
|
||||
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
commit=True,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_json_file(path: Path) -> bool:
|
||||
@classmethod
|
||||
def _validate_json_file(cls, path: Path) -> bool:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
|
||||
cls._load_active_payload(path)
|
||||
except RuntimeError:
|
||||
return False
|
||||
return isinstance(data, list)
|
||||
return True
|
||||
|
||||
async def _log_parse_failure(self, exc: Exception) -> None:
|
||||
await emit_business_log(
|
||||
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
),
|
||||
)
|
||||
|
||||
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
|
||||
@staticmethod
|
||||
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
else:
|
||||
invalid_count += 1
|
||||
if invalid_count:
|
||||
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
|
||||
)
|
||||
if not records:
|
||||
raise RuntimeError("CelesTrak active payload contains no satellite records")
|
||||
return records
|
||||
|
||||
@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
|
||||
ValidateCallback = Callable[[Path], bool]
|
||||
|
||||
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
|
||||
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
default_accept: str = "*/*",
|
||||
cache_root: Path | None = None,
|
||||
preflight_head: bool = True,
|
||||
cache_ttl_seconds: float = 0,
|
||||
) -> None:
|
||||
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
|
||||
self._user_agent = user_agent
|
||||
self._default_accept = default_accept
|
||||
self._preflight_head = preflight_head
|
||||
self._cache_ttl_seconds = cache_ttl_seconds
|
||||
|
||||
@staticmethod
|
||||
def _cache_key(url: str) -> str:
|
||||
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
|
||||
if not meta_path.exists():
|
||||
return {}
|
||||
try:
|
||||
return json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
data = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError, OSError:
|
||||
return {}
|
||||
|
||||
@staticmethod
|
||||
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
|
||||
return last_modified == str(meta.get("last_modified") or "").strip()
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _response_info(response: httpx.Response) -> dict[str, Any]:
|
||||
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
|
||||
raw_length = response.headers.get("content-length")
|
||||
encoded = response.headers.get("content-encoding", "identity") != "identity"
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": int(raw_length) if raw_length and not encoded else None,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
|
||||
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
|
||||
try:
|
||||
response = await client.head(url)
|
||||
if response.status_code >= 400:
|
||||
return {}
|
||||
content_length_raw = response.headers.get("content-length")
|
||||
content_length = int(content_length_raw) if content_length_raw else None
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": content_length,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
except (httpx.HTTPError, ValueError):
|
||||
return self._response_info(response)
|
||||
except httpx.HTTPError, ValueError:
|
||||
return {}
|
||||
|
||||
async def download_file(
|
||||
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
|
||||
) -> Path:
|
||||
self._cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
final_path, part_path, meta_path = self._cache_paths(url, extension)
|
||||
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
|
||||
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
|
||||
size = cached_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return cached_path
|
||||
meta = self._load_meta(meta_path)
|
||||
remote = await self.fetch_remote_info(client, url)
|
||||
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
if final_path.exists():
|
||||
local_size = final_path.stat().st_size
|
||||
size_match = expected_size is None or local_size == expected_size
|
||||
if self._validators_match(meta, remote) and size_match:
|
||||
if (
|
||||
(remote.get("etag") or remote.get("last_modified"))
|
||||
and self._validators_match(meta, remote)
|
||||
and size_match
|
||||
):
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
final_path.unlink(missing_ok=True)
|
||||
else:
|
||||
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
|
||||
headers["If-Range"] = str(remote.get("last_modified"))
|
||||
|
||||
async with client.stream("GET", url, headers=headers) as response:
|
||||
if response.status_code == 304 and final_path.exists():
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
if response.status_code == 304:
|
||||
if cached_path is None:
|
||||
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
|
||||
final_path.touch()
|
||||
size = final_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return final_path
|
||||
if response.status_code >= 400:
|
||||
body = (await response.aread()).decode("utf-8", errors="replace")
|
||||
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
|
||||
else:
|
||||
mode = "wb"
|
||||
resume_from = 0
|
||||
remote = self._response_info(response)
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
downloaded = resume_from
|
||||
last_emit_bytes = 0
|
||||
last_emit_time = time.monotonic()
|
||||
min_emit_bytes = (
|
||||
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
|
||||
max(expected_size // 150, 512 * 1024)
|
||||
if expected_size and expected_size > 0
|
||||
else 1024 * 1024
|
||||
)
|
||||
|
||||
with part_path.open(mode) as f:
|
||||
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
|
||||
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
|
||||
)
|
||||
|
||||
# Never replace the last good payload with an HTML error or invalid JSON.
|
||||
if validate_existing and not validate_existing(part_path):
|
||||
part_path.unlink(missing_ok=True)
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
part_path.replace(final_path)
|
||||
self._save_meta(
|
||||
meta_path,
|
||||
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
|
||||
},
|
||||
)
|
||||
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import compute_record_filter
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
@@ -930,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
"""Seed stored compute-center locations only from real source coordinates."""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
|
||||
.where(compute_record_filter(CollectedData))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
|
||||
99
backend/app/services/compute_configuration.py
Normal file
99
backend/app/services/compute_configuration.py
Normal file
@@ -0,0 +1,99 @@
|
||||
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import COUNTRY_CATALOG
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
CATEGORY = "compute_comparison"
|
||||
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
|
||||
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
|
||||
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def default_envelope() -> dict:
|
||||
return {
|
||||
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
|
||||
"revision": DEFAULT_REVISION,
|
||||
"is_default": True,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return dict(record.payload) if record else default_envelope()
|
||||
|
||||
|
||||
def configuration_response(envelope: dict) -> dict:
|
||||
return {
|
||||
**envelope,
|
||||
"countries": [
|
||||
{key: country[key] for key in ("code", "name", "name_en")}
|
||||
for country in COUNTRY_CATALOG
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
revision: str,
|
||||
config: ComputeConfiguration,
|
||||
actor: User,
|
||||
reset: bool = False,
|
||||
) -> dict:
|
||||
# The unique category plus row lock also serializes concurrent first writes.
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=default_envelope())
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if record.payload["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
envelope = (
|
||||
default_envelope()
|
||||
if reset
|
||||
else {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"is_default": reset,
|
||||
}
|
||||
)
|
||||
before = record.payload
|
||||
if reset:
|
||||
await db.delete(record)
|
||||
else:
|
||||
record.payload = envelope
|
||||
# Audit and configuration commit together: a published edit must be attributable.
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="compute_comparison.reset" if reset else "compute_comparison.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": envelope},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return envelope
|
||||
@@ -1,74 +1,16 @@
|
||||
"""Versioned, source-backed annual observations for compute comparisons."""
|
||||
|
||||
from copy import deepcopy
|
||||
from datetime import date
|
||||
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
DEFAULT_PARTICIPANTS,
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
)
|
||||
|
||||
HISTORY_VERSION = "compute-history-2026-09-22"
|
||||
HISTORY_YEARS = tuple(range(2020, 2027))
|
||||
REPORTS = {
|
||||
"caict-2021": {
|
||||
"title": "中国算力发展指数白皮书(2021)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2021,
|
||||
"published_at": None,
|
||||
"locator": "正文 11 页图 4(2020 年全球份额)",
|
||||
},
|
||||
"caict-2022": {
|
||||
"title": "中国算力发展指数白皮书(2022)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2022,
|
||||
"published_at": None,
|
||||
"locator": "正文 13 页图 5(2021 年全球份额)",
|
||||
},
|
||||
"caict-2023": {
|
||||
"title": "中国算力发展指数白皮书(2023)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2023,
|
||||
"published_at": None,
|
||||
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)",
|
||||
},
|
||||
"caict-2024": {
|
||||
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2024,
|
||||
"published_at": None,
|
||||
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)",
|
||||
},
|
||||
"innovation-2026": {
|
||||
"title": "全球人工智能创新指数报告 2026:科技日报报道",
|
||||
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
|
||||
"publisher": "科技日报(新浪转载)",
|
||||
"published_at": "2026-07-17",
|
||||
"publication_year": 2026,
|
||||
"locator": "2023–2025 年美国减中国的指数分差",
|
||||
},
|
||||
}
|
||||
GLOBAL_SHARES = {
|
||||
year: {"US": us, "CN": cn, "JP": jp}
|
||||
for year, (us, cn, jp) in {
|
||||
2020: (36, 31, 6), 2021: (34, 33, 5),
|
||||
2022: (34, 33, 4), 2023: (41, 31, 4),
|
||||
}.items()
|
||||
}
|
||||
|
||||
ABSOLUTE_COMPUTE = {2022: {"CN": 302}, 2023: {"CN": 435}}
|
||||
GAPS = {2023: 22.02, 2024: 19.96, 2025: 17.95}
|
||||
|
||||
|
||||
def _observations(mode: str, year: int) -> dict[str, float | int]:
|
||||
return (GLOBAL_SHARES if mode in {"global_share", "bilateral_share"} else ABSOLUTE_COMPUTE).get(
|
||||
year, {}
|
||||
)
|
||||
from app.services.compute_configuration import DEFAULT_CONFIGURATION
|
||||
|
||||
|
||||
def _participant_value(
|
||||
@@ -85,7 +27,11 @@ def _participant_value(
|
||||
if missing:
|
||||
return None, coverage
|
||||
if mode == "bilateral_share":
|
||||
if not set(members).issubset({"CN", "US"}) or not {"CN", "US"}.issubset(observations):
|
||||
if (
|
||||
not set(members).issubset({"CN", "US"})
|
||||
or not {"CN", "US"}.issubset(observations)
|
||||
or observations["US"] + observations["CN"] <= 0
|
||||
):
|
||||
coverage.update(
|
||||
{
|
||||
"status": "incompatible",
|
||||
@@ -96,7 +42,8 @@ def _participant_value(
|
||||
return (
|
||||
round(
|
||||
sum(observations[member] for member in members)
|
||||
/ (observations["US"] + observations["CN"]) * 100,
|
||||
/ (observations["US"] + observations["CN"])
|
||||
* 100,
|
||||
2,
|
||||
),
|
||||
coverage,
|
||||
@@ -104,16 +51,32 @@ def _participant_value(
|
||||
return sum(observations[member] for member in members), coverage
|
||||
|
||||
|
||||
def compute_history(countries: list[str], mode: str = "global_share") -> dict:
|
||||
def compute_history(
|
||||
countries: list[str],
|
||||
mode: str = "global_share",
|
||||
config: ComputeConfiguration | None = None,
|
||||
) -> dict:
|
||||
"""Aggregate only fully covered, same-year compatible observations."""
|
||||
config = config if config is not None else DEFAULT_CONFIGURATION
|
||||
metric = "global_share" if mode == "bilateral_share" else mode
|
||||
annual = {row.year: row for row in config.observations if row.metric == metric}
|
||||
gaps = {row.year: row for row in config.innovation}
|
||||
observed_years = [row.year for row in config.observations] + list(gaps)
|
||||
years = tuple(
|
||||
range(
|
||||
min(observed_years, default=date.today().year),
|
||||
max([date.today().year, *observed_years]) + 1,
|
||||
)
|
||||
)
|
||||
members_by_participant = participant_members(countries)
|
||||
rows = []
|
||||
for year in HISTORY_YEARS:
|
||||
observations = _observations(mode, year)
|
||||
for year in years:
|
||||
record = annual.get(year)
|
||||
observations = record.values if record else {}
|
||||
values, coverage = {}, {}
|
||||
for code, members in members_by_participant.items():
|
||||
values[code], coverage[code] = _participant_value(members, observations, mode)
|
||||
evidence_id = f"caict-{year + 1}" if observations else None
|
||||
evidence_id = record.source_id if record else None
|
||||
rows.append(
|
||||
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
|
||||
)
|
||||
@@ -121,11 +84,11 @@ def compute_history(countries: list[str], mode: str = "global_share") -> dict:
|
||||
selected_members = {member for members in members_by_participant.values() for member in members}
|
||||
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
|
||||
return {
|
||||
"version": HISTORY_VERSION,
|
||||
"provenance": "reviewed_official_reports",
|
||||
"reviewed_at": "2026-09-22",
|
||||
"version": f"compute-history-{config.reviewed_at}",
|
||||
"provenance": "curated_reports",
|
||||
"reviewed_at": config.reviewed_at.isoformat(),
|
||||
"time_basis": "retrospective",
|
||||
"years": HISTORY_YEARS,
|
||||
"years": years,
|
||||
"mode": mode,
|
||||
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
|
||||
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
|
||||
@@ -138,18 +101,22 @@ def compute_history(countries: list[str], mode: str = "global_share") -> dict:
|
||||
"innovation": [
|
||||
{
|
||||
"year": year,
|
||||
"value": GAPS.get(year) if has_innovation_scope else None,
|
||||
"evidence_id": "innovation-2026" if year in GAPS and has_innovation_scope else None,
|
||||
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
|
||||
"evidence_id": gaps[year].source_id
|
||||
if year in gaps and has_innovation_scope
|
||||
else None,
|
||||
}
|
||||
for year in HISTORY_YEARS
|
||||
for year in years
|
||||
],
|
||||
"evidence": deepcopy(REPORTS),
|
||||
"evidence": {
|
||||
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
|
||||
},
|
||||
"innovation_scope": ["US", "CN"],
|
||||
"limitations": [
|
||||
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
|
||||
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
|
||||
"全球份额不从中美或其他局部成员反推剩余地区。",
|
||||
"年度资料来自信通院官方报告,观测年与报告年份分列;跨年方法可比性仍需核验。",
|
||||
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
|
||||
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
|
||||
],
|
||||
}
|
||||
|
||||
125
backend/app/services/compute_history_defaults.json
Normal file
125
backend/app/services/compute_history_defaults.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"reviewed_at": "2026-09-22",
|
||||
"sources": [
|
||||
{
|
||||
"id": "caict-2021",
|
||||
"title": "中国算力发展指数白皮书(2021)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2021,
|
||||
"published_at": null,
|
||||
"locator": "正文 11 页图 4(2020 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2022",
|
||||
"title": "中国算力发展指数白皮书(2022)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2022,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2021 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2023",
|
||||
"title": "中国算力发展指数白皮书(2023)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2023,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2024",
|
||||
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2024,
|
||||
"published_at": null,
|
||||
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "innovation-2026",
|
||||
"title": "全球人工智能创新指数报告 2026:科技日报报道",
|
||||
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
|
||||
"publisher": "科技日报(新浪转载)",
|
||||
"report_year": 2026,
|
||||
"published_at": "2026-07-17",
|
||||
"locator": "2023–2025 年美国减中国的指数分差"
|
||||
}
|
||||
],
|
||||
"observations": [
|
||||
{
|
||||
"year": 2020,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2021",
|
||||
"values": {
|
||||
"US": 36,
|
||||
"CN": 31,
|
||||
"JP": 6
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2021,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2022",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"US": 41,
|
||||
"CN": 31,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"CN": 302
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"CN": 435
|
||||
}
|
||||
}
|
||||
],
|
||||
"innovation": [
|
||||
{
|
||||
"year": 2023,
|
||||
"value": 22.02,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2024,
|
||||
"value": 19.96,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2025,
|
||||
"value": 17.95,
|
||||
"source_id": "innovation-2026"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -22,6 +22,12 @@ from app.services.earth_layer_adapters import (
|
||||
get_earth_update_layers_for_source,
|
||||
)
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
from app.services.earth_outbox_maintenance import (
|
||||
configured_retention_days,
|
||||
purge_consumed_events,
|
||||
release_maintenance_lock,
|
||||
try_maintenance_lock,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -30,6 +36,7 @@ DEFAULT_DEBOUNCE_SECONDS = 0.25
|
||||
DEFAULT_MAX_WAIT_SECONDS = 1.5
|
||||
DELETE_FAST_FLUSH_SECONDS = 0.05
|
||||
LISTEN_KEEPALIVE_SECONDS = 5.0
|
||||
OUTBOX_RETENTION_CHECK_SECONDS = 60.0
|
||||
OUTBOX_POLL_LIMIT = 5000
|
||||
MAX_ENTITY_KEY_SAMPLES = 20
|
||||
MAX_SEEN_EVENT_IDS = 20000
|
||||
@@ -346,6 +353,7 @@ class EarthDbChangeListener:
|
||||
self._stop_event: asyncio.Event | None = None
|
||||
self._connection: asyncpg.Connection | None = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
self._next_retention_check = 0.0
|
||||
|
||||
def start(self) -> None:
|
||||
if self._task and not self._task.done():
|
||||
@@ -385,6 +393,7 @@ class EarthDbChangeListener:
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
await self._poll_outbox()
|
||||
await self._maybe_purge_outbox()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
@@ -459,6 +468,27 @@ class EarthDbChangeListener:
|
||||
context={"events": len(rows), "accepted": accepted_count},
|
||||
)
|
||||
|
||||
async def _maybe_purge_outbox(self) -> None:
|
||||
if self._connection is None or monotonic() < self._next_retention_check:
|
||||
return
|
||||
self._next_retention_check = monotonic() + OUTBOX_RETENTION_CHECK_SECONDS
|
||||
try:
|
||||
if not await try_maintenance_lock(self._connection):
|
||||
return
|
||||
try:
|
||||
await purge_consumed_events(
|
||||
self._connection,
|
||||
retention_days=configured_retention_days(),
|
||||
)
|
||||
finally:
|
||||
await release_maintenance_lock(self._connection)
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Earth outbox retention failed",
|
||||
event="earth.outbox.retention_failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
|
||||
def _on_notification(
|
||||
self,
|
||||
_connection: asyncpg.Connection,
|
||||
|
||||
@@ -99,7 +99,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
sources=frozenset({"media_news_archive", "earth_news_items"}),
|
||||
tables=frozenset({"earth_news_items"}),
|
||||
layers=("news",),
|
||||
cache_patterns=("summary*",),
|
||||
cache_patterns=(),
|
||||
refresh_strategy="reload",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
|
||||
@@ -2,25 +2,27 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from contextlib import suppress
|
||||
from dataclasses import dataclass, replace
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import Response
|
||||
from fastapi import HTTPException, Response
|
||||
|
||||
from app.core.cache import _RedisClient
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="earth_layer_cache")
|
||||
|
||||
EARTH_LAYER_CACHE_PREFIX = "earth:layer:v1"
|
||||
EARTH_LAYER_LOCK_PREFIX = "earth:layer:lock:v1"
|
||||
DEFAULT_LOCK_TTL_SECONDS = 10
|
||||
DEFAULT_LOCK_WAIT_SECONDS = 0.2
|
||||
DEFAULT_BUILD_WAIT_SECONDS = 30
|
||||
DEFAULT_MAX_FEATURES = 5000
|
||||
DEFAULT_MAX_BYTES = 5 * 1024 * 1024
|
||||
DEFAULT_BBOX_PRECISION_DEGREES = 0.1
|
||||
@@ -35,6 +37,7 @@ class EarthLayerCachePolicy:
|
||||
max_bytes: int = DEFAULT_MAX_BYTES
|
||||
lock_ttl_seconds: int = DEFAULT_LOCK_TTL_SECONDS
|
||||
lock_wait_seconds: float = DEFAULT_LOCK_WAIT_SECONDS
|
||||
build_wait_seconds: float = DEFAULT_BUILD_WAIT_SECONDS
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -49,6 +52,9 @@ class EarthLayerCacheResult:
|
||||
class EarthLayerCache:
|
||||
def __init__(self) -> None:
|
||||
self._client = None
|
||||
self._inflight: dict[
|
||||
tuple[asyncio.AbstractEventLoop, str], asyncio.Future[EarthLayerCacheResult | None]
|
||||
] = {}
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
@@ -82,17 +88,62 @@ class EarthLayerCache:
|
||||
value = json.loads(raw)
|
||||
return value if isinstance(value, dict) else None
|
||||
|
||||
def set_json(self, key: str, payload: dict[str, Any], ttl_seconds: int) -> None:
|
||||
self.client.setex(key, ttl_seconds, json.dumps(payload, ensure_ascii=False, default=str))
|
||||
def acquire_lock(self, key: str, ttl_seconds: int) -> str | None:
|
||||
token = uuid4().hex
|
||||
return (
|
||||
token if self.client.set(self.lock_key(key), token, nx=True, ex=ttl_seconds) else None
|
||||
)
|
||||
|
||||
def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
|
||||
return bool(self.client.set(self.lock_key(key), "1", nx=True, ex=ttl_seconds))
|
||||
def renew_lock(self, key: str, token: str, ttl_seconds: int) -> bool:
|
||||
return bool(
|
||||
self.client.eval(
|
||||
"""
|
||||
if redis.call('get', KEYS[1]) == ARGV[1] then
|
||||
return redis.call('expire', KEYS[1], ARGV[2])
|
||||
end
|
||||
return 0
|
||||
""",
|
||||
1,
|
||||
self.lock_key(key),
|
||||
token,
|
||||
ttl_seconds,
|
||||
)
|
||||
)
|
||||
|
||||
def release_lock(self, key: str) -> None:
|
||||
try:
|
||||
self.client.delete(self.lock_key(key))
|
||||
except Exception:
|
||||
pass
|
||||
def release_lock(self, key: str, token: str) -> None:
|
||||
self.client.eval(
|
||||
"""
|
||||
if redis.call('get', KEYS[1]) == ARGV[1] then
|
||||
return redis.call('del', KEYS[1])
|
||||
end
|
||||
return 0
|
||||
""",
|
||||
1,
|
||||
self.lock_key(key),
|
||||
token,
|
||||
)
|
||||
|
||||
def publish(
|
||||
self, key: str, token: str, payload: dict[str, Any], policy: EarthLayerCachePolicy
|
||||
) -> bool:
|
||||
return bool(
|
||||
self.client.eval(
|
||||
"""
|
||||
if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
|
||||
redis.call('setex', KEYS[2], ARGV[2], ARGV[4])
|
||||
redis.call('setex', KEYS[3], ARGV[3], ARGV[4])
|
||||
return 1
|
||||
""",
|
||||
3,
|
||||
self.lock_key(key),
|
||||
key,
|
||||
self.stale_key(key),
|
||||
token,
|
||||
policy.fresh_ttl_seconds,
|
||||
policy.stale_ttl_seconds,
|
||||
json.dumps(payload, ensure_ascii=False, default=str),
|
||||
)
|
||||
)
|
||||
|
||||
def delete_pattern(self, pattern: str = f"{EARTH_LAYER_CACHE_PREFIX}:*") -> int:
|
||||
keys = list(self.client.scan_iter(match=pattern))
|
||||
@@ -169,56 +220,122 @@ async def resolve_layer_payload(
|
||||
policy: EarthLayerCachePolicy,
|
||||
builder: Callable[[], Awaitable[dict[str, Any]]],
|
||||
) -> EarthLayerCacheResult:
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
cached = earth_layer_cache.get_json(key)
|
||||
if cached is not None:
|
||||
return _result(cached, state="hit", key=key)
|
||||
|
||||
lock_acquired = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
|
||||
if lock_acquired:
|
||||
try:
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
_write_fresh_and_stale(key, payload, policy)
|
||||
_log_cache_event("refresh", key, payload, started)
|
||||
return _result(payload, state="refresh", key=key)
|
||||
except Exception as exc:
|
||||
stale = _read_stale(key)
|
||||
if stale is not None:
|
||||
logger.warning_event(
|
||||
"Earth layer cache builder failed; returning stale payload",
|
||||
event="earth_layer_cache.stale_after_builder_error",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
return _result(stale, state="stale", key=key)
|
||||
raise
|
||||
finally:
|
||||
earth_layer_cache.release_lock(key)
|
||||
|
||||
flight_key = (asyncio.get_running_loop(), key)
|
||||
while flight_key in earth_layer_cache._inflight:
|
||||
stale = _read_stale(key)
|
||||
if stale is not None:
|
||||
return _result(stale, state="stale", key=key)
|
||||
future = earth_layer_cache._inflight[flight_key]
|
||||
done, _ = await asyncio.wait({future}, timeout=policy.build_wait_seconds)
|
||||
if not done:
|
||||
raise HTTPException(503, "Earth layer cache rebuild is still in progress")
|
||||
result = future.result()
|
||||
if result is not None:
|
||||
state = "hit" if result.state == "refresh" else result.state
|
||||
return replace(result, state=state)
|
||||
|
||||
await asyncio.sleep(policy.lock_wait_seconds)
|
||||
cached_after_wait = earth_layer_cache.get_json(key)
|
||||
if cached_after_wait is not None:
|
||||
return _result(cached_after_wait, state="hit", key=key)
|
||||
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
_log_cache_event("miss", key, payload, started)
|
||||
return _result(payload, state="miss", key=key)
|
||||
future = asyncio.get_running_loop().create_future()
|
||||
earth_layer_cache._inflight[flight_key] = future
|
||||
try:
|
||||
result = await _resolve_layer_payload(key=key, policy=policy, builder=builder)
|
||||
future.set_result(result)
|
||||
return result
|
||||
except asyncio.CancelledError:
|
||||
# A waiting request can take over using its own still-open DB session.
|
||||
future.set_result(None)
|
||||
raise
|
||||
except Exception as exc:
|
||||
future.set_exception(exc)
|
||||
future.exception()
|
||||
raise
|
||||
finally:
|
||||
earth_layer_cache._inflight.pop(flight_key)
|
||||
|
||||
|
||||
async def _resolve_layer_payload(
|
||||
*,
|
||||
key: str,
|
||||
policy: EarthLayerCachePolicy,
|
||||
builder: Callable[[], Awaitable[dict[str, Any]]],
|
||||
) -> EarthLayerCacheResult:
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
async with asyncio.timeout(policy.build_wait_seconds):
|
||||
while True:
|
||||
cached = earth_layer_cache.get_json(key)
|
||||
if cached is not None:
|
||||
return _result(cached, state="hit", key=key)
|
||||
token = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
|
||||
if token is not None:
|
||||
break
|
||||
stale = _read_stale(key)
|
||||
if stale is not None:
|
||||
return _result(stale, state="stale", key=key)
|
||||
await asyncio.sleep(policy.lock_wait_seconds)
|
||||
except TimeoutError:
|
||||
raise HTTPException(503, "Earth layer cache rebuild is still in progress") from None
|
||||
except Exception as exc:
|
||||
try:
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
except Exception:
|
||||
raise exc
|
||||
logger.warning_event(
|
||||
"Earth layer cache bypassed",
|
||||
event="earth_layer_cache.bypass",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
return _result(payload, state="bypass", key=key)
|
||||
|
||||
renewal = asyncio.create_task(_renew_cache_lock(key, token, policy))
|
||||
try:
|
||||
try:
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
except Exception as exc:
|
||||
stale = _read_stale(key)
|
||||
if stale is None:
|
||||
raise
|
||||
logger.warning_event(
|
||||
"Earth layer cache builder failed; returning stale payload",
|
||||
event="earth_layer_cache.stale_after_builder_error",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
return _result(stale, state="stale", key=key)
|
||||
try:
|
||||
published = earth_layer_cache.publish(key, token, payload, policy)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth layer cache publish failed",
|
||||
event="earth_layer_cache.bypass",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
published = False
|
||||
state = "refresh" if published else "bypass"
|
||||
_log_cache_event(state, key, payload, started)
|
||||
return _result(payload, state=state, key=key)
|
||||
finally:
|
||||
renewal.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await renewal
|
||||
try:
|
||||
earth_layer_cache.release_lock(key, token)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth layer cache lock release failed",
|
||||
event="earth_layer_cache.lock_release_failed",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def _renew_cache_lock(key: str, token: str, policy: EarthLayerCachePolicy) -> None:
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(policy.lock_ttl_seconds / 3)
|
||||
if not earth_layer_cache.renew_lock(key, token, policy.lock_ttl_seconds):
|
||||
raise RuntimeError("Earth layer cache lock ownership lost")
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth layer cache lock renewal stopped",
|
||||
event="earth_layer_cache.lock_renewal_failed",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) -> dict[str, Any]:
|
||||
budgeted = _truncate_features(payload, policy.max_features, "feature_budget")
|
||||
@@ -286,11 +403,6 @@ async def _build_budgeted_payload(
|
||||
return apply_payload_budget(payload, policy)
|
||||
|
||||
|
||||
def _write_fresh_and_stale(key: str, payload: dict[str, Any], policy: EarthLayerCachePolicy) -> None:
|
||||
earth_layer_cache.set_json(key, payload, policy.fresh_ttl_seconds)
|
||||
earth_layer_cache.set_json(earth_layer_cache.stale_key(key), payload, policy.stale_ttl_seconds)
|
||||
|
||||
|
||||
def _read_stale(key: str) -> dict[str, Any] | None:
|
||||
try:
|
||||
return earth_layer_cache.get_json(earth_layer_cache.stale_key(key))
|
||||
@@ -359,14 +471,16 @@ def _payload_size(payload: dict[str, Any]) -> int:
|
||||
|
||||
def _safe_key_part(value: Any) -> str:
|
||||
raw = str(value).strip().lower()
|
||||
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[:160]
|
||||
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[
|
||||
:160
|
||||
]
|
||||
|
||||
|
||||
def _layer_from_key(key: str) -> str:
|
||||
prefix = f"{EARTH_LAYER_CACHE_PREFIX}:"
|
||||
if not key.startswith(prefix):
|
||||
return "unknown"
|
||||
remainder = key[len(prefix):]
|
||||
remainder = key[len(prefix) :]
|
||||
return remainder.split(":", 1)[0]
|
||||
|
||||
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import codecs
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
@@ -1638,10 +1639,15 @@ async def record_earth_news_sources_health(
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def get_configured_sources_for_region(db: AsyncSession | None, region: str) -> list[NewsFeedSource]:
|
||||
async def get_configured_sources_for_region(
|
||||
db: AsyncSession | None,
|
||||
region: str,
|
||||
*,
|
||||
config_payload: dict[str, Any] | None = None,
|
||||
) -> list[NewsFeedSource]:
|
||||
if db is None:
|
||||
return get_sources_for_region(region)
|
||||
payload = await get_earth_news_sources_payload(db)
|
||||
payload = config_payload or await get_earth_news_sources_payload(db)
|
||||
sources = [
|
||||
source
|
||||
for raw in payload.get("sources", [])
|
||||
@@ -1800,6 +1806,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _decode_feed_response(response: httpx.Response) -> str:
|
||||
content = response.content
|
||||
for marker, encoding in (
|
||||
(codecs.BOM_UTF32_LE, "utf-32"),
|
||||
(codecs.BOM_UTF32_BE, "utf-32"),
|
||||
(codecs.BOM_UTF16_LE, "utf-16"),
|
||||
(codecs.BOM_UTF16_BE, "utf-16"),
|
||||
(codecs.BOM_UTF8, "utf-8-sig"),
|
||||
):
|
||||
if content.startswith(marker):
|
||||
return content.decode(encoding)
|
||||
|
||||
declaration = re.match(
|
||||
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
|
||||
content,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
encoding = response.charset_encoding or (
|
||||
declaration.group(1).decode("ascii") if declaration else "utf-8"
|
||||
)
|
||||
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
|
||||
return content.decode(encoding)
|
||||
|
||||
|
||||
def _parse_feed_entries(
|
||||
xml_text: str,
|
||||
source: NewsFeedSource,
|
||||
@@ -2453,7 +2483,9 @@ async def _fetch_single_feed_url(
|
||||
feed=feed,
|
||||
)
|
||||
try:
|
||||
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
|
||||
items = _parse_feed_entries(
|
||||
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
|
||||
)
|
||||
except Exception as exc:
|
||||
error, status = _format_source_fetch_error(
|
||||
status_code=status_code,
|
||||
@@ -2461,7 +2493,7 @@ async def _fetch_single_feed_url(
|
||||
body=response.text,
|
||||
parsed_count=0,
|
||||
)
|
||||
if error is None:
|
||||
if status == "empty":
|
||||
error = str(exc)
|
||||
status = "format_error"
|
||||
return source, [], error, _source_health_result(
|
||||
@@ -2704,7 +2736,6 @@ async def get_earth_news_payload(
|
||||
del provider_client
|
||||
active_region = region if region in REGION_ANCHORS else determine_focus_region(lat, lon)
|
||||
has_settings_db = db is not None and callable(getattr(db, "execute", None))
|
||||
source_config_payload = await get_earth_news_sources_payload(db) if has_settings_db else None
|
||||
sources = (
|
||||
await get_configured_sources_for_region(db, active_region)
|
||||
if has_settings_db
|
||||
@@ -2723,39 +2754,7 @@ async def get_earth_news_payload(
|
||||
locale=locale,
|
||||
)
|
||||
|
||||
from app.services.earth_news_store import (
|
||||
get_earth_news_feed_coverage,
|
||||
get_earth_news_freshness,
|
||||
list_earth_news_cruise_items,
|
||||
list_earth_news_items,
|
||||
upsert_earth_news_items,
|
||||
)
|
||||
|
||||
errors: list[str] = []
|
||||
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
|
||||
should_supplement = _needs_rss_supplement(item_count=item_count, newest_at=newest_at)
|
||||
if not should_supplement and callable(getattr(db, "execute", None)):
|
||||
expected_feed_keys = _expected_feed_keys(sources, active_region=active_region)
|
||||
if expected_feed_keys:
|
||||
covered_feed_keys = await get_earth_news_feed_coverage(
|
||||
db,
|
||||
active_region=active_region,
|
||||
recent_after=datetime.now(UTC) - timedelta(seconds=RSS_SUPPLEMENT_MAX_AGE_SECONDS),
|
||||
)
|
||||
should_supplement = bool(expected_feed_keys - covered_feed_keys)
|
||||
if should_supplement:
|
||||
if source_config_payload is None:
|
||||
fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(sources, active_region=active_region)
|
||||
else:
|
||||
fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(
|
||||
sources,
|
||||
active_region=active_region,
|
||||
config_payload=source_config_payload,
|
||||
)
|
||||
await record_earth_news_sources_health(db, health_by_source)
|
||||
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
await _enqueue_unverified_locations(ranked_fetched_items)
|
||||
await upsert_earth_news_items(db, ranked_fetched_items)
|
||||
from app.services.earth_news_store import list_earth_news_cruise_items, list_earth_news_items
|
||||
|
||||
items = await _call_store_list_items(
|
||||
list_earth_news_items,
|
||||
@@ -2765,7 +2764,7 @@ async def get_earth_news_payload(
|
||||
categories=categories,
|
||||
source_ids=source_ids,
|
||||
)
|
||||
if not source_ids:
|
||||
if items and not source_ids:
|
||||
ready_sources = {
|
||||
_news_item_source_id(item)
|
||||
for item in items
|
||||
@@ -2806,10 +2805,6 @@ async def get_earth_news_payload(
|
||||
)
|
||||
else:
|
||||
cruise_items = items
|
||||
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
|
||||
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
|
||||
stale = bool(errors and items)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
@@ -2817,8 +2812,8 @@ async def get_earth_news_payload(
|
||||
items=items,
|
||||
cruise_items=cruise_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=stale,
|
||||
errors=[],
|
||||
stale=False,
|
||||
categories=categories,
|
||||
source_ids=source_ids,
|
||||
limit=limit,
|
||||
|
||||
480
backend/app/services/earth_news_refresh.py
Normal file
480
backend/app/services/earth_news_refresh.py
Normal file
@@ -0,0 +1,480 @@
|
||||
"""Periodic RSS supplementation for the read-only Earth news feed."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from contextlib import suppress
|
||||
from dataclasses import replace
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from hashlib import sha256
|
||||
import json
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.cache import _RedisClient
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.services import earth_news
|
||||
from app.services.earth_news_store import (
|
||||
get_earth_news_feed_coverage,
|
||||
get_earth_news_freshness,
|
||||
upsert_earth_news_items,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__, service="earth_news")
|
||||
|
||||
REFRESH_INTERVAL_SECONDS = 10 * 60
|
||||
REFRESH_LOCK_TTL_SECONDS = 120
|
||||
REFRESH_LOCK_RENEW_SECONDS = 30
|
||||
SOURCE_FAILURE_COOLDOWN_SECONDS = 30 * 60
|
||||
REFRESH_LOCK_PREFIX = "earth:news:rss-refresh:v1"
|
||||
SOURCE_COOLDOWN_PREFIX = "earth:news:rss-cooldown:v1"
|
||||
REFRESH_STATUS_PREFIX = "earth:news:rss-refresh-status:v1"
|
||||
REFRESH_ACTIVE_PREFIX = "earth:news:rss-refresh-active:v1"
|
||||
REFRESH_STATUS_TTL_SECONDS = 10 * 60
|
||||
REFRESH_ACTIVE_TTL_SECONDS = 120
|
||||
REFRESH_STATUS_HEARTBEAT_SECONDS = 30
|
||||
|
||||
|
||||
def _refresh_status_key(refresh_id: str) -> str:
|
||||
return f"{REFRESH_STATUS_PREFIX}:{refresh_id}"
|
||||
|
||||
|
||||
def _refresh_active_key(region: str) -> str:
|
||||
return f"{REFRESH_ACTIVE_PREFIX}:{region}"
|
||||
|
||||
|
||||
def create_earth_news_refresh_status(region: str) -> tuple[dict[str, Any], bool]:
|
||||
"""Create one per-region active refresh status, or return the current one."""
|
||||
refresh_id = uuid4().hex
|
||||
status = {"refresh_id": refresh_id, "region": region, "status": "pending"}
|
||||
script = """
|
||||
local active_id = redis.call('get', KEYS[1])
|
||||
if active_id then
|
||||
local active_status = redis.call('get', ARGV[5] .. active_id)
|
||||
if active_status then
|
||||
local decoded = cjson.decode(active_status)
|
||||
if decoded.status == 'pending' or decoded.status == 'running' then
|
||||
return {active_id, active_status, 0}
|
||||
end
|
||||
end
|
||||
redis.call('del', KEYS[1])
|
||||
end
|
||||
redis.call('set', KEYS[1], ARGV[1], 'EX', ARGV[3])
|
||||
redis.call('set', KEYS[2], ARGV[2], 'EX', ARGV[4])
|
||||
return {ARGV[1], ARGV[2], 1}
|
||||
"""
|
||||
raw = _RedisClient.get_client().eval(
|
||||
script,
|
||||
2,
|
||||
_refresh_active_key(region),
|
||||
_refresh_status_key(refresh_id),
|
||||
refresh_id,
|
||||
json.dumps(status, separators=(",", ":")),
|
||||
REFRESH_ACTIVE_TTL_SECONDS,
|
||||
REFRESH_STATUS_TTL_SECONDS,
|
||||
f"{REFRESH_STATUS_PREFIX}:",
|
||||
)
|
||||
_, encoded, created = raw
|
||||
payload = json.loads(encoded)
|
||||
return payload, bool(int(created))
|
||||
|
||||
|
||||
def set_earth_news_refresh_status(
|
||||
refresh_id: str,
|
||||
region: str,
|
||||
status: str,
|
||||
*,
|
||||
error: dict[str, str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
payload: dict[str, Any] = {
|
||||
"refresh_id": refresh_id,
|
||||
"region": region,
|
||||
"status": status,
|
||||
}
|
||||
if error:
|
||||
payload["error"] = error
|
||||
script = """
|
||||
if redis.call('get', KEYS[1]) ~= ARGV[2] then return 0 end
|
||||
redis.call('set', KEYS[2], ARGV[1], 'EX', ARGV[3])
|
||||
if ARGV[4] == 'succeeded' or ARGV[4] == 'failed' then
|
||||
redis.call('del', KEYS[1])
|
||||
else
|
||||
redis.call('expire', KEYS[1], ARGV[5])
|
||||
end
|
||||
return 1
|
||||
"""
|
||||
_RedisClient.get_client().eval(
|
||||
script,
|
||||
2,
|
||||
_refresh_active_key(region),
|
||||
_refresh_status_key(refresh_id),
|
||||
json.dumps(payload, separators=(",", ":")),
|
||||
refresh_id,
|
||||
REFRESH_STATUS_TTL_SECONDS,
|
||||
status,
|
||||
REFRESH_ACTIVE_TTL_SECONDS,
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def touch_earth_news_refresh_status(refresh_id: str, region: str) -> bool:
|
||||
script = """
|
||||
if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
|
||||
redis.call('expire', KEYS[1], ARGV[2])
|
||||
redis.call('expire', KEYS[2], ARGV[3])
|
||||
return 1
|
||||
"""
|
||||
return bool(
|
||||
_RedisClient.get_client().eval(
|
||||
script,
|
||||
2,
|
||||
_refresh_active_key(region),
|
||||
_refresh_status_key(refresh_id),
|
||||
refresh_id,
|
||||
REFRESH_ACTIVE_TTL_SECONDS,
|
||||
REFRESH_STATUS_TTL_SECONDS,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def get_earth_news_refresh_status(refresh_id: str) -> dict[str, Any] | None:
|
||||
raw = _RedisClient.get_client().get(_refresh_status_key(refresh_id))
|
||||
if not raw:
|
||||
return None
|
||||
value = json.loads(raw)
|
||||
if not isinstance(value, dict):
|
||||
return None
|
||||
if value.get("status") in {"pending", "running"}:
|
||||
region = str(value.get("region") or "")
|
||||
active_id = _RedisClient.get_client().get(_refresh_active_key(region)) if region else None
|
||||
if active_id != refresh_id:
|
||||
return {
|
||||
**value,
|
||||
"status": "failed",
|
||||
"error": {
|
||||
"code": "expired",
|
||||
"message": "Refresh status expired. Submit a new refresh.",
|
||||
},
|
||||
}
|
||||
return value
|
||||
|
||||
|
||||
def _lock_key(region: str) -> str:
|
||||
return f"{REFRESH_LOCK_PREFIX}:{region}"
|
||||
|
||||
|
||||
def _cooldown_key(region: str, source_id: str, feed_id: str) -> str:
|
||||
identity = f"{region}\0{source_id}\0{feed_id}".encode("utf-8")
|
||||
return f"{SOURCE_COOLDOWN_PREFIX}:{sha256(identity).hexdigest()}"
|
||||
|
||||
|
||||
def _source_failure_cooldown_seconds(source: earth_news.NewsFeedSource | None) -> int:
|
||||
if source is None:
|
||||
return SOURCE_FAILURE_COOLDOWN_SECONDS
|
||||
configured_minutes = source.health_policy.get("cooldown_minutes")
|
||||
try:
|
||||
return max(60, int(float(configured_minutes) * 60))
|
||||
except TypeError, ValueError:
|
||||
return SOURCE_FAILURE_COOLDOWN_SECONDS
|
||||
|
||||
|
||||
def _select_feeds(
|
||||
sources: list[earth_news.NewsFeedSource],
|
||||
*,
|
||||
active_region: str,
|
||||
required_keys: set[tuple[str, str]],
|
||||
redis_client: Any | None,
|
||||
) -> tuple[list[earth_news.NewsFeedSource], set[tuple[str, str]]]:
|
||||
selected_sources: list[earth_news.NewsFeedSource] = []
|
||||
cooling_keys: set[tuple[str, str]] = set()
|
||||
for source in sources:
|
||||
selected_feeds = []
|
||||
source_cooling = False
|
||||
if redis_client is not None:
|
||||
try:
|
||||
source_cooling = bool(
|
||||
redis_client.exists(_cooldown_key(active_region, source.id, "__source__"))
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news source cooldown lookup failed",
|
||||
event="earth_news.rss.cooldown_lookup_failed",
|
||||
context={"region": active_region, "source_id": source.id, "error": str(exc)},
|
||||
)
|
||||
for feed in earth_news._source_feeds(source):
|
||||
feed_key = (source.id, feed.id)
|
||||
if feed_key not in required_keys:
|
||||
continue
|
||||
if source_cooling:
|
||||
cooling_keys.add(feed_key)
|
||||
continue
|
||||
if redis_client is not None:
|
||||
try:
|
||||
if redis_client.exists(_cooldown_key(active_region, *feed_key)):
|
||||
cooling_keys.add(feed_key)
|
||||
continue
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news source cooldown lookup failed",
|
||||
event="earth_news.rss.cooldown_lookup_failed",
|
||||
context={
|
||||
"region": active_region,
|
||||
"source_id": source.id,
|
||||
"error": str(exc),
|
||||
},
|
||||
)
|
||||
selected_feeds.append(feed)
|
||||
if selected_feeds:
|
||||
selected_sources.append(
|
||||
replace(
|
||||
source,
|
||||
feed_url=selected_feeds[0].url,
|
||||
feed_urls=tuple(feed.url for feed in selected_feeds),
|
||||
feeds=tuple(selected_feeds),
|
||||
)
|
||||
)
|
||||
return selected_sources, cooling_keys
|
||||
|
||||
|
||||
def _record_failed_feed_cooldowns(
|
||||
redis_client: Any | None,
|
||||
*,
|
||||
active_region: str,
|
||||
health_by_source: dict[str, dict[str, Any]],
|
||||
sources: list[earth_news.NewsFeedSource],
|
||||
) -> None:
|
||||
if redis_client is None:
|
||||
return
|
||||
cooldown_by_source = {source.id: source for source in sources}
|
||||
for source_id, health in health_by_source.items():
|
||||
feed_results = health.get("feed_results")
|
||||
if isinstance(feed_results, list) and feed_results:
|
||||
failed_feed_ids = [
|
||||
str(result.get("feed_id") or "")
|
||||
for result in feed_results
|
||||
if isinstance(result, dict) and not result.get("ok") and result.get("feed_id")
|
||||
]
|
||||
else:
|
||||
failed_feed_ids = [str(health.get("feed_id") or "__source__")]
|
||||
if health.get("ok"):
|
||||
failed_feed_ids = []
|
||||
for feed_id in failed_feed_ids:
|
||||
try:
|
||||
redis_client.set(
|
||||
_cooldown_key(active_region, source_id, feed_id),
|
||||
"1",
|
||||
ex=_source_failure_cooldown_seconds(cooldown_by_source.get(source_id)),
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news source failure cooldown write failed",
|
||||
event="earth_news.rss.cooldown_write_failed",
|
||||
context={"region": active_region, "source_id": source_id, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def _renew_lock(redis_client: Any, key: str, token: str, lease_lost: asyncio.Event) -> None:
|
||||
renew_script = """
|
||||
if redis.call('get', KEYS[1]) == ARGV[1] then
|
||||
return redis.call('expire', KEYS[1], ARGV[2])
|
||||
end
|
||||
return 0
|
||||
"""
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(REFRESH_LOCK_RENEW_SECONDS)
|
||||
if not redis_client.eval(renew_script, 1, key, token, REFRESH_LOCK_TTL_SECONDS):
|
||||
lease_lost.set()
|
||||
return
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
lease_lost.set()
|
||||
logger.warning_event(
|
||||
"Earth news refresh lock renewal failed",
|
||||
event="earth_news.rss.refresh_lock_renewal_failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def refresh_earth_news_region(active_region: str, *, force: bool = False) -> dict[str, Any]:
|
||||
"""Fetch only stale or uncovered feeds for one region, outside the request path."""
|
||||
if active_region not in earth_news.REGION_ANCHORS:
|
||||
raise ValueError(f"Unsupported Earth news region: {active_region}")
|
||||
|
||||
lock_key = _lock_key(active_region)
|
||||
token = uuid4().hex
|
||||
redis_client = None
|
||||
lock_acquired = False
|
||||
lock_available = False
|
||||
try:
|
||||
redis_client = _RedisClient.get_client()
|
||||
lock_acquired = bool(
|
||||
redis_client.set(lock_key, token, nx=True, ex=REFRESH_LOCK_TTL_SECONDS)
|
||||
)
|
||||
lock_available = True
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news refresh coordination unavailable",
|
||||
event="earth_news.rss.refresh_lock_unavailable",
|
||||
context={"region": active_region, "error": str(exc)},
|
||||
)
|
||||
|
||||
if not lock_available or not lock_acquired:
|
||||
state = "busy" if lock_available else "coordination_unavailable"
|
||||
return {"state": state, "fetched": 0, "upserted": 0}
|
||||
|
||||
lease_lost = asyncio.Event()
|
||||
renewal: asyncio.Task[None] | None = None
|
||||
if lock_acquired and redis_client is not None:
|
||||
renewal = asyncio.create_task(_renew_lock(redis_client, lock_key, token, lease_lost))
|
||||
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
source_config = await earth_news.get_earth_news_sources_payload(db)
|
||||
sources = await earth_news.get_configured_sources_for_region(
|
||||
db, active_region, config_payload=source_config
|
||||
)
|
||||
if not sources:
|
||||
return {
|
||||
"state": "failed",
|
||||
"error_code": "source_unavailable",
|
||||
"fetched": 0,
|
||||
"upserted": 0,
|
||||
}
|
||||
|
||||
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
|
||||
expected_keys = earth_news._expected_feed_keys(sources, active_region=active_region)
|
||||
if not expected_keys:
|
||||
return {
|
||||
"state": "failed",
|
||||
"error_code": "source_unavailable",
|
||||
"fetched": 0,
|
||||
"upserted": 0,
|
||||
}
|
||||
|
||||
now = datetime.now(UTC)
|
||||
needs_full_refresh = force or earth_news._needs_rss_supplement(
|
||||
item_count=item_count, newest_at=newest_at
|
||||
)
|
||||
covered_keys = await get_earth_news_feed_coverage(
|
||||
db,
|
||||
active_region=active_region,
|
||||
recent_after=now - timedelta(seconds=earth_news.RSS_SUPPLEMENT_MAX_AGE_SECONDS),
|
||||
)
|
||||
required_keys = expected_keys if needs_full_refresh else expected_keys - covered_keys
|
||||
if not required_keys:
|
||||
return {
|
||||
"state": "completed",
|
||||
"successful_sources": 0,
|
||||
"failed_sources": 0,
|
||||
"fetched": 0,
|
||||
"upserted": 0,
|
||||
}
|
||||
|
||||
selected_sources, cooling_keys = _select_feeds(
|
||||
sources,
|
||||
active_region=active_region,
|
||||
required_keys=required_keys,
|
||||
redis_client=redis_client,
|
||||
)
|
||||
if cooling_keys:
|
||||
logger.info_event(
|
||||
"Skipping failed Earth news feeds during cooldown",
|
||||
event="earth_news.rss.cooldown_active",
|
||||
context={"region": active_region, "feed_count": len(cooling_keys)},
|
||||
)
|
||||
if not selected_sources:
|
||||
return {
|
||||
"state": "failed",
|
||||
"error_code": "source_unavailable",
|
||||
"fetched": 0,
|
||||
"upserted": 0,
|
||||
}
|
||||
|
||||
fetched_items, _errors, health_by_source = (
|
||||
await earth_news._fetch_rss_items_for_sources(
|
||||
selected_sources,
|
||||
active_region=active_region,
|
||||
config_payload=source_config,
|
||||
)
|
||||
)
|
||||
if lease_lost.is_set():
|
||||
await db.rollback()
|
||||
return {"state": "busy", "fetched": 0, "upserted": 0}
|
||||
_record_failed_feed_cooldowns(
|
||||
redis_client,
|
||||
active_region=active_region,
|
||||
health_by_source=health_by_source,
|
||||
sources=selected_sources,
|
||||
)
|
||||
await earth_news.record_earth_news_sources_health(db, health_by_source)
|
||||
if lease_lost.is_set():
|
||||
await db.rollback()
|
||||
return {"state": "busy", "fetched": 0, "upserted": 0}
|
||||
ranked_items = earth_news._rank_and_trim_items(
|
||||
fetched_items, active_region=active_region
|
||||
)
|
||||
await earth_news._enqueue_unverified_locations(ranked_items)
|
||||
upserted = await upsert_earth_news_items(db, ranked_items)
|
||||
if lease_lost.is_set():
|
||||
await db.rollback()
|
||||
return {"state": "busy", "fetched": len(fetched_items), "upserted": 0}
|
||||
await db.commit()
|
||||
successful_sources = sum(bool(health.get("ok")) for health in health_by_source.values())
|
||||
failed_sources = max(0, len(selected_sources) - successful_sources)
|
||||
state = "completed" if successful_sources else "failed"
|
||||
failed_statuses = {
|
||||
str(health.get("status") or "")
|
||||
for health in health_by_source.values()
|
||||
if not health.get("ok")
|
||||
}
|
||||
error_code = None
|
||||
if not successful_sources:
|
||||
error_code = "timeout" if failed_statuses == {"timeout"} else "source_unavailable"
|
||||
return {
|
||||
"state": state,
|
||||
"error_code": error_code,
|
||||
"successful_sources": successful_sources,
|
||||
"failed_sources": failed_sources,
|
||||
"fetched": len(fetched_items),
|
||||
"upserted": upserted,
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news periodic RSS refresh failed",
|
||||
event="earth_news.rss.refresh_failed",
|
||||
context={"region": active_region, "error": str(exc)},
|
||||
)
|
||||
raise
|
||||
finally:
|
||||
if renewal is not None:
|
||||
renewal.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await renewal
|
||||
if lock_acquired and redis_client is not None:
|
||||
release_script = """
|
||||
if redis.call('get', KEYS[1]) == ARGV[1] then
|
||||
return redis.call('del', KEYS[1])
|
||||
end
|
||||
return 0
|
||||
"""
|
||||
try:
|
||||
redis_client.eval(release_script, 1, lock_key, token)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news refresh lock release failed",
|
||||
event="earth_news.rss.refresh_lock_release_failed",
|
||||
context={"region": active_region, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def refresh_earth_news_all_regions() -> None:
|
||||
for active_region in earth_news.REGION_ANCHORS:
|
||||
try:
|
||||
await refresh_earth_news_region(active_region)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
continue
|
||||
@@ -278,22 +278,25 @@ async def get_earth_news_feed_coverage(
|
||||
active_region: str,
|
||||
recent_after: datetime | None = None,
|
||||
) -> set[tuple[str, str]]:
|
||||
query = select(
|
||||
EarthNewsItem.id,
|
||||
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"),
|
||||
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"),
|
||||
news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
|
||||
source_id = func.coalesce(
|
||||
func.nullif(news_meta.op("->>")("source_id"), ""),
|
||||
func.split_part(EarthNewsItem.id, ":", 1),
|
||||
)
|
||||
feed_id = func.nullif(news_meta.op("->>")("feed_id"), "")
|
||||
query = select(
|
||||
source_id,
|
||||
feed_id,
|
||||
).where(source_id != "", feed_id.is_not(None)).distinct()
|
||||
if active_region != "global":
|
||||
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
|
||||
if recent_after is not None:
|
||||
query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after)
|
||||
result = await db.execute(query)
|
||||
coverage: set[tuple[str, str]] = set()
|
||||
for item_id, source_id, feed_id in result.all():
|
||||
for source_id, feed_id in result.all():
|
||||
normalized_source_id = str(source_id or "").strip()
|
||||
normalized_feed_id = str(feed_id or "").strip()
|
||||
if not normalized_source_id and isinstance(item_id, str) and ":" in item_id:
|
||||
normalized_source_id = item_id.split(":", 1)[0]
|
||||
if normalized_source_id and normalized_feed_id:
|
||||
coverage.add((normalized_source_id, normalized_feed_id))
|
||||
return coverage
|
||||
@@ -340,30 +343,49 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
|
||||
changed += 1
|
||||
continue
|
||||
|
||||
record.title = item.title
|
||||
record.summary = item.summary
|
||||
record.url = item.url
|
||||
record.source = item.source
|
||||
record.feed_name = item.feed_name
|
||||
record.region = item.feed_region
|
||||
record.homepage_url = item.homepage_url
|
||||
record.published_at = item.published_at
|
||||
visible_changed = False
|
||||
for field_name, value in (
|
||||
("title", item.title),
|
||||
("summary", item.summary),
|
||||
("url", item.url),
|
||||
("source", item.source),
|
||||
("feed_name", item.feed_name),
|
||||
("region", item.feed_region),
|
||||
("homepage_url", item.homepage_url),
|
||||
("published_at", item.published_at),
|
||||
):
|
||||
if getattr(record, field_name) != value:
|
||||
setattr(record, field_name, value)
|
||||
visible_changed = True
|
||||
record.last_seen_at = now
|
||||
location_meta = dict(record.location_meta or {})
|
||||
location_meta["news_meta"] = _news_meta_patch(item)
|
||||
record.location_meta = location_meta
|
||||
news_meta = _news_meta_patch(item)
|
||||
if location_meta.get("news_meta") != news_meta:
|
||||
location_meta["news_meta"] = news_meta
|
||||
record.location_meta = location_meta
|
||||
visible_changed = True
|
||||
if item.localizations:
|
||||
merged_localizations = {
|
||||
**dict(record.localizations or {}),
|
||||
**dict(item.localizations or {}),
|
||||
}
|
||||
record.content_language = item.content_language
|
||||
record.localizations = merged_localizations
|
||||
if record.content_language != item.content_language:
|
||||
record.content_language = item.content_language
|
||||
visible_changed = True
|
||||
if dict(record.localizations or {}) != merged_localizations:
|
||||
record.localizations = merged_localizations
|
||||
visible_changed = True
|
||||
if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at:
|
||||
record.enrichment_status = item.enrichment_status
|
||||
record.enrichment_error = item.enrichment_error
|
||||
record.enriched_at = item.enriched_at
|
||||
changed += 1
|
||||
for field_name, value in (
|
||||
("enrichment_status", item.enrichment_status),
|
||||
("enrichment_error", item.enrichment_error),
|
||||
("enriched_at", item.enriched_at),
|
||||
):
|
||||
if getattr(record, field_name) != value:
|
||||
setattr(record, field_name, value)
|
||||
visible_changed = True
|
||||
if visible_changed:
|
||||
changed += 1
|
||||
await db.flush()
|
||||
return changed
|
||||
|
||||
|
||||
184
backend/app/services/earth_outbox_maintenance.py
Normal file
184
backend/app/services/earth_outbox_maintenance.py
Normal file
@@ -0,0 +1,184 @@
|
||||
"""Bounded retention for consumed Earth change events."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import AsyncContextManager, Protocol
|
||||
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
INDEX_NAME = "idx_earth_data_change_events_consumed_retention"
|
||||
DEFAULT_RETENTION_DAYS = 1
|
||||
DEFAULT_BATCH_SIZE = 1000
|
||||
DEFAULT_PAUSE_SECONDS = 0.1
|
||||
MAINTENANCE_LOCK_KEY = (17001613, 1)
|
||||
|
||||
DELETE_BATCH_SQL = """
|
||||
WITH doomed AS (
|
||||
SELECT id
|
||||
FROM earth_data_change_events
|
||||
WHERE consumed_at IS NOT NULL AND consumed_at < $1
|
||||
ORDER BY consumed_at, id
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
), deleted AS (
|
||||
DELETE FROM earth_data_change_events AS events
|
||||
USING doomed
|
||||
WHERE events.id = doomed.id
|
||||
RETURNING events.id
|
||||
)
|
||||
SELECT count(*) FROM deleted
|
||||
"""
|
||||
|
||||
|
||||
class OutboxConnection(Protocol):
|
||||
async def fetchval(self, query: str, *args: object) -> object: ...
|
||||
|
||||
async def fetchrow(self, query: str, *args: object) -> object: ...
|
||||
|
||||
async def execute(self, query: str, *args: object) -> object: ...
|
||||
|
||||
def transaction(self) -> AsyncContextManager[object]: ...
|
||||
|
||||
|
||||
def configured_retention_days() -> int:
|
||||
raw = os.getenv("PLANET_EARTH_OUTBOX_RETENTION_DAYS", str(DEFAULT_RETENTION_DAYS))
|
||||
try:
|
||||
days = int(raw)
|
||||
if 1 <= days <= 3650:
|
||||
return days
|
||||
except ValueError:
|
||||
pass
|
||||
logger.warning_event(
|
||||
"Invalid Earth outbox retention; using default",
|
||||
event="earth.outbox.retention_invalid",
|
||||
context={"value": raw},
|
||||
)
|
||||
return DEFAULT_RETENTION_DAYS
|
||||
|
||||
|
||||
async def try_maintenance_lock(conn: OutboxConnection) -> bool:
|
||||
return bool(await conn.fetchval("SELECT pg_try_advisory_lock($1, $2)", *MAINTENANCE_LOCK_KEY))
|
||||
|
||||
|
||||
async def release_maintenance_lock(conn: OutboxConnection) -> None:
|
||||
await conn.fetchval("SELECT pg_advisory_unlock($1, $2)", *MAINTENANCE_LOCK_KEY)
|
||||
|
||||
|
||||
async def retention_index_state(conn: OutboxConnection) -> str:
|
||||
row = await conn.fetchrow(
|
||||
"""
|
||||
SELECT indexes.indisvalid, indexes.indisready
|
||||
FROM pg_index AS indexes
|
||||
JOIN pg_class AS index_table ON index_table.oid = indexes.indexrelid
|
||||
WHERE indexes.indrelid = 'earth_data_change_events'::regclass
|
||||
AND index_table.relname = $1
|
||||
""",
|
||||
INDEX_NAME,
|
||||
)
|
||||
if row is None:
|
||||
return "missing"
|
||||
return "ready" if row["indisvalid"] and row["indisready"] else "invalid"
|
||||
|
||||
|
||||
async def ensure_retention_index(conn: OutboxConnection) -> None:
|
||||
"""Call on an autocommit connection, never inside app init_db's transaction."""
|
||||
state = await retention_index_state(conn)
|
||||
if state == "invalid":
|
||||
await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {INDEX_NAME}")
|
||||
if state != "ready":
|
||||
await conn.execute(
|
||||
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {INDEX_NAME} "
|
||||
"ON earth_data_change_events (consumed_at, id) WHERE consumed_at IS NOT NULL"
|
||||
)
|
||||
if await retention_index_state(conn) != "ready":
|
||||
raise RuntimeError("Earth outbox retention index is not valid and ready")
|
||||
|
||||
|
||||
async def purge_consumed_events(
|
||||
conn: OutboxConnection,
|
||||
*,
|
||||
retention_days: int,
|
||||
batch_size: int = DEFAULT_BATCH_SIZE,
|
||||
max_batches: int = 1,
|
||||
pause_seconds: float = DEFAULT_PAUSE_SECONDS,
|
||||
) -> int:
|
||||
if not 1 <= retention_days <= 3650:
|
||||
raise ValueError("retention_days must be between 1 and 3650")
|
||||
if batch_size < 1 or max_batches < 1 or pause_seconds < 0:
|
||||
raise ValueError("batch_size/max_batches must be positive and pause_seconds non-negative")
|
||||
if await retention_index_state(conn) != "ready":
|
||||
logger.warning_event(
|
||||
"Skipping Earth outbox retention because its index is unavailable",
|
||||
event="earth.outbox.retention_index_unavailable",
|
||||
)
|
||||
return 0
|
||||
|
||||
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
|
||||
deleted_total = 0
|
||||
for batch_number in range(max_batches):
|
||||
deleted = int(await conn.fetchval(DELETE_BATCH_SQL, cutoff, batch_size) or 0)
|
||||
deleted_total += deleted
|
||||
if deleted < batch_size:
|
||||
break
|
||||
if batch_number + 1 < max_batches and pause_seconds:
|
||||
await asyncio.sleep(pause_seconds)
|
||||
if deleted_total:
|
||||
logger.info_event(
|
||||
"Purged consumed Earth outbox events",
|
||||
event="earth.outbox.retention_purged",
|
||||
context={"deleted": deleted_total, "retention_days": retention_days},
|
||||
)
|
||||
return deleted_total
|
||||
|
||||
|
||||
async def compact_consumed_events(
|
||||
conn: OutboxConnection,
|
||||
*,
|
||||
retention_days: int,
|
||||
lock_timeout_seconds: int = 10,
|
||||
statement_timeout_seconds: int = 900,
|
||||
) -> tuple[int, int]:
|
||||
"""One-time atomic rewrite; blocks table access until the transaction commits."""
|
||||
if not 1 <= retention_days <= 3650:
|
||||
raise ValueError("retention_days must be between 1 and 3650")
|
||||
if lock_timeout_seconds < 1 or statement_timeout_seconds < 1:
|
||||
raise ValueError("timeouts must be positive")
|
||||
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
|
||||
async with conn.transaction():
|
||||
await conn.fetchval(
|
||||
"SELECT set_config('lock_timeout', $1, true)", f"{lock_timeout_seconds}s"
|
||||
)
|
||||
await conn.fetchval(
|
||||
"SELECT set_config('statement_timeout', $1, true)", f"{statement_timeout_seconds}s"
|
||||
)
|
||||
await conn.execute("LOCK TABLE earth_data_change_events IN ACCESS EXCLUSIVE MODE")
|
||||
await conn.execute(
|
||||
"""
|
||||
CREATE TEMP TABLE earth_outbox_retained ON COMMIT DROP AS
|
||||
SELECT * FROM earth_data_change_events
|
||||
WHERE consumed_at IS NULL OR consumed_at >= $1
|
||||
""",
|
||||
cutoff,
|
||||
)
|
||||
retained = await conn.fetchrow("""
|
||||
SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
|
||||
FROM earth_outbox_retained
|
||||
""")
|
||||
if retained is None:
|
||||
raise RuntimeError("Unable to verify retained Earth outbox rows")
|
||||
await conn.execute("TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT")
|
||||
await conn.execute(
|
||||
"INSERT INTO earth_data_change_events SELECT * FROM earth_outbox_retained"
|
||||
)
|
||||
restored = await conn.fetchrow("""
|
||||
SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
|
||||
FROM earth_data_change_events
|
||||
""")
|
||||
if restored is None or dict(retained) != dict(restored):
|
||||
raise RuntimeError("Earth outbox compact verification failed; transaction rolled back")
|
||||
return int(retained["total"]), int(retained["pending"])
|
||||
@@ -2,9 +2,11 @@
|
||||
|
||||
import asyncio
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import time
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.date import DateTrigger
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
|
||||
@@ -14,19 +16,44 @@ from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
|
||||
from app.services.business_logs import (
|
||||
emit_business_log,
|
||||
emit_business_log_background,
|
||||
exception_context,
|
||||
)
|
||||
from app.services.collectors.registry import collector_registry
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
get_builtin_effective_candidate,
|
||||
save_connectivity_success,
|
||||
)
|
||||
from app.services.earth_news_refresh import (
|
||||
REFRESH_INTERVAL_SECONDS as EARTH_NEWS_REFRESH_INTERVAL_SECONDS,
|
||||
create_earth_news_refresh_status,
|
||||
refresh_earth_news_all_regions,
|
||||
refresh_earth_news_region,
|
||||
set_earth_news_refresh_status,
|
||||
touch_earth_news_refresh_status,
|
||||
REFRESH_STATUS_HEARTBEAT_SECONDS as EARTH_NEWS_STATUS_HEARTBEAT_SECONDS,
|
||||
)
|
||||
from app.services.earth_news import REGION_ANCHORS
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
|
||||
EARTH_NEWS_REFRESH_JOB_ID = "earth_news_rss_refresh"
|
||||
_pending_earth_news_refreshes: dict[str, dict[str, Any]] = {}
|
||||
_running_earth_news_refreshes: set[str] = set()
|
||||
_rerun_earth_news_refreshes: dict[str, dict[str, Any]] = {}
|
||||
EARTH_NEWS_REFRESH_WAIT_SECONDS = 300
|
||||
EARTH_NEWS_REFRESH_POLL_SECONDS = 1
|
||||
EARTH_NEWS_REFRESH_PUBLIC_ERRORS = {
|
||||
"source_unavailable": "No news source could be refreshed for this region. Please retry.",
|
||||
"timeout": "News sources timed out. Please try again.",
|
||||
"refresh_failed": "Could not refresh news. Please try again.",
|
||||
}
|
||||
|
||||
|
||||
def _collector_task_name(collector_name: str) -> str:
|
||||
@@ -86,7 +113,10 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
logger.info_event(
|
||||
"Scheduled collector",
|
||||
event="collector.schedule.updated",
|
||||
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
|
||||
context={
|
||||
"collector_name": datasource.source,
|
||||
"frequency_minutes": datasource.frequency_minutes,
|
||||
},
|
||||
)
|
||||
else:
|
||||
logger.info_event(
|
||||
@@ -133,7 +163,11 @@ async def run_collector_task(collector_name: str):
|
||||
category="collector",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource.id,
|
||||
"status": "skipped",
|
||||
},
|
||||
)
|
||||
return
|
||||
|
||||
@@ -153,9 +187,8 @@ async def run_collector_task(collector_name: str):
|
||||
if started_at is not None and started_at.tzinfo is None:
|
||||
started_at = started_at.replace(tzinfo=UTC)
|
||||
|
||||
is_stale = (
|
||||
started_at is not None
|
||||
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
|
||||
is_stale = started_at is not None and (now - started_at) > timedelta(
|
||||
minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES
|
||||
)
|
||||
if not is_stale:
|
||||
logger.warning_event(
|
||||
@@ -189,9 +222,7 @@ async def run_collector_task(collector_name: str):
|
||||
existing_running.phase = "failed"
|
||||
existing_running.completed_at = now
|
||||
existing_running.error_message = (
|
||||
f"{existing_error}\n{stale_reason}".strip()
|
||||
if existing_error
|
||||
else stale_reason
|
||||
f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
)
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
@@ -231,7 +262,11 @@ async def run_collector_task(collector_name: str):
|
||||
category="collector",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource_id,
|
||||
"status": "running",
|
||||
},
|
||||
)
|
||||
task_result = await collector.run(db)
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
@@ -265,7 +300,11 @@ async def run_collector_task(collector_name: str):
|
||||
logger.info_event(
|
||||
"Collector completed",
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource_id,
|
||||
"result": task_result,
|
||||
},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
@@ -300,7 +339,11 @@ async def run_collector_task(collector_name: str):
|
||||
level="warning",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource.id,
|
||||
"status": "cancelled",
|
||||
},
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
@@ -312,7 +355,11 @@ async def run_collector_task(collector_name: str):
|
||||
logger.exception_event(
|
||||
"Collector failed",
|
||||
event="collector.run.failed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource.id,
|
||||
"error": str(exc),
|
||||
},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
@@ -324,7 +371,11 @@ async def run_collector_task(collector_name: str):
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
|
||||
{
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource.id,
|
||||
"status": "failed",
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
@@ -349,7 +400,9 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
task.completed_at = datetime.now(UTC)
|
||||
existing_error = (task.error_message or "").strip()
|
||||
cleanup_error = "Marked failed automatically after stale running task cleanup"
|
||||
task.error_message = f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
|
||||
task.error_message = (
|
||||
f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
|
||||
)
|
||||
|
||||
if stale_tasks:
|
||||
await db.commit()
|
||||
@@ -365,6 +418,17 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
def start_scheduler() -> None:
|
||||
"""Start the scheduler."""
|
||||
if not scheduler.running:
|
||||
scheduler.add_job(
|
||||
refresh_earth_news_all_regions,
|
||||
trigger=IntervalTrigger(seconds=EARTH_NEWS_REFRESH_INTERVAL_SECONDS),
|
||||
id=EARTH_NEWS_REFRESH_JOB_ID,
|
||||
name="Earth news RSS refresh",
|
||||
replace_existing=True,
|
||||
next_run_time=datetime.now(UTC),
|
||||
max_instances=1,
|
||||
coalesce=True,
|
||||
misfire_grace_time=60,
|
||||
)
|
||||
scheduler.start()
|
||||
logger.info_event("Scheduler started", event="scheduler.started")
|
||||
|
||||
@@ -376,6 +440,205 @@ def stop_scheduler() -> None:
|
||||
logger.info_event("Scheduler stopped", event="scheduler.stopped")
|
||||
|
||||
|
||||
def schedule_earth_news_region_refresh(
|
||||
region: str, *, force: bool = False, refresh_id: str | None = None
|
||||
) -> bool:
|
||||
"""Queue one immediate, per-region RSS refresh and merge duplicate requests."""
|
||||
normalized_region = str(region or "").strip().lower()
|
||||
if normalized_region not in REGION_ANCHORS:
|
||||
raise ValueError(f"Unsupported Earth news region: {region}")
|
||||
|
||||
if normalized_region in _running_earth_news_refreshes:
|
||||
rerun = _rerun_earth_news_refreshes.setdefault(
|
||||
normalized_region, {"force": False, "refresh_id": None}
|
||||
)
|
||||
rerun["force"] = rerun["force"] or force
|
||||
rerun["refresh_id"] = refresh_id or rerun["refresh_id"]
|
||||
return False
|
||||
if normalized_region in _pending_earth_news_refreshes:
|
||||
pending = _pending_earth_news_refreshes[normalized_region]
|
||||
pending["force"] = pending["force"] or force
|
||||
pending["refresh_id"] = refresh_id or pending["refresh_id"]
|
||||
return False
|
||||
|
||||
_pending_earth_news_refreshes[normalized_region] = {
|
||||
"force": force,
|
||||
"refresh_id": refresh_id,
|
||||
}
|
||||
job_id = _earth_news_region_refresh_job_id(normalized_region)
|
||||
try:
|
||||
scheduler.add_job(
|
||||
_run_earth_news_region_refresh,
|
||||
trigger=DateTrigger(run_date=datetime.now(UTC)),
|
||||
id=job_id,
|
||||
name=f"Earth news RSS refresh: {normalized_region}",
|
||||
kwargs={"region": normalized_region},
|
||||
replace_existing=False,
|
||||
max_instances=1,
|
||||
coalesce=True,
|
||||
misfire_grace_time=60,
|
||||
)
|
||||
except Exception:
|
||||
_pending_earth_news_refreshes.pop(normalized_region, None)
|
||||
raise
|
||||
return True
|
||||
|
||||
|
||||
def request_earth_news_region_refresh(region: str) -> dict[str, Any]:
|
||||
normalized_region = str(region or "").strip().lower()
|
||||
if normalized_region not in REGION_ANCHORS:
|
||||
raise ValueError(f"Unsupported Earth news region: {region}")
|
||||
status, created = create_earth_news_refresh_status(normalized_region)
|
||||
if created:
|
||||
try:
|
||||
schedule_earth_news_region_refresh(
|
||||
normalized_region,
|
||||
force=True,
|
||||
refresh_id=status["refresh_id"],
|
||||
)
|
||||
except Exception:
|
||||
return set_earth_news_refresh_status(
|
||||
status["refresh_id"],
|
||||
normalized_region,
|
||||
"failed",
|
||||
error={
|
||||
"code": "refresh_failed",
|
||||
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
|
||||
},
|
||||
)
|
||||
return status
|
||||
|
||||
|
||||
def schedule_earth_news_refresh_for_all_regions(*, force: bool = False) -> None:
|
||||
for region in REGION_ANCHORS:
|
||||
schedule_earth_news_region_refresh(region, force=force)
|
||||
|
||||
|
||||
def _earth_news_region_refresh_job_id(region: str) -> str:
|
||||
return f"{EARTH_NEWS_REFRESH_JOB_ID}:{region}"
|
||||
|
||||
|
||||
async def _run_earth_news_region_refresh(region: str) -> None:
|
||||
request = _pending_earth_news_refreshes.pop(region, {"force": False, "refresh_id": None})
|
||||
force = bool(request["force"])
|
||||
refresh_id = request["refresh_id"]
|
||||
_running_earth_news_refreshes.add(region)
|
||||
status_heartbeat: asyncio.Task[None] | None = None
|
||||
try:
|
||||
if refresh_id:
|
||||
set_earth_news_refresh_status(refresh_id, region, "running")
|
||||
if not touch_earth_news_refresh_status(refresh_id, region):
|
||||
return
|
||||
status_heartbeat = asyncio.create_task(
|
||||
_earth_news_refresh_status_heartbeat(refresh_id, region)
|
||||
)
|
||||
deadline = time.monotonic() + EARTH_NEWS_REFRESH_WAIT_SECONDS
|
||||
while True:
|
||||
result = await refresh_earth_news_region(region, force=force)
|
||||
if result["state"] != "busy":
|
||||
break
|
||||
if time.monotonic() >= deadline:
|
||||
set_earth_news_refresh_status(
|
||||
refresh_id,
|
||||
region,
|
||||
"failed",
|
||||
error={
|
||||
"code": "refresh_failed",
|
||||
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
|
||||
},
|
||||
)
|
||||
return
|
||||
await asyncio.sleep(EARTH_NEWS_REFRESH_POLL_SECONDS)
|
||||
if result["state"] == "coordination_unavailable":
|
||||
error_code = "refresh_failed"
|
||||
elif result["state"] == "failed":
|
||||
error_code = result.get("error_code") or "source_unavailable"
|
||||
elif result["state"] == "completed" and result.get("successful_sources", 0) > 0:
|
||||
set_earth_news_refresh_status(refresh_id, region, "succeeded")
|
||||
return
|
||||
else:
|
||||
error_code = "source_unavailable"
|
||||
set_earth_news_refresh_status(
|
||||
refresh_id,
|
||||
region,
|
||||
"failed",
|
||||
error={
|
||||
"code": error_code,
|
||||
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS.get(
|
||||
error_code, EARTH_NEWS_REFRESH_PUBLIC_ERRORS["source_unavailable"]
|
||||
),
|
||||
},
|
||||
)
|
||||
else:
|
||||
await refresh_earth_news_region(region, force=force)
|
||||
except asyncio.CancelledError:
|
||||
if refresh_id:
|
||||
set_earth_news_refresh_status(
|
||||
refresh_id,
|
||||
region,
|
||||
"failed",
|
||||
error={
|
||||
"code": "refresh_failed",
|
||||
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
|
||||
},
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news tracked refresh failed",
|
||||
event="earth_news.rss.tracked_refresh_failed",
|
||||
context={"region": region, "error": str(exc)},
|
||||
)
|
||||
if refresh_id:
|
||||
try:
|
||||
set_earth_news_refresh_status(
|
||||
refresh_id,
|
||||
region,
|
||||
"failed",
|
||||
error={
|
||||
"code": "refresh_failed",
|
||||
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
|
||||
},
|
||||
)
|
||||
except Exception as status_exc:
|
||||
logger.warning_event(
|
||||
"Earth news refresh status update failed",
|
||||
event="earth_news.rss.refresh_status_update_failed",
|
||||
context={"region": region, "error": str(status_exc)},
|
||||
)
|
||||
finally:
|
||||
if status_heartbeat is not None:
|
||||
status_heartbeat.cancel()
|
||||
try:
|
||||
await status_heartbeat
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
_running_earth_news_refreshes.discard(region)
|
||||
rerun = _rerun_earth_news_refreshes.pop(region, None)
|
||||
if rerun is not None and scheduler.running:
|
||||
schedule_earth_news_region_refresh(
|
||||
region,
|
||||
force=rerun["force"],
|
||||
refresh_id=rerun["refresh_id"],
|
||||
)
|
||||
|
||||
|
||||
async def _earth_news_refresh_status_heartbeat(refresh_id: str, region: str) -> None:
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(EARTH_NEWS_STATUS_HEARTBEAT_SECONDS)
|
||||
if not touch_earth_news_refresh_status(refresh_id, region):
|
||||
return
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news refresh status heartbeat failed",
|
||||
event="earth_news.rss.refresh_status_heartbeat_failed",
|
||||
context={"region": region, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def sync_scheduler_with_datasources() -> None:
|
||||
"""Synchronize scheduler jobs with datasource table."""
|
||||
async with async_session_factory() as db:
|
||||
@@ -384,7 +647,10 @@ async def sync_scheduler_with_datasources() -> None:
|
||||
|
||||
configured_sources = {datasource.source for datasource in datasources}
|
||||
for job in list(scheduler.get_jobs()):
|
||||
if job.id not in configured_sources:
|
||||
is_earth_news_refresh = job.id == EARTH_NEWS_REFRESH_JOB_ID or job.id.startswith(
|
||||
f"{EARTH_NEWS_REFRESH_JOB_ID}:"
|
||||
)
|
||||
if job.id not in configured_sources and not is_earth_news_refresh:
|
||||
scheduler.remove_job(job.id)
|
||||
|
||||
for datasource in datasources:
|
||||
@@ -461,7 +727,9 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
return False
|
||||
|
||||
try:
|
||||
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
|
||||
task = asyncio.create_task(
|
||||
run_collector_task(collector_name), name=_collector_task_name(collector_name)
|
||||
)
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = task
|
||||
|
||||
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
|
||||
|
||||
@@ -8,6 +8,37 @@ from unittest.mock import AsyncMock, MagicMock
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def persisted_test_logs(monkeypatch):
|
||||
"""Keep application log persistence in memory, including business-log calls."""
|
||||
from app.models.system_log import ObservabilityEventGroup
|
||||
from app.services import persistent_logs
|
||||
|
||||
entries = []
|
||||
groups = {}
|
||||
|
||||
class LogSession:
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc, traceback):
|
||||
return False
|
||||
|
||||
def add(self, item):
|
||||
entries.append(item)
|
||||
if isinstance(item, ObservabilityEventGroup):
|
||||
groups[item.fingerprint] = item
|
||||
|
||||
async def get(self, model, key):
|
||||
return groups.get(key) if model is ObservabilityEventGroup else None
|
||||
|
||||
async def commit(self):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(persistent_logs, "async_session_factory", LogSession)
|
||||
return entries
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def bgp_collector_location_cache():
|
||||
"""Mirror app startup seeding for tests that call sync BGP helpers."""
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Tests for BGP observability helpers."""
|
||||
|
||||
import ipaddress
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
|
||||
@@ -21,13 +22,20 @@ from app.services.collectors.bgp_common import (
|
||||
create_bgp_anomalies_for_batch,
|
||||
save_bgp_observations_for_batch,
|
||||
)
|
||||
from app.services.bgp_enrichment import enrich_bgp_events_for_batch, extract_bgp_network_fields
|
||||
from app.services.bgp_enrichment import (
|
||||
enrich_bgp_events_for_batch,
|
||||
extract_bgp_network_fields,
|
||||
lookup_prefix_geography,
|
||||
)
|
||||
from app.services.bgp_incidents import (
|
||||
create_bgp_incidents_for_anomalies,
|
||||
infer_related_infrastructure,
|
||||
)
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.api.v1.visualization import convert_bgp_incidents_to_geojson, build_incident_geography_hints
|
||||
from app.api.v1.visualization import (
|
||||
convert_bgp_incidents_to_geojson,
|
||||
build_incident_geography_hints,
|
||||
)
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
@@ -58,8 +66,7 @@ class _FakeResult:
|
||||
def all(self):
|
||||
if self._rows and all(isinstance(row, BGPObservation) for row in self._rows):
|
||||
return [
|
||||
(row.prefix, row.origin_asn, row.collector, row.collector_geo)
|
||||
for row in self._rows
|
||||
(row.prefix, row.origin_asn, row.collector, row.collector_geo) for row in self._rows
|
||||
]
|
||||
return self._rows
|
||||
|
||||
@@ -93,6 +100,8 @@ class _FakeAsyncSession:
|
||||
self.commits = 0
|
||||
|
||||
async def execute(self, _stmt, _params=None):
|
||||
if str(_stmt).lstrip().startswith("SET LOCAL jit"):
|
||||
return _FakeResult([])
|
||||
if not self._results:
|
||||
return _FakeResult([])
|
||||
return _FakeResult(self._results.pop(0))
|
||||
@@ -107,6 +116,49 @@ class _FakeAsyncSession:
|
||||
self.commits += 1
|
||||
|
||||
|
||||
class _StrictFakeAsyncSession(_FakeAsyncSession):
|
||||
async def execute(self, stmt, params=None):
|
||||
if not str(stmt).lstrip().startswith("SET LOCAL jit") and not self._results:
|
||||
raise AssertionError(f"Unexpected BGP query: {stmt}")
|
||||
return await super().execute(stmt, params)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prefix_geography_keeps_source_priority_and_ipv6_range_parameters():
|
||||
calls = []
|
||||
|
||||
class GeoSession:
|
||||
async def execute(self, _stmt, params=None):
|
||||
if str(_stmt).lstrip().startswith("SET LOCAL jit"):
|
||||
return _FakeResult([])
|
||||
calls.append(dict(params))
|
||||
if params["source"] == "nro_delegated_prefix_geo":
|
||||
return _FakeResult(
|
||||
[
|
||||
{
|
||||
"metadata": {
|
||||
"country_code": "GB",
|
||||
"prefix": "2001:db8::/32",
|
||||
}
|
||||
}
|
||||
]
|
||||
)
|
||||
return _FakeResult([])
|
||||
|
||||
network = ipaddress.ip_network("2001:db8:1::/48")
|
||||
result = await lookup_prefix_geography(GeoSession(), [str(network), "invalid-prefix"])
|
||||
|
||||
assert result[str(network)]["source"] == "nro_delegated_stats"
|
||||
assert [call["source"] for call in calls] == [
|
||||
"opengeofeed_prefix_geo",
|
||||
"iptoasn_prefix_geo",
|
||||
"nro_delegated_prefix_geo",
|
||||
]
|
||||
assert all(call["family"] == "ipv6" for call in calls)
|
||||
assert all(call["range_start"] == str(network.network_address) for call in calls)
|
||||
assert all(call["range_end"] == str(network.broadcast_address) for call in calls)
|
||||
|
||||
|
||||
def test_normalize_bgp_event_from_live_payload():
|
||||
event = normalize_bgp_event(
|
||||
{
|
||||
@@ -449,7 +501,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
|
||||
"prefix": "203.0.113.0/24",
|
||||
"origin_asn": 64496,
|
||||
"as_path": [64500, 64496],
|
||||
"collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
|
||||
"collector_location": {
|
||||
"country": "NL",
|
||||
"city": "Amsterdam",
|
||||
"latitude": 52.3,
|
||||
"longitude": 4.9,
|
||||
},
|
||||
"enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
|
||||
}
|
||||
},
|
||||
@@ -460,7 +517,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
|
||||
"prefix": "203.0.113.0/24",
|
||||
"origin_asn": 64496,
|
||||
"as_path": [64510, 64520, 64530, 64540, 64496],
|
||||
"collector_location": {"country": "GB", "city": "London", "latitude": 51.5, "longitude": -0.1},
|
||||
"collector_location": {
|
||||
"country": "GB",
|
||||
"city": "London",
|
||||
"latitude": 51.5,
|
||||
"longitude": -0.1,
|
||||
},
|
||||
"enrichment": {"prefix_scope": {"regions": [{"country": "GB"}]}},
|
||||
}
|
||||
},
|
||||
@@ -481,7 +543,9 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
|
||||
def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
|
||||
base_timestamp = datetime(2026, 3, 27, 0, 0, tzinfo=UTC)
|
||||
events = []
|
||||
for index, event_type in enumerate(["announcement", "withdrawal", "announcement", "withdrawal"]):
|
||||
for index, event_type in enumerate(
|
||||
["announcement", "withdrawal", "announcement", "withdrawal"]
|
||||
):
|
||||
events.append(
|
||||
{
|
||||
"metadata": {
|
||||
@@ -491,7 +555,12 @@ def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
|
||||
"prefix": "198.51.100.0/24",
|
||||
"origin_asn": 64512,
|
||||
"as_path": [64500 + index, 64512] if event_type == "announcement" else [],
|
||||
"collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
|
||||
"collector_location": {
|
||||
"country": "NL",
|
||||
"city": "Amsterdam",
|
||||
"latitude": 52.3,
|
||||
"longitude": 4.9,
|
||||
},
|
||||
"enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
|
||||
}
|
||||
}
|
||||
@@ -920,7 +989,9 @@ async def test_create_bgp_incidents_for_anomalies_refreshes_existing_incident():
|
||||
|
||||
with patch(
|
||||
"app.services.bgp_incidents.infer_related_infrastructure",
|
||||
new=AsyncMock(return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}),
|
||||
new=AsyncMock(
|
||||
return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}
|
||||
),
|
||||
):
|
||||
created = await create_bgp_incidents_for_anomalies(
|
||||
db,
|
||||
@@ -1132,13 +1203,17 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
|
||||
source="ris_live_bgp",
|
||||
extra_data={"prefix": "203.0.113.0/24", "origin_asn": 64496},
|
||||
)
|
||||
db = _FakeAsyncSession([
|
||||
[],
|
||||
[],
|
||||
[],
|
||||
[previous_record],
|
||||
[],
|
||||
])
|
||||
db = _StrictFakeAsyncSession(
|
||||
[
|
||||
[], # Historical BGP observations
|
||||
[], # OpenGeoFeed prefix geography
|
||||
[], # IPtoASN prefix geography
|
||||
[], # NRO delegated prefix geography
|
||||
[], # PeeringDB ASN profiles
|
||||
[previous_record], # Prior collected snapshot for origin-change detection
|
||||
[], # Existing anomaly entity keys
|
||||
]
|
||||
)
|
||||
events = [
|
||||
{
|
||||
"reference_date": "2026-03-30T10:00:00Z",
|
||||
@@ -1178,6 +1253,7 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
|
||||
assert anomaly.anomaly_type == "origin_change"
|
||||
assert anomaly.prefix == "203.0.113.0/24"
|
||||
incident_mock.assert_awaited_once()
|
||||
assert not db._results
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -1197,14 +1273,18 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
|
||||
origin_asn=64496,
|
||||
new_origin_asn=64497,
|
||||
)
|
||||
db = _FakeAsyncSession([
|
||||
[],
|
||||
[],
|
||||
[],
|
||||
[previous_record],
|
||||
[existing_key],
|
||||
[existing_anomaly],
|
||||
])
|
||||
db = _StrictFakeAsyncSession(
|
||||
[
|
||||
[], # Historical BGP observations
|
||||
[], # OpenGeoFeed prefix geography
|
||||
[], # IPtoASN prefix geography
|
||||
[], # NRO delegated prefix geography
|
||||
[], # PeeringDB ASN profiles
|
||||
[previous_record], # Prior collected snapshot for origin-change detection
|
||||
[existing_key],
|
||||
[existing_anomaly],
|
||||
]
|
||||
)
|
||||
events = [
|
||||
{
|
||||
"reference_date": "2026-03-30T10:00:00Z",
|
||||
@@ -1235,6 +1315,7 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
|
||||
assert created == 0
|
||||
assert len(db.added) == 0
|
||||
incident_mock.assert_awaited_once()
|
||||
assert not db._results
|
||||
|
||||
|
||||
async def _bgp_test_client(db_session):
|
||||
@@ -1242,7 +1323,9 @@ async def _bgp_test_client(db_session):
|
||||
yield db_session
|
||||
|
||||
def override_get_current_user():
|
||||
return User(id=1, username="testuser", email="test@example.com", password_hash="x", role="admin")
|
||||
return User(
|
||||
id=1, username="testuser", email="test@example.com", password_hash="x", role="admin"
|
||||
)
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
app.dependency_overrides[get_current_user] = override_get_current_user
|
||||
@@ -1264,7 +1347,7 @@ async def test_bgp_events_api_lists_observations():
|
||||
origin_asn=64496,
|
||||
observed_at=datetime(2026, 3, 30, 10, 0, tzinfo=UTC),
|
||||
)
|
||||
db = _FakeAsyncSession([[observation]])
|
||||
db = _StrictFakeAsyncSession([[1], [observation]])
|
||||
client = await _bgp_test_client(db)
|
||||
|
||||
try:
|
||||
@@ -1278,6 +1361,7 @@ async def test_bgp_events_api_lists_observations():
|
||||
assert payload["total"] == 1
|
||||
assert payload["data"][0]["collector"] == "rrc00"
|
||||
assert payload["data"][0]["prefix"] == "203.0.113.0/24"
|
||||
assert not db._results
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -1376,7 +1460,18 @@ async def test_bgp_event_summary_api_returns_aggregates():
|
||||
event_type="withdrawal",
|
||||
observed_at=datetime(2026, 3, 30, 10, 5, tzinfo=UTC),
|
||||
)
|
||||
db = _FakeAsyncSession([[observation_one, observation_two]])
|
||||
observations = [observation_one, observation_two]
|
||||
db = _StrictFakeAsyncSession(
|
||||
[
|
||||
[len(observations)],
|
||||
[len({record.collector for record in observations})],
|
||||
[len({record.prefix for record in observations})],
|
||||
[
|
||||
(event_type, sum(record.event_type == event_type for record in observations))
|
||||
for event_type in sorted({record.event_type for record in observations})
|
||||
],
|
||||
]
|
||||
)
|
||||
client = await _bgp_test_client(db)
|
||||
|
||||
try:
|
||||
@@ -1392,6 +1487,7 @@ async def test_bgp_event_summary_api_returns_aggregates():
|
||||
assert payload["prefix_count"] == 2
|
||||
assert payload["by_type"]["announcement"] == 1
|
||||
assert payload["by_type"]["withdrawal"] == 1
|
||||
assert not db._results
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
22
backend/tests/test_bgp_prefix_geo.py
Normal file
22
backend/tests/test_bgp_prefix_geo.py
Normal file
@@ -0,0 +1,22 @@
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from app.db.bgp_prefix_geo import (
|
||||
BGP_PREFIX_GEO_FUNCTION_SQL,
|
||||
BGP_PREFIX_GEO_INDEX_SQL,
|
||||
install_bgp_prefix_geo_functions,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_database_installs_lookup_functions_without_large_index():
|
||||
connection = AsyncMock()
|
||||
|
||||
await install_bgp_prefix_geo_functions(connection)
|
||||
|
||||
statements = [str(call.args[0]) for call in connection.execute.call_args_list]
|
||||
assert len(statements) == len(BGP_PREFIX_GEO_FUNCTION_SQL) == 2
|
||||
assert all("CREATE OR REPLACE FUNCTION" in statement for statement in statements)
|
||||
assert all("CREATE INDEX" not in statement for statement in statements)
|
||||
assert "CREATE INDEX CONCURRENTLY" in BGP_PREFIX_GEO_INDEX_SQL
|
||||
333
backend/tests/test_card_images.py
Normal file
333
backend/tests/test_card_images.py
Normal file
@@ -0,0 +1,333 @@
|
||||
import io
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from PIL import Image
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.v1.card_images import router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
|
||||
from app.services.card_image_configuration import (
|
||||
effective_envelope,
|
||||
publish_configuration,
|
||||
resolve_image,
|
||||
)
|
||||
from app.services.card_image_defaults import bundled_configuration
|
||||
from app.services import card_image_files
|
||||
|
||||
|
||||
def configuration():
|
||||
assets = [
|
||||
CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
|
||||
for key in ["exact", "model", "group", "category"]
|
||||
]
|
||||
rules = [
|
||||
CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
|
||||
for key, scope, value in [
|
||||
("exact", "object", "123"),
|
||||
("model", "model", "m1"),
|
||||
("group", "group", "starlink"),
|
||||
("category", "category", "satellite"),
|
||||
]
|
||||
]
|
||||
return CardImageConfiguration(assets=assets, rules=rules)
|
||||
|
||||
|
||||
def test_precedence_disabled_fallback_and_explicit_no_image():
|
||||
config = configuration()
|
||||
context = dict(
|
||||
entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
|
||||
)
|
||||
for expected in ["exact", "model", "group", "category"]:
|
||||
assert resolve_image(config, context)["image"]["id"] == expected
|
||||
next(asset for asset in config.assets if asset.id == expected).enabled = False
|
||||
assert resolve_image(config, context)["image"] is None
|
||||
config = configuration()
|
||||
config.rules[0].asset_id = None
|
||||
config.rules[0].suppress = True
|
||||
result = resolve_image(config, context)
|
||||
assert result["suppressed"] and result["image"] is None
|
||||
|
||||
|
||||
def test_bundled_series_images_retain_truthful_source_metadata():
|
||||
config, _ = bundled_configuration()
|
||||
result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
|
||||
assert result["image"]["id"] == "starlink-solar-array"
|
||||
pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
|
||||
assert pending.rights_status == "permission_pending"
|
||||
assert (
|
||||
resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
|
||||
== pending.id
|
||||
)
|
||||
assert pending.enabled
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"url",
|
||||
[
|
||||
"javascript:alert(1)",
|
||||
"//evil.example/image.jpg",
|
||||
"/earth/assets/cards/../secret",
|
||||
"https://user:pass@example.org/a.jpg",
|
||||
],
|
||||
)
|
||||
def test_unsafe_urls_rejected(url):
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageAsset(id="test", title="test", url=url)
|
||||
|
||||
|
||||
def test_duplicate_targets_and_dangling_assets_rejected():
|
||||
data = configuration().model_dump()
|
||||
duplicate = {**data["rules"][0], "id": "another"}
|
||||
data["rules"].append(duplicate)
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageConfiguration.model_validate(data)
|
||||
data = configuration().model_dump()
|
||||
data["assets"] = []
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageConfiguration.model_validate(data)
|
||||
|
||||
|
||||
def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
|
||||
defaults, _ = bundled_configuration()
|
||||
payload = {
|
||||
"revision": "custom",
|
||||
"config": {"assets": [], "rules": []},
|
||||
"known_assets": [asset.id for asset in defaults.assets],
|
||||
"known_rules": [rule.id for rule in defaults.rules],
|
||||
}
|
||||
assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
|
||||
payload["known_assets"] = []
|
||||
assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
|
||||
def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
|
||||
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
|
||||
output = io.BytesIO()
|
||||
Image.new("RGB", (1600, 900), "blue").save(output, format)
|
||||
asset = card_image_files.store_image(output.getvalue(), "Facility")
|
||||
assert (asset.width, asset.height) == (1600, 900)
|
||||
assert len(list(tmp_path.iterdir())) == 3
|
||||
for size, url in asset.variants.items():
|
||||
with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
|
||||
assert max(image.size) == int(size)
|
||||
assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
|
||||
with pytest.raises(HTTPException):
|
||||
card_image_files.store_image(b"<svg></svg>", "bad")
|
||||
with pytest.raises(HTTPException):
|
||||
card_image_files.uploaded_path("../private")
|
||||
|
||||
|
||||
def fake_db(payload):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
record = SimpleNamespace(payload=payload)
|
||||
result = MagicMock()
|
||||
result.scalar_one.return_value = record
|
||||
result.scalar_one_or_none.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db, record
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_revision_and_audit():
|
||||
defaults, _ = bundled_configuration()
|
||||
payload = {
|
||||
"revision": "default",
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
db, record = fake_db(payload)
|
||||
revision = effective_envelope(payload)["revision"]
|
||||
result = await publish_configuration(
|
||||
db, revision, defaults, SimpleNamespace(id=1, username="admin")
|
||||
)
|
||||
assert result["revision"] != revision
|
||||
assert db.add.call_args.args[0].action == "card_images.update"
|
||||
before = deepcopy(record.payload)
|
||||
with pytest.raises(HTTPException) as error:
|
||||
await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
|
||||
assert error.value.status_code == 409 and record.payload == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configuration_requires_admin_and_validates_http_payload():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
defaults, _ = bundled_configuration()
|
||||
db, _ = fake_db(
|
||||
{
|
||||
"revision": "default",
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
|
||||
assert (await client.get("/configuration")).status_code == 403
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
role="admin", id=1, username="admin"
|
||||
)
|
||||
assert (await client.get("/configuration")).status_code == 200
|
||||
assert (
|
||||
await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
|
||||
).status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
role="admin", id=1, username="admin"
|
||||
)
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
asset = response.json()
|
||||
filename = asset["variants"]["640"].rsplit("/", 1)[1]
|
||||
image = await client.get(f"/files/{filename}")
|
||||
assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
|
||||
assert "immutable" in image.headers["cache-control"]
|
||||
assert (
|
||||
await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
|
||||
).status_code == 422
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
|
||||
assert (
|
||||
await client.post(
|
||||
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
|
||||
)
|
||||
).status_code == 403
|
||||
|
||||
|
||||
def test_all_object_types_have_a_default_without_overriding_exact_suppression():
|
||||
from typing import get_args
|
||||
from app.schemas.card_images import EntityType
|
||||
|
||||
config, _ = bundled_configuration()
|
||||
for entity_type in get_args(EntityType):
|
||||
result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
|
||||
assert result["image"] and result["match"]["value"] == "*"
|
||||
config = config.model_copy(deep=True)
|
||||
config.rules.append(
|
||||
CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
|
||||
)
|
||||
assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
|
||||
|
||||
|
||||
def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
|
||||
from app.services.card_image_defaults import previous_bundled_assets
|
||||
|
||||
old = previous_bundled_assets()["starlink-solar-array"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old.model_dump()], "rules": []},
|
||||
"known_assets": [old.id],
|
||||
"known_rules": [],
|
||||
}
|
||||
|
||||
def starlink():
|
||||
return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
|
||||
|
||||
assert starlink()["url"].endswith("starlink-orbit.webp")
|
||||
payload["config"]["assets"][0]["focus_x"] = 20
|
||||
assert starlink()["focus_x"] == 20
|
||||
assert starlink()["url"].endswith("starlink-orbit.webp")
|
||||
payload["config"]["assets"][0]["enabled"] = False
|
||||
assert starlink()["enabled"] is False
|
||||
payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
|
||||
assert starlink()["url"] == "https://example.org/custom.jpg"
|
||||
|
||||
|
||||
def test_compute_categories_and_bgp_get_separate_user_images():
|
||||
config, _ = bundled_configuration()
|
||||
for kind, filename in [
|
||||
("supercomputer", "supercomputer.png"),
|
||||
("gpu_cluster", "gpu-cluster.jpg"),
|
||||
("data_center", "data-center.png"),
|
||||
("cloud_center", "cloud-center.png"),
|
||||
("edge_node", "edge-node.png"),
|
||||
]:
|
||||
result = resolve_image(config, dict(entity_type="compute_center", category=kind))
|
||||
assert result["image"]["url"].endswith(filename)
|
||||
for kind in ("bgp", "bgp_collector"):
|
||||
result = resolve_image(config, dict(entity_type=kind, category=kind))
|
||||
assert result["image"]["url"].endswith(
|
||||
"bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
|
||||
)
|
||||
|
||||
|
||||
def test_saved_collector_default_migrates_without_changing_user_bindings():
|
||||
from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
|
||||
|
||||
old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
|
||||
old_asset = previous_bundled_assets()["network-switches"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
|
||||
"known_assets": [old_asset.id],
|
||||
"known_rules": [old_rule.id],
|
||||
}
|
||||
result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
|
||||
assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
|
||||
"image"
|
||||
]["url"].endswith("bgp-collector.png")
|
||||
payload["config"]["rules"][0]["enabled"] = False
|
||||
result = effective_envelope(payload)["config"]
|
||||
assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
|
||||
from app.services.card_image_targets import get_target
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
db = AsyncMock()
|
||||
result = MagicMock()
|
||||
result.mappings.return_value.first.return_value = {
|
||||
"entity_id": "epoch_ai_gpu:site",
|
||||
"name": "Site",
|
||||
"category": "cloud_center",
|
||||
"group": "Operator",
|
||||
"model": "Model",
|
||||
}
|
||||
db.execute.return_value = result
|
||||
context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
|
||||
query = db.execute.call_args.args[0]
|
||||
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
|
||||
assert "collected_data.source = 'epoch_ai_gpu'" in sql
|
||||
assert "collected_data.source_id = 'site'" in sql
|
||||
assert context["category"] == "cloud_center" and context["model"] == "Model"
|
||||
|
||||
|
||||
def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
|
||||
from app.services.card_image_defaults import previous_bundled_assets
|
||||
|
||||
old = previous_bundled_assets()["compute-gpu-cluster"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old.model_dump()], "rules": []},
|
||||
"known_assets": [old.id],
|
||||
"known_rules": [],
|
||||
}
|
||||
result = effective_envelope(payload)["config"]
|
||||
upgraded = next(a for a in result["assets"] if a["id"] == old.id)
|
||||
latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
|
||||
assert upgraded["variants"] == latest.variants
|
||||
assert upgraded["url"] == latest.url
|
||||
218
backend/tests/test_collector_downloads.py
Normal file
218
backend/tests/test_collector_downloads.py
Normal file
@@ -0,0 +1,218 @@
|
||||
"""Request-level regressions for quota-limited satellite downloads."""
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
|
||||
from app.services.collectors.downloads import ResumableFileDownloader
|
||||
|
||||
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
|
||||
|
||||
|
||||
def setup_collector(monkeypatch, tmp_path, handler):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/gp.php"
|
||||
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
|
||||
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
|
||||
original_client = httpx.AsyncClient
|
||||
monkeypatch.setattr(
|
||||
"app.services.collectors.celestrak.httpx.AsyncClient",
|
||||
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
|
||||
)
|
||||
return collector
|
||||
|
||||
|
||||
def expire(path):
|
||||
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
|
||||
os.utime(path, (old, old))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
|
||||
return httpx.Response(
|
||||
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert len(requests) == 1
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
|
||||
assert requests[1].headers["if-none-match"] == '"v1"'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
|
||||
assert requests == ["GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
|
||||
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
with pytest.raises(RuntimeError, match="validation failed"):
|
||||
await collector.fetch()
|
||||
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path, _, meta = collector._downloader._cache_paths(URL, ".json")
|
||||
path.write_text("[]")
|
||||
meta.write_text('{"etag":"old"}')
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
|
||||
assert "if-none-match" not in requests[0].headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
|
||||
)
|
||||
path, _, meta = downloader._cache_paths(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
expire(path)
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(304)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
for _ in range(2):
|
||||
await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert len(requests) == 1
|
||||
assert requests[0].headers["if-none-match"] == "v1"
|
||||
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
|
||||
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
|
||||
body = gzip.compress(raw)
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200,
|
||||
content=body,
|
||||
headers={"content-encoding": "gzip", "content-length": str(len(body))},
|
||||
)
|
||||
)
|
||||
) as client:
|
||||
path = await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert path.read_bytes() == raw
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
final, part, meta = downloader._cache_paths(URL, ".bin")
|
||||
part.parent.mkdir(parents=True)
|
||||
part.write_bytes(b"abc")
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.method == "HEAD":
|
||||
return httpx.Response(
|
||||
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
|
||||
)
|
||||
assert request.headers["range"] == "bytes=3-"
|
||||
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".bin")
|
||||
assert [r.method for r in requests] == ["HEAD", "GET"]
|
||||
assert final.read_bytes() == b"abcdef"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.url.params["GROUP"] == "active":
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
|
||||
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
|
||||
assert [r.method for r in requests] == ["GET", "GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generic_cache_without_validators_must_refresh(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
path = downloader.cached_file_path(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text("[]")
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".json")
|
||||
assert requests == ["HEAD", "GET"]
|
||||
assert path.read_text() == "[1]"
|
||||
222
backend/tests/test_compute_configuration.py
Normal file
222
backend/tests/test_compute_configuration.py
Normal file
@@ -0,0 +1,222 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.system_log import AuditLog
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
default_envelope,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import FastAPI, HTTPException, Response
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
|
||||
def default_config():
|
||||
return DEFAULT_CONFIGURATION.model_dump(mode="json")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation",
|
||||
[
|
||||
lambda value: value["observations"][0]["values"].update(CN=101),
|
||||
lambda value: value["observations"][0]["values"].update(CN=-1),
|
||||
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
|
||||
lambda value: value["observations"][0]["values"].update(CN=True),
|
||||
lambda value: value["observations"][0]["values"].update(EUROPE=10),
|
||||
lambda value: value["observations"][0].update(source_id="unknown"),
|
||||
lambda value: value["observations"][0].update(year=2026),
|
||||
lambda value: value["observations"].append(value["observations"][0]),
|
||||
lambda value: value["sources"].append(value["sources"][0]),
|
||||
lambda value: value["innovation"].append(value["innovation"][0]),
|
||||
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
|
||||
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_or_ambiguous_evidence(mutation):
|
||||
payload = default_config()
|
||||
mutation(payload)
|
||||
with pytest.raises(ValidationError):
|
||||
ComputeConfiguration.model_validate(payload)
|
||||
|
||||
|
||||
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
|
||||
payload = default_config()
|
||||
payload["sources"].append(
|
||||
{
|
||||
"id": "new-report",
|
||||
"title": "New report",
|
||||
"url": "https://example.com/report",
|
||||
"publisher": "Research group",
|
||||
"report_year": 2026,
|
||||
"locator": "Figure 2",
|
||||
}
|
||||
)
|
||||
payload["observations"].append(
|
||||
{
|
||||
"year": 2025,
|
||||
"metric": "global_share",
|
||||
"source_id": "new-report",
|
||||
"values": {"CN": 25, "US": 50, "JP": 0},
|
||||
}
|
||||
)
|
||||
config = ComputeConfiguration.model_validate(payload)
|
||||
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
|
||||
row = next(row for row in result["compute"] if row["year"] == 2025)
|
||||
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
|
||||
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
|
||||
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
|
||||
|
||||
|
||||
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
|
||||
payload = default_config()
|
||||
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
|
||||
result = compute_history(
|
||||
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
|
||||
)
|
||||
assert result["compute"][0]["values"] == {"CN": None, "US": None}
|
||||
payload.update(sources=[], observations=[], innovation=[])
|
||||
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
|
||||
assert all(row["values"]["CN"] is None for row in result["compute"])
|
||||
assert result["evidence"] == {}
|
||||
|
||||
|
||||
def fake_db(record=None):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = record
|
||||
result.scalar_one.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_install_uses_defaults_without_writing_database():
|
||||
db = fake_db()
|
||||
assert await read_configuration(db) == default_envelope()
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_series_reads_persisted_override_and_disables_http_cache():
|
||||
envelope = default_envelope()
|
||||
envelope["is_default"] = False
|
||||
envelope["revision"] = "custom"
|
||||
envelope["config"]["observations"][3]["values"]["CN"] = 30
|
||||
response = Response()
|
||||
data = await get_series(
|
||||
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
|
||||
)
|
||||
assert data["compute"][3]["values"]["CN"] == 30
|
||||
assert data["version"] == "custom"
|
||||
assert data["provenance"] == "admin_curated_reports"
|
||||
assert response.headers["Cache-Control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
actor = SimpleNamespace(id=1, username="admin")
|
||||
old = deepcopy(record.payload)
|
||||
result = await publish_configuration(
|
||||
db,
|
||||
revision=old["revision"],
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=actor,
|
||||
)
|
||||
assert result["revision"] != old["revision"]
|
||||
audit = db.add.call_args.args[0]
|
||||
assert isinstance(audit, AuditLog)
|
||||
assert audit.details == {"before": old, "after": result}
|
||||
assert audit.actor_id == actor.id
|
||||
db.commit.assert_awaited_once()
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await publish_configuration(
|
||||
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
|
||||
)
|
||||
assert failure.value.status_code == 409
|
||||
assert record.payload == result
|
||||
db.rollback.assert_awaited_once()
|
||||
db.commit.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reset_changes_revision_and_restores_bundled_data():
|
||||
record = SimpleNamespace(
|
||||
payload={**default_envelope(), "revision": "custom", "is_default": False}
|
||||
)
|
||||
result = await publish_configuration(
|
||||
fake_db(record),
|
||||
revision="custom",
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=SimpleNamespace(id=1, username="admin"),
|
||||
reset=True,
|
||||
)
|
||||
assert result["is_default"] is True
|
||||
assert result["config"] == default_config()
|
||||
assert result["revision"] != "custom"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
|
||||
async def test_non_admin_cannot_configure(role):
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await configuration_admin(SimpleNamespace(role=role))
|
||||
assert failure.value.status_code == 403
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_http_configuration_auth_validation_and_public_series():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
assert (await client.get("/series")).status_code == 200
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
id=1,
|
||||
username="admin",
|
||||
role="admin",
|
||||
)
|
||||
loaded = (await client.get("/configuration")).json()
|
||||
invalid = deepcopy(loaded["config"])
|
||||
invalid["observations"][0]["values"]["CN"] = 200
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": invalid,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
db.commit.assert_not_awaited()
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": loaded["config"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["revision"] != loaded["revision"]
|
||||
assert (
|
||||
await client.post(
|
||||
"/configuration/reset",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
},
|
||||
)
|
||||
).status_code == 409
|
||||
@@ -262,6 +262,7 @@ def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_cou
|
||||
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
|
||||
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
|
||||
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
|
||||
for report in absolute["evidence"].values():
|
||||
if report.get("report_year"):
|
||||
for row in absolute["compute"]:
|
||||
if row["evidence_id"]:
|
||||
report = absolute["evidence"][row["evidence_id"]]
|
||||
assert report["url"].startswith("https://www.caict.ac.cn/")
|
||||
|
||||
15
backend/tests/test_earth_layer_adapters.py
Normal file
15
backend/tests/test_earth_layer_adapters.py
Normal file
@@ -0,0 +1,15 @@
|
||||
from app.services.earth_layer_adapters import (
|
||||
get_earth_cache_patterns_for_source,
|
||||
get_earth_update_layers_for_change,
|
||||
)
|
||||
|
||||
|
||||
def test_news_changes_do_not_invalidate_earth_summary_cache():
|
||||
assert get_earth_update_layers_for_change("earth_news_items", "earth_news_items") == ["news"]
|
||||
assert get_earth_cache_patterns_for_source("earth_news_items") == []
|
||||
assert get_earth_cache_patterns_for_source("media_news_archive") == []
|
||||
|
||||
|
||||
def test_counted_layers_still_invalidate_earth_summary_cache():
|
||||
for source in ("aisstream_vessels", "arcgis_cables", "celestrak_tle", "top500", "ris_live_bgp"):
|
||||
assert "summary*" in get_earth_cache_patterns_for_source(source)
|
||||
@@ -1,9 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import Response
|
||||
from fastapi import FastAPI, HTTPException, Response
|
||||
|
||||
from app.api.v1 import visualization
|
||||
from app.services.earth_layer_cache import (
|
||||
@@ -49,6 +51,19 @@ class FakeRedis:
|
||||
deleted += 1 if self.store.pop(key, None) is not None else 0
|
||||
return deleted
|
||||
|
||||
def eval(self, script: str, numkeys: int, *args):
|
||||
self._maybe_fail()
|
||||
keys, values = args[:numkeys], args[numkeys:]
|
||||
if self.store.get(keys[0]) != values[0]:
|
||||
return 0
|
||||
if numkeys == 3:
|
||||
self.setex(keys[1], values[1], values[3])
|
||||
self.setex(keys[2], values[2], values[3])
|
||||
return 1
|
||||
if "'expire'" in script:
|
||||
return 1
|
||||
return self.delete(keys[0])
|
||||
|
||||
def scan_iter(self, match: str):
|
||||
self._maybe_fail()
|
||||
prefix = match.rstrip("*")
|
||||
@@ -120,7 +135,9 @@ async def test_resolve_layer_payload_writes_fresh_and_stale(fake_cache_client):
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_layer_payload_returns_stale_when_builder_fails(fake_cache_client):
|
||||
key = earth_layer_cache.key("bgp-incidents", status="active")
|
||||
fake_cache_client.store[f"{key}:stale"] = json.dumps({"type": "FeatureCollection", "features": []})
|
||||
fake_cache_client.store[f"{key}:stale"] = json.dumps(
|
||||
{"type": "FeatureCollection", "features": []}
|
||||
)
|
||||
|
||||
async def builder():
|
||||
raise RuntimeError("db exploded")
|
||||
@@ -165,6 +182,7 @@ async def test_resolve_layer_payload_bypasses_redis_failure():
|
||||
previous = earth_layer_cache._client
|
||||
earth_layer_cache._client = FakeRedis(fail=True)
|
||||
try:
|
||||
|
||||
async def builder():
|
||||
return {"type": "FeatureCollection", "features": [{"id": "safe"}]}
|
||||
|
||||
@@ -192,9 +210,13 @@ async def test_visualization_endpoint_sets_cache_headers(fake_cache_client, monk
|
||||
monkeypatch.setattr(visualization, "_build_satellites_geojson", fake_build_satellites_geojson)
|
||||
|
||||
first_response = Response()
|
||||
first = await visualization.get_satellites_geojson(limit=25, db=object(), response=first_response)
|
||||
first = await visualization.get_satellites_geojson(
|
||||
limit=25, db=object(), response=first_response
|
||||
)
|
||||
second_response = Response()
|
||||
second = await visualization.get_satellites_geojson(limit=25, db=object(), response=second_response)
|
||||
second = await visualization.get_satellites_geojson(
|
||||
limit=25, db=object(), response=second_response
|
||||
)
|
||||
|
||||
assert first == second
|
||||
assert calls == 1
|
||||
@@ -252,3 +274,284 @@ async def test_vessel_snapshot_uses_short_cache(fake_cache_client, monkeypatch):
|
||||
assert calls == 1
|
||||
assert first_response.headers["X-Planet-Cache"] == "refresh"
|
||||
assert second_response.headers["X-Planet-Cache"] == "hit"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_summary_requests_share_one_slow_build(monkeypatch):
|
||||
started = asyncio.Event()
|
||||
finish = asyncio.Event()
|
||||
calls = 0
|
||||
|
||||
async def build_summary(db):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
started.set()
|
||||
await finish.wait()
|
||||
return {"stats": {"vessel_count": 42}}
|
||||
|
||||
monkeypatch.setattr(visualization, "_build_visualization_geo_summary", build_summary)
|
||||
app = FastAPI()
|
||||
app.include_router(visualization.router, prefix="/api/v1/visualization")
|
||||
app.dependency_overrides[visualization.get_db] = lambda: object()
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
requests = [
|
||||
asyncio.create_task(client.get("/api/v1/visualization/geo/summary")) for _ in range(6)
|
||||
]
|
||||
await asyncio.wait_for(started.wait(), 1)
|
||||
# Keep the build open beyond the old 200 ms fallback to duplicate SQL.
|
||||
await asyncio.sleep(0.25)
|
||||
assert calls == 1
|
||||
finish.set()
|
||||
results = await asyncio.gather(*requests)
|
||||
assert all(response.status_code == 200 for response in results)
|
||||
assert all(response.json() == {"stats": {"vessel_count": 42}} for response in results)
|
||||
assert sorted(response.headers["X-Planet-Cache"] for response in results) == [
|
||||
"hit",
|
||||
"hit",
|
||||
"hit",
|
||||
"hit",
|
||||
"hit",
|
||||
"refresh",
|
||||
]
|
||||
assert not earth_layer_cache._inflight
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("error_type", [RuntimeError, TimeoutError])
|
||||
async def test_shared_builder_error_is_not_retried(error_type):
|
||||
started = asyncio.Event()
|
||||
finish = asyncio.Event()
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
started.set()
|
||||
await finish.wait()
|
||||
raise error_type("db failed")
|
||||
|
||||
requests = [
|
||||
asyncio.create_task(
|
||||
resolve_layer_payload(
|
||||
key="shared-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
|
||||
)
|
||||
)
|
||||
for _ in range(4)
|
||||
]
|
||||
await started.wait()
|
||||
await asyncio.sleep(0)
|
||||
finish.set()
|
||||
results = await asyncio.gather(*requests, return_exceptions=True)
|
||||
assert calls == 1
|
||||
assert all(isinstance(result, error_type) for result in results)
|
||||
assert not earth_layer_cache._inflight
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_redis_failure_shares_bypass_result(fake_cache_client):
|
||||
fake_cache_client.fail = True
|
||||
started = asyncio.Event()
|
||||
finish = asyncio.Event()
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
started.set()
|
||||
await finish.wait()
|
||||
return {"count": 42}
|
||||
|
||||
requests = [
|
||||
asyncio.create_task(
|
||||
resolve_layer_payload(
|
||||
key="shared-bypass", policy=EarthLayerCachePolicy(60, 120), builder=builder
|
||||
)
|
||||
)
|
||||
for _ in range(4)
|
||||
]
|
||||
await started.wait()
|
||||
await asyncio.sleep(0)
|
||||
finish.set()
|
||||
results = await asyncio.gather(*requests)
|
||||
assert calls == 1
|
||||
assert all(result.state == "bypass" and result.payload == {"count": 42} for result in results)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancelled_waiter_does_not_cancel_owner():
|
||||
started = asyncio.Event()
|
||||
finish = asyncio.Event()
|
||||
policy = EarthLayerCachePolicy(60, 120)
|
||||
|
||||
async def builder():
|
||||
started.set()
|
||||
await finish.wait()
|
||||
return {"count": 1}
|
||||
|
||||
owner = asyncio.create_task(
|
||||
resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
|
||||
)
|
||||
await started.wait()
|
||||
waiter = asyncio.create_task(
|
||||
resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
|
||||
)
|
||||
await asyncio.sleep(0)
|
||||
waiter.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await waiter
|
||||
finish.set()
|
||||
assert (await owner).payload == {"count": 1}
|
||||
assert not earth_layer_cache._inflight
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancelled_owner_allows_waiter_to_use_its_own_builder():
|
||||
started = asyncio.Event()
|
||||
policy = EarthLayerCachePolicy(60, 120)
|
||||
|
||||
async def owner_builder():
|
||||
started.set()
|
||||
await asyncio.Event().wait()
|
||||
|
||||
async def waiter_builder():
|
||||
return {"count": 2}
|
||||
|
||||
owner = asyncio.create_task(
|
||||
resolve_layer_payload(key="cancel-owner", policy=policy, builder=owner_builder)
|
||||
)
|
||||
await started.wait()
|
||||
waiter = asyncio.create_task(
|
||||
resolve_layer_payload(key="cancel-owner", policy=policy, builder=waiter_builder)
|
||||
)
|
||||
await asyncio.sleep(0)
|
||||
owner.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await owner
|
||||
assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 2}
|
||||
assert not earth_layer_cache._inflight
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cross_process_lock_waits_for_payload_instead_of_building(fake_cache_client):
|
||||
key = "remote-owner"
|
||||
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
|
||||
policy = EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005)
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return {"count": -1}
|
||||
|
||||
waiter = asyncio.create_task(resolve_layer_payload(key=key, policy=policy, builder=builder))
|
||||
await asyncio.sleep(0.03)
|
||||
assert calls == 0
|
||||
fake_cache_client.store[key] = json.dumps({"count": 3})
|
||||
assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 3}
|
||||
assert calls == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_timeout_does_not_bypass_other_owner(fake_cache_client):
|
||||
key = "remote-timeout"
|
||||
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return {}
|
||||
|
||||
with pytest.raises(HTTPException) as error:
|
||||
await resolve_layer_payload(
|
||||
key=key,
|
||||
policy=EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005, build_wait_seconds=0.02),
|
||||
builder=builder,
|
||||
)
|
||||
assert error.value.status_code == 503
|
||||
assert calls == 0
|
||||
assert not earth_layer_cache._inflight
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lost_owner_cannot_publish_or_release_new_lock(fake_cache_client):
|
||||
key = "lock-replaced"
|
||||
policy = EarthLayerCachePolicy(60, 120)
|
||||
|
||||
async def builder():
|
||||
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "new-owner"
|
||||
fake_cache_client.store[key] = json.dumps({"count": 99})
|
||||
return {"count": 1}
|
||||
|
||||
result = await resolve_layer_payload(key=key, policy=policy, builder=builder)
|
||||
assert result.payload == {"count": 1}
|
||||
assert result.state == "bypass"
|
||||
assert fake_cache_client.store[earth_layer_cache.lock_key(key)] == "new-owner"
|
||||
assert json.loads(fake_cache_client.store[key]) == {"count": 99}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_redis_publish_failure_returns_built_payload_without_retry(fake_cache_client):
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
fake_cache_client.fail = True
|
||||
return {"count": 7}
|
||||
|
||||
result = await resolve_layer_payload(
|
||||
key="publish-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
|
||||
)
|
||||
assert result.state == "bypass"
|
||||
assert result.payload == {"count": 7}
|
||||
assert calls == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_local_wait_timeout_leaves_owner_running():
|
||||
started = asyncio.Event()
|
||||
finish = asyncio.Event()
|
||||
policy = EarthLayerCachePolicy(60, 120, build_wait_seconds=0.02)
|
||||
|
||||
async def builder():
|
||||
started.set()
|
||||
await finish.wait()
|
||||
return {"count": 1}
|
||||
|
||||
owner = asyncio.create_task(
|
||||
resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
|
||||
)
|
||||
await started.wait()
|
||||
with pytest.raises(HTTPException) as error:
|
||||
await resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
|
||||
assert error.value.status_code == 503
|
||||
assert not owner.done()
|
||||
finish.set()
|
||||
assert (await owner).payload == {"count": 1}
|
||||
assert not earth_layer_cache._inflight
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_different_keys_can_build_concurrently():
|
||||
started = set()
|
||||
both_started = asyncio.Event()
|
||||
|
||||
async def build(key):
|
||||
started.add(key)
|
||||
if len(started) == 2:
|
||||
both_started.set()
|
||||
await asyncio.wait_for(both_started.wait(), 1)
|
||||
return {"key": key}
|
||||
|
||||
results = await asyncio.gather(
|
||||
*(
|
||||
resolve_layer_payload(
|
||||
key=key, policy=EarthLayerCachePolicy(60, 120), builder=lambda key=key: build(key)
|
||||
)
|
||||
for key in ("one", "two")
|
||||
)
|
||||
)
|
||||
assert {result.payload["key"] for result in results} == {"one", "two"}
|
||||
|
||||
@@ -3,6 +3,7 @@ from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from app.api.v1.news import get_earth_feed
|
||||
from app.services.earth_news import (
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
@@ -1201,7 +1202,6 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
|
||||
db=FakeDb(),
|
||||
)
|
||||
|
||||
assert captured["freshness_region"] == "europe"
|
||||
assert captured["items_region"] == "europe"
|
||||
assert captured["items_categories"] == {"business", "ecommerce"}
|
||||
assert captured["items_source_ids"][0] is None
|
||||
@@ -1237,71 +1237,41 @@ async def test_news_source_test_treats_type_reference_as_non_fetching():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
|
||||
async def test_earth_news_feed_get_does_not_fetch_write_or_queue_rss(monkeypatch):
|
||||
db = object()
|
||||
source = NewsFeedSource(
|
||||
id="test-feed",
|
||||
name="Test Feed",
|
||||
region="global",
|
||||
homepage_url="https://example.com",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
read_calls = 0
|
||||
async def fail_side_effect(*_args, **_kwargs):
|
||||
raise AssertionError("RSS feed reads must not fetch, persist, or enqueue work")
|
||||
|
||||
async def empty_news_items(*_args, **_kwargs):
|
||||
nonlocal read_calls
|
||||
read_calls += 1
|
||||
return []
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_side_effect)
|
||||
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fail_side_effect)
|
||||
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fail_side_effect)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_store.list_earth_news_items", empty_news_items
|
||||
)
|
||||
item = ParsedNewsItem(
|
||||
id="test-feed:init",
|
||||
title="Initial RSS story",
|
||||
summary="Initial summary",
|
||||
url="https://example.com/init",
|
||||
source="Test Feed",
|
||||
feed_name="Test Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
|
||||
payload = await get_earth_feed(
|
||||
lat=None,
|
||||
lon=None,
|
||||
region=None,
|
||||
categories=None,
|
||||
sources=None,
|
||||
limit=None,
|
||||
locale=None,
|
||||
db=db,
|
||||
)
|
||||
item.location_patch = {
|
||||
"latitude": 20.0,
|
||||
"longitude": 0.0,
|
||||
"location_label": "全球",
|
||||
"location_source": "region_anchor",
|
||||
"verified": False,
|
||||
"location_meta": {"target": None, "anchor": {"region": "global"}},
|
||||
}
|
||||
upserted = []
|
||||
enqueued = []
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 0, None
|
||||
|
||||
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
|
||||
return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}}
|
||||
|
||||
async def fake_upsert_earth_news_items(_db, items):
|
||||
upserted.extend(items)
|
||||
return len(items)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
return [item]
|
||||
|
||||
async def fake_enqueue_target_location_job(payload, **_kwargs):
|
||||
enqueued.append(payload)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
|
||||
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
|
||||
|
||||
payload = await get_earth_news_payload(db=db)
|
||||
|
||||
assert upserted[0].id == "test-feed:init"
|
||||
assert payload["items"][0]["id"] == "test-feed:init"
|
||||
assert payload["items"][0]["verified"] is False
|
||||
assert enqueued[0]["id"] == "test-feed:init"
|
||||
assert payload["items"] == []
|
||||
assert read_calls == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
|
||||
async def test_earth_news_payload_does_not_supplement_stale_database_items(monkeypatch):
|
||||
db = object()
|
||||
old_item = ParsedNewsItem(
|
||||
id="db:old",
|
||||
@@ -1314,41 +1284,17 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
old_item.location_patch = {
|
||||
"latitude": 20.0,
|
||||
"longitude": 0.0,
|
||||
"location_label": "全球",
|
||||
"location_source": "region_anchor",
|
||||
"verified": False,
|
||||
"location_meta": {"target": None, "anchor": {"region": "global"}},
|
||||
}
|
||||
fetched = []
|
||||
async def fail_fetch(*_args, **_kwargs):
|
||||
raise AssertionError("stale RSS supplementation belongs to the background worker")
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
|
||||
|
||||
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
|
||||
fetched.append(True)
|
||||
return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}}
|
||||
|
||||
async def fake_upsert_earth_news_items(_db, items):
|
||||
return len(items)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
|
||||
return [old_item]
|
||||
|
||||
async def fake_enqueue_target_location_job(_payload, **_kwargs):
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
|
||||
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
|
||||
|
||||
payload = await get_earth_news_payload(db=db)
|
||||
payload = await get_earth_news_payload(db=db, source_ids={"stored"})
|
||||
|
||||
assert fetched == [True]
|
||||
assert payload["items"][0]["id"] == "db:old"
|
||||
|
||||
|
||||
|
||||
86
backend/tests/test_earth_news_encoding.py
Normal file
86
backend/tests/test_earth_news_encoding.py
Normal file
@@ -0,0 +1,86 @@
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import (
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
ParsedNewsItem,
|
||||
_fetch_single_feed_url,
|
||||
_serialize_item,
|
||||
)
|
||||
|
||||
|
||||
async def fetch_feed(
|
||||
content: bytes, content_type: str = "text/xml"
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
|
||||
source = NewsFeedSource(
|
||||
id="ebrun",
|
||||
name="亿邦动力",
|
||||
region="asia-pacific",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
homepage_url="https://example.com/",
|
||||
)
|
||||
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
|
||||
transport = httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200, content=content, headers={"content-type": content_type}
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
return await _fetch_single_feed_url(client, source, feed)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("encoding", "declaration", "content_type"),
|
||||
[
|
||||
("gbk", 'encoding="GBK"', "text/xml"),
|
||||
("gb2312", "encoding='GB2312'", "application/rss+xml"),
|
||||
("gb18030", 'encoding="GB18030"', "text/xml"),
|
||||
("gbk", "", "text/xml; charset=gbk"),
|
||||
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
|
||||
("utf-8", "", "text/xml"),
|
||||
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
|
||||
("utf-16", 'encoding="UTF-16"', "text/xml"),
|
||||
("utf-32", 'encoding="UTF-32"', "text/xml"),
|
||||
],
|
||||
)
|
||||
async def test_feed_encoding_preserves_chinese_in_display(
|
||||
encoding: str, declaration: str, content_type: str
|
||||
):
|
||||
title = "618电商战报:零售额增长266%"
|
||||
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
|
||||
xml = f'''<?xml version="1.0" {declaration}?>
|
||||
<rss version="2.0"><channel><item>
|
||||
<title><![CDATA[{title}]]></title>
|
||||
<description><![CDATA[<p>{summary}</p>]]></description>
|
||||
<link>https://example.com/story</link>
|
||||
</item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
|
||||
|
||||
assert error is None
|
||||
assert health["ok"] is True
|
||||
assert len(items) == 1
|
||||
assert items[0].title == title
|
||||
assert items[0].summary == summary
|
||||
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
|
||||
assert payload["display_title"] == title
|
||||
assert payload["display_summary"] == summary
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
|
||||
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
|
||||
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
|
||||
<rss><channel><item><title>电商新闻</title>
|
||||
<link>https://example.com/story</link></item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode("gbk"))
|
||||
|
||||
assert items == []
|
||||
assert error
|
||||
assert health["ok"] is False
|
||||
assert health["status"] == "format_error"
|
||||
525
backend/tests/test_earth_news_refresh.py
Normal file
525
backend/tests/test_earth_news_refresh.py
Normal file
@@ -0,0 +1,525 @@
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import NewsFeedEndpoint, NewsFeedSource
|
||||
from app.services import earth_news_refresh
|
||||
from app.services.scheduler import (
|
||||
EARTH_NEWS_REFRESH_JOB_ID,
|
||||
_earth_news_region_refresh_job_id,
|
||||
schedule_earth_news_region_refresh,
|
||||
start_scheduler,
|
||||
sync_scheduler_with_datasources,
|
||||
)
|
||||
|
||||
|
||||
class FakeRedis:
|
||||
def __init__(self) -> None:
|
||||
self.values: dict[str, str] = {}
|
||||
|
||||
def set(self, key: str, value: str, *, nx: bool = False, ex: int | None = None) -> bool:
|
||||
del ex
|
||||
if nx and key in self.values:
|
||||
return False
|
||||
self.values[key] = value
|
||||
return True
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
return key in self.values
|
||||
|
||||
def eval(self, script: str, _key_count: int, key: str, token: str, *_args: object) -> int:
|
||||
if self.values.get(key) != token:
|
||||
return 0
|
||||
if "redis.call('del'" in script:
|
||||
del self.values[key]
|
||||
return 1
|
||||
|
||||
|
||||
class FakeStatusRedis:
|
||||
def __init__(self) -> None:
|
||||
self.values: dict[str, str] = {}
|
||||
self.expirations: dict[str, int] = {}
|
||||
|
||||
def get(self, key: str):
|
||||
return self.values.get(key)
|
||||
|
||||
def eval(self, script: str, key_count: int, *args):
|
||||
keys = args[:key_count]
|
||||
values = args[key_count:]
|
||||
if "local active_id" in script:
|
||||
active_key, status_key = keys
|
||||
refresh_id, encoded, active_ttl, status_ttl, status_prefix = values
|
||||
active_id = self.values.get(active_key)
|
||||
if active_id:
|
||||
active_status = self.values.get(f"{status_prefix}{active_id}")
|
||||
if active_status:
|
||||
status = json.loads(active_status)["status"]
|
||||
if status in {"pending", "running"}:
|
||||
return [active_id, active_status, 0]
|
||||
self.values.pop(active_key, None)
|
||||
self.values[active_key] = refresh_id
|
||||
self.values[status_key] = encoded
|
||||
self.expirations[active_key] = int(active_ttl)
|
||||
self.expirations[status_key] = int(status_ttl)
|
||||
return [refresh_id, encoded, 1]
|
||||
|
||||
active_key, status_key = keys
|
||||
encoded, refresh_id, status_ttl, status, active_ttl = values
|
||||
if self.values.get(active_key) != refresh_id:
|
||||
return 0
|
||||
self.values[status_key] = encoded
|
||||
self.expirations[status_key] = int(status_ttl)
|
||||
if status in {"succeeded", "failed"}:
|
||||
self.values.pop(active_key, None)
|
||||
else:
|
||||
self.expirations[active_key] = int(active_ttl)
|
||||
return 1
|
||||
|
||||
|
||||
class FakeSession:
|
||||
def __init__(self) -> None:
|
||||
self.commits = 0
|
||||
self.rollbacks = 0
|
||||
|
||||
async def commit(self) -> None:
|
||||
self.commits += 1
|
||||
|
||||
async def rollback(self) -> None:
|
||||
self.rollbacks += 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_region_refresh_fetches_only_missing_feed_coverage(monkeypatch):
|
||||
redis_client = FakeRedis()
|
||||
db = FakeSession()
|
||||
feeds = (
|
||||
NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global"),
|
||||
NewsFeedEndpoint("feed-b", "Feed B", "https://example.com/b.xml", region="global"),
|
||||
)
|
||||
source = NewsFeedSource(
|
||||
id="source-a",
|
||||
name="Source A",
|
||||
region="global",
|
||||
feed_url=feeds[0].url,
|
||||
homepage_url="https://example.com",
|
||||
feeds=feeds,
|
||||
)
|
||||
fetched_source_feeds: list[list[str]] = []
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_session_factory():
|
||||
yield db
|
||||
|
||||
async def fake_get_sources_payload(_db):
|
||||
return {"sources": []}
|
||||
|
||||
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
|
||||
assert config_payload == {"sources": []}
|
||||
return [source]
|
||||
|
||||
async def fake_freshness(_db, *, active_region):
|
||||
assert active_region == "global"
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_coverage(_db, *, active_region, recent_after):
|
||||
assert active_region == "global"
|
||||
assert recent_after is not None
|
||||
return {("source-a", "feed-a")}
|
||||
|
||||
async def fake_fetch(sources, **_kwargs):
|
||||
fetched_source_feeds.extend([[feed.id for feed in item.feeds] for item in sources])
|
||||
return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
|
||||
|
||||
async def fake_record_health(_db, _health):
|
||||
return None
|
||||
|
||||
async def fake_enqueue(_items):
|
||||
return None
|
||||
|
||||
async def fake_upsert(_db, items):
|
||||
assert items == []
|
||||
return 0
|
||||
|
||||
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
|
||||
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
|
||||
)
|
||||
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
|
||||
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.record_earth_news_sources_health", fake_record_health
|
||||
)
|
||||
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_enqueue)
|
||||
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
|
||||
|
||||
result = await earth_news_refresh.refresh_earth_news_region("global")
|
||||
|
||||
assert result["state"] == "completed"
|
||||
assert result["successful_sources"] == 1
|
||||
assert fetched_source_feeds == [["feed-b"]]
|
||||
assert db.commits == 1
|
||||
assert earth_news_refresh._lock_key("global") not in redis_client.values
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_region_refresh_reports_failure_when_all_sources_time_out(monkeypatch):
|
||||
redis_client = FakeRedis()
|
||||
db = FakeSession()
|
||||
feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global")
|
||||
source = NewsFeedSource(
|
||||
id="source-a",
|
||||
name="Source A",
|
||||
region="global",
|
||||
feed_url=feed.url,
|
||||
homepage_url="https://example.com",
|
||||
feeds=(feed,),
|
||||
)
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_session_factory():
|
||||
yield db
|
||||
|
||||
async def fake_get_sources_payload(_db):
|
||||
return {"sources": []}
|
||||
|
||||
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
|
||||
return [source]
|
||||
|
||||
async def fake_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_coverage(_db, *, active_region, recent_after):
|
||||
return set()
|
||||
|
||||
async def fake_fetch(*_args, **_kwargs):
|
||||
return (
|
||||
[],
|
||||
["private upstream details"],
|
||||
{
|
||||
"source-a": {
|
||||
"source_id": "source-a",
|
||||
"ok": False,
|
||||
"status": "timeout",
|
||||
"feed_results": [{"feed_id": "feed-a", "ok": False, "status": "timeout"}],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
async def fake_noop(*_args, **_kwargs):
|
||||
return None
|
||||
|
||||
async def fake_upsert(_db, _items):
|
||||
return 0
|
||||
|
||||
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
|
||||
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
|
||||
)
|
||||
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
|
||||
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
|
||||
monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
|
||||
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
|
||||
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
|
||||
|
||||
result = await earth_news_refresh.refresh_earth_news_region("global", force=True)
|
||||
|
||||
assert result["state"] == "failed"
|
||||
assert result["error_code"] == "timeout"
|
||||
assert db.commits == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_database_commit_failure_sets_refresh_failed_status(monkeypatch):
|
||||
class CommitFailureSession(FakeSession):
|
||||
async def commit(self) -> None:
|
||||
raise RuntimeError("database commit failed with sensitive context")
|
||||
|
||||
redis_client = FakeRedis()
|
||||
db = CommitFailureSession()
|
||||
feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="europe")
|
||||
source = NewsFeedSource(
|
||||
id="source-a",
|
||||
name="Source A",
|
||||
region="europe",
|
||||
feed_url=feed.url,
|
||||
homepage_url="https://example.com",
|
||||
feeds=(feed,),
|
||||
)
|
||||
scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
|
||||
statuses = []
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_session_factory():
|
||||
yield db
|
||||
|
||||
async def fake_get_sources_payload(_db):
|
||||
return {"sources": []}
|
||||
|
||||
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
|
||||
return [source]
|
||||
|
||||
async def fake_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_coverage(_db, *, active_region, recent_after):
|
||||
return set()
|
||||
|
||||
async def fake_fetch(*_args, **_kwargs):
|
||||
return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
|
||||
|
||||
async def fake_noop(*_args, **_kwargs):
|
||||
return None
|
||||
|
||||
async def fake_upsert(_db, _items):
|
||||
return 1
|
||||
|
||||
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
|
||||
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
|
||||
)
|
||||
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
|
||||
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
|
||||
monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
|
||||
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
|
||||
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
|
||||
monkeypatch.setattr(scheduler_module, "touch_earth_news_refresh_status", lambda *_args: True)
|
||||
monkeypatch.setattr(
|
||||
scheduler_module,
|
||||
"set_earth_news_refresh_status",
|
||||
lambda refresh_id, region, status, *, error=None: statuses.append(
|
||||
{"refresh_id": refresh_id, "region": region, "status": status, "error": error}
|
||||
),
|
||||
)
|
||||
scheduler_module._pending_earth_news_refreshes["europe"] = {
|
||||
"force": True,
|
||||
"refresh_id": "refresh-commit-failure",
|
||||
}
|
||||
|
||||
await scheduler_module._run_earth_news_region_refresh("europe")
|
||||
|
||||
assert [entry["status"] for entry in statuses] == ["running", "failed"]
|
||||
assert statuses[-1]["error"]["code"] == "refresh_failed"
|
||||
assert "sensitive context" not in repr(statuses)
|
||||
assert not redis_client.values
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_region_refresh_skips_when_cross_process_lock_is_held(monkeypatch):
|
||||
redis_client = FakeRedis()
|
||||
redis_client.values[earth_news_refresh._lock_key("global")] = "other-owner"
|
||||
|
||||
def fail_session_factory():
|
||||
raise AssertionError("a locked refresh must not open a database session")
|
||||
|
||||
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
|
||||
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
|
||||
|
||||
result = await earth_news_refresh.refresh_earth_news_region("global")
|
||||
|
||||
assert result["state"] == "busy"
|
||||
|
||||
|
||||
def test_failed_feed_enters_cooldown_without_blocking_other_sources():
|
||||
redis_client = FakeRedis()
|
||||
source = NewsFeedSource(
|
||||
id="source-a",
|
||||
name="Source A",
|
||||
region="global",
|
||||
feed_url="https://example.com/a.xml",
|
||||
homepage_url="https://example.com",
|
||||
feeds=(NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml"),),
|
||||
)
|
||||
health = {
|
||||
"source-a": {
|
||||
"ok": False,
|
||||
"feed_results": [{"feed_id": "feed-a", "ok": False}],
|
||||
}
|
||||
}
|
||||
|
||||
earth_news_refresh._record_failed_feed_cooldowns(
|
||||
redis_client,
|
||||
active_region="global",
|
||||
health_by_source=health,
|
||||
sources=[source],
|
||||
)
|
||||
selected, cooling = earth_news_refresh._select_feeds(
|
||||
[source],
|
||||
active_region="global",
|
||||
required_keys={("source-a", "feed-a")},
|
||||
redis_client=redis_client,
|
||||
)
|
||||
|
||||
assert selected == []
|
||||
assert cooling == {("source-a", "feed-a")}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_region_refresh_skips_when_redis_coordination_is_unavailable(monkeypatch):
|
||||
def fail_redis_client():
|
||||
raise RuntimeError("redis unavailable")
|
||||
|
||||
def fail_session_factory():
|
||||
raise AssertionError("uncoordinated processes must not start duplicate RSS fetches")
|
||||
|
||||
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", fail_redis_client)
|
||||
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
|
||||
|
||||
result = await earth_news_refresh.refresh_earth_news_region("global")
|
||||
|
||||
assert result["state"] == "coordination_unavailable"
|
||||
|
||||
|
||||
def test_refresh_status_deduplicates_active_region_and_bounds_completed_results(monkeypatch):
|
||||
redis_client = FakeStatusRedis()
|
||||
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
|
||||
|
||||
first, created = earth_news_refresh.create_earth_news_refresh_status("global")
|
||||
duplicate, duplicate_created = earth_news_refresh.create_earth_news_refresh_status("global")
|
||||
assert created is True
|
||||
assert duplicate_created is False
|
||||
assert duplicate == first
|
||||
|
||||
earth_news_refresh.set_earth_news_refresh_status(first["refresh_id"], "global", "running")
|
||||
running, running_created = earth_news_refresh.create_earth_news_refresh_status("global")
|
||||
assert running_created is False
|
||||
assert running["status"] == "running"
|
||||
|
||||
completed = earth_news_refresh.set_earth_news_refresh_status(
|
||||
first["refresh_id"], "global", "succeeded"
|
||||
)
|
||||
assert completed["status"] == "succeeded"
|
||||
assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
|
||||
|
||||
second, second_created = earth_news_refresh.create_earth_news_refresh_status("global")
|
||||
assert second_created is True
|
||||
assert second["refresh_id"] != first["refresh_id"]
|
||||
assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
|
||||
assert redis_client.expirations[earth_news_refresh._refresh_active_key("global")] == 120
|
||||
assert (
|
||||
redis_client.expirations[earth_news_refresh._refresh_status_key(second["refresh_id"])]
|
||||
== 600
|
||||
)
|
||||
status_keys = [
|
||||
key
|
||||
for key in redis_client.values
|
||||
if key.startswith(earth_news_refresh.REFRESH_STATUS_PREFIX)
|
||||
]
|
||||
assert len(status_keys) == 2
|
||||
|
||||
pending, pending_created = earth_news_refresh.create_earth_news_refresh_status("europe")
|
||||
assert pending_created is True
|
||||
redis_client.values.pop(earth_news_refresh._refresh_active_key("europe"), None)
|
||||
expired = earth_news_refresh.get_earth_news_refresh_status(pending["refresh_id"])
|
||||
assert expired["status"] == "failed"
|
||||
assert expired["error"]["code"] == "expired"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_refresh_scheduler_job_survives_datasource_sync(monkeypatch):
|
||||
class FakeJob:
|
||||
def __init__(self, job_id: str) -> None:
|
||||
self.id = job_id
|
||||
|
||||
class FakeScheduler:
|
||||
def __init__(self) -> None:
|
||||
self.running = False
|
||||
self.jobs = {
|
||||
"obsolete": FakeJob("obsolete"),
|
||||
_earth_news_region_refresh_job_id("global"): FakeJob(
|
||||
_earth_news_region_refresh_job_id("global")
|
||||
),
|
||||
}
|
||||
self.added: dict[str, object] = {}
|
||||
|
||||
def add_job(self, _func, **kwargs):
|
||||
self.added.update(kwargs)
|
||||
self.jobs[kwargs["id"]] = FakeJob(kwargs["id"])
|
||||
|
||||
def start(self) -> None:
|
||||
self.running = True
|
||||
|
||||
def get_jobs(self):
|
||||
return list(self.jobs.values())
|
||||
|
||||
def remove_job(self, job_id: str) -> None:
|
||||
self.jobs.pop(job_id, None)
|
||||
|
||||
class FakeResult:
|
||||
def scalars(self):
|
||||
return self
|
||||
|
||||
def all(self):
|
||||
return []
|
||||
|
||||
class FakeDb:
|
||||
async def execute(self, _query):
|
||||
return FakeResult()
|
||||
|
||||
fake_scheduler = FakeScheduler()
|
||||
|
||||
@asynccontextmanager
|
||||
async def fake_session_factory():
|
||||
yield FakeDb()
|
||||
|
||||
monkeypatch.setattr("app.services.scheduler.scheduler", fake_scheduler)
|
||||
monkeypatch.setattr("app.services.scheduler.async_session_factory", fake_session_factory)
|
||||
|
||||
start_scheduler()
|
||||
await sync_scheduler_with_datasources()
|
||||
|
||||
assert fake_scheduler.added["id"] == EARTH_NEWS_REFRESH_JOB_ID
|
||||
assert fake_scheduler.added["next_run_time"] <= datetime.now(UTC)
|
||||
assert EARTH_NEWS_REFRESH_JOB_ID in fake_scheduler.jobs
|
||||
assert _earth_news_region_refresh_job_id("global") in fake_scheduler.jobs
|
||||
assert "obsolete" not in fake_scheduler.jobs
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_region_refresh_requests_are_merged_and_force_is_preserved(monkeypatch):
|
||||
class FakeScheduler:
|
||||
running = True
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.jobs: dict[str, tuple[object, dict[str, object]]] = {}
|
||||
|
||||
def add_job(self, func, **kwargs):
|
||||
self.jobs[kwargs["id"]] = (func, kwargs)
|
||||
|
||||
scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
|
||||
fake_scheduler = FakeScheduler()
|
||||
monkeypatch.setattr(scheduler_module, "scheduler", fake_scheduler)
|
||||
captured: list[tuple[str, bool]] = []
|
||||
|
||||
async def fake_refresh(region: str, *, force: bool = False):
|
||||
captured.append((region, force))
|
||||
return {"locked": False, "fetched": 0, "upserted": 0}
|
||||
|
||||
monkeypatch.setattr(scheduler_module, "refresh_earth_news_region", fake_refresh)
|
||||
|
||||
assert schedule_earth_news_region_refresh("global") is True
|
||||
assert schedule_earth_news_region_refresh("global", force=True) is False
|
||||
assert len(fake_scheduler.jobs) == 1
|
||||
|
||||
job_id = _earth_news_region_refresh_job_id("global")
|
||||
func, kwargs = fake_scheduler.jobs.pop(job_id)
|
||||
await func(**kwargs["kwargs"])
|
||||
|
||||
assert captured == [("global", True)]
|
||||
assert job_id not in fake_scheduler.jobs
|
||||
140
backend/tests/test_earth_outbox_maintenance.py
Normal file
140
backend/tests/test_earth_outbox_maintenance.py
Normal file
@@ -0,0 +1,140 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services import earth_db_change_listener as listener_module
|
||||
from app.services.earth_db_change_listener import EarthDbChangeDispatcher, EarthDbChangeListener
|
||||
from app.services.earth_outbox_maintenance import (
|
||||
DELETE_BATCH_SQL,
|
||||
compact_consumed_events,
|
||||
ensure_retention_index,
|
||||
purge_consumed_events,
|
||||
)
|
||||
|
||||
|
||||
class FakeConnection:
|
||||
def __init__(self, *, index_state: str = "ready", batch_counts: list[int] | None = None):
|
||||
self.index_state = index_state
|
||||
self.batch_counts = list(batch_counts or [])
|
||||
self.calls: list[tuple[str, tuple[object, ...]]] = []
|
||||
self.retained = {"total": 13, "pending": 3}
|
||||
self.restored = dict(self.retained)
|
||||
self.rolled_back = False
|
||||
|
||||
async def fetchrow(self, query: str, *args: object):
|
||||
self.calls.append((query, args))
|
||||
if "pg_index" in query:
|
||||
if self.index_state == "missing":
|
||||
return None
|
||||
return {"indisvalid": self.index_state == "ready", "indisready": True}
|
||||
if "FROM earth_outbox_retained" in query:
|
||||
return self.retained
|
||||
return self.restored
|
||||
|
||||
async def fetchval(self, query: str, *args: object):
|
||||
self.calls.append((query, args))
|
||||
if query == DELETE_BATCH_SQL:
|
||||
return self.batch_counts.pop(0)
|
||||
return True
|
||||
|
||||
async def execute(self, query: str, *args: object):
|
||||
self.calls.append((query, args))
|
||||
if "CREATE INDEX CONCURRENTLY" in query:
|
||||
self.index_state = "ready"
|
||||
return "OK"
|
||||
|
||||
def transaction(self):
|
||||
connection = self
|
||||
|
||||
class Transaction:
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, _exc, _traceback):
|
||||
connection.rolled_back = exc_type is not None
|
||||
|
||||
return Transaction()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retention_skips_without_index():
|
||||
conn = FakeConnection(index_state="missing")
|
||||
assert await purge_consumed_events(conn, retention_days=1) == 0
|
||||
assert not any(query == DELETE_BATCH_SQL for query, _ in conn.calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retention_batches_only_consumed_old_rows():
|
||||
conn = FakeConnection(batch_counts=[1000, 19])
|
||||
deleted = await purge_consumed_events(
|
||||
conn, retention_days=1, batch_size=1000, max_batches=10, pause_seconds=0
|
||||
)
|
||||
assert deleted == 1019
|
||||
calls = [args for query, args in conn.calls if query == DELETE_BATCH_SQL]
|
||||
assert len(calls) == 2
|
||||
assert all(isinstance(args[0], datetime) and args[0].tzinfo is UTC for args in calls)
|
||||
assert "consumed_at IS NOT NULL AND consumed_at < $1" in DELETE_BATCH_SQL
|
||||
assert "FOR UPDATE SKIP LOCKED" in DELETE_BATCH_SQL
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_index_creation_is_concurrent_and_repairs_invalid_index():
|
||||
conn = FakeConnection(index_state="invalid")
|
||||
await ensure_retention_index(conn)
|
||||
statements = [query for query, _ in conn.calls]
|
||||
assert any("DROP INDEX CONCURRENTLY" in query for query in statements)
|
||||
assert any("CREATE INDEX CONCURRENTLY" in query for query in statements)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compact_retains_pending_and_recent_consumed_in_one_transaction():
|
||||
conn = FakeConnection()
|
||||
assert await compact_consumed_events(conn, retention_days=1) == (13, 3)
|
||||
statements = [query for query, _ in conn.calls]
|
||||
assert next(i for i, query in enumerate(statements) if query.startswith("LOCK TABLE")) < next(
|
||||
i for i, query in enumerate(statements) if query.startswith("TRUNCATE TABLE")
|
||||
)
|
||||
assert any("consumed_at IS NULL OR consumed_at >= $1" in query for query in statements)
|
||||
assert "TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT" in statements
|
||||
assert not conn.rolled_back
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compact_verification_failure_rolls_back():
|
||||
conn = FakeConnection()
|
||||
conn.restored = {"total": 12, "pending": 2}
|
||||
with pytest.raises(RuntimeError, match="verification failed"):
|
||||
await compact_consumed_events(conn, retention_days=1)
|
||||
assert conn.rolled_back
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_listener_runs_retention_at_most_once_per_interval(monkeypatch):
|
||||
calls: list[str] = []
|
||||
|
||||
async def acquire(_conn):
|
||||
calls.append("acquire")
|
||||
return True
|
||||
|
||||
async def purge(_conn, *, retention_days):
|
||||
assert retention_days == 1
|
||||
calls.append("purge")
|
||||
|
||||
async def release(_conn):
|
||||
calls.append("release")
|
||||
|
||||
monkeypatch.setattr(listener_module, "monotonic", lambda: 100.0)
|
||||
monkeypatch.setattr(listener_module, "try_maintenance_lock", acquire)
|
||||
monkeypatch.setattr(listener_module, "purge_consumed_events", purge)
|
||||
monkeypatch.setattr(listener_module, "release_maintenance_lock", release)
|
||||
monkeypatch.setattr(listener_module, "configured_retention_days", lambda: 1)
|
||||
dispatcher = EarthDbChangeDispatcher(broadcast_earth_update=lambda _: None)
|
||||
listener = EarthDbChangeListener(dsn="postgresql://example/db", dispatcher=dispatcher)
|
||||
listener._connection = FakeConnection()
|
||||
|
||||
await listener._maybe_purge_outbox()
|
||||
await listener._maybe_purge_outbox()
|
||||
|
||||
assert calls == ["acquire", "purge", "release"]
|
||||
@@ -10,7 +10,7 @@ from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
from app.services import business_logs
|
||||
from app.services import persistent_logs
|
||||
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup
|
||||
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup, SystemLog
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
@@ -100,6 +100,23 @@ def test_business_context_redacts_nested_sensitive_values():
|
||||
assert context["nested"]["safe"] == "visible"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_business_log_preserves_events_in_isolated_store(persisted_test_logs):
|
||||
await business_logs.emit_business_log(
|
||||
get_logger("tests.persistence"),
|
||||
event="test.business.completed",
|
||||
message="Business action completed",
|
||||
category="test",
|
||||
context={"api_key": "test-secret", "record_count": 2},
|
||||
)
|
||||
|
||||
logs = [entry for entry in persisted_test_logs if isinstance(entry, SystemLog)]
|
||||
assert len(logs) == 1
|
||||
assert logs[0].event == "test.business.completed"
|
||||
assert logs[0].context == {"api_key": "[REDACTED]", "record_count": 2}
|
||||
assert any(isinstance(entry, ObservabilityEventGroup) for entry in persisted_test_logs)
|
||||
|
||||
|
||||
def test_observability_fingerprint_normalizes_hls_fragments():
|
||||
first = persistent_logs.build_observability_fingerprint(
|
||||
source="earth-client",
|
||||
|
||||
142
backend/tests/test_news_refresh_api.py
Normal file
142
backend/tests/test_news_refresh_api.py
Normal file
@@ -0,0 +1,142 @@
|
||||
from types import SimpleNamespace
|
||||
from uuid import UUID
|
||||
import importlib
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.api.v1 import earth, news
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_post_news_refresh_only_schedules_background_region_check(monkeypatch):
|
||||
scheduled = []
|
||||
|
||||
def request(region: str):
|
||||
scheduled.append(region)
|
||||
return {"refresh_id": "refresh-europe", "region": region, "status": "pending"}
|
||||
|
||||
async def fail_network_refresh(*_args, **_kwargs):
|
||||
raise AssertionError("the POST handler must not perform RSS requests")
|
||||
|
||||
monkeypatch.setattr("app.services.scheduler.request_earth_news_region_refresh", request)
|
||||
monkeypatch.setattr("app.services.scheduler.refresh_earth_news_region", fail_network_refresh)
|
||||
|
||||
response = await news.request_earth_news_refresh(
|
||||
news.EarthNewsRefreshRequest(region="Europe", lat=48.8, lon=2.3)
|
||||
)
|
||||
|
||||
assert response == {"refresh_id": "refresh-europe", "region": "europe", "status": "pending"}
|
||||
assert scheduled == ["europe"]
|
||||
refresh_route = next(route for route in news.router.routes if route.path == "/refresh")
|
||||
assert refresh_route.status_code == 202
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_post_news_refresh_infers_region_and_rejects_unknown_region(monkeypatch):
|
||||
scheduled = []
|
||||
monkeypatch.setattr(
|
||||
"app.services.scheduler.request_earth_news_region_refresh",
|
||||
lambda region: scheduled.append(region)
|
||||
or {"refresh_id": "refresh-americas", "region": region, "status": "pending"},
|
||||
)
|
||||
|
||||
response = await news.request_earth_news_refresh(
|
||||
news.EarthNewsRefreshRequest(lat=40.7, lon=-74.0)
|
||||
)
|
||||
|
||||
assert response["region"] == "americas"
|
||||
assert response["status"] == "pending"
|
||||
assert scheduled == ["americas"]
|
||||
with pytest.raises(HTTPException) as error:
|
||||
await news.request_earth_news_refresh(news.EarthNewsRefreshRequest(region="unknown"))
|
||||
assert error.value.status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_news_refresh_returns_status_and_expired_shape(monkeypatch):
|
||||
current = {"refresh_id": "abc", "region": "global", "status": "running"}
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_refresh.get_earth_news_refresh_status",
|
||||
lambda _refresh_id: current,
|
||||
)
|
||||
|
||||
response = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
|
||||
|
||||
assert response is current
|
||||
assert response["status"] == "running"
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_refresh.get_earth_news_refresh_status", lambda _id: None
|
||||
)
|
||||
expired = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
|
||||
assert expired == {
|
||||
"refresh_id": UUID("00000000-0000-0000-0000-000000000abc").hex,
|
||||
"region": None,
|
||||
"status": "failed",
|
||||
"error": {
|
||||
"code": "expired",
|
||||
"message": "Refresh status expired. Submit a new refresh.",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_database_failure_sets_terminal_failed_status_without_leaking_error(monkeypatch):
|
||||
scheduler = importlib.import_module("app.services.scheduler")
|
||||
statuses = []
|
||||
|
||||
async def fail_refresh(_region, *, force=False):
|
||||
raise RuntimeError("postgres://private-token@host/database: commit failed")
|
||||
|
||||
monkeypatch.setattr(scheduler, "refresh_earth_news_region", fail_refresh)
|
||||
monkeypatch.setattr(scheduler, "touch_earth_news_refresh_status", lambda *_args: True)
|
||||
monkeypatch.setattr(
|
||||
scheduler,
|
||||
"set_earth_news_refresh_status",
|
||||
lambda refresh_id, region, state, *, error=None: statuses.append(
|
||||
{"refresh_id": refresh_id, "region": region, "status": state, "error": error}
|
||||
),
|
||||
)
|
||||
scheduler._pending_earth_news_refreshes["europe"] = {
|
||||
"force": True,
|
||||
"refresh_id": "refresh-db-failure",
|
||||
}
|
||||
|
||||
await scheduler._run_earth_news_region_refresh("europe")
|
||||
|
||||
assert [item["status"] for item in statuses] == ["running", "failed"]
|
||||
assert statuses[-1]["error"] == {
|
||||
"code": "refresh_failed",
|
||||
"message": "Could not refresh news. Please try again.",
|
||||
}
|
||||
assert "private-token" not in repr(statuses)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_news_source_save_and_reset_schedule_immediate_refresh(monkeypatch):
|
||||
scheduled = []
|
||||
monkeypatch.setattr(
|
||||
earth,
|
||||
"save_earth_news_sources_payload",
|
||||
lambda _db, _payload: _async_result({"status": "updated"}),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
earth,
|
||||
"reset_earth_news_sources_payload",
|
||||
lambda _db: _async_result({"status": "reset"}),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.scheduler.schedule_earth_news_refresh_for_all_regions",
|
||||
lambda *, force=False: scheduled.append(force),
|
||||
)
|
||||
|
||||
await earth.update_earth_news_sources(
|
||||
SimpleNamespace(model_dump=lambda: {}), _current_user=None, db=object()
|
||||
)
|
||||
await earth.reset_earth_news_sources(_current_user=None, db=object())
|
||||
|
||||
assert scheduled == [True, True]
|
||||
|
||||
|
||||
async def _async_result(value):
|
||||
return value
|
||||
@@ -741,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
candidate = compute_center_locations.LocationCandidate(
|
||||
latitude=45.764,
|
||||
longitude=4.8357,
|
||||
@@ -784,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
llm_candidate = compute_center_locations.LocationCandidate(
|
||||
latitude=45.764,
|
||||
longitude=4.8357,
|
||||
@@ -851,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
|
||||
async def build_without_shared_cache(**kwargs):
|
||||
return await kwargs["builder"]()
|
||||
|
||||
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
@@ -920,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
async def build_without_shared_cache(**kwargs):
|
||||
return await kwargs["builder"]()
|
||||
|
||||
# Exercise the fixture DB, never the running development server's Redis payload.
|
||||
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
@@ -1013,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query).lower()
|
||||
if "group by collected_data.source" in query_text:
|
||||
return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
@@ -1083,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
def _fake_ror(query):
|
||||
assert query == "Oak Ridge National Laboratory"
|
||||
return {
|
||||
@@ -1170,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
|
||||
monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
|
||||
|
||||
@@ -1380,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
|
||||
assert "FORBIDDEN_PRECISIONS" in text, (
|
||||
f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
|
||||
def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
|
||||
from app.core.compute_types import compute_site_type
|
||||
record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
|
||||
name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
|
||||
metadata={"location_precision": "site"})
|
||||
result = convert_compute_centers_to_geojson([record])
|
||||
assert compute_site_type(record) == kind
|
||||
assert result["features"][0]["properties"]["site_type"] == kind
|
||||
record.source = "epoch_ai_gpu"
|
||||
record.data_type = "gpu_cluster"
|
||||
record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
|
||||
assert compute_site_type(record) == kind
|
||||
assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind
|
||||
|
||||
@@ -8,6 +8,341 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.76.19] — 2026-11-09
|
||||
|
||||
Released: 2026-11-09
|
||||
|
||||
### Highlights
|
||||
- 改善 Earth 桌面与移动端卡片、新闻刷新和数据更新链路,降低重复抓取、缓存重建与历史事件维护的开销。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一区域卡片的展开与胶囊展示、悬停和拖拽跟随,分别应用算力/新闻卡片展开上限;提供细线/粗线引导线及移动端交互改进,修复语言切换后状态提示标题残留。
|
||||
- 新闻 GET 只读取已保存内容,RSS 补抓改为独立调度;手动刷新提供可追踪状态、超时和重试,完成展示后才提示成功。
|
||||
- 图层缓存按 key 合并并发构建,续租与发布校验锁持有者;纯新闻变更不再触发 summary 缓存重建。
|
||||
- BGP 前缀归属使用容错转换和可独立部署的 GiST 索引;Outbox 单次写入事件,跳过纯时间戳变更,并提供有界保留与维护脚本。
|
||||
- 卫星粒子以球体遮挡判断替代点精灵深度裁切;同步中英文文档,补充回归并隔离前端验证构建产物。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.18] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 收敛 Earth 新闻 breaking 交互、卡片 hover、面板稳定性与 ticker 玻璃扫光,并修复船只地表锚点显隐。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- breaking 数据、ticker 当前项、面板生命周期和选中态分离;展开面板时冻结卡片 DOM,后台更新在关闭时应用。
|
||||
- ticker 保留原有容器与滚动,breaking 扫光独立于静态上下边线,连续 breaking 切换不重启;卡片 hover/扫光统一复用等级色相。
|
||||
- 新闻卡片内容高度、首卡间距和 cruise 选中态统一;船只 marker 复用地表锚点遮挡,保持前后显隐与 BGP/算力对象一致。
|
||||
- 同步 Earth 前端状态机与渲染层中英文开发文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.17] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复 ticker 连续 breaking 新闻切换时扫光动画重启导致的闪烁、回跳和断帧。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- breaking 等级 class 只负责颜色,稳定的 active 状态负责动画生命周期。
|
||||
- ticker 与卡片扫光方向统一为左到右,连续 breaking 切换不重置时间轴。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.16] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复 ticker breaking class 每 120ms 重置导致的边框闪烁、扫光停滞和方向反转。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- watcher 仅在当前 breaking 等级变化时更新 class,动画可以完整播放。
|
||||
- ticker 与卡片扫光方向统一为左到右,ticker 扫光覆盖完整 ticker 区域。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.15] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 将 ticker 扫光限制在上下边线以内,消除 breaking 边框被扫光伪元素带动闪烁。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker 扫光与卡片扫光保持独立横向移动,边线保持稳定,只响应 hover/等级颜色。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.14] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复卡片扫光从中途出现的问题,让反射从卡片左边界完整扫到右边界。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 卡片反射改用内部 `background-position` 扫描,不再使用卡片外部 transform 起始。
|
||||
- ticker 上下边线保持静态,横向扫光独立运行。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.13] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修正 breaking hover 强度基准,hover 会在自身 breaking 高亮上继续增强。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker hover 从 breaking 原始色相和强度继续增强,不再回落到普通新闻强度。
|
||||
- breaking 卡片边框 hover 同样相对自身等级增强,保持红/橙/黄颜色连续。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.12] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复 breaking ticker 和黄卡 hover 被白色混合漂淡的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- hover 使用共享强度参数与等级色相,玻璃底复用深色 HUD surface。
|
||||
- 红色 ticker 保持红色边线,黄色卡片保持黄色 tint,不再被白色边框稀释。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.11] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修正 breaking 卡片整卡扫光层级,并将 ticker hover 亮度对齐共享 HUD 面板。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 卡片扫光覆层位于内容之上,整张卡片 hover/focus 时才扫过。
|
||||
- ticker 上下边线 hover 使用共享 HUD 边框 hover 强度,breaking 只替换色相。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.10] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 将 breaking 卡片扫光限制为当前 hover/focus 卡片,避免所有卡片同时动态闪烁。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 卡片反射层默认关闭,只在当前交互卡片上启动对应等级颜色扫光。
|
||||
- ticker 的全宽 breaking 扫光保持独立,不影响卡片浏览。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.9] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复 breaking 卡片黄色 hover 变淡,并补齐 ticker 上下边线的 hover 高亮反馈。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 卡片 hover 使用不透明等级色相和统一混合强度,红/橙/黄表现一致。
|
||||
- ticker hover 只强化上下两条边,普通新闻使用蓝色,breaking 使用对应等级颜色。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.8] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复展开新闻面板时后台 ticker / realtime 更新导致卡片重建和 hover 抖动的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 面板浏览期间冻结卡片 DOM,后台更新只写入最新 payload。
|
||||
- 关闭面板时一次性应用待处理新闻,保持滚动位置和 hover 稳定。
|
||||
- 展开面板时暂停隐藏 ticker,手动刷新仍可立即更新面板内容。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.7] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 统一 breaking 新闻卡片的 hover 交互,让通用 hover 行为叠加卡片自身等级颜色。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 红、橙、黄 breaking 卡片共用同一套 hover 提亮、边框增强和浮起逻辑。
|
||||
- 等级颜色通过卡片变量注入,避免为每种颜色复制 hover 样式。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.6] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 扩展 breaking ticker 的全域扫光效果,并拉开新闻卡片与上方状态内容的间距。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker 反射从局部光斑改为覆盖整条横向区域的扫光/闪烁带。
|
||||
- 桌面与移动新闻列表首卡预留顶部呼吸空间,保持内容区层次。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.5] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 增加 breaking 新闻的玻璃反射式警灯脉冲,并修复新闻卡片内容被 flex 压缩的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker、卡片边框和卡片反射统一使用同一 breaking 等级颜色。
|
||||
- breaking ticker 使用局部红/橙高光脉冲,卡片使用反射带扫过;面板底色保持蓝色。
|
||||
- 新闻卡片固定非收缩布局并预留最小内容高度,避免只剩媒体名和时间。
|
||||
- 同步中英文 Earth 前端开发文档,记录视觉状态与 reduced-motion 行为。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.4] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修正 breaking 新闻状态显示:ticker 仅在当前滚动到 breaking 新闻时变红,展开面板始终保持蓝色。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 选中、焦点、关闭动画与 breaking 数据状态彻底分离,关闭新闻栏不再留下橙色边线。
|
||||
- breaking 新闻按过期时间重新计算,单条新闻卡片继续保留自身 breaking 红色样式。
|
||||
- 增加 ticker breaking 状态单测和浏览器关闭/重开验证。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.3] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修复新闻栏关闭后残留橙色边线的问题,明确 breaking、展开面板和新闻选择的状态边界。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker 保持中性,breaking 等级只作用于展开新闻面板整体背景和阴影。
|
||||
- 关闭新闻面板时清理选中故事和过期焦点状态,breaking 按新闻过期时间重新计算。
|
||||
- 增加 breaking 状态单测,保持新闻滚动、选择和面板动画行为不变。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.2] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 优化光缆密集区的卫星点拾取,并将点状交互物件优先级泛化到其他密集图层。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 屏幕命中先统一比较卫星、BGP、算力和其他点状对象的距离,海缆线作为兜底候选,保护穿插在密集线缆中的小面积对象。
|
||||
- hover 与 click 共用同一候选排序,避免鼠标提示和最终选中对象不一致。
|
||||
- 补充密集区域拾取回归测试,清理中间态代码并同步渲染层说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.1] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修复选中 BGP 事件静态光晕在地表边缘被切断或陷入地表的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 抽取通用整体锚点遮挡能力,BGP 事件保持原有图标样式并复用圆角图标的前后显隐规则。
|
||||
- 选中事件光晕与事件图标共享地表锚点、前景层级和前后显隐路径,清理中间态代码与资源。
|
||||
- 补充中英文渲染层说明、回归测试和浏览器验证记录。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.0] — 2026-09-30
|
||||
|
||||
Released: 2026-09-30
|
||||
|
||||
### Highlights
|
||||
- 新增详情卡默认配图与后台配置,缺图时保持完整数据卡片,并清理已替换的旧资源。
|
||||
- 算力中心补齐超算、GPU 集群、数据中心、云计算中心和边缘计算节点类型及配图。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一地表物件身份、拾取优先级、连线目标、聚合展示和选中置顶;跨图层重叠时焦点对象保持在前景。
|
||||
- 详情卡保留全量显示,统一高度、字段顺序、拖拽落位和新闻展开交互;图片解析改用按对象索引的快速路径。
|
||||
- 合并卫星、轨迹、海缆和聚合渲染的重复计算,保留完整数据与现有交互;同步中英文 Earth、后台与运维文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.4] — 2026-09-29
|
||||
|
||||
Released: 2026-09-29
|
||||
|
||||
### Highlights
|
||||
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
|
||||
|
||||
### Improved / Fixed
|
||||
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
|
||||
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
|
||||
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
|
||||
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
|
||||
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.3] — 2026-09-27
|
||||
|
||||
Released: 2026-09-27
|
||||
|
||||
### Highlights
|
||||
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
|
||||
|
||||
### Improved / Fixed
|
||||
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
|
||||
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
|
||||
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
|
||||
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
|
||||
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.2] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
|
||||
|
||||
### Fixed
|
||||
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
|
||||
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
|
||||
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.1] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
|
||||
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
|
||||
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
|
||||
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
|
||||
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.0] — 2026-09-23
|
||||
|
||||
Released: 2026-09-23
|
||||
|
||||
@@ -79,13 +79,23 @@ git diff --unified=0 HEAD -- <path>
|
||||
|
||||
## Harness Commands
|
||||
|
||||
Windows desktop launcher tests run separately on Windows with
|
||||
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
|
||||
They use temporary desktop folders and mocked restart commands; no UAC prompt
|
||||
or real service restart is triggered. The WSL bridge regression suite,
|
||||
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
|
||||
disposable command substitutes. For a real entry-point check without restarting,
|
||||
run the repository `planet.cmd --install-only`, then the generated desktop
|
||||
launcher with the same flag. Installation and mocked checks do not prove a live
|
||||
elevated restart; report that separately when it has not been exercised.
|
||||
|
||||
| Tier | Command | What It Does |
|
||||
| --- | --- | --- |
|
||||
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
|
||||
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
|
||||
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, and documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
|
||||
@@ -151,11 +161,6 @@ PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
|
||||
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
|
||||
controls should reuse the shared component, shared token, or established CSS
|
||||
owner for that area instead of introducing a page-local lookalike.
|
||||
- `scripts/harness/frontend-rules-check.sh` warns when semantic
|
||||
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
|
||||
approved React and Earth CSS owner files. A warning means the reviewer should
|
||||
either move the style into the shared owner or document why this is a genuinely
|
||||
new visual family.
|
||||
|
||||
### Earth I18n Harness Rules
|
||||
|
||||
@@ -209,6 +214,12 @@ required tool is not visible there, `scripts/harness/lib.sh` asks the user's
|
||||
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
|
||||
path. This avoids hardcoding a dotfile while still covering agent environments
|
||||
that do not inherit the user's normal shell setup.
|
||||
Validation adds the resolved Bun directory to its process `PATH` so package
|
||||
scripts can invoke Bun as well. `test_tool_resolution.py` exercises this through
|
||||
the validation entry point with an isolated login-shell tool path.
|
||||
The validation build and preview share a temporary output directory that is
|
||||
removed after the run. This gives the smoke a consistent build while local
|
||||
development tools may rebuild `frontend/dist`.
|
||||
|
||||
Required for full local stack operation:
|
||||
|
||||
@@ -257,8 +268,7 @@ modify the host Docker environment.
|
||||
|
||||
No automatic hooks are installed in this phase. Manual reminders:
|
||||
|
||||
- Run `scripts/harness/quick-check.sh` before handing off small changes.
|
||||
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
|
||||
- Choose the verification scope using the [validation guidance in the agent guide](../AGENTS.md#validation).
|
||||
- Run `scripts/harness/security-check.sh` after touching config, auth,
|
||||
credentials, docs examples, or generated fixtures.
|
||||
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
|
||||
@@ -280,7 +290,7 @@ No automatic hooks are installed in this phase. Manual reminders:
|
||||
2. Check `CODEMAP.md` for entry points and ownership boundaries.
|
||||
3. Inspect existing tests and docs before editing.
|
||||
4. Make the smallest behavior-preserving or feature-scoped change.
|
||||
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
|
||||
5. Run the checks for the affected behavior and any gates required for that area.
|
||||
6. Update relevant docs when behavior, workflow, or operations change.
|
||||
7. For rendered frontend changes, verify the affected route with Playwright or
|
||||
the full harness smoke, because `bun run build` alone does not prove page
|
||||
@@ -291,8 +301,6 @@ No automatic hooks are installed in this phase. Manual reminders:
|
||||
1. Reproduce with a focused test or command.
|
||||
2. Patch the owning module, not a caller-side workaround.
|
||||
3. Run the focused regression test.
|
||||
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
|
||||
locally.
|
||||
|
||||
### Documentation Change
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
|
||||
|
||||
## Scope Rules
|
||||
|
||||
- Focus documentation changes on the primary document that owns the affected behavior. Do not trigger speculative updates across secondary or architectural docs unless the underlying contract or operator-facing workflow actually changed.
|
||||
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
|
||||
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
|
||||
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
|
||||
@@ -23,6 +24,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
|
||||
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
|
||||
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
|
||||
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
|
||||
- Internal bug fixes or optimizations that preserve existing APIs and workflows do not require updating user manuals or creating new documentation.
|
||||
|
||||
## Public Docs Rules
|
||||
|
||||
@@ -42,86 +44,17 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
|
||||
|
||||
## Recommended Checks
|
||||
|
||||
Run the checks that match the affected docs.
|
||||
|
||||
### Duplicate Bilingual Docs
|
||||
Always run the centralized harness documentation gate:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
same = []
|
||||
for en in sorted(Path("docs/technical/en").glob("*.md")):
|
||||
zh = Path("docs/technical/zh") / en.name
|
||||
if zh.exists() and en.read_text() == zh.read_text():
|
||||
same.append(en.name)
|
||||
if same:
|
||||
raise SystemExit("identical en/zh docs: " + ", ".join(same))
|
||||
print("no identical en/zh docs")
|
||||
PY
|
||||
scripts/harness/docs-consistency-check.sh
|
||||
```
|
||||
|
||||
### Language-Less Technical Links
|
||||
|
||||
```bash
|
||||
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
|
||||
```
|
||||
|
||||
This should return no matches.
|
||||
|
||||
### Public Docs Registry
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
|
||||
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
|
||||
known.add("README.md")
|
||||
|
||||
missing = []
|
||||
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
|
||||
if not readme.exists():
|
||||
continue
|
||||
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
|
||||
path = Path(href)
|
||||
if "docs/technical/" not in href:
|
||||
continue
|
||||
filename = path.name
|
||||
if filename not in known:
|
||||
missing.append(f"{readme}: {filename}")
|
||||
|
||||
if missing:
|
||||
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
|
||||
print("docs README links are whitelisted")
|
||||
PY
|
||||
```
|
||||
|
||||
### Public Bilingual Pairs
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
|
||||
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
|
||||
missing = []
|
||||
for filename in filenames:
|
||||
for lang in ("zh", "en"):
|
||||
path = Path("docs/technical") / lang / filename
|
||||
if not path.exists():
|
||||
missing.append(str(path))
|
||||
if missing:
|
||||
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
|
||||
print("public docs have zh/en file pairs")
|
||||
PY
|
||||
```
|
||||
|
||||
### Raw Filename Link Titles
|
||||
|
||||
```bash
|
||||
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
|
||||
```
|
||||
|
||||
This should return no matches for polished public docs.
|
||||
This single command automatically covers:
|
||||
1. Public docs whitelist and `DOCS_METADATA` synchronization (`frontend/src/pages/Docs/docs-content.ts` vs backend Gatekeeper)
|
||||
2. Technical doc bilingual file pairs and identical duplicate-content detection
|
||||
3. Public doc link validity and human-readable link titles (no raw `.md` filenames)
|
||||
4. Language-scoped internal links (`docs/technical/zh/` links pointing to `docs/technical/zh/`)
|
||||
5. Supported credential collector contracts and UI guide registration
|
||||
6. Manual console route coverage against actual admin manifests
|
||||
7. Documented `?section=` deep-link validity and UI route drift in active docs
|
||||
|
||||
@@ -28,8 +28,7 @@ compatibility note, not a replacement for existing rules or architecture docs.
|
||||
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
|
||||
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
|
||||
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
|
||||
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
|
||||
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
|
||||
| `.codex/skills/*/SKILL.md` | Present | Local workflows for cleanup, docs, goal-driven work, and release. |
|
||||
|
||||
## Existing CI Gates
|
||||
|
||||
@@ -57,8 +56,7 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
|
||||
## Release And Deploy Process
|
||||
|
||||
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
|
||||
`.claude/commands/release.md`.
|
||||
- Release workflow is documented in `.codex/skills/release/SKILL.md`.
|
||||
- Version-bearing files include `VERSION`, `frontend/package.json`,
|
||||
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
|
||||
`docs/version-history.md`.
|
||||
@@ -102,7 +100,6 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
|
||||
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
|
||||
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
|
||||
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
|
||||
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
|
||||
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
|
||||
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
|
||||
@@ -137,7 +134,7 @@ index is the routing layer; this table is the coverage/evidence layer.
|
||||
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
|
||||
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
|
||||
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
|
||||
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
|
||||
| `uiux` | Related visual surfaces should use one style system per product area. | Manual visual review. | Compare changed controls with the shared component on the rendered page. |
|
||||
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
|
||||
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
|
||||
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
|
||||
@@ -154,8 +151,8 @@ index is the routing layer; this table is the coverage/evidence layer.
|
||||
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
|
||||
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
|
||||
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
|
||||
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
|
||||
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
|
||||
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, viewport-font, zero-letter-spacing, and UI rules static check. |
|
||||
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, and admin-config-derived section deep-link consistency. |
|
||||
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
|
||||
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
|
||||
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |
|
||||
|
||||
@@ -30,7 +30,6 @@
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
|
||||
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
|
||||
@@ -1,715 +0,0 @@
|
||||
# Earth 天球背景与日月位置实施方案
|
||||
|
||||
## 目标
|
||||
|
||||
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
|
||||
|
||||
1. 用真实天球背景替换当前随机星点
|
||||
2. 在当前时间下显示太阳与月亮的相对位置
|
||||
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
|
||||
|
||||
本方案优先追求:
|
||||
|
||||
- 与当前 Three.js Earth 架构兼容
|
||||
- 风险可控
|
||||
- 先落地一版真实感明显提升的 V1
|
||||
- 为后续更严格的天文参考系升级预留余地
|
||||
|
||||
## 当前现状
|
||||
|
||||
当前 Earth 的基础条件已经具备:
|
||||
|
||||
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
|
||||
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
|
||||
|
||||
这意味着:
|
||||
|
||||
- 不需要重写 Earth
|
||||
- 可以在现有 scene/world 层新增一个 celestial layer
|
||||
- 第一阶段不必拆 Earth / satellite / cable 的参考系
|
||||
|
||||
## 总体策略
|
||||
|
||||
采用“两层现实”设计:
|
||||
|
||||
### 1. 世界层(world-space celestial layer)
|
||||
|
||||
用于放置:
|
||||
|
||||
- 天球背景
|
||||
- 太阳
|
||||
- 月亮
|
||||
- 太阳光方向
|
||||
|
||||
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
|
||||
|
||||
### 2. 地球层(earth-fixed layer)
|
||||
|
||||
继续保持当前结构:
|
||||
|
||||
- 海缆
|
||||
- 登陆点
|
||||
- 卫星点与轨迹
|
||||
- BGP 覆盖
|
||||
- 地球纹理、云层、地形
|
||||
|
||||
这些对象继续挂在 `earthObj` 下,不打断现有交互。
|
||||
|
||||
## 为什么先这样做
|
||||
|
||||
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
|
||||
如果现在直接做严格惯性参考系改造,会同时影响:
|
||||
|
||||
- `earthObj.rotation`
|
||||
- 卫星轨迹与锁定逻辑
|
||||
- 海缆与登陆点附着关系
|
||||
- resetView / autoRotate / hover / click 等交互链路
|
||||
|
||||
所以第一阶段只做:
|
||||
|
||||
- 真正的天空
|
||||
- 真正的日月方向
|
||||
- 不碰现有 Earth 附着对象的语义
|
||||
|
||||
## 推荐技术选型
|
||||
|
||||
### 天文计算库
|
||||
|
||||
推荐:
|
||||
|
||||
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
|
||||
原因:
|
||||
|
||||
- 有 JavaScript 版本
|
||||
- 支持 Sun / Moon 的矢量与坐标变换
|
||||
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
|
||||
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
|
||||
|
||||
不作为主选的库:
|
||||
|
||||
- [SunCalc](https://github.com/mourner/suncalc)
|
||||
|
||||
原因:
|
||||
|
||||
- 更偏本地观察者视角的太阳/月亮高度角
|
||||
- 用于“地面日出日落”很好
|
||||
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
|
||||
|
||||
### Three.js 表现层
|
||||
|
||||
推荐组合:
|
||||
|
||||
- 天球:内翻球壳 + 星图纹理
|
||||
- 太阳:`THREE.Sprite`
|
||||
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
|
||||
- 太阳光:`THREE.DirectionalLight`
|
||||
|
||||
参考:
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
|
||||
|
||||
### 1. 背景资源:全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源:
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求:
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影(equirectangular)
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮,避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据:星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层:
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先:Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先:Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略:
|
||||
|
||||
- V1:背景球壳只用全天星图,不立即生成全量恒星点
|
||||
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
|
||||
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
|
||||
|
||||
这样做的原因:
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是:
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分:
|
||||
|
||||
- 背景层:官方深空图 / 全天星图
|
||||
- 计算层:太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用:
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因:
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图,那么:
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现:
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤:
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式:
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的:
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在:
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合:
|
||||
|
||||
- 背景层:全天星图球壳
|
||||
- 点位层:亮星点
|
||||
- 动态层:太阳 / 月亮
|
||||
|
||||
这样有三个好处:
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源:NASA Tycho all-sky map
|
||||
- 用途:背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途:Phase 4 月相表现
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途:太阳 sprite halo
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A:先做可用版本(推荐)
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点:
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B:在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加:
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点:
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层(后续)
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载(后续)
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层,建议单独做离线脚本:
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责:
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果,不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1:真实天球背景
|
||||
|
||||
### 目标
|
||||
|
||||
用真实全天星图替换当前随机星点背景。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 新增一张全天星图纹理
|
||||
|
||||
建议路径:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
纹理要求:
|
||||
|
||||
- 等距矩形投影
|
||||
- 赤经/赤纬坐标展开
|
||||
- 无地平线、无地景遮挡
|
||||
- 尽量深色、弱干扰,适合大屏 HUD 叠加
|
||||
|
||||
2. 新增天球球壳
|
||||
|
||||
新增模块:
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
建议接口:
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date, camera, earth)
|
||||
export function disposeCelestialLayer()
|
||||
```
|
||||
|
||||
3. 实现一个大半径内翻球体
|
||||
|
||||
建议参数:
|
||||
|
||||
- 半径:`600 ~ 900`
|
||||
- 材质:`MeshBasicMaterial`
|
||||
- `side: THREE.BackSide`
|
||||
- 不受场景光照影响
|
||||
- 始终围绕场景中心
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 初始加载后背景不再是随机星点
|
||||
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
|
||||
- 不明显干扰海缆/卫星/BGP 的前景识别
|
||||
|
||||
## Phase 2:太阳与月亮真实位置
|
||||
|
||||
### 目标
|
||||
|
||||
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 在 `celestial.js` 内封装天体位置计算
|
||||
|
||||
建议函数:
|
||||
|
||||
```js
|
||||
function getSunDirection(date)
|
||||
function getMoonDirection(date)
|
||||
```
|
||||
|
||||
输出统一为 world-space `THREE.Vector3`
|
||||
|
||||
2. 太阳显示
|
||||
|
||||
- 一个暖色发光 sprite
|
||||
- 比月亮更大、更亮
|
||||
- 可选添加柔和 halo
|
||||
|
||||
3. 月亮显示
|
||||
|
||||
- 一个较小 sprite 或 sphere
|
||||
- 灰白偏冷色
|
||||
- 后续 Phase 3 再做月相
|
||||
|
||||
4. 更新频率
|
||||
|
||||
不要每帧重新做完整天文计算,建议:
|
||||
|
||||
- 每 30 秒或 60 秒重算一次真实位置
|
||||
- 渲染帧内做平滑过渡
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面可见太阳与月亮两个对象
|
||||
- 时间变化时位置会更新
|
||||
- 日月不会跟随地球局部旋转而错误附着
|
||||
|
||||
## Phase 3:太阳驱动地球受光
|
||||
|
||||
### 目标
|
||||
|
||||
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 替换或接管当前主定向光
|
||||
|
||||
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
|
||||
|
||||
建议改为:
|
||||
|
||||
- 保留环境补光
|
||||
- 主太阳光方向由 `sunDirection` 决定
|
||||
|
||||
2. 太阳光参数建议
|
||||
|
||||
- `DirectionalLight` 颜色偏暖白
|
||||
- 强度略高于当前主光
|
||||
- 保留一个弱背光作为氛围补偿,避免背面过死黑
|
||||
|
||||
3. 先不做物理级大气散射
|
||||
|
||||
第一版只要求:
|
||||
|
||||
- 亮面与暗面方向真实
|
||||
- 云层和大气仍保持当前风格
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 地球明暗面会随太阳方向改变
|
||||
- 太阳 sprite 和地球亮面方向一致
|
||||
- 不破坏现有海缆、卫星、BGP 的可见性
|
||||
|
||||
## Phase 4:月相与天文细节增强
|
||||
|
||||
### 目标
|
||||
|
||||
在日月真实位置基础上增加更强的“天文可信度”。
|
||||
|
||||
### 可选项
|
||||
|
||||
1. 月相
|
||||
|
||||
- 根据日月夹角计算 illuminated fraction
|
||||
- 用月相纹理或 shader 表达盈亏
|
||||
|
||||
2. 赤道/黄道辅助线
|
||||
|
||||
- 可作为开发调试层,不默认显示
|
||||
|
||||
3. 太阳 terminator 增强
|
||||
|
||||
- 给地球夜面加入更自然的 night tint
|
||||
- 未来可叠加城市夜光纹理
|
||||
|
||||
4. 天文时间入口
|
||||
|
||||
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 月亮不再只是一个静态圆点
|
||||
- 后续扩展行星或观测模式时无需推倒重来
|
||||
|
||||
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
|
||||
|
||||
### 目标
|
||||
|
||||
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
|
||||
|
||||
### 需要处理的问题
|
||||
|
||||
- 地球自转角与 UTC 的一致性
|
||||
- 赤道坐标系、地固坐标系、相机交互层分离
|
||||
- 卫星轨道显示与 Earth 旋转同步关系
|
||||
- resetView 和 autoRotate 的语义重定
|
||||
|
||||
### 风险
|
||||
|
||||
这一步会影响:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
因此不建议与 V1 同时推进。
|
||||
|
||||
## 代码改造清单
|
||||
|
||||
## 1. 新增文件
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理天球背景、太阳、月亮
|
||||
- 对外暴露 init/update/dispose
|
||||
|
||||
## 2. 修改 `constants.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
|
||||
新增:
|
||||
|
||||
```js
|
||||
export const CELESTIAL_CONFIG = {
|
||||
sphereRadius: 800,
|
||||
updateIntervalMs: 60000,
|
||||
sunSpriteScale: 28,
|
||||
moonSpriteScale: 16,
|
||||
sunLightIntensity: 1.25,
|
||||
ambientIntensity: 0.28,
|
||||
backLightIntensity: 0.18,
|
||||
};
|
||||
```
|
||||
|
||||
## 3. 修改 `main.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
1. `init()` 中:
|
||||
- 初始化 celestial layer
|
||||
2. `addLights()` 中:
|
||||
- 把固定太阳光改成可更新的 celestial sun light
|
||||
3. `animate()` 中:
|
||||
- 每帧调 `updateCelestialLayer()`
|
||||
4. `destroy()` 中:
|
||||
- 清理 celestial 资源
|
||||
|
||||
## 4. 修改 `earth.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
- `createStars()` 逐步退役
|
||||
- 第一阶段可先保留作为 fallback
|
||||
- 当真实星图加载成功后,不再显示随机星点
|
||||
|
||||
## 5. 新增资源
|
||||
|
||||
目录建议:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/`
|
||||
|
||||
建议至少包含:
|
||||
|
||||
- `starmap_equatorial_4k.jpg`
|
||||
- `sun_glow.png`
|
||||
- `moon_albedo_2k.jpg`
|
||||
|
||||
## 数据流设计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
|
||||
B --> C["创建天球球壳"]
|
||||
B --> D["创建太阳 sprite + 主定向光"]
|
||||
B --> E["创建月亮 sprite"]
|
||||
|
||||
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
|
||||
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
|
||||
H --> I["更新 sun sprite / moon sprite 位置"]
|
||||
H --> J["更新太阳 DirectionalLight 方向"]
|
||||
J --> K["地球昼夜方向变化"]
|
||||
```
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
### 1. 星图投影方向容易反
|
||||
|
||||
这会表现为:
|
||||
|
||||
- 星图左右镜像
|
||||
- 赤经方向颠倒
|
||||
- 日月位置和背景对不上
|
||||
|
||||
建议:
|
||||
|
||||
- 先做一个开发调试模式
|
||||
- 显示赤经/赤纬参考点,快速校正纹理朝向
|
||||
|
||||
### 2. 不要让天球跟随 Earth 旋转
|
||||
|
||||
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
|
||||
|
||||
### 3. 不要每帧做重型天文计算
|
||||
|
||||
真实位置更新应节流,否则会浪费 CPU。
|
||||
|
||||
### 4. 月亮先求“方向正确”,再求“月相精致”
|
||||
|
||||
月相属于第二步优化,不应阻塞 V1 上线。
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 新建 `celestial.js`
|
||||
2. 用星图球壳替换随机星点
|
||||
3. 接入 Astronomy Engine
|
||||
4. 加太阳/月亮 sprite
|
||||
5. 用太阳方向驱动主光
|
||||
6. 再决定要不要做月相和更严格参考系
|
||||
|
||||
## 最终建议
|
||||
|
||||
对于当前 Planet Earth,最稳妥的方案是:
|
||||
|
||||
- 先做真实天球背景
|
||||
- 再做真实太阳/月亮方向
|
||||
- 再让太阳驱动地球受光
|
||||
- 暂时不做 Earth 参考系重构
|
||||
|
||||
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。
|
||||
@@ -1,106 +0,0 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有:
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升:
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象:
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆:
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足,需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保:
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高,采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
|
||||
## Satellite Footprint Follow-Up Items
|
||||
|
||||
从技术文档迁出的 footprint 后续项,作为卫星覆盖能力的计划 backlog:
|
||||
|
||||
1. 为 `iridium-next` 新建独立 footprint adapter。
|
||||
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint。
|
||||
3. 如果未来拿到 GEO beam contour / operator metadata,再为 GEO 开 operator-specific footprint。
|
||||
@@ -209,6 +209,11 @@ def parse_response(self, html):
|
||||
|
||||
## VII. Scheduler
|
||||
|
||||
The `earth_news_rss_refresh` system job checks regional feeds on startup and
|
||||
every 10 minutes. Homepage GET requests read saved news. Datasource schedule
|
||||
synchronization preserves this system job. Freshness, locking, and cooldown
|
||||
details are in the [news source guide](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-sources.md).
|
||||
|
||||
```python
|
||||
# Register all collectors into scheduled tasks at startup
|
||||
def start_scheduler():
|
||||
@@ -467,6 +472,20 @@ Epoch uses its official public CSV, retaining theoretical FP16 performance, orig
|
||||
|
||||
Neither collector returns sample data on failure. Empty fetches or transformed results, schema errors, and losses exceeding 40% of the previous snapshot fail while retaining prior data. Incomplete writes are rolled back. TOP500 location migration copies only a unique exact name and canonical-country match to the new system ID, retaining the historical key; rank is never a migration key.
|
||||
|
||||
`compute_history.py` stores controlled imports from the reviewed research plan, including annual observations, sources, and a version. Reference periods and publication dates are distinct; unknown publication days remain unspecified. `/api/v1/data-products/compute/series` offers global share, fixed US + China bilateral share, and available FP32 absolute observations. Innovation gaps are independent and never expanded into inferred national scores.
|
||||
`compute_history_defaults.json` ships the original report data. `compute_configuration.py` reads a complete override from the `compute_comparison` category in `system_settings`; absent records use bundled defaults without migrations or manual initialization. `compute_history.py` performs same-year aggregation. Reference periods and publication dates remain distinct, and unknown publication days remain unspecified. `/api/v1/data-products/compute/series` reads configuration on each request with `Cache-Control: no-store`, offering global share, fixed US + China bilateral share and FP32 absolute observations. Innovation gaps remain independent. An empty override stays empty instead of silently restoring observations.
|
||||
|
||||
Under the same prefix, `GET/PUT /configuration` and `POST /configuration/reset` require admin/super_admin. Writes accept `{revision, config}` and reset accepts `revision`. Pydantic validates canonical countries, finite values, share totals, duplicate metric/year pairs and report references; each metric/year has one report basis. PostgreSQL category uniqueness and row locking serialize first writes and updates; stale revisions return 409. Configuration and before/after audit records commit in one transaction. Reset removes the override so subsequent reads use the current bundled defaults. `version` is a configuration revision, while `reviewed_at` is the supplied review date. Admin-curated data is not automatically labeled official. Source URLs are references, not scrape instructions.
|
||||
|
||||
`/api/v1/data-products/compute/facilities` reads successful snapshots with `compute_contract=v1`, applying country filters and per-source counts in SQL. Epoch excludes planned, decommissioned, superseded, and suspected duplicate records; TOP500 means list observations. Cross-source deduplication remains pending. Historical cutoffs use collection time, never pretending a recently fetched old source was available earlier. Geographic results are bounded and report truncation; total record counts are independent of this limit. Official site helpers are keyed by exact `source` + `source_id`; their access-map centers are site-level references only and retain their official URL in `source_note`.
|
||||
|
||||
|
||||
### CelesTrak download windows and caching
|
||||
|
||||
Both active and fallback-group downloads skip HEAD preflight and send GET directly: a preflight may consume the upstream download allowance before the body request. Valid complete cache entries are reused for two hours, then refreshed with conditional GET when validators exist, or ordinary GET otherwise. A 304 reuses validated cache; only the explicit “GP data has not updated” 403 permits a valid-cache fallback. Other 403 errors remain failures. Fallback groups also respect cache age rather than reusing old files indefinitely.
|
||||
|
||||
A nonempty GP record list is validated in `.part` before atomically replacing the cache; failed downloads preserve the previous valid payload. Length comes from the GET response, and compressed wire length is not compared with decoded bytes. Other collectors retain HEAD metadata and Range resume support. Restoring a raw cache does not populate the database: trigger a new collection task afterward.
|
||||
|
||||
|
||||
### Earth news encoding
|
||||
|
||||
Before parsing RSS/Atom, `earth_news.py` selects encoding from the BOM, HTTP charset, XML encoding declaration, then UTF-8, and decodes strictly. GBK/GB2312/GB18030 feeds no longer silently replace Chinese bytes with corrupted text. Unknown encodings or invalid bytes produce `format_error` instead of damaged titles. `test_earth_news_encoding.py` covers titles, summaries, display serialization and failure status.
|
||||
|
||||
@@ -31,8 +31,16 @@ flowchart LR
|
||||
5. The listener merges short-window same-layer events, invalidates cache, and broadcasts `earth_updates`.
|
||||
6. The outbox row is marked consumed only after successful broadcast; failed rows stay retryable.
|
||||
|
||||
Cache rebuilds are coalesced by key. Concurrent requests in one process share the build result or error, while cross-process cold misses wait for the Redis lock holder. Long builds renew the lease, and publishing or releasing checks ownership. If the owning request is cancelled, a waiter can take over with its own database session. Available stale data is returned immediately. Without stale data, a waiter returns HTTP 503 after the default 30 seconds. Redis coordination failures return the current build result, with concurrent requests in that process still coalesced. A failed build returns stale data when available and otherwise propagates the error.
|
||||
|
||||
DB changes no longer create default `earth_refresh` jobs, so they are not blocked by long same-source collection or clearing jobs. `earth_refresh` remains for manual cache clearing and non-DB refresh hints.
|
||||
|
||||
### Outbox Writes and Retention
|
||||
|
||||
The trigger obtains `event_id` from the sequence, then writes the payload containing that ID to `earth_data_change_events` once and sends `NOTIFY`. It does not update the same event after insertion. An `earth_news_items` update that only changes `last_seen_at` or `updated_at` emits no Earth change event; visible content changes still emit one. Those timestamps continue to record news-source health and freshness.
|
||||
|
||||
Consumed events are retained for one day by default, configurable through `PLANET_EARTH_OUTBOX_RETENTION_DAYS`. Every 60 seconds, the listener removes at most 1,000 expired consumed rows, and only when the separate retention index is valid; pending rows never match the deletion condition. A PostgreSQL advisory lock prevents concurrent cleanup across instances. The [standalone Outbox maintenance script](/home/ray/dev/linkong/planet/scripts/maintain_earth_outbox.py) creates the index with `CREATE INDEX CONCURRENTLY`, outside the application startup transaction. The script is read-only by default and also supports batched deletion and a one-time compact mode. See the [operations runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) for execution steps.
|
||||
|
||||
## Data Job Queue
|
||||
|
||||
`collection_tasks` is the unified job ledger. Workers claim `queued` jobs with PostgreSQL `FOR UPDATE SKIP LOCKED`; write jobs for the same `source` run serially, while different sources may run in parallel.
|
||||
|
||||
@@ -321,7 +321,7 @@ The new vessel list entry point is no longer the legacy `/api/v1/visualization/g
|
||||
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
|
||||
```
|
||||
|
||||
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](backend-collectors.md).
|
||||
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md).
|
||||
|
||||
## Custom REST / WebSocket Mapping Runtime
|
||||
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
# BGP Context
|
||||
|
||||
## Prefix geography lookup
|
||||
|
||||
Lookups preserve the OpenGeoFeed, IPtoASN, then NRO source order, address-family
|
||||
checks, and full start/end range containment. Within each source, the original
|
||||
prefix mask length and record ID determine ordering. `planet_bgp_try_inet` and
|
||||
`planet_bgp_try_masklen` return NULL for invalid historical values. Database
|
||||
initialization installs these lightweight functions. A separate maintenance
|
||||
script installs the combined GiST range index, as described in the
|
||||
[operations runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md).
|
||||
|
||||
## Current Goal
|
||||
|
||||
The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:
|
||||
|
||||
@@ -1,13 +1,19 @@
|
||||
# Earth Frontend Context
|
||||
|
||||
## HD surface and atmosphere materials
|
||||
|
||||
The visible HD surface uses a single-draw ShaderMaterial in `earth-surface-material.js`, retaining the R+0.48 mesh, depth policy and surface pick target. The existing controller still switches all four 8K maps, updates the sampler and disposes the previous texture only after success. Normals, view direction and astronomical sunlight share world coordinates. Land is matte; water has grazing-angle cool reflection without a white specular lobe. The existing geographic land/ocean texture owns the optical mask independently of border visibility, with a conservative chroma fallback until loaded and reflection suppression over ice. Original JPEGs are neither repainted nor downsampled.
|
||||
|
||||
Color textures are tagged sRGB with mipmaps. The r128 surface shader decodes sRGB; r160 uses hardware texture decoding. Both combine light in linear space and encode once. A local filmic curve affects only the surface instead of enabling global ACES exposure on the sky and data layers. The terminator spans normal dot products [-0.075, 0.16]; night shading has its own geographic texture ghosting and cool ambient branch. A 0.55 power of the locally tone-mapped texture retains shadow variation, followed by separate muted land and deep-indigo ocean tints and low ambient floors. Day/night mixing remains linear with one final sRGB output. Day exposure and reflections are unchanged; no synthetic city lights are added. One R×1.025 BackSide/Additive shell replaces the three rim shells, with depthWrite=false and depthTest=true; inner scattering is part of the surface pass. Independently switchable clouds use matte day/night shading at renderOrder 1.1. The sky uses the existing `starmap_deep_8k.jpg` without a tint overlay.
|
||||
|
||||
## Configurable compute observations
|
||||
|
||||
Compute cards read annual observations from `/api/v1/data-products/compute/series`; the admin Compute comparison section is the maintenance entry point. `compute-topic.js` fetches with no-store, listens for same-origin `planet-compute-configuration` notifications, and checks every 60 seconds while visible and enabled, plus when the page becomes visible. Observation refresh does not reload facility layers. `destroy()` releases the timer, channel and visibility listener. Country membership, full regional coverage and missing-value semantics remain backend-owned.
|
||||
|
||||
## Surface texture provenance
|
||||
|
||||
On 2026-09-21, `assets/8k_earth_daymap.jpg` was replaced with an 8192×4096 Web derivative of the paired Natural Earth I raster. Its loading path and rendering code remain unchanged. The source is a 16200×8100 EPSG:4326 image; the previous JPEG is preserved byte-for-byte under `resources/earth/archive/2026-09-21/`. Source/license records, the catalogue versus embedded-version discrepancy, SHA256 hashes, JPEG parameters, and recalculated world-file metadata live in `resources/earth/natural-earth-1/`. Large originals remain in Git-ignored `downloads/earth/`, outside the frontend release bundle.
|
||||
|
||||
The base-map selector defaults to Original basemap, listed first and saves the source chosen in the display menu. Natural Earth I remains available alongside “Original basemap”, a byte-identical copy of the September 21, 2026 archive; NASA 21K is source archive material only, while runtime delivery stays at one 8K texture. A new source replaces the old GPU texture only after successful loading and releases the old one then; failures retain the old texture and stale responses cannot overwrite the current source. This is a source switch, not an additional rendering layer. The physical texture is JPEG sRGB, does not bake political borders, and detailed boundary alignment is not yet validated. The starfield is restored byte-for-byte to commit `83a10a6c`: `celestial.js` matches that commit, `skyTint` and `skyBrightness` are removed, and `brightStarOpacity` is back to `0.88`; star density and color are no longer calibrated. The HD overlay uses a white `MeshBasicMaterial` with opacity `1` and `toneMapped:false`, with `texture-colors.js` centralizing sRGB read/write. The 0.128 path uses `encoding` plus `LinearTosRGB`; the 0.160.1 path uses `colorSpace` plus the normal colorspace fragment. There is no Phong white-light or colored twilight overlay, blue night tint, `cameraBoost`, or shoulder curve; daylight neutral scaling keeps brightness at or below the source maximum, and dark mode only applies neutral scaling. The original texture asset is unchanged. Natural Earth is intrinsically pale; this does not claim missing detail was restored. Earth still supports the Three 0.128 import-map runtime and frontend package Three 0.160.1 runtime, and WebGL1 derivatives remain enabled for `fwidth`.
|
||||
|
||||
|
||||
Day/night shading uses a narrow smooth transition centered on the solar horizon (normal dot product ±0.12). The day side is capped at the source brightness; night-side linear illumination is 0.24, rising smoothly to at most 0.60 in source shadows to retain dark ocean and forest detail. Shading uses a neutral scalar without added exposure, colored veils, rendering passes, or textures.
|
||||
|
||||
Basemaps and TV channels share `hud-select-menu.js` for the HUD popover, list, selection mark, keyboard navigation, and positioning; `tv-source-menu.js` only adapts channel copy. Basemaps disable search, pagination, and counts. The trigger owner identifies popover clicks as internal to the mobile drawer.
|
||||
This document describes the current real structure of the Earth display frontend. The focus is on helping future changes to the HUD, layers, media panel, real terrain, and BGP visualization avoid repeating past structural and state-sync pitfalls.
|
||||
@@ -66,7 +72,7 @@ Responsibilities:
|
||||
- Layer module integration
|
||||
- Earth-level state synchronization
|
||||
|
||||
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
|
||||
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. News-only changes leave the summary cache intact; a batch that also changes another layer still refreshes the summary. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
|
||||
|
||||
### 3. Earth Control Layer
|
||||
|
||||
@@ -247,7 +253,7 @@ All material, layer, satellite, BGP, cable, terrain, celestial, and other style
|
||||
|
||||
## Render hot paths
|
||||
|
||||
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object.
|
||||
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object. In dense cable areas, screen-distance picking compares satellites, BGP, compute and other point objects before using cable lines as the fallback; hover and click share the same ordering so small objects threaded through a line bundle remain selectable.
|
||||
|
||||
Satellite breathing runs in vertex shaders. `satellite-position-worker.js` moves full SGP4 position and initial trail calculations off the main thread; the main thread applies snapshots to interaction coordinates and render buffers. `satellite-propagation.js` supplies the same mathematics to the Worker, predicted orbits, and synchronous fallback. Reload, clear, and altitude changes must terminate prior work so stale snapshots cannot resurrect cleared layers.
|
||||
|
||||
@@ -284,7 +290,7 @@ The surface hover tooltip preference is persisted by `controls.js` as `shared.su
|
||||
|
||||
The real satellite altitude preference is persisted by `controls.js`, while the rendering state lives in `satellites.js`. When enabled, the real radius from SGP4 is compressed logarithmically into the current Earth visual radius range. When disabled, satellite dots, trails, and predicted orbits all return to the legacy same-sphere display. Toggling this setting must refresh satellite positions and clear trail buffers so a trail never mixes both height models. `maxRealAltitudeOffset = 25` is a visual cap tuned for the current camera and `earthRadius = 100`: GEO / MEO remain clearly higher than LEO, but the highest orbits stay within about 25% beyond the globe radius so selection targets, red trails, and the globe do not feel disconnected.
|
||||
|
||||
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for the whole future orbit, projecting the inertial orbit plane onto the current globe pose; that keeps the line closed and keeps the visual orbit inclination aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
|
||||
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for an open full-period prediction curve, projecting the inertial orbit plane onto the current globe pose. No first vertex or closing chord is appended; arc-length highlights travel toward increasing prediction time, and visual inclination stays aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
|
||||
|
||||
Boundary precision is stored separately by `country-boundaries.js` under `planet.earth.boundaries.highPrecisionEnabled`. When high precision is off, Earth keeps using the bundled low-precision `countries-admin0.min.geojson` fallback even if high-precision manifest/PMTiles files exist locally. When high precision is on but the artifact is missing, the Earth toolbar settings call `/api/v1/earth/boundaries/build` and poll progress. After success, `reloadCountryBoundaries()` hot-swaps the boundary layer without refreshing the page. Boundary hover is independent from interactable hover: a country polygon remains highlighted whenever the surface coordinate is inside it, while the tooltip can still prioritize a satellite, vessel, BGP marker, or other interactable.
|
||||
|
||||
@@ -365,10 +371,9 @@ If future cable, satellite, or news cruise is added, do not copy a new set of `m
|
||||
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) owns the Earth zoom state, and every zoom entry point must ultimately call `setZoomLevel()` to write the camera distance. Do not write `camera.position.z` from other modules, or the zoom percentage, drag sensitivity, and Interactable clustering thresholds will diverge again.
|
||||
|
||||
`globe-navigation.js` maps zoom to `sqrt(R² + (D₀² - R²) / zoom²)` with minimum surface clearance and the matching inverse. The 100% camera pose remains unchanged; 500% stays outside the globe. Mouse and single-touch dragging use ray/sphere intersections and quaternion rotation to keep the grabbed surface under the pointer, without an empirical sensitivity multiplier or frame-by-frame catch-up. Auto-rotation is suspended during dragging and pinching. Release inertia follows the measured axis; paused releases and cancellation do not fling. Returning from a pinch to one finger rebases the grab point. Click suppression uses displacement from pointer-down, including slow drags.
|
||||
`globe-navigation.js` retains the safe zoom-distance mapping. Mouse and single-touch drags uniformly use legacy screen deltas, 0.18 smoothing and release inertia, including movement outside the globe silhouette; ray/sphere intersection no longer gates dragging. Normal zoom retains legacy sensitivity. Once the globe exceeds the viewport, a smooth surface-distance/viewport cap limits excessive rotation at extreme zoom. Horizontal motion uses geographic yaw, vertical input changes tilt, and quaternion math prevents multi-revolution reversal. Pinch transitions and pointer capture remain intact.
|
||||
|
||||
|
||||
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps projection-based clustering, and `none` disables it. Vessels explicitly disable clustering and avoidance so incremental updates do not rebuild cluster topology. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
|
||||
Surface markers retain category icons and fixed geographic cluster membership. Numbered solid circles expand through a 320ms tween above 250% zoom; zoom never regroups members, and 500% has no aggregate badges. Vessels retain independent heading arrows without clustering. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md).
|
||||
|
||||
Wheel input has two paths. Traditional mouse wheels keep the 10% step and short animation, using `wheelZoomTarget` as the logical base for continuous wheel input. Trackpads and high-precision wheels use the pixel delta for continuous zoom and call `setZoomLevel()` directly instead of passing through the 10% stepped animation. The trackpad path also filters a short-window, old-direction residual delta after a real direction change so inertia tails do not pull a just-reversed zoom back in the previous direction.
|
||||
|
||||
@@ -411,13 +416,13 @@ For console structure, see:
|
||||
|
||||
Earth's existing Roboto Condensed face is now served locally from `assets/fonts/`, with its source and license in that directory and the shared declaration in `css/fonts.css`. The remote Google Fonts stylesheet no longer blocks the page; font family and weights are preserved.
|
||||
|
||||
`spatial-callouts.js` delegates layout to `spatial-callout-controller.js` and reuses `CalloutConnector` with stable region IDs. Layout is event-driven and retains valid old rectangles while work is pending; the projected source line follows the globe continuously. The layout reference is the direction of the actual source exit point, so a card stays near the object side instead of being pulled from a top-left geographic point to an opposite corner. Neighboring same-sector placements, shorter connectors, and fewer crossings are preferred; the solver may exchange free positions for at most three rounds rather than relayout everything each frame. In fullscreen Earth, nearby foreground positions are allowed so cards of normal size do not cross the whole globe merely to avoid it. In slow mode, sustained direction drift or connector crossings must persist for 1.8 seconds before proposing a joint reflow. Automatic proposals are at least 8 seconds apart and need an improvement of at least 18% plus a minimum absolute gain. Manual/inertial input retains the 72ms/12px response budget. Expansion and manual placement grant 12 seconds of reading time; hover, keyboard focus and dragging continue to lock a card, while residual pointer focus does not lock it forever. Manual placement is a preference: after its hold, automatic repair requires at least 45 degrees of source direction change. Restored positions are not permanent pins. Rear connectors retain the globe-outline cue without changing card identity. Local position/size persistence and reset-to-auto are part of the controller. A `ResizeObserver` handles resizing and collapse; configuration and source windows retain shared drag handles.
|
||||
`spatial-callouts.js` provides geographic projection and shared scopes. `spatial-callout-controller.js` selects expanded compute and news cards independently, prioritizing dragging or manually pinned cards, then the current focus and distance from the view center. `callout-limits.js` stores independent limits under `planet.earth.callout-limits.v1`; controls offer 1–3 of each, defaulting to one. `earth:callout-limits-change` applies changes immediately. Other cards become capsules following geographic anchors; hover temporarily expands them and click changes focus. Expanded cards occupy left/right stages with news above compute, including a destination for every card when a side holds more than two; peripheral overlap is allowed when space is limited. Rear capsules dim and shrink. Manual drops remain pinned until their region moves deep behind the globe. Dimensions and reading state remain separate from business coordinates, with shared drag foreground priority across both types.
|
||||
|
||||
Object presentation is shared through `PresentationController` and the new object-callout anchor/layout path. A click on a BGP, compute, vessel, satellite, or cable object draws the connector before placing the adjacent card. The source anchor begins at the actual click or original region label, then follows a visible highlight geometry sample as rotation approaches occlusion. The stable `layoutReferencePoint` is derived from the actual connector exit direction, so `packDirectionalEdgeCallouts` keeps cards on the corresponding screen edge and leaves the globe's center open; a card may move inward only when that edge lane is unavailable, and it may never cover its own connector exit point. The bounded joint assignment considers up to 24 beam states and 32 candidates, avoids existing HUD rectangles, then allows at most three safe exchanges to reduce directional crossings; the previous solver remains a fallback and the work is not a full relayout every frame. A slow source-direction gate also checks whether the actual card target reverses more than 90 degrees from the source before committing a change. Manual drag and hover locks take priority. The source point itself remains a per-frame visual connection target; fixed objects track live positions while cards remain draggable. Rear-side objects retain a globe-outline direction cue, a subdued connector, and a hollow source point; these are orientation guides rather than geographic points. Country candidates are capped at 96 with a hard limit of 128 and cached, so GeoJSON is not scanned every frame. Compute cards keep their original point independent from dynamic source-point movement, and the shared card endpoint slides along the obstacle perimeter. Cable presentation prefers the true ray-hit local point, then samples highlighted geometry by `cableId`. Mobile keeps the existing mini popup and opens drawer details. A stale asynchronous presentation must never dismiss a newer selection.
|
||||
|
||||
`configVisible` and `cardsEnabled` are separate. X/Escape closes only configuration; disabling the switch releases connectors and restores the previous map query/layer state. Configuration and source windows bind HUD typography directly so reparenting cannot enlarge their text. The main card presents the selected compute metric as a large share percentage or an absolute value with unit, observation year, historical context, and source entry. Missing comparable data is explicit; no empty trend chart is rendered without observations. Facility records and coverage remain complete in the expanded auxiliary area, where sample counts are evidence context rather than a compute value or global coverage rate. Long lists open only when needed. Mobile preserves the same single metric meaning in its accordion. Local position/size persistence and reset-to-auto remain available. Stable DOM, dimensions, and reading state survive updates, and stale responses cannot re-enable cards.
|
||||
`configVisible` and `cardsEnabled` are separate. Closing the configuration window with X or Escape preserves card visibility. The desktop title bar uses the news panel’s card icon and selected style, with `aria-pressed` reflecting `cardsEnabled`. Turning off cards through this button or the mobile switch releases the connectors and restores the previous map query and layer state. Configuration and source windows bind HUD typography directly so reparenting cannot enlarge their text. The main card presents the selected compute metric as a large share percentage or an absolute value with unit, observation year, historical context, and source entry. Missing comparable data is explicit; no empty trend chart is rendered without observations. Facility records and coverage remain complete in the expanded auxiliary area, where sample counts are evidence context rather than a compute value or global coverage rate. Long lists open only when needed. Mobile preserves the same single metric meaning in its accordion. Local position/size persistence remains available. Stable DOM, dimensions, and reading state survive updates, and stale responses cannot re-enable cards.
|
||||
|
||||
Motion presentation uses the shared `main.js` input state. Sustained drift or crossings trigger a joint assignment, rather than each card competing independently for leftover slots. `callout-region-path.js` provides the short connection and terminal stub used both for rendering and intersection scoring. Candidate paths are precomputed and layout states are cloned only after pruning. Existing cards remain visible during planning; queued reflows survive frame throttling.
|
||||
Expanded cards follow committed targets with smooth inertia, pausing during hover or keyboard reading; capsules independently follow geographic projections. Connector anchors still update each frame. `shared.calloutLineStyle` persists thin/bold preferences, synchronized from desktop/mobile controls to `body.dataset.calloutLineStyle` and consumed by shared connectors without changing business anchors.
|
||||
|
||||
The shared `CalloutConnector` calculates polyline length from its points instead of reading `getTotalLength()` after each SVG write, avoiding repeated synchronous layout during motion. Initial draw animation is retained, so existing cruise presentations receive the same optimization.
|
||||
|
||||
@@ -427,16 +432,26 @@ On mobile, compute uses a compact regional overview in the existing drawer. Rows
|
||||
|
||||
### Boundary surface attachment
|
||||
|
||||
Normal borders, coastlines, claim lines, highlights and landing points use `earth-surface.js` to share the HD sphere UV-triangle surface. Borders draw after the texture but before landing points. Highlight lines are always transparent-queue objects with depth testing, preventing through-globe display or overpainting markers. Geometry is computed and cached at construction, not reprojected per frame. See the [render-layer specification](earth-render-layer-order.md).
|
||||
Normal borders, coastlines, claim lines, highlights and landing points use `earth-surface.js` to share the HD sphere UV-triangle surface. Borders draw after the texture but before landing points. Highlight lines are always transparent-queue objects with depth testing, preventing through-globe display or overpainting markers. Geometry is computed and cached at construction, not reprojected per frame. See the [render-layer specification](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
|
||||
|
||||
### News ticker edge fading
|
||||
### News ticker edge fading and breaking state machine
|
||||
|
||||
A separate pseudo-element paints the ticker background and horizontal rules, fading to full transparency outside the text bounds. The shell has no rectangular backdrop filter, box shadow, or text mask. The region label sizes to its content and the scrolling text follows directly without fading. Alert states change only the gradient rule color so they cannot reintroduce rectangular shadow edges.
|
||||
A separate pseudo-element paints the ticker background and horizontal rules, fading to full transparency outside the text bounds. The shell has no rectangular backdrop filter, box shadow, or text mask. The region label sizes to its content and the scrolling text follows directly without fading.
|
||||
|
||||
The previous breaking rule was global shell promotion: if the payload contained any unexpired breaking story, the highest level (`critical > breaking > watch`) was applied to the ticker or expanded panel. That made a closed panel, or a ticker currently showing a normal story, retain an orange/red state. `breaking_expires_at` eventually cleared the state, while selection, opening and closing did not change the underlying breaking data.
|
||||
|
||||
The current state machine keeps data, ticker visibility, panel lifecycle and selection separate:
|
||||
|
||||
- `getNewsBreakingState()` derives the data state from `items` and `cruise_items`, ignores expired stories, and ranks `critical > breaking > watch`. The expiry timestamp only schedules the next recomputation.
|
||||
- A ticker viewport watcher runs every 120 ms and chooses the item with the greatest viewport overlap, breaking ties by distance to the viewport center. Only when that current item has a non-`none` level does the ticker switch its horizontal rules to `has-breaking-*`; a brief no-overlap gap at the duplicated-track seam retains the active state so the sweep timeline and borders do not restart.
|
||||
- The panel lifecycle is `closed → opening → open → closing → closed`. Opening and closing control morphing, visibility and ticker hiding; they do not mutate the breaking data. During `opening/open`, the current card DOM stays stable while incoming payloads are deferred and applied once on close, so realtime updates cannot rebuild a hovered card.
|
||||
- `selectedCruiseStoryId` is independent of breaking severity. Selection, focus, hover and close only change the selected story and reading position; close also clears the selection and filter popover.
|
||||
- The expanded panel always keeps its blue glass surface. Breaking colors remain on the matching story cards, so opening the panel never turns the entire glass panel red; the ticker can still show a red signal when the currently scrolling story is breaking.
|
||||
|
||||
The glass-reflection effect is implemented: when the current ticker story enters a breaking state, its `::after` carries a short, feathered light cloud across the ticker glass area; the matching breaking card gets a full-card reflection sweep through its own `::after`. The panel base remains blue and the static top/bottom rules do not participate in the sweep. `prefers-reduced-motion` disables the animation and keeps a low-intensity static cue. This visual sub-state stays attached to the current breaking story and matching card, independent of whether the panel is open.
|
||||
|
||||
Compute cards directly reuse the shared `.hud-panel` background, border, shadow, and glass treatment. Summary, essentials, and detail areas add no separate black surface. Their headers share inset spacing with the layer panel through `.hud-panel__header--inset` / `.layer-panel-header`; primary numbers remain prominent and essential information stays visible.
|
||||
|
||||
Saved placement restoration migrates legacy central positions: only old central coordinate locks without the current position marker return to automatic layout, retaining dimensions. Existing edge positions are preserved and versioned. New drags save `positionVersion=2` and continue to honor deliberate placement. The existing storage key is retained instead of clearing every personalized position.
|
||||
|
||||
|
||||
### First frame and spatial card entrance
|
||||
|
||||
@@ -447,3 +462,135 @@ The root attributes `data-earth-first-frame-ms` and `data-earth-surface-ready-ms
|
||||
Positioned cards stay hidden while the shared connector draws for 420 ms, then fade in using the HUD 180 ms opacity timing. Targets remain fixed during drawing; subsequent motion does not replay entrance. Reduced-motion preferences skip this animation.
|
||||
|
||||
Object clicks, search focus and gesture confirmation share `PresentationController`. Cable selection updates selection/legend state without independently opening the card. InfoCard reveal callbacks are versioned so staging or hiding invalidates older callbacks. Connector completion gates card reveal; cancelled cruises no longer hide a newer selection. Object details reuse the regional-card edge candidates and collision scoring, with dimensions measured while staged. Selecting a mobile search result collapses the search drawer so the focused object, connector and compact card remain visible.
|
||||
|
||||
### Satellite particles and smooth rendering
|
||||
|
||||
Satellites use one `THREE.Points` batch with `PointsMaterial`: a 32×32 white disc texture of radius 0.42, `alphaTest=0.5`, and `sizeAttenuation=false`. The dark backdrop batch and simulated sphere lighting are removed. `satellite-display.js` calibrates against satellitemap screenshot pixels: point squares are 2.4px at zooms up to 150% and 8px at 500%, with a smoothstep transition. The disc fills 84% of the square, yielding visible cores of about 2px and 6–7px respectively. Blue/orange/green/purple/rose use the reference palette directly without the prototype's extra exposure. Point sizes retain the DPR-cap-2 adjustment without reducing globe resolution. Ordinary breathing defaults off but remains an opt-in setting. Existing hover/selected rings, halos, open predicted orbits and forward-flow animation remain intact.
|
||||
|
||||
Satellite point sprites use `depthTest=false`. The vertex shader tests the camera-to-satellite segment against the Earth sphere at `CONFIG.earthRadius * 0.998`, moving truly occluded points outside clip space while retaining complete visible sprites near the limb. Orbits, coverage and business coordinates retain their existing paths.
|
||||
|
||||
Real GP/TLE propagation stays in the Worker at 250ms intervals. `satellite-motion.js` caches sample radii and interpolates radius-preserving positions each frame, shared by points, picking, trail tips, selection rings and coverage. `satellite-trail-geometry.js` merges redundant ribbon segments only when the projected error remains below 0.2 pixels across supported zooms. All satellites and 10 history samples are retained; curved or near-camera trails keep their original segments. Only active buffer ranges are uploaded. Trail width is capped at 60% of the point size and tapers toward the tail, avoiding a thick fragmented rim at distant views. `satellite-footprint-geometry.js` retains the full 260×170 coverage grid, created once. The vertex shader evaluates the same surface projection, while local basis and extent uniforms update each frame instead of rebuilding tens of thousands of vertices. Starlink/Iridium footprints share blue `#3399ff`; sunlight changes opacity only, with real coverage models preserved.
|
||||
|
||||
`SATELLITE_CATEGORIES` owns real inclination categories and the blue/orange/green/purple/rose/gray palette. Selecting a satellite automatically switches to its legend. Legend categories directly control points, trails, labels and picking; hiding a selected category clears selection and coverage. Layers and settings have no separate filter entry. Explicit search selection re-enables its category. `satellite-labels.js` shows actual names and models (NORAD identifiers when unavailable) at the configured zoom threshold (**400%** by default, adjustable from **250% to 500%**). The slider below Close-up labels sets a persisted desktop/mobile count limit from 5 to 36 (default 36), immediately rescanning candidates on change. Actual visible counts still depend on available positions and occlusion. It checks occlusion and overlap in the central 65% of the view, scans candidates every 160ms and follows active labels each frame.
|
||||
|
||||
Settings → Display → Show FPS persists through `shared.showFps`, defaulting off. `frame-stats.js` displays actual frame-interval statistics beside the brand, updating once per second; its native tooltip contains mean interval, P95, main-thread CPU time, draw calls and an approximately 30-second rolling mean FPS (total frames / total duration, not an average of per-second rates). Slow foreground frames remain in the sample; visibility changes reset the sampling window. A single high-FPS sample is not a sustained-performance guarantee. Performance acceptance uses an actual canvas of at least 1920×1080, samples at least 30 seconds and targets a mean of 60 FPS while allowing occasional low frames. Compare distant and 400% views with matching viewport, complete data, layers and playback state, and verify category hiding, search selection and FPS persistence after reload.
|
||||
|
||||
The boundary loading tint, land fill, base and HD layers share the `earth-surface.js` geometry factory. Tint and fill share R+0.32, fixing pre-texture diamond rings without changing boundary/HD controls. Intersection and far-zoom depth margins are covered by `earth-surface.test.js`; see [render layer order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
|
||||
|
||||
Cables and landing points share HD-surface coordinates through `cable-surface.js`. `cable-batches.js` controls related halos with per-instance emphasis cleared on deselection. `interactable.js` uses `surface-marker-occlusion.js` for whole-badge visibility and picking near the limb. BGP change events retain their original icons; collectors use activity-colored broadcast badges. Sky material uses the original brightness (multiplier 1.0).
|
||||
|
||||
`toolbar-hover.js` owns the zoom submenu and radial toolbar as one state machine. Magnifier hover opens zoom; trigger, transparent gap bridge and vertical controls form a continuous keep-open region. Leaving first runs the submenu's 220ms exit, then checks the latest pointer location and pinned state before collapsing the parent. Remaining in the radial wake region or pinning the parent closes only zoom; reentry cancels stale timers. CSS hover/focus-within no longer owns visibility independently. Zoom clicks retain parent pinning, with keyboard and touch entry preserved.
|
||||
|
||||
The radial menu inherits global `--hud-scale` (1920×1080 reference, clamped to 0.7–1) instead of fixing its scale at 1. Buttons, icons, fan radii, bottom spacing and the zoom submenu scale together. Base fan coordinates are scaled in CSS transforms; hover wake bounds use the same coordinate scale to prevent hit-region drift under DPI changes. Scaling changes geometry only, preserving animation timing and pinned/hover state transitions.
|
||||
|
||||
### Sky color output
|
||||
|
||||
The star map is decoded as sRGB. `celestial.js` uses shared `preserveTextureOutput()` to encode only the sky material back to display sRGB. The Three.js r128 renderer retains its existing global linear output so Earth, satellites and other layers remain unchanged. `skyBrightness=1.0` retains source brightness; missing output encoding displays decoded dark values directly and cannot be corrected by raising the multiplier alone. Validate with the same map, camera and orientation, comparing rendered pixels and the 1.0/1.5 response instead of checking asset loading only.
|
||||
|
||||
|
||||
### Direct legend filtering
|
||||
|
||||
The current interactive object continues to determine the legend mode; enabling a layer with its existing switch also selects its legend, without adding a filter entry. Cables filter by route name, compute facilities by type, vessels by type and anchored state, and BGP by severity and collector activity. Coverage and impacted-region entries control their respective effects. The geographic legend separately controls lines, land fill and ocean fill without changing the shared land mask data. The obsolete BGP connections/hubs entry is removed.
|
||||
|
||||
`legend-filters.js` owns independent per-mode session filters. `legend.js` renders shared desktop/mobile toggle buttons with `aria-pressed`, preserving keyboard focus when refreshed. `interactable.js` filters draw batches, cluster candidates and picking while retaining complete source data, counts and object identities. Live vessel type changes, data reload and layer toggles respect filters. Hiding a selected item clears its card and overlays; explicit search selection reveals its category. Vessel track requests are versioned so late responses cannot resurrect a hidden track.
|
||||
|
||||
### Stable cards and display preferences
|
||||
|
||||
`callout-rest.js` retains a 450ms interaction-settle gate and subtle floating capped at 1.5px. Hover, keyboard reading, dragging and reduced-motion pause the float. Decorative offsets never enter persistence or placement solving, and the central boundary remains protected.
|
||||
|
||||
Display settings add `shared.satelliteLabelsEnabled` (true by default), `shared.satelliteLabelMinZoom` (2.5–5, default 4) and `shared.skyBrightness` (0–2, default 1). Desktop/mobile controls, reload and reset share runtime state. Sky brightness multiplies the sky material in linear space with its existing local sRGB output, leaving Earth exposure unchanged. Compute legend counts reflect located types. Chip and stacked SVG glyphs remain mapped to `site_type`; a GPU-only dataset does not invent other types or coordinates.
|
||||
|
||||
Focus state and per-type limits decide expanded cards versus capsules. One controller owns left/right stage destinations, capsule following and hover expansion. Manual pinning takes priority; reading pauses floating and automatic expanded-card movement. The older `callout-dock-motion.js` / `callout-routing.js` utilities no longer decide this presentation path.
|
||||
|
||||
|
||||
### Cluster badges and Spiderfy
|
||||
|
||||
Ordinary surface markers retain GPU batches. Each aggregate uses the same projected children in `cluster-presentation.js` through collapse, expansion and the final resting state. A 26px solid numbered circle represents the group, with an amber ring for mixed categories. Membership changes only with data or filters, never through intermediate groups while zooming.
|
||||
|
||||
`cluster-motion.js` drives a 320ms cubic tween after visible icons are prepared and decoded, with encoded textures cached. Children interpolate from the group center to their real projections; `marker-stack.js` owns final display offsets. Animation and resting endpoints retain the same DOM nodes, with continuous reversal.
|
||||
|
||||
The UI's `globeZoomForDistance()` expands groups above 250%, with return hysteresis and no aggregate badges at 500%. Only co-located or extremely close geographic anchors (about 0.00015 degrees) share a compact cross-layer stack. Screen overlap and coarse city buckets never merge different locations. Each icon retains its own real projection as its baseline, with the entire stack bounded to 24px per axis and small groups exposing approximately 8–12px edges. Stable IDs define resting order; selecting a rear item brings it forward, with projected children at `z-index: 1000` and unclustered locked overlays at shared `renderOrder >= 100`, so focus stays above overlapping layers; clearing focus restores its slot. Ordinary expansion ends exactly at real projected positions; coordinates and marker positions remain unchanged. Data refresh removes retired member instances and synchronizes GPU ownership. Satellites and AIS vessels stay separate.
|
||||
|
||||
### Regional news cards
|
||||
|
||||
`news-region-cards.js` reuses `renderNewsCardContent` from `info-card.js` and the cruise adapter's news presentation mapping. Cruise and regional cards share the news template, 📰 icon and gold header. Regional metadata is reduced to source, category and time. Each card owns an independent cancellable `typewriter.js` instance, while an invisible sizing layer reserves the full summary height. Summaries are not truncated. A fixed content area with internal scrolling keeps new headlines and typing from changing placement. Automatic stowing pauses typing and rotation; expanding resumes, and the seven-second reading interval begins after typing completes.
|
||||
|
||||
`news-region-model.js` owns four isolated queues and remaining reading time, using `CRUISE_CONFIG.dwellMs` (seven seconds). It supports per-card pause, navigation, hidden-page resume and current-ID preservation across refreshes. Existing `getCruiseNewsItems()` data updates through `earth:news-payload-updated`, retaining source/category/content-readiness filtering. No new collection requests or global/other-region backfill are introduced.
|
||||
|
||||
News and compute submit scopes through `createCalloutGroups` to one `createSpatialCallouts` controller, sharing destination planning, drag locks, size changes and obstacle handling. Resize state is looked up by element identity; news layout keys use `news:<region>` to avoid compute-key collisions. Existing regional connector styling and routing remain unchanged, as do normal object-detail connectors.
|
||||
|
||||
The opt-in switch persists under `planet.earth.news-region-cards.v1` and lives in the existing news panel. Mobile moves the same nodes into the news drawer and removes their map connectors. Destruction releases cards, animations, observers and subscriptions. Verification covers region isolation, pause/resume, current-story retention, scope removal, simultaneous 1080p layout, mobile and returning to desktop.
|
||||
|
||||
The news cruise adapter completes `presentItem` only after typing finishes, so `CruiseSequencer` dwell never cuts off long text. The wait checks sequence ownership every frame and exits on manual interruption. Regional-news toggles reflect `aria-pressed` with a highlighted background and border. Compute headers retain country flags and use small representative emoji for regional groups.
|
||||
|
||||
### Card DPI and frame-stage boundaries
|
||||
|
||||
Compute, regional news and normal/cruise detail cards share `--hud-detail-card-width` and the same `info-panel.css` typography, metadata and summary rules. Compute cards, charts, icons, spacing and controls use root `--hud-scale`; manually resized inline pixel dimensions are normalized to a scale expression. SVG viewBox units scale with the chart viewport rather than receiving a second multiplier. Europe uses a small EU monogram with the title's font metrics, avoiding Windows flag-emoji fallback alignment.
|
||||
|
||||
The configuration menu retains its independent 340px width and original height limit rather than inheriting detail-card geometry scaling. Compute headers keep the shared HUD specification; larger primary values, charts and content spacing support the common width. Regional motion is owned by the JS tween: CSS position transitions and forced transform compositing are removed. Final left/top values align to device pixels, including decorative float, and connectors use that same displayed position. Regional cards use peripheral stowing when full content does not fit.
|
||||
|
||||
Global selections are heterogeneous: satellites are GeoJSON objects, cables are lines, and only surface interaction markers carry `userData.icon_layer_id`. `markerPresentationId` generates stack identity only for that marker contract, returning null for other selections instead of dereferencing missing fields during the frame loop.
|
||||
|
||||
`frame-stages.js` isolates independent update stages so a failed card, label or layer cannot skip other stages and scene rendering. It reports the stage/error, retries at one-second intervals or immediately after selection changes, and resumes normally on success. Fixing the cause and observing no runtime errors in normal scenarios remain acceptance requirements; isolation does not replace the fix.
|
||||
|
||||
All object-selection confirmations use `showSelectionStatus(name)` and “Selected: name”, including direct clicks, cluster children, search and motion confirmation. `confirmInteractionCandidate` and its feedback option are input-neutral. The cable layer updates selection/legend state while the interaction entry emits one notification. Obsolete lock/location/motion-confirmation message keys are removed; actual motion-mode changes and gesture guidance retain their mode label.
|
||||
|
||||
### Detail-card information cleanup
|
||||
|
||||
Normal details retain identity, business properties, timestamps and useful reliability context. Satellite cards omit the three rendering diagnostics (coverage capability, current presentation and coverage model) without changing coverage controls. News details omit feed identity, source type and fetch channel, retaining publisher, category, region, publication time and summary. Facility cards omit geocoding provenance, confidence scores, reasoning, matched names and verification timestamps while retaining location precision and verification status. Identical GPU model/platform values appear once. Internal source status is omitted; compute precision/measurement and original-source access remain.
|
||||
|
||||
Collector cards omit the redundant activity summary while retaining observation counts, event times and coverage. BGP event confidence remains relevant to alert reliability. Cables omit a single-coordinate row and custom points omit internal record IDs. Empty vessel enrichment no longer displays a caching message or enrichment confidence scores; identity, navigation, available profile data and source attribution remain. Desktop and mobile share field definitions. Underlying data and location-correction operations remain intact.
|
||||
|
||||
### Performance with multiple cards and the full satellite set
|
||||
|
||||
Regional news and compute cards retain their shared dark surfaces, borders, shadows and text while avoiding per-card backdrop filtering of the changing WebGL/video scene and SVG edge distortion. `contain: layout style` limits layout propagation. Pixel-aligned positioning, card count, content, typing and float timing remain intact.
|
||||
|
||||
Each `CalloutConnector` SVG covers its path plus node/shadow padding rather than the entire viewport; path coordinates remain in screen space. Geographic anchors cache unchanged local coordinates, invalidate on coordinate changes, and still update world projection and front/rear visibility each frame. Obstacle tests compare rectangle coordinates directly without allocating bounds for every candidate pair. HUD obstacle measurement precedes that frame's card-style writes.
|
||||
|
||||
Satellite interpolation copies exact endpoint samples and retains radius-preserving interpolation between them. Trail compaction only adds an early rejection for high-altitude trails that already fail the existing error bound. Repeated identical hover/selection indices avoid full alpha-buffer rewrites. Static cable and landing-point local matrices are cached; landing-point scale changes explicitly rebuild their matrix while parent-world transforms propagate normally. No satellites, trail samples or rendering pixels are removed.
|
||||
|
||||
For development diagnostics, append `?profileFrames=1` to the Earth URL. `data-earth-frame-profile` reports per-stage CPU mean, maximum and count every second; profiling is disabled normally. `data-earth-gpu-renderer` identifies the GPU. Performance evidence must include actual canvas pixel dimensions, fully visible card count, satellite visibility, zoom, video state and 30-second average FPS. Pending cards or smaller viewports do not establish a 1080p full-scene result. CPU stage durations are not GPU timings.
|
||||
|
||||
## Detail-card images
|
||||
|
||||
`card-images.js` integrates optional imagery into the existing `showInfoCard` path used by selection and cruise, with one presenter implementation for desktop and mobile. It requests `/api/v1/earth/card-images/resolve` on selection using NORAD IDs or source-qualified object IDs. The server derives models, groups and categories from current catalog data. A bounded 100-identity, 30-second cache avoids repeated lookups; a same-origin storage revision event refreshes open cards after an admin save. No image work runs in the animation loop.
|
||||
|
||||
The shared `card-images.css` places imagery behind the rightmost 68% of the existing body. Horizontal and bottom gradients intersect using standard/WebKit mask composition. The image uses 0.85 opacity with `saturate(1.2) contrast(1.1)`; its containing layer fades to 1, without an additional dark scrim or group opacity reduction. Text, headers, DPI sizing, scrolling, dragging and connectors retain their own layout. Decode precedes the fade; reduced motion skips animation. Missing/broken images produce a data-only card. Selection tokens guard both resolution and decoding, so late results cannot replace newer selections or reopen a closed card. The header source link exposes attribution and license information.
|
||||
|
||||
`card_image_configuration.py` stores assets and assignments in a separate system setting, using revision checks, row locks and audit transactions. Collected records are never edited. Resolution prefers object, model, group and category in that order; explicit per-object suppression stops fallback. Disabled assets cannot resolve; permission metadata remains truthful and separate from the enabled flag. Schemas reject duplicate targets and dangling references. Built-in assets come from the same manifest copied into the backend container, without seeding the database. Known built-in IDs preserve edits/removals while merging newly shipped entries; previous_defaults replaces old bundled URLs while preserving fields changed from the prior default and all custom image URLs.
|
||||
|
||||
Pillow validates still JPEG/PNG/WebP uploads and generates 640/1280-bound WebP derivatives, preserving original files. Content-addressed files are written atomically and served through a read-only public endpoint; management, upload, catalog search and preview require administrators. See the operations runbook for storage/backup. The Frontier photo is facility-specific; Starlink remains a series illustration. Qianfan series imagery is enabled as requested, while its permission metadata remains unverified.
|
||||
|
||||
Category `*` covers satellites, compute facilities, cables, landing stations, vessels, BGP incidents, BGP collectors and custom points, after specific category rules. Stable keys are MMSI, incident_key, collector_id and the custom-point ID. Missing identities in old snapshots allow only a type illustration. Rebind a collected object if its upstream source_id changes.
|
||||
|
||||
|
||||
### Five compute facility types and default imagery
|
||||
|
||||
Supported types are `supercomputer`, `gpu_cluster`, `data_center`, `cloud_center` and `edge_node`. `compute-site-types.js` centralizes frontend titles, labels, counts and fallback; backend `compute_types.py` shares classification between GeoJSON and image lookup. An explicit supported metadata.site_type wins, followed by data_type and known source defaults. Unrecognized facilities use a generic data-center presentation instead of becoming GPU clusters. No facilities are fabricated or existing records reclassified by name.
|
||||
|
||||
Each type supports legend filtering, marker icons, desktop/mobile details and image identity. New infrastructure cards use identity, operator, capacity, power and location fields without GPU counts or TOP500 ranks. The map accepts known sources or records declaring a supported compute type. Regional facilities retain snapshot/country constraints; infrastructure records do not require a GPU measurement contract and do not imply a global compute share. Built-in collectors remain TOP500 and Epoch AI; adding types does not create records.
|
||||
|
||||
Seven user-supplied images provide defaults for the five facility types, BGP collectors and BGP events. All five categories are available in image management before data exists. Exact assignments retain priority. Manifest previous_defaults / previous_rules migrate old bundled sources and unchanged collector assignments while preserving custom crop, enabled state and specific bindings.
|
||||
|
||||
|
||||
### Frame-work reductions with unchanged presentation
|
||||
|
||||
Cluster presentation reuses each parent's world matrix and local camera position within one update, refreshing on every frame. Fully collapsed members without child DOM still participate in hidden-state reconciliation, but skip invisible endpoint projections. Unchanged count and ARIA text no longer cause repeated DOM writes.
|
||||
|
||||
Static point batches refresh on data, selection, filtering, distance scaling or explicit invalidation; selected overlays continue pulsing every frame. Parsed colors cache immutable strings/numbers only, never mutable caller-owned Color objects. Satellite dot/halo overlays update only the required ancestor transform rather than traversing all Earth descendants twice.
|
||||
|
||||
Trail sampling combines the radius bound and initial error scan, preserving the existing threshold and subdivision indices. No satellites, trajectory samples, card content or resolution are removed. Deterministic comparison covered 10,000 histories with identical indices. Local Bun microbenchmarks (not whole-page FPS): 480 cluster members took about 1.19→0.90 ms collapsed and 4.47→2.86 ms expanded; scanning 16,621 trails took about 11.50→8.94 ms. End-to-end FPS requires a separate 30-second foreground comparison at identical resolution and scene/card/video settings.
|
||||
|
||||
Regional news collapse buttons explicitly invoke the shared HUD controller. Their bodies use hud-panel__body--collapsible and aria-expanded follows the actual state. Only user clicks collapse cards; layout and performance code never hide their contents automatically.
|
||||
|
||||
Floating object details and regional news share `--hud-detail-card-max-height`: 390px times HUD scale, capped at viewport height minus 24px. A min-height:0 flex chain keeps the header fixed and scrolls only the image-backed body. Short cards keep their natural height. BGP incidents prioritize type/severity/status/impact/time/summary; collectors prioritize identity/status/recent activity; facilities prioritize name/capacity/hardware, with location verification and provenance later. Desktop and mobile field order share CARD_CONFIG.
|
||||
|
||||
|
||||
### Detail identity and connector consistency
|
||||
|
||||
BGP collector indexes use collector names. Shared marker indexes omit empty IDs and refuse cross-refresh replacement for ambiguous duplicates; a clicked marker still in the current layer retains its own reference. Stack presentation shares icon_item_id with the index. BGP incidents prefer incident_key, while raw anomalies use source plus ID to prevent numeric collisions across feeds.
|
||||
|
||||
Satellite detail connectors identify the selected NORAD and relocate its index when the catalog array changes. Selection highlights rebind to that NORAD after refresh; a removed target clears selection. Uninitialized zero positions are not valid anchors. Cable anchors remain confined to the selected cable geometry, landing points use the clicked marker, and regional news/compute cards retain their regional anchors.
|
||||
|
||||
|
||||
When compute image configuration has no model/group rules, resolution uses a bounded fast path: the stable source prefix supplies the supercomputer/GPU category, then object and category rules are checked without scanning the collected catalog. Adding model or group rules automatically uses the full catalog resolver. Frontend identities, collected data and other object types are unchanged.
|
||||
|
||||
@@ -1,84 +1,35 @@
|
||||
# Intelligent Planet Interactable Clustering
|
||||
# Earth interactable clustering
|
||||
|
||||
Earth interactable icons are managed by `createInteractableLayer`. Rendering still uses Three.js `Points`, and picking, hover, locked state, tooltips, and cruise focus still depend on marker `userData`; the clustering strategy only decides which markers are represented by a cluster dot.
|
||||
`interactable.js` owns fixed geographic membership, `cluster-motion.js` owns timed tweening, and `cluster-presentation.js` owns circular badges, projected children, picking and Spiderfy. Independent markers remain GPU Points. Each clustered child uses the same DOM icon throughout its animation and at the final position.
|
||||
|
||||
## Strategies
|
||||
## Membership and zoom
|
||||
|
||||
`cluster.strategy` supports three modes:
|
||||
Spherical buckets use real `icon_base_position` anchors. Defaults are distance 15 in Earth units, at least two members and at most 14 per group. Data, filter and visibility changes invalidate membership; rotation and zoom do not. Badges stay at the spherical centroid without screen avoidance.
|
||||
|
||||
- `stable-spherical`: stable spherical clustering. It clusters by local 3D positions on the globe and caches topology by zoom band. Rotation and small zoom changes within the same band only update projection and size. Use it for semi-static layers such as BGP, compute centers, and Earth interactables.
|
||||
- `dynamic-screen`: dynamic screen-space clustering. This keeps the previous projection-based behavior and evaluates visible relationships per frame. Use it for high-frequency realtime layers such as vessels, or as a fallback.
|
||||
- `none`: no clustering. Every marker is shown independently. Use it for low-count layers or precision-first views.
|
||||
Use `cluster: { enabled: true }` or `cluster: false`. Options are `distance`, `minCount`, `maxMarkersPerDot`, `disableAboveZoom` (2.5) and `expansionHysteresis` (0.08). Zoom uses the UI's `globeZoomForDistance()` mapping: expand above 250%, collapse below 230%, and show no aggregate badges at 500%. Satellites and AIS vessels remain separate.
|
||||
|
||||
`cluster: false` is equivalent to `strategy: "none"`. If a layer enables clustering without declaring a strategy, it keeps the compatible `dynamic-screen` behavior.
|
||||
## Motion and Spiderfy
|
||||
|
||||
## Stable Spherical
|
||||
Both directions use a 320ms easeInOutCubic tween, complete after wheel input stops, and reverse continuously. Children interpolate directly from the centroid to their real local positions and retain the same DOM nodes at the endpoint. Because this chord can temporarily enter the sphere, occlusion uses a radial surface anchor while projection retains the interpolated position; intermediate frames remain visible.
|
||||
|
||||
`stable-spherical` moves cluster identity from screen distance to globe distance:
|
||||
Only co-located or extremely close geographic anchors (about 0.00015 degrees) share a compact cross-layer stack. Screen overlap and coarse city buckets never merge different locations. Each icon retains its own real projection as its baseline, with the entire stack bounded to 24px per axis and small groups exposing approximately 8–12px edges. Stable IDs define resting order; selecting a rear item brings it forward, and clearing focus restores its slot. Ordinary expansion ends exactly at real projected positions; coordinates and marker positions remain unchanged.
|
||||
|
||||
- Each marker uses `icon_base_position` as its geographic anchor.
|
||||
- By default, zoom levels above `2.5` force clustering off and show every marker as its original icon.
|
||||
- The current zoom maps to a discrete band; rotation and small zoom changes inside the same band do not recompute topology.
|
||||
- Clusters are recomputed only when the band, data revision, visibility, or filter state changes.
|
||||
- A cluster centroid is computed from member 3D positions and normalized back to the globe shell, so the cluster dot stays rigidly aligned to its geographic center.
|
||||
- Cluster dots do not participate in 2D avoidance, so screen-space repulsion cannot push them away from their real geographic projection.
|
||||
- Band changes use a small hysteresis margin so zooming at a boundary does not repeatedly bounce between two bands.
|
||||
- Newly created marker and cluster dots run a short scale + opacity ease. This is only a rendering transition; it does not change marker coordinates, picking objects, or locked state.
|
||||
## Interaction and BGP
|
||||
|
||||
The stable strategy uses spherical bucket/hash neighbor lookup and must not use an all-pairs loop. Most frames only pay projection and material-size cost; topology cost is paid only on band or data changes.
|
||||
Clicking a badge expands it and pauses rotation. Legs have 14px transparent hit areas; hover highlights the matching leg and child, and either click selects the same business object. The canvas container accepts wheel input over the canvas and projected markers while excluding HUD scrolling.
|
||||
|
||||
## Configuration
|
||||
Projected textures bake the original category tint. BGP events retain severity colors. Badges containing BGP and expanded BGP event children use an expanding/fading halo based on `BGP_CONFIG.ring`; replaced GPU effects are hidden to avoid duplicate rings at old anchors. Reduced-motion preference disables the pulse.
|
||||
|
||||
```js
|
||||
const computeCenterIconLayer = createInteractableLayer({
|
||||
id: "computeCenters",
|
||||
// ...
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
minCount: 2,
|
||||
maxMarkersPerDot: 14,
|
||||
transitionMs: 220,
|
||||
bandHysteresis: 0.08,
|
||||
disableAboveZoom: 2.5,
|
||||
bands: [
|
||||
{ key: "far", maxZoom: 1.7, distance: 15 },
|
||||
{ key: "mid", maxZoom: 2.6, distance: 8 },
|
||||
{ key: "near", maxZoom: 3.5, distance: 4 },
|
||||
{ key: "detail", maxZoom: Infinity, distance: 0 },
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
## Removed paths
|
||||
|
||||
Realtime layers can stay dynamic:
|
||||
The low-zoom compact-dot desktop/mobile controls, persistence field, textures, rendering/picking branches and translations are removed. Dynamic screen clustering, discrete zoom bands, GPU aggregate dots and the old creation-time fade/scale transition are also removed. There is no legacy strategy fallback.
|
||||
|
||||
```js
|
||||
const vesselIconLayer = createInteractableLayer({
|
||||
id: "vessels",
|
||||
// ...
|
||||
cluster: {
|
||||
strategy: "dynamic-screen",
|
||||
enabled: true,
|
||||
maxMarkersPerDot: 10,
|
||||
},
|
||||
});
|
||||
```
|
||||
## Regression coverage
|
||||
|
||||
## Tuning
|
||||
Tests cover 320ms completion, reversal continuity, visible intermediate frames, retained endpoint nodes, 500% expansion, immutable source coordinates, filtering cleanup, wheel target routing, linked leg hover/click and projected BGP category colors: `cluster-motion.test.js`, `cluster-presentation.test.js`, `globe-navigation.test.js`, `marker-badges.test.js`.
|
||||
|
||||
- `distance` is the 3D globe-distance threshold and uses the same unit as `CONFIG.earthRadius`. Larger values cluster more aggressively.
|
||||
- The farthest band usually uses a larger `distance`; the nearest detail band usually uses `0` to split clusters into original icons.
|
||||
- `bandHysteresis` controls band-boundary stickiness. Too little can flicker near thresholds; too much can make band changes feel late.
|
||||
- `transitionMs` controls cluster split/merge easing. Keep it around 160-260ms; longer durations can make realtime layers feel sluggish.
|
||||
- `disableAboveZoom` controls the precision-view threshold. It defaults to `2.5`; above that zoom, no cluster dots are generated. Set it to `false` to disable the hard threshold.
|
||||
- High-frequency realtime layers should prefer `dynamic-screen` so frequent data changes do not trigger stable topology recomputation.
|
||||
- If a layer behaves poorly, switch it back to `dynamic-screen` or use `cluster: false`.
|
||||
## Refresh and first-frame preparation
|
||||
|
||||
## Acceptance Checks
|
||||
A reused cluster ID does not imply reused Object3D instances: stale children are reconciled against current instances. Ownership changes immediately rebuild the GPU batch, and topology invalidation clears its old signature.
|
||||
|
||||
- Rotating the globe inside one zoom band should not cause clusters to flicker or regroup.
|
||||
- Cluster dots should stay aligned with the globe-surface centroid and should not be pushed by avoidance.
|
||||
- Zooming into the detail band should restore the layer's original icon textures and click behavior.
|
||||
- Zoom levels above 250% should not show cluster dots.
|
||||
- Realtime layers such as vessels should reflect incoming updates immediately.
|
||||
Expansion prepares visible icons and waits for image decoding before starting its 320ms clock. Encoded URLs are reused per texture, and invisible rear members are not instantiated. Regression tests cover repeated refresh/zoom, preparation slower than the tween duration, selection/restoration and connector anchors.
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# Earth Layer Style Property Index
|
||||
|
||||
`EARTH_MATERIAL_CONFIG.surface.night` centralizes night texture contrast (detailGamma=0.55) and separate land/ocean tint and ambient values. It only contributes to the night branch, keeps day exposure unchanged, and reuses the geographic mask without another texture or draw call.
|
||||
|
||||
## Modern object presentation
|
||||
|
||||
- Surface: `EARTH_MATERIAL_CONFIG.surface` owns exposure=0.9, landSaturation=0.9, oceanAbsorption=0.36, oceanDeepColor=[0.004,0.012,0.032], night retention and Fresnel parameters. `atmosphere` owns the single shell's radius, color and strength. Base and terrain specular highlights are disabled.
|
||||
- Cables blend a cool normal tint with their category color, consistently with the legend. Selected pulses follow cumulative route length with 6 repetitions and speed 4.5 in the existing batch draw. Drawing no longer writes depth; original Line pick objects, cable IDs and state logic remain intact.
|
||||
- Landing points use the 128×128 monochrome `marker-landing-point.svg` micro-core instead of a glossy procedural ball; Sprite picking and billboard size logic remain intact.
|
||||
- Compute markers use canonical SVG inner glyphs over shared gradient plates, pale rims, top highlights and shadows. GPU/AI uses purple stacked layers, supercomputers purple chips, and infrastructure teal server racks. Estimated badges and picking remain intact.
|
||||
- Satellites share vivid real-inclination category colors with the legend and predicted orbits. Normal markers are prototype hard discs without outlines or sphere shading; breathing defaults off. Coverage stays blue and sunlight only affects opacity.
|
||||
|
||||
|
||||
|
||||
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
|
||||
|
||||
## Naming Conventions
|
||||
@@ -20,15 +32,11 @@ This document records the material, color, opacity, line width, radius offset, a
|
||||
| Earth base radius | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
|
||||
| Earth base color | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
|
||||
| Earth base emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
|
||||
| Earth base specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
|
||||
| Earth base shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
|
||||
| Earth base specular | `EARTH_MATERIAL_CONFIG.specular` | `0x000000` | `MeshPhongMaterial.specular` |
|
||||
| Earth base shininess | `EARTH_MATERIAL_CONFIG.shininess` | `0` | `MeshPhongMaterial.shininess` |
|
||||
| Earth base opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
|
||||
| HD texture radius offset | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.48` | Standalone HD texture sphere radius; must keep enough depth separation from the land/ocean base and Earth base sphere to avoid far-zoom z-fighting |
|
||||
| HD texture opacity | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | HD texture `MeshPhongMaterial.opacity` |
|
||||
| HD texture renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |
|
||||
| HD texture specular | `EARTH_MATERIAL_CONFIG.textureOverlaySpecular` | `0x05080d` | Reduces specular highlight in direct-light areas to avoid blown-out texture |
|
||||
| HD texture shininess | `EARTH_MATERIAL_CONFIG.textureOverlayShininess` | `4` | Reduces specular concentration |
|
||||
| HD texture color multiplier | inline | `0xffffff` | `_earthTextureOverlayMaterial.color` |
|
||||
|
||||
## Earth Occluder and Day/Night
|
||||
|
||||
@@ -36,7 +44,7 @@ This document records the material, color, opacity, line width, radius offset, a
|
||||
| --- | --- | --- | --- |
|
||||
| Occluder radius factor | `EARTH_MATERIAL_CONFIG.occluderRadiusFactor` | `0.999` | Depth occluder sphere radius |
|
||||
| Occluder segments | `EARTH_MATERIAL_CONFIG.occluderSegments` | `48` | Occluder geometry segments |
|
||||
| Occluder renderOrder | inline | `-1` | `occluder.renderOrder` |
|
||||
| Occluder renderOrder | inline | `0.5` | `occluder.renderOrder` |
|
||||
| Day/night sun direction | `EARTH_MATERIAL_CONFIG.dayNight.sunDirection` | `{ x: 1, y: 0.2, z: 0.4 }` | Custom day/night shader |
|
||||
| Night-side minimum brightness | `EARTH_MATERIAL_CONFIG.dayNight.nightFloor` | `0.24` | Shader uniform |
|
||||
| Day-side boost | `EARTH_MATERIAL_CONFIG.dayNight.dayBoost` | `1.12` | Shader uniform |
|
||||
@@ -50,27 +58,12 @@ This document records the material, color, opacity, line width, radius offset, a
|
||||
|
||||
| Name | Variable | Current Value | Location / Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Inner atmosphere radius factor | `EARTH_MATERIAL_CONFIG.atmosInnerRadiusFactor` | `1.01` | `atmosInnerGeo` |
|
||||
| Inner atmosphere segments | `EARTH_MATERIAL_CONFIG.atmosInnerSegments` | `64` | `atmosInnerGeo` |
|
||||
| Inner atmosphere color | `EARTH_MATERIAL_CONFIG.atmosInnerColor` | `[0.25, 0.62, 1.0]` | Shader RGB |
|
||||
| Inner atmosphere rim power | `EARTH_MATERIAL_CONFIG.atmosInnerRimPower` | `3.2` | Shader rim falloff |
|
||||
| Inner atmosphere intensity | `EARTH_MATERIAL_CONFIG.atmosInnerIntensity` | `0.18` | Shader alpha multiplier |
|
||||
| Outer atmosphere radius factor | `EARTH_MATERIAL_CONFIG.atmosOuterRadiusFactor` | `1.016` | `atmosOuterGeo` |
|
||||
| Outer atmosphere segments | `EARTH_MATERIAL_CONFIG.atmosOuterSegments` | `48` | `atmosOuterGeo` |
|
||||
| Outer atmosphere color | `EARTH_MATERIAL_CONFIG.atmosOuterColor` | `[0.18, 0.45, 0.9]` | Shader RGB |
|
||||
| Outer atmosphere rim power | `EARTH_MATERIAL_CONFIG.atmosOuterRimPower` | `5.0` | Shader rim falloff |
|
||||
| Outer atmosphere intensity | `EARTH_MATERIAL_CONFIG.atmosOuterIntensity` | `0.02` | Shader alpha multiplier |
|
||||
| Atmosphere blending | inline | `THREE.AdditiveBlending` | `ShaderMaterial.blending` |
|
||||
| Atmosphere renderOrder | inline | `1` | `atmosInner/Outer.renderOrder` |
|
||||
| No-HD-texture rim glow color | `EARTH_MATERIAL_CONFIG.rimGlowColor` | `[0.35, 0.65, 1.0]` | Fresnel shell RGB when HD texture is hidden or unavailable |
|
||||
| No-HD-texture rim glow power | `EARTH_MATERIAL_CONFIG.rimGlowPower` | `3.8` | Shader rim falloff; higher = narrower edge |
|
||||
| No-HD-texture rim glow intensity | `EARTH_MATERIAL_CONFIG.rimGlowIntensity` | `0.28` | Shader alpha multiplier |
|
||||
| No-HD-texture rim glow segments | `EARTH_MATERIAL_CONFIG.rimGlowSegments` | `64` | `earth-rim-glow` geometry segments |
|
||||
| Cloud layer radius offset | `CLOUD_LAYER_CONFIG.radiusOffset` | `3` | Cloud sphere radius |
|
||||
| Cloud layer segments | `CLOUD_LAYER_CONFIG.widthSegments / heightSegments` | `64 / 64` | Cloud sphere geometry |
|
||||
| Cloud layer opacity | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshPhongMaterial.opacity` |
|
||||
| Cloud layer opacity | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshBasicMaterial.opacity` |
|
||||
| Cloud texture | `CLOUD_LAYER_CONFIG.textureUrl` | `"./assets/earth_clouds_1024.png"` | Cloud texture map |
|
||||
| Cloud blending | inline | `THREE.AdditiveBlending` | `MeshPhongMaterial.blending` |
|
||||
| Cloud blending | inline | `THREE.AdditiveBlending` | `MeshBasicMaterial.blending` |
|
||||
|
||||
## Land/Ocean Base and Country Borders
|
||||
|
||||
@@ -94,7 +87,7 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Land/ocean base renderOrder | `COUNTRY_BOUNDARY_CONFIG.landRenderOrder` | `0.86` | `country-land-ocean.renderOrder` |
|
||||
| Land/ocean mask size | `landMaskWidth / landMaskHeight` | `2048 / 1024` | Canvas / DataTexture size |
|
||||
| Country tint color | `COUNTRY_BOUNDARY_CONFIG.tintColor` | `0x0b1830` | Tint when HD texture is off |
|
||||
| Country tint radius offset | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.04` | `country-tint` radius |
|
||||
| Country tint radius offset | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.32` | `country-tint` radius |
|
||||
| Country tint renderOrder | `COUNTRY_BOUNDARY_CONFIG.tintRenderOrder` | `0.2` | `country-tint.renderOrder` |
|
||||
| Border line color | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | Normal border line |
|
||||
| Border line opacity | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | Normal border line opacity |
|
||||
@@ -123,8 +116,8 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Terrain opacity | `TERRAIN_CONFIG.opacity` | `0.68` | `MeshPhongMaterial.opacity` |
|
||||
| Terrain color | `TERRAIN_CONFIG.color` | `0x8aa884` | `MeshPhongMaterial.color` |
|
||||
| Terrain emissive | `TERRAIN_CONFIG.emissive` | `0x030704` | Reduces self-emission to preserve terrain shading |
|
||||
| Terrain specular | `TERRAIN_CONFIG.specular` | `0x344438` | Gives terrain local sheen without boosting HD texture brightness |
|
||||
| Terrain shininess | `TERRAIN_CONFIG.shininess` | `16` | Tightens terrain highlight |
|
||||
| Terrain specular | `TERRAIN_CONFIG.specular` | `0x000000` | Matte terrain; no white specular lobe |
|
||||
| Terrain shininess | `TERRAIN_CONFIG.shininess` | `0` | Specular disabled |
|
||||
| Terrain renderOrder | inline | `1.2` | `terrain.renderOrder` |
|
||||
| Terrain polygonOffset | inline | `factor -1`, `units -1` | Reduces z-fighting near sphere surface |
|
||||
|
||||
@@ -148,11 +141,11 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Name | Variable | Current Value | Location / Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Default cable color | `CABLE_COLORS.default` | `0xffff44` | Used when no data color available |
|
||||
| Cable radius offset | `CABLE_CONFIG.line.altitudeOffset` | `0.2` | Cable line radius |
|
||||
| Cable radius offset | `CABLE_CONFIG.line.altitudeOffset` | `0.48` | Cable line radius |
|
||||
| Cable line width | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
|
||||
| Cable opacity | `CABLE_CONFIG.line.opacity` | `1.0` | Cable line opacity |
|
||||
| Cable renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | Cable line level |
|
||||
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | Same surface height as cable lines, avoiding a floating marker |
|
||||
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | Same surface height as cable lines, avoiding a floating marker |
|
||||
| Landing point sprite height | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` base height |
|
||||
| Landing point reference FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | Matches the current Earth camera FOV |
|
||||
| Landing point scale minimum | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | Minimum multiplier after roughly 200% zoom, limiting high-zoom screen footprint; `3 * 0.16 = 0.48` |
|
||||
@@ -162,7 +155,7 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Landing point opacity | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
|
||||
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | Same level as cable lines; `depthTest: false` keeps the ball whole, while camera-to-center globe occlusion hides back-side points |
|
||||
| Landing point dim brightness | `landingPointVisual.dimBrightness` | `0.62` | Dim state color multiplier |
|
||||
| Related landing point opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | Highlight pulse |
|
||||
| Related landing halo extent | `landingPointVisual.related.haloScale / haloFalloff` | `1.65 / 8` | Expands only the halo canvas; core size and color stay fixed |
|
||||
| Dimmed landing point color | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | Dim state color; avoids dark base showing through as a dark hole |
|
||||
| Dimmed landing point opacity | `landingPointVisual.dimmed.opacity` | `0.78` | Dim state opacity; no longer uses low alpha blending with dark base |
|
||||
|
||||
@@ -174,14 +167,11 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Satellite altitude compression scale | `SATELLITE_CONFIG.altitudeCompressionKm` | `1200` | TLE/SGP4 altitude compression; larger values soften high-orbit separation |
|
||||
| Satellite real-altitude cap | `SATELLITE_CONFIG.maxDisplayAltitudeKm` | `40000` | Altitude clamp for the compressed display mapping, covering GEO-range altitude |
|
||||
| Satellite real-altitude display offset range | `minRealAltitudeOffset / maxRealAltitudeOffset` | `4 / 25` | Compressed offset added to `CONFIG.earthRadius`; `25` is about one quarter of the current globe radius `100`, keeping high-orbit separation visible without pushing GEO/MEO trails too far away |
|
||||
| Satellite dot base pixel size | `SATELLITE_CONFIG.dotBaseSize` | `2.8` | Point shader size |
|
||||
| Satellite backdrop dot scale | `SATELLITE_CONFIG.dotBackdropScale` | `1.28` | Backdrop dot size |
|
||||
| Satellite dot opacity range | `dotOpacityMin / dotOpacityMax` | `0.42 / 1.0` | Breathing animation |
|
||||
| Satellite dot breathing speed | `SATELLITE_CONFIG.dotBreathingSpeed` | `0.12` | Dot opacity animation |
|
||||
| Satellite backdrop renderOrder | inline | `5` | `satelliteBackdropPoints.renderOrder` |
|
||||
| Satellite dot base pixel size | `SATELLITE_CONFIG.dotBaseSize` | `2.2` | Legacy interaction hit-radius basis; not visual size |
|
||||
| Satellite dot opacity range | `dotOpacityMin / dotOpacityMax` | `0.72 / 1.0` | Selected-point breathing; ordinary breathing defaults off |
|
||||
| Satellite dot renderOrder | inline | `6` | `satellitePoints.renderOrder` |
|
||||
| Satellite trail length | `SATELLITE_CONFIG.trailLength` | `10` | Trail buffer |
|
||||
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` | Ribbon shader uniform |
|
||||
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` (cap) | Also capped at 60% of point size with tapered short tails |
|
||||
| Selected ring size | `SATELLITE_CONFIG.ringSize` | `0.07` | Hover / locked ring sprite |
|
||||
| Satellite overlay renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | Locked ring / halo / orbit |
|
||||
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill and Iridium coverage ring; must stay above land / texture / terrain surface layers |
|
||||
@@ -216,8 +206,8 @@ AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel
|
||||
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover overlay size multiplier |
|
||||
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked overlay size multiplier, with pulse |
|
||||
| Dimmed scale / opacity | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | Dim state |
|
||||
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | Marker texture |
|
||||
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | Marker texture |
|
||||
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#a855f7"` | Marker texture |
|
||||
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#a855f7"` | Marker texture |
|
||||
| Linked color | `COMPUTE_CENTER_CONFIG.colors.linked` | `"#f8fafc"` | Linked state |
|
||||
| Compute center renderOrder | local `COMPUTE_CENTER_RENDER_ORDER` | `4.5` | Surface facility below satellites |
|
||||
|
||||
@@ -259,7 +249,54 @@ AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel
|
||||
| Moon halo scale | `CELESTIAL_CONFIG.moonHaloScale` | `62` | Moon halo |
|
||||
| Sun light color / intensity | `sunLightColor / sunLightIntensity` | `0xfff4df / 1.02` | Scene light |
|
||||
| Back light color / intensity | `backLightColor / backLightIntensity` | `0x2b4c78 / 0.3` | Scene light |
|
||||
| Star count | `STARFIELD_CONFIG.count` | `8000` | `createStars()` |
|
||||
| Star radius range | `minRadius + radiusJitter` | `800 + 200` | Random distribution |
|
||||
| Star color | `STARFIELD_CONFIG.color` | `0xffffff` | `PointsMaterial.color` |
|
||||
| Star size | `STARFIELD_CONFIG.size` | `0.5` | `PointsMaterial.size` |
|
||||
|
||||
|
||||
## Badge taxonomy and palette
|
||||
|
||||
| Type | Primary color | Inner glyph |
|
||||
| --- | --- | --- |
|
||||
| Datacenter/infrastructure | Teal `#14b8a6` → `#0d9488`, rim `#5eead4` | Server rack |
|
||||
| GPU/AI | Purple `#a855f7` → `#7e22ce`, rim `#d8b4fe` | Stacked compute layers |
|
||||
| Supercomputer | Same purple family | Chip |
|
||||
| Cable landing point | Amber `#f59e0b`, pale-gold `#fef3c7` rim | Glowing circular core |
|
||||
| BGP collector | Blue and existing activity heat colors | Rounded badge containing the broadcast-tower glyph |
|
||||
| BGP event | Low/medium/high/critical: blue/amber/orange/red | Irregular flat event symbols, no badge plate |
|
||||
| Vessel | Cargo blue, tanker orange, passenger teal, fishing amber, military indigo, other silver | Heading arrow; anchored state retained |
|
||||
| Custom marker | Place green, note yellow, alert red, default blue | Pin, note sheet, warning triangle |
|
||||
|
||||
BGP change events retain irregular flat symbols as the exception; collectors use rounded badges; other finite surface objects share cached badge textures without a new DOM node or draw object per marker. Vessels retain heading buckets and vertex colors rather than multiplying buckets by vessel type. Satellites use distinct category colors and hard discs without an outline; orbit and footprint calculations stay intact. The low-zoom dot control, persisted field and rendering/picking branches are removed.
|
||||
|
||||
|
||||
### Stable satellite palette
|
||||
|
||||
Category colors follow [satellitemap.space](https://satellitemap.space/) while retaining this project's real inclination classes: 0–30° blue `#3399ff`, 30–60° orange `#ff801a`, 60–90° green `#00e680`, 90–120° purple `#cc33ff`, 120–180° rose `#ff3366`, unknown gray `#94a3b8`. A satellite keeps its hue across day/night; predicted orbits no longer use an unrelated yellow-purple gradient.
|
||||
|
||||
All coverage uses `#3399ff`, opacity 0.34 and day-side opacity multiplier 1.65. Normal points have no dark outline; one `PointsMaterial` batch uses alphaTest=0.5. `satellitePixelSize()` smoothly varies visual size from 2.4 to 8.0.
|
||||
|
||||
### Selected orbit and panel outlines
|
||||
|
||||
Prediction covers the full period without truncating or fading the endpoints and without an appended closing vertex. Highlights travel forward along cumulative arc length in the same Line draw. TV and shared HUD shells use square corners, including the player clip, without changing panel size or dragging. Temporary frame-statistics instrumentation and its QA-only DOM attributes are removed from release code.
|
||||
|
||||
### Landing points, limb badges and subtle sky brightness
|
||||
|
||||
Cables and landing points share R+0.48 and renderOrder=1. `cable-surface.js` generates both on the actual surface triangles, removing ideal-sphere versus surface endpoint parallax. Normal points are yellow discs with a pale ring and no persistent halo. Only related points get a subtle pulsing halo in the existing batch shader; unrelated points dim, and deselection restores the normal yellow state.
|
||||
|
||||
Rounded badges use anchor occlusion through `surface-marker-occlusion.js`: the GPU keeps a visible anchor's whole badge and hides it when Earth occludes the anchor. Screen-space picking shares the same check. Only badge layers replace per-pixel depth clipping; BGP change events keep their flat symbols and existing depth behavior. Collectors retain activity colors inside rounded plates. Normal, hover, locked and cluster badges share the path.
|
||||
|
||||
Only the sky texture material receives `CELESTIAL_CONFIG.skyBrightness = 1.0`; Earth exposure, celestial lighting and business-layer brightness stay unchanged.
|
||||
|
||||
Vessels are independent moving objects: their pre-badge heading arrows, anchored dots and hover/selected glow are restored. They do not use the fixed-facility rounded plates or whole-badge occlusion path.
|
||||
|
||||
Normal and related landing points retain the same orange core and yellow rim. Related state adds a wider soft pulsing halo without changing core color, opacity or size. Deselecting clears the halo; unrelated points retain the dim state.
|
||||
|
||||
|
||||
### Sky color output
|
||||
|
||||
The star map is decoded as sRGB. `celestial.js` uses shared `preserveTextureOutput()` to encode only the sky material back to display sRGB. The Three.js r128 renderer retains its existing global linear output so Earth, satellites and other layers remain unchanged. `skyBrightness=1.0` retains source brightness; missing output encoding displays decoded dark values directly and cannot be corrected by raising the multiplier alone. Validate with the same map, camera and orientation, comparing rendered pixels and the 1.0/1.5 response instead of checking asset loading only.
|
||||
|
||||
Satellite hover rings use the hovered satellite inclination category color, matching the point and legend. Changing the hovered object resolves its color again; position updates preserve it.
|
||||
|
||||
All map cards use the root HUD DPI scale. Regional and cruise news share header icon, colors, typography and padding; regional cards simplify metadata and add independent rotation controls. Expanded cards retain full content while other regions use capsules that temporarily open with the same card styling on hover. Rear capsules dim and shrink without removing business objects. Thin/bold lines share `CalloutConnector`; bold styling widens the white line, adds dark outlines and enlarges endpoints. Compute metrics, trend charts and title icons follow HUD scaling. Europe uses a compact EU monogram; other region emoji follow the same scale.
|
||||
|
||||
Compute markers use five canonical SVGs: compute-supercomputer.svg, compute-gpu-cluster.svg, compute-hdd-network.svg, compute-cloud.svg and compute-edge.svg. They retain the compute layer purple badge and existing selection behavior.
|
||||
|
||||
@@ -84,6 +84,20 @@ Importance levels are fixed: `low` 0–34, `medium` 35–59, `high` 60–79, and
|
||||
|
||||
## Configuration and Cache
|
||||
|
||||
`GET /api/v1/news/earth-feed` reads saved news. The scheduler's
|
||||
`earth_news_rss_refresh` job supplements RSS on startup and every 10 minutes,
|
||||
checking freshness and feed coverage by region. Redis locks coordinate refreshes
|
||||
across processes. Failed feeds use the source's configured cooldown while other
|
||||
sources continue. Health timestamp updates alone do not broadcast content changes.
|
||||
|
||||
`POST /api/v1/news/refresh` accepts a region or view coordinates and returns
|
||||
HTTP 202 with a `refresh_id` shared by active manual requests for that region.
|
||||
`GET /api/v1/news/refresh/{refresh_id}` reports pending, running, succeeded, or
|
||||
failed state. On success, the frontend reloads news and updates the panel before
|
||||
showing success. Existing content stays readable during the wait. Failure
|
||||
re-enables the button for retry. Saving or resetting source configuration also
|
||||
schedules an immediate background check.
|
||||
|
||||
`GET /api/v1/earth/news-sources` returns the default or saved configuration. `PUT /api/v1/earth/news-sources` saves it, increments `cache_version`, and clears the process region cache. `POST /api/v1/earth/news-sources/reset` restores defaults. `POST /api/v1/earth/news-sources/test` tests one RSS/Atom/Aggregated source without writing news items.
|
||||
|
||||
## Manual News Content
|
||||
@@ -160,13 +174,13 @@ The Web Earth category chips only store the current browser preference; changing
|
||||
|
||||
When `sources` is omitted, the service layer prefers stories that already have displayable title and summary content for the requested `locale`, supplements candidates from enabled sources, and rotates sources so one source's newest pending items cannot occupy all 12 default slots. An explicit `sources` filter remains precise and does not supplement other sources. The database query layer owns region, category, source, and ordering constraints only; it does not own locale presentation policy.
|
||||
|
||||
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next `earth-feed` request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.
|
||||
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not prove that items are stored or visible under the current region/category filters. Saving or resetting sources increments the configuration version, clears cache and immediately schedules background refreshes for all regions. Background work supplements feeds without recent stored items; `earth-feed` requests remain read-only.
|
||||
|
||||
## Regional Balancing and Async Enrichment
|
||||
|
||||
`earth_news_items` is the current-state table for EarthFeed. When no explicit `sources` filter is present, the endpoint sorts by Breaking state, region, and publish time, then applies a regional round-robin so Asia Pacific or any other high-volume source cannot fill the entire global view and cruise queue. The default region order is Americas, Europe, Middle East / Africa, Asia Pacific, then Global; unknown regions participate after the known regions. Regional views still follow the active-region-plus-global rule and do not mix unrelated regions into the regional panel.
|
||||
|
||||
Both `items` and `cruise_items` are enqueued for target-location and localization enrichment, but the frontend must not wait for AI before rendering. If the requested display locale is not ready yet, Web Earth falls back to the original `title / summary` so a card does not show a "translation pending" placeholder when readable source content already exists. After translation, classification, Breaking, or target-location enrichment completes, the same `earth_news_items` row is updated and Earth receives a news reload / patch.
|
||||
RSS refresh and content-write paths enqueue location and localization enrichment; `earth-feed` reads no longer create those tasks. The frontend renders existing content without waiting for AI, subject to the active locale’s content-availability filter. English mode does not fall back to Chinese source text. After translation, classification, Breaking, or target-location enrichment completes, the same `earth_news_items` row is updated and Earth receives a news reload / patch.
|
||||
|
||||
Target-location enrichment uses two Redis Streams queues:
|
||||
|
||||
@@ -236,7 +250,11 @@ flowchart LR
|
||||
ManualAPI --> Store
|
||||
|
||||
Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||||
NewsAPI --> Resolver["Source Resolver"]
|
||||
NewsAPI --> Config
|
||||
Earth --> RefreshAPI["POST /api/v1/news/refresh"]
|
||||
RefreshAPI --> Scheduler["RSS refresh scheduler"]
|
||||
ConfigAPI --> Scheduler
|
||||
Scheduler --> Resolver["Source Resolver"]
|
||||
Resolver --> Config
|
||||
Resolver --> Cache["Region Feed Cache"]
|
||||
Resolver --> Fetcher["RSS / Atom Fetcher"]
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Earth Render Layer Order
|
||||
|
||||
The compute topic reuses the compute-center Interactable icons, depth behavior, and layer order. Country/year changes replace that layer's data and clear prior selection. Its HUD is screen UI excluded from Earth picking in `main.js`. Regional cards are tethered in screen space; rear anchors do not draw connectors through the globe. Facility details reuse the existing card and mobile details drawer. The existing outer atmosphere shell remains, while the starfield is restored to commit `83a10a6c` (`brightStarOpacity:0.88`; no tint/brightness calibration). The HD overlay is a white `MeshBasicMaterial` with opacity 1 and `toneMapped:false`; `texture-colors.js` selects the r128 `encoding`/`LinearTosRGB` path or r160 `colorSpace`/colorspace fragment. No Phong white light, colored twilight, blue night tint, camera boost, or shoulder curve is added. No new post-processing, GPU draw pass, or high-resolution texture was added; the compute topic adds 45 local SVG flag assets.
|
||||
The compute topic reuses the compute-center Interactable icons, depth behavior, and layer order. Country/year changes replace that layer's data and clear prior selection. Its HUD is screen UI excluded from Earth picking in `main.js`. Regional cards are tethered in screen space; rear anchors do not draw connectors through the globe. Facility details reuse the existing card and mobile details drawer. The visible HD surface uses a single-draw ShaderMaterial in `earth-surface-material.js`, retaining the R+0.48 mesh, depth policy and surface pick target. The existing controller still switches all four 8K maps, updates the sampler and disposes the previous texture only after success. Normals, view direction and astronomical sunlight share world coordinates. Land is matte; water has grazing-angle cool reflection without a white specular lobe. The existing geographic land/ocean texture owns the optical mask independently of border visibility, with a conservative chroma fallback until loaded and reflection suppression over ice. Original JPEGs are neither repainted nor downsampled. Color textures are tagged sRGB with mipmaps. The r128 surface shader decodes sRGB; r160 uses hardware texture decoding. Both combine light in linear space and encode once. A local filmic curve affects only the surface instead of enabling global ACES exposure on the sky and data layers. The terminator spans normal dot products [-0.12, 0.28]; night attenuation retains 0.12–0.32 linear illumination after tone mapping to avoid crushing source shadows twice. One R×1.025 BackSide/Additive shell replaces the three rim shells, with depthWrite=false and depthTest=true; inner scattering is part of the surface pass. Independently switchable clouds use matte day/night shading at renderOrder 1.1. The sky uses the existing `starmap_deep_8k.jpg` without a tint overlay.
|
||||
|
||||
This document records the current Earth renderer's layer order and the intent of each layer. When adjusting `renderOrder`, radius offsets, depth strategy, or pointer interaction, update this document accordingly.
|
||||
|
||||
@@ -16,13 +16,13 @@ Note: the layer control panel order and the registration / startup load order ar
|
||||
| Order | Layer | Source | Render / Radius Strategy | Depth / Interaction Strategy | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| -1000 | Celestial background mesh | `celestial.js` | Background sphere | Not part of surface picking | Behind all Earth content. |
|
||||
| -1 | Earth occluder sphere | `earth.js` | Invisible inner sphere | Writes depth buffer | Occludes objects behind the Earth. |
|
||||
| 0.5 | Earth occluder sphere | `earth.js` | Invisible inner sphere | Writes depth buffer | Occludes objects behind the Earth. |
|
||||
| 0 | Earth base sphere | `earth.js` | `CONFIG.earthRadius` | Surface picking fallback target | Dark base; still visible when all optional map layers are off. |
|
||||
| 0.2 | Country dark tint | `country-boundaries.js` | `tintAltitudeOffset` | Raycast disabled | Used when HD texture is off. |
|
||||
| 0.86 | Land/ocean base fill | `country-boundaries.js` | `landAltitudeOffset = 0.32`; ocean `#010609`, land `#080f1b` | Raycast disabled | Base map remains usable even when country borders are off; radius is separated from the base sphere to avoid far-zoom z-fighting. |
|
||||
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
|
||||
| 1 | Atmospheric glow and clouds | `earth.js` | Atmosphere / cloud spheres | Not in normal object selection path | Cloud layer controlled by the "Cloud Layer" toggle. |
|
||||
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order remains `1`; cable altitude offset stays `0.2`, while landing-point anchors use the HD surface offset `0.48` | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
|
||||
| 1 / 1.1 | Atmosphere / clouds | `earth.js` | One atmosphere shell at R×1.025; clouds at R+3 | Atmosphere uses BackSide, depthWrite=false, depthTest=true; cloud toggle retained | Inner scattering shares the surface pass; no fullscreen post-processing. |
|
||||
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order remains `1`; cables and landing-point anchors share the HD surface offset `0.48` and the same triangle-surface coordinates | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
|
||||
| 1.2 | Real terrain | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` plus terrain displacement | Raycast disabled | Terrain overlays HD texture; temporarily hidden when HD texture is off, restores to prior state when re-enabled. |
|
||||
| 2.05 | Grid lines | `earth.js` | `CONFIG.earthRadius + 0.14` | Raycast disabled | Low-opacity lines over HD texture. |
|
||||
| 0.97–0.99 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
|
||||
@@ -31,17 +31,16 @@ Note: the layer control panel order and the registration / startup load order ar
|
||||
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
|
||||
| 3-4.5 | BGP collectors, event rings, and event markers | `bgp.js`, `interactable.js` | Collector markers use `BGP_COLLECTOR_RENDER_ORDER = 4.4`; event markers use `BGP_EVENT_RENDER_ORDER = 4.5`; overlays remain in BGP-owned groups | Screen-space Interactable picking; BGP collectors/events may use collision avoidance where appropriate | BGP collectors share the surface-facility band with vessels; BGP events share the compute-center band. |
|
||||
| 4.3 | AIS vessel track lines | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`; `CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | Follows vessel selection, not independently picked | Recent track for the selected vessel, below vessel markers. |
|
||||
| 4.4 | AIS vessel markers | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`; business radius `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`; normal markers are bucketed `THREE.Points`, hover / locked states are single-point overlays | `depthTest: true`; `main.js` uses screen-space picking for front-facing markers; `cluster: false` and `avoidance: false` | Moving vessels use triangular point textures, anchored / slow vessels use dots. Dense waterways may overlap and do not participate in dynamic screen clustering, avoiding Points rebuilds during rotation. |
|
||||
| 4.4 | AIS vessel markers | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`; business radius `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`; normal markers are bucketed `THREE.Points`, hover / locked states are single-point overlays; whole-anchor occlusion matches BGP and compute markers at the globe limb | `depthTest: false`, with visibility decided by anchor occlusion; `main.js` uses screen-space picking for front-facing markers; `cluster: false` and `avoidance: false` | Moving vessels use triangular point textures, anchored / slow vessels use dots. Dense waterways may overlap and do not participate in dynamic screen clustering, avoiding Points rebuilds during rotation. |
|
||||
| 4.5 | Compute centers | `compute-centers.js`, `interactable.js` | `COMPUTE_CENTER_RENDER_ORDER` | Screen-space Interactable picking, with surface-icon collision handling | Surface facilities, below satellites. |
|
||||
| 5 | Satellite background dot | `satellites.js` | Fixed renderOrder; by default TLE/SGP4 altitude is compressed to `CONFIG.earthRadius + 4..25`; with real altitude disabled or propagation failed, uses `fallbackAltitudeOffset = 8` | Screen-space satellite picking | Below satellite dots. |
|
||||
| 6 | Satellite dots | `satellites.js` | Same compressed / fallback height as satellite backdrop dots | Screen-space satellite picking | Satellite dots above footprints and compute centers. |
|
||||
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw a closed inertial orbit; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
|
||||
| 6 | Satellite dots | `satellites.js` | SGP4 altitude compressed to R+4..25; fallback R+8 | `depthTest=false`; vertex shader tests camera-to-satellite segment occlusion against an R×0.998 sphere; screen-space picking | Visible sprites remain complete; truly occluded points move outside clip space instead of being partially depth-clipped near the limb. |
|
||||
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw an open full-period inertial curve with forward-flow highlights; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
|
||||
| 98-100 | Sun / moon halo and sprite | `celestial.js` | Fixed renderOrder | Celestial picking disabled | Foreground celestial sprites. |
|
||||
|
||||
## Full-set rendering and updates
|
||||
|
||||
- Batching changes GPU submission, not the number of satellites, cable segments, or landing points. It does not filter data by the visible hemisphere. Every adjacent cable vertex pair remains present, and picking still returns the original business object.
|
||||
- Satellite foreground and backdrop remain two `Points` draws. Vertex shaders compute breathing from static per-point parameters and a shared frame-time uniform; existing hover/locked state still controls point masking.
|
||||
- Normal satellites use one hard-disc `Points` batch without a backdrop batch. Optional breathing remains in the vertex shader, defaulting off; category filters and hover/locked masking share the same visibility state.
|
||||
- `satellite-position-worker.js` uses the same Three.js / SGP4 versions as the main thread. `satellite-propagation.js` owns shared orbit, display-altitude, and fallback calculations. Full positions and initial trail samples use transferable arrays; only one calculation is in flight. Reload, clear, and altitude-mode changes terminate the previous Worker.
|
||||
- Worker startup failure or `SATELLITE_CONFIG.workerStartupTimeoutMs` expiry falls back to the shared synchronous calculation. Counts, APIs, the existing update interval, and trail length are unchanged.
|
||||
- Verify full draw ranges, per-item picking, locked overlays, trails, rear-side occlusion, visibility toggles, and resource release after clearing. Frame-time comparisons require identical data, view, and resolution.
|
||||
@@ -88,3 +87,22 @@ Maintenance rules:
|
||||
### Startup loading versus drawing order
|
||||
|
||||
The selected full-resolution surface loads before independent data requests; boundary construction no longer blocks its first frame. This does not change renderOrder, depth behavior, geometry precision or image resolution. Data refresh retains the surface, with texture disposal limited to successful replacement or scene teardown.
|
||||
|
||||
### Loading-state boundary diamond regression
|
||||
|
||||
`v0.58.0` moved land fill to R+0.32 and HD imagery to R+0.48, but left the tint placeholder at R+0.04 with a 64×32 mesh. Its chords intersected the 128×128 base, producing latitude-ring diamonds until loaded imagery covered them. The old geometry penetrated the base by approximately 0.153 at tested samples.
|
||||
|
||||
Base, tint, land fill and HD surface now use `createEarthSurfaceGeometry()` and one UV triangulation. Tint and land share R+0.32; neither writes depth, so their existing renderOrder lets loaded fill replace the tint. HD remains R+0.48. Occluder, terrain, clouds, picking and geographic display policy are unchanged. Do not mask intersecting geometry by changing renderOrder alone, disabling depth tests or hiding the loading globe.
|
||||
|
||||
`earth-surface.test.js` ray-tests the actual loading tint and verifies at least two 24-bit depth steps of separation at 50%, 100% and 500%. Browser checks cover delayed textures, boundary toggles, rotation/zoom, and land/HD completion.
|
||||
|
||||
### Surface marker alignment and limb visibility
|
||||
|
||||
Cables and landing points share R+0.48, the `earthSurfacePoint` triangle projection and renderOrder=1. Related landing halos use an emphasis attribute in the existing batch, adding no Sprite or draw call. Rounded badges and vessel heading arrows use GPU anchor ray/sphere occlusion: draw the entire billboard when its anchor is visible, otherwise clip it entirely. CPU picking uses the same test to reject rear anchors. The selected BGP event's static overlay halo uses the event icon's R+0.48 anchor, foreground order and whole-anchor front/rear visibility; the existing animated BGP event ring remains a separate effect with its original behavior.
|
||||
|
||||
|
||||
The `.map-clusters` projection layer uses z-index 8 below HUD and whole-marker occlusion from real surface anchors. Ordinary points retain their Three.js layers; transitioning members render only once in the projection layer. Spider legs use widened transparent hit areas; hover highlights the matching line and child, and both clicks resolve the same business object. Wheel input over projected markers and legs continues to zoom the globe.
|
||||
|
||||
Co-located markers share a compact projection stack across categories; screen collisions never merge different locations. A selected child uses projection `z-index: 1000`, while the locked overlay for an unclustered marker uses the shared foreground `renderOrder >= 100`, so focus stays on top across layers; clearing focus restores stable ID order. The aggregate badge stays below a selected child during its opening tween. `icon_presentation_screen` lets detail connectors follow display offsets while markers retain their real anchors. Ownership changes immediately rebuild the corresponding GPU batch.
|
||||
|
||||
Regional news and compute cards share the `#earth-region-callouts` spatial root (z-index 20), with independent data and reading state. Within that root, expanded cards use 50, dragged foreground cards 55, hover-expanded cards 60, ordinary capsules 30 and dimmed/shrunk rear capsules 8. Per-type expanded counts persist independently. Full cards occupy the globe’s flanks, while capsules follow geographic anchors; hover temporarily expands them and click changes focus. News connectors retain representative regional anchors across story rotation. One controller excludes owned cards when measuring HUD obstacles and handles size changes. Thin/bold lines share `body.dataset.calloutLineStyle`. Mobile news cards live in the drawer without map connectors.
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Admin Frontend Context
|
||||
|
||||
## Compute comparison configuration
|
||||
|
||||
The Earth content `compute_comparison` section mounts `ComputeComparisonSettings.tsx`. Structured forms maintain report sources, annual country observations and innovation gaps using the shared Panel, Input, Select, Button and confirmation dialog; operators do not edit JSON. Drafts survive section switches, full-page unload warns about unsaved work, and discarding a draft or restoring defaults requires confirmation. Management data loads on activation. Publishing sends the backend revision to prevent overwriting concurrent edits, and failures preserve the draft. The content area scrolls internally while the heading and publish controls stay in the first viewport. `compute-comparison-model.ts` owns types and same-origin update notifications; `compute-comparison.css` only owns this section's layout.
|
||||
|
||||
This document describes the current real structure of the console frontend. The goal is to help future page development, table refactoring, layout governance, and state consolidation quickly find the right entry points.
|
||||
|
||||
Related references:
|
||||
@@ -71,6 +75,8 @@ This keeps Earth, AI, collection management, and other multi-section pages from
|
||||
|
||||
## Datasource Collection Queue
|
||||
|
||||
`datasource-task-queue.ts` reconciles attempts by task ID. Retrying a source adds a new attempt and preserves failed/completed/cancelled history; source matching is allowed only for active placeholders without an ID. Late events cannot reactivate terminal attempts, and older task polls cannot overwrite the latest datasource task. Task status uses `ListChecks`; refresh retains `RefreshCw`.
|
||||
|
||||
The Admin datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
|
||||
|
||||
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
|
||||
@@ -389,7 +395,7 @@ Current boundary:
|
||||
- `TV Livestream` owns the Earth media-panel source configuration.
|
||||
- `Branding` owns Earth HUD brand assets. `Logo URL` and `Title Image URL` use inline upload controls inside the fields; the upload buttons keep the primary `TactileButton` style, and dropping an image onto the matching field shows a low-saturation drag reaction instead of the old global asset-picker toolbar.
|
||||
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
|
||||
- `Base Map`, `Layer Resources`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
|
||||
- `Base Map`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
|
||||
|
||||
Do not add Earth experience resources or collection-lifecycle tabs back into `/settings`; collection belongs to `/collection-management`, and Earth display resources belong to `/earth-content`.
|
||||
|
||||
@@ -448,3 +454,11 @@ Therefore:
|
||||
For Earth-related structure, see:
|
||||
|
||||
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
|
||||
|
||||
## Detail-card image workspace
|
||||
|
||||
`CardImageSettings.tsx` replaces the `layer_resources` placeholder in Earth Content, retaining the shared page frame, section navigation, controls and confirmation dialogs. Its persistent component preserves drafts across section changes. List and editor own their scrolling with a constrained height chain; narrow layouts stack these regions. `CardImagePreview.tsx` reuses the actual Earth mask stylesheet.
|
||||
|
||||
The image library supports upload/URL entry, attribution, licensing, enabled state and crop focus. Referenced assets cannot be deleted until assignments are removed. Assignments use database-backed object/model/group/category choices and support edit, disable, removal and explicit per-object suppression. Draft preview calls the same resolver used after publication. Only Save publishes; conflicts retain drafts, and the editing fieldset is disabled during save/upload. Uploading alone does not publish an asset.
|
||||
|
||||
Under `/earth/card-images`, configuration GET/PUT, upload POST, targets GET and preview POST require administrators. Resolve and content-addressed files are public read-only routes. Configuration writes use revision checks and audit transactions.
|
||||
|
||||
@@ -204,7 +204,7 @@ TV livestreams and boundary precision moved to `/earth-content`; collectors and
|
||||
- **TV Livestream**: manages sources shown in the Earth media panel.
|
||||
- **News Content**: browses news grouped by RSS source and manual group. RSS items remain read-only; manual groups support create, JSON import, edit, delete, and reprocess.
|
||||
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
|
||||
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
|
||||
- **Base Map**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
|
||||
|
||||
The Earth page settings gear also includes Boundary Precision. Switching to High Precision starts a local background download/build, like a game update package, when no high-precision asset exists yet. Progress is shown as a percentage, and the result applies automatically after success without a page reload. Switching back to Low Precision only changes the local display preference.
|
||||
|
||||
@@ -289,11 +289,28 @@ The bottom-left legend follows the focused or enabled layer. Covered: cables, sa
|
||||
|
||||
AIS vessel legend colors by type: cargo, tanker, passenger, fishing, military, moored/slow, other. Triangles indicate moving vessels; dots indicate moored or slow targets.
|
||||
|
||||
Click legend entries to hide or restore their route, category or visual effect; hidden entries appear dimmed. Each layer retains its filter state for the current page. Selecting another object or enabling a layer switches to its legend without a separate filter button. Hiding a selected object dismisses its details and coverage; explicitly selecting a hidden object through search reveals its category. Filtering does not change total statistics. Vessel-type and anchored/slow filters compose; geographic fill controls are visible when the base map is shown.
|
||||
|
||||
Satellite names and NORAD identifiers appear in a limited set at 400% zoom and above, with point size growing smoothly with zoom. Settings → Display → Show FPS displays actual frame rate beside the brand; hover for an approximately 30-second average. The toggle persists. The radial menu and zoom submenu follow interface DPI scaling; moving from the magnifier across the gap to the plus/minus controls keeps the submenu open.
|
||||
|
||||
### Search
|
||||
|
||||
Search finds cables, landing points, satellites, compute centers, BGP events, BGP observers. Results jump to and focus the object.
|
||||
|
||||
Click a BGP, compute, vessel, satellite, or cable object on the globe to draw a connector from the object before revealing a nearby detail card. The card follows the source direction toward the nearest usable screen edge, leaving breathing room around the globe; it never covers its own connector exit point. As the globe rotates, the source moves smoothly toward a visible part of the highlight. Fully hidden objects retain an outline direction cue with a lighter line and hollow source point. Cards stay readable and remain draggable; their connection points adjust along the edge automatically. Mobile shows a compact popup first, with full details available in the drawer. Dropdown menus and tool lists retain their existing opening behavior.
|
||||
Click a BGP, compute, vessel, satellite, or cable object on the globe to draw a connector from the object before revealing a nearby detail card. The card follows the source direction toward the nearest usable screen edge, leaving breathing room around the globe; it never covers its own connector exit point. As the globe rotates, the source moves smoothly toward a visible part of the highlight. Fully hidden objects retain an outline direction cue with a lighter line and hollow source point. Cards follow the connector direction smoothly and remain draggable; their connection points adjust along the edge automatically. Mobile shows a compact popup first, with full details available in the drawer. Dropdown menus and tool lists retain their existing opening behavior.
|
||||
|
||||
### Four regional news cards
|
||||
|
||||
Clicking refresh shows a refreshing state while existing news stays readable.
|
||||
Success is shown after the latest results are displayed. If refresh fails, the
|
||||
page explains the failure and enables the button for another attempt. Repeated
|
||||
clicks during the wait do not start duplicate refreshes.
|
||||
|
||||
Open the news panel from the top ticker, then choose “Show regional news cards” in its header. Four cards cover the Americas, Europe, Middle East & Africa, and Asia-Pacific. They remain visible after the panel closes, and the switch is remembered. Its background and border highlight while enabled; click again to turn it off.
|
||||
|
||||
Each card rotates only displayable stories from its own region after its typewriter summary finishes and a seven-second reading interval elapses. Hovering, keyboard interaction, dragging or collapsing pauses that card while other regions continue. Controls provide previous/next, pause/resume and a source link. Refresh preserves the current story where possible. Empty regions stay empty instead of borrowing unrelated stories.
|
||||
|
||||
Cards reuse the cruise-news gold header and 📰 icon, preserve full summaries and simplify source metadata, with the compute cards' dragging, collapsing and automatic placement. News and compute cards share one placement plan. Connectors identify representative regional anchors, not precise event locations. On mobile, enable “Region cards” from the news page and read them in the drawer.
|
||||
|
||||
### Choosing a News Live Stream
|
||||
|
||||
@@ -323,7 +340,7 @@ The batch continues within the current Earth page when you close the candidate p
|
||||
|
||||
### Settings
|
||||
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only / recognized-gesture whitelist, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, day-night mode, panel toggles), motion debug mode / input source / skeleton-only / recognized-gesture whitelist, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
|
||||
News categories use the same chip selector as Cruise Modules. They only filter the news panel and news cruise items in the current browser; they do not affect layers, TV, data points, basemap, boundaries, collector jobs, or admin news-source configuration.
|
||||
|
||||
@@ -412,12 +429,48 @@ Docs supports: category navigation, Markdown rendering, tables and code blocks,
|
||||
|
||||
## Compute intelligence
|
||||
|
||||
Open the analysis icon beside Compute Centers, then enable Show region cards. Connectors draw to their target positions before the cards fade in; subsequent globe motion does not replay entrance. Closing the configuration window or pressing Escape leaves the cards enabled; the analysis icon reopens configuration, while the switch disables the cards. Anchors rotate with Earth while cards remain readable. Dragging or expanding grants reading time; after a substantial change in geographic direction, cards can move together when that materially reduces crossing or detours. Active interaction prevents movement, while manual positions remain preferences rather than permanent pins. Rear regions retain their state without a connector drawn through the globe. Existing layer, news, and media windows remain available.
|
||||
Administrators maintain card data in **Earth content → Compute comparison**. Add a report title, URL, publisher, report year and page/figure locator under Sources, then add country values by observation year, metric and source under Observations. Save and publish applies the complete configuration. Global shares use the world denominator; absolute values use FP32 EFlop/s. Bilateral shares are derived from same-year US and China observations, and regions require every member. Do not mix FP16/FP64 capacity, regional estimates or facility counts. Omit missing countries; enter zero only when the source explicitly reports zero.
|
||||
|
||||
Innovation gap observations remain independent US-minus-China values. Referenced reports cannot be removed until their observations are removed or reassigned. Duplicate metric/year rows, missing sources and global shares totaling over 100% are rejected. Concurrent edits return a conflict while preserving the local draft. Restore defaults requires confirmation and restores the bundled report data. Publishing and restoring are audited. Same-origin Earth pages receive a refresh notification; other visible pages check once per minute and hidden pages check when visible again. Reports must still be reviewed manually; configuring a URL does not scrape a report.
|
||||
|
||||
Open the analysis icon beside Compute Centers, then click **Show region cards** to the left of the close button in the title bar. Connectors draw to their target positions before the cards fade in; subsequent globe motion does not replay entrance. Closing the configuration window or pressing Escape leaves the cards enabled; the analysis icon reopens the panel. Click the highlighted card icon to hide the cards. Anchors rotate with Earth while cards remain readable. Dragging or expanding grants reading time; after a substantial change in geographic direction, cards can move together when that materially reduces crossing or detours. Active interaction prevents movement, while manual positions remain preferences rather than permanent pins. Rear regions retain their state without a connector drawn through the globe. Existing layer, news, and media windows remain available.
|
||||
|
||||
The picker groups the canonical comparison catalog as China, US, Europe, APAC excluding China, and Other; the groups are display buckets, not statistical denominators. It offers period and metric selection. Bilateral share still uses US + China as its fixed denominator; unavailable measures stay missing for other participants. Regional anchors are label positions, never facility locations. Source buttons open report evidence and facility buttons reuse the existing detail card. Mobile uses a compact regional overview in the drawer. Each row is an accordion: tapping the same header collapses it, tapping another collapses the old row, and at most one long card is open. The expanded card keeps trends, region members, facilities, media, and sources; there is no region subpage. Settings remain a separate on-demand page.
|
||||
|
||||
The default national metric is global compute-device scale, covering basic, intelligent, and supercomputer capacity with observations from 2020–2023. The reviewed absolute observations are 302 for 2022 (`caict2023` evidence) and 435 for 2023 (`caict2024` evidence). Japan's global-share observations are 6%/5%/4%/4% for 2020/2021/2022/2023. Product groups APAC and OTHER do not mean the report's Japan or “other” categories, and EUROPE remains missing until its membership matches the report. Bilateral share always uses US + China as its denominator. A missing primary value may show the latest reported year (currently 2023); explicitly selecting 2026 still shows missing and does not backfill. Unconfirmed comparability does not produce year-over-year change. National report metrics remain separate from public facility samples; coverage counts remain auxiliary evidence, never computed capacity or a global coverage rate. TOP500 and Epoch may overlap, so source-record counts are not a global census. API review metadata is `reviewed_at=2026-09-22` with the matching version date. An independent Epoch share mode is not implemented.
|
||||
The default national metric is global compute-device scale, covering basic, intelligent, and supercomputer capacity with observations from 2020–2023. The reviewed absolute observations are 302 for 2022 (`caict2023` evidence) and 435 for 2023 (`caict2024` evidence). Japan's global-share observations are 6%/5%/4%/4% for 2020/2021/2022/2023. Product groups APAC and OTHER do not mean the report's Japan or “other” categories, and EUROPE remains missing until its membership matches the report. Bilateral share always uses US + China as its denominator. A missing primary value may show the latest reported year (currently 2023); explicitly selecting 2026 still shows missing and does not backfill. Unconfirmed comparability does not produce year-over-year change. National report metrics remain separate from public facility samples; coverage counts remain auxiliary evidence, never computed capacity or a global coverage rate. TOP500 and Epoch may overlap, so source-record counts are not a global census. The bundled reports were reviewed on 2026-09-22; published overrides use their maintained review date and observations. An independent Epoch share mode is not implemented.
|
||||
|
||||
This prototype shows up to six content cards. The main card centers on the selected compute metric: a large share percentage or absolute value with its unit, observation year, historical context, and source entry. Missing comparable data is shown as “No comparable data yet”; a trend is omitted when there are no observations rather than rendered as an empty chart. Facility records and coverage metadata remain complete but move to the expanded auxiliary area; sample counts are evidence context, never a compute value or global coverage rate. Region previews may show member counts, and long lists open only when needed. Desktop compact cards are about 268px wide with content-driven height; expanded cards support user resizing up to 480px. Mobile keeps the same single metric meaning in the one-card accordion rather than promoting facility counts. Local card position and size are saved, with a reset-to-auto action. The content layout leaves room for later image and video integration, but those media data are not connected yet. When outside space is insufficient, cards can travel through the globe's foreground instead of teleporting or shrinking to labels.
|
||||
Select up to six compute participants sharing peripheral space with four regional news cards. Settings → Panels independently sets the expanded compute and news card counts to 1, 2 or 3, defaulting to one each and retaining the choice. Other cards become capsules; hover temporarily opens the full content and click changes the focused region. Full cards retain metrics, years, trends and sources, with explicit missing values and internal scrolling. Expanded cards occupy the globe’s flanks while capsules follow regional projections and dim on the rear side. Dragging retains a manual position until the region moves deep behind the globe. Settings → Display → Connector Style selects thin or bold lines.
|
||||
|
||||
When the map cards switch is off, the overview summary still loads. Turning it on shares the map loading event; a newer request cancels the previous one. Drawer event handling follows `composedPath()` so replacing inner HTML cannot accidentally collapse the active row. The shared HUD collapsible panel animation is reused and respects reduced-motion preferences. Facility names remain in a scroll region capped at 240px; rows are left-aligned, wrap long names, and retain at least 42px of touch height.
|
||||
|
||||
## Surface and object appearance
|
||||
|
||||
The real surface retains all four map choices. Day/night shading follows the sun, with uniform illumination when disabled. Selected cables show traveling pulses and landing points use amber micro-markers. Facilities retain category symbols, with numbered solid circles for aggregates. Satellites retain self-glow/ground-coverage modes, trails and their existing orbital motion. BGP severity, vessel heading, related highlights, detail cards and mobile drawers retain their established interactions.
|
||||
|
||||
### Satellite orbits and collection attempts
|
||||
|
||||
Selecting a satellite shows an open predicted arc with highlights moving in the travel direction. Satellites, trails and footprints remain synchronized. Coverage stays blue across day/night; particle and orbit category colors match the legend. Dragging continues outside the globe silhouette, with reduced sensitivity at extreme zoom. The TV panel has square corners like other cards.
|
||||
|
||||
Retriggering a datasource creates a new task while older failures remain in the failed group. Task Status uses a checklist icon, distinct from Refresh. CelesTrak can reuse valid recent download cache; each new collection attempt still gets its own task and updates collected data.
|
||||
|
||||
|
||||
Display → Satellites → Close-up labels defaults on; turning it off hides satellite map names while retaining detail cards. The zoom-threshold slider below the switch ranges from 250% to 500% (default 400%); labels appear at or above the selected zoom and hide below it. The count slider sets a saved limit of 5–36 labels (default 36); occlusion or limited space may result in fewer visible labels. Sky brightness ranges from 0–200%, defaulting to 100%. Both choices persist. Compute region cards wait for drag/globe motion and new-card placement to settle before necessary avoidance. Otherwise they float subtly around stable positions and pause during interaction. Compute legend category counts describe currently located markers.
|
||||
|
||||
|
||||
Surface aggregates appear as solid numbered circles. Click to expand members; only co-located or extremely close geographic markers form compact stacks at their location, with exposed clickable edges. Ordinary markers stay at their real projected positions. Selecting a rear icon brings it forward and opens its details; clearing focus restores its original position. Ordinary groups split above 250%; no aggregate badge remains at 500%, and co-located markers expand directly.
|
||||
|
||||
Object detail cards show identity, business properties and update times without rendering diagnostics, collection channels or coordinate-matching internals. Facility location precision and verification status, BGP alert confidence, and news publisher, publication time and summary remain visible.
|
||||
|
||||
Detail popups share the news-card height cap. Headers remain fixed while long bodies scroll; key event information and device specifications precede location verification, provenance and related objects.
|
||||
|
||||
### Detail-card images
|
||||
|
||||
Open Earth Content → Layer resources → Detail card images. Upload a still JPEG, PNG or WebP, or enter an image URL. Set its title, source, credit, license and crop focus; the preview uses the actual feathered card background.
|
||||
|
||||
In Assignments, select satellites, compute facilities, cables, landing stations, vessels, BGP incidents, BGP collectors and custom points, then an object/model/group/category. Search selects current catalog objects and stable identifiers; missing model data does not produce invented choices. Apply an assignment to the draft, then Save to publish. Object rules take precedence over model, group and category rules. “No image for this object” suppresses all fallback images for that object.
|
||||
|
||||
Verify an object to see its draft match before publishing. Disabled assignments allow more general rules to match. Remove assignments before deleting an image. Conflicting saves preserve your draft.
|
||||
|
||||
Earth fades available imagery behind the existing detail data. Missing or failed images leave a complete data-only card. The header image icon links to the source and exposes attribution/license in its tooltip. Every map object type has a bundled default: exact and series images take precedence over explicitly labeled type illustrations. Starlink, BeiDou-3 and Qianfan are assigned; Qianfan retains its unverified permission metadata. Fresh installs need no setup. Upgrades merge missing defaults after services restart while preserving custom assignments and explicit suppression.
|
||||
|
||||
Compute legends and details support supercomputers, GPU clusters, data centers, cloud computing centers and edge nodes. Manage their defaults under Compute facilities → Category in image management. Types without collected records show a zero count. BGP collectors and incidents have separate default imagery.
|
||||
|
||||
@@ -1,6 +1,20 @@
|
||||
# `planet.sh` Startup Performance Optimization
|
||||
|
||||
## Background
|
||||
## Windows desktop launcher
|
||||
|
||||
The launcher prefers a new Windows Terminal window and explicitly runs PowerShell 7 (`pwsh.exe`), independently of the default Terminal profile. Without PowerShell 7 it falls back to Windows PowerShell; without Terminal it starts the available PowerShell directly. The CMD bootstrap also prefers PowerShell 7. Ordinary entries request elevation; entries already running as administrator still dispatch to the preferred terminal without another elevation request. The child validates its privileges and enters WSL directly, preventing recursive windows. The existing post-restart WSL shell is retained.
|
||||
|
||||
On first use, open the actual project folder in Windows Explorer and double-click the repository's `planet.cmd`. It derives the WSL distribution and Linux project directory from its own location, creates `Planet.cmd` on the current user's actual Windows desktop, then follows the normal restart flow. Both `\\wsl$\distribution\...` and `\\wsl.localhost\distribution\...` work. Local Windows-drive checkouts use the default WSL distribution and `wslpath`. Windows resolves the desktop location, including redirected desktops such as OneDrive; WSL, zsh and project dependencies must already be installed.
|
||||
|
||||
Subsequent desktop launches use the bound target, request the existing administrator elevation and call `planet.sh restart --allow-lan`. No test ports are embedded: defaults remain frontend 3000, backend 8000 and AI Provider 8010, subject to explicit `DEFAULT_*_PORT` environment overrides. Successful restarts open Earth using the actual `ports.env` values and retain the WSL shell; failures display their error without opening a success page. `scripts/windows/planet-launcher.ps1` owns path resolution, desktop installation and elevation; `scripts/windows/restart.zsh` only bridges to the existing lifecycle script.
|
||||
|
||||
Use `planet.cmd --install-only` on Windows to generate or validate a launcher without restarting services. Re-running the repository entry updates the same target's launcher. Other projects and unmanaged files are preserved by choosing a target-specific filename. After moving a checkout, regenerate from its new location; old launchers never guess another directory. Desktop launchers require the scripts in the checkout and are not standalone installation packages.
|
||||
|
||||
Do not copy the raw repository `planet.cmd` to the desktop by itself: first installation needs the adjacent `scripts/windows/` directory. On another computer, generate a new launcher from that computer's checkout instead of reusing an old desktop copy. Server paths such as `/home/linkong/planet` and personal workstation paths both work without machine-specific Git edits. Run under the Windows account that owns the WSL distribution; elevating under a different account may expose a different set of registered distributions. Installation itself does not require administrator privileges; elevation is requested for the actual restart.
|
||||
|
||||
Run Windows regression checks with `powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`. They cover target resolution, special characters, generated launchers and name collisions. `scripts/harness/test_windows_restart.py` uses disposable command substitutes to check restart arguments, browser opening after success and shell retention after failure. The latter runs in quick-check without starting real services.
|
||||
|
||||
## Startup background
|
||||
|
||||
Use `zsh ./planet.sh start --non-motion-agent` for daily startup; use `init` when preparing a new environment. When investigating latency, distinguish initial dependency downloads, container readiness, and application initialization using the stage timestamps.
|
||||
|
||||
|
||||
@@ -311,6 +311,8 @@ For each newly confirmed cause, update both language tables with a stable code,
|
||||
<!-- planet-error-catalog:start -->
|
||||
| Error code | Matching fragments (semicolon-separated) | Cause / established failure | Remedy |
|
||||
| --- | --- | --- | --- |
|
||||
| P_CELESTRAK_NOT_UPDATED | GP data has not updated since your last successful | CelesTrak rejected a repeated download within an update window. HEAD 200 followed by GET 403 can mean preflight consumed the allowance, not an unreachable egress route. | Use the collector without HEAD preflight and preserve validated raw cache. Without valid cache, wait for the next GP update or configure Space-Track instead of retrying repeatedly. |
|
||||
| P_WINDOWS_LAUNCHER_TARGET_MISSING | P_WINDOWS_LAUNCHER_TARGET_MISSING | The bound WSL project is unavailable or lacks launcher scripts, commonly after moving a checkout, losing access to a distribution or copying another machine's desktop launcher. | Run the repository planet.cmd from the current checkout to regenerate this machine's desktop launcher and verify the WSL distribution; do not hardcode another machine's path. |
|
||||
| P_PROXY_EXTERNAL | PLANET_PROXY_EXTERNAL | Docker proxy settings belong to another configuration source or have been edited manually. | Inspect daemon.json, systemd proxy settings and the Planet ownership record before changing ownership; existing settings are preserved. |
|
||||
| P_AUTH_BOOTSTRAP_MISSING | DEFAULT_LOGIN_USER_MISSING | The active backend database is confirmed to lack default accounts, often in an isolated preview that skipped initialization. | Verify database ownership and run the existing account initializer without resetting existing passwords or clearing data. |
|
||||
| P_AUTH_PROFILE_INVALID | AUTH_IDENTITY_RESPONSE_INVALID | Login succeeds but the identity response lacks required fields or incorrectly rejects a stored local email address. | Load the complete persisted identity, separate response serialization from registration validation, and run default-account identity regressions. |
|
||||
@@ -487,3 +489,51 @@ The old `frontend/public/earth/assets/8k_earth_daymap.jpg` path remains but now
|
||||
High-precision boundary outputs live in a Git-ignored generated directory. Pulling does not deliver them automatically. Outputs lacking the required new policy metadata are rejected in favor of the shipped policy fallback; regenerate and publish the static artifacts through the existing boundary workflow to retain high precision. This fallback does not reduce basemap resolution.
|
||||
|
||||
Run `PLANET_HARNESS_DOCKER_SMOKE=1 .venv/bin/python scripts/harness/test_backend_runtime_assets.py` to exercise the actual Docker context and production COPY rules, verify the shared policy/geometry and builder imports, and exclude environment files. This probe does not start the full application stack.
|
||||
|
||||
## Detail-card image storage
|
||||
|
||||
Bundled originals and their manifest live under `frontend/public/earth/assets/cards/`, with card-sized derivatives in `previews/`. After changing source assets, run `uv run --frozen --project . python scripts/prepare_card_images.py`, verify source hashes and permissions, then commit originals, manifest and generated WebP files. Preview generation follows the manifest enabled flag; permission records are retained unchanged. Backend images copy the same manifest while frontend builds distribute the files; deploy both images from the same revision.
|
||||
|
||||
Uploads default to the repository-root `data/earth-card-images/`; `PLANET_CARD_IMAGE_DIR` can select an absolute directory. Production containers need a persistent mount, shared across backend replicas. Back up that directory together with the PostgreSQL `earth_card_images` system setting. Database backups alone cannot restore uploaded bytes. Original and 640/1280 WebP files use content-addressed URLs under `/api/v1/earth/card-images/files/`, already covered by the API proxy. Removing configuration does not immediately delete physical files while older clients may still use them.
|
||||
|
||||
## Database cache and historical event maintenance
|
||||
|
||||
Consumed Outbox events default to one day of retention, configurable through
|
||||
`PLANET_EARTH_OUTBOX_RETENTION_DAYS`. Once its maintenance index is valid, the
|
||||
listener deletes at most 1,000 expired consumed rows per minute. Run initial
|
||||
maintenance in a maintenance window. The first command only inspects state,
|
||||
while `--apply` enables changes.
|
||||
|
||||
```bash
|
||||
uv run --frozen python scripts/maintain_earth_outbox.py
|
||||
uv run --frozen python scripts/maintain_earth_outbox.py --apply
|
||||
uv run --frozen python scripts/install_bgp_prefix_geo_indexes.py
|
||||
```
|
||||
|
||||
For a severely bloated historical Outbox, use
|
||||
`scripts/maintain_earth_outbox.py --compact --apply` in a maintenance window.
|
||||
It exclusively locks the table, stages pending and retained events, refills the
|
||||
original table transactionally, verifies total and pending counts, and then
|
||||
maintains the index. Other access blocks during the scan and refill. Reserve
|
||||
temporary disk space and pause backend hot reload to prevent startup DDL from
|
||||
conflicting with concurrent index installation. Check transaction outcomes and
|
||||
invalid indexes before retrying a failed operation. Measure reclaimed database
|
||||
space with `pg_total_relation_size`. Linux page cache has a separate lifecycle.
|
||||
|
||||
The WSL memory budget is configured in `%USERPROFILE%\.wslconfig`. The following
|
||||
configuration was validated on a 32 GiB host. Preserve existing network settings,
|
||||
restart WSL, restore services, and check `free -m`, Windows available memory,
|
||||
and application load.
|
||||
|
||||
```ini
|
||||
[wsl2]
|
||||
memory=8GB
|
||||
swap=4GB
|
||||
|
||||
[experimental]
|
||||
autoMemoryReclaim=gradual
|
||||
hostAddressLoopback=true
|
||||
```
|
||||
|
||||
`hostAddressLoopback` allows Windows to reach mirrored WSL services through an
|
||||
IPv4 address assigned to the host, enabling local checks of the LAN address.
|
||||
|
||||
@@ -57,6 +57,12 @@ Once in, verify:
|
||||
- The settings panel can switch rotate / cruise / motion modes; motion settings can select the input source and allowed gestures; view settings can switch hover tooltip content, and satellite settings can toggle real-altitude layering and track display
|
||||
- The base map defaults to Original basemap, listed first; the display menu can switch and save the map source. Natural Earth I remains available, NASA 21K is source archive material only, and runtime uses one 8K texture at a time.
|
||||
|
||||
After selecting an object or enabling a layer, click its legend entries to filter directly. Display settings can enable FPS; satellite names appear at zoom levels of 400% or above.
|
||||
|
||||
Open the news panel from the top ticker and enable “Show regional news cards” to rotate separate queues for the Americas, Europe, Middle East & Africa and Asia-Pacific. Each card supports independent pause, navigation, dragging and collapsing.
|
||||
|
||||
Refresh keeps existing news readable until the latest results are displayed, then reports success. If it fails, follow the message and retry. Settings → Panels independently expands 1–3 compute and news cards; other regions remain available as capsules that open on hover.
|
||||
|
||||
## 5. Recover a Lost Password
|
||||
|
||||
Open `/forgot-password`, enter your email, receive a code, then enter the code plus a new password. The same confirmation is shown for unknown emails (to avoid enumeration).
|
||||
@@ -74,4 +80,14 @@ Open `/forgot-password`, enter your email, receive a code, then enter the code p
|
||||
|
||||
## Try compute intelligence
|
||||
|
||||
Open the analysis icon beside Compute Centers and enable Show region cards. Closing configuration keeps the cards active; the switch controls their visibility. Resize or collapse desktop cards and inspect observed history, sources, and facility details. On mobile, use the Compute drawer tab. National indicators and facility samples are separate. “—” means missing evidence, not zero capacity.
|
||||
To maintain card values, an administrator opens **Earth content → Compute comparison**, adds report sources, enters comparable annual country observations, then chooses **Save and publish**. Omit missing countries; facility counts never replace compute shares. Bundled defaults can be restored and Earth cards automatically check published configuration.
|
||||
|
||||
Open the analysis icon beside Compute Centers. In the title bar, click **Show region cards** to the left of the close button. The cards stay visible when you close the panel. Click the same icon again to hide them. Resize or collapse desktop cards and inspect observed history, sources, and facility details. On mobile, use the Compute drawer tab. National indicators and facility samples are separate. “—” means missing evidence, not zero capacity.
|
||||
|
||||
### Inspect satellites and retry collection
|
||||
|
||||
Enable satellites in Earth and select a particle to inspect its open predicted orbit, travel-direction highlights and supported footprint. Dragging continues beyond the globe silhouette. A failed datasource can be triggered again: the new task is separate, and older failures remain in the current queue’s failed group.
|
||||
|
||||
Regional cards remain around the globe while connectors follow rotation. Full cards remain visible when space is limited. Valid drops retain their new location; central or out-of-bounds drops settle nearby automatically. Detail cards retain business information and essential reliability context.
|
||||
|
||||
Manage detail-card imagery under Earth Content → Layer resources: upload an image, assign an object or group, preview and save. Bundled exact, series and type illustrations cover all map object types. Fresh installs need no import; pull updates and restart both services to merge missing defaults. Custom assignments and explicit suppression remain intact.
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
- [智能星球卫星覆盖策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-satellite-footprint-policy.md):卫星 footprint 的显示边界和策略
|
||||
- [BGP 态势上下文](/home/ray/dev/linkong/planet/docs/technical/zh/earth-bgp-context.md):BGP 在智能星球中的渲染、聚合和观测站实现
|
||||
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md):`Interactable` 的接口、生命周期和接入示例
|
||||
- [智能星球可交互图标聚类策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-clustering.md):可插拔 cluster strategy、稳定球面聚类和动态屏幕聚类的适用边界
|
||||
- [智能星球可交互图标聚类策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-clustering.md):固定地理聚合、时间补间、Spiderfy 放射展开及交互验收
|
||||
- [智能星球工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索、设置、新闻、图层浮层的关闭矩阵
|
||||
- [智能星球新闻源配置](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-sources.md):默认新闻源、Feed 子项、源属性标签、内容类型、重要度规则和配置接口
|
||||
|
||||
|
||||
@@ -200,6 +200,10 @@ def parse_response(self, html):
|
||||
|
||||
## 七、调度机制
|
||||
|
||||
Earth RSS 补抓使用独立的 `earth_news_rss_refresh` 系统作业,在启动时及每
|
||||
10 分钟检查区域 Feed。首页 GET 读取已保存新闻。同步数据源作业时保留该系统
|
||||
作业,具体新鲜度、锁和失败冷却见[新闻源配置](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-sources.md)。
|
||||
|
||||
```python
|
||||
# 启动时注册所有采集器到定时任务
|
||||
def start_scheduler():
|
||||
@@ -498,6 +502,20 @@ Epoch 改用官方公开 CSV,保留 FP16 理论性能、原始日期、状态
|
||||
|
||||
两个采集器不再生成示例回退。空原文、转换后空结果、结构错误或相对上次快照减少超过 40% 会失败并保留旧快照;写入失败回滚不完整记录。TOP500 身份切换仅在旧位置名称、规范国家完全一致且唯一时复制位置到新系统 ID,旧位置保留用于历史查询,绝不按名次迁移。
|
||||
|
||||
`compute_history.py` 保存从已核对研究计划受控导入的年度数值、来源和版本,参考期与发表时间分开;未知具体发表日保持未提供。`/api/v1/data-products/compute/series` 支持全球份额、固定中美分母的双边份额及已知 FP32 绝对量。创新分差独立返回,不反推单国指数。
|
||||
`compute_history_defaults.json` 提供原有报告数据,`compute_configuration.py` 从 `system_settings` 的 `compute_comparison` 分类读取完整数据库覆盖;没有记录时使用默认值,不需要迁移或人工初始化。`compute_history.py` 只处理同年聚合,参考期与发表时间分开;未知具体发表日保持未提供。`/api/v1/data-products/compute/series` 支持全球份额、固定中美分母的双边份额及 FP32 绝对量,逐次读取配置并返回 `Cache-Control: no-store`。创新分差独立返回,不反推单国指数。空配置保留缺失,不回填默认观测。
|
||||
|
||||
同前缀的 `GET/PUT /configuration` 和 `POST /configuration/reset` 仅允许 admin/super_admin。写入体为 `{revision, config}`,重置只需要 `revision`。Pydantic 校验国家目录、有限数值、比例总和、重复指标年份和报告引用;每个指标年份只允许一个报告口径。PostgreSQL 唯一分类与行锁串行化首次写入和更新,旧版本返回 409。审计记录和配置在同一事务提交;重置移除覆盖,下次读取随版本默认值。`version` 为配置修订标识,`reviewed_at` 为录入的核验日期,不能将管理员配置一概称为官方报告。来源链接只作引用,不触发抓取。
|
||||
|
||||
`/api/v1/data-products/compute/facilities` 仅读取 `compute_contract=v1` 的成功快照,数据库执行国家过滤和按来源计数。Epoch 默认排除规划、退役、已被替代和疑似重复记录;TOP500 表示榜单观测,跨源仍未合并。历史选择按采集时间截止,不将最近抓到的旧资料冒充过去已知的快照。地理结果有返回上限与截断标记,记录统计不受该上限影响。官方 site helper 按精确 `source` + `source_id` 匹配;access map 中心只表示 site 级参考点,并在 `source_note` 保留官方 URL。
|
||||
|
||||
|
||||
### CelesTrak 下载窗口与缓存
|
||||
|
||||
CelesTrak active 和 fallback group 均禁用 HEAD 预检,直接 GET,避免预检被上游记为下载后使正文请求收到 403。完整有效缓存两小时内复用;到期后进行条件 GET(有 ETag/Last-Modified 时)或普通 GET。304 复用已校验缓存;仅明确的“GP data has not updated”403 可以回退到有效缓存,其他 403 不伪装成功。fallback group 同样检查缓存时效,不永久使用旧文件。
|
||||
|
||||
下载在 `.part` 上校验非空 GP 记录列表后原子替换缓存,失败保留上一份有效文件。内容长度以 GET 响应为准;压缩响应不拿压缩长度校验解压后的字节数。其他采集器继续支持 HEAD 元数据和 Range 断点续传。原始缓存不等于已入库数据,恢复缓存后仍须触发新采集任务。
|
||||
|
||||
|
||||
### Earth 新闻编码
|
||||
|
||||
`earth_news.py` 在解析 RSS/Atom 前按 BOM、HTTP charset、XML encoding 声明和默认 UTF-8 的顺序确定编码,并严格解码。GBK/GB2312/GB18030 源不会再被隐式替换为乱码;未知编码或非法字节标记为 `format_error`,不写入损坏标题。`test_earth_news_encoding.py` 覆盖中文标题、摘要、展示序列化与失败状态。
|
||||
|
||||
@@ -31,8 +31,16 @@ flowchart LR
|
||||
5. listener 短窗口合并同 layer 事件,失效缓存并广播 `earth_updates`。
|
||||
6. 广播成功后标记 outbox consumed;失败时保留待重试。
|
||||
|
||||
缓存重建按 key 合并。同进程的并发请求共享构建结果或错误,冷缓存的跨进程请求等待 Redis 锁持有者完成。长构建会续租,发布和释放操作校验锁持有者。持有请求取消时,等待者可使用自己的数据库 session 接手。可用的 stale 缓存会直接返回;没有 stale 时,等待超过默认 30 秒返回 HTTP 503。Redis 协调失败时返回本次构建结果,同进程并发仍会合并。构建失败优先返回 stale,否则传播错误。
|
||||
|
||||
DB 变化不再默认创建 `earth_refresh` 任务,因此不会被同 source 的长采集或删除任务阻塞。`earth_refresh` 只保留给手动清缓存和非 DB 变化刷新提示。
|
||||
|
||||
### Outbox 写入与保留
|
||||
|
||||
触发器先从序列取得 `event_id`,再将含该 ID 的 payload 单次写入 `earth_data_change_events` 并发送 `NOTIFY`,避免插入后再次更新同一事件。`earth_news_items` 仅更新 `last_seen_at`、`updated_at` 时不产生 Earth 变化事件;标题等可见内容变化仍会产生事件。这两个时间戳继续记录新闻来源的健康与新鲜度。
|
||||
|
||||
已消费事件默认保留 1 天,可用 `PLANET_EARTH_OUTBOX_RETENTION_DAYS` 调整。listener 每 60 秒最多清理 1000 条过期已消费事件,且仅在独立维护索引有效时运行;未消费事件不进入清理条件。多实例通过 PostgreSQL advisory lock 避免同时清理。索引由[独立 Outbox 维护脚本](/home/ray/dev/linkong/planet/scripts/maintain_earth_outbox.py)以 `CREATE INDEX CONCURRENTLY` 建立,不在应用启动事务中创建。该脚本默认只读,也提供分批清理和一次性 compact 模式;具体执行步骤见[运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md)。
|
||||
|
||||
## 数据作业队列
|
||||
|
||||
`collection_tasks` 是统一 job ledger。worker 使用 PostgreSQL `FOR UPDATE SKIP LOCKED` 领取 `queued` 任务;同一 `source` 的写任务串行,不同 source 可并行。
|
||||
|
||||
@@ -323,7 +323,7 @@ AISStream 使用 WebSocket 实时流,采集器只写入 `ais_raw_observations`
|
||||
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
|
||||
```
|
||||
|
||||
该接口查询 `vessel_current_state` 当前状态表,只返回有效窗口内每个 MMSI 的最新点;原始 `ais_raw_observations` 继续保留给轨迹、审计和态势分析,但不再由展示接口临时扫描聚合。Earth 前端统一传全球 bbox,不随当前镜头视口反复请求。实时更新使用 `/ws` 的 `vessels` 全局订阅,按 MMSI 推送确认状态并原位更新;超出单帧上限时拆包,保留所有更新。首次加载、重连和删除校准仍使用快照,详见[采集器架构](backend-collectors.md)。
|
||||
该接口查询 `vessel_current_state` 当前状态表,只返回有效窗口内每个 MMSI 的最新点;原始 `ais_raw_observations` 继续保留给轨迹、审计和态势分析,但不再由展示接口临时扫描聚合。Earth 前端统一传全球 bbox,不随当前镜头视口反复请求。实时更新使用 `/ws` 的 `vessels` 全局订阅,按 MMSI 推送确认状态并原位更新;超出单帧上限时拆包,保留所有更新。首次加载、重连和删除校准仍使用快照,详见[采集器架构](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)。
|
||||
|
||||
## 自定义 REST / WebSocket 映射运行时
|
||||
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
# BGP 态势上下文
|
||||
|
||||
## 网段归属查询
|
||||
|
||||
前缀归属按 OpenGeoFeed、IPtoASN、NRO 顺序查找,并校验 IPv4/IPv6 地址族及
|
||||
完整起止区间包含关系。来源内仍按原始 prefix 掩码长度和记录 ID 排序。
|
||||
`planet_bgp_try_inet` 和 `planet_bgp_try_masklen` 将无效历史地址转换为 NULL。
|
||||
数据库初始化安装这些轻量函数,区间联合 GiST 索引由独立维护脚本部署。
|
||||
索引安装及验证见[运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md)。
|
||||
|
||||
## 当前目标
|
||||
|
||||
BGP 模块正在从一个只展示异常的演示功能,演进为分层观测管线:
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
# 智能星球前端结构
|
||||
|
||||
## 高清地表与大气材质
|
||||
|
||||
高清地表由 `earth-surface-material.js` 的单次绘制 `ShaderMaterial` 负责,保持原有 R+0.48 网格、深度策略和地表拾取对象。四张 8K 底图仍由原控制器切换,成功后更新采样 uniform 并释放旧纹理,失败保留原图。法线、视线和真实太阳方向统一到世界坐标;土地哑光,海洋只有掠射角冷色反射,没有白色镜面高光。海洋区分优先复用现有海陆地理纹理,未加载时使用保守色差估计,冰雪抑制反射;关闭国界线不移除光学遮罩。原始 JPEG 不重绘、不降采样。
|
||||
|
||||
颜色纹理标记 sRGB 并启用 mipmaps。r128 在地表 Shader 中解码 sRGB,r160 由纹理硬件格式解码;二者在线性空间合成光照后只编码一次。局部 filmic 曲线只作用于地表,不打开全局 ACES 曝光,避免改变星空和业务图层。晨昏范围为法线点积 [-0.075, 0.16],夜侧单独计算地理纹理透光与冷色环境漫射:对局部色调映射后的纹理用 0.55 次幂保留暗部层次,再分别应用陆地冷灰绿、海洋深靛蓝的纹理系数和极弱环境底色;最后在线性空间按晨昏系数混合昼夜并统一输出 sRGB。日面曝光和反射不变,不添加虚构城市灯光。大气收敛为 R×1.025 的一个 BackSide/Additive 壳,depthWrite=false、depthTest=true,内沿散射并入地表;云图独立开关,使用哑光昼夜材质并在 1.1 顺序绘制。星图使用已有 `starmap_deep_8k.jpg`,不加染色遮罩。
|
||||
|
||||
## 地表贴图资源记录
|
||||
|
||||
2026-09-21,`assets/8k_earth_daymap.jpg` 替换为 Natural Earth I 配套栅格的 8192×4096 Web 派生图,加载路径和渲染代码未变。源图为 EPSG:4326、16200×8100;旧 JPG 在仓库 `resources/earth/archive/2026-09-21/` 逐字节保留。来源、许可、源包/目录版本差异、SHA256、JPEG 参数及按导出尺寸重算的定位文件见 `resources/earth/natural-earth-1/`。原始大文件位于 Git 忽略的 `downloads/earth/`,不进入前端发布包。
|
||||
|
||||
底图选择器默认使用“原始底图”,并将该选项放在首位,显示菜单可切换并在本地保存选择。已有 Natural Earth I 保留可用,并增加“原始底图”选项,使用 2026-09-21 归档原图的逐字节副本;NASA 21K 只作为源归档,运行时统一使用一张 8K 贴图。新贴图成功加载后才替换并释放旧 GPU 纹理,失败时保留旧纹理,过期响应不会覆盖当前结果;这是源切换,不新增渲染层。物理贴图为 JPEG sRGB,未烘焙政治边界,也未完成逐段边界对齐验收。星空逐字节恢复到上次提交前的 commit `83a10a6c`:`celestial.js` 与该提交一致,删除 `skyTint`、`skyBrightness`,`brightStarOpacity` 恢复为 `.88`,不再校准星空密度和颜色。高清 overlay 使用白色 `MeshBasicMaterial`、opacity 1、`toneMapped:false`,由 `texture-colors.js` 统一 sRGB 读写;r128 使用 `encoding`/`LinearTosRGB`,r160 使用 `colorSpace`/colorspace fragment。不叠加 Phong 白光、彩色暮光、蓝色夜染、`cameraBoost` 或 shoulder 曲线;daylight 中性亮度不超过原图最大值,dark 仅做 neutral 缩放,原图资源不变。Natural Earth 原图本身偏浅,不声称恢复不存在的细节。为 WebGL1 的 `fwidth` 开启 derivatives。
|
||||
|
||||
|
||||
昼夜明暗使用以太阳地平线为中心的窄平滑过渡(法线点积 ±0.12)。日面保持原图亮度上限;夜面线性光照系数为 0.24,原图暗部最高保留 0.60,按源亮度连续过渡,以保持暗色海洋和森林细节。只使用中性标量,不增加曝光、彩色罩层、额外渲染通道或纹理。
|
||||
|
||||
地表底图与新闻频道复用 `hud-select-menu.js` 的 HUD 浮层、列表、选中标记、键盘导航和定位;`tv-source-menu.js` 仅保留频道文案适配。底图关闭搜索、分页和计数,菜单通过触发按钮归属参与移动抽屉的内部点击判断。
|
||||
本文件描述当前智能星球前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
|
||||
@@ -19,6 +21,8 @@
|
||||
|
||||
## 当前目标
|
||||
|
||||
算力卡片的年度观测由 `/api/v1/data-products/compute/series` 返回,后台“算力对比”配置为唯一维护入口。`compute-topic.js` 以 no-store 请求读取数据,接收同源 `planet-compute-configuration` 发布通知,并在可见且专题启用时每 60 秒及页面恢复可见时检查更新;刷新观测不重拉设施图层。`destroy()` 释放计时器、通知通道和可见性监听。国家目录、区域完整覆盖条件与缺失语义仍由后端统一处理。
|
||||
|
||||
Earth 前端不是普通管理页,它是独立的大屏展示前端。当前产品目标是:
|
||||
|
||||
- 维持地球视图的空间感和可读性
|
||||
@@ -66,7 +70,7 @@ React 路由入口:
|
||||
- 各图层集成
|
||||
- Earth 级别状态同步
|
||||
|
||||
Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,普通图层仍通过 `/api/v1/visualization/...` 接口重新 GET;船舶使用下文的 `vessels` 专用确认状态增量通道。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload`、`reload` 或 `delta` 策略。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉,summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,普通图层仍通过 `/api/v1/visualization/...` 接口重新 GET;船舶使用下文的 `vessels` 专用确认状态增量通道。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload`、`reload` 或 `delta` 策略。纯新闻变更不使 summary 缓存失效;同一批变更还涉及其他图层时,summary 仍随之刷新。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉,summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
|
||||
### 3. 地球控制层
|
||||
|
||||
@@ -200,7 +204,7 @@ Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/h
|
||||
- `关于`:配置 Earth 设置里的 About 卡片,包括 logo、眉标、标题、版本、描述和元信息条目;Earth 运行时从 `/earth/about` 读取,失败时回退默认内容。
|
||||
- `国界精度`:构建边界、刷新状态和恢复默认属于这个分区内部动作,不应放在页面全局工具栏。
|
||||
- `电视直播`:列表同时区分内置源、采集源和自定义源;卡片状态表达启用、停用、新建或错误。内置源不能删除,采集源和自定义源可以删除。新增直播源在点击加号后才进入草稿状态,保存后固化到列表,取消则销毁草稿。
|
||||
- `底图资源`、`图层资源`、`3D 模型`、`新闻锚点策略`:如果后端能力未接入,控制台应明确显示待接入空态,不应混入 TV 或品牌配置。
|
||||
- `底图资源`、`3D 模型`、`新闻锚点策略`:如果后端能力未接入,控制台应明确显示待接入空态,不应混入 TV 或品牌配置。
|
||||
|
||||
TV 预览需要尽量复用 Earth 运行时的直播卡片结构和状态标签,这样控制台看到的内置、直播加载中、播放源、地区、语言等信息与 Earth 中实际显示保持一致。
|
||||
|
||||
@@ -229,7 +233,7 @@ TV 预览需要尽量复用 Earth 运行时的直播卡片结构和状态标签
|
||||
|
||||
## 渲染热路径
|
||||
|
||||
`cable-batches.js` 将全部海缆线段与全部登陆点分别合批绘制;`cables.js` 继续拥有原始业务对象、拾取、遮挡和选择状态。避免在后续功能中把这些代理对象重新加入逐对象绘制,或把绘制批次当作业务对象返回给详情卡。
|
||||
`cable-batches.js` 将全部海缆线段与全部登陆点分别合批绘制;`cables.js` 继续拥有原始业务对象、拾取、遮挡和选择状态。避免在后续功能中把这些代理对象重新加入逐对象绘制,或把绘制批次当作业务对象返回给详情卡。密集线缆区的拾取先在屏幕距离上比较卫星、BGP、算力和其他点状对象,海缆线作为没有更近点状对象时的兜底;hover 与 click 共用该候选顺序,保护穿插在线缆中的小面积物件。
|
||||
|
||||
卫星的逐点呼吸在 GPU 中计算。全量 SGP4 位置和初始轨迹计算通过 `satellite-position-worker.js` 离开主线程;主线程只接收位置快照、更新共享交互坐标及绘制缓冲。轨道数学由 `satellite-propagation.js` 同时供 Worker、预测轨道和同步降级路径使用,避免多套公式漂移。数据重载、清空和高度切换必须同步终止旧计算,不能让旧快照恢复已清空的图层。
|
||||
|
||||
@@ -406,7 +410,7 @@ AISStream 与 BarentsWatch 共用 `/ws` 的 `vessels` 增量通道。Earth 使
|
||||
|
||||
登陆点是当前明确保留的例外:它曾接入 `Interactable`,但 pin 类 SVG 在地球边缘会被 `THREE.Points` 的深度测试裁切成碎片;关闭 depthTest 又会破坏背面遮挡语义。因此登陆点退回 `cables.js` 内的专用 `THREE.Sprite` 路径,并改为 canvas 生成的黄色扁平球纹理。它的 `altitudeOffset` 和 `renderOrder` 与海缆线一致,避免漂在海缆之上;Sprite 本体关闭 `depthTest` 保持球完整,背面可见性由 `isFacingCamera()` 的球体遮挡判断控制。
|
||||
|
||||
图标资源可以继续用 canvas draw,也可以放到 `frontend/public/earth/assets/icons/` 后由 `Interactable` 预加载。asset 路径不会在每帧读取;图层加载阶段通过 `preloadAssets()` 只加载一次 SVG / 图片,之后按 `icon source + state + bucket + color` 生成 `CanvasTexture` 并复用。当前算力中心已经从 `assets/icons/compute-supercomputer.svg`、`assets/icons/compute-gpu-cluster.svg` 和备用 `assets/icons/compute-hdd-network.svg` 读取图标,再在 canvas 上叠加未确认位置的 `?` badge。算力中心后端在启动链路只渲染源数据自带坐标或 `compute_center_locations` 维表坐标;手动候选采集会调用 ROR 和 Nominatim/OpenStreetMap,并在 GeoJSON 或候选响应中返回位置精度、置信度、来源说明和核验时间;前端详情卡展示这些字段。
|
||||
图标资源可以继续用 canvas draw,也可以放到 `frontend/public/earth/assets/icons/` 后由 `Interactable` 预加载。asset 路径不会在每帧读取;图层加载阶段通过 `preloadAssets()` 只加载一次 SVG / 图片,之后按 `icon source + state + bucket + color` 生成 `CanvasTexture` 并复用。当前算力中心已经从 `assets/icons/compute-supercomputer.svg`、`assets/icons/compute-gpu-cluster.svg` 和备用 `assets/icons/compute-hdd-network.svg` 读取图标,再在 canvas 上叠加未确认位置的 `?` badge。算力中心后端在启动链路只渲染源数据自带坐标或 `compute_center_locations` 维表坐标;手动候选采集会调用 ROR 和 Nominatim/OpenStreetMap,并在 GeoJSON 或候选响应中返回位置精度、置信度、来源说明和核验时间;前端详情卡保留位置精度和核验状态;定位来源、置信分、匹配过程等仍保留在数据与位置校准流程中,不再逐项占据常规详情卡。
|
||||
|
||||
算力中心图层行左上角的通知气泡显示 GeoJSON `unresolved` 数量。这个数字表示“完全没有可信坐标、不能渲染到地球上”的记录,不等同于地图上带 `?` 的已定位待确认点。点击气泡会在图层面板右侧打开固定信息卡,信息卡内容区内部滚动,不随鼠标 hover 消失。列表中的单条 `采集` 只展示候选;顶部 `一键采用` 会按当前列表顺序逐条采集、保存最高置信候选,成功一条就移除一条、重新编号,并通过 `earth:compute-center-unresolved-count-change` 同步气泡数量。批量结束后再触发 `earth:compute-center-location-saved` 刷新真实图层。
|
||||
|
||||
@@ -426,7 +430,7 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
|
||||
|
||||
跨 Interactable 的同坐标关系也在公共层记录,但真实位置必须始终以 `icon_base_position` 为准。缩放、避让、聚合和后续 spiderfy 展开都只能改变屏幕表现,不能写回 `marker.position` 或 `THREE.Points` 里的业务锚点;巡航定位、详情卡、搜索定位和 picking 返回对象都必须落回真实经纬度。多个图标归入同一个经纬度 key 时,公共层只写 `icon_avoidance_*` 元数据,供业务层弱化 halo 或显示聚合提示;真正的低缩放聚合应通过独立 cluster glyph / screen layout 层实现,而不是把对象沿地表切平面挪开。
|
||||
|
||||
`Interactable` 的单点显示只由全局地图缩放决定:170% 及以下强制显示小圆点,超过 170% 显示原图标。cluster 现在由 `cluster.strategy` 决定:`stable-spherical` 使用离散 zoom band 和 3D 球面分桶,BGP、算力中心和 Earth interactable 在同一 band 内旋转或细微缩放时不会重新计算聚合拓扑;`dynamic-screen` 保留屏幕空间聚类;`none` 关闭聚类。船只显式关闭聚类与避让,保证增量更新时不重建聚类拓扑。稳定球面聚类的 cluster 圆点刚性落在成员 3D 质心投影上,不参与 2D 避让,避免缩放时被推离真实地理位置。cluster 圆点大小随包含对象数量增长,数量过多时按稳定地理顺序拆成多个较小圆点;数量默认只在 hover tooltip 中显示。这个过程只设置 `icon_cluster_*` 展示元数据和重建渲染 Points,不改变每个 marker 的真实经纬度。
|
||||
地表标记保留分类图标,聚合使用固定地理成员及带数量的实心圆。缩放只驱动 320ms 展开/收拢补间,不重新分档聚类;超过 250% 展开,500% 无聚合圆标。船只关闭聚合,保留独立航向箭头。实现与验收见 [聚合说明](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-clustering.md)。
|
||||
|
||||
接口细节、生命周期和接入示例见:
|
||||
|
||||
@@ -437,6 +441,8 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
|
||||
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一维护智能星球缩放状态,所有入口最终都必须调用 `setZoomLevel()` 写相机距离。不要在其他模块直接写 `camera.position.z`,否则缩放百分比、拖拽灵敏度和 Interactable 聚合阈值会再次分叉。
|
||||
|
||||
鼠标与单指统一使用原有屏幕位移驱动、0.18 平滑和松手惯性。按下后拖出球体轮廓仍连续旋转,不再调用射线/球面命中作为拖动前提。常规缩放保留原灵敏度;球体超出视口后平滑加入按相机距地表距离和视口尺寸计算的旋转上限,修正极大倍率的放大效应。左右绕地理轴、上下改倾角,四元数保持多圈不反向;双指缩放切换与 pointer capture 保留。
|
||||
|
||||
滚轮输入分两类处理:传统鼠标滚轮保留 10% 档位和短动画,并用 `wheelZoomTarget` 作为连续滚动的逻辑基准;触控板 / 高精度滚轮按 pixel delta 连续缩放,直接调用 `setZoomLevel()`,不走 10% 档位动画。触控板路径还会过滤反向后的短窗口旧方向小 residual delta,避免惯性尾巴把用户刚反向的缩放又拉回旧方向。
|
||||
|
||||
该提示每 90ms 最多更新一次,显示 760ms 后淡出。它是视角反馈,不是数据加载进度,也不应该写进图层 loading 状态。
|
||||
@@ -445,7 +451,7 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
|
||||
|
||||
`globe-navigation.js` 统一投影缩放与表面拖动。相机距离采用 `sqrt(R² + (D₀² - R²) / zoom²)`,并保留最小地表净空;100% 视角不变,500% 仍在球外,反向换算也使用同一公式。滚轮、按钮、捏合、搜索与巡航继续共用 `setZoomLevel()`。
|
||||
|
||||
鼠标和单指通过相机射线取得球面抓点,以四元数将同一地表点旋转到当前接触位置;按住期间不使用经验倍率或追赶平滑,自转也不叠加。松手后沿实际旋转轴衰减惯性,停住再松手或取消手势不触发甩动。双指进入缩放时清除拖动惯性,回到单指时重建抓点。拖动阈值按起点至当前位置的累计位移判断,避免慢拖被当作点击。
|
||||
鼠标和单指在按下时按实际相机视锥选择并固定拖动模式:球体完整放入视口时,使用旧版屏幕位移、缩放灵敏度、0.18 追赶平滑和松手衰减惯性,指针移出球面仍保持原方向;球体放大到被视口裁切且按下命中球面时,才用射线抓点与四元数逐像素跟手。大球拖动移出球面后暂停更新并清除甩动速度,不再用可能反向的最近点代替真实交点;从球外开始则使用屏幕位移拖动。一趟拖动中不切换模式,自转不叠加。双指进入缩放时清除惯性,回到单指时重新判断模式并建立抓点。拖动阈值仍按起点累计位移判断,取消手势不产生惯性。
|
||||
|
||||
### `data-status-target`
|
||||
|
||||
@@ -493,7 +499,7 @@ Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/
|
||||
|
||||
卫星真实高度开关由 `controls.js` 持久化,实际渲染状态在 `satellites.js`。开启时,SGP4 得到的真实半径会按对数压缩到当前 Earth 视觉半径范围;关闭时,卫星点、轨迹和预测轨道都回到旧版同层球面。切换时必须刷新卫星位置并清理轨迹缓存,避免同一条轨迹混入两个高度模型。`maxRealAltitudeOffset = 25` 是当前相机和 `earthRadius = 100` 下的视觉上限:它让 GEO / MEO 比 LEO 明显更高,但把最高轨道控制在地球半径外约 25%,避免选择点、红色轨迹和主体地球之间出现过大的空场。
|
||||
|
||||
SGP4 传播输出是惯性系位置,不能直接当成 Earth 的经纬度固定坐标使用。`satellites.js` 会用当前时间的 `gstime` 把 ECI/TEME 位置转换到 ECF,再映射到 `latLonToVector3()` 使用的 Three.js 坐标轴。卫星点和短尾迹使用随采样时间变化的地固坐标,表示相对当前地球表面的实际位置;锁定后的预测轨道线使用锁定时刻固定的 `gstime`,把未来一圈惯性轨道投到当前地球姿态上显示,因此会闭合,并且轨道面倾角应与详情卡一致。fallback 预测轨道也必须使用真正的 RAAN + inclination 轨道平面公式,不能把 inclination 当成恒定纬度。
|
||||
SGP4 传播输出是惯性系位置,不能直接当成 Earth 的经纬度固定坐标使用。`satellites.js` 会用当前时间的 `gstime` 把 ECI/TEME 位置转换到 ECF,再映射到 `latLonToVector3()` 使用的 Three.js 坐标轴。卫星点和短尾迹使用随采样时间变化的地固坐标,表示相对当前地球表面的实际位置;锁定后的预测轨道线使用锁定时刻固定的 `gstime`,把未来一个完整周期的惯性轨道投到当前地球姿态上显示为开放弧线;不追加首点或用直线强制封口,轨道面倾角与详情卡一致。流光按累计弧长和未来采样方向移动。fallback 预测轨道也必须使用真正的 RAAN + inclination 轨道平面公式,不能把 inclination 当成恒定纬度。
|
||||
|
||||
国界精度偏好独立存储在 `country-boundaries.js` 的 `planet.earth.boundaries.highPrecisionEnabled`。未开启高精时,即使本机已经有高精 manifest/PMTiles,也继续加载低精 `countries-admin0.min.geojson` fallback;开启高精但高精产物缺失时,智能星球工具栏设置会调用 `/api/v1/earth/boundaries/build` 启动后台构建并轮询进度。构建成功后调用 `reloadCountryBoundaries()` 热切换,不再刷新整个页面。国界 hover 与 tooltip 解耦:只要地表坐标落在国界 polygon 内就保持高亮;如果鼠标同时命中卫星、船只、BGP 等 interactable,tooltip 显示 interactable 信息,但国界高亮不应闪烁。
|
||||
|
||||
@@ -543,13 +549,13 @@ Earth 原有 Roboto Condensed 字体已随 `assets/fonts/` 本地分发,许可
|
||||
|
||||
`compute-topic.js` 管理参考期、口径、参与方和来源,读取后端 `compute_catalog.py` 提供的规范名称、分组与地区标签锚点。当前目录覆盖 45 个已有坐标的参与方,未知历史指标返回缺失。参与方接口不再只接受中美;区域分组不参与分母计算,坐标不能复用为设施定位。
|
||||
|
||||
`spatial-callouts.js` 将布局委托给新的 `spatial-callout-controller.js`,按稳定地区 ID 复用 `CalloutConnector` 管理卡片。布局采用事件驱动;新计算尚未完成时保留有效旧矩形,投影 source 连线持续跟随地球。布局参考点改为实际出线方向,因此卡片保持在对象同侧,不会被左上地理点牵到右下角。solver 优先同扇区的邻近位置、较短连线和较少交叉;最多交换三轮空位改善 assignment,不在每帧全量重排。全屏地球允许附近前景位置,正常大小尽量留在球外,不为了完全避开球体而跨越整盘面。慢速模式下,方向明显偏移或连线持续交叉 1.8 秒后才能提议整体重排,两次自动提议至少间隔 8 秒;只有整体代价降低至少 18% 且超过最低收益门槛才采用。手动/惯性仍在 72ms/12px 预算内响应。展开与手动摆放获得 12 秒阅读缓冲;鼠标悬停、键盘焦点及拖动继续锁定,普通点击留下的焦点不永久锁卡。手动位置保持为偏好,缓冲结束且地球出线方向变化至少 45° 后才允许自动调整,旧保存位置不作为永久锁定。背面保留球体轮廓方向提示且不改变卡片身份。本地位置/尺寸保存与“恢复自动布局”由控制器处理。`ResizeObserver` 响应卡片调整尺寸和收起;选择器与来源窗复用共享拖动句柄。
|
||||
`spatial-callouts.js` 提供地理投影与共享分组,`spatial-callout-controller.js` 为算力和新闻分别选择展开卡片,优先保留正在拖拽或手动放置的卡片,再按当前关注区域与距视图中心的距离补齐。`callout-limits.js` 在 `planet.earth.callout-limits.v1` 保存两类独立上限,界面提供 1–3 张,默认各 1 张;修改后通过 `earth:callout-limits-change` 立即重算。其余卡片使用随地理锚点移动的胶囊,悬停临时展开,点击切换关注区域。完整卡片分配到左右两侧,新闻优先在上,同侧超过两张时仍为每张分配目标位置;空间有限时允许周边重叠。背面胶囊渐淡缩小,手动拖放保留位置,区域深度进入背面时才解除固定。尺寸、阅读状态与业务坐标独立,拖动前景层级由新闻和算力共用。
|
||||
|
||||
对象展示统一通过 `PresentationController` 及新的 object-callout anchor/layout 路径。点击真实 BGP、算力中心、船只、卫星或海缆对象时,先绘制连接线,再把卡片放到对象旁边。源锚点从实际点击位置或原始区域标签开始;接近遮挡时,随着旋转自动平滑移向仍可见的高亮几何采样。`layoutReferencePoint` 取实际出线点的方向参考,`packDirectionalEdgeCallouts` 优先把卡片放到对应方向的屏幕边缘,给地球中间留空;边缘 lane 不可用时才内收,卡片不能盖住自己的出线点。有界联合分配最多使用 24 个 beam 状态和 32 个候选,避开已有 HUD,再最多三轮安全交换减少方向交叉;旧 solver 作为 fallback,不逐帧全量重排。slow 模式还检查实际卡片目标相对 source 反向超过 90° 后才提交变化;manual/hover 锁优先。source point 仍每帧更新为连线目标;固定对象跟踪动态位置,卡片仍可拖动。完全背面的对象保留球体轮廓方向提示、较淡连线和空心源点,这些是方向引导而非地理点。国家候选最多 96 条、硬上限 128,并使用缓存,不在每帧扫描 GeoJSON。算力卡继续让原始 point 独立于动态 source point,保持阅读稳定;共享卡片端接点沿障碍物周界滑移。海缆展示优先真实 ray-hit 的 local point,再按 `cableId` 采样高亮几何。移动端沿用 mini popup,点击后打开抽屉详情。旧异步 presentation 不能 dismiss 新的 selection。
|
||||
|
||||
配置窗的 `configVisible` 与卡片的 `cardsEnabled` 分离;X/Escape 只关闭配置,开关停用时才释放连线并恢复专题查询前的图层状态。配置和来源窗直接绑定 HUD 字号,避免拖到 `container` 后继承大号正文。主卡围绕当前选择的算力指标,显示大号百分比或带单位的绝对量、观测年份、历史上下文和来源入口。缺少可比数据时明确提示,不在没有观测值时绘制空趋势图。设施记录和覆盖信息仍完整保留在展开后的辅助区,样本数只是证据上下文,不能当作算力值或全球覆盖率。长列表仅在需要时展开;手机手风琴保持同一单一占比/绝对量语义。本地位置/尺寸保存与“恢复自动布局”仍可用;无变化的 DOM、尺寸、阅读状态应保留,过时响应不能重新启用卡片。
|
||||
配置窗的 `configVisible` 与卡片的 `cardsEnabled` 分离;X/Escape 关闭配置窗并保留卡片状态。桌面标题栏的卡片按钮与新闻面板共用图标和选中样式,其 `aria-pressed` 跟随 `cardsEnabled`。通过该按钮或移动端开关停用卡片时,释放连线并恢复专题查询前的图层状态。配置和来源窗直接绑定 HUD 字号,避免拖到 `container` 后继承大号正文。主卡围绕当前选择的算力指标,显示大号百分比或带单位的绝对量、观测年份、历史上下文和来源入口。缺少可比数据时明确提示,不在没有观测值时绘制空趋势图。设施记录和覆盖信息仍完整保留在展开后的辅助区,样本数只是证据上下文,不能当作算力值或全球覆盖率。长列表仅在需要时展开;手机手风琴保持同一单一占比/绝对量语义。本地位置/尺寸保存仍可用;无变化的 DOM、尺寸、阅读状态应保留,过时响应不能重新启用卡片。
|
||||
|
||||
动捕/地球运动模式由 `main.js` 的 `isDragging`、`activeTouchPoints`、`hasActiveGlobeInertia` 接入。方向错位或交叉触发的是一组卡片联合优化,而非逐卡抢占剩余空位。`callout-region-path.js` 提供短直连加末端短折线;实际绘制、交叉评分及候选搜索共用路径几何。候选路径只计算一次,裁剪候选后才复制布局状态,避免重排期间不必要的分配。新布局计算期间保留现有卡片;队列不会因帧间隔而丢失重排请求。
|
||||
展开卡片以平滑惯性追随已提交目标,悬停和键盘阅读时暂停该移动;胶囊独立跟随地理投影。连线锚点继续逐帧更新。`shared.calloutLineStyle` 保存细线/粗线偏好,桌面和移动控件同步到 `body.dataset.calloutLineStyle`,由共享连接线读取,不改变业务锚点。
|
||||
|
||||
共享 `CalloutConnector` 直接从折线点计算长度,不在每次写入 SVG 点后调用 `getTotalLength()`,避免多条移动连线触发反复同步布局。原有首次绘线动画保留;这一优化同时适用于原巡航展示。
|
||||
|
||||
@@ -559,16 +565,26 @@ Earth 原有 Roboto Condensed 字体已随 `assets/fonts/` 本地分发,许可
|
||||
|
||||
### 国界贴地与登陆点
|
||||
|
||||
普通国界、海岸、主张线、高亮及登陆点使用 `earth-surface.js` 对齐高清球面的 UV 三角形。国界与高亮在贴图之后、登陆点之前绘制,高亮开启深度测试且始终位于透明渲染队列;不再透过球体显示或盖住登陆点。几何构建时计算并缓存,不增加每帧重投影。具体顺序见[渲染图层规范](earth-render-layer-order.md)。
|
||||
普通国界、海岸、主张线、高亮及登陆点使用 `earth-surface.js` 对齐高清球面的 UV 三角形。国界与高亮在贴图之后、登陆点之前绘制,高亮开启深度测试且始终位于透明渲染队列;不再透过球体显示或盖住登陆点。几何构建时计算并缓存,不增加每帧重投影。具体顺序见[渲染图层规范](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)。
|
||||
|
||||
### 新闻栏端部渐变
|
||||
### 新闻栏端部渐变与 breaking 状态机
|
||||
|
||||
新闻栏的底色与上下细线由独立伪元素绘制,两端在正文范围之外渐变到完全透明;外框不使用矩形 backdrop-filter、box-shadow 或文字遮罩。地区标签按内容占宽,正文紧接标签,滚动文字不做渐隐。警报状态只改变渐变细线的颜色,避免重新引入矩形阴影边界。
|
||||
新闻栏的底色与上下细线由独立伪元素绘制,两端在正文范围之外渐变到完全透明;外框不使用矩形 backdrop-filter、box-shadow 或文字遮罩。地区标签按内容占宽,正文紧接标签,滚动文字不做渐隐。
|
||||
|
||||
此前的 breaking 规则是:只要本次 payload 中存在未过期的 breaking 新闻,就按 `critical > breaking > watch` 取最高等级,把等级直接映射到新闻壳层。这样会让整条 ticker 或展开面板在当前显示的新闻并非 breaking、甚至面板已经关闭时仍保留橙/红状态,造成“关闭后残留边缘”和“整个面板变色”。`breaking_expires_at` 到期后才清除该状态;选中、打开和关闭不会改变 breaking 数据本身。
|
||||
|
||||
当前状态机将数据态、ticker 显示态、面板生命周期和选中态分开:
|
||||
|
||||
- 数据态由 `getNewsBreakingState()` 从 `items` 与 `cruise_items` 合并计算,只接受未过期项目,并按 `critical > breaking > watch` 排序;到期时间只负责安排下一次重新计算。
|
||||
- ticker 的显示态由滚动视口 watcher 每 120ms 取视口内重叠最多、距离中心最近的当前项目。只有当前项目的 breaking 等级非 `none` 时,ticker 的上下细线才切换为对应的 `has-breaking-*` 颜色;循环换项产生的短暂无重叠空档会保留 active 状态,避免扫光时间轴重启或边线闪烁。
|
||||
- 面板生命周期 `panelPhase` 只有 `closed → opening → open → closing → closed`。打开/关闭只控制形变、可见性和 ticker 的隐藏,不改变 breaking 数据态;进入 `opening/open` 后保留当前卡片 DOM,后台 payload 先暂存,关闭时再一次性应用,避免 hover 卡片被实时更新重建。
|
||||
- `selectedCruiseStoryId` 独立于 breaking 等级。选择、聚焦、悬停和关闭只更新选中项与阅读位置;关闭时清空选中项及筛选弹层。
|
||||
- 展开面板始终使用蓝色玻璃底。breaking 只在对应新闻卡片上使用等级颜色,因此点开面板不会把整块玻璃染成红色,之后滚动到 breaking 新闻时仍可让 ticker 显示红色提示。
|
||||
|
||||
breaking 的玻璃反射效果已实现:ticker 当前项目进入 breaking 状态时,`::after` 用短而四周羽化的光团横穿 ticker 玻璃区域;展开面板中只在对应 breaking 新闻卡片上用 `::after` 扫过整卡反射带。面板底色保持蓝色,静态上下边线不参与扫光;`prefers-reduced-motion` 下停用动画并保留低强度静态提示。该视觉子状态挂在当前 breaking 项目和对应卡片上,不与面板是否打开耦合。
|
||||
|
||||
算力卡片外观直接复用共享 `.hud-panel` 的背景、边框、阴影和玻璃效果,摘要、基础信息及详情区不再叠加单独黑底。标题栏与图层面板共用 `.hud-panel__header--inset` / `.layer-panel-header` 的内边距;主数字继续突出,基础信息保持常显。
|
||||
|
||||
保存位置恢复时会迁移旧版居中拖放坐标:只解除位于屏幕核心区域、且没有新版位置标记的旧坐标锁,保留宽高;旧版边缘位置保持不动并更新标记。新拖放保存 `positionVersion=2`,以后仍尊重用户的新位置。迁移沿用原存储键,避免因全量清空布局而丢失所有个性化位置。
|
||||
|
||||
|
||||
### 首帧与空间卡片入场
|
||||
|
||||
@@ -579,3 +595,135 @@ Earth 由 `earth-runtime-plugin.mjs` 将本地模块合并为带内容哈希的
|
||||
空间卡片定位后保持隐藏,先复用 `CalloutConnector` 的 420 ms 绘线动画,再按共享 HUD 的 180 ms 淡入时长显示卡片。绘线期间固定目标位置,后续跟随运动不重播入场;系统减少动态效果时直接呈现。
|
||||
|
||||
物件点击、搜索定位及手势确认统一通过 `PresentationController`。海缆选择处理器只更新选择与图例,不再绕过控制器提前显示卡片;共享 InfoCard 的显示回调带版本标记,新的隐藏/预布局请求会使旧回调失效。所有连线卡片以绘线完成事件作为显示门槛,取消的旧巡航不能关闭新的选择卡片。物件详情复用算力卡的边缘候选与避让评分,按隐藏状态下测得的真实尺寸定位置;手机搜索结果被选中时收起搜索抽屉,让地图定位、引导线与精简卡片可见。
|
||||
|
||||
### 卫星粒子与平滑显示
|
||||
|
||||
卫星由单批 `THREE.Points` 与 `PointsMaterial` 绘制:32×32 白色圆盘贴图、半径 0.42、`alphaTest=0.5`、`sizeAttenuation=false`,不再绘制深色背景点或伪球面明暗。`satellite-display.js` 按 satellitemap 的截图像素校准:150% 及以下的点方框为 2.4px、500% 为 8px,实心圆盘占 84%,对应约 2px 与 6–7px 的可见点径,中间 smoothstep 过渡。蓝/橙/绿/紫/玫红直接使用参考色板,不额外叠加原型曝光。点尺寸保留 DPR 上限 2 的折算,地球渲染分辨率不变。普通呼吸默认关闭,设置仍可单独开启;悬停/选中环、光晕、开放预测轨道及流光保持现有交互。
|
||||
|
||||
卫星点精灵设置 `depthTest=false`,顶点着色器用相机到卫星的线段与地球球体做遮挡测试,半径为 `CONFIG.earthRadius * 0.998`。真正被球体遮挡的点移出裁剪空间,可见点则整点显示,避免近地平线被逐像素深度裁切。轨道、覆盖与业务位置仍使用原有路径。
|
||||
|
||||
真实 GP/TLE 仍在 Worker 每 250ms 传播。`satellite-motion.js` 缓存采样半径并逐帧半径保持插值,粒子、拾取、短尾迹末端、选中环和覆盖共享同一位置。`satellite-trail-geometry.js` 仅在所有支持缩放下的投影误差不超过 0.2 像素时合并冗余尾迹段;保留全部卫星和 10 个历史采样,曲率或近相机条件不满足时保留原分段。仅上传活跃缓冲区,尾迹宽度不超过点尺寸的 60%,尾端收细,避免远景形成粗碎线圈。`satellite-footprint-geometry.js` 保留 260×170 的完整覆盖网格,仅创建一次;同一地表投影公式在顶点着色器计算,每帧更新局部基向量和范围参数,避免移动时重建数万个顶点。Starlink/Iridium 覆盖统一蓝色 `#3399ff`,日照只调整透明度,真实覆盖模型不变。
|
||||
|
||||
`SATELLITE_CATEGORIES` 统一真实倾角分类及蓝/橙/绿/紫/玫红/灰色。选中卫星时自动切换到卫星图例,直接点击图例分类控制点、拖尾、标签和拾取,隐藏已选分类会清除选择与覆盖;图层和设置中不另设筛选入口。搜索明确选择隐藏类别的卫星时重新显示该类。`satellite-labels.js` 在缩放达到设置阈值(默认 **400%**,可调范围 **250%–500%**)时显示真实名称与型号(缺省为 NORAD 编号),数量上限可在近景标签开关下通过滑块设置为 5–36 个,默认 36;桌面和移动端共享持久化设置,修改后立即重新选取候选。实际显示数量仍受遮挡和可用位置限制,中央 65% 范围内做遮挡和重叠检查;候选扫描每 160ms 一次,已选标签逐帧跟随。
|
||||
|
||||
“设置 → 显示 → 显示 FPS”由 `shared.showFps` 持久化,默认关闭。`frame-stats.js` 在品牌右侧显示实际帧间隔统计,每秒更新;原生提示包含平均帧间隔、P95、主线程 CPU 时间、Draw calls 和约 30 秒滚动平均 FPS(总帧数 / 总时长,不平均每秒 FPS)。前台慢帧必须计入,隐藏页切换时重置窗口;不能把个别高帧样本当成稳定性能保证。性能验收至少使用 1920×1080 实际画布,连续采样至少 30 秒,以平均 60 FPS 为目标,允许偶发低帧。验证应使用同一视口、缩放、完整数据、图层和播放状态,对比远景及 400% 近景,并检查分类隐藏、搜索选中和 FPS 刷新后的持久化。
|
||||
|
||||
国界加载占位球、海陆填色、主球和高清层使用 `earth-surface.js` 的统一网格工厂;占位球与海陆填色共用 R+0.32,修复纹理加载前的菱形纬圈。国界和高清开关行为保持不变;几何穿插与远距深度余量由 `earth-surface.test.js` 覆盖,详见[渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)。
|
||||
|
||||
海缆与登陆点使用 `cable-surface.js` 共用高清地表坐标;关联光晕由 `cable-batches.js` 的逐实例 emphasis 控制,解除选择后清零。圆角徽标在 `interactable.js` 中接入 `surface-marker-occlusion.js`,以锚点决定整块显隐及拾取,避免近地平线被地球切开。BGP 变化事件保留原图标,观测站采用带活动色的广播塔圆角徽标;星空材质使用原始亮度(系数 1.0)。
|
||||
|
||||
缩放子菜单与环形工具菜单由 `toolbar-hover.js` 的统一状态机控制。放大镜 hover 打开子菜单;触发器、透明桥接间隙和纵向按钮区构成连续保活区域。离开该区域先执行子菜单 220ms 收起,再根据最新指针位置及固定展开状态决定是否收起外层。鼠标仍在环形唤醒区域或外层已 toggle 固定时,仅收起子菜单;迅速返回会取消旧计时器。CSS 不再用 hover/focus-within 独立控制显隐,点击缩放按钮不会撤销外层固定状态;键盘与触摸入口仍保留。
|
||||
|
||||
扇形菜单继承全局 `--hud-scale`(1920×1080 基准,范围 0.7–1),不再固定为 1。按钮、图标、扇形半径、底部留白与缩放子菜单一起缩放;轨道基准坐标仅在 CSS 变换时乘比例,hover 唤醒范围使用相同坐标比例,避免 DPI 变化后的视觉位置与命中区域错位。缩放仅改变几何尺寸,不改变展开动画时序及 pinned/hover 状态机。
|
||||
|
||||
### 星空颜色输出
|
||||
|
||||
星空贴图按 sRGB 解码;`celestial.js` 通过共享 `preserveTextureOutput()` 仅为星空材质补齐屏幕 sRGB 输出。Three.js r128 的全局 renderer 仍保持既有线性输出,避免改变地球、卫星及其他图层。`skyBrightness=1.0` 表示原图亮度;缺少输出编码时,解码后的暗部会再次以线性数值显示,不能靠提高倍率修正。验证应固定星图、相机与视角,对比渲染像素和 1.0/1.5 两档响应,不能只检查加载成功。
|
||||
|
||||
|
||||
### 图例直接筛选
|
||||
|
||||
当前交互物件继续决定图例类型;通过现有开关启用图层也会切换到对应图例,不增加筛选入口。海缆按线路名称、算力按类型、船只按船型及停泊状态、BGP 按事件级别和观测站活跃度筛选;覆盖范围和影响区域各自控制对应效果。国界图例分别控制线条、陆地填色与海洋填色,海陆 mask 数据本身不变。已不存在的 BGP“事件连线 / 枢纽”行被移除。
|
||||
|
||||
`legend-filters.js` 保存各类独立的页面内筛选状态;`legend.js` 统一绘制带 `aria-pressed` 的图例按钮,桌面和移动端共享同一状态,刷新图例保留键盘焦点。`interactable.js` 在绘制批次、聚类候选及拾取前应用筛选,保留完整原始数据、统计总量和对象身份。实时船只类型变化、数据重载及图层开关不会绕过筛选;隐藏选中项清除卡片和覆盖,显式搜索选中隐藏项则重新显示相应类别。船只轨迹请求使用版本检查,避免隐藏后迟到响应恢复旧轨迹。
|
||||
|
||||
### 稳定卡片与显示偏好
|
||||
|
||||
`callout-rest.js` 保留 450ms 的动作稳定门槛与最多 1.5px 的轻微漂浮。鼠标悬停、键盘阅读、拖拽与 reduced-motion 暂停漂浮;装饰偏移不参与持久化和布局求解,中央边界始终受保护。
|
||||
|
||||
显示设置新增 `shared.satelliteLabelsEnabled`(默认 true)以及 `shared.satelliteLabelMinZoom`(2.5–5,默认 4)与 `shared.skyBrightness`(0–2,默认 1)。桌面与移动控件、刷新、重置共用同一运行时状态。星空倍率在线性空间作用于星空材质,沿用局部 sRGB 输出,不修改地球曝光。算力图例显示实际已定位分类数量,芯片与堆叠 SVG 继续按 `site_type` 区分;数据集中只有 GPU 时不伪造其他类型或坐标。
|
||||
|
||||
展开卡片与胶囊由关注状态和各自数量上限控制。左右两侧舞台位置、胶囊跟随与悬停展开共享控制器;手动固定优先,阅读时暂停漂浮及展开卡片的自动移动。旧 `callout-dock-motion.js` / `callout-routing.js` 工具不再决定这一展示路径。
|
||||
|
||||
|
||||
### 聚合圆标与 Spiderfy
|
||||
|
||||
普通地表标记继续由 `interactable.js` 批量绘制;聚合组使用 `cluster-presentation.js` 的同一组投影节点贯穿合并、展开和展开后停留。圆标为 26px 实心渐变圆,显示真实成员数,混合类别带琥珀外圈。分组只随数据或筛选更新,缩放过程中不反复拆成中间子簇。
|
||||
|
||||
`cluster-motion.js` 用固定 320ms 三次缓动控制进度。可见图标准备和解码完成后才开始计时,纹理编码按缓存复用;子点从组中心补间到真实标记投影,最终显示偏移由 `marker-stack.js` 统一计算。动画与终点保持同一个 DOM 节点,反向切换从当前进度续接。
|
||||
|
||||
聚合判定使用界面的 `globeZoomForDistance()`;超过 250% 展开,缩回阈值有滞后,因此 500% 不显示聚合圆标。只有同坐标或极近地理锚点(约 0.00015°)跨图层紧凑堆叠;屏幕重叠和城市粗网格不会把不同地点合并。每个图标以自己的真实投影为基准,整叠在每个轴向最多错开 24px,小组露出约 8–12px 点击边缘。稳定 ID 决定初始顺序,选中后排对象时移到前端;聚合子点使用 `z-index: 1000`,未聚合图标使用共享的锁定覆盖层 `renderOrder >= 100`,跨图层重叠时仍由焦点对象在最上面,取消焦点后恢复原顺序。普通点展开终点严格等于真实投影,堆叠不改变经纬度或 marker 位置。数据刷新按实际成员实例清除旧节点,并同步重绘 GPU 隐藏批次;卫星与 AIS 船只不接入。
|
||||
|
||||
### 区域新闻卡片
|
||||
|
||||
`news-region-cards.js` 复用 `info-card.js` 的 `renderNewsCardContent` 与 `news-cruise-adapter.js` 的新闻展示映射。巡航与区域卡片共用新闻内容模板和 📰 图标、金色标题栏;区域卡片精简为来源/类型/时间三项元信息。`typewriter.js` 为每张卡片提供独立、可取消的打字机实例,先用隐藏排版层预留完整摘要高度,避免打字时卡片持续变高。摘要不截断;新闻卡片预留固定内容高度,长内容内部滚动,换标题和打字不改变落位。手动折叠时暂停打字及轮播;展开后继续,完整打完后再开始 7 秒阅读计时。
|
||||
|
||||
`news-region-model.js` 维护四个互相隔离的队列与剩余阅读时间,使用 `CRUISE_CONFIG.dwellMs`(7 秒),支持单卡暂停、手动翻页、页面隐藏恢复和数据刷新保留当前 ID。数据来自已有 `getCruiseNewsItems()`,随 `earth:news-payload-updated` 更新,沿用来源、类别和可展示内容过滤;不新增采集请求,不将 global 或其他区域补入空队列。
|
||||
|
||||
新闻与算力通过 `createCalloutGroups` 向同一个 `createSpatialCallouts` 提交卡片,统一求解目标位置、拖拽锁定、尺寸变化和遮挡。尺寸观察以元素身份查找状态,不依赖地区字符串;新闻布局键为 `news:<region>`,避免与算力地区冲突。仍使用既有区域卡片连线,不改变普通点击详情的连线方式。
|
||||
|
||||
启用开关保存在 `planet.earth.news-region-cards.v1`,默认关闭,入口位于现有新闻面板。手机将相同节点放入新闻抽屉并退出地图连线布局。销毁时释放卡片、动画、尺寸/布局观察与事件订阅。验证覆盖队列隔离、暂停与继续、刷新保留条目、共享布局增删、1080p 同时显示、移动布局和切回桌面。
|
||||
|
||||
新闻巡航适配器在正文打字完成后才结束 `presentItem`,因此 `CruiseSequencer` 的停留计时不会截断长文。等待每帧检查当前序列,手动切换或退出可立即中断。新闻区域开关使用 `aria-pressed` 同步高亮背景与边框;算力区域标题保留国旗,区域组合使用小尺寸代表 emoji。
|
||||
|
||||
### 卡片 DPI 与渲染帧边界
|
||||
|
||||
算力卡片、新闻区域卡片与普通/巡航详情卡共用 `--hud-detail-card-width`,标题、正文、元信息及摘要直接使用同一套 `info-panel.css` 规则。算力卡片、趋势图、图标、间距和控件尺寸均随根节点 `--hud-scale` 缩放;手动调整大小生成的内联像素尺寸会转换为相同比例的表达式。SVG 图表内部的 viewBox 单位随图表整体缩放,避免重复乘倍率。欧洲标题使用与正文同字体基线的小型 EU 字标,避免 Windows 将旗帜 emoji 展示为基线不同的字母。
|
||||
|
||||
配置菜单保留独立的 340px 宽度和原有高度上限,不套用详情卡的宽高缩放。算力卡片标题保持共享 HUD 规格,内容区用更大的主数值、趋势图和说明间距支撑统一宽度。区域卡片的移动由 JS 时间补间独占,取消通用 HUD 的 CSS 位移过渡和强制 transform 合成层;最终 left/top 对齐设备像素,漂浮继续保留,连线使用同一显示位置。空间不足允许周边少量遮挡,卡片内容保持显示。
|
||||
|
||||
全局选中对象是异构数据:卫星为 GeoJSON 对象,海缆是线对象,只有地表交互标记带 `userData.icon_layer_id`。`markerPresentationId` 仅为带该标识的对象生成堆叠身份,其余返回空,防止选中卫星时在帧内访问不存在的字段。
|
||||
|
||||
`frame-stages.js` 为各独立更新阶段建立异常边界;某个卡片、标签或图层更新失败不会截断其他阶段和主场景绘制。失败会记录阶段和异常,按一秒间隔重试,切换选中对象时允许立即重试;成功后恢复正常更新。根因修复和正常场景无运行异常仍是验收要求,保护机制不替代修复。
|
||||
|
||||
物件选中结果统一由 `showSelectionStatus(name)` 生成“已选择:名称”。普通点击、聚合子点、搜索定位和动捕确认共用该入口;`confirmInteractionCandidate` 的名称和反馈参数不再暗示仅适用于动捕。海缆图层只更新选中和图例状态,由交互入口通知一次,避免重复提示。已删除旧的锁定/定位/动捕确认消息键;真正的动捕模式切换和手势引导仍可标注动捕。
|
||||
|
||||
### 详情卡信息精简
|
||||
|
||||
常规详情保留对象身份、业务属性、时间与必要的数据可信度提示。卫星卡删除覆盖能力、当前显示和覆盖模型三项渲染说明,覆盖显示设置仍然有效;新闻详情删除 RSS 来源、源类型及抓取通道,保留媒体、分类、区域、发布时间和摘要。算力设施删除定位过程的来源、置信分、解析依据、匹配名称和核验时间,保留位置精度与核验状态;相同的 GPU 型号与芯片/平台只显示一次。来源状态等内部元信息不再展示,实际算力的精度/测量口径和原始来源仍保留。
|
||||
|
||||
BGP 观测站移除重复的日常活跃度汇总,保留观测计数、事件时间、覆盖范围;BGP 事件置信度仍用于判断告警可靠性。海缆不再显示没有单一含义的经纬度行,自定义点不再显示内部记录 ID。船舶资料为空时不显示“资料缓存中”,也不展示补全置信分;船舶身份、动态、资料及来源保持可用。桌面和移动详情共用字段配置,底层数据与定位操作不受影响。
|
||||
|
||||
### 多卡片与全量卫星的性能路径
|
||||
|
||||
区域新闻与算力卡片保留共享深色表面、边框、阴影和文字,取消每张卡片对动态 WebGL/视频背景的实时 backdrop 滤镜及边缘 SVG 扭曲滤镜,并用 `contain: layout style` 限定布局影响。卡片仍按设备像素定位,未改动数量、内容、打字机或漂浮时序。
|
||||
|
||||
`CalloutConnector` 的 SVG 视口只覆盖自身路径及节点/阴影留白,路径仍使用原始屏幕坐标;不再为每条短连线保留整屏 SVG 面积。地理锚点缓存不变的本地坐标,坐标变更会失效,世界变换和正背面判断逐帧更新。遮挡检查直接比较矩形,避免每个候选点与每个障碍物组合重复分配对象;HUD 障碍测量集中在该帧写入卡片样式前。
|
||||
|
||||
卫星插值在端点直接复制采样数据,中间帧继续使用半径保持插值;尾迹仅提前拒绝原有误差界限下必不合格的高轨弦合并。重复选中/悬停索引不重写全量透明度缓冲。海缆与登陆点的固定本地矩阵不再逐帧重建,登陆点缩放变化显式更新矩阵,地球父级的世界矩阵仍正常传播。上述优化不减少卫星、轨迹样本或渲染分辨率。
|
||||
|
||||
开发诊断可在 Earth URL 加 `?profileFrames=1`:`data-earth-frame-profile` 每秒输出各更新阶段的 CPU 均值、最大值和次数,默认关闭。GPU 名称见 `data-earth-gpu-renderer`。性能验收同时记录实际 canvas 像素尺寸、可见完整卡片数、卫星图层状态、缩放、直播状态和 30 秒平均 FPS;卡片未完成落位或仅调整到较小视口的读数不属于 1080p 全场景成绩。CPU 阶段计时不等于 GPU 耗时。
|
||||
|
||||
## 详情卡配图
|
||||
|
||||
`card-images.js` 在普通详情与巡航复用的 `showInfoCard` 路径中管理可选配图;桌面与移动详情共用 presenter。选择物件时按稳定标识请求 `/api/v1/earth/card-images/resolve`,卫星使用 NORAD,设施使用 `source:source_id`,海缆和登陆站使用各自来源 ID。数据适配只传身份,型号、分组及分类由后端当前记录解析,不在点击处理器中分散匹配。响应缓存最多 100 个身份、30 秒;同源后台保存通过 storage 版本事件使缓存失效并刷新当前卡片。请求不进入动画帧循环。
|
||||
|
||||
`card-images.css` 同时用于后台预览和真实详情正文。独立绝对定位图片层覆盖右侧 68%,横向与自底向上的渐变通过 `mask-composite: intersect` 相交;WebKit 使用 `source-in`。图片自身不透明度为 0.85,使用 `saturate(1.2) contrast(1.1)`;图片层淡入终点为 1,不再叠加暗色遮罩或整体降透明度。内容在上层,标题保持原样,蒙版不作用于文字。原有 DPI、滚动、拖拽及连线不受图片尺寸影响;图片解码后淡入,reduced-motion 下直接显示。加载失败、无图、关闭或切到其他对象都会隐藏旧图片;异步请求与解码均检查选择代次,避免串图。标题中的图片来源链接提供署名和授权信息。
|
||||
|
||||
后端 `card_image_configuration.py` 用独立的 `system_settings.earth_card_images` 保存资产与关联,沿用配置版本检查、行锁和审计事务,不改采集记录、不建并行采集状态。匹配顺序是物件、型号、分组、分类;专属“无图”阻断回退,停用资产不能被解析;授权状态作为真实来源记录保留,与启用开关分开。重复目标和悬空关联在 schema 层拒绝。默认值来自 `assets/cards/manifest.json`,无需数据库种子;保存时记录已知内置 ID,升级合并新内置项,按清单 previous_defaults 替换旧内置图片地址,同时保留用户修改的属性,保留已编辑或删除的旧项。容器构建同步复制同一清单,避免两份默认配置漂移。
|
||||
|
||||
上传经 Pillow 校验 JPEG/PNG/WebP、尺寸及解码结果,原图保留,另生成 640/1280 边界的 WebP,正文按显示宽度与设备像素比选用。文件按内容哈希命名并原子写入,经公共只读 files 接口分发;管理、上传、目标搜索和预览需管理员权限。文件存储与备份见运维手册。内置 Frontier 实景只绑定该设施;Starlink 为系列示意图,不假定 V2/V3。千帆按用户要求启用系列配图,来源授权仍标记待确认。
|
||||
|
||||
默认分类 `*` 覆盖卫星、算力设施、海缆、登陆站、船舶、BGP 事件、BGP 观测站和自定义点,在具体分类之后匹配。船舶使用 MMSI,BGP 使用 incident_key,观测站使用 collector_id,自定义点使用既有 ID;旧快照缺少身份时仅允许类型通用图,不冒充专属匹配。采集来源的 source_id 若因上游改名发生变化,需要重新关联。
|
||||
|
||||
|
||||
### 五类算力设施与默认配图
|
||||
|
||||
算力类型为 `supercomputer`(超算中心)、`gpu_cluster`(GPU 集群)、`data_center`(数据中心)、`cloud_center`(云计算中心)和 `edge_node`(边缘计算节点)。前端 `compute-site-types.js` 统一标题、类型标签、计数与缺省类型;后端 `compute_types.py` 为 GeoJSON 与图片查询共用分类规则:有效的 metadata.site_type 优先,其次 data_type,最后使用已知采集源的类型。未识别设施按通用数据中心展示,不强制改成 GPU 集群。类型扩展不添加虚构设施或改变已采集记录的分类。
|
||||
|
||||
五类均接入图例独立筛选、地图图标、详情卡、移动详情及图片身份。新类型详情复用设施身份、运营方、算力、功耗和位置字段;不沿用 GPU 数量或 TOP500 排名。主地图接受现有采集来源或明确声明算力类型的当前记录;地区设施查询保留快照时间和国家条件,三类基础设施不要求具有 GPU 算力计量合同,也不据此推算全球算力份额。当前内置采集器仍为 TOP500 / Epoch AI,新增类型不会自动产生记录。
|
||||
|
||||
七张用户指定配图分别作为五类算力设施、BGP 观测站、BGP 事件的默认图。图库分类选项包含五类,无需等待某类已有数据。专属图片优先于类型图;旧内置图片及未自定义的 BGP 观测站关联按 manifest 的 previous_defaults / previous_rules 更新,手工修改的裁切、启用状态与专属绑定保留。
|
||||
|
||||
|
||||
### 保持显示行为的帧计算优化
|
||||
|
||||
聚合展示在一次 update 内按父节点复用世界矩阵和相机局部坐标;跨帧强制刷新缓存,因此旋转、缩放、切换视角和筛选不会使用旧投影。完全收起且没有子 DOM 的聚合成员仍执行隐藏状态维护,但不计算看不见的展开坐标。计数和无变化的 ARIA 文案不重复写入 DOM。
|
||||
|
||||
静态点批次仅在数据、选择、筛选、距离缩放或显式刷新时更新;选中物件的呼吸效果继续逐帧更新覆盖层。颜色缓存只接受不可变的字符串/数字,调用方可变 Color 对象不会被缓存。卫星圆点/光晕只更新其需要的地球父变换,避免各自遍历整棵地球节点。
|
||||
|
||||
轨迹采样合并半径界限和第一轮误差扫描,使用原来的阈值与递归分割结果;未减少卫星数量、轨迹点、分辨率或任何卡片内容。确定性对比检查了 10,000 组历史样本,输出索引一致。局部 Bun 微基准(不是整页 FPS):480 个聚合成员收起约 1.19→0.90 ms,展开约 4.47→2.86 ms;16,621 条轨迹扫描约 11.50→8.94 ms。端到端 FPS 仍需同分辨率、同卡片/卫星/视频状态、前台连续 30 秒单独比较。
|
||||
|
||||
区域新闻卡的折叠按钮显式调用共享 HUD 控制器,正文复用 hud-panel__body--collapsible;aria-expanded 与真实折叠状态同步。仅用户点击触发折叠,布局和性能逻辑不自动隐藏卡片内容。
|
||||
|
||||
物件详情浮窗与区域新闻卡共享 `--hud-detail-card-max-height`:390px 基准乘 HUD 缩放,并受视口减 24px 限制。标题固定,配图正文容器与内容组成 min-height:0 的 flex 链,仅正文滚动;内容少的卡片自然保持较短高度。BGP 事件优先显示事件/严重度/状态/影响区域/时间/摘要;观测站优先身份、状态及近期活跃;算力设施优先名称、算力和硬件,定位核验及来源后置。字段顺序来自共享 CARD_CONFIG,移动详情同步使用同一顺序。
|
||||
|
||||
|
||||
### 详情卡身份与连线一致性
|
||||
|
||||
BGP 观测站以 collector 编号建立标记索引。共享索引不登记空 ID,重复 ID 不参与跨刷新替换;仍在当前图层中的点击对象直接保留自身引用。堆叠展示使用同一个 icon_item_id,避免详情、连线和选中置顶分别采用不同身份。BGP 聚合事件优先使用 incident_key,原始异常使用来源加 ID,防止不同来源的数字编号碰撞。
|
||||
|
||||
卫星详情连线以 NORAD 识别卫星,并在数据数组替换时重新定位下标;选择高亮也在目录刷新后按同一 NORAD 重绑,目标消失则清除选择。轨道位置尚未就绪时不把零坐标当作连线锚点。海缆仍沿被选中海缆自己的几何取锚点,登陆点使用被点击标记,区域新闻/算力卡继续使用其区域锚点。
|
||||
|
||||
|
||||
算力详情配图解析在没有型号/分组规则时使用快速路径:由稳定 source 前缀确定超算或 GPU 分类,先匹配对象规则,再匹配分类默认图,跳过完整采集目录查询。只要管理员配置了型号或分组规则,就自动回退到完整目录解析;这不会改变前端身份、采集数据或其他物件类型。
|
||||
|
||||
@@ -1,84 +1,41 @@
|
||||
# 智能星球可交互图标聚类策略
|
||||
# 智能星球可交互图标聚合
|
||||
|
||||
智能星球的可交互图标由 `createInteractableLayer` 统一管理。图标仍然使用 Three.js `Points` 渲染,拾取、hover、locked、tooltip 和巡航焦点继续依赖 marker 的 `userData`;聚类策略只决定“哪些 marker 被合成一个 cluster dot”。
|
||||
`interactable.js` 负责固定地理成员,`cluster-motion.js` 负责时间补间,`cluster-presentation.js` 负责圆形聚合、展开图标、拾取与 Spiderfy。普通独立标记继续使用 GPU Points;聚合成员从动画开始到终点由同一 DOM 图标显示,不再切回另一套位置。
|
||||
|
||||
## 策略
|
||||
## 成员与缩放
|
||||
|
||||
`cluster.strategy` 支持三种模式:
|
||||
聚合使用真实 `icon_base_position` 和球面分桶,默认距离 15(与地球半径同单位)、至少 2 个成员、每组至多 14 个。数据、筛选、显隐变化使缓存失效;旋转和缩放不改变成员。圆标锚在成员的球面质心,不参与屏幕避让。
|
||||
|
||||
- `stable-spherical`:稳定球面聚类。按地球局部 3D 坐标聚合,并用 zoom band 缓存拓扑;旋转和同档缩放只更新投影和尺寸,不重算谁和谁聚在一起。适合 BGP、超算/GPU 中心和 Earth interactable 这类半静态图层。
|
||||
- `dynamic-screen`:动态屏幕聚类。保留原来的屏幕空间聚类逻辑,每帧按可见投影关系判断。适合船舶等高频实时图层,也可作为稳定策略的回退。
|
||||
- `none`:不聚类。所有 marker 独立显示,适合低数量或需要精确展示的图层。
|
||||
`cluster: { enabled: true }` 启用,`cluster: false` 关闭。可调项为 `distance`、`minCount`、`maxMarkersPerDot`、`disableAboveZoom`(默认 2.5)和 `expansionHysteresis`(默认 0.08)。界面缩放统一通过 `globeZoomForDistance()` 换算,超过 250% 自动展开,低于 230% 收拢;500% 不显示聚合圆标。卫星和 AIS 船只不接入本机制。
|
||||
|
||||
`cluster: false` 等价于 `strategy: "none"`。未显式声明 `strategy` 时,保持兼容行为:开启聚类的旧图层继续走 `dynamic-screen`。
|
||||
## 动画与同坐标展开
|
||||
|
||||
## Stable Spherical
|
||||
展开和收拢统一使用 320ms easeInOutCubic;停止滚轮后仍按时间完成,反向输入从当前进度续接。成员直接从质心插值到真实局部坐标,终点 DOM 保留,因此不存在动画终点与实际图标之间的切换。插值弦线可能暂入球体,显隐使用同方向球壳锚点判断,投影仍使用原插值位置,避免中途被误裁掉。
|
||||
|
||||
`stable-spherical` 的核心是把聚类身份从屏幕距离迁到球面距离:
|
||||
只有同坐标或极近地理锚点(约 0.00015°)跨图层紧凑堆叠;屏幕重叠和城市粗网格不会把不同地点合并。每个图标以自己的真实投影为基准,整叠在每个轴向最多错开 24px,小组露出约 8–12px 点击边缘。稳定 ID 决定初始顺序,选中后排对象时移到前端,取消焦点后恢复原顺序。普通点展开终点严格等于真实投影,堆叠不改变经纬度或 marker 位置。
|
||||
|
||||
- 每个 marker 使用 `icon_base_position` 作为真实地理锚点。
|
||||
- 默认 zoom 超过 `2.5` 时强制关闭聚类,所有 marker 展开为原图标。
|
||||
- 当前 zoom 只映射到离散 band;同一 band 内旋转地球或细微缩放不会重算拓扑。
|
||||
- 跨 band、数据变更、图层显隐变化时才重新计算 cluster。
|
||||
- cluster 质心由成员 3D 坐标平均后 normalize 回球壳半径,因此 cluster dot 刚性贴在地理质心投影上。
|
||||
- cluster dot 不参与 2D 避让,避免被屏幕排斥推离真实地理位置。
|
||||
- band 切换带有少量 hysteresis,避免缩放停在临界点时在两个 band 之间来回跳。
|
||||
- 新生成的 marker / cluster dot 会执行短 scale + opacity ease;这只是渲染过渡,不改变 marker 的真实经纬度、拾取对象或 locked 状态。
|
||||
## 交互与 BGP
|
||||
|
||||
稳定策略使用球面 bucket/hash 邻域查询,禁止用全量双循环。这样大多数帧只承担投影与材质尺寸更新,聚类成本只在 band 或数据版本变化时支付。
|
||||
点击圆标展开并暂停自转。支线有 14px 透明命中区域,悬停同时强调对应支线与子点;点击任一处选择同一业务对象。画布容器统一处理画布和投影层的滚轮,HUD 的滚动不触发地球缩放。
|
||||
|
||||
## 配置示例
|
||||
投影图标生成纹理时烘入原分类色。BGP 事件沿用告警颜色;包含 BGP 的聚合圆标及展开的事件图标使用扩散淡出光环,周期与大小来自 `BGP_CONFIG.ring`。GPU 中被投影替代的事件光环隐藏,避免在旧锚点重复绘制。减少动态效果偏好关闭脉冲动画。
|
||||
|
||||
```js
|
||||
const computeCenterIconLayer = createInteractableLayer({
|
||||
id: "computeCenters",
|
||||
// ...
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
minCount: 2,
|
||||
maxMarkersPerDot: 14,
|
||||
transitionMs: 220,
|
||||
bandHysteresis: 0.08,
|
||||
disableAboveZoom: 2.5,
|
||||
bands: [
|
||||
{ key: "far", maxZoom: 1.7, distance: 15 },
|
||||
{ key: "mid", maxZoom: 2.6, distance: 8 },
|
||||
{ key: "near", maxZoom: 3.5, distance: 4 },
|
||||
{ key: "detail", maxZoom: Infinity, distance: 0 },
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
## 已删除的旧实现
|
||||
|
||||
实时层可以保留动态策略:
|
||||
低缩放圆点的桌面/移动端设置、存储字段、纹理、绘制/拾取分支和翻译均已删除。动态屏幕聚合、离散缩放分档、GPU 聚合圆点与旧的创建时缩放淡入路径也已删除,不提供旧策略回退。
|
||||
|
||||
```js
|
||||
const vesselIconLayer = createInteractableLayer({
|
||||
id: "vessels",
|
||||
// ...
|
||||
cluster: {
|
||||
strategy: "dynamic-screen",
|
||||
enabled: true,
|
||||
maxMarkersPerDot: 10,
|
||||
},
|
||||
});
|
||||
```
|
||||
## 回归验证
|
||||
|
||||
## 调参原则
|
||||
- 320ms 到达真实终点、反向连续、中间帧可见、终点保持同一节点。
|
||||
- 500% 无聚合圆标,同坐标成员可分别选中,源坐标保持不变。
|
||||
- 图层筛选/移除清理失效节点与 GPU 隐藏状态。
|
||||
- 图标和支线上滚轮可缩放,HUD 滚动除外。
|
||||
- 支线 hover 只联动对应子点;投影 BGP 纹理保留分类色。
|
||||
|
||||
- `distance` 是球面 3D 距离阈值,单位与 `CONFIG.earthRadius` 一致。值越大,越容易聚合。
|
||||
- 最远 band 用较大的 `distance` 降低视觉密度;最近 band 通常设为 `0`,让图标完全解散。
|
||||
- `bandHysteresis` 控制 band 边界滞回。值太小容易临界闪烁,值太大会让切换略显迟钝。
|
||||
- `transitionMs` 控制聚散过渡时间。建议保持在 160-260ms,过长会让实时层显得拖泥带水。
|
||||
- `disableAboveZoom` 控制精细查看阈值。默认 `2.5`,超过后不再生成 cluster;设为 `false` 可关闭这个硬阈值。
|
||||
- 高实时性图层优先用 `dynamic-screen`,避免数据频繁变更时触发稳定策略的拓扑重算。
|
||||
- 若某图层出现异常,可临时切回 `dynamic-screen` 或 `cluster: false`。
|
||||
对应测试:`cluster-motion.test.js`、`cluster-presentation.test.js`、`globe-navigation.test.js`、`marker-badges.test.js`。
|
||||
|
||||
## 验收重点
|
||||
## 刷新与首帧准备
|
||||
|
||||
- 同一 zoom band 内旋转地球,cluster 不应闪烁或重新聚散。
|
||||
- cluster dot 应跟随地球表面质心,不被避让逻辑推开。
|
||||
- 放大到 detail band 后,应恢复该图层原本的图标纹理和点击行为。
|
||||
- zoom 超过 250% 后不应再显示 cluster dot。
|
||||
- vessel 等实时层更新后,图标和 cluster 应立即反映最新数据。
|
||||
数据刷新即使沿用聚合 ID,也必须按当前 Object3D 实例清理旧子节点。显示所有权变更立即重绘对应 GPU 批次;聚合拓扑失效同时废弃旧签名,避免旧圆标与独立图标并存。
|
||||
|
||||
展开前先准备可见图标并等待图片解码,再开始 320ms 计时。同一纹理复用编码结果,不为背面的不可见成员创建图标,避免纹理准备耗时吞掉动画。回归覆盖重复刷新/缩放、首次纹理准备超过动画时长、选中前置/取消复位和卡片引线。
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# 智能星球图层样式属性索引
|
||||
|
||||
`EARTH_MATERIAL_CONFIG.surface.night` 集中夜侧纹理对比(detailGamma=0.55)及海陆各自的 tint/ambient 参数。它只参与夜侧混合,不调整日面曝光;沿用现有地理遮罩,无新增贴图或 Draw Call。
|
||||
|
||||
## 现代物件视觉
|
||||
|
||||
- 地表:`EARTH_MATERIAL_CONFIG.surface` 集中 exposure=0.9、landSaturation=0.9、oceanAbsorption=0.36、oceanDeepColor=[0.004,0.012,0.032]、夜面保留与 Fresnel 参数;`atmosphere` 集中单壳半径、颜色和强度。低清基座及地形关闭镜面高光。
|
||||
- 海缆:常态冷色与原分类色混合,图例与线路保持一致;选中后参考流光沿真实线路累计长度移动,6 个周期、速度 4.5。复用同一批量 draw,绘制材质不写深度;原 Line 对象、cableId、选中逻辑不变。
|
||||
- 登陆点:`marker-landing-point.svg` 为 128×128 琥珀微核单色源,取代带高光的程序球;Sprite 拾取和 billboard 批量绘制保持原尺寸逻辑。
|
||||
- 算力:三个规范 SVG 只负责内置符号;`marker-badges.js` 与 `interactable.js` 共同绘制渐变方块底板、浅色边圈、上沿高光与投影。GPU/AI 使用电紫堆叠层,超算使用电紫芯片,数据中心/基础设施使用青绿机柜。估算标志和原拾取/联动保留。
|
||||
- 卫星:真实倾角分类与图例、预测轨道共享鲜明类别色;普通态使用原型硬边圆盘,无衬边或球面明暗,默认不呼吸。覆盖统一蓝色,日照只改变透明度。
|
||||
|
||||
|
||||
|
||||
本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和
|
||||
`renderOrder` 等样式属性。层级关系请配合
|
||||
[智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
|
||||
@@ -23,15 +35,11 @@
|
||||
| 智能星球基座半径 | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
|
||||
| 智能星球基座颜色 | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
|
||||
| 智能星球基座 emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
|
||||
| 智能星球基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
|
||||
| 智能星球基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
|
||||
| 智能星球基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x000000` | `MeshPhongMaterial.specular` |
|
||||
| 智能星球基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `0` | `MeshPhongMaterial.shininess` |
|
||||
| 智能星球基座 opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
|
||||
| 高清材质半径偏移 | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.48` | 独立高清材质球半径;必须与海陆基座和地球基座保持足够深度间距,避免远距 z-fighting |
|
||||
| 高清材质透明度 | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | 高清材质 `MeshPhongMaterial.opacity` |
|
||||
| 高清材质 renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |
|
||||
| 高清材质 specular | `EARTH_MATERIAL_CONFIG.textureOverlaySpecular` | `0x05080d` | 降低直射区域镜面高光,避免贴图死白 |
|
||||
| 高清材质 shininess | `EARTH_MATERIAL_CONFIG.textureOverlayShininess` | `4` | 降低高光集中度 |
|
||||
| 高清材质颜色乘色 | inline | `0xffffff` | `_earthTextureOverlayMaterial.color` |
|
||||
|
||||
## Earth 遮挡与昼夜
|
||||
|
||||
@@ -39,7 +47,7 @@
|
||||
| --- | --- | --- | --- |
|
||||
| 遮挡球半径系数 | `EARTH_MATERIAL_CONFIG.occluderRadiusFactor` | `0.999` | 深度遮挡球半径 |
|
||||
| 遮挡球分段 | `EARTH_MATERIAL_CONFIG.occluderSegments` | `48` | 遮挡球几何分段 |
|
||||
| 遮挡球 renderOrder | inline | `-1` | `occluder.renderOrder` |
|
||||
| 遮挡球 renderOrder | inline | `0.5` | `occluder.renderOrder` |
|
||||
| 昼夜太阳方向 | `EARTH_MATERIAL_CONFIG.dayNight.sunDirection` | `{ x: 1, y: 0.2, z: 0.4 }` | 自定义 day/night shader |
|
||||
| 夜侧最低亮度 | `EARTH_MATERIAL_CONFIG.dayNight.nightFloor` | `0.24` | shader uniform |
|
||||
| 日侧增强 | `EARTH_MATERIAL_CONFIG.dayNight.dayBoost` | `1.12` | shader uniform |
|
||||
@@ -53,30 +61,13 @@
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 内层大气半径系数 | `EARTH_MATERIAL_CONFIG.atmosInnerRadiusFactor` | `1.01` | `atmosInnerGeo` |
|
||||
| 内层大气分段 | `EARTH_MATERIAL_CONFIG.atmosInnerSegments` | `64` | `atmosInnerGeo` |
|
||||
| 内层大气颜色 | `EARTH_MATERIAL_CONFIG.atmosInnerColor` | `[0.25, 0.62, 1.0]` | shader RGB |
|
||||
| 内层大气 rim power | `EARTH_MATERIAL_CONFIG.atmosInnerRimPower` | `3.2` | shader rim |
|
||||
| 内层大气强度 | `EARTH_MATERIAL_CONFIG.atmosInnerIntensity` | `0.18` | shader alpha multiplier |
|
||||
| 外层大气半径系数 | `EARTH_MATERIAL_CONFIG.atmosOuterRadiusFactor` | `1.0025` | `atmosOuterGeo` |
|
||||
| 外层大气分段 | `EARTH_MATERIAL_CONFIG.atmosOuterSegments` | `48` | `atmosOuterGeo` |
|
||||
| 外层大气颜色 | `EARTH_MATERIAL_CONFIG.atmosOuterColor` | `[0.18, 0.45, 0.9]` | shader RGB |
|
||||
| 外层大气 rim power | `EARTH_MATERIAL_CONFIG.atmosOuterRimPower` | `9.0` | shader rim |
|
||||
| 外层大气强度 | `EARTH_MATERIAL_CONFIG.atmosOuterIntensity` | `0.0025` | shader alpha multiplier |
|
||||
| 大气辉光 blending | inline | `THREE.AdditiveBlending` | `ShaderMaterial.blending` |
|
||||
| 大气辉光 renderOrder | inline | `1` | `atmosInner/Outer.renderOrder` |
|
||||
| 无高清材质边缘光颜色 | `EARTH_MATERIAL_CONFIG.rimGlowColor` | `[0.42, 0.72, 1.0]` | 高清材质隐藏或不可用时的 Fresnel shell RGB |
|
||||
| 无高清材质边缘光半径系数 | `EARTH_MATERIAL_CONFIG.rimGlowRadiusFactor` | `1.0035` | `earth-rim-glow` 外扩球壳半径 |
|
||||
| 无高清材质边缘光 rim power | `EARTH_MATERIAL_CONFIG.rimGlowPower` | `3.4` | shader rim 衰减;值越大边缘越窄 |
|
||||
| 无高清材质边缘光强度 | `EARTH_MATERIAL_CONFIG.rimGlowIntensity` | `0.24` | shader alpha multiplier |
|
||||
| 无高清材质边缘光分段 | `EARTH_MATERIAL_CONFIG.rimGlowSegments` | `96` | `earth-rim-glow` 几何分段 |
|
||||
| 无高清材质边缘光 renderOrder | `EARTH_MATERIAL_CONFIG.rimGlowRenderOrder` | `1.08` | `_earthRimGlow.renderOrder` |
|
||||
| 无高清材质边缘光 depthTest | inline | `false` | 避免被海陆基座或地表填充遮住 |
|
||||
| 云图半径偏移 | `CLOUD_LAYER_CONFIG.radiusOffset` | `3` | 云层球半径 |
|
||||
| 云图分段 | `CLOUD_LAYER_CONFIG.widthSegments / heightSegments` | `64 / 64` | 云层球几何分段 |
|
||||
| 云图透明度 | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshPhongMaterial.opacity` |
|
||||
| 云图透明度 | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshBasicMaterial.opacity` |
|
||||
| 云图贴图 | `CLOUD_LAYER_CONFIG.textureUrl` | `"./assets/earth_clouds_1024.png"` | 云层贴图 |
|
||||
| 云图 blending | inline | `THREE.AdditiveBlending` | `MeshPhongMaterial.blending` |
|
||||
| 云图 blending | inline | `THREE.AdditiveBlending` | `MeshBasicMaterial.blending` |
|
||||
|
||||
## 海陆基座与国界
|
||||
|
||||
@@ -100,7 +91,7 @@
|
||||
| 海陆基座 renderOrder | `COUNTRY_BOUNDARY_CONFIG.landRenderOrder` | `0.86` | `country-land-ocean.renderOrder` |
|
||||
| 海陆 mask 尺寸 | `landMaskWidth / landMaskHeight` | `2048 / 1024` | canvas / DataTexture 尺寸 |
|
||||
| 国界 tint 颜色 | `COUNTRY_BOUNDARY_CONFIG.tintColor` | `0x0b1830` | 高清材质关闭时 tint |
|
||||
| 国界 tint 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.04` | `country-tint` 半径 |
|
||||
| 国界 tint 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.32` | `country-tint` 半径 |
|
||||
| 国界 tint renderOrder | `COUNTRY_BOUNDARY_CONFIG.tintRenderOrder` | `0.2` | `country-tint.renderOrder` |
|
||||
| 国界线颜色 | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | 普通国界线 |
|
||||
| 国界线透明度 | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | 普通国界线 opacity |
|
||||
@@ -129,8 +120,8 @@
|
||||
| 地形透明度 | `TERRAIN_CONFIG.opacity` | `0.68` | `MeshPhongMaterial.opacity` |
|
||||
| 地形颜色 | `TERRAIN_CONFIG.color` | `0x8aa884` | `MeshPhongMaterial.color` |
|
||||
| 地形 emissive | `TERRAIN_CONFIG.emissive` | `0x030704` | 降低自发光,恢复地形明暗层次 |
|
||||
| 地形 specular | `TERRAIN_CONFIG.specular` | `0x344438` | 给地形局部光泽,不抬高高清贴图直射亮度 |
|
||||
| 地形 shininess | `TERRAIN_CONFIG.shininess` | `16` | 收紧地形高光,增强起伏辨识 |
|
||||
| 地形 specular | `TERRAIN_CONFIG.specular` | `0x000000` | 给地形局部光泽,不抬高高清贴图直射亮度 |
|
||||
| 地形 shininess | `TERRAIN_CONFIG.shininess` | `0` | 关闭镜面高光 |
|
||||
| 地形 renderOrder | inline | `1.2` | `terrain.renderOrder` |
|
||||
| 地形 polygonOffset | inline | `factor -1`, `units -1` | 降低贴近球面时的闪烁 |
|
||||
|
||||
@@ -154,11 +145,11 @@
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 默认海缆颜色 | `CABLE_COLORS.default` | `0xffff44` | 无数据颜色时使用 |
|
||||
| 海缆半径偏移 | `CABLE_CONFIG.line.altitudeOffset` | `0.2` | 海缆线半径 |
|
||||
| 海缆半径偏移 | `CABLE_CONFIG.line.altitudeOffset` | `0.48` | 海缆线半径 |
|
||||
| 海缆线宽 | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
|
||||
| 海缆透明度 | `CABLE_CONFIG.line.opacity` | `1.0` | 海缆线 opacity |
|
||||
| 海缆 renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | 海缆线层级 |
|
||||
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | 与海缆线同层贴地,避免凌空 |
|
||||
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | 与海缆线同层贴地,避免凌空 |
|
||||
| 登陆点 sprite 高度 | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` 基准高度 |
|
||||
| 登陆点缩放参考 FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | 与当前 Earth 相机 FOV 一致 |
|
||||
| 登陆点缩放下限 | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | 地球放到 200% 之后的最小倍率,限制高倍 zoom 下的屏幕占比;`3 * 0.16 = 0.48` |
|
||||
@@ -168,7 +159,7 @@
|
||||
| 登陆点透明度 | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
|
||||
| 登陆点 renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | 与海缆线同层;`depthTest: false` 保持球体完整,背面通过相机到球心的球体遮挡判断隐藏 |
|
||||
| 登陆点 dim 亮度系数 | `landingPointVisual.dimBrightness` | `0.62` | dim 状态颜色乘数 |
|
||||
| 相关登陆点高亮 opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | 高亮脉冲 |
|
||||
| 关联登陆点光晕范围 | `landingPointVisual.related.haloScale / haloFalloff` | `1.65 / 8` | 仅扩大光晕画布,圆点大小和颜色不变 |
|
||||
| 非相关登陆点颜色 | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | dim 状态颜色,避免黑色基座透出成暗洞 |
|
||||
| 非相关登陆点 opacity | `landingPointVisual.dimmed.opacity` | `0.78` | dim 状态透明度,不再用低 alpha 混黑底 |
|
||||
|
||||
@@ -180,17 +171,12 @@
|
||||
| 卫星真实高度压缩尺度 | `SATELLITE_CONFIG.altitudeCompressionKm` | `1200` | TLE/SGP4 高度压缩映射,数值越大高轨差异越温和 |
|
||||
| 卫星真实高度上限 | `SATELLITE_CONFIG.maxDisplayAltitudeKm` | `40000` | 压缩映射的高度截断,覆盖 GEO 附近高度 |
|
||||
| 卫星真实高度显示偏移范围 | `minRealAltitudeOffset / maxRealAltitudeOffset` | `4 / 25` | 压缩后映射到 `CONFIG.earthRadius + offset`;`25` 约为当前地球半径 `100` 的四分之一,用于保留高轨分层同时避免 GEO/MEO 轨迹离主体过远 |
|
||||
| 卫星点基础像素大小 | `SATELLITE_CONFIG.dotBaseSize` | `2.8` | 点 shader size |
|
||||
| 卫星背景点缩放 | `SATELLITE_CONFIG.dotBackdropScale` | `1.28` | 背景点大小 |
|
||||
| 卫星点透明度范围 | `dotOpacityMin / dotOpacityMax` | `0.42 / 1.0` | 呼吸动画 |
|
||||
| 卫星点呼吸速度 | `SATELLITE_CONFIG.dotBreathingSpeed` | `0.12` | 点 opacity 动画 |
|
||||
| 卫星背景点颜色 | inline | `0x0b1626` | backdrop point baseColor |
|
||||
| 卫星背景点透明度 | inline | `0.42` | backdrop point opacity |
|
||||
| 卫星点透明度 | inline | `0.9` | point material opacity |
|
||||
| 卫星背景点 renderOrder | inline | `5` | `satelliteBackdropPoints.renderOrder` |
|
||||
| 卫星点基础像素大小 | `SATELLITE_CONFIG.dotBaseSize` | `2.2` | 旧交互命中半径计算基准;不控制视觉尺寸 |
|
||||
| 卫星点透明度范围 | `dotOpacityMin / dotOpacityMax` | `0.72 / 1.0` | 选中点呼吸动画;普通点默认不呼吸 |
|
||||
| 卫星点透明度 | inline | `1.0` | 硬边圆盘普通态 |
|
||||
| 卫星点 renderOrder | inline | `6` | `satellitePoints.renderOrder` |
|
||||
| 卫星轨迹长度 | `SATELLITE_CONFIG.trailLength` | `10` | trail buffer |
|
||||
| 卫星轨迹线宽 | `SATELLITE_CONFIG.trailLineWidth` | `3` | ribbon shader uniform |
|
||||
| 卫星轨迹线宽 | `SATELLITE_CONFIG.trailLineWidth` | `3`(上限) | 实际宽度同时限制为点径的 60%,保留渐细短拖尾 |
|
||||
| 选中 ring 大小 | `SATELLITE_CONFIG.ringSize` | `0.07` | hover / locked ring sprite |
|
||||
| 卫星覆盖层 renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | locked ring / halo / orbit |
|
||||
| 自发光选中点颜色 | inline default | `"#ffd25a"` | `showSelfGlowStyle()` |
|
||||
@@ -235,8 +221,8 @@ AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。
|
||||
| hover 缩放 | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | hover overlay 尺寸倍率 |
|
||||
| locked 缩放 | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | locked overlay 尺寸倍率,并叠加 pulse |
|
||||
| dimmed 缩放 / 透明度 | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | dim 状态 |
|
||||
| 超算颜色 | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | marker texture |
|
||||
| GPU 集群颜色 | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | marker texture |
|
||||
| 超算颜色 | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#a855f7"` | marker texture |
|
||||
| GPU 集群颜色 | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#a855f7"` | marker texture |
|
||||
| 关联颜色 | `COMPUTE_CENTER_CONFIG.colors.linked` | `"#f8fafc"` | 关联态 |
|
||||
| 算力中心 renderOrder | local `COMPUTE_CENTER_RENDER_ORDER` | `4.5` | 地表设施低于卫星点 |
|
||||
|
||||
@@ -282,7 +268,54 @@ AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。
|
||||
| 月亮 halo 缩放 | `CELESTIAL_CONFIG.moonHaloScale` | `62` | 月亮 halo |
|
||||
| 太阳光颜色 / 强度 | `sunLightColor / sunLightIntensity` | `0xfff4df / 1.02` | scene light |
|
||||
| 背光颜色 / 强度 | `backLightColor / backLightIntensity` | `0x2b4c78 / 0.3` | scene light |
|
||||
| 星空点数量 | `STARFIELD_CONFIG.count` | `8000` | `createStars()` |
|
||||
| 星空半径范围 | `minRadius + radiusJitter` | `800 + 200` | 随机分布 |
|
||||
| 星空点颜色 | `STARFIELD_CONFIG.color` | `0xffffff` | `PointsMaterial.color` |
|
||||
| 星空点大小 | `STARFIELD_CONFIG.size` | `0.5` | `PointsMaterial.size` |
|
||||
|
||||
|
||||
## 方块徽标分类与配色
|
||||
|
||||
| 类型 | 主色 | 内置图标 |
|
||||
| --- | --- | --- |
|
||||
| 数据中心/基础设施 | 青绿 `#14b8a6` → `#0d9488`,边框 `#5eead4` | 服务器机柜 |
|
||||
| GPU/AI | 电紫 `#a855f7` → `#7e22ce`,边框 `#d8b4fe` | 堆叠算力层 |
|
||||
| 超级计算机 | 同一电紫 | 芯片 |
|
||||
| 海缆登陆点 | 琥珀 `#f59e0b`,浅金 `#fef3c7` 边圈 | 发光圆核 |
|
||||
| BGP 观测站 | 蓝色及原活动热度色 | 圆角矩形底板内置广播塔 |
|
||||
| BGP 事件 | 低/中/高/严重:蓝/琥珀/橙/红 | 不规则扁平符号(三角、感叹号、波动、爆发、泄漏),不使用方块 |
|
||||
| 船舶 | 货轮蓝、油轮橙、客船青绿、渔船琥珀、军舰靛紫、其他银灰 | 独立航向箭头,无徽标底板;停泊为圆点,悬停/选中发光 |
|
||||
| 自定义标记 | 地点绿、备注黄、告警红、默认蓝 | 地点针、备注页、告警三角 |
|
||||
|
||||
BGP 变化事件保持不规则扁平符号作为唯一例外,观测站使用圆角徽标;其他有限地表物件使用缓存的徽标纹理,不为每个图标新增 DOM 或独立绘制对象。船舶继续按航向分桶,以顶点色区分船型,不将船型与航向相乘扩张批次数。卫星使用参考的鲜明类别色与无衬边硬圆盘;轨道和覆盖计算不变。低缩放圆点的设置、持久化字段和绘制/拾取分支已删除。
|
||||
|
||||
|
||||
### 卫星统一色板
|
||||
|
||||
参考 [satellitemap.space](https://satellitemap.space/) 的类别色;本地 3d.html 中的模拟壳层与数量不作为项目分类依据。当前项目仍按真实轨道倾角分类:0–30° 蓝 `#3399ff`、30–60° 橙 `#ff801a`、60–90° 绿 `#00e680`、90–120° 紫 `#cc33ff`、120–180° 玫红 `#ff3366`,未知灰 `#94a3b8`。同一颗卫星跨昼夜保持同一颜色,预测轨道不再使用无关的黄紫渐变。
|
||||
|
||||
覆盖统一 `#3399ff`,opacity=0.34,日侧透明度增益=1.65。普通点不带深色衬边,单批 `PointsMaterial` 使用 alphaTest=0.5;视觉尺寸由 `satellitePixelSize()` 在 2.4–8.0 之间平滑变化。
|
||||
|
||||
### 选中轨道与面板轮廓
|
||||
|
||||
预测轨道绘制完整周期的开放曲线,不追加首点,不截短采样范围或隐藏端部;按累计弧长沿未来采样方向推进亮色流光,复用同一 Line 绘制。电视与通用 HUD 外壳恢复方角,播放器裁剪区域同样为方角;不改变窗口尺寸与拖拽。发布代码移除临时逐帧性能统计,不保留只用于验收的 DOM 诊断接口。
|
||||
|
||||
### 登陆点、侧面徽标与星空微调
|
||||
|
||||
海缆与登陆点共用 R+0.48、renderOrder=1,均通过 `cable-surface.js` 的地表三角网格坐标生成,避免理想球与实际地表之间的端点视差。普通点为黄色实心圆与浅色圈,不带常驻光晕;只有 `related` 点在原有合批 Shader 中产生轻微脉动光晕,其他点变暗,取消高亮恢复普通黄色点。
|
||||
|
||||
圆角徽标改为锚点遮挡:`surface-marker-occlusion.js` 在 GPU 判断锚点是否被球体挡住,正面徽标整块显示、背面整块隐藏;屏幕拾取复用相同判断。仅徽标层关闭逐像素深度裁切;BGP 变化事件保持原扁平符号与深度行为,观测站保留活动热度色并使用圆角底板。悬停、选中和聚合徽标使用同一遮挡路径。
|
||||
|
||||
星空纹理材质独立使用 `CELESTIAL_CONFIG.skyBrightness = 1.0`;不改变地球曝光、天体灯光或业务图层亮度。
|
||||
|
||||
船只为独立移动对象,恢复徽标改造前的航向箭头、停泊圆点和悬停/选中光晕,不接入固定地表物件的圆角底板及整块徽标遮挡路径。
|
||||
|
||||
登陆点普通态和关联态均保持橙色内芯、黄色外圈;关联态只扩大外侧柔光区域并脉动,不改变核心颜色、透明度或尺寸。取消关联后光晕归零;非关联点继续使用暗态。
|
||||
|
||||
|
||||
### 星空颜色输出
|
||||
|
||||
星空贴图按 sRGB 解码;`celestial.js` 通过共享 `preserveTextureOutput()` 仅为星空材质补齐屏幕 sRGB 输出。Three.js r128 的全局 renderer 仍保持既有线性输出,避免改变地球、卫星及其他图层。`skyBrightness=1.0` 表示原图亮度;缺少输出编码时,解码后的暗部会再次以线性数值显示,不能靠提高倍率修正。验证应固定星图、相机与视角,对比渲染像素和 1.0/1.5 两档响应,不能只检查加载成功。
|
||||
|
||||
卫星 hover 小圈直接使用当前卫星的倾角类别色,与粒子和图例一致;切换悬停对象时重新取色,位置更新不覆盖颜色。
|
||||
|
||||
所有地表卡片统一使用根 HUD DPI 比例。新闻区域卡片与巡航新闻复用标题图标、配色、字号及内边距;区域版精简元信息并增加独立轮播控件。展开卡片保留完整内容,其余区域使用胶囊,悬停以相同卡片样式临时展开;背面胶囊渐淡并缩小,不隐藏业务对象。细线/粗线引导线共用 `CalloutConnector`,粗线增加白色线宽与深色描边,端点同步放大。算力指标数字、趋势图及标题图标随 HUD 缩放。欧洲使用小型 EU 字标,其他区域使用受同一比例约束的代表 emoji。
|
||||
|
||||
算力图标按五类选择:超算芯片、GPU 层叠、数据中心机柜、云计算云朵、边缘节点连接。几何分别来自 compute-supercomputer.svg、compute-gpu-cluster.svg、compute-hdd-network.svg、compute-cloud.svg、compute-edge.svg,保持算力图层紫色底板与原有选中行为。
|
||||
|
||||
@@ -84,6 +84,17 @@ Earth 态势新闻使用 `/api/v1/news/earth-feed` 输出给前端。新闻源
|
||||
|
||||
## 配置与缓存
|
||||
|
||||
`GET /api/v1/news/earth-feed` 读取已保存新闻。RSS 补抓由调度器的
|
||||
`earth_news_rss_refresh` 作业在启动时及每 10 分钟执行,按区域检查新鲜度和
|
||||
Feed 覆盖,并通过 Redis 锁合并多进程抓取。失败 Feed 按来源配置的冷却时间
|
||||
重试,其他来源继续处理。仅更新时间戳的健康记录不会广播新闻内容变化。
|
||||
|
||||
`POST /api/v1/news/refresh` 接收区域或视角坐标,返回 HTTP 202 和
|
||||
`refresh_id`,同区域进行中的手动刷新共享该标识。
|
||||
`GET /api/v1/news/refresh/{refresh_id}` 返回等待、执行、成功或失败状态。
|
||||
前端在成功后重新读取新闻并更新面板,再显示刷新成功。刷新期间保留已有内容,
|
||||
失败后恢复按钮并允许重试。保存或重置新闻源配置也会立即安排后台检查。
|
||||
|
||||
`GET /api/v1/earth/news-sources` 返回默认或已保存配置。`PUT /api/v1/earth/news-sources` 保存配置并递增 `cache_version`,同时清理进程内 region cache。`POST /api/v1/earth/news-sources/reset` 恢复默认源。`POST /api/v1/earth/news-sources/test` 只测试单个 RSS/Atom/Aggregated 源,不写入新闻表。
|
||||
|
||||
## 手动新闻内容
|
||||
@@ -160,13 +171,13 @@ Web 星球端的新闻类型按钮只保存当前浏览器的显示偏好;偏
|
||||
|
||||
未指定 `sources` 时,服务层会优先选择当前 `locale` 已有可展示标题和摘要的新闻,并从当前启用来源补齐候选后做来源轮转,避免一个来源的最新待处理条目占满默认 12 条。指定 `sources` 时保持精确来源过滤,不做跨来源补齐。数据库查询层只负责区域、类型、来源和排序条件,不包含语言展示策略。
|
||||
|
||||
源测试只证明当前 RSS/Atom/XML 能解析到条目,不等于这些条目已经入库展示。展示链路还会检查区域、类型过滤和数据库新鲜度。保存或重置新闻源会递增配置版本并清理缓存;如果当前启用的 Feed 子项在库里没有近期条目,下一次 `earth-feed` 请求会补抓,避免新启用的 36氪、亿邦被旧 Google News 缓存挡住。
|
||||
源测试只证明当前 RSS/Atom/XML 能解析到条目,不等于这些条目已经入库展示。展示链路仍检查区域和类型过滤。保存或重置新闻源会递增配置版本、清理缓存并立即安排所有区域的后台刷新;缺少近期条目的 Feed 由后台补抓,`earth-feed` 请求保持只读。
|
||||
|
||||
## 区域均衡与异步精修
|
||||
|
||||
`earth_news_items` 是 EarthFeed 的当前状态表。接口在没有显式 `sources` 过滤时会先按 Breaking、区域、发布时间排序,再做区域轮转,避免亚太或任一高频来源把全球视图和巡航队列全部占满。默认区域顺序是美洲、欧洲、中东与非洲、亚太、全球;未知区域只在已知区域之后参与轮转。区域视图仍遵守“当前区域 + global”的规则,不会把其它区域混进区域面板。
|
||||
|
||||
`items` 和 `cruise_items` 都会进入目标位置与本地化精修队列,但前端不能等 AI 完成后再展示。标题或摘要没有目标语言翻译时,Web Earth 先显示原始 `title / summary`,避免卡片出现“新闻汉化中”而实际内容已经可读。后台完成翻译、分类、Breaking 或目标坐标后,会更新同一条 `earth_news_items` 并通过 Earth news reload / patch 刷新前端。
|
||||
补抓和内容写入链路负责提交目标位置与本地化精修,`earth-feed` 读取不再创建这些任务。前端不能等待 AI 才绘制已有内容;展示语言仍受当前语言的可用内容过滤,英文模式不回退到中文原文。后台完成翻译、分类、Breaking 或目标坐标后,会更新同一条 `earth_news_items` 并通过 Earth news reload / patch 刷新前端。
|
||||
|
||||
目标位置队列使用 Redis Streams 两级队列:
|
||||
|
||||
@@ -236,7 +247,11 @@ flowchart LR
|
||||
ManualAPI --> Store
|
||||
|
||||
Earth["Earth 新闻面板"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||||
NewsAPI --> Resolver["Source Resolver"]
|
||||
NewsAPI --> Config
|
||||
Earth --> RefreshAPI["POST /api/v1/news/refresh"]
|
||||
RefreshAPI --> Scheduler["RSS refresh scheduler"]
|
||||
ConfigAPI --> Scheduler
|
||||
Scheduler --> Resolver["Source Resolver"]
|
||||
Resolver --> Config
|
||||
Resolver --> Cache["Region Feed Cache"]
|
||||
Resolver --> Fetcher["RSS / Atom Fetcher"]
|
||||
|
||||
@@ -12,18 +12,18 @@
|
||||
|
||||
## 地表图层栈
|
||||
|
||||
算力专题沿用原算力中心 Interactable 图标、深度与层次顺序;国家/年份切换会替换该层数据并清除旧选中状态。专题 HUD 属于屏幕 UI,在 `main.js` 的指针排除列表中,不参与地球 picking;区域数据卡在屏幕空间由地理锚点牵引,背面不画穿球连线,设施详情复用原详情卡和移动端详情抽屉。外层大气壳保持原有实现;星空逐字节恢复到 commit `83a10a6c`,`brightStarOpacity` 为 `.88`,删除 `skyTint`/`skyBrightness`,不再校准密度和颜色。高清 overlay 使用白色 `MeshBasicMaterial`、opacity 1、`toneMapped:false`;`texture-colors.js` 统一选择 r128 的 `encoding`/`LinearTosRGB` 或 r160 的 `colorSpace`/colorspace fragment。不叠加 Phong 白光、彩色暮光、蓝色夜染、camera boost 或 shoulder 曲线。没有新增后处理、GPU draw pass 或高分辨率纹理;算力专题新增 45 个本地 SVG 旗帜资源。
|
||||
算力专题沿用原算力中心 Interactable 图标、深度与层次顺序;国家/年份切换会替换该层数据并清除旧选中状态。专题 HUD 属于屏幕 UI,在 `main.js` 的指针排除列表中,不参与地球 picking;区域数据卡在屏幕空间由地理锚点牵引,背面不画穿球连线,设施详情复用原详情卡和移动端详情抽屉。高清地表由 `earth-surface-material.js` 的单次绘制 `ShaderMaterial` 负责,保持原有 R+0.48 网格、深度策略和地表拾取对象。四张 8K 底图仍由原控制器切换,成功后更新采样 uniform 并释放旧纹理,失败保留原图。法线、视线和真实太阳方向统一到世界坐标;土地哑光,海洋只有掠射角冷色反射,没有白色镜面高光。海洋区分优先复用现有海陆地理纹理,未加载时使用保守色差估计,冰雪抑制反射;关闭国界线不移除光学遮罩。原始 JPEG 不重绘、不降采样。 颜色纹理标记 sRGB 并启用 mipmaps。r128 在地表 Shader 中解码 sRGB,r160 由纹理硬件格式解码;二者在线性空间合成光照后只编码一次。局部 filmic 曲线只作用于地表,不打开全局 ACES 曝光,避免改变星空和业务图层。晨昏范围为法线点积 [-0.12, 0.28],夜面在线性色调映射后保留 0.12–0.32 光照,避免暗部被曲线再次压黑。大气收敛为 R×1.025 的一个 BackSide/Additive 壳,depthWrite=false、depthTest=true,内沿散射并入地表;云图独立开关,使用哑光昼夜材质并在 1.1 顺序绘制。星图使用已有 `starmap_deep_8k.jpg`,不加染色遮罩。
|
||||
|
||||
| 顺序 | 图层 | 来源 | 渲染 / 半径策略 | 深度 / 交互策略 | 备注 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| -1000 | 天体背景 mesh | `celestial.js` | 背景球 | 不参与地表拾取 | 位于所有智能星球内容之后。 |
|
||||
| -1 | 智能星球遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
|
||||
| 0.5 | 智能星球遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
|
||||
| 0 | 智能星球基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
|
||||
| 0.2 | 国界暗色 tint | `country-boundaries.js` | `tintAltitudeOffset` | 禁用 raycast | 高清材质关闭时使用。 |
|
||||
| 0.2 | 国界暗色 tint | `country-boundaries.js` | `tintAltitudeOffset = 0.32`,与海陆填色共用网格及半径 | depthTest=true、depthWrite=false,禁用 raycast | 加载占位及高清材质关闭时使用;填色就绪后按 renderOrder 覆盖。 |
|
||||
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset = 0.32`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用;半径与基座球拉开以避免远距 z-fighting。 |
|
||||
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充;半径必须高于海陆基座并与基座球保持足够间距。 |
|
||||
| 1 | 大气辉光和云图 | `earth.js` | 大气 / 云层球 | 不走普通对象选择路径 | 云图由“大气云图”图层开关控制。 |
|
||||
| 1 | 海缆 / 登陆点 | `cables.js`, `cable-batches.js` | 海缆全量线段合并为一个 `LineSegments`;登陆点全量使用实例化 billboard;`renderOrder = 1`、海缆半径偏移仍为 `0.2`;登陆点锚点改为高清地表的 `0.48` | 原始 `Line` / `Sprite` 仅作为逐项拾取和选择状态对象,材质不再单独绘制;登陆点仍按球体遮挡判断背面可见性,批量材质 `depthTest: false` | 样式通过线缆样式纹理和登陆点实例属性同步;保留颜色、脉冲、尺寸、显隐与点击语义。 |
|
||||
| 1 / 1.1 | 大气辉光 / 云图 | `earth.js` | 单层大气 R×1.025;云图 R+3 | 大气 BackSide、depthWrite=false、depthTest=true;保留云图开关 | 内沿光照并入高清地表,不新增全屏后处理。 |
|
||||
| 1 | 海缆 / 登陆点 | `cables.js`, `cable-batches.js` | 海缆全量线段合并为一个 `LineSegments`;登陆点全量使用实例化 billboard;`renderOrder = 1`、海缆与登陆点统一使用高清地表 `0.48` 偏移和同源三角表面坐标 | 原始 `Line` / `Sprite` 仅作为逐项拾取和选择状态对象,材质不再单独绘制;登陆点仍按球体遮挡判断背面可见性,批量材质 `depthTest: false` | 样式通过线缆样式纹理和登陆点实例属性同步;保留颜色、脉冲、尺寸、显隐与点击语义。 |
|
||||
| 1.2 | 真实地形 | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` 加地形位移 | 禁用 raycast | 地形压过高清材质;高清材质关闭时临时隐藏,重新开启后恢复原状态。 |
|
||||
| 2.05 | 经纬线 | `earth.js` | `CONFIG.earthRadius + 0.14` | 禁用 raycast | 低透明度显示在高清材质上。 |
|
||||
| 0.97–0.99 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`;claim 线不再额外抬高 | `depthTest: true`,禁用 raycast | 线层使用独立 line geometry 与 `renderOrder` 控制,但半径与高清材质壳完全一致,避免转动地球时与高清贴图出现视差。 |
|
||||
@@ -32,17 +32,16 @@
|
||||
| 3 | 卫星 footprint 填充 / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested;Iridium adapter 的 fill / ring 也使用同一 renderOrder | Footprint 在 land / texture / terrain 和国界线之上,但在算力中心和卫星之下。 |
|
||||
| 3-4.5 | BGP 观测站、事件扩散圈和事件 marker | `bgp.js`, `interactable.js` | BGP 观测站和事件 marker 均使用 `Interactable` 批量 `THREE.Points`;事件 marker 使用 `BGP_EVENT_RENDER_ORDER = 4.5`;观测站主图标使用 `BGP_COLLECTOR_RENDER_ORDER = 4.4` 和 `BGP_CONFIG.collectorAltitudeOffset = 0.2`;事件 overlay 进入 `bgp-event-overlay-layer`;观测站 halo 和覆盖扇形进入 `bgp-collector-radar-layer` | BGP 事件和观测站都通过 `Interactable` 屏幕空间 picking,并参与同坐标避让 | BGP 观测站主图标与船只同层;BGP 事件与算力中心同层;向外扩散圈、观测站雷达/覆盖动画继续由 BGP 业务逻辑驱动。 |
|
||||
| 4.3 | AIS 船只轨迹线 | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`;`CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | 跟随船只显隐,不单独参与拾取 | 选中船只后显示最近轨迹,低于船只 marker。 |
|
||||
| 4.4 | AIS 船只 marker | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`;业务高度为 `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`;普通 marker 为分桶 `THREE.Points`,hover / locked 为单点 `THREE.Points` overlay | `depthTest: true`;`main.js` 使用屏幕空间 picking,只取正面 marker;`cluster: false` 且 `avoidance: false` | 航行船只用三角点纹理,停泊/低速用圆点;密集海域允许重叠,不参与动态屏幕聚类,避免旋转时重建 Points;低于算力中心 `4.5`。 |
|
||||
| 4.4 | AIS 船只 marker | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`;业务高度为 `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`;普通 marker 为分桶 `THREE.Points`,hover / locked 为单点 `THREE.Points` overlay;使用锚点整块遮挡,与 BGP/算力图标在地球边缘保持同一显隐策略 | `depthTest: false`,由锚点前后显隐控制;`main.js` 使用屏幕空间 picking,只取正面 marker;`cluster: false` 且 `avoidance: false` | 航行船只用三角点纹理,停泊/低速用圆点;密集海域允许重叠,不参与动态屏幕聚类,避免旋转时重建 Points;低于算力中心 `4.5`。 |
|
||||
| 4.5 | 算力中心 | `compute-centers.js`, `interactable.js` | 使用 `COMPUTE_CENTER_RENDER_ORDER` 并由 `Interactable` 绘制 | 通过 `Interactable` 屏幕空间 picking,参与同坐标避让 | 地表设施层,保持在卫星下方。登陆点已下沉到海缆层。 |
|
||||
| 5 | 卫星背景点 | `satellites.js` | 固定 renderOrder;默认按 TLE/SGP4 真实高度压缩到 `CONFIG.earthRadius + 4..25`,关闭真实高度或传播失败时回退到 `fallbackAltitudeOffset = 8` | 屏幕空间卫星拾取 | 位于卫星点下方。 |
|
||||
| 6 | 卫星点 | `satellites.js` | 与卫星背景点使用同一压缩高度 / fallback 高度 | 屏幕空间卫星拾取 | 卫星点压过 footprint 和算力中心。 |
|
||||
| 12+ | 卫星锁定 ring、halo、预测轨道 | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` 及偏移;预测轨道使用同一真实高度开关,并固定锁定时刻的地球姿态来绘制闭合惯性轨道;关闭真实高度时回到同层球面 | 卫星覆盖层路径 | 用于选中 / 锁定卫星强调。 |
|
||||
| 6 | 卫星点 | `satellites.js` | SGP4 高度压缩为 R+4..25;回退高度 R+8 | `depthTest=false`;顶点着色器以 R×0.998 球体测试相机到卫星的线段遮挡,屏幕空间拾取 | 可见点完整显示,真正被球体遮挡的点移出裁剪空间,避免近地平线逐像素裁切。 |
|
||||
| 12+ | 卫星锁定 ring、halo、预测轨道 | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` 及偏移;预测轨道使用同一真实高度开关,并固定锁定时刻的地球姿态来绘制 完整周期开放惯性轨道与方向流光;关闭真实高度时回到同层球面 | 卫星覆盖层路径 | 用于选中 / 锁定卫星强调。 |
|
||||
| 98-100 | 太阳 / 月亮 halo 和 sprite | `celestial.js` | 固定 renderOrder | 天体拾取禁用 | 前景天体 sprite。 |
|
||||
|
||||
## 全量绘制与更新约束
|
||||
|
||||
- 合批只改变 GPU 提交方式,不减少卫星、线段或登陆点数量,也不按相机半球裁剪数据。海缆每对相邻顶点都保留,拾取仍返回原始业务对象。
|
||||
- 卫星普通点与背景点保持两个 `Points` 绘制;呼吸动画由顶点着色器使用静态逐点参数和每帧统一时间计算。hover / locked 的隐藏标记仍由原有选择状态控制。
|
||||
- 卫星普通态为单批硬边圆盘 `Points`,无背景点批次;可选呼吸仍由顶点着色器计算,默认关闭。分类过滤与 hover / locked 的点隐藏保持统一。
|
||||
- `satellite-position-worker.js` 使用与主线程相同的 Three.js / SGP4 版本,通过 `satellite-propagation.js` 共享轨道、显示高度和 fallback 计算;全量位置及初始轨迹以可转移数组交给主线程。主线程维持一个在途计算,不堆积过期帧;重载、清空和高度模式变更会终止旧 Worker。
|
||||
- Worker 启动失败或超过 `SATELLITE_CONFIG.workerStartupTimeoutMs` 时退回共享的同步计算,避免图层无限等待。计数、数据接口、原有更新周期和轨迹长度不变。
|
||||
- 验证时同时检查全量 draw range、逐项拾取、锁定覆盖层、轨迹、背面遮挡、开关及清空后的资源释放;比较帧耗时必须使用相同数据、相同视角和分辨率。
|
||||
@@ -90,3 +89,22 @@ Earth 的地表不是单一 mesh,而是多层近似同心球:基座球、海
|
||||
### 启动加载与绘制顺序
|
||||
|
||||
运行时优先完成选中的完整高清地表,再启动相互独立的数据图层请求;国界构建不再阻塞地表首帧。该启动优化不改变各图层的 renderOrder、深度策略、几何精度或贴图分辨率。数据刷新不清除当前地表,纹理仅在替换成功或场景销毁时释放。
|
||||
|
||||
### 国界加载态菱形格回归防护
|
||||
|
||||
`v0.58.0` 曾将海陆填色抬至 R+0.32、高清贴图抬至 R+0.48,但遗留的占位 tint 仍为 R+0.04、64×32 网格。它与 128×128 主球的弦面交叉,会在海陆/高清纹理未就绪时形成纬圈菱形格;加载后只被上层盖住,并非问题消失。实测旧几何在采样处切入主球约 0.153。
|
||||
|
||||
基座、占位 tint、海陆填色和高清地表统一由 `createEarthSurfaceGeometry()` 使用相同 UV 三角网格。tint 和 land 共用 R+0.32;两者不写深度,靠原 renderOrder 完成填色替换,高清贴图仍为 R+0.48。遮挡球、地形、云图、国界拾取及政治显示策略不变。不能仅靠改 renderOrder、关闭深度测试或隐藏加载球掩盖几何穿插。
|
||||
|
||||
`earth-surface.test.js` 用实际创建的加载态 tint 做径向相交检查,并在 50% / 100% / 500% 下验证至少两个 24-bit 深度单位的前后余量。浏览器验收需覆盖延迟贴图、国界开关、远近距旋转和填色/高清就绪后的状态。
|
||||
|
||||
### 地表标记对齐与侧面显隐
|
||||
|
||||
海缆和登陆点现在统一使用 R+0.48 及 `earthSurfacePoint` 三角表面投影,并保持 renderOrder=1。登陆点关联光晕进入原实例批次的 emphasis 属性,不增加独立 Sprite 或 draw call。侧面的完整圆角徽标和船舶航向箭头采用 GPU 锚点射线/球体遮挡:锚点可见则整个 billboard 绘制,锚点被地球挡住则整块裁除。CPU 拾取使用同样的射线判断,避免点到背面图标。选中 BGP 事件的静态覆盖光晕使用事件图标相同的 R+0.48 锚点、前景顺序和整块前后显隐;BGP 发散圆环仍保留原有独立动画语义。
|
||||
|
||||
|
||||
聚合投影层 `.map-clusters` 的 z-index 为 8,低于 HUD;以真实地表锚点做整标遮挡。普通点仍使用原 Three.js 图层,过渡中的成员只在投影层显示一次,避免双图标。Spiderfy 引线有透明加宽命中区域,悬停联动高亮引线与子点,点击返回同一业务对象;滚轮在图标与引线上继续缩放地球。
|
||||
|
||||
跨图层同坐标/极近坐标标记在同一投影层内紧凑堆叠,不以屏幕碰撞合并不同地点;选中项使用投影子点 `z-index: 1000`,普通未聚合图标的锁定覆盖层使用共享的前景 `renderOrder >= 100`,因此跨图层重叠时仍由焦点对象在最上层;取消后恢复稳定 ID 顺序。聚合徽标在展开过渡期间保持低于选中子点。`icon_presentation_screen` 仅供详情引线跟随显示偏移,真实锚点仍由原 marker 持有;隐藏状态变化立即重建对应 GPU 批次。
|
||||
|
||||
区域新闻卡片与算力卡片共用 `#earth-region-callouts` 空间布局根节点(z-index 20),各自数据与阅读状态独立。根节点内部的展开卡片为 50、拖拽前景为 55、悬停展开为 60,普通胶囊为 30,背面胶囊降至 8 并渐淡缩小。各类同时展开数独立保存,完整卡片在地球两侧分配位置,其余以胶囊跟随地理锚点;悬停临时展开,点击改变关注区域。新闻引线始终连接大区代表锚点,内容轮播不改变锚点。共享控制器排除自身卡片后测量 HUD 障碍并处理尺寸变化。细线/粗线样式统一来自 `body.dataset.calloutLineStyle`。手机新闻卡片进入抽屉,不绘制地图连线。
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# 控制台前端结构
|
||||
|
||||
## 算力对比配置
|
||||
|
||||
“智能星球内容”的 `compute_comparison` 分区挂载独立 `ComputeComparisonSettings.tsx`,通过结构化表单维护来源、年度国家观测及创新分差,复用现有 Panel、Input、Select、Button 和确认弹窗,不让运营人员编辑 JSON。草稿在同页切换分区时保留,整页离开提示未保存内容;手动重新加载和恢复默认需确认。组件仅激活时读取管理接口,保存使用后端 revision 防止覆盖其他编辑者,失败保留草稿。表单内容区负责内部滚动,标题和发布操作留在首屏。`compute-comparison-model.ts` 集中类型与同源更新通知,`compute-comparison.css` 只负责该分区布局。
|
||||
|
||||
本文件描述当前控制台前端的真实结构,目标是帮助后续页面开发、表格改造、布局治理和状态收口时快速找到正确入口。
|
||||
|
||||
相关规则建议一起参考:
|
||||
@@ -71,6 +75,8 @@
|
||||
|
||||
## 数据源采集队列
|
||||
|
||||
队列由 `datasource-task-queue.ts` 按独立 task ID 合并状态。同一数据源再次采集会新增一条任务,旧失败/完成/取消记录保留;仅尚未取得 task ID 的活动占位项允许按数据源绑定。迟到推送不能把终态改回进行中,旧任务轮询也不能覆盖数据源当前的新任务。任务状态按钮使用 `ListChecks`,刷新按钮仍使用 `RefreshCw`。
|
||||
|
||||
Admin 的数据源页把单源触发、表格勾选触发和触发全部统一接入浏览器下载列表式采集队列:
|
||||
|
||||
- 队列状态由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 管理,只保存当前会话中的可见任务,不用 `localStorage` 伪造历史。
|
||||
@@ -420,7 +426,7 @@ AI 配置不再挂在 `/settings` 下;`/playground` 应跳转到 `/ai?section=
|
||||
- `电视直播` 迁移原直播源配置,继续管理 Earth 媒体面板内容源。
|
||||
- `品牌标识` 管理 Earth HUD 的品牌资源。`Logo 地址` 与 `标题图地址` 使用字段内上传控件,上传按钮保持 `TactileButton` 的 primary 样式;图片拖到对应字段时显示低饱和拖拽反应,避免回到旧的全局“选择资产/上传”工具栏。
|
||||
- `国界精度` 管理 Earth 静态国界资产:provider 状态、低精 fallback、高精 manifest/PMTiles、源配置 JSON 和构建动作。
|
||||
- `地球底图`、`图层资源`、`三维素材`、`新闻锚点策略` 是占位页,只显示模块待接入,不造假接口或假数据。
|
||||
- `地球底图`、`三维素材`、`新闻锚点策略` 是占位页,只显示模块待接入,不造假接口或假数据。
|
||||
|
||||
系统级 `/settings` 不应再新增 Earth 体验资源或采集生命周期 tab;采集相关入口属于 `/collection-management`,Earth 展示资源属于 `/earth-content`。
|
||||
|
||||
@@ -452,3 +458,11 @@ AI 配置不再挂在 `/settings` 下;`/playground` 应跳转到 `/ai?section=
|
||||
详细经验见:
|
||||
|
||||
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
|
||||
|
||||
## 图层资源中的详情卡配图
|
||||
|
||||
`CardImageSettings.tsx` 替换智能星球内容页 `layer_resources` 的占位区,沿用 `PageFrame`、分区导航、Panel、Input、Select、Switch 和确认弹窗。组件常驻并按 active 隐藏,切换分区保留草稿;图片库和规则各有内部滚动,父级保持 `min-height:0`,窄屏改为上下工作区。`CardImagePreview.tsx` 引用真实详情卡的蒙版 CSS,不维护第二套羽化参数。
|
||||
|
||||
图片库支持上传或地址录入、来源/授权、启停、焦点预览;关联中的资产须先解除规则才能删除。规则编辑从后端当前目录搜索物件或选择已有型号/分组/分类,支持停用、改绑、删除和专属无图。预览接口接收草稿,复用发布后的同一解析函数,返回命中范围。只有顶栏保存才发布,409 时保留草稿;上传返回的文件仍需保存资产配置。保存期间禁用编辑区,防止返回结果覆盖保存期间的新输入。
|
||||
|
||||
接口前缀 `/earth/card-images`:管理员 `GET/PUT configuration`、`POST upload`、`GET targets`、`POST preview`;公共 `GET resolve`、`GET files/{filename}`。配图配置读写保持版本和审计,文件路径不接受任意目录。
|
||||
|
||||
@@ -203,7 +203,7 @@ Base URL 输入框尾端的插头图标会触发轻量连接测试:检查代
|
||||
- **电视直播**:维护智能星球媒体面板里的直播源。
|
||||
- **新闻内容**:按 RSS 来源和手动新闻组查看新闻。RSS 新闻保持只读;手动新闻组可以新增、批量导入 JSON、编辑、删除和重新处理。
|
||||
- **国界精度**:查看当前国界 provider、低精 fallback、高精 PMTiles/manifest 状态,编辑本机源配置并手动构建。
|
||||
- **地球底图**、**图层资源**、**三维素材**、**新闻锚点策略**:目前是待接入占位页,不展示假数据。
|
||||
- **地球底图**、**三维素材**、**新闻锚点策略**:目前是待接入占位页,不展示假数据。
|
||||
|
||||
智能星球页面工具栏齿轮中也有“国界精度”。切到“高精”时,如果本机尚未构建高精资产,会像游戏更新包一样启动后台下载/构建并显示百分比;构建成功后自动应用,无需刷新。切回“低精”只切换本机显示偏好,不重新下载。
|
||||
|
||||
@@ -288,11 +288,27 @@ Base URL 输入框尾端的插头图标会触发轻量连接测试:检查代
|
||||
|
||||
AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军舰、停泊/低速、其他船只。三角形对应航行船只标记,圆点对应停泊或低速状态。
|
||||
|
||||
直接点击图例项可隐藏或恢复对应线路、类别或效果,隐藏项会变暗。各图层独立保留本页筛选状态;切换到另一个物件或启用图层后,对应图例自动出现,无需额外筛选按钮。隐藏已选物件会收起其详情与覆盖;搜索明确选择隐藏物件时,会重新显示其类别。筛选只影响地图显示,不改变统计总数。船型与“停泊/低速”可以同时筛选;国界中的填色控制在基座地图可见时生效。
|
||||
|
||||
卫星在 400% 及以上缩放显示有限数量的真实名称与 NORAD 编号;点径随缩放平滑变化。设置 → 显示 → 显示 FPS 可在品牌右侧查看实时帧率,悬停可查看约 30 秒平均成绩,开关会记住。扇形菜单与缩放子菜单会随界面 DPI 比例适配;鼠标从放大镜经过间隙移到加减按钮时,子菜单保持打开。
|
||||
|
||||
### 搜索
|
||||
|
||||
支持查找海缆、登陆点、卫星、算力中心、BGP 事件、BGP 观测站。结果可快速定位并打开详情。
|
||||
|
||||
点击地球上的 BGP、算力中心、船只、卫星或海缆后,先从物件引出连接线,再沿同一方向把详情卡尽量放到屏幕边缘,中间给地球留出呼吸空间;卡片不能盖住自己的出线点。地球转动时,连线起点会自动平滑移向仍可见的高亮部分;整个对象转到背面后,保留球体轮廓上的方向提示,以较淡连线和空心端点区分。卡片位置保持稳定,仍可拖动,接点会随卡片沿边缘调整,无需手动编辑连线。手机上先显示精简卡片,点开后在抽屉中查看完整详情。下拉菜单与工具列表保持原有打开方式。
|
||||
点击地球上的 BGP、算力中心、船只、卫星或海缆后,先从物件引出连接线,再沿同一方向把详情卡尽量放到屏幕边缘,中间给地球留出呼吸空间;卡片不能盖住自己的出线点。地球转动时,连线起点会自动平滑移向仍可见的高亮部分;整个对象转到背面后,保留球体轮廓上的方向提示,以较淡连线和空心端点区分。卡片随出线方向平滑跟随,仍可拖动,接点会沿卡片边缘调整,无需手动编辑连线。手机上先显示精简卡片,点开后在抽屉中查看完整详情。下拉菜单与工具列表保持原有打开方式。
|
||||
|
||||
### 四大区新闻卡片
|
||||
|
||||
点击新闻面板的刷新按钮后,会显示“正在刷新”,并保留已有新闻供继续阅读。
|
||||
完成更新并显示最新结果后提示刷新成功;失败时说明原因,按钮恢复可用,
|
||||
可以再次刷新。等待期间重复点击不会重复启动刷新。
|
||||
|
||||
点击顶部新闻条打开态势新闻面板,再点右上角“显示区域新闻卡片”,即可同时显示美洲、欧洲、中东与非洲、亚太四张卡片。关闭新闻面板后,区域卡片仍保留,开关会记住,启用时背景和边框高亮,再次点击关闭。
|
||||
|
||||
每张卡片只轮播本区已可显示的新闻,摘要以打字机效果完整显示后,停留 7 秒再换一条;悬停、键盘操作、拖拽或折叠时暂停当前卡片,其他区域继续。底部可上一条、下一条、暂停/继续,或打开原文。刷新新闻时尽量保留正在阅读的条目;本区无可用新闻会显示空态,不拿其他地区填充。
|
||||
|
||||
卡片复用巡航新闻的金色标题栏与 📰 图标,保留完整摘要并精简来源字段,可像算力区域卡片一样拖拽、折叠,松手后自动判断落点。新闻和算力卡片同时开启时统一安排位置。引线连接大区的代表锚点,不表示每篇报道的精确事发地。手机上从新闻页的“区域卡片”开启,在抽屉内依次阅读四大区。
|
||||
|
||||
### 新闻直播源选择
|
||||
|
||||
@@ -322,7 +338,7 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
### 设置
|
||||
|
||||
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、低缩放圆点、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼 / 识别动作白名单、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
|
||||
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼 / 识别动作白名单、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
|
||||
|
||||
新闻类型使用与巡航模块一致的标签选择器,只筛选当前浏览器里的新闻面板和新闻巡航条目,不影响图层、TV、数据点、底图、边界、采集任务或后台新闻源配置。
|
||||
|
||||
@@ -411,12 +427,48 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
## 算力与资源态势
|
||||
|
||||
点击算力中心图层行的“算力态势分析”打开配置窗,再开启“显示区域卡片”,地球周围会先绘制到目标位置的连接线,随后数据卡平滑出现;再次转动时不会重播入场。关闭配置窗或按 Escape 仅收起配置,卡片继续显示;再次点击分析按钮可重开配置,关闭显示开关才会停用卡片。区域锚点随地球旋转,卡片保持朝向读者;背面区域有状态提示,不绘制穿过地球的连线。其他图层、新闻和媒体窗口保持可用。
|
||||
管理员可在控制台“智能星球内容 → 算力对比”维护卡片数据。先在“报告来源”录入报告标题、链接、发布机构、报告年份与页码,再在“年度观测”按年份选择指标、来源和国家数值,最后“保存并发布”。占比以全球为分母,绝对量统一为 FP32 EFlop/s;中美双边份额由同年全球份额计算,区域仅在成员数据完整时汇总。不要混入 FP16/FP64、地区估计值或设施记录数;空缺国家应不录入,明确的零值才填 0。
|
||||
|
||||
“创新指数分差”独立维护美国减中国的观测,不反推单国指数。来源被观测引用时不能直接移除;重复指标年份、无效来源、占比总和超过 100% 等内容会拒绝发布。多人同时编辑时,后保存者收到冲突提示,草稿保留以便核对。“恢复默认”需要确认,会恢复随版本提供的报告数据;保存和恢复均有操作审计。同源页面收到发布通知后更新,其他正在展示的地球页每分钟检查一次,后台标签页恢复可见时也会检查。数值仍需人工核验报告,配置链接不会自动抓取报告。
|
||||
|
||||
点击算力中心图层行的“算力态势分析”打开配置窗,再点击标题栏关闭按钮左侧的“显示区域卡片”图标。地球周围会先绘制到目标位置的连接线,随后数据卡平滑出现;再次转动时不会重播入场。关闭配置窗或按 Escape 仅收起配置,卡片继续显示;再次点击分析按钮可重开配置,点击已高亮的卡片图标可隐藏卡片。区域锚点随地球旋转,卡片保持朝向读者;背面区域有状态提示,不绘制穿过地球的连线。其他图层、新闻和媒体窗口保持可用。
|
||||
|
||||
选择器按规范目录分为中国、美国、欧洲、不含中国的亚太和其他五组;这些分组只用于展示,不参与统计分母,可改变参考期与指标口径。双边份额的分母仍固定为美国加中国,其他参与方没有该口径时显示缺失。区域中心只用于卡片定位,不代表设施位置。点击来源查看报告,点击设施按钮使用原有详情卡。手机通过底部抽屉的“算力”页查看紧凑地区概览:同一列表中的短卡采用手风琴,重复点击卡头收起,点击另一张先收起旧卡,最多同时展开一张长卡。展开详情保留趋势、区域成员、设施、媒体和来源,不设地区二级页面;设置仍是单独按需页。
|
||||
|
||||
默认主指标为全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。已核对的绝对量为 2022 年 302(证据 `caict2023`)和 2023 年 435(证据 `caict2024`)。日本全球份额为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 APAC、OTHER 不能直接等同报告里的日本或“其他”;EUROPE 成员集合尚未与报告对齐,仍显示缺失。双边份额分母固定为美国加中国,不受日本加入影响。主指标缺失时可显示最近有报告的年份(当前到 2023),明确选择 2026 仍显示缺失,不回填最近值。不可比性未确认时不展示同比变化。国家报告指标与公开设施样本独立,覆盖 count 只保留在辅助证据区,不能当作算力值或全球覆盖率。TOP500 与 Epoch 可能重复收录,来源记录数不代表全球设施总数。API 审计字段为 `reviewed_at=2026-09-22`,版本与该日期一致。独立 Epoch 份额模式尚未实现。
|
||||
默认主指标为全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。已核对的绝对量为 2022 年 302(证据 `caict2023`)和 2023 年 435(证据 `caict2024`)。日本全球份额为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 APAC、OTHER 不能直接等同报告里的日本或“其他”;EUROPE 成员集合尚未与报告对齐,仍显示缺失。双边份额分母固定为美国加中国,不受日本加入影响。主指标缺失时可显示最近有报告的年份(当前到 2023),明确选择 2026 仍显示缺失,不回填最近值。不可比性未确认时不展示同比变化。国家报告指标与公开设施样本独立,覆盖 count 只保留在辅助证据区,不能当作算力值或全球覆盖率。TOP500 与 Epoch 可能重复收录,来源记录数不代表全球设施总数。默认报告核验日期为 2026-09-22;后台发布后以维护的核验日期和实际观测为准。独立 Epoch 份额模式尚未实现。
|
||||
|
||||
当前原型最多同时显示六张内容卡。主卡围绕当前选择的算力指标:大号百分比或带单位的绝对量、观测年份、历史上下文和来源入口。缺少可比数据时明确显示“暂无可比数据”;没有观测值时不绘制空趋势图。设施记录和覆盖信息仍完整保留,但移到展开后的辅助区;样本数只是证据上下文,不能当作算力值或全球覆盖率。区域预览可以显示成员数,长列表仅在需要时展开。桌面卡片约宽 268 像素,高度随内容自适应;展开卡片支持用户调整至最高 480 像素。手机手风琴单卡也保持同一占比/绝对量语义,不把设施条数提升为主数字。位置和尺寸保存于本地,并可恢复自动布局。拖动或展开后先留出阅读时间;地区随地球明显转到另一侧或上下方后,卡片才会在能减少连线交叉、绕行的情况下成组平滑换位。鼠标正在卡片上操作时不移动,手动位置不会被立即抢回,也不会永久阻止必要调整。图片、视频等内容可以沿用内容区布局,当前尚未接入这些媒体数据。空间不足时卡片可以经过球体前景移动,不再靠瞬移避让或缩成小标签。
|
||||
最多选择六个算力参与方,与四大区新闻卡片共用周边空间。“设置 → 面板”可分别将算力卡和新闻卡同时展开数设为 1、2 或 3,默认各 1 张,选择会保存。其余卡片显示为胶囊,悬停可临时查看完整内容,点击可切换关注区域。完整卡片保留业务指标、年份、趋势和来源,缺失值明确标注;长内容内部滚动。展开卡片沿地球两侧安排位置,胶囊随区域投影移动,背面胶囊变淡。拖放保留手动位置,区域深度转到背面后恢复自动跟随。“设置 → 显示 → 引导线样式”可选择细线或粗线。
|
||||
|
||||
地图卡片开关关闭时仍会加载概览摘要,打开后共享地图加载事件;新请求会取消旧请求。抽屉事件使用 `composedPath()`,即使通过 innerHTML 换节点也不会误收起当前卡片。复用共享 HUD 折叠面板动画,并尊重 reduced-motion。设施名单使用最高 240px 的内部滚动区域,名称左对齐并正常换行,每行至少保留 42px 触控高度。
|
||||
|
||||
## 地表与物件视觉
|
||||
|
||||
真实地表保留原有四种底图选择;日夜模式按太阳方向控制明暗,关闭后均匀照亮。海缆选中时沿线显示流动脉冲,登陆点使用琥珀微标;设施保留分类图标,聚合以带数量的实心圆显示。卫星仍可切换自身发光/真实地表覆盖,显示轨迹并沿既有轨道运动。BGP 告警、船舶方向、关联高亮、详情卡与移动端抽屉继续使用原有操作。
|
||||
|
||||
### 卫星轨道与采集任务
|
||||
|
||||
选中卫星后,预测轨道以开放弧线显示,线上流光表示运行方向;卫星、轨迹与覆盖投影继续同步移动。覆盖区昼夜保持蓝色,粒子和轨道按图例保持固定类别色。拖动从地球移到轮廓外仍可继续旋转,大倍率下灵敏度会减小。电视窗口使用与其他卡片一致的方角。
|
||||
|
||||
再次触发同一数据源会创建新任务,旧失败项留在失败分组。“任务状态”使用清单图标,与刷新操作区分。CelesTrak 在短时间内可复用有效下载缓存;重新采集仍会生成独立任务并更新采集数据。
|
||||
|
||||
|
||||
显示设置中的“卫星 → 近景标签”默认开启,关闭后隐藏地图上的卫星名称,详情卡仍可使用。开关下的“近景标签显示阈值”滑块可设为 250%–500%,默认 400%;达到该缩放比例时显示标签,低于阈值时隐藏。数量滑块可设为 5–36 个,默认 36;这是显示上限,遮挡或空间不足时实际数量可能更少,设置会保存。“星空亮度”滑块范围为 0–200%,默认 100%,两项会记住选择。算力区域卡片在拖拽、地球运动或新卡片落位稳定后才进行必要避让;平时只在原位附近轻微漂浮,鼠标操作时暂停,减少阅读时频繁换位。算力图例里的分类数量只统计当前已定位点。
|
||||
|
||||
|
||||
地表点位聚合后显示带数量的实心圆,点击可展开成员;只有同坐标或极近坐标点位在原位置附近紧凑堆叠,普通点始终回到各自真实位置;堆叠露出可点击边缘,点击后排图标会将它置前并打开详情,取消焦点后恢复原位。放大超过 250% 后普通聚合自动拆开,500% 不再保留聚合圆标,同坐标点直接展开。
|
||||
|
||||
详情卡按物件展示身份、业务参数和更新时间,隐藏覆盖渲染、采集通道、坐标匹配过程等内部信息。算力设施仍显示位置精度与核验状态,BGP 事件仍显示告警置信度;新闻保留媒体来源、发布时间和摘要。
|
||||
|
||||
详情浮窗与新闻卡共用高度上限,标题固定、长正文内部滚动;关键事件信息和设备参数优先,位置核验、来源及关联信息后置。
|
||||
|
||||
### 详情卡配图
|
||||
|
||||
在“智能星球内容 → 图层资源 → 详情卡配图”维护图片。图片库中可上传 JPEG、PNG、WebP 或填写图片地址,设置名称、来源、署名、授权和裁切焦点;预览显示实际的右侧羽化效果。
|
||||
|
||||
切到“适用规则”,选择卫星、算力设施、海缆、登陆站、船舶、BGP 事件、BGP 观测站和自定义点,再选择指定物件、型号、分组或分类。搜索使用实际已采集对象及编号;没有型号数据时不会编造选项。选好图片后保存关联到草稿,再点顶栏“保存”发布。指定物件的图片优先于型号、分组、分类;“此物件不使用配图”可关闭它的所有回退图片。
|
||||
|
||||
“验证物件配图”可在发布前检查某个对象会用哪张图、命中了哪条规则。停用关联后可继续匹配更通用的规则;删除图片前须先解除其关联。多人同时修改时,保存冲突会保留当前草稿。
|
||||
|
||||
地球详情卡在有图时淡入正文背景,缺图或图片加载失败时继续显示完整数据。标题中的图片图标可查看来源,悬停显示署名和授权。全部地图物件类型均有内置默认图片:优先使用专属图和系列图,否则使用明确标注的类型通用示意。Starlink、北斗三号、千帆等已关联;千帆仍保留授权待确认的来源记录。新安装无需配置,旧安装更新并重启服务后自动补入缺少的默认项;自定义绑定和明确无图规则保留。
|
||||
|
||||
算力图例和详情支持超算中心、GPU 集群、数据中心、云计算中心、边缘计算节点五类;配图管理中选择“算力设施 → 分类”即可分别维护默认图。没有已采集记录的类型显示数量 0。BGP 观测站与 BGP 事件使用独立默认图。
|
||||
|
||||
@@ -1,6 +1,20 @@
|
||||
# planet.sh 启动性能优化
|
||||
|
||||
## 背景
|
||||
## Windows 桌面启动器
|
||||
|
||||
启动窗口优先使用 Windows Terminal,并显式运行 PowerShell 7(`pwsh.exe`),不依赖 Terminal 默认配置文件。未安装 PowerShell 7 时回退到 Windows PowerShell;未安装 Terminal 时直接启动可用的 PowerShell。CMD 初始引导同样优先 PowerShell 7。普通入口请求提权,已经以管理员运行的入口也会进入新 Terminal 窗口,但不会再次请求提权;子窗口校验权限后直接执行 WSL,避免递归打开窗口。重启完成后仍保留原有 WSL shell。
|
||||
|
||||
首次在 Windows 文件资源管理器中打开实际项目目录,双击仓库根目录的 `planet.cmd`。它从自身位置识别 WSL 发行版和 Linux 项目目录,在 Windows 当前用户的实际桌面生成 `Planet.cmd`,随后进入正常重启流程。支持 `\\wsl$\发行版\...` 与 `\\wsl.localhost\发行版\...`;仓库位于 Windows 本地磁盘时,使用默认 WSL 发行版及 `wslpath` 映射。系统桌面位置由 Windows 获取,兼容不同用户名和 OneDrive 等桌面重定向;WSL、zsh 及项目依赖仍须先准备好。
|
||||
|
||||
后续双击桌面副本,使用已绑定的发行版与目录,申请原有管理员权限,并调用 `planet.sh restart --allow-lan`。不写入测试端口;默认前端 3000、后端 8000、AI Provider 8010,显式 `DEFAULT_*_PORT` 环境配置仍有效。成功后按实际 `ports.env` 端口打开 Earth,终端保留 WSL shell;失败只显示错误,不打开成功页面。源文件负责生成入口,`scripts/windows/planet-launcher.ps1` 负责 Windows 路径、桌面安装与提权,`scripts/windows/restart.zsh` 只衔接现有重启流程。
|
||||
|
||||
只生成或检查副本而不重启服务,可在 Windows 中运行 `planet.cmd --install-only`。从仓库再次运行会刷新同一目标的副本;已有其他项目或非托管 `Planet.cmd` 时另存带目标标识的文件,不覆盖。目录移动后从新位置重新运行仓库入口,旧副本不会猜测新目录。桌面副本依赖项目中的脚本,不是可脱离仓库运行的安装包。
|
||||
|
||||
不要把仓库原始 `planet.cmd` 单独复制到桌面,它需要相邻的 `scripts/windows/` 才能完成首次生成。换电脑时应从新电脑的项目目录重新生成,不能直接沿用旧电脑的桌面副本。服务器的 `/home/linkong/planet` 与个人电脑的其他路径均可使用,不必修改提交到 Git 的路径。应在拥有目标 WSL 发行版的 Windows 账号下运行;若提权时改用其他账号,其 WSL 发行版注册可能不同。首次生成不要求管理员权限,真正重启时才申请提权。
|
||||
|
||||
Windows 回归命令为 `powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`,覆盖路径识别、特殊字符、生成副本及名称冲突。`scripts/harness/test_windows_restart.py` 用临时替身验证重启参数、成功打开页面和失败保留 shell,已接入 quick-check,不启动实际服务。
|
||||
|
||||
## 启动背景
|
||||
|
||||
日常启动使用 `zsh ./planet.sh start --non-motion-agent`;新环境首次准备才需要 `init`。排查耗时时,应区分首次依赖下载、容器就绪和应用初始化,结合阶段日志时间判断。
|
||||
|
||||
|
||||
@@ -311,6 +311,8 @@ OTP 一次性验证码走 Redis,key 格式 `otp:{purpose}:{email}`,TTL 600
|
||||
<!-- planet-error-catalog:start -->
|
||||
| 错误编号 | 匹配片段(分号分隔) | 原因/已确认的故障现象 | 处理建议 |
|
||||
| --- | --- | --- | --- |
|
||||
| P_CELESTRAK_NOT_UPDATED | GP data has not updated since your last successful | CelesTrak 拒绝同一更新窗口的重复下载;若日志是 HEAD 200 后 GET 403,预检可能已消耗下载额度,这不表示出口不通。 | 使用不发 HEAD 的采集器版本,保留并校验原始下载缓存;无有效缓存时等待下一次 GP 更新或配置 Space-Track,避免连续重试。 |
|
||||
| P_WINDOWS_LAUNCHER_TARGET_MISSING | P_WINDOWS_LAUNCHER_TARGET_MISSING | 桌面启动器绑定的 WSL 项目目录不可访问或缺少启动脚本,常见于项目移动、发行版不可用或复制旧机器的副本。 | 从当前项目目录运行仓库 planet.cmd 重新生成本机桌面副本,检查 WSL 发行版;不修改成其他电脑的固定路径。 |
|
||||
| P_PROXY_EXTERNAL | PLANET_PROXY_EXTERNAL | 当前 Docker 代理由其他配置管理,或已被人工修改,不能安全自动覆盖。 | 检查 daemon.json、systemd 代理及 Planet 管理记录,确认归属后再调整;脚本保留现有配置。 |
|
||||
| P_AUTH_BOOTSTRAP_MISSING | DEFAULT_LOGIN_USER_MISSING | 已确认当前后端数据库缺少默认账户,常见于跳过初始化的隔离预览。 | 核对数据库归属后运行现有默认账户初始化,不重置已有账户或清空数据库。 |
|
||||
| P_AUTH_PROFILE_INVALID | AUTH_IDENTITY_RESPONSE_INVALID | 已确认登录成功但身份响应字段缺失,或响应错误拒绝既有本地邮箱。 | 从真实用户记录加载完整身份,区分响应序列化与注册输入校验;运行默认账户身份回归。 |
|
||||
@@ -487,3 +489,46 @@ uv run pytest backend/tests/test_otp_service.py
|
||||
高精国界产物位于被 Git 忽略的生成目录,不会随拉取自动交付。旧产物缺少新规则元数据时会被拒绝,页面使用已交付的政策基础国界;需保持高精效果时,应按既有国界构建流程重新生成并发布对应静态产物。底图分辨率不受这一回退影响。
|
||||
|
||||
容器资源回归使用 `PLANET_HARNESS_DOCKER_SMOKE=1 .venv/bin/python scripts/harness/test_backend_runtime_assets.py`。它通过真实 Docker 构建上下文验证生产 COPY 规则、共享规则/几何及构建脚本,同时检查环境文件未打包;不启动整套服务。
|
||||
|
||||
## 详情卡图片存储
|
||||
|
||||
|
||||
内置原图及清单位于 `frontend/public/earth/assets/cards/`,卡片尺寸预览位于其 `previews/` 子目录。新增或更换原图后运行 `uv run --frozen --project . python scripts/prepare_card_images.py`,检查来源哈希及授权状态,再提交原图、清单和生成的 WebP。预览生成遵循清单 enabled:明确启用的素材生成预览,授权记录保持原样。后端镜像复制同一 manifest,前端构建分发图片,因此两个镜像应使用同一版本发布。
|
||||
|
||||
人工上传默认写入仓库根的 `data/earth-card-images/`;可通过 `PLANET_CARD_IMAGE_DIR` 指定绝对存储目录。生产容器须将该目录挂载到持久卷,多副本须共享同一目录。备份同时包含 PostgreSQL 中 `system_settings` 的 `earth_card_images` 配置及该图片目录;仅备份数据库无法恢复上传文件。原图和 640/1280 WebP 使用内容哈希路径,经 `/api/v1/earth/card-images/files/` 提供,现有 `/api/` 代理可直接转发。移除配置不会立即删除物理文件,以免破坏仍在读取旧版本的客户端。
|
||||
|
||||
## 数据库缓存与历史事件维护
|
||||
|
||||
已消费 Outbox 事件默认保留一天,`PLANET_EARTH_OUTBOX_RETENTION_DAYS` 可设置
|
||||
保留天数。监听器每分钟最多删除 1000 条过期已消费记录,并要求维护索引有效。
|
||||
首次部署应在维护窗口运行以下命令。默认命令只检查,添加 `--apply` 才修改。
|
||||
|
||||
```bash
|
||||
uv run --frozen python scripts/maintain_earth_outbox.py
|
||||
uv run --frozen python scripts/maintain_earth_outbox.py --apply
|
||||
uv run --frozen python scripts/install_bgp_prefix_geo_indexes.py
|
||||
```
|
||||
|
||||
历史 Outbox 严重膨胀时,可在维护窗口使用
|
||||
`scripts/maintain_earth_outbox.py --compact --apply`。该模式持排他锁,将未消费和
|
||||
保留期内事件暂存,再事务性重填原表,核对总数和未消费数后提交,并继续维护
|
||||
索引。扫描及回填期间其他访问会阻塞。预留临时数据空间,停止后端热重载,
|
||||
避免应用启动 DDL 与并发索引安装争锁。失败时检查事务结果及无效索引后再重试。
|
||||
压缩后的空间以 `pg_total_relation_size` 为准,Linux Page Cache 另行回收。
|
||||
|
||||
WSL 宿主机可在 `%USERPROFILE%\.wslconfig` 设置内存预算。32 GiB 宿主机上的
|
||||
一组已验证配置如下,现有网络设置应一并保留。修改后须重启 WSL 并恢复服务,
|
||||
随后验证 `free -m`、Windows 可用内存和真实负载。
|
||||
|
||||
```ini
|
||||
[wsl2]
|
||||
memory=8GB
|
||||
swap=4GB
|
||||
|
||||
[experimental]
|
||||
autoMemoryReclaim=gradual
|
||||
hostAddressLoopback=true
|
||||
```
|
||||
|
||||
`hostAddressLoopback` 允许 Windows 使用自身 IPv4 地址访问 mirrored 模式中的
|
||||
WSL 服务,用于本机验证局域网地址入口。
|
||||
|
||||
@@ -57,6 +57,12 @@
|
||||
- 设置面板的旋转 / 巡航 / 动捕模式可以切换;动捕设置可以选择输入源和允许识别的动作;视图设置里可以切换悬停提示,卫星相关设置里可以打开或关闭真实高度分层和轨迹显示
|
||||
- 底图默认使用“原始底图”,并将该选项放在首位;显示菜单可以切换并保存底图源。Natural Earth I 仍保留可用,NASA 21K 只作源归档,运行时同一时间只使用一张 8K 贴图。
|
||||
|
||||
点击物件或开启图层后,可直接点击对应图例项筛选;无需在图层中寻找额外入口。设置 → 显示可开启 FPS,卫星名称在缩放达到 400% 后显示。
|
||||
|
||||
顶部新闻条可打开态势新闻面板;启用“显示区域新闻卡片”后,美洲、欧洲、中东与非洲、亚太分别轮播本区新闻。每张卡片可单独暂停、翻页、拖动与折叠。
|
||||
|
||||
刷新新闻时可继续阅读已有内容,等待“刷新成功”后查看最新结果;失败时可按提示重试。“设置 → 面板”可分别选择算力卡和新闻卡展开 1–3 张,其他区域保留为可悬停展开的胶囊。
|
||||
|
||||
## 5. 找回密码
|
||||
|
||||
忘记密码时打开 `/forgot-password`,输入邮箱并接收验证码,再输入验证码 + 新密码即可。系统对未注册的邮箱也会返回同样的提示(防止账号枚举)。
|
||||
@@ -74,4 +80,14 @@
|
||||
|
||||
## 试用算力专题
|
||||
|
||||
在算力中心旁打开“算力态势分析”,开启“显示区域卡片”,选择地区并旋转地球。关闭配置窗会保留卡片,显示开关才控制启停。桌面卡片可调整大小、收起并查看真实历史图;点击来源或设施进入详情。手机使用底部“算力”页。国家指标与设施样本分开,“—”表示缺数据,不代表算力为零。
|
||||
维护卡片数值:管理员打开“智能星球内容 → 算力对比”,先维护报告来源,再填写年度国家观测并“保存并发布”。只录入同年同口径数据,缺失国家不填,设施数量不会替代算力份额。可恢复内置默认值,地球卡片自动检查已发布配置。
|
||||
|
||||
在算力中心旁打开“算力态势分析”,点击标题栏关闭按钮左侧的“显示区域卡片”图标,再选择地区并旋转地球。关闭配置窗后卡片会继续显示,再次点击该图标可隐藏卡片。桌面卡片可调整大小、收起并查看真实历史图;点击来源或设施进入详情。手机使用底部“算力”页。国家指标与设施样本分开,“—”表示缺数据,不代表算力为零。
|
||||
|
||||
### 查看卫星与重试采集
|
||||
|
||||
在 Earth 启用卫星并选中一个粒子,可查看开放预测轨道、方向流光及受支持的覆盖投影。拖动移出球体仍会继续旋转。数据源采集失败后可重新触发;新任务独立显示,旧失败记录保留在当前任务队列的失败分组。
|
||||
|
||||
区域卡片保持在地球周边,旋转时主要变化是连线。空间不足时保留完整内容,允许少量遮挡;合法落点会保留;落在中央或越界时会自动就近调整。详情卡只保留业务信息和必要的可信度提示。
|
||||
|
||||
详情卡配图可在“智能星球内容 → 图层资源”管理:上传图片,选择适用物件或分组,预览后保存。内置专属图、系列图和类型通用示意覆盖全部地图物件;新安装无需导入,旧项目更新后重启前后端即可补齐默认配图。用户自定义关联和明确禁用的设置保留。
|
||||
|
||||
@@ -16,12 +16,36 @@
|
||||
## Current Version
|
||||
|
||||
- `main` 当前主线历史推导到:`0.16.5`
|
||||
- `dev` 当前开发分支历史推导到:`0.75.0`
|
||||
- `dev` 当前开发分支历史推导到:`0.76.19`
|
||||
|
||||
## Timeline
|
||||
|
||||
| Version | Type | Branch | Commit | Summary |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `0.76.19` | improvement | `dev` | `v0.76.19` | 改善 Earth 卡片与移动交互、新闻后台刷新、图层缓存、BGP 前缀查询及 Outbox 维护 |
|
||||
| `0.76.18` | bugfix | `dev` | `v0.76.18` | 收敛 breaking 新闻交互、卡片 hover/扫光、面板稳定性、船只锚点显隐并同步开发文档 |
|
||||
| `0.76.17` | bugfix | `dev` | `v0.76.17` | 稳定 breaking active 动画状态,修复连续新闻切换时扫光重启闪烁 |
|
||||
| `0.76.16` | bugfix | `dev` | `v0.76.16` | 修复 breaking class 重置导致的边框闪烁、扫光停滞与方向反转 |
|
||||
| `0.76.15` | bugfix | `dev` | `v0.76.15` | ticker 扫光限制在上下边线以内,消除 breaking 边框闪烁并与卡片特效分离 |
|
||||
| `0.76.14` | bugfix | `dev` | `v0.76.14` | 修复卡片扫光起点,改为从卡片左边界到右边界的内部背景扫描 |
|
||||
| `0.76.13` | bugfix | `dev` | `v0.76.13` | breaking hover 以自身高亮为基准继续增强,修复红 ticker/卡片变暗 |
|
||||
| `0.76.12` | bugfix | `dev` | `v0.76.12` | 修复 breaking ticker 与黄卡 hover 被白色混合漂淡,保留等级色相 |
|
||||
| `0.76.11` | bugfix | `dev` | `v0.76.11` | breaking 卡片整卡扫光及 ticker hover 强度对齐共享 HUD |
|
||||
| `0.76.10` | bugfix | `dev` | `v0.76.10` | breaking 卡片扫光仅在 hover/focus 时启动,避免全列表同时闪烁 |
|
||||
| `0.76.9` | bugfix | `dev` | `v0.76.9` | 统一 breaking 卡片 hover 强度并增加 ticker 上下边线 hover 高亮 |
|
||||
| `0.76.8` | bugfix | `dev` | `v0.76.8` | 展开新闻面板时冻结卡片 DOM,隔离 ticker/realtime 更新并修复 hover 抖动 |
|
||||
| `0.76.7` | bugfix | `dev` | `v0.76.7` | 统一 breaking 卡片 hover 行为并通过卡片等级变量注入颜色 tint |
|
||||
| `0.76.6` | bugfix | `dev` | `v0.76.6` | breaking ticker 全域扫光闪烁,新闻列表首卡增加顶部呼吸空间 |
|
||||
| `0.76.5` | bugfix | `dev` | `v0.76.5` | breaking 玻璃反射式警灯脉冲、等级颜色统一及新闻卡片内容高度修复 |
|
||||
| `0.76.4` | bugfix | `dev` | `v0.76.4` | 修正 breaking 新闻 ticker 与面板状态边界,当前滚动到 breaking 项时才显示红色 |
|
||||
| `0.76.3` | bugfix | `dev` | `v0.76.3` | 修复新闻栏关闭后残留 breaking 橙色边线,分离 ticker、面板、选择和过期状态 |
|
||||
| `0.76.2` | bugfix | `dev` | `v0.76.2` | 优化光缆密集区卫星点拾取,统一点状对象与线缆候选排序并补齐回归验证 |
|
||||
| `0.76.1` | bugfix | `dev` | `v0.76.1` | 修复选中 BGP 事件静态光晕地表遮挡,统一事件锚点显隐与前景偏移,清理中间态逻辑并补齐验证 |
|
||||
| `0.76.0` | feature | `dev` | `v0.76.0` | 详情卡默认配图与后台配置、五类算力中心、统一地表拾取与选中置顶、卡片交互修复及 Earth 性能优化 |
|
||||
| `0.75.4` | improvement | `dev` | `v0.75.4` | 图例统一筛选、卫星 Points 点径与性能优化、FPS、菜单 DPI/hover、星空 sRGB、地表对齐及新闻编码修复,清理废弃代码文档 |
|
||||
| `0.75.3` | improvement | `dev` | `v0.75.3` | 地球海陆材质与可读夜侧、连续拖拽、现代徽标、卫星平滑开放轨道及流光,修复 CelesTrak 缓存与任务历史并清理废弃实现 |
|
||||
| `0.75.2` | bugfix | `dev` | `v0.75.2` | Windows 启动器优先 Terminal 与 PowerShell 7,修复管理员入口分流,补齐兼容回退与提权测试 |
|
||||
| `0.75.1` | improvement | `dev` | `v0.75.1` | Windows 启动器自动生成本机桌面入口,后台维护算力比较数据,恢复小球惯性与大球跟手分流并更新 agent 策略 |
|
||||
| `0.75.0` | feature | `dev` | `v0.75.0` | 四图切换默认原始底图,新增算力区域态势卡片,统一空间连线与重排,优化启动和缩放拖动并补齐地理规则交付 |
|
||||
| `0.74.7` | improvement | `dev` | `v0.74.7` | Docker 镜像网络故障临时回退,精简副行进度与失败显示,修复数据库端口冲突及启动重试并补齐实际验证 |
|
||||
| `0.74.6` | improvement | `dev` | `v0.74.6` | 自动检测 Docker 构建代理与直连,统一运维错误诊断,支持跳过 AI Provider 构建并修复 Compose 回退与失败退出状态 |
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<link rel="icon" href="data:," />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, viewport-fit=cover" />
|
||||
<title>智能星球计划</title>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "planet-frontend",
|
||||
"version": "0.75.0",
|
||||
"version": "0.76.19",
|
||||
"private": true,
|
||||
"packageManager": "bun@1",
|
||||
"dependencies": {
|
||||
|
||||
185
frontend/public/earth/assets/cards/README.md
Normal file
185
frontend/public/earth/assets/cards/README.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# 内置详情卡配图 / Bundled detail-card images
|
||||
|
||||
图片与默认匹配规则一起随仓库分发,无需外网下载、后台上传或手动导入数据库。新安装直接读取清单,已有配置自动补入新默认项;旧内置图片地址随版本替换,用户修改的裁切、名称等属性保留,用户编辑、删除和明确无图规则保留。
|
||||
|
||||
覆盖全部地图物件类型。专属图 → 型号图 → 分组图 → 具体分类 → 类型通用图;通用图明确标为示意,不代表当前设施或船舶的实际照片。千帆按用户要求作为系列默认配图启用,但开放再分发许可仍未确认,清单保留真实状态。
|
||||
|
||||
Bundled files and bindings work after checkout without external downloads, uploads or database imports. Existing settings merge new defaults and upgrade old bundled sources while preserving edited fields; custom edits, removals and suppression survive. Type illustrations are not photographs of the selected object. Qianfan is enabled as requested; its open redistribution permission remains unverified.
|
||||
|
||||
## 素材与来源 / Assets and credits
|
||||
|
||||
### Starlink 太阳翼展开示意图
|
||||
|
||||
- 文件 / File: [starlink-orbit.webp](starlink-orbit.webp) · 768 × 768
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"constellation_group": "starlink", "scope": "series_representative", "example_norad_ids": [100779]}`
|
||||
- 说明 / Note: 用户指定的 Starlink 系列默认配图;不是单颗卫星实拍或已核实的硬件型号。
|
||||
|
||||
### GPS Block IIIA 在轨示意图
|
||||
|
||||
- 文件 / File: [gps-block-iiia-orbit.jpg](gps-block-iiia-orbit.jpg) · 3150 × 3150
|
||||
- 来源 / Source: [U.S. Air Force](https://commons.wikimedia.org/wiki/File:GPS_Block_IIIA.jpg)
|
||||
- 许可 / License: Public domain (U.S. federal government work) — https://commons.wikimedia.org/wiki/File:GPS_Block_IIIA.jpg#Licensing
|
||||
- 默认关联 / Defaults: `{"norad_ids": [43873, 44506, 45854, 46826, 48859, 55268], "scope": "model_representative"}`
|
||||
- 说明 / Note: 只关联已核实的 GPS IIIA 卫星,不按 NAVSTAR 前缀应用到全部 GPS 世代。
|
||||
|
||||
### Landsat 8 在轨示意图
|
||||
|
||||
- 文件 / File: [landsat-8-orbit.jpg](landsat-8-orbit.jpg) · 2000 × 1126
|
||||
- 来源 / Source: [NASA/Goddard Space Flight Center Conceptual Image Lab](https://svs.gsfc.nasa.gov/10812/)
|
||||
- 许可 / License: Public domain (NASA SVS) — https://svs.gsfc.nasa.gov/help/
|
||||
- 默认关联 / Defaults: `{"norad_ids": [39084], "scope": "spacecraft_illustration"}`
|
||||
- 说明 / Note:
|
||||
|
||||
### Landsat 9 在轨示意图
|
||||
|
||||
- 文件 / File: [landsat-9-orbit.jpg](landsat-9-orbit.jpg) · 3840 × 2160
|
||||
- 来源 / Source: [NASA's Goddard Space Flight Center/Conceptual Image Lab](https://svs.gsfc.nasa.gov/13259/)
|
||||
- 许可 / License: Public domain (NASA SVS) — https://svs.gsfc.nasa.gov/help/
|
||||
- 默认关联 / Defaults: `{"norad_ids": [49260], "scope": "spacecraft_illustration"}`
|
||||
- 说明 / Note:
|
||||
|
||||
### NOAA-21 / JPSS-2 在轨示意图
|
||||
|
||||
- 文件 / File: [noaa-21-jpss-2-orbit.jpg](noaa-21-jpss-2-orbit.jpg) · 971 × 648
|
||||
- 来源 / Source: [NOAA/NESDIS; artist Lamont W. Harvey](https://www.nesdis.noaa.gov/imagery/media-stills-animation/jpss/mission-satellite/jpss-beauty-renders)
|
||||
- 许可 / License: NOAA public-domain content policy; retain credit — https://oceanservice.noaa.gov/about/faq.html
|
||||
- 默认关联 / Defaults: `{"norad_ids": [54234], "scope": "spacecraft_illustration"}`
|
||||
- 说明 / Note: JPSS-2 对应 NOAA-21;不要关联到 NOAA-20 / JPSS-1。
|
||||
|
||||
### 北斗三号展览模型照片
|
||||
|
||||
- 文件 / File: [beidou-3-mockup.jpg](beidou-3-mockup.jpg) · 2680 × 2011
|
||||
- 来源 / Source: [AKAMGO yalms; crop published by Shujianyang](https://commons.wikimedia.org/wiki/File:Beidou-3_Satellite_Mockup.jpg)
|
||||
- 许可 / License: CC-BY-SA-4.0 — https://creativecommons.org/licenses/by-sa/4.0/
|
||||
- 默认关联 / Defaults: `{"series": "beidou-3", "scope": "series_representative", "example_norad_ids": [44204, 44337, 44709, 43683, 45344, 45807, 56564], "constellation_group": "beidou-3"}`
|
||||
- 说明 / Note: 展览模型照片,原始照片主题为 IGSO/GEO。只作北斗三号系列代表图,不把模型当成在轨实拍,不用于声称 MEO 或某颗卫星的精确外形。
|
||||
|
||||
### Galileo IOV 卫星模型照片
|
||||
|
||||
- 文件 / File: [galileo-iov-model.jpg](galileo-iov-model.jpg) · 1536 × 1075
|
||||
- 来源 / Source: [Science Museum Group; model by Paragon Creative, York (2014)](https://commons.wikimedia.org/wiki/File:Galileo_satellite_model.jpg)
|
||||
- 许可 / License: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0/
|
||||
- 默认关联 / Defaults: `{"norad_ids": [37846, 37847, 38857], "scope": "model_representative"}`
|
||||
- 说明 / Note: IOV 型模型,不应用到 Galileo FOC 或第二代卫星。只列当前采集到的三个 IOV 编号。
|
||||
|
||||
### 千帆卫星动画示意图
|
||||
|
||||
- 文件 / File: [qianfan-animation-still.jpg](qianfan-animation-still.jpg) · 860 × 540
|
||||
- 来源 / Source: [CCTV, as credited by Raumfahrtkalender](https://www.raumfahrtkalender.de/uebersichten/nutzlasten/nutzlast/25140)
|
||||
- 许可 / License: Not established; no open redistribution license found for this still
|
||||
- 默认关联 / Defaults: `{"series": "qianfan", "scope": "series_representative", "example_norad_ids": [61556], "constellation_group": "qianfan"}`
|
||||
- 说明 / Note: CCTV 千帆动画静帧,按用户要求用于系列默认配图;非单颗卫星实拍,开放再分发授权仍未确认。
|
||||
|
||||
### Frontier 超级计算机实景
|
||||
|
||||
- 文件 / File: [frontier-supercomputer.jpg](frontier-supercomputer.jpg) · 3000 × 2000
|
||||
- 来源 / Source: [Oak Ridge National Laboratory](https://commons.wikimedia.org/wiki/File:Frontier_supercomputer_(10).jpg)
|
||||
- 许可 / License: CC-BY-2.0 — https://creativecommons.org/licenses/by/2.0/
|
||||
- 默认关联 / Defaults: `{"entity_ids": ["epoch_ai_gpu:epoch_oak_ridge_nl_frontier"], "scope": "specific_facility"}`
|
||||
- 说明 / Note: Frontier 实景,仅关联 Oak Ridge NL Frontier,不作为其他机房的通用实景。
|
||||
|
||||
### 通用卫星示意图(Landsat 9)
|
||||
|
||||
- 文件 / File: [landsat-9-orbit.jpg](landsat-9-orbit.jpg) · 3840 × 2160
|
||||
- 来源 / Source: [NASA's Goddard Space Flight Center/Conceptual Image Lab](https://svs.gsfc.nasa.gov/13259/)
|
||||
- 许可 / License: Public domain (NASA SVS) — https://svs.gsfc.nasa.gov/help/
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "satellite", "scope": "category", "value": "*"}]}`
|
||||
- 说明 / Note: 类型通用示意,不代表当前选中卫星的实际外观。
|
||||
|
||||
### 数据中心通用示意图
|
||||
|
||||
- 文件 / File: [data-center.png](data-center.png) · 1700 × 1133
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "compute_center", "scope": "category", "value": "*"}, {"entity_type": "compute_center", "scope": "category", "value": "data_center"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
### BGP 事件网络设备示意图
|
||||
|
||||
- 文件 / File: [bgp-event.jpg](bgp-event.jpg) · 600 × 401
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "bgp", "scope": "category", "value": "*"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
### 海缆登陆站通用示意(ACE Duynefontein)
|
||||
|
||||
- 文件 / File: [cable-landing-station.jpg](cable-landing-station.jpg) · 1600 × 777
|
||||
- 来源 / Source: [John.combey](https://commons.wikimedia.org/wiki/File:ACE%27s_Duynefontein_landing_station.jpg)
|
||||
- 许可 / License: CC-BY-SA-4.0 — https://creativecommons.org/licenses/by-sa/4.0/
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "landing_point", "scope": "category", "value": "*"}]}`
|
||||
- 说明 / Note: 类型通用示意,不代表当前物件的实际现场或外观。
|
||||
|
||||
### 海底光缆通用示意图
|
||||
|
||||
- 文件 / File: [subsea-cable.png](subsea-cable.png) · 800 × 500
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "cable", "scope": "category", "value": "*"}]}`
|
||||
- 说明 / Note: 用户指定的海底光缆通用示意图,不代表当前海缆的实际外观。
|
||||
|
||||
### 船舶通用示意
|
||||
|
||||
- 文件 / File: [vessel-at-sea.jpg](vessel-at-sea.jpg) · 2000 × 1600
|
||||
- 来源 / Source: [U.S. Navy / Navy Media Content Services](https://www.dvidshub.net/image/99922/container-ship-sea)
|
||||
- 许可 / License: Public domain — https://www.dvidshub.net/image/99922/container-ship-sea
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "vessel", "scope": "category", "value": "*"}]}`
|
||||
- 说明 / Note: 类型通用示意,不代表当前物件的实际现场或外观。
|
||||
|
||||
### 地理点位通用示意
|
||||
|
||||
- 文件 / File: [earth-location.jpg](earth-location.jpg) · 1280 × 640
|
||||
- 来源 / Source: [NASA GSFC / Reto Stöckli, Robert Simmon; composition by Meow](https://commons.wikimedia.org/wiki/File:Blue_Marble_2002.jpg)
|
||||
- 许可 / License: Public domain (NASA) — https://commons.wikimedia.org/wiki/File:Blue_Marble_2002.jpg#Licensing
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "earth_interactable", "scope": "category", "value": "*"}]}`
|
||||
- 说明 / Note: 类型通用示意,不代表当前物件的实际现场或外观。
|
||||
|
||||
### 超算中心通用示意图
|
||||
|
||||
- 文件 / File: [supercomputer.png](supercomputer.png) · 1024 × 717
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "compute_center", "scope": "category", "value": "supercomputer"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
### GPU 集群通用示意图
|
||||
|
||||
- 文件 / File: [gpu-cluster.jpg](gpu-cluster.jpg) · 936 × 528
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "compute_center", "scope": "category", "value": "gpu_cluster"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
### 云计算中心通用示意图
|
||||
|
||||
- 文件 / File: [cloud-center.png](cloud-center.png) · 680 × 400
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "compute_center", "scope": "category", "value": "cloud_center"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
### 边缘计算节点通用示意图
|
||||
|
||||
- 文件 / File: [edge-node.png](edge-node.png) · 614 × 345
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "compute_center", "scope": "category", "value": "edge_node"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
### BGP 观测站通用示意图
|
||||
|
||||
- 文件 / File: [bgp-collector.png](bgp-collector.png) · 326 × 235
|
||||
- 来源 / Source: 用户提供 / User supplied
|
||||
- 许可 / License: 未提供授权信息
|
||||
- 默认关联 / Defaults: `{"defaults": [{"entity_type": "bgp_collector", "scope": "category", "value": "*", "id": "builtin-network-switches-default-1"}]}`
|
||||
- 说明 / Note: 用户指定的类型通用示意图,不作为当前物件的实景证明。
|
||||
|
||||
## 预览与维护 / Derivatives
|
||||
|
||||
原始素材保留下载字节;previews/ 按最长边 640 / 1280 像素等比缩小,不放大、不裁切,编码为 WebP 并保留透明通道。来源许可和署名继续适用于衍生文件;BY-SA 素材的预览继续遵守同一许可。manifest.json 保存原图和预览的尺寸、哈希、来源及修改说明。
|
||||
|
||||
Original downloaded raster bytes are retained. WebP derivatives fit 640 / 1280 pixels without upscaling or cropping. Credits and licenses, including ShareAlike terms, continue to apply. The manifest records checksums and modification descriptions.
|
||||
|
||||
重新生成 / Regenerate: `uv run --frozen --project . python scripts/prepare_card_images.py`。前后端须发布同一版本;后端只打包清单,前端分发图片。
|
||||
BIN
frontend/public/earth/assets/cards/beidou-3-mockup.jpg
Normal file
BIN
frontend/public/earth/assets/cards/beidou-3-mockup.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.7 MiB |
BIN
frontend/public/earth/assets/cards/bgp-collector.png
Normal file
BIN
frontend/public/earth/assets/cards/bgp-collector.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
frontend/public/earth/assets/cards/bgp-event.jpg
Normal file
BIN
frontend/public/earth/assets/cards/bgp-event.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
BIN
frontend/public/earth/assets/cards/cable-landing-station.jpg
Normal file
BIN
frontend/public/earth/assets/cards/cable-landing-station.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 184 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user