release: bump version to 0.31.1
This commit is contained in:
rayd1o
34
docs/plans/README.md
Normal file
34
docs/plans/README.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Plans Docs
|
||||
|
||||
这里放“未来实施方案和未完成计划”的文档，重点回答：
|
||||
|
||||
- 我们准备做什么
|
||||
- 为什么要做
|
||||
- 分几期做
|
||||
- 当前差距和下一步是什么
|
||||
|
||||
适合放入这里的内容：
|
||||
|
||||
- Earth / BGP / 地形 / 天球实施方案
|
||||
- AI Playground 发展计划
|
||||
- backend / datasource / agent roadmap
|
||||
- UE5 MVP 方案
|
||||
|
||||
当前重点入口：
|
||||
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容：
|
||||
|
||||
- 当前代码结构说明
|
||||
- 组件现状和实现入口
|
||||
- 已经落地的技术上下文说明
|
||||
|
||||
这些应放入：
|
||||
|
||||
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
|
||||
647
docs/plans/agents-agent-architecture-plan.md
Normal file
647
docs/plans/agents-agent-architecture-plan.md
Normal file
@@ -0,0 +1,647 @@
|
||||
# Agent Architecture Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the agent architecture for Planet.
|
||||
|
||||
The architecture is intentionally broader than datasource health checking.
|
||||
|
||||
It is designed to support both:
|
||||
|
||||
- datasource health governance
|
||||
- future situational-awareness workflows
|
||||
|
||||
The core idea is to avoid building a one-off "repair broken API links" agent.
|
||||
|
||||
Instead, Planet should grow a reusable agent runtime that can:
|
||||
|
||||
- collect evidence
|
||||
- evaluate signals
|
||||
- reason over incomplete information
|
||||
- generate proposals
|
||||
- produce assessments
|
||||
- execute limited actions under policy
|
||||
|
||||
|
||||
## Design Goal
|
||||
|
||||
Build an agent foundation that can evolve in this order:
|
||||
|
||||
1. datasource health checks
|
||||
2. datasource repair proposals
|
||||
3. signal correlation
|
||||
4. situational assessments
|
||||
5. controlled runtime actions
|
||||
|
||||
This means the architecture should treat datasource health as one use case of the larger agent system, not as the whole system.
|
||||
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. Separate evidence from reasoning
|
||||
|
||||
- raw signals should be gathered first
|
||||
- deterministic checks should run before LLM reasoning
|
||||
|
||||
2. Agents do not own the defaults
|
||||
|
||||
- repository defaults remain human-owned
|
||||
- agents operate on runtime state, proposals, and overrides
|
||||
|
||||
3. Reasoning and action are different responsibilities
|
||||
|
||||
- many agents should be read-only or propose-only
|
||||
- only tightly controlled flows may apply changes
|
||||
|
||||
4. Shared runtime, specialized roles
|
||||
|
||||
- multiple agent roles should share the same object model and orchestration patterns
|
||||
- health and situational-awareness agents should not invent incompatible payloads
|
||||
|
||||
5. Auditability is mandatory
|
||||
|
||||
- every proposal, assessment, and applied action should be attributable
|
||||
|
||||
|
||||
## System Layers
|
||||
|
||||
Planet agent architecture should be split into four layers.
|
||||
|
||||
### 1. Signal Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- gather raw evidence from internal and external systems
|
||||
|
||||
Example sources:
|
||||
|
||||
- collector outputs
|
||||
- datasource health checks
|
||||
- logs
|
||||
- snapshots
|
||||
- alerts
|
||||
- web search results
|
||||
- scraped pages
|
||||
- external APIs
|
||||
- operator inputs
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- fetch
|
||||
- normalize
|
||||
- timestamp
|
||||
- tag with source and trust level
|
||||
|
||||
This layer should not make high-level judgments.
|
||||
|
||||
|
||||
### 2. Evaluation Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- perform deterministic analysis
|
||||
|
||||
Examples:
|
||||
|
||||
- reachability checks
|
||||
- schema validation
|
||||
- threshold checks
|
||||
- time-window comparisons
|
||||
- anomaly counters
|
||||
- completeness checks
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- classify signals into machine-readable findings
|
||||
- attach deterministic evidence
|
||||
|
||||
This layer should avoid LLM dependency whenever possible.
|
||||
|
||||
|
||||
### 3. Reasoning Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- use LLMs when semantic interpretation or incomplete-information reasoning is needed
|
||||
|
||||
Examples:
|
||||
|
||||
- endpoint migration inference
|
||||
- multi-source event correlation
|
||||
- causality hypotheses
|
||||
- ambiguity reduction
|
||||
- assessment narrative generation
|
||||
- action recommendation generation
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- synthesize evidence
|
||||
- produce hypotheses
|
||||
- rank confidence
|
||||
- explain reasoning boundaries
|
||||
|
||||
This is the main place where `aiprovider` and web search are used.
|
||||
|
||||
|
||||
### 4. Action Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- convert proposals or assessments into controlled system actions
|
||||
|
||||
Examples:
|
||||
|
||||
- create runtime override
|
||||
- create proposal
|
||||
- publish alert
|
||||
- update operator task queue
|
||||
- generate summary artifact
|
||||
- trigger follow-up verification
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- enforce policy
|
||||
- enforce approval requirements
|
||||
- verify post-action outcomes
|
||||
- record audit trails
|
||||
|
||||
|
||||
## Architecture Sketch
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Collectors / Logs / Snapshots / External APIs"] --> B["Signal Layer"]
|
||||
W["Web Search / Page Fetch / Docs"] --> B
|
||||
B --> C["Evaluation Layer"]
|
||||
C --> D["Findings"]
|
||||
D --> E["Reasoning Layer (LLM + Tools)"]
|
||||
E --> F["Proposals"]
|
||||
E --> G["Assessments"]
|
||||
F --> H["Action Layer"]
|
||||
H --> I["Runtime Overrides / Alerts / Tasks"]
|
||||
H --> J["Verification Loop"]
|
||||
J --> B
|
||||
|
||||
K["Policy Engine"] --> H
|
||||
L["Audit / History Store"] --> H
|
||||
L --> E
|
||||
L --> C
|
||||
```
|
||||
|
||||
|
||||
## Agent Roles
|
||||
|
||||
The first version should define these logical roles.
|
||||
|
||||
### 1. Health Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- datasource health governance
|
||||
|
||||
Inputs:
|
||||
|
||||
- datasource metadata
|
||||
- current endpoint
|
||||
- latest health records
|
||||
- latest failures
|
||||
- deterministic findings
|
||||
|
||||
Outputs:
|
||||
|
||||
- health interpretation
|
||||
- repair proposal
|
||||
- confidence
|
||||
- evidence references
|
||||
|
||||
Typical action level:
|
||||
|
||||
- propose-only
|
||||
|
||||
|
||||
### 2. Correlation Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- identify whether multiple signals describe the same event or related events
|
||||
|
||||
Inputs:
|
||||
|
||||
- findings from multiple collectors
|
||||
- time windows
|
||||
- region / ASN / prefix / cable relationships
|
||||
- prior incidents
|
||||
|
||||
Outputs:
|
||||
|
||||
- grouped event candidates
|
||||
- correlation rationale
|
||||
- confidence per relationship
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only
|
||||
|
||||
|
||||
### 3. Assessment Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- produce situational-awareness outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- grouped events
|
||||
- findings
|
||||
- current context
|
||||
- historical context
|
||||
- operator constraints
|
||||
|
||||
Outputs:
|
||||
|
||||
- structured assessment
|
||||
- risk summary
|
||||
- evidence-backed recommendations
|
||||
- missing-information list
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only or propose-only
|
||||
|
||||
|
||||
### 4. Recovery Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- carry low-risk proposals into controlled runtime actions
|
||||
|
||||
Inputs:
|
||||
|
||||
- approved proposal
|
||||
- policy constraints
|
||||
- trusted-domain rules
|
||||
- verification checks
|
||||
|
||||
Outputs:
|
||||
|
||||
- applied override
|
||||
- failed application
|
||||
- rollback request
|
||||
|
||||
Typical action level:
|
||||
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Shared Object Model
|
||||
|
||||
All agents should work on a shared object model.
|
||||
|
||||
That prevents the health subsystem and situational-awareness subsystem from drifting into incompatible payloads.
|
||||
|
||||
### Signal
|
||||
|
||||
Represents a raw observed fact.
|
||||
|
||||
Examples:
|
||||
|
||||
- a datasource returned HTTP 404
|
||||
- a collector returned empty results
|
||||
- BGP updates spiked in one region
|
||||
- a known endpoint now redirects elsewhere
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sig_123",
|
||||
"type": "datasource.http_failure",
|
||||
"source": "ris_live_bgp",
|
||||
"occurred_at": "2026-04-08T10:00:00Z",
|
||||
"severity": "medium",
|
||||
"payload": {},
|
||||
"trust": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Finding
|
||||
|
||||
Represents a deterministic or semi-deterministic interpretation of one or more signals.
|
||||
|
||||
Examples:
|
||||
|
||||
- `schema_changed`
|
||||
- `endpoint_unreachable`
|
||||
- `data_volume_abnormally_low`
|
||||
- `event_cluster_detected`
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "find_123",
|
||||
"type": "datasource.schema_changed",
|
||||
"source_ids": ["sig_123"],
|
||||
"confidence": 0.92,
|
||||
"evidence": [],
|
||||
"details": {}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Proposal
|
||||
|
||||
Represents a recommended action, not an already-applied action.
|
||||
|
||||
Examples:
|
||||
|
||||
- switch endpoint to new URL
|
||||
- disable bad override
|
||||
- escalate issue for manual review
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "prop_123",
|
||||
"kind": "endpoint_override",
|
||||
"target": "telegeography_cables",
|
||||
"confidence": 0.84,
|
||||
"reason": "Official docs now point to a new API path",
|
||||
"payload": {},
|
||||
"evidence_urls": [],
|
||||
"status": "proposed"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Assessment
|
||||
|
||||
Represents a structured situational-awareness output for operators or downstream systems.
|
||||
|
||||
Examples:
|
||||
|
||||
- current network posture summary
|
||||
- incident impact assessment
|
||||
- risk and response recommendations
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "assess_123",
|
||||
"scope": "regional-network",
|
||||
"risk_level": "high",
|
||||
"summary": "Regional routing instability is increasing.",
|
||||
"key_risks": [],
|
||||
"evidence": [],
|
||||
"recommendations": [],
|
||||
"missing_data": []
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## State Machine
|
||||
|
||||
The shared orchestration flow should look like this:
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Collect
|
||||
Collect --> Validate
|
||||
Validate --> Classify
|
||||
Classify --> Reason
|
||||
Reason --> Propose
|
||||
Reason --> Assess
|
||||
Propose --> Review
|
||||
Review --> Apply
|
||||
Apply --> Verify
|
||||
Verify --> Archive
|
||||
Assess --> Archive
|
||||
Archive --> [*]
|
||||
```
|
||||
|
||||
Definitions:
|
||||
|
||||
- `Collect`: gather signals
|
||||
- `Validate`: run deterministic checks
|
||||
- `Classify`: create findings
|
||||
- `Reason`: invoke LLM reasoning when needed
|
||||
- `Propose`: create change proposals
|
||||
- `Review`: policy or human approval
|
||||
- `Apply`: perform limited runtime action
|
||||
- `Verify`: confirm action effect
|
||||
- `Archive`: store artifacts and decisions
|
||||
|
||||
|
||||
## Permission Model
|
||||
|
||||
Each agent role should be assigned one of these action levels.
|
||||
|
||||
### `read-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- read signals
|
||||
- search web
|
||||
- fetch pages
|
||||
- read internal state
|
||||
- generate findings and assessments
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate config
|
||||
- write overrides
|
||||
- change live runtime behavior
|
||||
|
||||
|
||||
### `propose-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `read-only`
|
||||
- create proposals
|
||||
- create review tasks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- apply live changes
|
||||
|
||||
|
||||
### `apply-limited`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `propose-only`
|
||||
- write approved runtime overrides
|
||||
- trigger verification checks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate repository defaults
|
||||
- make destructive data changes
|
||||
- bypass policy engine
|
||||
|
||||
|
||||
## Runtime Components
|
||||
|
||||
The first durable architecture should introduce these components.
|
||||
|
||||
### 1. Signal Store
|
||||
|
||||
Stores normalized evidence and health outputs.
|
||||
|
||||
|
||||
### 2. Finding Store
|
||||
|
||||
Stores deterministic classifications that can be reused by multiple agents.
|
||||
|
||||
|
||||
### 3. Proposal Store
|
||||
|
||||
Stores recommended actions with evidence and confidence.
|
||||
|
||||
|
||||
### 4. Assessment Store
|
||||
|
||||
Stores structured situational-awareness outputs.
|
||||
|
||||
|
||||
### 5. Policy Engine
|
||||
|
||||
Decides:
|
||||
|
||||
- whether agent may run
|
||||
- whether proposal requires review
|
||||
- whether proposal may auto-apply
|
||||
- whether post-apply verification passed
|
||||
|
||||
|
||||
### 6. Override Store
|
||||
|
||||
Stores runtime-only configuration changes.
|
||||
|
||||
This is where endpoint repairs should live.
|
||||
|
||||
|
||||
## Relation To `aiprovider`
|
||||
|
||||
`aiprovider` should remain the model gateway.
|
||||
|
||||
It should not become the full agent runtime.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `aiprovider`
|
||||
- provider adaptation
|
||||
- prompt transport
|
||||
- model execution
|
||||
- protocol compatibility
|
||||
|
||||
- agent runtime
|
||||
- orchestration
|
||||
- signal handling
|
||||
- tool selection
|
||||
- proposal generation
|
||||
- policy and audit
|
||||
|
||||
This keeps provider concerns and agent behavior concerns separate.
|
||||
|
||||
|
||||
## Relation To Datasource Health
|
||||
|
||||
Datasource health becomes one vertical slice of this architecture.
|
||||
|
||||
Mapping:
|
||||
|
||||
- signal:
|
||||
- endpoint unreachable
|
||||
- schema mismatch
|
||||
- bad content type
|
||||
- finding:
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `moved_endpoint_suspected`
|
||||
- proposal:
|
||||
- runtime override suggestion
|
||||
- assessment:
|
||||
- datasource health summary for operators
|
||||
|
||||
|
||||
## Relation To Situational Awareness
|
||||
|
||||
Future situational-awareness capabilities should reuse the same flow:
|
||||
|
||||
- raw telemetry becomes signals
|
||||
- anomaly detection becomes findings
|
||||
- LLM correlation becomes reasoning
|
||||
- operator-facing output becomes assessments
|
||||
- policy-approved mitigations become actions
|
||||
|
||||
This lets the platform evolve from operational health governance into broader cyber/network posture workflows without changing the architecture.
|
||||
|
||||
|
||||
## Suggested Delivery Sequence
|
||||
|
||||
### Phase A
|
||||
|
||||
- finalize shared object model
|
||||
- implement health-oriented signal and finding storage
|
||||
|
||||
### Phase B
|
||||
|
||||
- implement Health Agent
|
||||
- generate proposals only
|
||||
|
||||
### Phase C
|
||||
|
||||
- implement Assessment Agent
|
||||
- expose structured assessments via API
|
||||
|
||||
### Phase D
|
||||
|
||||
- implement Correlation Agent
|
||||
- support multi-source incident grouping
|
||||
|
||||
### Phase E
|
||||
|
||||
- implement Recovery Agent with policy-gated runtime actions
|
||||
|
||||
|
||||
## Recommended First Build
|
||||
|
||||
The first build should not try to implement every agent role.
|
||||
|
||||
Recommended initial slice:
|
||||
|
||||
- shared object model
|
||||
- health signals
|
||||
- health findings
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
This gives immediate value while preserving the longer-term architecture.
|
||||
|
||||
|
||||
## Non-Goals For The First Iteration
|
||||
|
||||
- repository YAML auto-rewrites
|
||||
- unrestricted autonomous action
|
||||
- full incident graph reasoning
|
||||
- automatic large-scale remediation
|
||||
- agent-owned configuration source of truth
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should treat agents as a reusable runtime for evidence, reasoning, proposals, and assessments.
|
||||
|
||||
The datasource health use case is the first practical entrypoint, but the architecture should already assume future situational-awareness expansion.
|
||||
|
||||
The safest path is:
|
||||
|
||||
- deterministic checks first
|
||||
- agent reasoning second
|
||||
- proposals before actions
|
||||
- runtime overrides instead of default mutation
|
||||
346
docs/plans/agents-agent-runtime-roadmap.md
Normal file
346
docs/plans/agents-agent-runtime-roadmap.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# Agent Runtime Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
This document connects three existing planning threads into one implementation roadmap:
|
||||
|
||||
- `aiprovider` as the model gateway
|
||||
- datasource health governance as the first practical agent use case
|
||||
- situational awareness as the broader long-term target
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
|
||||
Planet should evolve in layers:
|
||||
|
||||
1. stable model gateway
|
||||
2. deterministic health and evidence collection
|
||||
3. agent runtime for reasoning and proposal generation
|
||||
4. situational-awareness assessments and controlled actions
|
||||
|
||||
This prevents the system from collapsing into a single giant "AI feature" with unclear boundaries.
|
||||
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U["Frontend / Backend APIs / Operators"] --> B["Planet Backend"]
|
||||
B --> H["Datasource Health Services"]
|
||||
B --> R["Agent Runtime"]
|
||||
R --> P["aiprovider"]
|
||||
P --> M["OpenAI / Anthropic / MiniMax / Ollama / Local Models"]
|
||||
|
||||
C["Collectors / Snapshots / Logs / Alerts / BGP Signals"] --> S["Signal Store"]
|
||||
H --> S
|
||||
S --> E["Evaluation Layer"]
|
||||
E --> F["Findings"]
|
||||
F --> R
|
||||
|
||||
W["Web Search / Page Fetch / Docs Fetch"] --> R
|
||||
R --> PR["Proposals"]
|
||||
R --> AS["Assessments"]
|
||||
|
||||
PR --> O["Runtime Overrides / Review Queue / Tasks"]
|
||||
AS --> SA["Situational Awareness APIs / UI"]
|
||||
|
||||
O --> V["Verification Loop"]
|
||||
V --> S
|
||||
```
|
||||
|
||||
|
||||
## Role Boundaries
|
||||
|
||||
### `aiprovider`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- provider compatibility
|
||||
- protocol adaptation
|
||||
- auth and model transport
|
||||
- request/response normalization
|
||||
|
||||
Not responsible for:
|
||||
|
||||
- agent orchestration
|
||||
- business workflows
|
||||
- datasource repair policy
|
||||
- situational-awareness domain logic
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- stable business APIs
|
||||
- auth and permissions
|
||||
- task orchestration
|
||||
- health records
|
||||
- proposal and override persistence
|
||||
- assessment exposure
|
||||
|
||||
|
||||
### Agent Runtime
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- consume findings and context
|
||||
- invoke LLMs via `aiprovider`
|
||||
- invoke tools such as web search
|
||||
- create proposals
|
||||
- create assessments
|
||||
- route to policy-controlled action paths
|
||||
|
||||
|
||||
## Delivery Sequence
|
||||
|
||||
## Stage 1: Gateway Foundation
|
||||
|
||||
Status:
|
||||
|
||||
- already in place
|
||||
|
||||
Delivered by current work:
|
||||
|
||||
- `aiprovider`
|
||||
- multi-provider compatibility
|
||||
- backend AI facade
|
||||
- MiniMax / Anthropic-compatible support
|
||||
- request-id propagation
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- the system already has a stable way to call models
|
||||
|
||||
|
||||
## Stage 2: Datasource Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- establish deterministic health observability
|
||||
|
||||
Key work:
|
||||
|
||||
- health check task runner
|
||||
- health result table
|
||||
- datasource health APIs
|
||||
- UI visibility
|
||||
- collector endpoint override precedence cleanup
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet knows which collectors are healthy before asking an LLM anything
|
||||
|
||||
|
||||
## Stage 3: Health Agent
|
||||
|
||||
Goal:
|
||||
|
||||
- let the first agent role operate on health failures
|
||||
|
||||
Key work:
|
||||
|
||||
- convert health failures into signals/findings
|
||||
- invoke agent only for failed or suspicious cases
|
||||
- produce repair proposals with evidence and confidence
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet can suggest endpoint repairs without mutating defaults
|
||||
|
||||
|
||||
## Stage 4: Runtime Repair Application
|
||||
|
||||
Goal:
|
||||
|
||||
- safely apply approved datasource repair proposals
|
||||
|
||||
Key work:
|
||||
|
||||
- override storage
|
||||
- policy-gated apply flow
|
||||
- verification after apply
|
||||
- rollback path
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- datasource repair becomes operationally useful without polluting repository defaults
|
||||
|
||||
|
||||
## Stage 5: Situational Awareness Assessments
|
||||
|
||||
Goal:
|
||||
|
||||
- reuse the same runtime for broader operator-facing assessment
|
||||
|
||||
Key work:
|
||||
|
||||
- normalize telemetry and incident evidence into signals/findings
|
||||
- build Assessment Agent
|
||||
- expose structured assessments through backend APIs and UI
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- LLM output becomes evidence-backed situational summary, not just ad hoc chat output
|
||||
|
||||
|
||||
## Stage 6: Correlation and Controlled Actions
|
||||
|
||||
Goal:
|
||||
|
||||
- connect multiple sources into higher-level posture and event groupings
|
||||
|
||||
Key work:
|
||||
|
||||
- event correlation
|
||||
- incident grouping
|
||||
- recommendation scoring
|
||||
- controlled action routing
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet becomes a true agent-assisted situational-awareness system
|
||||
|
||||
|
||||
## Implementation Tracks
|
||||
|
||||
These tracks can progress in parallel, but they should stay loosely coupled.
|
||||
|
||||
### Track A: Config and Runtime Resolution
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource defaults
|
||||
- overrides
|
||||
- runtime precedence
|
||||
- audit trails
|
||||
|
||||
First milestone:
|
||||
|
||||
- health-safe override layer
|
||||
|
||||
|
||||
### Track B: Health and Evidence
|
||||
|
||||
Scope:
|
||||
|
||||
- deterministic checks
|
||||
- failure categorization
|
||||
- signal and finding persistence
|
||||
|
||||
First milestone:
|
||||
|
||||
- datasource health record system
|
||||
|
||||
|
||||
### Track C: Agent Runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- shared object model
|
||||
- orchestration flow
|
||||
- prompt/tool pipeline
|
||||
- policy integration
|
||||
|
||||
First milestone:
|
||||
|
||||
- Health Agent proposal pipeline
|
||||
|
||||
|
||||
### Track D: Situational Awareness
|
||||
|
||||
Scope:
|
||||
|
||||
- assessment schema
|
||||
- multi-source context assembly
|
||||
- operator-facing outputs
|
||||
|
||||
First milestone:
|
||||
|
||||
- structured assessment API
|
||||
|
||||
|
||||
## Shared Artifacts
|
||||
|
||||
To avoid fragmentation, these artifacts should be shared across all future agent work.
|
||||
|
||||
### Shared object model
|
||||
|
||||
- `Signal`
|
||||
- `Finding`
|
||||
- `Proposal`
|
||||
- `Assessment`
|
||||
|
||||
### Shared orchestration flow
|
||||
|
||||
- collect
|
||||
- validate
|
||||
- classify
|
||||
- reason
|
||||
- propose or assess
|
||||
- review or apply
|
||||
- verify
|
||||
- archive
|
||||
|
||||
### Shared policy model
|
||||
|
||||
- read-only
|
||||
- propose-only
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Recommended Next Concrete Steps
|
||||
|
||||
1. Build Stage 2 first
|
||||
|
||||
- datasource health records
|
||||
- deterministic checks
|
||||
- no automatic repair
|
||||
|
||||
2. Then build Stage 3
|
||||
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
3. Then Stage 4
|
||||
|
||||
- override apply flow
|
||||
- rollback and verification
|
||||
|
||||
4. Only after that start Stage 5
|
||||
|
||||
- broader situational-awareness assessment workflows
|
||||
|
||||
|
||||
## Why This Order
|
||||
|
||||
Because situational-awareness quality depends on reliable upstream data.
|
||||
|
||||
If datasource health is weak:
|
||||
|
||||
- agent reasoning quality will degrade
|
||||
- false explanations will increase
|
||||
- assessment trust will drop
|
||||
|
||||
So datasource health is not a side task.
|
||||
|
||||
It is the first operational foundation for the later situational-awareness system.
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should be built as:
|
||||
|
||||
- `aiprovider` for model access
|
||||
- backend services for orchestration and persistence
|
||||
- datasource health as the first evidence-governance layer
|
||||
- agent runtime as the reusable reasoning core
|
||||
- situational awareness as the long-term application layer
|
||||
|
||||
That path keeps the architecture coherent and lets each phase produce useful functionality without forcing a rewrite later.
|
||||
486
docs/plans/agents-datasource-health-plan.md
Normal file
486
docs/plans/agents-datasource-health-plan.md
Normal file
@@ -0,0 +1,486 @@
|
||||
# Datasource Health Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines a phased plan for datasource health governance.
|
||||
|
||||
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
|
||||
|
||||
The key principle is:
|
||||
|
||||
- do not let runtime automation rewrite repository default config
|
||||
|
||||
Instead, split responsibilities across:
|
||||
|
||||
- default config
|
||||
- runtime overrides
|
||||
- health check records
|
||||
- agent-generated repair proposals
|
||||
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
|
||||
|
||||
These upstream dependencies can fail in several ways:
|
||||
|
||||
- endpoint becomes unreachable
|
||||
- endpoint still responds but schema changes
|
||||
- content-type changes
|
||||
- website shuts down or moves
|
||||
- mirror link disappears
|
||||
- HTML structure changes and scraping fails
|
||||
- endpoint requires a new path or new host
|
||||
|
||||
We want a system that can:
|
||||
|
||||
- detect datasource health degradation early
|
||||
- identify likely cause
|
||||
- search for updated endpoints when reasonable
|
||||
- apply safe runtime fixes without polluting default repo config
|
||||
- preserve auditability and rollback
|
||||
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. Default config is stable
|
||||
|
||||
- `backend/app/core/data_sources.yaml` remains the repository baseline.
|
||||
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
|
||||
|
||||
2. Runtime fixes are isolated
|
||||
|
||||
- Emergency or adaptive fixes should live in a runtime override layer.
|
||||
- Overrides should be reversible and auditable.
|
||||
|
||||
3. Deterministic checks come first
|
||||
|
||||
- Use normal programmatic health checks before using LLMs.
|
||||
- Only call an agent when deterministic checks indicate a meaningful failure.
|
||||
|
||||
4. Agents suggest before they mutate
|
||||
|
||||
- Agents should produce proposals with evidence and confidence.
|
||||
- Application of a proposal should be controlled by policy.
|
||||
|
||||
5. Every repair is attributable
|
||||
|
||||
- Store what changed, why, who or what suggested it, and when it was applied.
|
||||
|
||||
|
||||
## Configuration Layers
|
||||
|
||||
Recommended runtime precedence:
|
||||
|
||||
1. datasource endpoint override
|
||||
2. datasource DB endpoint override
|
||||
3. repository default YAML
|
||||
4. collector internal fallback logic
|
||||
|
||||
Definitions:
|
||||
|
||||
- repository default YAML:
|
||||
- `backend/app/core/data_sources.yaml`
|
||||
- versioned baseline
|
||||
- datasource DB endpoint override:
|
||||
- existing `DataSourceConfig.endpoint`
|
||||
- current runtime override entrypoint
|
||||
- datasource endpoint override:
|
||||
- a dedicated new override table
|
||||
- used for health-repair and proposal application
|
||||
- collector internal fallback logic:
|
||||
- final defensive fallback
|
||||
- should be minimized over time
|
||||
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### 1. Deterministic Health Checks
|
||||
|
||||
Each collector gets a health profile with checks such as:
|
||||
|
||||
- endpoint resolves
|
||||
- HTTP request succeeds
|
||||
- status code is acceptable
|
||||
- content-type is expected
|
||||
- body parses successfully
|
||||
- minimum structural fields exist
|
||||
- sample item count is plausible
|
||||
- latency is within threshold
|
||||
|
||||
Output states:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
|
||||
|
||||
### 2. Agent-Assisted Repair Discovery
|
||||
|
||||
Only triggered when deterministic health checks fail or return suspicious structure.
|
||||
|
||||
Agent responsibilities:
|
||||
|
||||
- search for current official endpoint or replacement path
|
||||
- inspect likely upstream documentation or landing pages
|
||||
- compare candidate endpoint output to collector expectations
|
||||
- produce a repair proposal with confidence and evidence
|
||||
|
||||
Agent should not directly modify repository defaults.
|
||||
|
||||
|
||||
### 3. Safe Runtime Repair Application
|
||||
|
||||
Repair proposals can be:
|
||||
|
||||
- reviewed manually
|
||||
- auto-applied only under strict low-risk policy
|
||||
|
||||
Auto-apply should be limited to cases like:
|
||||
|
||||
- same trusted domain
|
||||
- highly similar response structure
|
||||
- repeated successful verification
|
||||
- confidence above threshold
|
||||
|
||||
|
||||
## Phased Delivery Plan
|
||||
|
||||
## Phase 1: Deterministic Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- build health observability without automated repair
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource health check task runner
|
||||
- datasource health result persistence
|
||||
- endpoint reachability + parse checks
|
||||
- dashboard or API visibility into health status
|
||||
|
||||
Deliverables:
|
||||
|
||||
- health check service
|
||||
- health check record table
|
||||
- status endpoint
|
||||
- scheduled or manual check trigger
|
||||
|
||||
No agent usage yet.
|
||||
|
||||
|
||||
## Phase 2: Agent Repair Proposals
|
||||
|
||||
Goal:
|
||||
|
||||
- let agent investigate failing sources and propose updated endpoints
|
||||
|
||||
Scope:
|
||||
|
||||
- invoke agent only when datasource health is `failed` or `schema_changed`
|
||||
- web search + page inspection
|
||||
- candidate endpoint extraction
|
||||
- proposal persistence
|
||||
|
||||
Deliverables:
|
||||
|
||||
- repair proposal schema
|
||||
- proposal generation pipeline
|
||||
- confidence and evidence model
|
||||
- operator review view or API
|
||||
|
||||
Still no automatic config mutation.
|
||||
|
||||
|
||||
## Phase 3: Runtime Overrides
|
||||
|
||||
Goal:
|
||||
|
||||
- allow approved proposals to take effect safely at runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- add dedicated override storage
|
||||
- runtime resolution prefers override over default config
|
||||
- proposal application writes override only
|
||||
|
||||
Deliverables:
|
||||
|
||||
- endpoint override table
|
||||
- override-aware resolution logic
|
||||
- apply/reject endpoints
|
||||
- rollback endpoint
|
||||
|
||||
Repository default YAML remains untouched.
|
||||
|
||||
|
||||
## Phase 4: Limited Auto-Apply
|
||||
|
||||
Goal:
|
||||
|
||||
- safely automate a narrow slice of low-risk repairs
|
||||
|
||||
Scope:
|
||||
|
||||
- policy engine for auto-apply
|
||||
- same-domain or trusted-domain checks
|
||||
- structure validation
|
||||
- staged verification after apply
|
||||
|
||||
Deliverables:
|
||||
|
||||
- auto-apply rules
|
||||
- audit logs
|
||||
- automatic post-apply health verification
|
||||
- auto-disable or rollback on regression
|
||||
|
||||
|
||||
## Data Model Draft
|
||||
|
||||
### datasource_health_checks
|
||||
|
||||
Purpose:
|
||||
|
||||
- store each health evaluation result
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
`details` can store structured diagnostic data such as:
|
||||
|
||||
- parsed fields
|
||||
- schema mismatch summary
|
||||
- retry count
|
||||
- exception class
|
||||
|
||||
|
||||
### datasource_repair_proposals
|
||||
|
||||
Purpose:
|
||||
|
||||
- store agent-generated repair suggestions
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `old_endpoint`
|
||||
- `candidate_endpoint`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `evidence_urls`
|
||||
- `evidence_summary`
|
||||
- `status`
|
||||
- `created_by`
|
||||
- `created_at`
|
||||
- `reviewed_at`
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `proposed`
|
||||
- `approved`
|
||||
- `rejected`
|
||||
- `applied`
|
||||
- `expired`
|
||||
|
||||
|
||||
### datasource_endpoint_overrides
|
||||
|
||||
Purpose:
|
||||
|
||||
- runtime endpoint override layer
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint`
|
||||
- `reason`
|
||||
- `source`
|
||||
- `proposal_id`
|
||||
- `enabled`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
Suggested `source` values:
|
||||
|
||||
- `manual`
|
||||
- `health-agent`
|
||||
- `migration`
|
||||
|
||||
|
||||
## API Draft
|
||||
|
||||
### Health
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
### Repair proposals
|
||||
|
||||
- `GET /api/v1/datasources/{id}/repair-proposals`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
|
||||
|
||||
### Overrides
|
||||
|
||||
- `GET /api/v1/datasources/{id}/overrides`
|
||||
- `POST /api/v1/datasources/{id}/overrides`
|
||||
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
|
||||
|
||||
## Agent Contract Draft
|
||||
|
||||
When deterministic health fails, the agent should receive:
|
||||
|
||||
- datasource name
|
||||
- collector name
|
||||
- current endpoint
|
||||
- current failure mode
|
||||
- expected response shape summary
|
||||
- known trusted domains
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "proposal",
|
||||
"candidate_endpoint": "https://example.com/api/v2/data",
|
||||
"confidence": 0.86,
|
||||
"reason": "Official docs now point to v2 endpoint",
|
||||
"evidence_urls": [
|
||||
"https://example.com/docs/api",
|
||||
"https://example.com/changelog"
|
||||
],
|
||||
"notes": "Response shape appears compatible after light field remapping"
|
||||
}
|
||||
```
|
||||
|
||||
The agent should never output "rewrite the default yaml" as its primary action.
|
||||
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
### Risk: wrong endpoint chosen by agent
|
||||
|
||||
Mitigation:
|
||||
|
||||
- use trusted-domain allowlists
|
||||
- require evidence URLs
|
||||
- require confidence threshold
|
||||
- add manual review for medium-risk sources
|
||||
|
||||
|
||||
### Risk: endpoint responds but schema silently changed
|
||||
|
||||
Mitigation:
|
||||
|
||||
- deterministic schema checks
|
||||
- parse and sample validation
|
||||
- content-type checks
|
||||
- collector-specific required fields
|
||||
|
||||
|
||||
### Risk: automatic runtime override causes hidden drift
|
||||
|
||||
Mitigation:
|
||||
|
||||
- store all overrides explicitly
|
||||
- mark source of override
|
||||
- keep default YAML unchanged
|
||||
- expose active overrides in API/UI
|
||||
|
||||
|
||||
### Risk: persistent bad override breaks data collection
|
||||
|
||||
Mitigation:
|
||||
|
||||
- allow rollback
|
||||
- keep parent/default endpoint visible
|
||||
- re-run verification after apply
|
||||
- auto-disable override on repeated failure
|
||||
|
||||
|
||||
## Operational Policy Recommendations
|
||||
|
||||
1. Do not auto-apply for high-value or high-fragility sources initially.
|
||||
|
||||
2. Use manual approval for:
|
||||
|
||||
- scraped HTML sources
|
||||
- unofficial mirrors
|
||||
- sources with auth or rate-limit complexity
|
||||
- sources with legal or trust ambiguity
|
||||
|
||||
3. Allow auto-apply only for:
|
||||
|
||||
- same-domain version bumps
|
||||
- obvious official migration paths
|
||||
- repeated passing verification
|
||||
|
||||
4. Expose health + proposal + override state together in one operator view.
|
||||
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Phase 1
|
||||
- health result table
|
||||
- deterministic checks
|
||||
- API and UI visibility
|
||||
|
||||
2. Phase 2
|
||||
- proposal table
|
||||
- agent prompt/output contract
|
||||
- proposal generation job
|
||||
|
||||
3. Phase 3
|
||||
- runtime override table
|
||||
- resolver precedence update
|
||||
- apply/reject endpoints
|
||||
|
||||
4. Phase 4
|
||||
- auto-apply rules
|
||||
- rollback policy
|
||||
- operator automation
|
||||
|
||||
|
||||
## Out Of Scope For The First Iteration
|
||||
|
||||
- direct automatic mutation of repository default YAML
|
||||
- automatic git commits by repair agents
|
||||
- unrestricted autonomous endpoint replacement
|
||||
- fully generalized schema remapping engine
|
||||
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted results
|
||||
- manual visibility
|
||||
- no automatic repair
|
||||
|
||||
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.
|
||||
478
docs/plans/agents-datasource-health-stage2-tasks.md
Normal file
478
docs/plans/agents-datasource-health-stage2-tasks.md
Normal file
@@ -0,0 +1,478 @@
|
||||
# Datasource Health Stage 2 Tasks
|
||||
|
||||
## Goal
|
||||
|
||||
Stage 2 focuses on the first practical operational layer:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted health results
|
||||
- health visibility through API and UI
|
||||
- no agent-assisted repair yet
|
||||
|
||||
This stage should make Planet capable of answering:
|
||||
|
||||
- which collectors are healthy
|
||||
- which collectors are degraded
|
||||
- which collectors are failing
|
||||
- why they are failing at a basic deterministic level
|
||||
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- datasource health data model
|
||||
- deterministic health check service
|
||||
- manual and scheduled health check triggers
|
||||
- health result APIs
|
||||
- frontend visibility
|
||||
|
||||
Excluded:
|
||||
|
||||
- LLM reasoning
|
||||
- web-search-based repair proposals
|
||||
- automatic endpoint rewriting
|
||||
- runtime override application
|
||||
|
||||
|
||||
## Delivery Target
|
||||
|
||||
At the end of Stage 2, an operator should be able to:
|
||||
|
||||
1. see health status for each collector
|
||||
2. trigger a health check manually
|
||||
3. inspect the latest failure reason
|
||||
4. inspect the last checked endpoint
|
||||
5. understand whether the problem is:
|
||||
- unreachable
|
||||
- auth-related
|
||||
- rate-limit-related
|
||||
- schema-related
|
||||
- empty-data-related
|
||||
|
||||
|
||||
## Work Breakdown
|
||||
|
||||
## A. Data Model
|
||||
|
||||
### A1. Add datasource health record table
|
||||
|
||||
Create a new model, for example:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
Suggested status enum values:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
- `empty_result`
|
||||
|
||||
|
||||
### A2. Add datasource health summary fields
|
||||
|
||||
Option A:
|
||||
|
||||
- keep summary only in the health check table
|
||||
|
||||
Option B:
|
||||
|
||||
- also add summary fields on `data_sources`
|
||||
|
||||
Recommended first step:
|
||||
|
||||
- do not mutate `data_sources` schema yet
|
||||
- derive summary from the latest health record
|
||||
|
||||
|
||||
### A3. Migration task
|
||||
|
||||
Add migration for the health table.
|
||||
|
||||
Deliverables:
|
||||
|
||||
- migration file
|
||||
- model registration
|
||||
|
||||
|
||||
## B. Health Check Engine
|
||||
|
||||
### B1. Define health check service
|
||||
|
||||
Add a new service module, for example:
|
||||
|
||||
- `backend/app/services/datasource_health.py`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- resolve effective endpoint
|
||||
- execute deterministic check
|
||||
- classify result
|
||||
- persist health record
|
||||
|
||||
|
||||
### B2. Define shared result schema
|
||||
|
||||
Create a typed result object, for example:
|
||||
|
||||
- `HealthCheckResult`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `status`
|
||||
- `endpoint_checked`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
|
||||
|
||||
### B3. Implement base deterministic checks
|
||||
|
||||
Every datasource should go through a minimal baseline check:
|
||||
|
||||
1. resolve endpoint
|
||||
2. perform request
|
||||
3. measure latency
|
||||
4. inspect status code
|
||||
5. inspect content type
|
||||
6. inspect body shape
|
||||
|
||||
Classification rules:
|
||||
|
||||
- network error -> `failed`
|
||||
- HTTP 401/403 -> `auth_required`
|
||||
- HTTP 429 -> `rate_limited`
|
||||
- HTTP 404/410 -> `failed`
|
||||
- parse failure -> `schema_changed`
|
||||
- zero or suspiciously empty results -> `empty_result` or `degraded`
|
||||
- valid parse -> `healthy`
|
||||
|
||||
|
||||
### B4. Add collector-aware adapters
|
||||
|
||||
Some collectors do not use the same fetch semantics.
|
||||
|
||||
Add adapter profiles such as:
|
||||
|
||||
- `http_json`
|
||||
- `http_csv`
|
||||
- `html_scrape`
|
||||
- `stream_probe`
|
||||
- `auth_session_http`
|
||||
|
||||
Initial mapping suggestion:
|
||||
|
||||
- `huggingface`, `peeringdb`, `cloudflare` -> `http_json`
|
||||
- `fao` -> `http_csv`
|
||||
- `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape`
|
||||
- `ris_live` -> `stream_probe`
|
||||
- `spacetrack` -> `auth_session_http`
|
||||
|
||||
|
||||
### B5. Add sample validation hooks
|
||||
|
||||
For each adapter, add a lightweight validation rule.
|
||||
|
||||
Examples:
|
||||
|
||||
- JSON array length > 0
|
||||
- CSV rows > 1
|
||||
- HTML page contains expected table or script patterns
|
||||
- stream source yields at least one valid event within timeout
|
||||
|
||||
|
||||
## C. Persistence and Query Layer
|
||||
|
||||
### C1. Save every check run
|
||||
|
||||
Each health check should insert a record.
|
||||
|
||||
Do not overwrite history in Stage 2.
|
||||
|
||||
|
||||
### C2. Add latest-health query helpers
|
||||
|
||||
Add helper functions to fetch:
|
||||
|
||||
- latest health record by datasource
|
||||
- latest failed health record
|
||||
- recent health history
|
||||
|
||||
|
||||
### C3. Optional retention policy
|
||||
|
||||
For Stage 2, retention can be deferred.
|
||||
|
||||
If desired, keep only:
|
||||
|
||||
- last N records per datasource
|
||||
|
||||
|
||||
## D. API Layer
|
||||
|
||||
### D1. Add health list endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- current endpoint
|
||||
- latest health status
|
||||
- last checked time
|
||||
- short reason
|
||||
|
||||
|
||||
### D2. Add per-datasource health detail endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- latest record
|
||||
- recent history
|
||||
- detailed classification fields
|
||||
|
||||
|
||||
### D3. Add manual health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
|
||||
Behavior:
|
||||
|
||||
- run a health check now
|
||||
- persist the result
|
||||
- return the new record
|
||||
|
||||
|
||||
### D4. Add bulk health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
Behavior:
|
||||
|
||||
- enqueue or run health checks for all active datasources
|
||||
|
||||
|
||||
## E. Scheduling
|
||||
|
||||
### E1. Add health scheduler task
|
||||
|
||||
Decide scheduling strategy.
|
||||
|
||||
Recommended first version:
|
||||
|
||||
- run collector jobs and health checks separately
|
||||
- health checks run on a lower frequency
|
||||
|
||||
Suggested frequency:
|
||||
|
||||
- every 6h or 12h for most datasources
|
||||
- optionally on-demand only in the very first cut
|
||||
|
||||
|
||||
### E2. Prevent health check collision with collection
|
||||
|
||||
Rules:
|
||||
|
||||
- health checks should not disrupt active collection
|
||||
- they should use light requests
|
||||
- if a collector is currently running, health check may:
|
||||
- skip
|
||||
- or use a lightweight endpoint probe only
|
||||
|
||||
|
||||
## F. Frontend
|
||||
|
||||
### F1. Add health columns to datasource list
|
||||
|
||||
Update:
|
||||
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
|
||||
Suggested new columns:
|
||||
|
||||
- health status
|
||||
- last checked
|
||||
- reason summary
|
||||
|
||||
|
||||
### F2. Add manual health check action
|
||||
|
||||
Per datasource:
|
||||
|
||||
- button or dropdown action:
|
||||
- `健康检查`
|
||||
|
||||
|
||||
### F3. Add health detail drawer or modal
|
||||
|
||||
Show:
|
||||
|
||||
- endpoint checked
|
||||
- status
|
||||
- HTTP status
|
||||
- content type
|
||||
- sample count
|
||||
- error message
|
||||
- last few results
|
||||
|
||||
|
||||
### F4. Add basic visual language
|
||||
|
||||
Suggested colors:
|
||||
|
||||
- green -> healthy
|
||||
- yellow -> degraded
|
||||
- orange -> rate-limited / auth-required
|
||||
- red -> failed / schema-changed
|
||||
|
||||
|
||||
## G. Observability
|
||||
|
||||
### G1. Structured logging
|
||||
|
||||
Every health check should log:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- endpoint
|
||||
- status
|
||||
- latency
|
||||
- failure class
|
||||
|
||||
|
||||
### G2. Optional metrics
|
||||
|
||||
If metrics are added later, useful counters include:
|
||||
|
||||
- health checks total
|
||||
- health checks failed
|
||||
- schema changes detected
|
||||
- rate limited checks
|
||||
|
||||
|
||||
## H. Tests
|
||||
|
||||
### H1. Unit tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- status classification
|
||||
- content type classification
|
||||
- adapter behavior
|
||||
- latest-health query helpers
|
||||
|
||||
|
||||
### H2. API tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- health endpoints require auth
|
||||
- manual trigger endpoint works
|
||||
- list endpoint returns latest status
|
||||
|
||||
|
||||
### H3. Failure-path tests
|
||||
|
||||
Add coverage for:
|
||||
|
||||
- HTTP 404
|
||||
- HTTP 429
|
||||
- invalid JSON
|
||||
- empty response
|
||||
- parse mismatch
|
||||
|
||||
|
||||
## Suggested File Plan
|
||||
|
||||
Possible implementation files:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
- `backend/app/services/datasource_health.py`
|
||||
- `backend/app/schemas/datasource_health.py`
|
||||
- `backend/app/api/v1/datasource_health.py`
|
||||
- migration file under the project migration system
|
||||
|
||||
Likely touched existing files:
|
||||
|
||||
- `backend/app/api/main.py`
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
- `backend/tests/test_api.py`
|
||||
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. Add model and migration
|
||||
2. Add service and result schema
|
||||
3. Add deterministic adapters
|
||||
4. Add manual trigger API
|
||||
5. Add list/detail API
|
||||
6. Add frontend visibility
|
||||
7. Add scheduled checks
|
||||
8. Expand tests
|
||||
|
||||
|
||||
## Minimal First Milestone
|
||||
|
||||
If we want the fastest useful slice, do this first:
|
||||
|
||||
1. health table
|
||||
2. deterministic check service
|
||||
3. manual per-datasource health check API
|
||||
4. latest health list API
|
||||
5. frontend status badge column
|
||||
|
||||
That is enough to start operating the system and will provide the input layer for Stage 3.
|
||||
|
||||
|
||||
## Dependency On Later Stages
|
||||
|
||||
Stage 2 outputs become direct inputs for Stage 3.
|
||||
|
||||
Specifically:
|
||||
|
||||
- failed or schema-changed health records become agent triggers
|
||||
- health history becomes repair context
|
||||
- endpoint_checked becomes proposal baseline
|
||||
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Stage 2 is done when:
|
||||
|
||||
- every active datasource can be health-checked deterministically
|
||||
- the latest health state is visible in API and UI
|
||||
- operators can manually trigger checks
|
||||
- failures are categorized into stable machine-readable statuses
|
||||
- no LLM is required for core health visibility
|
||||
309
docs/plans/agents-situational-awareness-foundation-plan.md
Normal file
309
docs/plans/agents-situational-awareness-foundation-plan.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# Situational Awareness Foundation Plan
|
||||
|
||||
## 定位
|
||||
|
||||
当前这套 AI 能力应被视为 `态势感知服务底座`，而不是完整的态势感知产品。
|
||||
|
||||
也就是说，现阶段的目标不是：
|
||||
|
||||
- 做一个“什么都能分析”的万能 AI 页面
|
||||
- 让模型在证据不足时替代人工研判
|
||||
- 过早把页面做成完整指挥大屏
|
||||
|
||||
现阶段真正要做的是：
|
||||
|
||||
- 先把 `model gateway / backend facade / evidence injection / page-specific brief` 这几层边界搭稳
|
||||
- 让系统能够在已有证据上稳定地产出“可读、可回看、可扩展”的摘要
|
||||
- 为后续更强的数据联动、agent 推理和 assessment 结构化输出预留好接口与数据模型
|
||||
|
||||
## 当前现实约束
|
||||
|
||||
### 1. 数据维度不足
|
||||
|
||||
目前系统能提供的主要证据仍集中在：
|
||||
|
||||
- BGP incidents / anomalies / events
|
||||
- collector coverage
|
||||
- datasource health / platform alerts
|
||||
- prefix geography 的部分归属信息
|
||||
|
||||
当前明显还缺：
|
||||
|
||||
- 流量异常与业务指标
|
||||
- 电商、支付、物流等业务侧指标
|
||||
- 更丰富的资产、链路、区域、行业画像
|
||||
- 外部舆情、公告、运营商状态、基础设施事件等背景信息
|
||||
|
||||
这意味着：
|
||||
|
||||
- 模型现在可以做“基于现有证据的摘要与归纳”
|
||||
- 但还不能可靠地做“跨维度因果研判”
|
||||
|
||||
### 2. 维度之间联动还弱
|
||||
|
||||
目前不同模块之间更多是“并列展示”，还不是“强关联分析”：
|
||||
|
||||
- 系统告警和 BGP 事件还没有统一事件模型
|
||||
- collector bias 与真实区域热度还没有完全剥离
|
||||
- datasource health 与 BGP 风险、业务影响之间还没有稳定映射
|
||||
|
||||
这意味着：
|
||||
|
||||
- 当前更适合做 `brief / overview / operator notes`
|
||||
- 还不适合过度承诺“自动态势判断”
|
||||
|
||||
### 3. 结构化 assessment 还未成为主输出
|
||||
|
||||
虽然已经有 BGP brief、系统告警 brief、态势告警 brief，但目前主输出仍偏向：
|
||||
|
||||
- 文本摘要
|
||||
- facts/context 附带证据
|
||||
|
||||
后续真正要服务态势感知，需要更稳定的结构化输出，例如：
|
||||
|
||||
- summary
|
||||
- key risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing data
|
||||
|
||||
## 当前基座已经具备的能力
|
||||
|
||||
### 1. AI 调用边界已经明确
|
||||
|
||||
- `aiprovider` 负责模型协议与 provider 兼容
|
||||
- `backend` 负责业务 API、证据整合和鉴权
|
||||
- `frontend` 负责页面入口与结果展示
|
||||
|
||||
### 2. 页面级 AI 入口已经开始成型
|
||||
|
||||
当前已经有或正在收口的入口：
|
||||
|
||||
- `Playground`
|
||||
- 用于链路验证与 provider 诊断
|
||||
- `BGP AI 简报`
|
||||
- 用于 BGP 事实摘要和区域风险归纳
|
||||
- `Alerts`
|
||||
- 用于系统告警、BGP 告警、态势告警三类入口
|
||||
|
||||
### 3. 证据优先的方向已经建立
|
||||
|
||||
已经不再只依赖人工在 Playground 中手填 prompt，系统开始具备：
|
||||
|
||||
- 从真实业务数据生成事实输入
|
||||
- 保存 facts/context 快照
|
||||
- 回看 AI 输出时同时回看证据
|
||||
|
||||
这一步非常关键，因为它决定后面能否从“玩具 demo”走向“有运维价值的系统”。
|
||||
|
||||
## 近期收尾建议
|
||||
|
||||
这些事情都属于“底座收口”，值得做，但不应该再继续重产品包装。
|
||||
|
||||
### 1. 统一 Alerts 页面
|
||||
|
||||
已采用：
|
||||
|
||||
- 一个 `Alerts` 页面
|
||||
- 三个 tab：
|
||||
- `系统告警`
|
||||
- `BGP 告警`
|
||||
- `态势告警`
|
||||
|
||||
收尾重点：
|
||||
|
||||
- 保持 tab 的文案、摘要卡和 AI 简报交互一致
|
||||
- 不额外扩展成多个独立二级页面
|
||||
|
||||
### 2. 保持 Playground 为测试台
|
||||
|
||||
原则：
|
||||
|
||||
- Playground 只承担链路验证、provider 状态诊断、请求结果观察
|
||||
- 不继续堆“万能业务分析器”式交互
|
||||
|
||||
### 3. 把 brief 能力当服务能力而不是页面特效
|
||||
|
||||
页面现在能看到按钮和结果，这很好，但更重要的是：
|
||||
|
||||
- 后端接口稳定
|
||||
- facts/context 可追踪
|
||||
- 输出结构后续可升级
|
||||
|
||||
### 4. 导航结构先收口，不继续平铺一级菜单
|
||||
|
||||
随着后续能力扩展，系统很可能继续新增：
|
||||
|
||||
- 海缆
|
||||
- 算力中心
|
||||
- 战争信息
|
||||
- 电商分析
|
||||
- 其他专题观测页
|
||||
|
||||
如果继续把这些入口全部平铺在左侧一级菜单中，会带来两个问题：
|
||||
|
||||
- 一级菜单过长，用户难以判断先进入哪个上下文
|
||||
- `观测页 / 告警页 / 研判页 / 运维页` 的职责边界会被混在一起
|
||||
|
||||
因此近期应明确采用分组导航，而不是继续扩展平铺菜单。
|
||||
|
||||
推荐的导航分组如下：
|
||||
|
||||
- `总览`
|
||||
- 仪表盘
|
||||
- Earth
|
||||
- `专题观测`
|
||||
- BGP 观测
|
||||
- 采集数据
|
||||
- 后续可扩展：海缆、算力中心、战争信息、电商分析
|
||||
- `告警与研判`
|
||||
- Alerts
|
||||
- `运维与配置`
|
||||
- 数据源
|
||||
- AI Playground
|
||||
- 用户管理
|
||||
- 系统配置
|
||||
|
||||
这套结构的含义是：
|
||||
|
||||
- `专题观测` 页面负责看某个维度本身
|
||||
- `Alerts` 负责跨模块风险与值班工作台
|
||||
- `Playground` 保持为测试台，不挤占业务导航语义
|
||||
|
||||
短期收尾时，应优先重组现有入口，而不是继续增加新的一级菜单。
|
||||
|
||||
## 后续路线
|
||||
|
||||
## Phase 1：服务底座稳固
|
||||
|
||||
目标：
|
||||
|
||||
- 不追求“更炫的 AI 页面”
|
||||
- 先把当前接口、证据、存储和页面入口收稳
|
||||
|
||||
工作项：
|
||||
|
||||
- 统一页面级 AI 入口模式
|
||||
- 统一 brief response schema
|
||||
- 保证 facts/context 在前后端都可回看
|
||||
- 继续清理 mock 和临时分支逻辑
|
||||
|
||||
完成标准：
|
||||
|
||||
- 每个 AI 入口都是真实链路
|
||||
- 每个 AI 结果都能追溯到证据输入
|
||||
|
||||
## Phase 2：Evidence-first Assessment
|
||||
|
||||
目标：
|
||||
|
||||
- 从“文本摘要”升级成“结构化 assessment”
|
||||
|
||||
工作项：
|
||||
|
||||
- 为 brief/assessment 定义统一 schema
|
||||
- 固化：
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing_data
|
||||
- 页面以结构化区块展示，而不只是大段文本
|
||||
|
||||
完成标准：
|
||||
|
||||
- AI 输出可持久化、可比较、可审计
|
||||
|
||||
## Phase 3：多维证据接入
|
||||
|
||||
目标：
|
||||
|
||||
- 让“态势感知”真正拥有更多维度，而不是只靠 BGP 与系统告警
|
||||
|
||||
优先接入方向：
|
||||
|
||||
- datasource health findings
|
||||
- 流量或业务指标
|
||||
- 区域/资产/链路映射
|
||||
- 外部事件与公告
|
||||
- 业务垂直数据，例如电商分析相关指标
|
||||
|
||||
完成标准：
|
||||
|
||||
- AI 能基于多个维度做交叉说明
|
||||
- 不再只围绕单一模块自说自话
|
||||
|
||||
## Phase 4：Correlation Layer
|
||||
|
||||
目标：
|
||||
|
||||
- 不同来源的信号不再只是并列，而是形成统一的事件关联
|
||||
|
||||
工作项：
|
||||
|
||||
- 统一 signal/finding 模型
|
||||
- 跨模块事件聚合
|
||||
- 证据来源权重
|
||||
- collector bias 与真实热度分离
|
||||
|
||||
完成标准：
|
||||
|
||||
- 系统能回答“这些异常是不是同一件事”
|
||||
- 系统能回答“哪些结论只是观测偏差”
|
||||
|
||||
## Phase 5：Agent-assisted Situational Awareness
|
||||
|
||||
目标：
|
||||
|
||||
- 在证据足够的前提下，再让 agent 负责更复杂的推理与建议
|
||||
|
||||
工作项：
|
||||
|
||||
- 复用现有 agent runtime 规划
|
||||
- 引入 web search / docs fetch / repair proposal 等能力
|
||||
- 但始终坚持：
|
||||
- evidence first
|
||||
- proposal before action
|
||||
- no silent mutation of defaults
|
||||
|
||||
完成标准：
|
||||
|
||||
- agent 成为证据驱动的分析层
|
||||
- 而不是一个“万能猜测层”
|
||||
|
||||
## 设计原则
|
||||
|
||||
### 1. 先底座，后产品化
|
||||
|
||||
先把服务链路和证据模型做好，再做更大的页面表达。
|
||||
|
||||
### 2. 先证据，后判断
|
||||
|
||||
事实输入应先稳定，再让模型做归纳。
|
||||
|
||||
### 3. 先专用 brief，后统一态势层
|
||||
|
||||
先让各业务页有各自可信的 AI 入口，再考虑统一态势页。
|
||||
|
||||
### 4. 先 proposal，后自动动作
|
||||
|
||||
涉及修复、覆盖、写配置、调任务的动作，都应经过 proposal 和审计。
|
||||
|
||||
## 当前建议结论
|
||||
|
||||
对现在这个项目，最合理的定位是：
|
||||
|
||||
- `Playground` 是测试台
|
||||
- `BGP / Alerts` 是第一批业务 AI 入口
|
||||
- `aiprovider + backend AI facade + evidence snapshots` 是核心服务底座
|
||||
|
||||
现阶段不需要追求“已经具备完整态势感知能力”。
|
||||
|
||||
现阶段真正的成功标准是：
|
||||
|
||||
- 这套底座可用
|
||||
- 可回看
|
||||
- 可扩展
|
||||
- 不自欺欺人
|
||||
402
docs/plans/backend-collected-data-history-plan.md
Normal file
402
docs/plans/backend-collected-data-history-plan.md
Normal file
@@ -0,0 +1,402 @@
|
||||
# 采集数据历史快照化改造方案
|
||||
|
||||
## 背景
|
||||
|
||||
当前系统的 `collected_data` 更接近“当前结果表”：
|
||||
|
||||
- 同一个 `source + source_id` 会被更新覆盖
|
||||
- 前端列表页默认读取这张表
|
||||
- `collection_tasks` 只记录任务执行状态，不直接承载数据版本语义
|
||||
|
||||
这套方式适合管理后台，但不利于后续做态势感知、时间回放、趋势分析和版本对比。
|
||||
如果后面需要回答下面这类问题，当前模型会比较吃力：
|
||||
|
||||
- 某条实体在过去 7 天如何变化
|
||||
- 某次采集相比上次新增了什么、删除了什么、值变了什么
|
||||
- 某个时刻地图上“当时的世界状态”是什么
|
||||
- 告警是在第几次采集后触发的
|
||||
|
||||
因此建议把采集数据改造成“历史快照 + 当前视图”模型。
|
||||
|
||||
## 目标
|
||||
|
||||
1. 每次触发采集都保留一份独立快照，历史可追溯。
|
||||
2. 管理后台默认仍然只看“当前最新状态”，不增加使用复杂度。
|
||||
3. 后续支持：
|
||||
- 时间线回放
|
||||
- 两次采集差异对比
|
||||
- 趋势分析
|
||||
- 按快照回溯告警和地图状态
|
||||
4. 尽量兼容现有接口，降低改造成本。
|
||||
|
||||
## 结论
|
||||
|
||||
不建议继续用以下两种单一模式：
|
||||
|
||||
- 直接覆盖旧数据
|
||||
问题：没有历史，无法回溯。
|
||||
|
||||
- 软删除旧数据再全量新增
|
||||
问题：语义不清，历史和“当前无效”混在一起，后续统计复杂。
|
||||
|
||||
推荐方案：
|
||||
|
||||
- 保留历史事实表
|
||||
- 维护当前视图
|
||||
- 每次采集对应一个明确的快照批次
|
||||
|
||||
## 推荐数据模型
|
||||
|
||||
### 方案概览
|
||||
|
||||
建议拆成三层：
|
||||
|
||||
1. `collection_tasks`
|
||||
继续作为采集任务表，表示“这次采集任务”。
|
||||
|
||||
2. `data_snapshots`
|
||||
新增快照表，表示“某个数据源在某次任务中产出的一个快照批次”。
|
||||
|
||||
3. `collected_data`
|
||||
从“当前结果表”升级为“历史事实表”，每一行归属于一个快照。
|
||||
|
||||
同时再提供一个“当前视图”：
|
||||
|
||||
- SQL View / 物化视图 / API 查询层封装均可
|
||||
- 语义是“每个 `source + source_id` 的最新有效记录”
|
||||
|
||||
### 新增表：`data_snapshots`
|
||||
|
||||
建议字段：
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|---|---|---|
|
||||
| `id` | bigint PK | 快照主键 |
|
||||
| `datasource_id` | int | 对应数据源 |
|
||||
| `task_id` | int | 对应采集任务 |
|
||||
| `source` | varchar(100) | 数据源名，如 `top500` |
|
||||
| `snapshot_key` | varchar(100) | 可选，业务快照标识 |
|
||||
| `reference_date` | timestamptz nullable | 这批数据的参考时间 |
|
||||
| `started_at` | timestamptz | 快照开始时间 |
|
||||
| `completed_at` | timestamptz | 快照完成时间 |
|
||||
| `record_count` | int | 快照总记录数 |
|
||||
| `status` | varchar(20) | `running/success/failed/partial` |
|
||||
| `is_current` | bool | 当前是否是该数据源最新快照 |
|
||||
| `parent_snapshot_id` | bigint nullable | 上一版快照，可用于 diff |
|
||||
| `summary` | jsonb | 本次快照统计摘要 |
|
||||
|
||||
说明：
|
||||
|
||||
- `collection_tasks` 偏“执行过程”
|
||||
- `data_snapshots` 偏“数据版本”
|
||||
- 一个任务通常对应一个快照，但保留分层更清晰
|
||||
|
||||
### 升级表：`collected_data`
|
||||
|
||||
建议新增字段：
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|---|---|---|
|
||||
| `snapshot_id` | bigint not null | 归属快照 |
|
||||
| `task_id` | int nullable | 归属任务，便于追查 |
|
||||
| `entity_key` | varchar(255) | 实体稳定键，通常可由 `source + source_id` 派生 |
|
||||
| `is_current` | bool | 当前是否为该实体最新记录 |
|
||||
| `previous_record_id` | bigint nullable | 上一个版本的记录 |
|
||||
| `change_type` | varchar(20) | `created/updated/unchanged/deleted` |
|
||||
| `change_summary` | jsonb | 字段变化摘要 |
|
||||
| `deleted_at` | timestamptz nullable | 对应“本次快照中消失”的实体 |
|
||||
|
||||
保留现有字段：
|
||||
|
||||
- `source`
|
||||
- `source_id`
|
||||
- `data_type`
|
||||
- `name`
|
||||
- `title`
|
||||
- `description`
|
||||
- `country`
|
||||
- `city`
|
||||
- `latitude`
|
||||
- `longitude`
|
||||
- `value`
|
||||
- `unit`
|
||||
- `metadata`
|
||||
- `collected_at`
|
||||
- `reference_date`
|
||||
- `is_valid`
|
||||
|
||||
### 当前视图
|
||||
|
||||
建议新增一个只读视图：
|
||||
|
||||
`current_collected_data`
|
||||
|
||||
语义：
|
||||
|
||||
- 对每个 `source + source_id` 只保留最新一条 `is_current = true` 且 `deleted_at is null` 的记录
|
||||
|
||||
这样：
|
||||
|
||||
- 管理后台继续像现在一样查“当前数据”
|
||||
- 历史分析查 `collected_data`
|
||||
|
||||
## 写入策略
|
||||
|
||||
### 触发按钮语义
|
||||
|
||||
“触发”不再理解为“覆盖旧表”，而是：
|
||||
|
||||
- 启动一次新的采集任务
|
||||
- 生成一个新的快照
|
||||
- 将本次结果写入历史事实表
|
||||
- 再更新当前视图标记
|
||||
|
||||
### 写入流程
|
||||
|
||||
1. 创建 `collection_tasks` 记录，状态 `running`
|
||||
2. 创建 `data_snapshots` 记录，状态 `running`
|
||||
3. 采集器拉取原始数据并标准化
|
||||
4. 为每条记录生成 `entity_key`
|
||||
- 推荐：`{source}:{source_id}`
|
||||
5. 将本次记录批量写入 `collected_data`
|
||||
6. 与上一个快照做比对，计算：
|
||||
- 新增
|
||||
- 更新
|
||||
- 未变
|
||||
- 删除
|
||||
7. 更新本批记录的：
|
||||
- `change_type`
|
||||
- `previous_record_id`
|
||||
- `is_current`
|
||||
8. 将上一批同实体记录的 `is_current` 置为 `false`
|
||||
9. 将本次快照未出现但上一版存在的实体标记为 `deleted`
|
||||
10. 更新 `data_snapshots.status = success`
|
||||
11. 更新 `collection_tasks.status = success`
|
||||
|
||||
### 删除语义
|
||||
|
||||
这里不建议真的删记录。
|
||||
建议采用“逻辑消失”模型：
|
||||
|
||||
- 历史行永远保留
|
||||
- 如果某实体在新快照里消失：
|
||||
- 上一条历史记录补一条“删除状态记录”或标记 `change_type = deleted`
|
||||
- 同时该实体不再出现在当前视图
|
||||
|
||||
这样最适合态势感知。
|
||||
|
||||
## API 改造建议
|
||||
|
||||
### 保持现有接口默认行为
|
||||
|
||||
现有接口：
|
||||
|
||||
- `GET /api/v1/collected`
|
||||
- `GET /api/v1/collected/{id}`
|
||||
- `GET /api/v1/collected/summary`
|
||||
|
||||
建议默认仍返回“当前视图”，避免前端全面重写。
|
||||
|
||||
### 新增历史查询能力
|
||||
|
||||
建议新增参数或新接口：
|
||||
|
||||
#### 1. 当前/历史切换
|
||||
|
||||
`GET /api/v1/collected?mode=current|history`
|
||||
|
||||
- `current`：默认，查当前视图
|
||||
- `history`：查历史事实表
|
||||
|
||||
#### 2. 按快照查询
|
||||
|
||||
`GET /api/v1/collected?snapshot_id=123`
|
||||
|
||||
#### 3. 快照列表
|
||||
|
||||
`GET /api/v1/snapshots`
|
||||
|
||||
支持筛选：
|
||||
|
||||
- `datasource_id`
|
||||
- `source`
|
||||
- `status`
|
||||
- `date_from/date_to`
|
||||
|
||||
#### 4. 快照详情
|
||||
|
||||
`GET /api/v1/snapshots/{id}`
|
||||
|
||||
返回：
|
||||
|
||||
- 快照基础信息
|
||||
- 统计摘要
|
||||
- 与上一版的 diff 摘要
|
||||
|
||||
#### 5. 快照 diff
|
||||
|
||||
`GET /api/v1/snapshots/{id}/diff?base_snapshot_id=122`
|
||||
|
||||
返回：
|
||||
|
||||
- `created`
|
||||
- `updated`
|
||||
- `deleted`
|
||||
- `unchanged`
|
||||
|
||||
## 前端改造建议
|
||||
|
||||
### 1. 数据列表页
|
||||
|
||||
默认仍看当前数据，不改用户使用习惯。
|
||||
|
||||
建议新增：
|
||||
|
||||
- “视图模式”
|
||||
- 当前数据
|
||||
- 历史数据
|
||||
- “快照时间”筛选
|
||||
- “只看变化项”筛选
|
||||
|
||||
### 2. 数据详情页
|
||||
|
||||
详情页建议展示：
|
||||
|
||||
- 当前记录基础信息
|
||||
- 元数据动态字段
|
||||
- 所属快照
|
||||
- 上一版本对比入口
|
||||
- 历史版本时间线
|
||||
|
||||
### 3. 数据源管理页
|
||||
|
||||
“触发”按钮文案建议改成更准确的：
|
||||
|
||||
- `立即采集`
|
||||
|
||||
并在详情里补：
|
||||
|
||||
- 最近一次快照时间
|
||||
- 最近一次快照记录数
|
||||
- 最近一次变化数
|
||||
|
||||
## 迁移方案
|
||||
|
||||
### Phase 1：兼容式落地
|
||||
|
||||
目标：先保留当前页面可用。
|
||||
|
||||
改动：
|
||||
|
||||
1. 新增 `data_snapshots`
|
||||
2. 给 `collected_data` 增加：
|
||||
- `snapshot_id`
|
||||
- `task_id`
|
||||
- `entity_key`
|
||||
- `is_current`
|
||||
- `previous_record_id`
|
||||
- `change_type`
|
||||
- `change_summary`
|
||||
- `deleted_at`
|
||||
3. 现有数据全部补成一个“初始化快照”
|
||||
4. 现有 `/collected` 默认改查当前视图
|
||||
|
||||
优点：
|
||||
|
||||
- 前端几乎无感
|
||||
- 风险最小
|
||||
|
||||
### Phase 2：启用差异计算
|
||||
|
||||
目标：采集后可知道本次改了什么。
|
||||
|
||||
改动：
|
||||
|
||||
1. 写入时做新旧快照比对
|
||||
2. 写 `change_type`
|
||||
3. 生成快照摘要
|
||||
|
||||
### Phase 3：前端态势感知能力
|
||||
|
||||
目标：支持历史回放和趋势分析。
|
||||
|
||||
改动：
|
||||
|
||||
1. 快照时间线
|
||||
2. 版本 diff 页面
|
||||
3. 地图时间回放
|
||||
4. 告警和快照关联
|
||||
|
||||
## 唯一性与索引建议
|
||||
|
||||
### 建议保留的业务唯一性
|
||||
|
||||
在“同一个快照内部”，建议唯一：
|
||||
|
||||
- `(snapshot_id, source, source_id)`
|
||||
|
||||
不要在整张历史表上强加：
|
||||
|
||||
- `(source, source_id)` 唯一
|
||||
|
||||
因为历史表本来就应该允许同一实体跨快照存在多条版本。
|
||||
|
||||
### 建议索引
|
||||
|
||||
- `idx_collected_data_snapshot_id`
|
||||
- `idx_collected_data_source_source_id`
|
||||
- `idx_collected_data_entity_key`
|
||||
- `idx_collected_data_is_current`
|
||||
- `idx_collected_data_reference_date`
|
||||
- `idx_snapshots_source_completed_at`
|
||||
|
||||
## 风险点
|
||||
|
||||
1. 存储量会明显增加
|
||||
- 需要评估保留周期
|
||||
- 可以考虑冷热分层
|
||||
|
||||
2. 写入复杂度上升
|
||||
- 需要批量 upsert / diff 逻辑
|
||||
|
||||
3. 当前接口语义会从“表”变成“视图”
|
||||
- 文档必须同步
|
||||
|
||||
4. 某些采集器缺稳定 `source_id`
|
||||
- 需要补齐实体稳定键策略
|
||||
|
||||
## 对当前项目的具体建议
|
||||
|
||||
结合当前代码，推荐这样落地：
|
||||
|
||||
### 短期
|
||||
|
||||
1. 先设计并落表：
|
||||
- `data_snapshots`
|
||||
- `collected_data` 新字段
|
||||
2. 采集完成后每次新增快照
|
||||
3. `/api/v1/collected` 默认查 `is_current = true`
|
||||
|
||||
### 中期
|
||||
|
||||
1. 在 `BaseCollector._save_data()` 中改成：
|
||||
- 生成快照
|
||||
- 批量写历史
|
||||
- 标记当前
|
||||
2. 将 `CollectionTask.id` 关联到 `snapshot.task_id`
|
||||
|
||||
### 长期
|
||||
|
||||
1. 地图接口支持按 `snapshot_id` 查询
|
||||
2. 仪表盘支持“最近一次快照变化量”
|
||||
3. 告警支持绑定到快照版本
|
||||
|
||||
## 最终建议
|
||||
|
||||
最终建议采用：
|
||||
|
||||
- 历史事实表：保存每次采集结果
|
||||
- 当前视图：服务管理后台默认查询
|
||||
- 快照表：承载版本批次和 diff 语义
|
||||
|
||||
这样既能保留历史，又不会把当前页面全部推翻重做，是最适合后续做态势感知的一条路径。
|
||||
48
docs/plans/backend-system-settings-plan.md
Normal file
48
docs/plans/backend-system-settings-plan.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# 系统配置中心开发计划
|
||||
|
||||
## 目标
|
||||
|
||||
将当前仅保存于内存中的“系统配置”页面升级为真正可用的配置中心，优先服务以下两类能力：
|
||||
|
||||
1. 系统级配置持久化
|
||||
2. 采集调度配置管理
|
||||
|
||||
## 第一阶段范围
|
||||
|
||||
### 1. 系统配置持久化
|
||||
|
||||
- 新增 `system_settings` 表，用于保存分类配置
|
||||
- 将系统、通知、安全配置从进程内存迁移到数据库
|
||||
- 提供统一读取接口，页面刷新和服务重启后保持不丢失
|
||||
|
||||
### 2. 采集调度配置接入真实数据源
|
||||
|
||||
- 统一内置采集器默认定义
|
||||
- 启动时自动初始化 `data_sources` 表
|
||||
- 配置页允许修改：
|
||||
- 是否启用
|
||||
- 采集频率（分钟）
|
||||
- 优先级
|
||||
- 修改后实时同步到调度器
|
||||
|
||||
### 3. 前端配置页重构
|
||||
|
||||
- 将当前通用模板页调整为项目专用配置中心
|
||||
- 增加“采集调度”Tab
|
||||
- 保留“系统显示 / 通知 / 安全”三类配置
|
||||
- 将设置页正式接入主路由
|
||||
|
||||
## 非本阶段内容
|
||||
|
||||
- 邮件发送能力本身
|
||||
- 配置审计历史
|
||||
- 敏感凭证加密管理
|
||||
- 多租户或按角色细粒度配置
|
||||
|
||||
## 验收标准
|
||||
|
||||
- 设置项修改后重启服务仍然存在
|
||||
- 配置页可以查看并修改所有内置采集器的启停与采集频率
|
||||
- 调整采集频率后，调度器任务随之更新
|
||||
- `/settings` 页面可从主导航进入并正常工作
|
||||
|
||||
296
docs/plans/earth-bgp-earth-rendering-plan.md
Normal file
296
docs/plans/earth-bgp-earth-rendering-plan.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# BGP Earth Rendering Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This document defines how the BGP `region activity layer` and `incident layer` should coexist on Earth without conflicting.
|
||||
|
||||
The main question it answers is:
|
||||
|
||||
- how to add a regional observability background layer
|
||||
- without weakening the current incident-first event focus
|
||||
|
||||
## Core Principle
|
||||
|
||||
The Earth design should follow a strict semantic hierarchy:
|
||||
|
||||
- `collector layer` = observation infrastructure
|
||||
- `region activity layer` = background situational awareness
|
||||
- `incident layer` = focal high-confidence event objects
|
||||
|
||||
In short:
|
||||
|
||||
- collectors prove the network is observing
|
||||
- regions show where routing behavior is active or abnormal
|
||||
- incidents show the concrete event worth clicking
|
||||
|
||||
Region aggregation is therefore not a replacement for incident rendering.
|
||||
It is the context layer that makes sparse incident markers legible.
|
||||
|
||||
## Rendering Hierarchy
|
||||
|
||||
Recommended visual stack order:
|
||||
|
||||
1. collector network / collector halos
|
||||
2. region activity glow
|
||||
3. incident markers and incident pulses
|
||||
|
||||
This ordering should always hold.
|
||||
|
||||
Why:
|
||||
|
||||
- collectors should stay visible but quiet
|
||||
- regions should create ambient activity presence
|
||||
- incidents must remain the first thing users notice as a concrete event
|
||||
|
||||
## Role Separation
|
||||
|
||||
### Region Layer
|
||||
|
||||
The region layer answers:
|
||||
|
||||
- where is routing activity building up
|
||||
- where is there current noise or instability
|
||||
- which part of the world is currently worth looking at
|
||||
|
||||
The region layer should feel:
|
||||
|
||||
- broad
|
||||
- ambient
|
||||
- low-frequency
|
||||
- contextual
|
||||
|
||||
### Incident Layer
|
||||
|
||||
The incident layer answers:
|
||||
|
||||
- which exact event should the user inspect
|
||||
- where is the highest-confidence routing event located right now
|
||||
|
||||
The incident layer should feel:
|
||||
|
||||
- sharp
|
||||
- compact
|
||||
- high-contrast
|
||||
- intentionally clickable
|
||||
|
||||
## Non-Conflict Rules
|
||||
|
||||
To avoid visual and semantic conflict, these implementation rules should be treated as hard constraints:
|
||||
|
||||
1. region markers must not use the same symbol language as incidents
|
||||
2. region emphasis must stay weaker than incident emphasis
|
||||
3. region animation frequency must stay lower than incident animation frequency
|
||||
4. incident markers must always render above region glows
|
||||
5. region layer should support the event, not compete with it
|
||||
|
||||
If a user notices the region layer first but misses the incident marker, the region layer is too strong.
|
||||
|
||||
If a user only sees isolated incident points and cannot feel broader activity context, the region layer is too weak.
|
||||
|
||||
## Region Rendering Rules
|
||||
|
||||
The region layer should not be rendered as a second kind of incident point.
|
||||
|
||||
Recommended representation:
|
||||
|
||||
- diffuse glow
|
||||
- halo
|
||||
- low-detail pulse
|
||||
- soft center, not a sharp icon
|
||||
|
||||
### Status Mapping
|
||||
|
||||
#### `observing`
|
||||
|
||||
- weak glow
|
||||
- cool color, such as cyan or blue
|
||||
- little to no pulse
|
||||
- purpose: keep the globe alive during calm periods
|
||||
|
||||
#### `anomaly`
|
||||
|
||||
- stronger glow
|
||||
- warmer color, such as amber
|
||||
- gentle breathing or low-frequency pulse
|
||||
- purpose: show that a region is experiencing abnormal routing noise
|
||||
|
||||
#### `incident`
|
||||
|
||||
- strongest regional background emphasis
|
||||
- still clearly weaker than the incident marker itself
|
||||
- purpose: lift the surrounding area so the focal event does not feel isolated
|
||||
|
||||
### Region Visual Characteristics
|
||||
|
||||
Recommended properties:
|
||||
|
||||
- large radius
|
||||
- low opacity
|
||||
- soft edge
|
||||
- low-contrast outline or no outline
|
||||
- low pulse amplitude
|
||||
|
||||
Avoid:
|
||||
|
||||
- sharp symbol shapes
|
||||
- strong icon silhouettes
|
||||
- bright hard-edged centers
|
||||
- incident-like pulse language
|
||||
|
||||
## Incident Rendering Rules
|
||||
|
||||
The incident layer should remain visually sharper and more explicit than region activity.
|
||||
|
||||
Recommended qualities:
|
||||
|
||||
- clear event symbol
|
||||
- compact hot core
|
||||
- one or two outward ring pulses
|
||||
- high contrast
|
||||
- clear click target
|
||||
|
||||
The incident layer should read as:
|
||||
|
||||
- focal
|
||||
- deliberate
|
||||
- high-confidence
|
||||
|
||||
while the region layer should read as:
|
||||
|
||||
- contextual
|
||||
- ambient
|
||||
- supporting
|
||||
|
||||
## Region And Incident In The Same Area
|
||||
|
||||
When a region contains one or more incidents:
|
||||
|
||||
- the region glow may intensify
|
||||
- but the incident marker must remain the dominant local feature
|
||||
|
||||
Interpretation should be:
|
||||
|
||||
- `region` says this area is in an event state
|
||||
- `incident marker` says this is the concrete event object
|
||||
|
||||
So a region with `incident` status is not itself the event marker.
|
||||
It is the background state around the event.
|
||||
|
||||
## Interaction Model
|
||||
|
||||
Interaction should also preserve hierarchy.
|
||||
|
||||
### Click Region
|
||||
|
||||
Open a regional situation view, such as:
|
||||
|
||||
- region name
|
||||
- observation count
|
||||
- anomaly count
|
||||
- incident count
|
||||
- affected prefix count
|
||||
- affected ASN count
|
||||
- recent incidents in the region
|
||||
|
||||
### Click Incident
|
||||
|
||||
Keep the current incident-focused detail interaction.
|
||||
|
||||
This creates a natural two-step flow:
|
||||
|
||||
1. region gives context
|
||||
2. incident gives detail
|
||||
|
||||
## Layer Relationship To Existing BGP Elements
|
||||
|
||||
### Collector Layer
|
||||
|
||||
Collectors should remain:
|
||||
|
||||
- quieter than regions
|
||||
- more infrastructural than semantic
|
||||
- proof of coverage, not proof of incident
|
||||
|
||||
### Region Layer
|
||||
|
||||
Regions should become:
|
||||
|
||||
- the main ambient activity layer
|
||||
- the bridge between collectors and incidents
|
||||
- the answer to low-density map quietness
|
||||
|
||||
### Incident Layer
|
||||
|
||||
Incidents should remain:
|
||||
|
||||
- the most legible event layer
|
||||
- sparse but dominant
|
||||
- compact and symbol-driven
|
||||
|
||||
## Practical Visual Test
|
||||
|
||||
Use this test when tuning the Earth implementation:
|
||||
|
||||
### Calm Period
|
||||
|
||||
Expected result:
|
||||
|
||||
- collectors visible
|
||||
- some weak region glows present
|
||||
- no region feels alarm-heavy
|
||||
- globe still feels alive
|
||||
|
||||
### Anomaly Period
|
||||
|
||||
Expected result:
|
||||
|
||||
- one or more regions brighten noticeably
|
||||
- user can sense the active area before clicking
|
||||
- still no confusion between region background and incident objects
|
||||
|
||||
### Incident Period
|
||||
|
||||
Expected result:
|
||||
|
||||
- region provides broader context
|
||||
- incident marker is the first explicit focal object the eye lands on
|
||||
- user can immediately tell both:
|
||||
- which region is active
|
||||
- which specific event to inspect
|
||||
|
||||
## Failure Modes To Avoid
|
||||
|
||||
### Region Too Strong
|
||||
|
||||
Symptoms:
|
||||
|
||||
- incident markers disappear into the glow
|
||||
- users treat the region center as the main event
|
||||
- the map feels like area flooding instead of event focus
|
||||
|
||||
### Region Too Weak
|
||||
|
||||
Symptoms:
|
||||
|
||||
- incident markers still feel isolated
|
||||
- low-incident periods still look visually empty
|
||||
- users cannot tell where routing activity is generally happening
|
||||
|
||||
### Region Uses Incident Language
|
||||
|
||||
Symptoms:
|
||||
|
||||
- region and incident both look like event markers
|
||||
- users cannot distinguish context from event
|
||||
|
||||
## Final Design Rule
|
||||
|
||||
The desired reading order is:
|
||||
|
||||
1. see the specific incident marker
|
||||
2. feel the active region around it
|
||||
3. understand that collectors and background activity keep the globe alive even during quieter periods
|
||||
|
||||
In one sentence:
|
||||
|
||||
`incident is the point; region is the field.`
|
||||
487
docs/plans/earth-bgp-observability-plan.md
Normal file
487
docs/plans/earth-bgp-observability-plan.md
Normal file
@@ -0,0 +1,487 @@
|
||||
# BGP Observability Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Build a global routing observability capability on top of:
|
||||
|
||||
- [RIPE RIS Live](https://ris-live.ripe.net/)
|
||||
- [CAIDA BGPStream data access overview](https://bgpstream.caida.org/docs/overview/data-access)
|
||||
|
||||
The target is to support:
|
||||
|
||||
- real-time routing event ingestion
|
||||
- historical replay and baseline analysis
|
||||
- anomaly detection
|
||||
- Earth big-screen visualization
|
||||
|
||||
## Important Scope Note
|
||||
|
||||
These data sources expose the BGP control plane, not user traffic itself.
|
||||
|
||||
That means the system can infer:
|
||||
|
||||
- route propagation direction
|
||||
- prefix reachability changes
|
||||
- AS path changes
|
||||
- visibility changes across collectors
|
||||
|
||||
But it cannot directly measure:
|
||||
|
||||
- exact application traffic volume
|
||||
- exact user packet path
|
||||
- real bandwidth consumption between countries or operators
|
||||
|
||||
Product wording should therefore use phrases like:
|
||||
|
||||
- global routing propagation
|
||||
- route visibility
|
||||
- control-plane anomalies
|
||||
- suspected path diversion
|
||||
|
||||
Instead of claiming direct traffic measurement.
|
||||
|
||||
## Data Source Roles
|
||||
|
||||
### RIS Live
|
||||
|
||||
Use RIS Live as the real-time feed.
|
||||
|
||||
Recommended usage:
|
||||
|
||||
- subscribe to update streams over WebSocket
|
||||
- ingest announcements and withdrawals continuously
|
||||
- trigger low-latency alerts
|
||||
|
||||
Best suited for:
|
||||
|
||||
- hijack suspicion
|
||||
- withdrawal bursts
|
||||
- real-time path changes
|
||||
- live Earth event overlay
|
||||
|
||||
### BGPStream
|
||||
|
||||
Use BGPStream as the historical and replay layer.
|
||||
|
||||
Recommended usage:
|
||||
|
||||
- backfill time windows
|
||||
- build normal baselines
|
||||
- compare current events against history
|
||||
- support investigations and playback
|
||||
|
||||
Best suited for:
|
||||
|
||||
- historical anomaly confirmation
|
||||
- baseline path frequency
|
||||
- visibility baselines
|
||||
- postmortem analysis
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["RIS Live WebSocket"] --> B["Realtime Collector"]
|
||||
C["BGPStream Historical Access"] --> D["Backfill Collector"]
|
||||
B --> E["Normalization Layer"]
|
||||
D --> E
|
||||
E --> F["data_snapshots"]
|
||||
E --> G["collected_data"]
|
||||
E --> H["bgp_anomalies"]
|
||||
H --> I["Alerts API"]
|
||||
G --> J["Visualization API"]
|
||||
H --> J
|
||||
J --> K["Earth Big Screen"]
|
||||
```
|
||||
|
||||
## Storage Design
|
||||
|
||||
The current project already has:
|
||||
|
||||
- [data_snapshot.py](/home/ray/dev/linkong/planet/backend/app/models/data_snapshot.py)
|
||||
- [collected_data.py](/home/ray/dev/linkong/planet/backend/app/models/collected_data.py)
|
||||
|
||||
So the lowest-risk path is:
|
||||
|
||||
1. keep raw and normalized BGP events in `collected_data`
|
||||
2. use `data_snapshots` to group each ingest window
|
||||
3. add a dedicated anomaly table for higher-value derived events
|
||||
|
||||
## Proposed Data Types
|
||||
|
||||
### `collected_data`
|
||||
|
||||
Use these `source` values:
|
||||
|
||||
- `ris_live_bgp`
|
||||
- `bgpstream_bgp`
|
||||
|
||||
Use these `data_type` values:
|
||||
|
||||
- `bgp_update`
|
||||
- `bgp_rib`
|
||||
- `bgp_visibility`
|
||||
- `bgp_path_change`
|
||||
|
||||
Recommended stable fields:
|
||||
|
||||
- `source`
|
||||
- `source_id`
|
||||
- `entity_key`
|
||||
- `data_type`
|
||||
- `name`
|
||||
- `reference_date`
|
||||
- `metadata`
|
||||
|
||||
Recommended `entity_key` strategy:
|
||||
|
||||
- event entity: `collector|peer|prefix|event_time`
|
||||
- prefix state entity: `collector|peer|prefix`
|
||||
- origin state entity: `prefix|origin_asn`
|
||||
|
||||
### `metadata` schema for raw events
|
||||
|
||||
Store the normalized event payload in `metadata`:
|
||||
|
||||
```json
|
||||
{
|
||||
"project": "ris-live",
|
||||
"collector": "rrc00",
|
||||
"peer_asn": 3333,
|
||||
"peer_ip": "2001:db8::1",
|
||||
"event_type": "announcement",
|
||||
"prefix": "203.0.113.0/24",
|
||||
"origin_asn": 64496,
|
||||
"as_path": [3333, 64500, 64496],
|
||||
"communities": ["3333:100", "64500:1"],
|
||||
"next_hop": "192.0.2.1",
|
||||
"med": 0,
|
||||
"local_pref": null,
|
||||
"timestamp": "2026-03-26T08:00:00Z",
|
||||
"raw_message": {}
|
||||
}
|
||||
```
|
||||
|
||||
### New anomaly table
|
||||
|
||||
Add a new table, recommended name: `bgp_anomalies`
|
||||
|
||||
Suggested columns:
|
||||
|
||||
- `id`
|
||||
- `snapshot_id`
|
||||
- `task_id`
|
||||
- `source`
|
||||
- `anomaly_type`
|
||||
- `severity`
|
||||
- `status`
|
||||
- `entity_key`
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `new_origin_asn`
|
||||
- `peer_scope`
|
||||
- `started_at`
|
||||
- `ended_at`
|
||||
- `confidence`
|
||||
- `summary`
|
||||
- `evidence`
|
||||
- `created_at`
|
||||
|
||||
This table should represent derived intelligence, not raw updates.
|
||||
|
||||
## Collector Design
|
||||
|
||||
## 1. `RISLiveCollector`
|
||||
|
||||
Responsibility:
|
||||
|
||||
- maintain WebSocket connection
|
||||
- subscribe to relevant message types
|
||||
- normalize messages
|
||||
- write event batches into snapshots
|
||||
- optionally emit derived anomalies in near real time
|
||||
|
||||
Suggested runtime mode:
|
||||
|
||||
- long-running background task
|
||||
|
||||
Suggested snapshot strategy:
|
||||
|
||||
- one snapshot per rolling time window
|
||||
- for example every 1 minute or every 5 minutes
|
||||
|
||||
## 2. `BGPStreamBackfillCollector`
|
||||
|
||||
Responsibility:
|
||||
|
||||
- fetch historical data windows
|
||||
- normalize to the same schema as real-time data
|
||||
- build baselines
|
||||
- re-run anomaly rules on past windows if needed
|
||||
|
||||
Suggested runtime mode:
|
||||
|
||||
- scheduled task
|
||||
- or ad hoc task for investigations
|
||||
|
||||
Suggested snapshot strategy:
|
||||
|
||||
- one snapshot per historical query window
|
||||
|
||||
## Normalization Rules
|
||||
|
||||
Normalize both sources into the same internal event model.
|
||||
|
||||
Required normalized fields:
|
||||
|
||||
- `collector`
|
||||
- `peer_asn`
|
||||
- `peer_ip`
|
||||
- `event_type`
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `as_path`
|
||||
- `timestamp`
|
||||
|
||||
Derived normalized fields:
|
||||
|
||||
- `as_path_length`
|
||||
- `country_guess`
|
||||
- `prefix_length`
|
||||
- `is_more_specific`
|
||||
- `visibility_weight`
|
||||
|
||||
## Anomaly Detection Rules
|
||||
|
||||
Start with these five rules first.
|
||||
|
||||
### 1. Origin ASN Change
|
||||
|
||||
Trigger when:
|
||||
|
||||
- the same prefix is announced by a new origin ASN not seen in the baseline window
|
||||
|
||||
Use for:
|
||||
|
||||
- hijack suspicion
|
||||
- origin drift detection
|
||||
|
||||
### 2. More-Specific Burst
|
||||
|
||||
Trigger when:
|
||||
|
||||
- a more-specific prefix appears suddenly
|
||||
- especially from an unexpected origin ASN
|
||||
|
||||
Use for:
|
||||
|
||||
- subprefix hijack suspicion
|
||||
|
||||
### 3. Mass Withdrawal
|
||||
|
||||
Trigger when:
|
||||
|
||||
- the same prefix or ASN sees many withdrawals across collectors within a short window
|
||||
|
||||
Use for:
|
||||
|
||||
- outage suspicion
|
||||
- regional incident detection
|
||||
|
||||
### 4. Path Deviation
|
||||
|
||||
Trigger when:
|
||||
|
||||
- AS path length jumps sharply
|
||||
- or a rarely seen transit ASN appears
|
||||
- or path frequency drops below baseline norms
|
||||
|
||||
Use for:
|
||||
|
||||
- route leak suspicion
|
||||
- unusual path diversion
|
||||
|
||||
### 5. Visibility Drop
|
||||
|
||||
Trigger when:
|
||||
|
||||
- a prefix is visible from far fewer collectors/peers than its baseline
|
||||
|
||||
Use for:
|
||||
|
||||
- regional reachability degradation
|
||||
|
||||
## Baseline Strategy
|
||||
|
||||
Use BGPStream historical data to build:
|
||||
|
||||
- common origin ASN per prefix
|
||||
- common AS path patterns
|
||||
- collector visibility distribution
|
||||
- normal withdrawal frequency
|
||||
|
||||
Recommended baseline windows:
|
||||
|
||||
- short baseline: last 24 hours
|
||||
- medium baseline: last 7 days
|
||||
- long baseline: last 30 days
|
||||
|
||||
The first implementation can start with only the 7-day baseline.
|
||||
|
||||
## API Design
|
||||
|
||||
### Raw event API
|
||||
|
||||
Add endpoints like:
|
||||
|
||||
- `GET /api/v1/bgp/events`
|
||||
- `GET /api/v1/bgp/events/{id}`
|
||||
|
||||
Suggested filters:
|
||||
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `peer_asn`
|
||||
- `collector`
|
||||
- `event_type`
|
||||
- `time_from`
|
||||
- `time_to`
|
||||
- `source`
|
||||
|
||||
### Anomaly API
|
||||
|
||||
Add endpoints like:
|
||||
|
||||
- `GET /api/v1/bgp/anomalies`
|
||||
- `GET /api/v1/bgp/anomalies/{id}`
|
||||
- `GET /api/v1/bgp/anomalies/summary`
|
||||
|
||||
Suggested filters:
|
||||
|
||||
- `severity`
|
||||
- `anomaly_type`
|
||||
- `status`
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `time_from`
|
||||
- `time_to`
|
||||
|
||||
### Visualization API
|
||||
|
||||
Add an Earth-oriented endpoint like:
|
||||
|
||||
- `GET /api/v1/visualization/geo/bgp-anomalies`
|
||||
|
||||
Recommended feature shapes:
|
||||
|
||||
- point: collector locations
|
||||
- arc: inferred propagation or suspicious path edge
|
||||
- pulse point: active anomaly hotspot
|
||||
|
||||
## Earth Big-Screen Design
|
||||
|
||||
Recommended layers:
|
||||
|
||||
### Layer 1: Collector layer
|
||||
|
||||
Show known collector locations and current activity intensity.
|
||||
|
||||
### Layer 2: Route propagation arcs
|
||||
|
||||
Use arcs for:
|
||||
|
||||
- origin ASN country to collector country
|
||||
- or collector-to-collector visibility edges
|
||||
|
||||
Important note:
|
||||
|
||||
This is an inferred propagation view, not real packet flow.
|
||||
|
||||
### Layer 3: Active anomaly overlay
|
||||
|
||||
Show:
|
||||
|
||||
- hijack suspicion in red
|
||||
- mass withdrawal in orange
|
||||
- visibility drop in yellow
|
||||
- path deviation in blue
|
||||
|
||||
### Layer 4: Time playback
|
||||
|
||||
Use `data_snapshots` to replay:
|
||||
|
||||
- minute-by-minute route changes
|
||||
- anomaly expansion
|
||||
- recovery timeline
|
||||
|
||||
## Alerting Strategy
|
||||
|
||||
Map anomaly severity to the current alert system.
|
||||
|
||||
Recommended severity mapping:
|
||||
|
||||
- `critical`
|
||||
- likely hijack
|
||||
- very large withdrawal burst
|
||||
- `high`
|
||||
- clear origin change
|
||||
- large visibility drop
|
||||
- `medium`
|
||||
- unusual path change
|
||||
- moderate more-specific burst
|
||||
- `low`
|
||||
- weak or localized anomalies
|
||||
|
||||
## Delivery Plan
|
||||
|
||||
### Phase 1
|
||||
|
||||
- add `RISLiveCollector`
|
||||
- normalize updates into `collected_data`
|
||||
- create `bgp_anomalies`
|
||||
- implement 3 rules:
|
||||
- origin change
|
||||
- more-specific burst
|
||||
- mass withdrawal
|
||||
|
||||
### Phase 2
|
||||
|
||||
- add `BGPStreamBackfillCollector`
|
||||
- build 7-day baseline
|
||||
- implement:
|
||||
- path deviation
|
||||
- visibility drop
|
||||
|
||||
### Phase 3
|
||||
|
||||
- add Earth visualization layer
|
||||
- add time playback
|
||||
- add anomaly filtering and drilldown
|
||||
|
||||
## Practical Implementation Notes
|
||||
|
||||
- Start with IPv4 first, then add IPv6 after the event schema is stable.
|
||||
- Store the original raw payload in `metadata.raw_message` for traceability.
|
||||
- Deduplicate events by a stable hash of collector, peer, prefix, type, and timestamp.
|
||||
- Keep anomaly generation idempotent so replay and backfill do not create duplicate alerts.
|
||||
- Expect noisy data and partial views; confidence scoring matters.
|
||||
|
||||
## Recommended First Patch Set
|
||||
|
||||
The first code milestone should include:
|
||||
|
||||
1. `backend/app/services/collectors/ris_live.py`
|
||||
2. `backend/app/services/collectors/bgpstream.py`
|
||||
3. `backend/app/models/bgp_anomaly.py`
|
||||
4. `backend/app/api/v1/bgp.py`
|
||||
5. `backend/app/api/v1/visualization.py`
|
||||
add BGP anomaly geo endpoint
|
||||
6. `frontend/src/pages`
|
||||
add a BGP anomaly list or summary page
|
||||
7. `frontend/public/earth/js`
|
||||
add BGP anomaly rendering layer
|
||||
|
||||
## Sources
|
||||
|
||||
- [RIPE RIS Live](https://ris-live.ripe.net/)
|
||||
- [CAIDA BGPStream Data Access Overview](https://bgpstream.caida.org/docs/overview/data-access)
|
||||
422
docs/plans/earth-bgp-region-aggregation-plan.md
Normal file
422
docs/plans/earth-bgp-region-aggregation-plan.md
Normal file
@@ -0,0 +1,422 @@
|
||||
# BGP Region Aggregation Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This document refines the current BGP `activity layer` into an implementation-ready regional aggregation design.
|
||||
|
||||
Primary product goal:
|
||||
|
||||
- turn sparse prefix-level observations, anomalies, and incidents into a readable `regional observability layer`
|
||||
- keep Earth visually alive during low-incident periods
|
||||
- make `incident markers` remain the highest-confidence foreground layer instead of replacing them
|
||||
|
||||
This layer is not a new collector, detector, or raw storage table.
|
||||
It is an aggregation/view-model layer:
|
||||
|
||||
`observations -> enrichment -> anomalies/incidents -> geography mapping -> region aggregation -> Earth/UI activity layer`
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
- collector presence alone proves coverage, but does not communicate `where routing is currently active or noisy`
|
||||
|
||||
So the missing middle layer is:
|
||||
|
||||
- `collectors` show that observation exists
|
||||
- `regions` show where activity is building up
|
||||
- `incidents` show the specific high-confidence focus events
|
||||
|
||||
## Scope
|
||||
|
||||
This plan is specifically for:
|
||||
|
||||
- a backend aggregation service
|
||||
- a summary API for console/stats
|
||||
- a GeoJSON API for Earth rendering
|
||||
- an Earth background activity layer that supports, but does not replace, incident markers
|
||||
|
||||
This plan does not attempt to solve:
|
||||
|
||||
- exact prefix geolocation quality
|
||||
- polygon-heavy geopolitical visualization
|
||||
- persistent materialized region tables in v1
|
||||
|
||||
## Region Layer Definition
|
||||
|
||||
Recommended conceptual model:
|
||||
|
||||
- `region layer` = background situational awareness
|
||||
- `incident layer` = focal event markers
|
||||
|
||||
That means:
|
||||
|
||||
- region activity should answer `where is routing behavior currently active or abnormal`
|
||||
- incident markers should answer `which concrete event should the user click`
|
||||
|
||||
## Recommended Output Model
|
||||
|
||||
Suggested backend output object:
|
||||
|
||||
## `BGPRegionActivity`
|
||||
|
||||
```json
|
||||
{
|
||||
"region_key": "sea",
|
||||
"region_name": "Southeast Asia",
|
||||
"center_lat": 1.3521,
|
||||
"center_lon": 103.8198,
|
||||
"observation_count": 128,
|
||||
"anomaly_count": 9,
|
||||
"incident_count": 2,
|
||||
"activity_score": 17.6,
|
||||
"status": "incident",
|
||||
"affected_prefix_count": 14,
|
||||
"affected_asn_count": 6,
|
||||
"collector_count": 5,
|
||||
"first_seen_at": "2026-04-02T10:00:00Z",
|
||||
"last_seen_at": "2026-04-02T10:12:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Fields To Keep In MVP
|
||||
|
||||
- `region_key`
|
||||
- `region_name`
|
||||
- `center_lat`
|
||||
- `center_lon`
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
- `activity_score`
|
||||
- `status`
|
||||
- `affected_prefix_count`
|
||||
- `affected_asn_count`
|
||||
- `collector_count`
|
||||
- `first_seen_at`
|
||||
- `last_seen_at`
|
||||
|
||||
### Fields To Delay
|
||||
|
||||
These are useful, but not required for the first implementation:
|
||||
|
||||
- `bounding_box`
|
||||
- `top_incident_types`
|
||||
- `top_prefixes`
|
||||
- polygon geometry
|
||||
|
||||
## Region Definition Strategy
|
||||
|
||||
### Recommendation
|
||||
|
||||
Use a static region-definition table first.
|
||||
|
||||
Examples:
|
||||
|
||||
- `north_america`
|
||||
- `south_america`
|
||||
- `western_europe`
|
||||
- `eastern_europe`
|
||||
- `east_asia`
|
||||
- `southeast_asia`
|
||||
- `south_asia`
|
||||
- `middle_east`
|
||||
- `north_africa`
|
||||
- `sub_saharan_africa`
|
||||
- `oceania`
|
||||
|
||||
Why this is the right v1 choice:
|
||||
|
||||
- stable UI semantics
|
||||
- strong readability on Earth
|
||||
- easier debugging and explanation
|
||||
- lower implementation cost than geohash or H3 grids
|
||||
|
||||
### Not Recommended For V1
|
||||
|
||||
- geohash cell aggregation
|
||||
- H3 aggregation
|
||||
- fine-grained lat/lon bucket maps
|
||||
|
||||
Those are more flexible, but they make the map feel fragmented and less explainable.
|
||||
|
||||
## Geography Mapping Strategy
|
||||
|
||||
Do not reduce the implementation to only `prefix -> exact geo`.
|
||||
|
||||
The region layer should follow the same geography-priority logic already implied by the current BGP direction:
|
||||
|
||||
1. `prefix_geography`
|
||||
2. `prefix_scope`
|
||||
3. `ASN organization region`
|
||||
4. `collector centroid` fallback
|
||||
|
||||
This matters because exact prefix geography will often be incomplete or approximate.
|
||||
The region layer should stay robust even when only partial enrichment is available.
|
||||
|
||||
## Backend Design
|
||||
|
||||
Recommended new service file:
|
||||
|
||||
- [backend/app/services/bgp_regions.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_regions.py)
|
||||
|
||||
Suggested responsibilities:
|
||||
|
||||
- `map_record_to_region(...)`
|
||||
- `aggregate_region_activity(...)`
|
||||
- `build_region_geojson(...)`
|
||||
- `resolve_activity_status(...)`
|
||||
- `compute_activity_score(...)`
|
||||
|
||||
### Data Source Inputs
|
||||
|
||||
Use a recent rolling window, default `15 minutes`, and aggregate from:
|
||||
|
||||
- `BGPObservation`
|
||||
- `BGPAnomaly`
|
||||
- active `BGPIncident`
|
||||
|
||||
### Aggregation Flow
|
||||
|
||||
1. query observations in the time window
|
||||
2. query anomalies in the same window
|
||||
3. query active incidents in the same window or active status set
|
||||
4. resolve each record to a best-effort region
|
||||
5. accumulate per-region counters
|
||||
6. compute score and status
|
||||
7. return region activity list
|
||||
|
||||
## Status Model
|
||||
|
||||
Recommended status buckets:
|
||||
|
||||
- `idle`
|
||||
- `observing`
|
||||
- `anomaly`
|
||||
- `incident`
|
||||
|
||||
Suggested rule:
|
||||
|
||||
```text
|
||||
if incident_count > 0: incident
|
||||
elif anomaly_count > 0: anomaly
|
||||
elif observation_count > 0: observing
|
||||
else: idle
|
||||
```
|
||||
|
||||
This aligns well with the current Earth status language and keeps the visual mapping simple.
|
||||
|
||||
## Activity Score
|
||||
|
||||
The score should be a tunable heuristic, not a fixed truth model.
|
||||
|
||||
Recommended v1 formula:
|
||||
|
||||
```text
|
||||
activity_score =
|
||||
min(observation_count, 50) * 0.03
|
||||
+ anomaly_count * 1.2
|
||||
+ incident_count * 5.0
|
||||
```
|
||||
|
||||
Why cap observations:
|
||||
|
||||
- observation volume is usually much larger than anomaly or incident volume
|
||||
- uncapped observation counts would overwhelm the score
|
||||
- capped observation counts preserve baseline presence without drowning real abnormality
|
||||
|
||||
### Practical Guidance
|
||||
|
||||
- treat coefficients as configuration-like constants
|
||||
- expect to retune after looking at real data
|
||||
- keep `incident` weight dominant
|
||||
|
||||
## API Design
|
||||
|
||||
### 1. Summary/List API
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `/api/v1/bgp/regions/activity`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"window_minutes": 15,
|
||||
"regions": []
|
||||
}
|
||||
```
|
||||
|
||||
Use cases:
|
||||
|
||||
- BGP console summaries
|
||||
- right-side Earth stats
|
||||
- future region list panels
|
||||
|
||||
### 2. GeoJSON API
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `/api/v1/visualization/geo/bgp-regions`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": []
|
||||
}
|
||||
```
|
||||
|
||||
Each feature should include:
|
||||
|
||||
- `geometry`
|
||||
- v1: `Point`
|
||||
- later: optional `Polygon`
|
||||
- `properties`
|
||||
- `region_key`
|
||||
- `region_name`
|
||||
- `status`
|
||||
- `activity_score`
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
- `affected_prefix_count`
|
||||
- `affected_asn_count`
|
||||
- `collector_count`
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
- `region layer` = ambient background activity
|
||||
- `incident marker` = focal event object
|
||||
|
||||
Do not replace incident markers with region markers.
|
||||
|
||||
### Region Visual Rules
|
||||
|
||||
Suggested mapping:
|
||||
|
||||
- `observing`
|
||||
- weak glow
|
||||
- low pulse or no pulse
|
||||
- `anomaly`
|
||||
- stronger glow
|
||||
- more visible pulse
|
||||
- `incident`
|
||||
- strongest regional emphasis
|
||||
- but still visually secondary to the incident marker itself
|
||||
|
||||
### Region Labels
|
||||
|
||||
Good v2 enhancement:
|
||||
|
||||
- show region name
|
||||
- show counts like `2 incidents / 5 anomalies`
|
||||
|
||||
This is useful, but should come after the core aggregation and Earth glow layer are working.
|
||||
|
||||
## Interaction Model
|
||||
|
||||
### Click Region
|
||||
|
||||
Recommended detail payload:
|
||||
|
||||
- region name
|
||||
- observation/anomaly/incident counts in the selected window
|
||||
- affected prefix count
|
||||
- affected ASN count
|
||||
- collector count
|
||||
- recent incidents in the region
|
||||
|
||||
### Click Incident
|
||||
|
||||
Keep the current incident-detail flow.
|
||||
|
||||
Interaction should feel hierarchical:
|
||||
|
||||
1. region gives situational context
|
||||
2. incident gives event focus
|
||||
|
||||
## MVP Implementation Order
|
||||
|
||||
### Step 1
|
||||
|
||||
Define static `REGIONS` in code or config.
|
||||
|
||||
### Step 2
|
||||
|
||||
Map geography-enriched BGP records into regions using the fallback chain.
|
||||
|
||||
### Step 3
|
||||
|
||||
Aggregate recent window counts:
|
||||
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
|
||||
### Step 4
|
||||
|
||||
Compute `activity_score` and `status`.
|
||||
|
||||
### Step 5
|
||||
|
||||
Expose:
|
||||
|
||||
- `/api/v1/bgp/regions/activity`
|
||||
- `/api/v1/visualization/geo/bgp-regions`
|
||||
|
||||
### Step 6
|
||||
|
||||
Render region glows on Earth behind incident markers.
|
||||
|
||||
## Out Of Scope For MVP
|
||||
|
||||
- persistent materialized region tables
|
||||
- geohash or H3 support
|
||||
- polygon-filled regional overlays
|
||||
- detailed top-prefix ranking in the first release
|
||||
- complicated scoring personalization
|
||||
|
||||
## Risks And Constraints
|
||||
|
||||
### Geography Quality
|
||||
|
||||
Prefix geography is approximate and incomplete.
|
||||
The region layer must tolerate fallback-based placement.
|
||||
|
||||
### Query Cost
|
||||
|
||||
Dynamic aggregation is the right v1 choice, but repeated short-window queries may eventually need:
|
||||
|
||||
- in-process caching
|
||||
- scheduled pre-aggregation
|
||||
- materialized summaries
|
||||
|
||||
### UI Overcrowding
|
||||
|
||||
If region glow, collector activity, and incidents all become too strong at once, Earth readability will regress.
|
||||
The region layer must remain supportive, not dominant.
|
||||
|
||||
## Final Recommendation
|
||||
|
||||
The current BGP roadmap should explicitly add:
|
||||
|
||||
- `region aggregation` as the concrete implementation of the missing `activity layer`
|
||||
|
||||
The recommended product interpretation is:
|
||||
|
||||
- `collectors` prove observation coverage
|
||||
- `regions` communicate live routing activity and abnormality
|
||||
- `incidents` remain the clearest high-confidence event objects
|
||||
|
||||
In one sentence:
|
||||
|
||||
`region aggregation is not a replacement for incidents; it is the situational background that makes sparse incidents feel legible on Earth.`
|
||||
715
docs/plans/earth-celestial-background-plan.md
Normal file
715
docs/plans/earth-celestial-background-plan.md
Normal file
@@ -0,0 +1,715 @@
|
||||
# Earth 天球背景与日月位置实施方案
|
||||
|
||||
## 目标
|
||||
|
||||
为 Earth 大屏增加一套真正可用的天文背景层，覆盖三件事：
|
||||
|
||||
1. 用真实天球背景替换当前随机星点
|
||||
2. 在当前时间下显示太阳与月亮的相对位置
|
||||
3. 让太阳方向同时驱动地球受光，形成更可信的昼夜关系
|
||||
|
||||
本方案优先追求：
|
||||
|
||||
- 与当前 Three.js Earth 架构兼容
|
||||
- 风险可控
|
||||
- 先落地一版真实感明显提升的 V1
|
||||
- 为后续更严格的天文参考系升级预留余地
|
||||
|
||||
## 当前现状
|
||||
|
||||
当前 Earth 的基础条件已经具备：
|
||||
|
||||
- 地球、云层、地形、网格都基于 Three.js，主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点，不是真实星图
|
||||
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`，位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 主循环 `animate()` 已稳定运行，可在其中接入天体更新逻辑
|
||||
|
||||
这意味着：
|
||||
|
||||
- 不需要重写 Earth
|
||||
- 可以在现有 scene/world 层新增一个 celestial layer
|
||||
- 第一阶段不必拆 Earth / satellite / cable 的参考系
|
||||
|
||||
## 总体策略
|
||||
|
||||
采用“两层现实”设计：
|
||||
|
||||
### 1. 世界层（world-space celestial layer）
|
||||
|
||||
用于放置：
|
||||
|
||||
- 天球背景
|
||||
- 太阳
|
||||
- 月亮
|
||||
- 太阳光方向
|
||||
|
||||
这些对象不挂在 `earthObj` 上，而是直接放在 `scene` 中。
|
||||
|
||||
### 2. 地球层（earth-fixed layer）
|
||||
|
||||
继续保持当前结构：
|
||||
|
||||
- 海缆
|
||||
- 登陆点
|
||||
- 卫星点与轨迹
|
||||
- BGP 覆盖
|
||||
- 地球纹理、云层、地形
|
||||
|
||||
这些对象继续挂在 `earthObj` 下，不打断现有交互。
|
||||
|
||||
## 为什么先这样做
|
||||
|
||||
当前用户交互是“拖动地球本体”，而不是“移动相机绕惯性系观测”。
|
||||
如果现在直接做严格惯性参考系改造，会同时影响：
|
||||
|
||||
- `earthObj.rotation`
|
||||
- 卫星轨迹与锁定逻辑
|
||||
- 海缆与登陆点附着关系
|
||||
- resetView / autoRotate / hover / click 等交互链路
|
||||
|
||||
所以第一阶段只做：
|
||||
|
||||
- 真正的天空
|
||||
- 真正的日月方向
|
||||
- 不碰现有 Earth 附着对象的语义
|
||||
|
||||
## 推荐技术选型
|
||||
|
||||
### 天文计算库
|
||||
|
||||
推荐：
|
||||
|
||||
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
|
||||
原因：
|
||||
|
||||
- 有 JavaScript 版本
|
||||
- 支持 Sun / Moon 的矢量与坐标变换
|
||||
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
|
||||
- 后续若要加行星、月相、黄道、赤道网，也能继续沿用
|
||||
|
||||
不作为主选的库：
|
||||
|
||||
- [SunCalc](https://github.com/mourner/suncalc)
|
||||
|
||||
原因：
|
||||
|
||||
- 更偏本地观察者视角的太阳/月亮高度角
|
||||
- 用于“地面日出日落”很好
|
||||
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
|
||||
|
||||
### Three.js 表现层
|
||||
|
||||
推荐组合：
|
||||
|
||||
- 天球：内翻球壳 + 星图纹理
|
||||
- 太阳：`THREE.Sprite`
|
||||
- 月亮：`THREE.Sprite` 或小型 `THREE.Mesh`
|
||||
- 太阳光：`THREE.DirectionalLight`
|
||||
|
||||
参考：
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈，本方案明确分成两类资源：
|
||||
|
||||
### 1. 背景资源：全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源：
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点，而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换，也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求：
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影（equirectangular）
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮，避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图，而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录：
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据：星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层：
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先：Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先：Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略：
|
||||
|
||||
- V1：背景球壳只用全天星图，不立即生成全量恒星点
|
||||
- V2：只挑选亮星（例如星等 `< 5.5`）生成恒星点层
|
||||
- V3：如果确实需要更丰富的星场，再逐步扩展到更深星等
|
||||
|
||||
这样做的原因：
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板：
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是：
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分：
|
||||
|
||||
- 背景层：官方深空图 / 全天星图
|
||||
- 计算层：太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”，而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用：
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因：
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图，那么：
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现：
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤：
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式：
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时，需要做一次项目内坐标轴校准：
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的：
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在：
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合：
|
||||
|
||||
- 背景层：全天星图球壳
|
||||
- 点位层：亮星点
|
||||
- 动态层：太阳 / 月亮
|
||||
|
||||
这样有三个好处：
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源：NASA Tycho all-sky map
|
||||
- 用途：背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途：Phase 4 月相表现
|
||||
- 路径建议：
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途：太阳 sprite halo
|
||||
- 路径建议：
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层，建议新增一个预处理后的轻量数据文件：
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段：
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表，而是先离线裁剪成：
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A：先做可用版本（推荐）
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点：
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B：在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加：
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点：
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层（后续）
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载（后续）
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层，建议单独做离线脚本：
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责：
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果，不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1：真实天球背景
|
||||
|
||||
### 目标
|
||||
|
||||
用真实全天星图替换当前随机星点背景。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 新增一张全天星图纹理
|
||||
|
||||
建议路径：
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
纹理要求：
|
||||
|
||||
- 等距矩形投影
|
||||
- 赤经/赤纬坐标展开
|
||||
- 无地平线、无地景遮挡
|
||||
- 尽量深色、弱干扰，适合大屏 HUD 叠加
|
||||
|
||||
2. 新增天球球壳
|
||||
|
||||
新增模块：
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
建议接口：
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date, camera, earth)
|
||||
export function disposeCelestialLayer()
|
||||
```
|
||||
|
||||
3. 实现一个大半径内翻球体
|
||||
|
||||
建议参数：
|
||||
|
||||
- 半径：`600 ~ 900`
|
||||
- 材质：`MeshBasicMaterial`
|
||||
- `side: THREE.BackSide`
|
||||
- 不受场景光照影响
|
||||
- 始终围绕场景中心
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 初始加载后背景不再是随机星点
|
||||
- 旋转地球时，背景保持为稳定天球而不是跟地球一起转
|
||||
- 不明显干扰海缆/卫星/BGP 的前景识别
|
||||
|
||||
## Phase 2：太阳与月亮真实位置
|
||||
|
||||
### 目标
|
||||
|
||||
在当前 UTC 时间下，计算太阳与月亮在天球中的方向，并显示出来。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 在 `celestial.js` 内封装天体位置计算
|
||||
|
||||
建议函数：
|
||||
|
||||
```js
|
||||
function getSunDirection(date)
|
||||
function getMoonDirection(date)
|
||||
```
|
||||
|
||||
输出统一为 world-space `THREE.Vector3`
|
||||
|
||||
2. 太阳显示
|
||||
|
||||
- 一个暖色发光 sprite
|
||||
- 比月亮更大、更亮
|
||||
- 可选添加柔和 halo
|
||||
|
||||
3. 月亮显示
|
||||
|
||||
- 一个较小 sprite 或 sphere
|
||||
- 灰白偏冷色
|
||||
- 后续 Phase 3 再做月相
|
||||
|
||||
4. 更新频率
|
||||
|
||||
不要每帧重新做完整天文计算，建议：
|
||||
|
||||
- 每 30 秒或 60 秒重算一次真实位置
|
||||
- 渲染帧内做平滑过渡
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面可见太阳与月亮两个对象
|
||||
- 时间变化时位置会更新
|
||||
- 日月不会跟随地球局部旋转而错误附着
|
||||
|
||||
## Phase 3：太阳驱动地球受光
|
||||
|
||||
### 目标
|
||||
|
||||
让地球光照方向与太阳方向一致，不再使用写死的固定主光。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 替换或接管当前主定向光
|
||||
|
||||
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
|
||||
|
||||
建议改为：
|
||||
|
||||
- 保留环境补光
|
||||
- 主太阳光方向由 `sunDirection` 决定
|
||||
|
||||
2. 太阳光参数建议
|
||||
|
||||
- `DirectionalLight` 颜色偏暖白
|
||||
- 强度略高于当前主光
|
||||
- 保留一个弱背光作为氛围补偿，避免背面过死黑
|
||||
|
||||
3. 先不做物理级大气散射
|
||||
|
||||
第一版只要求：
|
||||
|
||||
- 亮面与暗面方向真实
|
||||
- 云层和大气仍保持当前风格
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 地球明暗面会随太阳方向改变
|
||||
- 太阳 sprite 和地球亮面方向一致
|
||||
- 不破坏现有海缆、卫星、BGP 的可见性
|
||||
|
||||
## Phase 4：月相与天文细节增强
|
||||
|
||||
### 目标
|
||||
|
||||
在日月真实位置基础上增加更强的“天文可信度”。
|
||||
|
||||
### 可选项
|
||||
|
||||
1. 月相
|
||||
|
||||
- 根据日月夹角计算 illuminated fraction
|
||||
- 用月相纹理或 shader 表达盈亏
|
||||
|
||||
2. 赤道/黄道辅助线
|
||||
|
||||
- 可作为开发调试层，不默认显示
|
||||
|
||||
3. 太阳 terminator 增强
|
||||
|
||||
- 给地球夜面加入更自然的 night tint
|
||||
- 未来可叠加城市夜光纹理
|
||||
|
||||
4. 天文时间入口
|
||||
|
||||
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 月亮不再只是一个静态圆点
|
||||
- 后续扩展行星或观测模式时无需推倒重来
|
||||
|
||||
## Phase 5：严格参考系升级（可选，不作为 V1 必做）
|
||||
|
||||
### 目标
|
||||
|
||||
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
|
||||
|
||||
### 需要处理的问题
|
||||
|
||||
- 地球自转角与 UTC 的一致性
|
||||
- 赤道坐标系、地固坐标系、相机交互层分离
|
||||
- 卫星轨道显示与 Earth 旋转同步关系
|
||||
- resetView 和 autoRotate 的语义重定
|
||||
|
||||
### 风险
|
||||
|
||||
这一步会影响：
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
因此不建议与 V1 同时推进。
|
||||
|
||||
## 代码改造清单
|
||||
|
||||
## 1. 新增文件
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
职责：
|
||||
|
||||
- 管理天球背景、太阳、月亮
|
||||
- 对外暴露 init/update/dispose
|
||||
|
||||
## 2. 修改 `constants.js`
|
||||
|
||||
文件：
|
||||
|
||||
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
|
||||
新增：
|
||||
|
||||
```js
|
||||
export const CELESTIAL_CONFIG = {
|
||||
sphereRadius: 800,
|
||||
updateIntervalMs: 60000,
|
||||
sunSpriteScale: 28,
|
||||
moonSpriteScale: 16,
|
||||
sunLightIntensity: 1.25,
|
||||
ambientIntensity: 0.28,
|
||||
backLightIntensity: 0.18,
|
||||
};
|
||||
```
|
||||
|
||||
## 3. 修改 `main.js`
|
||||
|
||||
文件：
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
主要改动：
|
||||
|
||||
1. `init()` 中：
|
||||
- 初始化 celestial layer
|
||||
2. `addLights()` 中：
|
||||
- 把固定太阳光改成可更新的 celestial sun light
|
||||
3. `animate()` 中：
|
||||
- 每帧调 `updateCelestialLayer()`
|
||||
4. `destroy()` 中：
|
||||
- 清理 celestial 资源
|
||||
|
||||
## 4. 修改 `earth.js`
|
||||
|
||||
文件：
|
||||
|
||||
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
主要改动：
|
||||
|
||||
- `createStars()` 逐步退役
|
||||
- 第一阶段可先保留作为 fallback
|
||||
- 当真实星图加载成功后，不再显示随机星点
|
||||
|
||||
## 5. 新增资源
|
||||
|
||||
目录建议：
|
||||
|
||||
- `frontend/public/earth/assets/celestial/`
|
||||
|
||||
建议至少包含：
|
||||
|
||||
- `starmap_equatorial_4k.jpg`
|
||||
- `sun_glow.png`
|
||||
- `moon_albedo_2k.jpg`
|
||||
|
||||
## 数据流设计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
|
||||
B --> C["创建天球球壳"]
|
||||
B --> D["创建太阳 sprite + 主定向光"]
|
||||
B --> E["创建月亮 sprite"]
|
||||
|
||||
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
|
||||
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
|
||||
H --> I["更新 sun sprite / moon sprite 位置"]
|
||||
H --> J["更新太阳 DirectionalLight 方向"]
|
||||
J --> K["地球昼夜方向变化"]
|
||||
```
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
### 1. 星图投影方向容易反
|
||||
|
||||
这会表现为：
|
||||
|
||||
- 星图左右镜像
|
||||
- 赤经方向颠倒
|
||||
- 日月位置和背景对不上
|
||||
|
||||
建议：
|
||||
|
||||
- 先做一个开发调试模式
|
||||
- 显示赤经/赤纬参考点，快速校正纹理朝向
|
||||
|
||||
### 2. 不要让天球跟随 Earth 旋转
|
||||
|
||||
天球背景和日月必须属于 scene/world，而不是 `earthObj`。
|
||||
|
||||
### 3. 不要每帧做重型天文计算
|
||||
|
||||
真实位置更新应节流，否则会浪费 CPU。
|
||||
|
||||
### 4. 月亮先求“方向正确”，再求“月相精致”
|
||||
|
||||
月相属于第二步优化，不应阻塞 V1 上线。
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 新建 `celestial.js`
|
||||
2. 用星图球壳替换随机星点
|
||||
3. 接入 Astronomy Engine
|
||||
4. 加太阳/月亮 sprite
|
||||
5. 用太阳方向驱动主光
|
||||
6. 再决定要不要做月相和更严格参考系
|
||||
|
||||
## 最终建议
|
||||
|
||||
对于当前 Planet Earth，最稳妥的方案是：
|
||||
|
||||
- 先做真实天球背景
|
||||
- 再做真实太阳/月亮方向
|
||||
- 再让太阳驱动地球受光
|
||||
- 暂时不做 Earth 参考系重构
|
||||
|
||||
这样可以在不破坏现有 Earth 交互和图层系统的前提下，显著提升空间感、真实感和演示说服力。
|
||||
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时，显示“预测轨道”而不是只有历史尾迹：
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有：
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升：
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹，预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象：
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆：
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足，需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保：
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高，采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
216
docs/plans/earth-prefix-geography-plan.md
Normal file
216
docs/plans/earth-prefix-geography-plan.md
Normal file
@@ -0,0 +1,216 @@
|
||||
# Prefix Geography Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Make Earth BGP incidents `prefix-centric` instead of `collector-centric`.
|
||||
|
||||
The map should primarily answer:
|
||||
|
||||
- where a prefix-related event is likely centered
|
||||
- which regions the prefix is likely associated with
|
||||
- which collectors observed the event as evidence
|
||||
|
||||
It should not continue to imply that the event is located at the collector itself unless no better geography is available.
|
||||
|
||||
## Why Current Geography Is Not Enough
|
||||
|
||||
Current incident geography can still collapse back to collector-derived regions because:
|
||||
|
||||
1. `prefix_scope` is currently built mostly from observed collector regions and historical observation regions.
|
||||
2. `origin_asn_profile` currently comes from `peeringdb_network`, which is useful for ASN footprint hints but not sufficient as a primary prefix location source.
|
||||
3. `collector centroid` is still a common fallback and therefore dominates sparse incidents.
|
||||
|
||||
This makes Earth feel like a collector map with event decorations instead of a prefix impact map.
|
||||
|
||||
## Data Source Layers
|
||||
|
||||
Prefix geography should be built from four layers, ordered by confidence.
|
||||
|
||||
### Layer 1. Prefix-to-country / prefix-to-region
|
||||
|
||||
This is the primary source layer and the current missing piece.
|
||||
|
||||
Recommended sources:
|
||||
|
||||
1. `IPtoASN / IPtoCountry`
|
||||
- URL: <https://iptoasn.com/>
|
||||
- Good fit for this project because it provides downloadable IPv4/IPv6 range-to-ASN and range-to-country mappings.
|
||||
- Best use:
|
||||
- map a prefix to country code
|
||||
- enrich prefixes with coarse regional placement
|
||||
|
||||
2. `OpenGeoFeed`
|
||||
- URL: <https://opengeofeed.org/faq/>
|
||||
- Best use:
|
||||
- override coarse country mappings when the prefix holder publishes a geofeed
|
||||
- provide a more realistic deployment/service region than whois-style registration country
|
||||
|
||||
### Layer 2. Registry allocation fallback
|
||||
|
||||
Use these only as fallback signals, not as a ground-truth physical location.
|
||||
|
||||
Candidate inputs:
|
||||
|
||||
- RIR delegated stats
|
||||
- `inetnum` / `inet6num` whois
|
||||
|
||||
Best use:
|
||||
|
||||
- detect registration country / allocation region
|
||||
- provide fallback when no direct prefix geolocation dataset is available
|
||||
|
||||
### Layer 3. ASN footprint hints
|
||||
|
||||
Existing in this project:
|
||||
|
||||
- `peeringdb_network`
|
||||
- `peeringdb_facility`
|
||||
- `peeringdb_ixp`
|
||||
|
||||
Best use:
|
||||
|
||||
- derive ASN city/country footprint
|
||||
- identify likely exchange/facility regions
|
||||
- act as secondary evidence when prefix-specific geography is unavailable
|
||||
|
||||
### Layer 4. Observation evidence
|
||||
|
||||
Existing in this project:
|
||||
|
||||
- `RIPE RIS Live`
|
||||
- `CAIDA BGPStream Backfill`
|
||||
|
||||
Best use:
|
||||
|
||||
- prove who observed the event
|
||||
- derive affected observation regions
|
||||
- support impact evidence
|
||||
|
||||
This should remain the final fallback and evidence layer, not the primary event geography.
|
||||
|
||||
## Recommended Geography Priority
|
||||
|
||||
The backend should compute incident geography with this order:
|
||||
|
||||
1. `prefix_geography`
|
||||
- prefix-to-country / region / geofeed-backed result
|
||||
2. `asn_region`
|
||||
- ASN organization / facility / IXP footprint
|
||||
3. `collector_centroid`
|
||||
- observed collector regions only as final fallback
|
||||
|
||||
Returned GeoJSON should keep exposing the selected mode through:
|
||||
|
||||
- `geography_mode = prefix_geography | asn_region | collector_centroid`
|
||||
|
||||
## Proposed Backend Changes
|
||||
|
||||
### 1. Add a dedicated prefix geography dataset
|
||||
|
||||
New datasource candidates:
|
||||
|
||||
- `ip2asn_prefix_geo`
|
||||
- optionally `opengeofeed_prefix_geo`
|
||||
|
||||
Suggested storage model:
|
||||
|
||||
- keep downloaded rows in `CollectedData` first for speed of integration
|
||||
- later move to a dedicated table if lookup volume grows
|
||||
|
||||
Minimum normalized fields:
|
||||
|
||||
- `range_start`
|
||||
- `range_end`
|
||||
- `prefix`
|
||||
- `country`
|
||||
- `continent`
|
||||
- `asn`
|
||||
- `as_name`
|
||||
- `source`
|
||||
- `confidence`
|
||||
|
||||
### 2. Add prefix geography enrichment
|
||||
|
||||
Extend:
|
||||
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
|
||||
New enrichment payload should include:
|
||||
|
||||
- `prefix_geography`
|
||||
- `country`
|
||||
- `continent`
|
||||
- `regions`
|
||||
- `source`
|
||||
- `confidence`
|
||||
|
||||
This should be separate from the current `prefix_scope`.
|
||||
|
||||
Suggested distinction:
|
||||
|
||||
- `prefix_scope`
|
||||
- observation-derived scope hint
|
||||
- `prefix_geography`
|
||||
- prefix-centric geography estimate
|
||||
|
||||
### 3. Update incident visualization geography selection
|
||||
|
||||
Extend:
|
||||
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
|
||||
Selection order:
|
||||
|
||||
1. `prefix_geography.regions`
|
||||
2. ASN geography hints from PeeringDB-derived profile
|
||||
3. observation-derived `affected_regions`
|
||||
|
||||
### 4. Keep evidence visible in the frontend
|
||||
|
||||
Earth should distinguish:
|
||||
|
||||
- event center = prefix geography estimate
|
||||
- evidence lines / collectors = observation proof
|
||||
|
||||
This keeps the event meaningful for non-expert users without losing collector evidence.
|
||||
|
||||
## Earth UX Result
|
||||
|
||||
After this change, a user should see:
|
||||
|
||||
- an incident marker near the estimated affected prefix region
|
||||
- collectors as supporting evidence, not as the event center itself
|
||||
- cables / landing points / nearby infrastructure as weak correlation around the estimated region
|
||||
|
||||
This makes BGP incidents readable as “where the event is likely happening or affecting”, instead of “which station saw it”.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
### Phase 1
|
||||
|
||||
1. Add `IPtoASN / IPtoCountry` datasource support
|
||||
2. Normalize rows into lookup-friendly format
|
||||
3. Enrich BGP events with `prefix_geography`
|
||||
4. Switch incident geography priority to prefer `prefix_geography`
|
||||
|
||||
### Phase 2
|
||||
|
||||
5. Add `OpenGeoFeed` support
|
||||
6. Let geofeed override coarse country-level prefix geography
|
||||
7. Add confidence scoring per geography source
|
||||
|
||||
### Phase 3
|
||||
|
||||
8. Add RIR / whois fallback
|
||||
9. Add better ASN regional footprint from PeeringDB facilities / IXPs
|
||||
10. Refine Earth visual semantics for prefix geography vs observation evidence
|
||||
|
||||
## Recommendation
|
||||
|
||||
The best next engineering move is:
|
||||
|
||||
1. integrate `IPtoASN / IPtoCountry`
|
||||
2. model `prefix_geography` separately from `prefix_scope`
|
||||
3. only then continue refining incident map placement
|
||||
|
||||
Without this layer, any further Earth tuning will still be constrained by collector-centric data.
|
||||
472
docs/plans/earth-real-terrain-plan.md
Normal file
472
docs/plans/earth-real-terrain-plan.md
Normal file
@@ -0,0 +1,472 @@
|
||||
# Earth Real Terrain Plan
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 页当前的“程序噪声假地形”替换成基于真实 DEM 的可用地形层，使 `地形 terrain` 开关真正显示全球海拔起伏，而不是占位效果。
|
||||
|
||||
当前占位实现位于：
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
具体问题：
|
||||
|
||||
- `createTerrain()` 直接对球体顶点应用 `simplex noise`
|
||||
- 没有真实海拔数据来源
|
||||
- 没有分辨率分层
|
||||
- 没有和当前相机/视角配套的性能控制
|
||||
|
||||
## Constraints
|
||||
|
||||
本计划必须贴合当前 Earth 架构，而不是引入一套全新的地形引擎：
|
||||
|
||||
- 地球主体仍然是一个 Three.js sphere
|
||||
- 海缆、登陆点、卫星、BGP 都已经建立在当前球体坐标系之上
|
||||
- 不能为了地形把整页改成 Cesium/MapLibre Globe 之类的全栈替换
|
||||
- 第一阶段优先做“真实可用”，不是一步到位做摄影测量级地形
|
||||
|
||||
## Recommended Data Source
|
||||
|
||||
### Primary recommendation
|
||||
|
||||
使用公开的 Terrarium 编码高程瓦片作为浏览器端高度来源，第一阶段优先接入：
|
||||
|
||||
- Mapzen/AWS `Terrarium` elevation tiles
|
||||
参考：[Mapzen terrain tile format / Terrarium](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
|
||||
原因：
|
||||
|
||||
- 已经是全球瓦片化高程
|
||||
- 浏览器端按 tile 请求，最适合当前 Earth 这种在线 globe
|
||||
- 编码简单稳定：
|
||||
- `heightMeters = (R * 256 + G + B / 256) - 32768`
|
||||
- 不需要我们先离线拼整球 DEM
|
||||
|
||||
### Data quality upgrade path
|
||||
|
||||
如果后面第一阶段效果确认可用，再逐步升级到底层源：
|
||||
|
||||
- Copernicus DEM GLO-30
|
||||
参考：[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
- 或用 Copernicus / SRTM / ASTER 等离线切成我们自己的 terrain tiles
|
||||
|
||||
这条升级路径适合第二阶段，不建议一开始就直接自建全球瓦片服务。
|
||||
|
||||
## Why Not Replace the Engine
|
||||
|
||||
不建议为了地形直接切到 Cesium terrain / quantized mesh 引擎，原因：
|
||||
|
||||
- 现有 Earth 业务对象都依附当前球面坐标
|
||||
- 切引擎会同时波及：
|
||||
- 海缆绘制
|
||||
- 卫星/轨迹
|
||||
- BGP 标记
|
||||
- HUD 与交互
|
||||
- 这是“重做一页”，不是“给地形层接真实数据”
|
||||
|
||||
所以推荐路线是：
|
||||
|
||||
- 保持当前 sphere globe
|
||||
- 为 sphere 增加真实高度位移层
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
分三期推进。
|
||||
|
||||
### Phase 1 — Global Heightmap Terrain Overlay
|
||||
|
||||
目标：
|
||||
|
||||
- 地形层切换后显示真实海拔起伏
|
||||
- 全球范围可用
|
||||
- 性能可控
|
||||
|
||||
做法：
|
||||
|
||||
1. 新增 terrain 数据模块
|
||||
|
||||
建议文件：
|
||||
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
职责：
|
||||
|
||||
- 选择 DEM zoom level
|
||||
- 请求 Terrarium tiles
|
||||
- 解码 tile 高程
|
||||
- 将高程重采样到当前地形球体网格
|
||||
|
||||
2. 替换 `createTerrain()`
|
||||
|
||||
当前：
|
||||
|
||||
- 在 [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) 中同步生成噪声地形
|
||||
|
||||
调整后：
|
||||
|
||||
- `createTerrain()` 只负责创建 terrain mesh 骨架
|
||||
- 真正的顶点位移由 terrain 模块异步注入
|
||||
|
||||
3. 第一阶段采用“整球低分辨率位移”
|
||||
|
||||
不要一上来做动态 patch stitching。第一阶段更稳的办法是：
|
||||
|
||||
- 保留一张全球 terrain sphere
|
||||
- 使用较低分辨率几何
|
||||
- 例如 `SphereGeometry(radius, 192, 192)` 或 `256/256`
|
||||
- 运行时按一个固定地形 zoom（如 `z=4` 或 `z=5`）抓取覆盖全球的 Terrarium tiles
|
||||
- 将 tile 解码后重投影到经纬度采样网格
|
||||
- 将每个球面顶点按真实高度抬升
|
||||
|
||||
这样第一阶段就能做到：
|
||||
|
||||
- 有真实地形
|
||||
- 不需要复杂的局部 LOD
|
||||
- 不会让现有球体对象体系爆炸
|
||||
|
||||
### Phase 2 — View-Aware Refinement
|
||||
|
||||
目标：
|
||||
|
||||
- 正面可见区域更精细
|
||||
- 背面与远处维持低成本
|
||||
|
||||
做法：
|
||||
|
||||
- 引入“基础全球地形 + 当前视角高分局部补丁”
|
||||
- 正面区域额外抓更高 zoom 的高程 tile
|
||||
- 只替换局部顶点位移或局部 overlay mesh
|
||||
|
||||
这一阶段适合在第一阶段稳定后做。
|
||||
|
||||
### Phase 3 — Normals / Shading / Terrain UX
|
||||
|
||||
目标：
|
||||
|
||||
- 地形不仅有起伏，还更好看、更可读
|
||||
|
||||
包括：
|
||||
|
||||
- 根据高度生成更合理的 normals
|
||||
- 调整 terrain material，使山脉/高原更易读
|
||||
- 可选加入：
|
||||
- hillshade
|
||||
- contour lines
|
||||
- snowline / bathymetry tint
|
||||
|
||||
## Calibration Overlay Before More Terrain Tuning
|
||||
|
||||
在当前项目里，terrain 看起来“不像真地形”，不一定只是 DEM 或 exaggeration 不够，也可能是因为缺少稳定参照物。
|
||||
|
||||
没有清晰的海岸线、国界线和地表分层时，人眼很难判断：
|
||||
|
||||
- 山脉是不是在应该高的地方高
|
||||
- terrain 是否真的贴在正确的大陆位置上
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否存在偏移
|
||||
|
||||
这里要明确区分两件事：
|
||||
|
||||
- 国界线不会修好错误的 terrain
|
||||
- 但海岸线 / 国界线会让我们更容易判断 terrain 有没有贴准
|
||||
|
||||
所以在继续盲调 terrain 参数之前，建议先插入一个“校准参照层”阶段。
|
||||
|
||||
### Recommended order for the calibration layer
|
||||
|
||||
1. 海岸线
|
||||
2. 国界线
|
||||
3. 再继续调 terrain
|
||||
|
||||
原因：
|
||||
|
||||
- 海岸线比国界线更基础，也更接近真实地表边界
|
||||
- 判断 terrain 是否贴准，最重要的是大陆边缘和山脉/海岸关系
|
||||
- 国界线更多是政治边界，只能作为辅助参照
|
||||
|
||||
如果只加国界线，不加海岸线，效果仍然可能会怪，因为：
|
||||
|
||||
- 很多国界线本来就是人为直线
|
||||
- 它们并不总是跟真实地形走
|
||||
|
||||
### Suggested layer order during debugging
|
||||
|
||||
建议调试期临时把地球层次明确成：
|
||||
|
||||
1. base earth texture
|
||||
2. coastline / borders overlay
|
||||
3. terrain relief
|
||||
4. cables / landing points / bgp / satellites
|
||||
|
||||
这样会比现在更容易判断：
|
||||
|
||||
- 山脉是否位于正确区域
|
||||
- terrain 是否和地表对齐
|
||||
- 国界/海岸是否漂移
|
||||
|
||||
### Suggested data source for the calibration overlay
|
||||
|
||||
优先用 `Natural Earth` 的轻量全球矢量数据：
|
||||
|
||||
- 海岸线（coastline）
|
||||
- Admin 0 国界线（country borders）
|
||||
|
||||
优点：
|
||||
|
||||
- 全球一致
|
||||
- 轻量
|
||||
- 很适合当前 Three.js globe 做 overlay
|
||||
|
||||
### Recommended execution path
|
||||
|
||||
#### Phase A — Add reference overlays
|
||||
|
||||
先加两层可开关的参考线：
|
||||
|
||||
- 海岸线
|
||||
- 国界线
|
||||
|
||||
这两层的目标不是最终美术表现，而是调试 / 校准。
|
||||
|
||||
#### Phase B — Recalibrate terrain against coastline
|
||||
|
||||
有了海岸线以后，再重新看 terrain：
|
||||
|
||||
- terrain 是否和大陆边缘错位
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否有固定偏移
|
||||
|
||||
#### Phase C — Decide whether to keep the current terrain path
|
||||
|
||||
这时再决定后面的路线：
|
||||
|
||||
- 如果发现真实高程整体是对的，只是缺少 shading / readability
|
||||
继续保留当前 DEM + terrain overlay 路线
|
||||
- 如果发现整球采样投影、本初子午线或 overlay 关系本身就很别扭
|
||||
再考虑重做 terrain pipeline
|
||||
|
||||
### Practical recommendation
|
||||
|
||||
当前阶段不建议“从头开始重做 terrain”。
|
||||
|
||||
更稳的策略是：
|
||||
|
||||
- 暂停继续盲调 terrain 参数
|
||||
- 先补海岸线 / 国界线作为校准参照层
|
||||
- 再基于参照层判断 terrain 是“参数没调好”，还是“整条实现路径有偏移”
|
||||
|
||||
## Recommended Geometry Model
|
||||
|
||||
### First usable model
|
||||
|
||||
保留一层独立 terrain sphere：
|
||||
|
||||
- base earth sphere：贴纹理、昼夜、海洋
|
||||
- terrain sphere：略高于地球半径，真实高程位移
|
||||
|
||||
建议：
|
||||
|
||||
- `terrainBaseRadius = CONFIG.earthRadius + 0.2`
|
||||
- 高度缩放使用真实米制换算，再乘一个可调 exaggeration
|
||||
|
||||
示例关系：
|
||||
|
||||
- `heightWorld = (elevationMeters / 6371000) * CONFIG.earthRadius * exaggeration`
|
||||
|
||||
建议第一阶段 `exaggeration = 1.3 ~ 1.8`
|
||||
|
||||
因为完全真实比例在全球球体上会太平，看不出来。
|
||||
|
||||
## Tile Decoding Plan
|
||||
|
||||
### Terrarium decode
|
||||
|
||||
对于每个高程 tile 像素：
|
||||
|
||||
```text
|
||||
heightMeters = (R * 256 + G + B / 256) - 32768
|
||||
```
|
||||
|
||||
### Sampling path
|
||||
|
||||
对于 terrain mesh 上每个顶点：
|
||||
|
||||
1. 将顶点方向转成经纬度
|
||||
2. 将经纬度映射到 Web Mercator tile 坐标
|
||||
3. 找到对应的 tile 和像素
|
||||
4. 解码高程
|
||||
5. 将顶点沿法线方向抬升
|
||||
|
||||
### Needed helpers
|
||||
|
||||
建议新增：
|
||||
|
||||
- `latLonToTileXY(lat, lon, z)`
|
||||
- `tilePixelFromLatLon(lat, lon, z, tileSize)`
|
||||
- `decodeTerrariumHeight(r, g, b)`
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
为了不让地形开关每次重开都重新抓全量 tile：
|
||||
|
||||
- terrain tile 按 `z/x/y` 存到内存缓存
|
||||
- terrain mesh 结果也缓存一份
|
||||
- 当用户关闭/开启 terrain：
|
||||
- 直接复用已有位移结果
|
||||
|
||||
建议：
|
||||
|
||||
- `Map<string, Float32Array | ImageBitmap>`
|
||||
|
||||
## Material Strategy
|
||||
|
||||
第一阶段不要复杂化。
|
||||
|
||||
建议 terrain material：
|
||||
|
||||
- 半透明低饱和地形色
|
||||
- 比 base earth 稍亮或稍偏冷
|
||||
- 保留当前 HUD 风格下的可读性
|
||||
|
||||
第一阶段不需要：
|
||||
|
||||
- 真实土地覆被纹理
|
||||
- 独立卫星影像贴 terrain
|
||||
|
||||
因为那会和现有地球纹理、云层、昼夜 shader 打架。
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Files to change
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 重写 `createTerrain()`
|
||||
- 删除 simplex noise 占位逻辑
|
||||
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 初始化 terrain 数据加载
|
||||
- 控制 terrain readiness / loading message
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
- `toggleTerrain` 逻辑保持，但应能区分：
|
||||
- mesh 已就绪
|
||||
- 正在加载
|
||||
- [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 新增 `TERRAIN_CONFIG`
|
||||
- 新文件：
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
### Suggested new config
|
||||
|
||||
建议新增：
|
||||
|
||||
```js
|
||||
export const TERRAIN_CONFIG = {
|
||||
enabled: true,
|
||||
tileSize: 256,
|
||||
baseZoom: 4,
|
||||
baseRadiusOffset: 0.2,
|
||||
exaggeration: 1.5,
|
||||
opacity: 0.55,
|
||||
color: 0x6c876f,
|
||||
maxConcurrentRequests: 8,
|
||||
cacheEnabled: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Loading UX
|
||||
|
||||
地形第一次开启时，不能像现在一样瞬时切换。
|
||||
|
||||
建议：
|
||||
|
||||
- 如果地形数据尚未准备：
|
||||
- 顶部状态条显示：`正在加载真实地形数据...`
|
||||
- 完成后：
|
||||
- `真实地形已就绪`
|
||||
|
||||
如果加载失败：
|
||||
|
||||
- 保留 base earth
|
||||
- 显示轻量错误提示
|
||||
- 不要让 terrain 开关卡死在“开”状态
|
||||
|
||||
## Risks
|
||||
|
||||
### 1. Global tile count too high
|
||||
|
||||
即使 `z=5` 全球 tile 数也不少。
|
||||
|
||||
缓解：
|
||||
|
||||
- 第一阶段限定低 zoom
|
||||
- 并发上限
|
||||
- 缓存
|
||||
|
||||
### 2. Mesh resolution too low
|
||||
|
||||
如果球面分段太低，山脉会被抹平。
|
||||
|
||||
缓解：
|
||||
|
||||
- 第一阶段先选一个中等分辨率
|
||||
- 用 exaggeration 保证可见性
|
||||
|
||||
### 3. Existing overlays may z-fight with terrain
|
||||
|
||||
海缆、登陆点、BGP、卫星相关对象都假设地球半径固定。
|
||||
|
||||
缓解：
|
||||
|
||||
- terrain sphere 单独作为 overlay
|
||||
- overlay 保持略低或略高的固定 offset
|
||||
- 必要时局部调整 landing point / cable altitude offset
|
||||
|
||||
### 4. Mercator sampling distortion near poles
|
||||
|
||||
Web Mercator 在高纬会有失真。
|
||||
|
||||
缓解：
|
||||
|
||||
- 第一阶段接受
|
||||
- 后续若需要更严格极区质量，再上 geodetic reprojection pipeline
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
第一阶段完成后，应满足：
|
||||
|
||||
1. `地形 terrain` 开关开启时，地表起伏明显不再是随机噪声
|
||||
2. 喜马拉雅、安第斯、落基山、东非高原等全球大尺度地形可辨认
|
||||
3. 关闭/重新开启 terrain 不重复全量请求
|
||||
4. 不破坏：
|
||||
- 海缆
|
||||
- 卫星
|
||||
- BGP
|
||||
- 地球昼夜
|
||||
- 天球层
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. 引入 `TERRAIN_CONFIG`
|
||||
2. 新建 `terrain.js`
|
||||
3. 实现 Terrarium tile 请求与 decode
|
||||
4. 用低 zoom 全球 tile 构建真实 terrain sphere
|
||||
5. 接管 `toggleTerrain()`
|
||||
6. 调整 terrain material 和高度 exaggeration
|
||||
7. 做缓存
|
||||
8. 再考虑第二阶段局部高分 refinement
|
||||
|
||||
## Source References
|
||||
|
||||
- Mapzen Terrarium / AWS terrain tiles
|
||||
[Mapzen Terrain Tile Service](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
- Terrarium tile experiments / format background
|
||||
[mapzen/terrarium](https://github.com/mapzen/terrarium)
|
||||
- Copernicus DEM overview
|
||||
[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
|
||||
## Recommendation Summary
|
||||
|
||||
如果现在就要开始做，我建议直接按这条路线开工：
|
||||
|
||||
- 第一阶段接入 Terrarium 全球高程 tile
|
||||
- 替换掉当前 simplex 假地形
|
||||
- 先做一层真实可见的全球 terrain overlay
|
||||
- 等第一阶段稳定，再做视角高分 refinement
|
||||
|
||||
这是对当前项目风险最低、最贴合现有 Earth 架构的一条路。
|
||||
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Earth Renderer / Logic Separation Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/earth-architecture-refactor.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 前端继续往“逻辑层 / 状态层 / 渲染层”分离推进，降低后续这几类工作的耦合成本：
|
||||
|
||||
- Three.js 渲染重构
|
||||
- 部分图层替换实现
|
||||
- 未来 UE / Cesium 客户端迁移
|
||||
- Earth 行为逻辑复用
|
||||
|
||||
## Why This Matters
|
||||
|
||||
当前 Earth 已经有一些良好分层，例如：
|
||||
|
||||
- 图层显隐入口
|
||||
- Cable state 枚举与状态 map
|
||||
- 交互逻辑与实际视觉效果的部分分离
|
||||
|
||||
但还没有形成一套更明确的统一规则。现在的风险是：
|
||||
|
||||
- 同一类对象的 hover / locked / hidden / loading 语义不一致
|
||||
- 状态和渲染更新散落在多个模块
|
||||
- 后续再加新图层时容易复制旧逻辑
|
||||
|
||||
## Target Architecture
|
||||
|
||||
Earth 对每类对象都尽量拆成三层：
|
||||
|
||||
1. `state layer`
|
||||
- 保存对象状态
|
||||
- 例如：`normal / hovered / locked / hidden / loading`
|
||||
|
||||
2. `logic layer`
|
||||
- 处理点击、悬停、锁定、过滤、显隐切换
|
||||
- 不直接关心 Three.js 具体材质怎么改
|
||||
|
||||
3. `renderer layer`
|
||||
- 根据状态更新 Three.js / HUD 外观
|
||||
- 是最容易针对不同渲染引擎替换的一层
|
||||
|
||||
## Current Good Signals
|
||||
|
||||
当前已经接近这条方向的地方：
|
||||
|
||||
- cable 状态管理
|
||||
- 部分 landing point 状态同步
|
||||
- layer button 的统一状态入口
|
||||
- tooltip / legend / info-card 开始朝状态驱动靠拢
|
||||
|
||||
## Next Steps
|
||||
|
||||
### 1. Standardize object state enums
|
||||
|
||||
优先为这些对象建立更稳定的状态语义：
|
||||
|
||||
- cables
|
||||
- satellites
|
||||
- landing points
|
||||
- BGP markers
|
||||
- media / news 面板入口按钮
|
||||
|
||||
### 2. Unify state-to-visual adapters
|
||||
|
||||
为各模块建立更清晰的渲染适配函数，例如：
|
||||
|
||||
- `applyCableVisualState()`
|
||||
- `applySatelliteVisualState()`
|
||||
- `applyBGPVisualState()`
|
||||
|
||||
要求：
|
||||
|
||||
- 逻辑层只改状态
|
||||
- 视觉层负责把状态映射到材质、透明度、发光、尺寸、文字
|
||||
|
||||
### 3. Separate Earth UI state from render state
|
||||
|
||||
HUD / 面板 / 图层按钮状态也需要和渲染状态分离：
|
||||
|
||||
- `loading`
|
||||
- `active`
|
||||
- `locked`
|
||||
- `hidden`
|
||||
- `error`
|
||||
|
||||
不要再让 UI 通过“猜渲染结果”推导业务状态。
|
||||
|
||||
### 4. Prepare migration-safe boundaries
|
||||
|
||||
后续如果做 UE / Cesium 客户端，尽量保留：
|
||||
|
||||
- 状态枚举
|
||||
- 交互规则
|
||||
- 数据层接口
|
||||
|
||||
只替换：
|
||||
|
||||
- Three.js 具体渲染实现
|
||||
- HUD 展示实现
|
||||
|
||||
## Practical Rule
|
||||
|
||||
后续 Earth 新功能开发时，优先问三个问题：
|
||||
|
||||
1. 这个状态由谁持有？
|
||||
2. 这个交互逻辑在哪一层处理？
|
||||
3. 这个视觉变化是否能在不改逻辑的情况下单独替换？
|
||||
|
||||
如果答不上来，就说明还在把状态、逻辑、渲染揉在一起。
|
||||
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Earth WebGL Instancing Satellites Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/webgl-instancing-satellites.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
把 Earth 卫星渲染从当前方案继续推进到更适合高数量卫星的 instancing 方向，目标是：
|
||||
|
||||
- 支持更多卫星
|
||||
- 降低渲染压力
|
||||
- 仍然保留当前数据层和交互层
|
||||
|
||||
## Why It Matters
|
||||
|
||||
当前卫星系统已经具备：
|
||||
|
||||
- 数据加载
|
||||
- 轨迹
|
||||
- 选择/锁定
|
||||
- 图例
|
||||
- 相关区域联动
|
||||
|
||||
但当卫星数量持续增加时，渲染层会越来越接近瓶颈。
|
||||
|
||||
## Recommended Direction
|
||||
|
||||
优先调研并原型验证：
|
||||
|
||||
- `InstancedBufferGeometry + custom shader`
|
||||
|
||||
而不是一开始就推倒重写成 raw WebGL。
|
||||
|
||||
原因：
|
||||
|
||||
- 仍能保留 Three.js 主架构
|
||||
- 更容易渐进迁移
|
||||
- 比继续堆普通点渲染更有上限
|
||||
|
||||
## What Should Stay
|
||||
|
||||
尽量保留这些层：
|
||||
|
||||
- 卫星数据获取
|
||||
- 位置计算
|
||||
- 锁定/悬停逻辑
|
||||
- legend / info-card / 相关联动
|
||||
|
||||
主要替换的是：
|
||||
|
||||
- 卫星点渲染实现
|
||||
- 颜色/大小等实例属性更新方式
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Prototype
|
||||
|
||||
- 用 instancing 做最小原型
|
||||
- 先只渲染卫星点
|
||||
- 不碰轨迹系统
|
||||
|
||||
### Phase 2: Integrate
|
||||
|
||||
- 接入当前 `satellites.js` 数据层
|
||||
- 保留当前选择和高亮语义
|
||||
|
||||
### Phase 3: Tune
|
||||
|
||||
- 调整可视大小
|
||||
- 调整选中高亮方式
|
||||
- 评估是否需要分层 LOD
|
||||
|
||||
## Risks
|
||||
|
||||
1. 透明度排序更复杂
|
||||
2. Shader 调试成本更高
|
||||
3. 选中态和 hover 态不能简单复用旧材质逻辑
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 在更高卫星数量下保持可接受帧率
|
||||
2. 不破坏现有锁定/高亮语义
|
||||
3. 图例、信息卡、相关卫星联动仍然成立
|
||||
361
docs/plans/frontend-ai-playground-development-plan.md
Normal file
361
docs/plans/frontend-ai-playground-development-plan.md
Normal file
@@ -0,0 +1,361 @@
|
||||
# AI Playground Development Plan
|
||||
|
||||
## 目标
|
||||
|
||||
这份计划用于统一 `aiprovider`、`backend AI facade`、`Playground` 页面，以及后续 `BGP / 告警 / 数据源健康` 等 AI 入口的演进方向。
|
||||
|
||||
当前原则：
|
||||
|
||||
- `aiprovider` 继续作为独立模型网关
|
||||
- `backend` 继续作为稳定业务入口
|
||||
- `frontend` 负责测试台和业务 UI
|
||||
- 先做“可控、可验证、可解释”的 AI 能力，再逐步引入 agent/tool calling
|
||||
|
||||
## 当前已完成
|
||||
|
||||
### 1. AI 网关基础层
|
||||
|
||||
已完成：
|
||||
|
||||
- 独立 `aiprovider` 服务
|
||||
- `backend -> aiprovider -> model provider` 调用链
|
||||
- `provider/status` 与 `situational-awareness/analyze` 稳定接口
|
||||
- `X-Request-ID` 透传
|
||||
- 轻量超时与重试
|
||||
- MiniMax / Anthropic-compatible / OpenAI-compatible / Ollama 适配
|
||||
|
||||
相关文件：
|
||||
|
||||
- [backend/app/api/v1/ai.py](/home/ray/dev/linkong/planet/backend/app/api/v1/ai.py)
|
||||
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
|
||||
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
|
||||
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
|
||||
### 2. 本地运行与配置打通
|
||||
|
||||
已完成：
|
||||
|
||||
- `planet.sh` 启动链路纳入 `aiprovider`
|
||||
- `planet.sh` 启动完成后输出 Playground 入口
|
||||
- `docker-compose.yml` 为 `aiprovider` 加入 `env_file`
|
||||
- `backend/.env` 与 `aiprovider/.env` 两侧 service token 对齐
|
||||
- `Playground` 状态缓存，避免页面切换时每次都重新请求 provider 状态
|
||||
|
||||
相关文件：
|
||||
|
||||
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
|
||||
- [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml)
|
||||
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
|
||||
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
|
||||
|
||||
### 3. Playground UI 基础版
|
||||
|
||||
已完成：
|
||||
|
||||
- 新增前端路由 `/playground`
|
||||
- 左侧 `Provider 状态 + 测试说明`
|
||||
- 右侧 `请求 / 结果` Tabs
|
||||
- `Provider 状态` 支持手动刷新
|
||||
- `测试说明` 支持折叠
|
||||
- 内部区域采用细滚动条
|
||||
- 页面布局开始遵循“单屏工作区 + 模块内部滚动”规范
|
||||
|
||||
相关文件：
|
||||
|
||||
- [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
- [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
- [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
### 4. 前端布局规范沉淀
|
||||
|
||||
已完成：
|
||||
|
||||
- 把“一屏工作区、主模块优先、模块内部滚动”的规范文档化
|
||||
- 明确 `BGP` 页面为当前参考实现
|
||||
|
||||
相关文件：
|
||||
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
|
||||
## 当前限制
|
||||
|
||||
### 1. Playground 还是 prompt playground，不是 agent playground
|
||||
|
||||
当前 `Playground` 的 `观察项 / 目标 / 约束条件` 都是人工输入。
|
||||
|
||||
模型现在拿到的是：
|
||||
|
||||
- 你手工输入的结构化字段
|
||||
- 后端传递的少量静态上下文
|
||||
|
||||
模型现在拿不到：
|
||||
|
||||
- 实时 BGP 事件
|
||||
- 真实告警列表
|
||||
- 数据源健康状态
|
||||
- 自动检索结果
|
||||
- tool calling / skills / 自主取数
|
||||
|
||||
### 2. `situational-awareness/analyze` 还是通用提示词接口
|
||||
|
||||
当前更适合：
|
||||
|
||||
- 测试链路
|
||||
- 测试模型输出风格
|
||||
- 验证不同 provider 是否正常返回
|
||||
|
||||
当前还不适合：
|
||||
|
||||
- 直接当真实态势系统主入口
|
||||
- 让用户手工维护长期分析模板
|
||||
- 代替专用业务研判接口
|
||||
|
||||
### 3. 还没有可验证的真实业务输入注入
|
||||
|
||||
目前最缺的是：
|
||||
|
||||
- 从业务系统自动整理“事实输入”
|
||||
- 再把这些事实喂给 AI
|
||||
|
||||
而不是继续让用户在 Playground 手工输入真实事件摘要。
|
||||
|
||||
## 短期计划
|
||||
|
||||
### Phase A: Playground 收敛为稳定测试台
|
||||
|
||||
目标：
|
||||
|
||||
- 保持 Playground 简洁可用
|
||||
- 不再继续堆“高级参数”
|
||||
|
||||
工作项：
|
||||
|
||||
- 继续微调左侧 `Provider 状态` 与 `测试说明` 的空间策略
|
||||
- 保持 `请求 / 结果` 为单一主工作区
|
||||
- 不引入盲填式高级字段
|
||||
- 统一滚动条、卡片、溢出行为
|
||||
|
||||
完成标准：
|
||||
|
||||
- 笔记本视口下依然可用
|
||||
- 各模块标题可见
|
||||
- 主要阅读区始终是右侧 Tabs
|
||||
|
||||
### Phase B: BGP AI 简报
|
||||
|
||||
目标：
|
||||
|
||||
- 不再依赖手工填写“观察项”
|
||||
- 让系统自动把真实 BGP 数据注入 AI
|
||||
- 让 BGP 页面逐步从“摘要汇总”升级为“证据驱动的区域态势分析”
|
||||
|
||||
建议实现：
|
||||
|
||||
- 新增专用后端接口，例如：
|
||||
- `POST /api/v1/ai/bgp/brief`
|
||||
- 后端自动读取：
|
||||
- incidents summary
|
||||
- anomalies
|
||||
- recent events
|
||||
- collector coverage summary
|
||||
- 后端将结构化事实注入 `context / observations`
|
||||
- 前端在 BGP 页面增加“生成 AI 简报”
|
||||
|
||||
当前阶段说明：
|
||||
|
||||
- 第一版 `BGP AI 简报` 允许先落地为“值班摘要生成器”
|
||||
- 也就是先把 incidents / anomalies / events / collector coverage 自动注入
|
||||
- 允许模型先做事实摘要、风险归纳、建议动作
|
||||
|
||||
但这不应被视为 Phase B 的最终形态。
|
||||
|
||||
Phase B 后续还需要补齐：
|
||||
|
||||
- prefix geography 证据注入
|
||||
- `iptoasn`
|
||||
- `opengeofeed`
|
||||
- `nro_delegated`
|
||||
- 基于 `affected_regions` 与 prefix geography 的区域聚合
|
||||
- 区分“真实区域热度”与“collector coverage 偏差”
|
||||
- 对高风险 prefix / ASN 给出更明确的国家、城市、运营商归属线索
|
||||
- 让 AI 输出明确回答：
|
||||
- 哪些区域正在异常升温
|
||||
- 哪些结论只是观测站偏差
|
||||
- 当前还缺哪些区域证据
|
||||
|
||||
完成标准：
|
||||
|
||||
- 用户不需要手工录入 BGP 观察项
|
||||
- AI 输出能明确区分“事实”和“研判”
|
||||
- AI 不只是复述总量和最近几条事件，还能利用 prefix geography 与 affected regions 做区域态势判断
|
||||
- 输出中能明确指出：
|
||||
- 高风险区域
|
||||
- 区域证据来源
|
||||
- collector coverage 偏差对判断的影响
|
||||
|
||||
### Phase C: 告警 / 数据源健康 AI 简报
|
||||
|
||||
目标：
|
||||
|
||||
- 复用同样模式，扩展到其他模块
|
||||
|
||||
建议入口：
|
||||
|
||||
- `Alerts` 页面：异常与告警摘要
|
||||
- `DataSources` 页面：采集失败与健康状态总结
|
||||
|
||||
原则：
|
||||
|
||||
- 每个业务页优先做“专用 AI 简报”
|
||||
- 不优先做“万能大聊天框”
|
||||
|
||||
## 中期计划
|
||||
|
||||
### 1. Assessment Layer
|
||||
|
||||
目标：
|
||||
|
||||
- 不只返回自由文本
|
||||
- 返回结构化的 assessment
|
||||
|
||||
建议输出字段：
|
||||
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- recommendations
|
||||
- confidence
|
||||
- missing_data
|
||||
|
||||
这样后续才能：
|
||||
|
||||
- 持久化
|
||||
- 回看
|
||||
- 对比不同时间的 AI 结论
|
||||
- 在 Earth / Dashboard / BGP 页面稳定展示
|
||||
|
||||
### 2. Evidence-first Runtime
|
||||
|
||||
目标：
|
||||
|
||||
- 所有 AI 分析先取真实数据，再调模型
|
||||
|
||||
原则：
|
||||
|
||||
- 先 evidence
|
||||
- 再 prompt
|
||||
- 最后才是自由生成
|
||||
|
||||
优先要做的不是更强聊天，而是：
|
||||
|
||||
- 更稳定的数据注入
|
||||
- 更一致的事实模板
|
||||
- 更清晰的结果结构
|
||||
|
||||
### 3. 按页面提供专用入口
|
||||
|
||||
目标：
|
||||
|
||||
- 让 AI 成为业务视图的一部分，而不是孤立 playground
|
||||
|
||||
优先顺序建议：
|
||||
|
||||
1. `BGP` AI 简报
|
||||
2. `Alerts` AI 简报
|
||||
3. `DataSources` 健康研判
|
||||
4. `Dashboard` 总览总结
|
||||
|
||||
## 长期计划
|
||||
|
||||
### 1. Tool Calling / Agent Runtime
|
||||
|
||||
只有在以下基础稳定后再推进：
|
||||
|
||||
- 数据源健康信号稳定
|
||||
- BGP / Alerts / Datasource evidence 注入稳定
|
||||
- assessment 结构稳定
|
||||
|
||||
长期可做能力：
|
||||
|
||||
- AI 调用受控工具查询业务数据
|
||||
- AI 调用检索/web search 做外部验证
|
||||
- AI 生成建议而不是直接修改系统
|
||||
- 审核后触发受控动作
|
||||
|
||||
### 2. 受控动作与闭环
|
||||
|
||||
潜在方向：
|
||||
|
||||
- 根据健康异常生成修复建议
|
||||
- 根据态势变化生成处理建议
|
||||
- 进入 review queue
|
||||
- 审批后执行
|
||||
- 验证结果并形成闭环
|
||||
|
||||
### 3. 多模块统一 AI 体验
|
||||
|
||||
长期目标不是一个孤立 Playground，而是：
|
||||
|
||||
- 每个业务页都有自己的 AI 入口
|
||||
- 共享统一的 backend AI facade
|
||||
- 共享统一的 assessment 结构
|
||||
- 共享统一的 evidence 注入与审计链路
|
||||
|
||||
## 设计决策总结
|
||||
|
||||
### 为什么保留 `aiprovider`
|
||||
|
||||
因为它已经很好地承担了：
|
||||
|
||||
- provider 适配
|
||||
- 协议兼容
|
||||
- service token 边界
|
||||
- 独立重启与部署
|
||||
|
||||
因此短期内不建议把它并回 `backend`。
|
||||
|
||||
### 为什么 Playground 不做成万能聊天页
|
||||
|
||||
因为当前更需要的是：
|
||||
|
||||
- 稳定测试链路
|
||||
- 可验证业务输入
|
||||
- 专用分析入口
|
||||
|
||||
而不是一个泛化但没有真实数据支撑的聊天框。
|
||||
|
||||
### 为什么优先做专用 AI 简报
|
||||
|
||||
因为：
|
||||
|
||||
- 数据可以自动注入
|
||||
- 用户心智更清晰
|
||||
- 输出更容易结构化
|
||||
- 更容易校验事实与研判是否一致
|
||||
|
||||
## 下一步建议
|
||||
|
||||
按优先级建议接下来这样做：
|
||||
|
||||
1. 稳住 `Playground` 当前布局，不再大幅重做
|
||||
2. 在 `BGP` 页面新增专用 “AI 简报” 入口
|
||||
3. 后端新增 `BGP brief` 专用接口，自动注入真实数据
|
||||
4. 补齐 `BGP brief` 的区域态势证据层
|
||||
5. 把 AI 输出逐步从自由文本升级为结构化 assessment
|
||||
|
||||
### BGP Brief 后续子项
|
||||
|
||||
为避免把“已有 AI 简报”误判成“区域分析已完成”，这里单独记录 `BGP brief` 的后续 backlog：
|
||||
|
||||
1. 把高风险 prefix 命中的 `iptoasn / opengeofeed / nro_delegated` 结果注入 brief context
|
||||
2. 按国家/城市聚合 active incidents、anomalies、affected prefixes，生成区域热点事实层
|
||||
3. 把 collector coverage 与区域热点并排注入，避免模型把观测偏差误判成区域风险
|
||||
4. 对高风险 ASN / prefix 追加归属线索，如国家、城市、可能运营商或注册区域
|
||||
5. 在输出结构中单独增加：
|
||||
- 区域态势
|
||||
- 证据来源
|
||||
- 观测偏差说明
|
||||
- 缺失区域证据
|
||||
1015
docs/plans/ue5-mvp-fused-plan.md
Normal file
1015
docs/plans/ue5-mvp-fused-plan.md
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user