release: bump version to 0.50.0
This commit is contained in:
@@ -25,6 +25,9 @@
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
|
||||
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
|
||||
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
|
||||
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
|
||||
- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
|
||||
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
|
||||
- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
|
||||
@@ -32,6 +35,7 @@
|
||||
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/plans/docs-gatekeeper-auth-plan.md)
|
||||
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/plans/location-resolver-shared-pipeline-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [Lightweight Agent Orchestrator 与 WebSearch 证据层计划](/home/ray/dev/linkong/planet/docs/plans/agents-light-orchestrator-websearch-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
407
docs/plans/agents-light-orchestrator-websearch-plan.md
Normal file
407
docs/plans/agents-light-orchestrator-websearch-plan.md
Normal file
@@ -0,0 +1,407 @@
|
||||
# Lightweight Agent Orchestrator and WebSearch Evidence Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Planet should not turn `aiprovider` into a general-purpose agent runtime.
|
||||
|
||||
`aiprovider` should remain the model gateway:
|
||||
|
||||
- provider compatibility
|
||||
- protocol adaptation
|
||||
- model authentication
|
||||
- request and response normalization
|
||||
|
||||
Agent behavior belongs in the backend, where Planet already owns business state,
|
||||
permissions, persistence, evidence records, and operator workflows.
|
||||
|
||||
The recommended direction is a lightweight backend Agent Orchestrator with a
|
||||
controlled tool layer. The first version should use fixed workflows instead of a
|
||||
free-form tool-calling loop.
|
||||
|
||||
|
||||
## Architecture Decision
|
||||
|
||||
Use this boundary:
|
||||
|
||||
```text
|
||||
aiprovider = model adapter only
|
||||
backend Agent = task orchestration + tools + evidence + policy + business rules
|
||||
```
|
||||
|
||||
This keeps model transport separate from Planet-specific behavior. It also lets
|
||||
OpenAI, MiniMax, Anthropic-compatible providers, Ollama, and later providers all
|
||||
reuse the same backend tools.
|
||||
|
||||
Recommended module shape:
|
||||
|
||||
```text
|
||||
backend/app/services/
|
||||
ai/
|
||||
agent_orchestrator.py
|
||||
tool_registry.py
|
||||
prompts.py
|
||||
schemas.py
|
||||
ai_tools/
|
||||
web_search.py
|
||||
web_fetch.py
|
||||
geo_resolve.py
|
||||
internal_data_query.py
|
||||
incident_query.py
|
||||
evidence_store.py
|
||||
situation/
|
||||
bgp_analyzer.py
|
||||
risk_scoring.py
|
||||
event_correlator.py
|
||||
alert_policy.py
|
||||
|
||||
aiprovider/
|
||||
provider_service.py
|
||||
main.py
|
||||
```
|
||||
|
||||
|
||||
## Phase 1: Controlled Workflow Agent
|
||||
|
||||
The first implementation should not be a full OpenClaw/Codex-style agent loop.
|
||||
Planet's immediate needs are better served by explicit workflows:
|
||||
|
||||
1. `tutorial_refresh`
|
||||
2. `geo_correction`
|
||||
3. `situation_brief`
|
||||
|
||||
Each workflow should:
|
||||
|
||||
1. collect evidence with backend tools
|
||||
2. normalize and store evidence
|
||||
3. call `AIProviderClient` through the configured global provider/model/key
|
||||
4. validate the result with Pydantic schemas
|
||||
5. return a proposal, candidate, or brief instead of directly mutating critical state
|
||||
|
||||
For location correction, the flow should be:
|
||||
|
||||
```text
|
||||
object name / type / current coordinate / description
|
||||
-> web_search
|
||||
-> web_fetch for selected results
|
||||
-> geo_resolve for city/site coordinates
|
||||
-> LLM structured extraction
|
||||
-> schema validation and confidence scoring
|
||||
-> pending review candidate
|
||||
```
|
||||
|
||||
The LLM output must be constrained to a schema such as:
|
||||
|
||||
```json
|
||||
{
|
||||
"object_id": "string",
|
||||
"object_type": "datacenter|ixp|submarine_cable|asn|city|facility|satellite",
|
||||
"current_location": {
|
||||
"lat": 0,
|
||||
"lon": 0
|
||||
},
|
||||
"suggested_location": {
|
||||
"lat": 0,
|
||||
"lon": 0
|
||||
},
|
||||
"confidence": 0.82,
|
||||
"reason": "short evidence-backed explanation",
|
||||
"evidence": [
|
||||
{
|
||||
"title": "source title",
|
||||
"url": "https://example.com/source",
|
||||
"quote": "short supporting excerpt",
|
||||
"retrieved_at": "2026-05-10T00:00:00Z"
|
||||
}
|
||||
],
|
||||
"needs_human_review": true
|
||||
}
|
||||
```
|
||||
|
||||
The LLM may generate a suggestion, but it must not directly write final
|
||||
coordinates into the dimension tables.
|
||||
|
||||
|
||||
## Phase 2: Backend Tool Registry
|
||||
|
||||
Add a small Python tool interface in the backend:
|
||||
|
||||
```python
|
||||
class ToolResult(BaseModel):
|
||||
ok: bool
|
||||
data: Any = None
|
||||
error: str | None = None
|
||||
evidence: list[dict] = []
|
||||
```
|
||||
|
||||
Register tools through a backend registry:
|
||||
|
||||
```text
|
||||
web_search
|
||||
web_fetch
|
||||
geo_resolve
|
||||
internal_data_query
|
||||
incident_query
|
||||
evidence_store
|
||||
```
|
||||
|
||||
Do not put WebSearch inside `aiprovider`.
|
||||
|
||||
Reasons:
|
||||
|
||||
- search is a business tool, not a model-provider feature
|
||||
- search evidence must be stored and audited by the backend
|
||||
- different LLM providers should share the same search pipeline
|
||||
- Planet may switch between Tavily, Brave, Exa, SearXNG, or MiniMax MCP without
|
||||
changing model transport
|
||||
|
||||
The first WebSearch implementation should be an HTTP evidence provider. Tavily is
|
||||
the recommended first default because it is simple to call from the existing
|
||||
`httpx` backend stack and returns LLM/RAG-friendly search results. The interface
|
||||
should remain provider-neutral so Brave, Exa, SearXNG, or MiniMax MCP can be
|
||||
added later.
|
||||
|
||||
WebSearch configuration should live under PostgreSQL `system_settings` with the
|
||||
rest of external integrations:
|
||||
|
||||
```text
|
||||
external_integrations.web_search
|
||||
enabled
|
||||
provider
|
||||
api_key
|
||||
base_url
|
||||
max_results
|
||||
timeout_seconds
|
||||
```
|
||||
|
||||
Secret resolution should follow the existing settings pattern:
|
||||
|
||||
1. saved PostgreSQL secret
|
||||
2. provider-specific environment variable, for example `TAVILY_API_KEY`
|
||||
3. generic fallback `WEB_SEARCH_API_KEY`
|
||||
|
||||
|
||||
## Phase 3: Limited Agent Loop
|
||||
|
||||
After the fixed workflows are stable, the backend can add a limited agent loop:
|
||||
|
||||
```text
|
||||
LLM sees an allowed tool list
|
||||
-> LLM requests a tool call
|
||||
-> backend validates and executes the tool
|
||||
-> tool result is added to context
|
||||
-> LLM continues
|
||||
-> final structured output after at most N steps
|
||||
```
|
||||
|
||||
Guardrails:
|
||||
|
||||
- max tool steps: 3 to 5
|
||||
- only read-only tools may run automatically
|
||||
- writes go to pending review first
|
||||
- all web evidence must be persisted
|
||||
- all final outputs must pass schema validation
|
||||
- prompts must include explicit evidence boundaries
|
||||
|
||||
Permission levels:
|
||||
|
||||
```text
|
||||
L0: pure analysis, no tools
|
||||
L1: read-only tools, web_search / web_fetch / internal_query
|
||||
L2: proposal generation, write pending review records
|
||||
L3: low-risk notifications and briefs
|
||||
L4: database mutation or alert triggering, human confirmation required
|
||||
```
|
||||
|
||||
|
||||
## Situational Awareness Boundary
|
||||
|
||||
Planet's situational-awareness layer should not rely on the LLM as the primary
|
||||
risk engine.
|
||||
|
||||
Use deterministic analysis for:
|
||||
|
||||
- anomaly type
|
||||
- affected prefixes
|
||||
- affected ASNs
|
||||
- geographic scope
|
||||
- duration
|
||||
- severity score
|
||||
- confidence
|
||||
- related events
|
||||
- raw evidence
|
||||
|
||||
Use the LLM for:
|
||||
|
||||
- readable summaries
|
||||
- risk explanation
|
||||
- likely impact narrative
|
||||
- next recommended actions
|
||||
- missing data requests
|
||||
|
||||
In short:
|
||||
|
||||
```text
|
||||
deterministic services compute the score
|
||||
LLM explains the evidence and options
|
||||
```
|
||||
|
||||
Proactive alerts should be triggered by deterministic rules or scheduled jobs,
|
||||
then optionally summarized by the Agent Orchestrator.
|
||||
|
||||
|
||||
## Persistence Model
|
||||
|
||||
Add lightweight persistence for auditability:
|
||||
|
||||
```text
|
||||
ai_tasks
|
||||
id
|
||||
task_type
|
||||
status
|
||||
input_json
|
||||
output_json
|
||||
model
|
||||
created_at
|
||||
finished_at
|
||||
error
|
||||
|
||||
ai_evidence
|
||||
id
|
||||
task_id
|
||||
source_type
|
||||
title
|
||||
url
|
||||
snippet
|
||||
content_hash
|
||||
retrieved_at
|
||||
credibility_score
|
||||
|
||||
ai_briefs
|
||||
id
|
||||
brief_type
|
||||
severity
|
||||
title
|
||||
summary
|
||||
evidence_ids
|
||||
related_entity_ids
|
||||
created_at
|
||||
acknowledged_at
|
||||
|
||||
ai_location_suggestions
|
||||
id
|
||||
object_type
|
||||
object_id
|
||||
old_lat
|
||||
old_lon
|
||||
new_lat
|
||||
new_lon
|
||||
confidence
|
||||
reason
|
||||
evidence_ids
|
||||
status
|
||||
```
|
||||
|
||||
The tables can be introduced incrementally. The first implementation may start
|
||||
with `ai_tasks` and `ai_evidence`, then add specialized tables when the UI needs
|
||||
review queues and acknowledgement state.
|
||||
|
||||
|
||||
## MVP Scope
|
||||
|
||||
The MVP should deliver three fixed capabilities:
|
||||
|
||||
### 1. Tutorial Refresh
|
||||
|
||||
Input:
|
||||
|
||||
- provider or tutorial topic
|
||||
- current tutorial text
|
||||
- known stale point, when available
|
||||
|
||||
Tools:
|
||||
|
||||
- `web_search`
|
||||
- `web_fetch`
|
||||
|
||||
Output:
|
||||
|
||||
- updated Markdown
|
||||
- source list
|
||||
- verification status
|
||||
|
||||
### 2. Geo Correction
|
||||
|
||||
Input:
|
||||
|
||||
- object id
|
||||
- object name
|
||||
- object type
|
||||
- current coordinates
|
||||
- source description
|
||||
|
||||
Tools:
|
||||
|
||||
- `web_search`
|
||||
- `web_fetch`
|
||||
- `geo_resolve`
|
||||
|
||||
Output:
|
||||
|
||||
- `LocationCorrection` JSON
|
||||
- evidence list
|
||||
- pending review candidate
|
||||
|
||||
### 3. Situation Brief
|
||||
|
||||
Input:
|
||||
|
||||
- anomaly event
|
||||
- deterministic findings
|
||||
- internal data summary
|
||||
|
||||
Tools:
|
||||
|
||||
- `internal_data_query`
|
||||
- optional `web_search`
|
||||
|
||||
Output:
|
||||
|
||||
- `SituationBrief` JSON
|
||||
- risk explanation
|
||||
- recommended actions
|
||||
- missing evidence list
|
||||
|
||||
|
||||
## Test Plan
|
||||
|
||||
Backend tests:
|
||||
|
||||
- WebSearch settings persist to `system_settings` and mask secrets in API responses.
|
||||
- Env fallback resolves provider-specific keys before `WEB_SEARCH_API_KEY`.
|
||||
- WebSearch provider normalizes success, empty results, 401, 429, and timeout responses.
|
||||
- `tutorial_refresh` uses evidence when available and marks output unverified when no evidence exists.
|
||||
- `geo_correction` returns pending review candidates and never writes final coordinates directly.
|
||||
- `situation_brief` accepts deterministic findings and returns schema-valid summaries.
|
||||
- Agent outputs fail closed when schema validation fails.
|
||||
|
||||
Frontend tests:
|
||||
|
||||
- WebSearch settings card shows configured state, masked key, connection test result, and save feedback.
|
||||
- Candidate review UI can display evidence links and pending location suggestions.
|
||||
- Situation brief UI can show evidence-backed summaries without exposing raw secrets.
|
||||
|
||||
Regression tests:
|
||||
|
||||
- existing `aiprovider` status and analysis calls remain unchanged
|
||||
- current LLM provider configuration remains the global model source
|
||||
- location pipeline tests continue to pass
|
||||
- datasource credential guide tests continue to pass
|
||||
|
||||
|
||||
## Assumptions
|
||||
|
||||
- `aiprovider` remains model-adapter-only.
|
||||
- Backend tools are implemented directly in Python first; MCP support is optional and later.
|
||||
- Search is evidence collection, not model transport.
|
||||
- Writes to important domain tables require human confirmation.
|
||||
- Deterministic analysis owns risk scores; LLM output is explanatory and evidence-backed.
|
||||
191
docs/plans/earth-motion-capture-gesture-control-plan.md
Normal file
191
docs/plans/earth-motion-capture-gesture-control-plan.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# Earth Motion Capture Gesture Control Plan
|
||||
|
||||
## Goal
|
||||
|
||||
为 Planet Earth 大屏和未来 3D 展示增加一套解耦的动作捕捉手势控制能力。实时输入分成两条路线:网页端可直接通过浏览器 `getUserMedia` 在本机识别;高级设备可继续使用本机 Motion Capture Edge Agent。两条路线都只输出轻量语义事件,客户端负责把“手势事件”映射到“具体交互函数”。
|
||||
|
||||
首版面向两颗 Logitech C1000 RGB 摄像头,但必须保持单摄像头兼容。后续任何 USB 摄像头、手机摄像头、RTSP/HTTP/WebRTC 视频源都应通过输入适配器接入,而不是改 Earth 渲染端。
|
||||
|
||||
## Architecture
|
||||
|
||||
实时链路分两种 provider,但进入 Earth 后协议一致:
|
||||
|
||||
```text
|
||||
Browser camera -> browser-local recognizer -> Motion Provider events -> Earth control functions
|
||||
Camera(s)/RTSP/HTTP -> Local Motion Capture Agent -> local WebSocket -> Motion Provider events -> Earth control functions
|
||||
```
|
||||
|
||||
关键原则:
|
||||
|
||||
- 实时控制不经过 SaaS 云端。
|
||||
- 实时控制不复用现有新闻、RSS、聚合数据接口。
|
||||
- 浏览器 provider 和 Agent provider 都不向云端上传视频帧,只输出低带宽语义事件。
|
||||
- Web/3D 客户端只消费统一事件并执行映射,不把具体输入源写进 Earth 交互逻辑。
|
||||
- 双摄首版用于冗余和稳定性,不承诺完整 3D 姿态重建。
|
||||
|
||||
## Motion Providers
|
||||
|
||||
Earth 使用统一 Motion Provider 抽象:
|
||||
|
||||
- `browser_camera`:默认 provider。使用 `getUserMedia` 获取摄像头,在浏览器本地加载 MediaPipe Tasks Vision,输出 `gesture` / `skeleton` / `status` 事件。适合 SaaS、WSL、Windows 浏览器、大屏演示和“不安装 app”的用户。
|
||||
- `motion_agent`:连接本地 Agent WebSocket。适合双摄、USB index、RTSP/HTTP 视频源、边缘设备和客户端集成。
|
||||
|
||||
设置项保存在 `planet.earth.settings.v2.shared.motionProvider`。`?motionProvider=browser` 强制浏览器摄像头,`?motionProvider=agent` 或 `?motionAgent=ws://...` 强制 Motion Agent。
|
||||
|
||||
## Motion Capture Agent
|
||||
|
||||
Agent 是本地 Edge 服务,职责包括:
|
||||
|
||||
- 读取摄像头:默认 USB index,支持单摄、双摄和未来 URL 视频源。
|
||||
- 运行识别:首版使用 OpenCV + MediaPipe;识别引擎藏在接口后,未来可替换为 ONNX、TensorRT、C++ 或 Rust worker。
|
||||
- 输出事件:通过 WebSocket 推送 `gesture`、`status`、`heartbeat`。
|
||||
- 控制节流:负责置信度阈值、防抖、冷却时间和连续手势限频。
|
||||
- 健康状态:报告摄像头数量、当前模式、识别 FPS、最近手势和错误。
|
||||
- 明确失败:缺少 CV 依赖、摄像头打不开、无可用输入时给出可读错误。
|
||||
|
||||
Python 不应成为性能瓶颈:重计算在 OpenCV/MediaPipe 原生代码中完成,Python 只做编排、状态机和事件推送。事件消息通常小于 1KB,频率不超过 20Hz。
|
||||
|
||||
## Event Protocol
|
||||
|
||||
本地默认地址:
|
||||
|
||||
```text
|
||||
ws://127.0.0.1:8765/ws/gestures
|
||||
```
|
||||
|
||||
事件类型:
|
||||
|
||||
- `gesture`
|
||||
- `status`
|
||||
- `heartbeat`
|
||||
|
||||
手势语义:
|
||||
|
||||
- `rotate_left`:左挥手,地球向左旋转。
|
||||
- `rotate_right`:右挥手,地球向右旋转。
|
||||
- `zoom_in`:双手张开,地球放大。
|
||||
- `zoom_out`:双手合拢,地球缩小。
|
||||
- `confirm`:握拳或确认动作,触发当前交互确认。
|
||||
|
||||
最小事件字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "gesture",
|
||||
"gesture": "rotate_left",
|
||||
"phase": "discrete",
|
||||
"confidence": 0.92,
|
||||
"intensity": 0.8,
|
||||
"timestamp_ms": 1770000000000,
|
||||
"seq": 42,
|
||||
"source": "motion-agent",
|
||||
"mode": "single",
|
||||
"payload": {}
|
||||
}
|
||||
```
|
||||
|
||||
## Earth Client Integration
|
||||
|
||||
Earth 前端新增 motion-control adapter:
|
||||
|
||||
- 连接本地 Agent WebSocket。
|
||||
- 处理断线、重连、心跳和状态。
|
||||
- 过滤低置信度事件。
|
||||
- 将手势映射到 Earth 控制函数。
|
||||
- Agent 离线时不影响普通鼠标、触摸、巡航和图层交互。
|
||||
|
||||
Earth 端只暴露最小动作入口:
|
||||
|
||||
- `applyMotionRotate(direction, intensity)`
|
||||
- `applyMotionZoom(direction, intensity)`
|
||||
- `applyMotionConfirm()`
|
||||
|
||||
动作捕捉不直接操作 Three.js 内部对象,也不修改图层业务模块。
|
||||
|
||||
## SaaS Strategy
|
||||
|
||||
未来网页端做成 SaaS 后,默认实时手势链路仍在浏览器本地完成,不走云端 RPC。高级现场设备可选本地 Agent:
|
||||
|
||||
```text
|
||||
Browser SaaS page -> getUserMedia -> browser-local recognizer
|
||||
Browser SaaS page -> local secure bridge -> Local Motion Capture Agent (advanced)
|
||||
Cloud SaaS -> config/auth/status only
|
||||
```
|
||||
|
||||
原因:
|
||||
|
||||
- 云端 RPC 会增加网络 RTT 和抖动。
|
||||
- 上传摄像头帧有隐私和带宽风险。
|
||||
- 大屏交互需要稳定体感延迟,云端只适合做配置、授权、设备状态和审计。
|
||||
|
||||
浏览器摄像头要求 HTTPS 或 localhost。Agent 模式在本地部署可使用 `ws://127.0.0.1:8765`;生产 HTTPS SaaS 若要接 Agent,需要补 `wss://127.0.0.1` 或等价本地安全桥接,避免浏览器混合内容限制。
|
||||
|
||||
## Latency Budget
|
||||
|
||||
目标体感延迟:
|
||||
|
||||
- 摄像头采集:16-33ms。
|
||||
- 识别:8-25ms。
|
||||
- 状态机:小于 2ms。
|
||||
- 本地 WebSocket:1-5ms。
|
||||
- 浏览器渲染:约 16ms。
|
||||
|
||||
实验室目标:从动作被识别到 Earth 响应 p95 小于 50ms;摄像头到画面响应端到端小于 120ms。
|
||||
|
||||
## Implementation Milestones
|
||||
|
||||
1. 保存本计划并注册到 `docs/plans/README.md`。
|
||||
2. 新增独立 motion agent 包,提供 CLI、配置、摄像头输入抽象、事件模型和 WebSocket server。
|
||||
3. 新增手势状态机,支持阈值、防抖、冷却和限频。
|
||||
4. 新增 Earth motion-control provider manager,默认接浏览器摄像头 provider,可切换到 Motion Agent provider。
|
||||
5. 增加 Agent 单元测试、协议测试和前端 adapter 静态验证。
|
||||
6. 更新中英文用户手册和 Earth 前端开发上下文。
|
||||
|
||||
## Debug Mode Addition
|
||||
|
||||
**当前状态**:Browser Camera provider 会在调试面板中显示本地 `<video>` 预览并叠加骨架;`只显示骨骼` 可关闭视频底图。Motion Agent provider 仍只发送 `skeleton` 事件,不传原始摄像头帧。
|
||||
|
||||
Earth 设置中增加“动捕调试模式” switch,并增加“动捕输入源”选择。开启后,Earth 会启动当前 provider 并显示独立 HUD 调试面板。Browser Camera 模式下调试面板可以显示本机浏览器视频预览;Motion Agent 模式下只画归一化骨架点和关节连线,不传原始摄像头画面。
|
||||
|
||||
Motion Agent 增加 `skeleton` 事件:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "skeleton",
|
||||
"camera_id": "usb:0",
|
||||
"matched_gesture": "rotate_left",
|
||||
"confidence": 0.91,
|
||||
"joints": [{ "id": "left_wrist", "x": 0.42, "y": 0.61, "confidence": 0.98 }],
|
||||
"bones": [["left_shoulder", "left_elbow"]]
|
||||
}
|
||||
```
|
||||
|
||||
调试颜色约定:
|
||||
|
||||
- 未匹配动作:红色骨架。
|
||||
- 已匹配动作:绿色骨架,并显示匹配到的动作名。
|
||||
|
||||
权限先预留 `data-gatekeeper-permission="earth.motion_debug"` 标记,后续由 Gatekeeper 决定 switch 是否可见/可用。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Agent 单元测试:
|
||||
- 事件模型可序列化。
|
||||
- 低置信度手势被忽略。
|
||||
- 冷却期内重复手势被忽略。
|
||||
- 冷却后新手势可再次输出。
|
||||
- 无摄像头/缺依赖时错误可读。
|
||||
- Agent 协议测试:
|
||||
- `gesture`、`status`、`heartbeat` 字段稳定。
|
||||
- WebSocket 广播只发送语义事件。
|
||||
- Earth 前端验证:
|
||||
- motion-control provider manager 能消费浏览器 provider 和 Agent provider 的 mock 消息。
|
||||
- browser provider 在 mock `getUserMedia` 成功时进入 active 状态。
|
||||
- browser provider 在权限拒绝、无摄像头或非安全上下文时给出可读错误。
|
||||
- `skeleton` 事件能触发 `earth:motion-debug-frame`。
|
||||
- Agent 离线时不抛异常。
|
||||
- `rotate_left/right`、`zoom_in/out`、`confirm` 映射到 Earth 动作函数。
|
||||
- 文档验证:
|
||||
- 计划文档存在。
|
||||
- `docs/plans/README.md` 有入口。
|
||||
- 中英文使用说明不互相矛盾。
|
||||
66
docs/plans/earth-motion-gesture-interaction-v2-plan.md
Normal file
66
docs/plans/earth-motion-gesture-interaction-v2-plan.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Earth Motion Gesture Interaction V2 Plan
|
||||
|
||||
**状态**:已实现主体交互,并按实测调整。当前浏览器识别保留右手导航、头部切目标、左手上下切动捕图层、双手张开/收拢缩放;双手上举确认暂时关闭。Motion 目标展示已改为 `CruiseSequencer` + `PresentationController` 的 persistent 展示。
|
||||
|
||||
## Summary
|
||||
|
||||
把动捕从“几个单点手势触发函数”升级为一套更像大屏遥控器的交互层:右手负责地球导航,头部负责候选切换,左手上下切换动捕候选图层,双手负责缩放,调试面板支持“只显示骨骼”和暂停匹配。进入动捕模式后,Earth 自动软选中屏幕中心附近的正面可交互目标;确认动作预留为把目标升级为锁定,并用巡航/引导线式详情打开,不再模拟鼠标点击。
|
||||
|
||||
## Key Changes
|
||||
|
||||
- 手势语义 v1 固定为稳健小集:
|
||||
- 修正当前左右挥手语义反向问题:手势名以用户感知方向为准,provider 层输出正确 `rotate_left` / `rotate_right`。
|
||||
- 右手左/右/上/下挥控制地球水平/垂直旋转,新增 `rotate_up`、`rotate_down`。
|
||||
- 双手张开/靠近明确映射为 `zoom_in` / `zoom_out`。
|
||||
- 头往左/右歪新增 `focus_prev` / `focus_next`,在当前自动候选目标之间切换。
|
||||
- 左手上/下挥新增 `layer_prev` / `layer_next`,切换当前动捕候选图层并聚焦该图层最近目标。
|
||||
- 双手确认手势暂时关闭,避免与缩放和站姿误触混淆;协议仍保留 `confirm`。
|
||||
|
||||
- Motion Provider / Protocol:
|
||||
- 扩展 `MOTION_GESTURES`,新增 `rotate_up`、`rotate_down`、`focus_prev`、`focus_next`、`layer_prev`、`layer_next`。
|
||||
- Browser Camera provider 扩展 pose joints,保留肩/肘/腕,增加头部关键点,用于判断头歪。
|
||||
- 右手作为导航手;左手独立控制动捕候选图层。
|
||||
- 每类手势使用独立阈值和 cooldown,避免缩放/确认/旋转互相误触。
|
||||
|
||||
- Earth 交互层:
|
||||
- Motion adapter 支持水平/垂直旋转和 focus 切换 callback。
|
||||
- 进入动捕模式后,周期性从可交互对象中选出屏幕中心最近、位于地球正面的候选。
|
||||
- 软选中目标独立于 `lockedObject`,用 hover/linked 视觉态展示,不立即打开详情。
|
||||
- `focus_prev` / `focus_next` 在候选列表中切换;列表按屏幕中心距离、正面可见性、当前图层可见性排序。
|
||||
- `confirm` 预留为将软选中目标升级为 locked,并打开引导线详情;若没有候选,显示状态提示。
|
||||
|
||||
- 调试面板:
|
||||
- 在动捕 HUD / drawer 内增加“只显示骨骼”开关。
|
||||
- 增加“停止匹配动作”开关:暂停 gesture 执行,但不关闭摄像头预览或骨架绘制。
|
||||
- 设置持久化到 `planet.earth.settings.v2.shared.motionDebugSkeletonOnly`。
|
||||
- 开启后 canvas 不绘制视频帧,只绘制深色背景 + 红/绿骨骼线;摄像头仍继续用于识别。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Browser provider 单元测试:
|
||||
- 右手左/右挥输出的 `rotate_left` / `rotate_right` 与用户语义一致。
|
||||
- 右手上/下挥输出 `rotate_up` / `rotate_down`。
|
||||
- 双手张开输出 `zoom_in`,双手靠近输出 `zoom_out`。
|
||||
- 头部左右倾斜输出 `focus_prev` / `focus_next`。
|
||||
- 双手确认动作暂时不会触发。
|
||||
|
||||
- Motion adapter 测试:
|
||||
- 新增 gesture 能通过 `normalizeGestureMessage`。
|
||||
- `rotate_up/down` 调用垂直旋转逻辑。
|
||||
- `focus_prev/focus_next` 调用候选切换 callback。
|
||||
- `confirm` 在协议层保持兼容;浏览器 provider 当前不主动发出。
|
||||
|
||||
- Earth 前端验证:
|
||||
- 开启动捕模式后,屏幕中心附近正面目标自动软选中。
|
||||
- 头歪能在候选之间切换。
|
||||
- 左手上下切换图层后会在新图层中选择最近目标并展示 persistent 引导线详情。
|
||||
- 右手上下挥能旋转到南北方向目标。
|
||||
- “只显示骨骼”开关持久化,刷新后状态保持。
|
||||
- `bun --check` 覆盖新增/修改 Earth JS 模块,现有 motion tests 全绿。
|
||||
|
||||
## Assumptions
|
||||
|
||||
- v1 采用“右手导航、头部切候选、左手切图层、双手缩放”的交互模型;确认手势保留协议但暂时关闭浏览器识别。
|
||||
- 自动选中是 soft focus,不覆盖现有 mouse locked selection;只有 `confirm` 才真正锁定目标。
|
||||
- 骨骼-only 只影响调试画面,不关闭摄像头、不影响识别。
|
||||
- Motion Agent 协议可以接收新增 gesture 名;旧 agent 只发旧 gesture 时仍兼容。
|
||||
67
docs/plans/earth-presentation-decoupled-architecture-plan.md
Normal file
67
docs/plans/earth-presentation-decoupled-architecture-plan.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Earth Presentation Decoupled Architecture Plan
|
||||
|
||||
## Goal
|
||||
|
||||
把 Earth 页面里的“详情卡片、连接器、隐藏策略、跟随更新”从具体业务交互里拆出来,形成统一的 Presentation 层。第一阶段只迁 Motion 动捕展示,修复卡片被鼠标移动误隐藏、连接器 interactable 端不贴合本体的问题;BGP/News 巡航保持现状,避免改变原有轮播体验。
|
||||
|
||||
## Current Issues
|
||||
|
||||
- Motion 展示复用了巡航卡片,但隐藏判断仍散落在 `main.js` 的 hover/mousemove 分支里,导致鼠标移动时卡片可能被 `hideInfoCard()` 清掉。
|
||||
- Motion 连接器 source 端目前主要使用屏幕点坐标,缺少本体视觉边界,线无法稳定贴住 marker、卫星或海缆本体。
|
||||
- 卡片、连接器、目标本体和生命周期策略耦合在 adapter 内,不利于后续把点击详情、动捕、巡航统一管理。
|
||||
|
||||
## Phase 1 Scope
|
||||
|
||||
- 新增 `PresentationController`。
|
||||
- Motion 使用 `PresentationController` 管理卡片、连接器和 persistent 生命周期。
|
||||
- BGP/News adapter 不迁移,继续使用现有 `CruiseSequencer`、卡片位置、线动画和 dwell/advance 行为。
|
||||
- InfoCard 和 CalloutConnector 继续作为底层 renderer,不重写 UI。
|
||||
|
||||
## Presentation Interface
|
||||
|
||||
`PresentationController.present(request)` 接收:
|
||||
|
||||
- `id`: presentation 唯一 id。
|
||||
- `owner`: `motion | cruise | click | hover`。
|
||||
- `card`: 提供 `render({ reveal })` 和 `hide()`。
|
||||
- `connector`: 提供 `sourceProvider`、`targetProvider`、`options`,由 controller 调用 `createConnectorPath()` 和 `connector.render()`。
|
||||
- `lifetime`: `persistent | timeout | sequenced`,Motion 默认 `persistent`。
|
||||
- `onDismiss(reason)`: 替换、关闭、停止等清理回调。
|
||||
|
||||
`PresentationController.update()` 每帧重算 active connector 的 source/target anchor。`dismiss(reason)` 统一清理卡片、连接器和计时器。
|
||||
|
||||
## Motion Integration
|
||||
|
||||
- Motion adapter 不再直接管理 `showInfoCard + connector.render + hideInfoCard`。
|
||||
- Motion request 使用 `owner: "motion"` 和 `lifetime: { mode: "persistent" }`。
|
||||
- Motion 切目标时替换当前 presentation。
|
||||
- Motion 关闭、页面销毁或用户关闭展示时 dismiss。
|
||||
- Motion source anchor 使用视觉近似矩形:
|
||||
- BGP / compute / vessel marker: 投影中心 + marker 尺寸近似。
|
||||
- satellite: 当前卫星位置 + point size 近似。
|
||||
- cable: localCenter + 小矩形近似。
|
||||
|
||||
## Cruise Compatibility
|
||||
|
||||
- BGP/News 第一阶段不迁移。
|
||||
- `CruiseSequencer` 的 `auto_advance` 不改。
|
||||
- 原巡航的 dwell、hide、advance、卡片固定锚点、连接器动画时序不改。
|
||||
- 后续迁移 BGP/News 前必须先补回归测试,再只替换渲染层,不改排序、聚焦和时序。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- `presentation-controller.test.js`
|
||||
- `persistent` 不自动隐藏。
|
||||
- `timeout` 按配置隐藏。
|
||||
- 新 presentation 替换旧 presentation,并触发旧 `onDismiss("replace")`。
|
||||
- `dismiss(reason)` 清理卡片、连接器、计时器。
|
||||
- `update()` 重新获取 source/target anchor 并重绘 connector。
|
||||
- Motion 手动验证:
|
||||
- Motion 展示后移动鼠标,卡片不消失。
|
||||
- Motion 切目标后旧卡片和旧线被替换。
|
||||
- 卡片拖动、窗口 resize、地球旋转、卫星移动时 connector 两端跟随。
|
||||
- source 端贴近 interactable 视觉边缘。
|
||||
- 巡航回归:
|
||||
- BGP/News 自动轮播、dwell、隐藏、进入下一条不变。
|
||||
- 移动端 popup/drawer 行为不变。
|
||||
|
||||
Reference in New Issue
Block a user