release: bump version to 0.71.0

2026-06-11 16:47:24 +08:00
parent 8c204717cd
commit 899e3bce43
56 changed files with 4618 additions and 260 deletions
--- a/docs/plans/README.md
+++ b/docs/plans/README.md
@@ -33,6 +33,7 @@
 - [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
 - [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
 - [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
+- [Motion Agent v2 控制协议与 3D 标定路线](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md)
 - [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
 - [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
 - [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
--- a/docs/plans/earth-motion-capture-gesture-control-plan.md
+++ b/docs/plans/earth-motion-capture-gesture-control-plan.md
@@ -1,5 +1,7 @@
 # Earth Motion Capture Gesture Control Plan

+> Update: Motion Agent process/device control, UE/Web shared command protocol, dual-camera redundant fusion, and the next 3D calibration route are now tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md). This document remains useful for the original provider split and gesture-control intent.
+
 ## Goal

 为 Planet Earth 大屏和未来 3D 展示增加一套解耦的动作捕捉手势控制能力。实时输入分成两条路线：网页端可直接通过浏览器 `getUserMedia` 在本机识别；高级设备可继续使用本机 Motion Capture Edge Agent。两条路线都只输出轻量语义事件，客户端负责把“手势事件”映射到“具体交互函数”。
--- a/docs/plans/earth-motion-gesture-interaction-v2-plan.md
+++ b/docs/plans/earth-motion-gesture-interaction-v2-plan.md
@@ -2,6 +2,8 @@

 **状态**：已实现主体交互，并按实测调整。当前浏览器识别保留右手导航、头部切目标、左手上下切动捕图层、双手张开/收拢缩放；双手上举确认暂时关闭。Motion 目标展示已改为 `CruiseSequencer` + `PresentationController` 的 persistent 展示。

+> Update: Agent-side bidirectional commands, UE/Web shared device control, dual-camera redundant fusion, and future calibrated 3D mode are tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
+
 ## Summary

 把动捕从“几个单点手势触发函数”升级为一套更像大屏遥控器的交互层：右手负责地球导航，头部负责候选切换，左手上下切换动捕候选图层，双手负责缩放，调试面板支持“只显示骨骼”和暂停匹配。进入动捕模式后，Earth 自动软选中屏幕中心附近的正面可交互目标；确认动作预留为把目标升级为锁定，并用巡航/引导线式详情打开，不再模拟鼠标点击。
--- a/docs/plans/motion-agent-v2-control-protocol-plan.md
+++ b/docs/plans/motion-agent-v2-control-protocol-plan.md
@@ -0,0 +1,175 @@
+# Motion Agent v2 Control Protocol And 3D Calibration Roadmap
+
+## Summary
+
+Motion Agent v2 turns the local motion service into a shared WebSocket control plane for the Web Earth page and UE clients. The agent process is still started externally through `planet.sh`, a desktop service, or UE process management. Once the process is running, clients can open cameras, close cameras, switch input mode, arm or pause recognition, and inspect status through the same bidirectional WebSocket protocol.
+
+This phase implements robust single-camera and dual-camera redundant fusion. True calibrated 3D skeleton reconstruction is deliberately reserved for the v3 calibration phase.
+
+## v2 Protocol
+
+The default endpoint remains:
+
+```text
+ws://127.0.0.1:8765/ws/gestures
+```
+
+The agent emits:
+
+- `gesture`
+- `skeleton`
+- `status`
+- `heartbeat`
+- `command_result`
+
+Clients send:
+
+- `open_devices`
+- `close_devices`
+- `rescan_devices`
+- `set_armed`
+- `set_paused`
+- `set_input_mode`
+- `set_camera_config`
+- `set_fusion_config`
+- `set_debug_options`
+- `set_enabled_gestures`
+- `get_status`
+- `ping`
+
+All v2 messages include additive compatibility fields such as `protocol_version`, `request_id`, `camera_id`, `input_mode`, and optional `fusion`. Older push-only clients can continue to consume `gesture`, `skeleton`, `status`, and `heartbeat`.
+
+Example command:
+
+```json
+{
+  "type": "command",
+  "command": "open_devices",
+  "request_id": "req-001",
+  "payload": {
+    "input_mode": "dual_redundant",
+    "camera_indexes": [0, 1]
+  }
+}
+```
+
+Example gesture whitelist command:
+
+```json
+{
+  "type": "command",
+  "command": "set_enabled_gestures",
+  "request_id": "earth-motion-enabled-gestures",
+  "payload": {
+    "gestures": ["rotate_left", "rotate_right", "zoom_in", "zoom_out", "confirm"]
+  }
+}
+```
+
+Example result:
+
+```json
+{
+  "type": "command_result",
+  "protocol_version": "motion.v2",
+  "request_id": "req-001",
+  "command": "open_devices",
+  "ok": true,
+  "status": {
+    "armed": false,
+    "paused": false,
+    "input_mode": "dual_redundant",
+    "devices_open": true,
+    "active_camera_ids": ["usb:0", "usb:1"]
+  }
+}
+```
+
+## Device And Control State
+
+Process startup is not a WebSocket feature: the server must exist before a WebSocket client can connect. UE should start the agent as an external process or depend on a system service, then connect to the WebSocket endpoint.
+
+Device wake and control wake are WebSocket features:
+
+- `open_devices` opens USB cameras or URL cameras.
+- `close_devices` releases them.
+- `set_armed` enables gesture execution.
+- `set_paused` pauses recognition without closing the connection.
+
+When `armed=false`, the agent may still emit skeleton and status events, but it does not emit actionable gesture events.
+
+The Earth settings dialog owns a user-facing gesture whitelist. Unchecked gestures are ignored in the Earth client and are also sent to the Motion Agent through `set_enabled_gestures`, so the server does not broadcast disabled actions to UE/Web consumers. The default whitelist enables the full v2 gesture set; disabling gestures is a local display/control preference and does not change the installed recognition model.
+
+## Input Modes
+
+- `single`: one camera.
+- `dual_redundant`: two or more cameras observe the same gesture. Matching observations in a short window are fused into a higher-confidence event.
+- `single_fallback`: the primary camera is preferred and a secondary input is used as fallback.
+- `calibrated_3d`: reserved for v3 and should not be enabled unless a calibration profile exists.
+
+The v2 dual-camera mode is redundant fusion, not 3D reconstruction. It is meant to improve reliability under occlusion and camera noise without requiring calibration.
+
+## v3 3D Calibration Roadmap
+
+The next phase is `Motion Agent v3 3D Calibration`. It upgrades from redundant fusion to calibrated multi-camera skeleton fusion.
+
+Planned capabilities:
+
+- Camera intrinsics: focal length, distortion, resolution.
+- Camera extrinsics: relative position, rotation, and baseline distance.
+- Calibration workflow: checkerboard, AprilTag, or ArUco board.
+- Local calibration profile JSON with query, load, reset, and validation commands.
+- `skeleton_3d` event with world-space joints, confidence, and source cameras.
+- UE coordinate mapping from Motion Agent coordinates to UE world or widget coordinates.
+
+Reserved v3 input configuration:
+
+```json
+{
+  "input_mode": "calibrated_3d",
+  "calibration_profile": "desk-dual-camera-v1"
+}
+```
+
+Reserved v3 event:
+
+```json
+{
+  "type": "skeleton_3d",
+  "protocol_version": "motion.v3",
+  "profile": "desk-dual-camera-v1",
+  "joints": [
+    {
+      "name": "right_wrist",
+      "x": 0.42,
+      "y": 1.13,
+      "z": 0.76,
+      "confidence": 0.91
+    }
+  ]
+}
+```
+
+## Test Plan
+
+- Command/result roundtrip for device open, close, rescan, armed, paused, and status.
+- Gesture protocol accepts the full Earth v2 gesture set.
+- Earth settings can disable individual gestures; disabled gestures are ignored locally and filtered server-side through `set_enabled_gestures`.
+- Motion Agent `status` reports the current enabled gesture list for Web/UE diagnostics.
+- Single and dual redundant fusion emit compatible gesture payloads.
+- Conflicting dual-camera observations below the confidence delta are ignored.
+- Web Earth `motion-agent-provider` can send commands over the same socket it uses for events.
+- UE mock clients can operate the service without browser-only assumptions.
+- Dry-run mode can test commands, status, skeleton, and fusion behavior without camera dependencies.
+
+## Current Limitations
+
+- Production recognition uses a real MediaPipe pose pipeline and heuristic gesture recognizer. It still needs environment-specific threshold tuning, camera framing validation, and long-running reliability checks before it can be treated as calibration-free.
+- Dual-camera v2 does not triangulate 3D joint positions.
+- `calibrated_3d` is documented as a reserved mode and must not be treated as implemented until v3 lands.
+
+## Implementation Status
+
+- Implemented: bidirectional command/result protocol, device lifecycle controls, dry-run mode, subprocess recognition worker, MediaPipe pose recognition, gesture whitelist, single-camera mode, dual-redundant/fallback scaffolding, Web client integration, and default `planet.sh` lifecycle integration.
+- Remaining v2 hardening: tune recognition thresholds across camera placements, exercise UE command/control integration, and run longer soak tests for device reconnect and dual-camera conflicts.
+- Planned v3: calibrated multi-camera 3D skeleton fusion and Motion Agent-to-UE coordinate calibration.