release: bump version to 0.71.0
This commit is contained in:
linkong
175
docs/plans/motion-agent-v2-control-protocol-plan.md
Normal file
175
docs/plans/motion-agent-v2-control-protocol-plan.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# Motion Agent v2 Control Protocol And 3D Calibration Roadmap
|
||||
|
||||
## Summary
|
||||
|
||||
Motion Agent v2 turns the local motion service into a shared WebSocket control plane for the Web Earth page and UE clients. The agent process is still started externally through `planet.sh`, a desktop service, or UE process management. Once the process is running, clients can open cameras, close cameras, switch input mode, arm or pause recognition, and inspect status through the same bidirectional WebSocket protocol.
|
||||
|
||||
This phase implements robust single-camera and dual-camera redundant fusion. True calibrated 3D skeleton reconstruction is deliberately reserved for the v3 calibration phase.
|
||||
|
||||
## v2 Protocol
|
||||
|
||||
The default endpoint remains:
|
||||
|
||||
```text
|
||||
ws://127.0.0.1:8765/ws/gestures
|
||||
```
|
||||
|
||||
The agent emits:
|
||||
|
||||
- `gesture`
|
||||
- `skeleton`
|
||||
- `status`
|
||||
- `heartbeat`
|
||||
- `command_result`
|
||||
|
||||
Clients send:
|
||||
|
||||
- `open_devices`
|
||||
- `close_devices`
|
||||
- `rescan_devices`
|
||||
- `set_armed`
|
||||
- `set_paused`
|
||||
- `set_input_mode`
|
||||
- `set_camera_config`
|
||||
- `set_fusion_config`
|
||||
- `set_debug_options`
|
||||
- `set_enabled_gestures`
|
||||
- `get_status`
|
||||
- `ping`
|
||||
|
||||
All v2 messages include additive compatibility fields such as `protocol_version`, `request_id`, `camera_id`, `input_mode`, and optional `fusion`. Older push-only clients can continue to consume `gesture`, `skeleton`, `status`, and `heartbeat`.
|
||||
|
||||
Example command:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "command",
|
||||
"command": "open_devices",
|
||||
"request_id": "req-001",
|
||||
"payload": {
|
||||
"input_mode": "dual_redundant",
|
||||
"camera_indexes": [0, 1]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Example gesture whitelist command:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "command",
|
||||
"command": "set_enabled_gestures",
|
||||
"request_id": "earth-motion-enabled-gestures",
|
||||
"payload": {
|
||||
"gestures": ["rotate_left", "rotate_right", "zoom_in", "zoom_out", "confirm"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Example result:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "command_result",
|
||||
"protocol_version": "motion.v2",
|
||||
"request_id": "req-001",
|
||||
"command": "open_devices",
|
||||
"ok": true,
|
||||
"status": {
|
||||
"armed": false,
|
||||
"paused": false,
|
||||
"input_mode": "dual_redundant",
|
||||
"devices_open": true,
|
||||
"active_camera_ids": ["usb:0", "usb:1"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Device And Control State
|
||||
|
||||
Process startup is not a WebSocket feature: the server must exist before a WebSocket client can connect. UE should start the agent as an external process or depend on a system service, then connect to the WebSocket endpoint.
|
||||
|
||||
Device wake and control wake are WebSocket features:
|
||||
|
||||
- `open_devices` opens USB cameras or URL cameras.
|
||||
- `close_devices` releases them.
|
||||
- `set_armed` enables gesture execution.
|
||||
- `set_paused` pauses recognition without closing the connection.
|
||||
|
||||
When `armed=false`, the agent may still emit skeleton and status events, but it does not emit actionable gesture events.
|
||||
|
||||
The Earth settings dialog owns a user-facing gesture whitelist. Unchecked gestures are ignored in the Earth client and are also sent to the Motion Agent through `set_enabled_gestures`, so the server does not broadcast disabled actions to UE/Web consumers. The default whitelist enables the full v2 gesture set; disabling gestures is a local display/control preference and does not change the installed recognition model.
|
||||
|
||||
## Input Modes
|
||||
|
||||
- `single`: one camera.
|
||||
- `dual_redundant`: two or more cameras observe the same gesture. Matching observations in a short window are fused into a higher-confidence event.
|
||||
- `single_fallback`: the primary camera is preferred and a secondary input is used as fallback.
|
||||
- `calibrated_3d`: reserved for v3 and should not be enabled unless a calibration profile exists.
|
||||
|
||||
The v2 dual-camera mode is redundant fusion, not 3D reconstruction. It is meant to improve reliability under occlusion and camera noise without requiring calibration.
|
||||
|
||||
## v3 3D Calibration Roadmap
|
||||
|
||||
The next phase is `Motion Agent v3 3D Calibration`. It upgrades from redundant fusion to calibrated multi-camera skeleton fusion.
|
||||
|
||||
Planned capabilities:
|
||||
|
||||
- Camera intrinsics: focal length, distortion, resolution.
|
||||
- Camera extrinsics: relative position, rotation, and baseline distance.
|
||||
- Calibration workflow: checkerboard, AprilTag, or ArUco board.
|
||||
- Local calibration profile JSON with query, load, reset, and validation commands.
|
||||
- `skeleton_3d` event with world-space joints, confidence, and source cameras.
|
||||
- UE coordinate mapping from Motion Agent coordinates to UE world or widget coordinates.
|
||||
|
||||
Reserved v3 input configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"input_mode": "calibrated_3d",
|
||||
"calibration_profile": "desk-dual-camera-v1"
|
||||
}
|
||||
```
|
||||
|
||||
Reserved v3 event:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "skeleton_3d",
|
||||
"protocol_version": "motion.v3",
|
||||
"profile": "desk-dual-camera-v1",
|
||||
"joints": [
|
||||
{
|
||||
"name": "right_wrist",
|
||||
"x": 0.42,
|
||||
"y": 1.13,
|
||||
"z": 0.76,
|
||||
"confidence": 0.91
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Command/result roundtrip for device open, close, rescan, armed, paused, and status.
|
||||
- Gesture protocol accepts the full Earth v2 gesture set.
|
||||
- Earth settings can disable individual gestures; disabled gestures are ignored locally and filtered server-side through `set_enabled_gestures`.
|
||||
- Motion Agent `status` reports the current enabled gesture list for Web/UE diagnostics.
|
||||
- Single and dual redundant fusion emit compatible gesture payloads.
|
||||
- Conflicting dual-camera observations below the confidence delta are ignored.
|
||||
- Web Earth `motion-agent-provider` can send commands over the same socket it uses for events.
|
||||
- UE mock clients can operate the service without browser-only assumptions.
|
||||
- Dry-run mode can test commands, status, skeleton, and fusion behavior without camera dependencies.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- Production recognition uses a real MediaPipe pose pipeline and heuristic gesture recognizer. It still needs environment-specific threshold tuning, camera framing validation, and long-running reliability checks before it can be treated as calibration-free.
|
||||
- Dual-camera v2 does not triangulate 3D joint positions.
|
||||
- `calibrated_3d` is documented as a reserved mode and must not be treated as implemented until v3 lands.
|
||||
|
||||
## Implementation Status
|
||||
|
||||
- Implemented: bidirectional command/result protocol, device lifecycle controls, dry-run mode, subprocess recognition worker, MediaPipe pose recognition, gesture whitelist, single-camera mode, dual-redundant/fallback scaffolding, Web client integration, and default `planet.sh` lifecycle integration.
|
||||
- Remaining v2 hardening: tune recognition thresholds across camera placements, exercise UE command/control integration, and run longer soak tests for device reconnect and dual-camera conflicts.
|
||||
- Planned v3: calibrated multi-camera 3D skeleton fusion and Motion Agent-to-UE coordinate calibration.
|
||||
Reference in New Issue
Block a user