planet/docs/technical/en/ops-planet-sh-startup.md

# `planet.sh` Startup Performance Optimization

## Background

Use `zsh ./planet.sh start --non-motion-agent` for daily startup; use `init` when preparing a new environment. When investigating latency, distinguish initial dependency downloads, container readiness, and application initialization using the stage timestamps.

Before preparing the AI Provider image, startup verifies the backend's actual database connection and recreates a missing port mapping once while preserving the volume. A terminated backend process or Uvicorn initialization, ASGI loading, import, or syntax failure stops waiting and identical retries immediately. Normally slow initialization keeps its existing timeout budget. AI Provider readiness probes the host `/health` endpoint directly, without waiting for Docker's first scheduled health check.

`planet.sh` manages start, stop, restart, health checks, and logs for all local services. The previous implementation had several startup issues:

1. AI Provider rebuilt every time, even when code had not changed.
2. Port cleanup could wait up to 45 seconds.
3. Port bind detection used a Python subprocess, adding about 300 ms per call.
4. Plain `restart` and `restart -b` behaved differently.

## Issue 1: AI Provider Rebuilt Every Time

### Root Cause

The build stamp file lived under `/tmp/`. After WSL or Linux restart, `/tmp` is cleared, so the `stamp_non_empty` condition failed and the script decided to rebuild:

```bash
# All three conditions had to be true to skip rebuild
image_exists AND stamp_non_empty AND fingerprint_match
```

### Fix

The stamp file moved from a temporary location to a persistent cache path:

```bash
AI_PROVIDER_BUILD_STAMP_FILE="${XDG_CACHE_HOME:-$HOME/.cache}/planet/aiprovider_build.sha256"
```

Writing the stamp creates the directory first:

```bash
write_ai_provider_build_stamp() {
    mkdir -p "$(dirname "$AI_PROVIDER_BUILD_STAMP_FILE")"
    compute_ai_provider_build_fingerprint > "$AI_PROVIDER_BUILD_STAMP_FILE"
}
```

### Fingerprint Scope

The fingerprint hashes file contents from `aiprovider/`, the Dockerfile, the root manifest and lockfile, and the provider dependency information. It does not traverse frontend assets or downloaded data. `.env` and `.env.*` are excluded because they are runtime configuration. The Dockerfile applies its fingerprint label after dependency installation so a changed build marker alone does not invalidate dependency layers.

### Docker Build Context

AI Provider only needs root `pyproject.toml`, `uv.lock`, and `aiprovider/` source code. Sending the entire repository as Docker build context wastes time on frontend assets, PDFs, historical data, and Unreal files.

The root `.dockerignore` now narrows the context:

```dockerignore
**

!pyproject.toml
!uv.lock
!aiprovider/
!aiprovider/**

aiprovider/.env
aiprovider/.env.*
!aiprovider/.env.example
```

The Dockerfile copies only AI Provider inputs:

```dockerfile
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --only-group aiprovider

COPY aiprovider /app/aiprovider
```

`uv sync` uses a BuildKit cache mount. The first build may still depend on network speed, but later builds reuse `/root/.cache/uv`.

The `aiprovider` dependency group in the root `pyproject.toml` uses the same `uv.lock` and installs only the API, HTTP client, settings, and ASGI runtime dependencies. The image excludes backend collectors and OpenCV / MediaPipe motion dependencies. The build fingerprint label comes after dependency installation and code copying, so a fingerprint change alone does not invalidate dependency layers. The container starts the installed `.venv/bin/python` directly, without runtime dependency synchronization. Dependency changes must update this group and the lockfile and validate image imports and `/health`.

### Runtime Configuration

Before starting AI Provider, `planet.sh` generates a current-user runtime env-file and passes it to Compose or the manual `docker run` fallback. The default path is `${XDG_STATE_HOME:-$HOME/.local/state}/planet/aiprovider_runtime.env`. Configuration priority:

1. `aiprovider/.env`
2. simple `export AI_...=...` or `AI_...=...` lines from `~/.zshrc`

The default parser is static and only covers AI Provider, image, and proxy variables. It avoids executing interactive shell initialization. Complex shell expansion can be enabled explicitly:

```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```

To ignore personal shell config during debugging:

```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```

### Skip-Rebuild Behavior

When the fingerprint matches, the script skips `docker compose build` and starts the existing container:

```bash
docker start planet_aiprovider
```

`docker stop` stops the container without deleting the image. `start` and `restart` retain stopped containers instead of scanning and deleting all exited containers on the host. An unchanged AI Provider can be reused, while Compose still reconciles database configuration. Existing recreation paths remain responsible for image or configuration changes.

## Issue 2: Slow Port Cleanup

### Cause

`wait_for_port_release` could wait up to 45 seconds by default: 15 attempts times 3 seconds.

### Fix

Background process cleanup now uses a 3-second timeout: TERM, 1.5 seconds, KILL, 1.5 seconds.

```bash
PORT_RELEASE_ATTEMPTS=15
PORT_RELEASE_INTERVAL=0.2

wait_for_port_release "$port" 15 0.2
```

`wait_for_port_release` accepts optional parameters so different situations can choose different timeouts.

## Issue 3: Port Detection Used Python

### Cause

`can_bind_port` used `python3 -c "import socket..."`; each call cost about 300 ms.

### Fix

Prefer system tools and keep Python as a fallback:

```bash
can_bind_port() {
    local port="$1"
    if command -v ss >/dev/null 2>&1; then
        ! ss -tlnH 2>/dev/null | awk '{print $4}' | grep -qE ":${port}$"
        return
    fi
    if command -v lsof >/dev/null 2>&1; then
        [ -z "$(lsof -tiTCP:"${port}" -sTCP:LISTEN 2>/dev/null)" ]
        return
    fi
    python3 - "$port" <<'PY'
import sys, socket
p = int(sys.argv[1])
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
    s.bind(("", p)); s.close(); sys.exit(0)
except OSError:
    sys.exit(1)
PY
}
```

Frontend startup now has an additional pre-start cleanup retry layer:

- `PORT_PRESTART_RETRIES`: defaults to 3 attempts.
- `PORT_PRESTART_RETRY_INTERVAL`: defaults to 2 seconds.

`kill_port_if_requested()` first cleans listener PIDs visible in the current environment. It only checks for Windows-side listeners when the script detects WSL, no local listener PID is visible, and the port still cannot bind. In that WSL-only path it requests Administrator PowerShell to delete stale `portproxy` rules, stop services that own the port, or force-stop the owning process. If the administrator request is canceled, or a system service such as `iphlpsvc` refuses to stop, the script prints the Windows listener details and Administrator PowerShell recovery commands, then stops startup immediately instead of launching the service into the same port error. If the frontend Vite process only discovers `Port 3000 is already in use` after launch, the script prints the same Windows listener recovery commands. Non-WSL environments do not run the Windows cleanup path. `--allow-lan` now exposes `3000` / `8000` / `8010` directly and no longer starts an extra Windows forwarding process; old persistent portproxy rules should be removed.

## Issue 4: `restart` Behavior

Before the stamp path fix:

- `restart -b`: stop all services, check fingerprint, rebuild only when needed, then start.
- plain `restart`: stop all services, then often rebuild AI Provider because `/tmp` lost the stamp.

After moving the stamp file, plain `restart` uses the same `stop + start` behavior and the same fingerprint check as `restart -b`.

## State Files, Logs, and Failed-Start Cleanup

`planet.sh` no longer writes PID files, logs, or runtime env-files to fixed `/tmp/planet_*` paths. The default state directory is:

```bash
${XDG_STATE_HOME:-$HOME/.local/state}/planet
```

At startup the script creates this directory and tries to set it to `700`. The current files include:

- `backend.pid` / `frontend.pid` / `motion_agent.pid`
- `backend.log` / `frontend.log` / `motion_agent.log`
- `aiprovider_build.log`
- `aiprovider_runtime.env`
- `ports.env`

PID writes validate that the PID is a positive integer, include a trailing newline, and try to set file mode `600`. PID reads ignore invalid content instead of passing it to `kill`.

After a successful `start`, the script records the ports in `ports.env`. Later `./planet.sh health` calls prefer the last started ports; if the state file is missing, health checks fall back to the defaults `8000`, `3000`, `8010`, and `8765`. This avoids checking default ports after starting with custom ports.

Startup now has light failed-start cleanup. If `start` exits before completing, the script only cleans local processes that this run already started: backend, frontend, and Motion Agent. It does not stop services after a successful start. AI Provider, PostgreSQL, and Redis keep their existing container lifecycle behavior.

## Health Checks and Hardening

HTTP readiness checks now use `curl -fsS --max-time`, so 4xx and 5xx responses are no longer treated as healthy.

Process termination now validates:

- signal names are limited to `TERM`, `KILL`, `INT`, and `HUP`;
- PIDs must be positive integers;
- process group IDs must be positive integers.

This prevents bad PID files or invalid signals from reaching `kill`.

Frontend and Motion Agent startup failures now call `print_port_listener_details()`, matching backend port diagnostics. The Windows-side listener and cleanup path still only runs when WSL is detected.

## Cross-Platform Notes

The script is currently Linux-first with WSL enhancements. Normal Linux runs do not execute the PowerShell path; WSL gets extra Windows listener, portproxy, and camera guidance.

To make this single script fully portable across Linux, macOS, and WSL, the remaining platform differences should be wrapped behind compatibility helpers:

- `stat --format`, `sort -V`, and `xargs -r` are GNU-style and are not fully compatible with default macOS BSD tools.
- `hostname -I`, `ss`, `fuser`, and `systemctl` are usually unavailable on macOS.
- `tac` may be missing on macOS; use `awk` or Python as a fallback.
- Docker Desktop on macOS does not use `systemctl` daemon diagnostics.
- Camera auto-detection relies on `/dev/video*` / `v4l2-ctl`, which is Linux-specific; macOS should use explicit camera URLs or a separate AVFoundation detector.

The recommended direction is a small platform compatibility layer for port listener detection, version comparison, file metadata, reverse tail, LAN IP discovery, and Docker daemon diagnostics, instead of scattering more platform branches throughout service startup logic.

## Production Delivery Boundary

`planet.sh` is a local development convenience script, not the production startup entrypoint. Production delivery should use Kubernetes `Deployment`, `Service`, `Ingress`, and readiness/liveness probes for ports, health checks, restarts, and rolling upgrades. This removes the need for a host script to reclaim local ports and avoids running the Vite dev server in production.

The production frontend shape is `vite build` static output served by nginx/Caddy or an equivalent HTTP server. Do not use `bun run dev` or `vite preview` in production. The project does not maintain a parallel Webpack build chain; if a future enterprise requirement needs closer Webpack-ecosystem compatibility, run an Rsbuild/Rspack spike first. Electron should only be evaluated when the official target becomes an offline desktop application.

## Default Motion Agent Startup

`planet.sh` now starts the local Motion Agent by default during `start` and full `restart`. This makes the Earth page, UE clients, and debug clients able to connect to `ws://127.0.0.1:8765/ws/gestures` immediately. If the machine has no usable camera, implicit default startup falls back to dry-run protocol mode and does not block backend/frontend startup. Explicit Motion Agent startup through `--motion-agent`, camera indexes, camera URLs, or WSL USB options still treats live camera failures as real errors.

To skip Motion Agent for this run:

```bash
./planet.sh start --non-motion-agent
./planet.sh restart --non-motion-agent
```

Common options:

- `--non-motion-agent`: do not start Motion Agent for this `start` or full `restart`.
- `--motion-agent` / `-m`: explicitly start or restart the Motion Agent for this command; live camera failures are reported as failures.
- `--motion-agent-port <port>`: override the default WebSocket port `8765`.
- `--motion-agent-mode <mode>`: choose `auto`, `single`, `dual_redundant`, `single_fallback`, or `calibrated_3d`; `dual` is kept as a compatibility alias for redundant dual-camera mode.
- `--motion-agent-camera-indexes <indexes>`: override auto-detected camera indexes, for example `0` or `0,1`. The same can be provided through `MOTION_AGENT_CAMERA_INDEXES=0,1`.
- `--motion-agent-camera-urls <urls>`: use RTSP/HTTP camera streams, useful for WSL, phone cameras, or network cameras. The same can be provided through `MOTION_AGENT_CAMERA_URLS=...`.
- `--motion-agent-wsl-usbipd`: in WSL, try to attach the single detected Windows USB camera through `usbipd-win`.
- `--motion-agent-wsl-usbipd-busid <BUSID>`: in WSL, attach the camera matching a `usbipd list` BUSID; use this when multiple cameras are present.
- `--motion-agent-dry-run`: start only the protocol service without opening cameras or loading CV dependencies; useful for Web client debugging.

Non-dry-run live mode checks `mediapipe` and `opencv-python` before startup. If the current `.venv` is missing them, the script automatically runs:

```bash
uv add mediapipe opencv-python
```

To disable startup-time Python CV dependency auto-install:

```bash
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start
```

`./planet.sh init` also performs a WSL host dependency preflight for `usbipd-win`. If `usbipd.exe` is missing, the script first tries `winget install -e --id dorssel.usbipd-win`, then reuses the repository-bundled dorssel.usbipd-win MSI fallback. If that cached MSI is missing or an architecture-specific MSI is needed, it downloads one and requests an Administrator PowerShell installation. This is best-effort: failure prints next steps but does not block normal initialization. Use `./planet.sh init --non-motion-agent` to skip this preflight.

Live mode auto-detects `/dev/video*`, then prefers an OpenCV probe to keep only indexes that can open and return frames before passing them to the Motion Agent. In WSL/USB camera setups, one camera can expose multiple `/dev/video*` nodes, and some of them are metadata or non-capture nodes; the script skips those unreadable indexes. In WSL, Windows cameras usually do not appear as `/dev/video*` automatically. Check available devices first:

Live capture defaults to low-latency settings: `640x360` input and roughly `15Hz` recognition events. The worker uses latest-frame reader threads and keeps only the newest frame from each camera, so a slow MediaPipe frame does not make the recognizer drain stale camera backlog. The skeleton debug stream is disabled by default and is only sent at roughly `8Hz` while the Earth motion debug panel is open, so normal gesture control is not slowed down by debug data. Status events report both capture FPS and recognition FPS to separate camera throughput issues from recognition cost.

```bash
ls /dev/video*
```

To override auto-detection, pass indexes explicitly:

```bash
./planet.sh start --motion-agent --motion-agent-camera-indexes 1,2
```

In WSL, the more general path is to connect a phone or network camera through an RTSP/HTTP stream:

```bash
./planet.sh start --motion-agent --motion-agent-camera-urls http://192.168.1.20:8080/video
```

To use a Windows USB camera directly from WSL, let the script call `usbipd-win`. This is opt-in because an attached camera is usually temporarily unavailable to Windows apps while WSL owns it.

When there is only one camera:

```bash
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
```

When there are multiple cameras, inspect the BUSID first and pass it explicitly:

```bash
usbipd.exe list
./planet.sh start --motion-agent --motion-agent-wsl-usbipd-busid 3-2
```

If `usbipd attach` says the device is not shared or bound, the script tries to open an Administrator PowerShell to run `usbipd bind`, then retries attach. If UAC is canceled or automatic bind fails, run this manually from an Administrator PowerShell:

```powershell
usbipd bind --busid 3-2
usbipd attach --wsl --busid 3-2
```

If WSL has no `/dev/video*` and no `--motion-agent-camera-urls` is provided, implicit default startup falls back to dry-run. Explicit live startup stops and prints guidance. Choose one of:

```bash
./planet.sh start --motion-agent --motion-agent-camera-urls http://<phone-ip>:8080/video
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
./planet.sh start --motion-agent --motion-agent-dry-run
```

For explicit live startup, automatic dry-run fallback only happens when `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` is set.

`--non-motion-agent` is the command-level opt-out. Environment variables can still tune how the service starts:

```bash
MOTION_AGENT_DRY_RUN=1 ./planet.sh start
```

Logs:

```bash
./planet.sh log -m
```

To expose it together with the frontend on the LAN:

```bash
./planet.sh start --allow-lan --motion-agent
```

In this mode the Motion Agent binds `0.0.0.0`, and startup output prints both the local WebSocket URL and the recommended LAN WebSocket URL. When opening Earth from another LAN browser, point `motionAgent` at the display machine:

```text
http://<LAN_IP>:3000/earth?motion=1&motionAgent=ws://<LAN_IP>:8765/ws/gestures
```

`./planet.sh stop` also stops a script-managed Motion Agent. `./planet.sh health` reports its online/offline status. The Earth page still requires `?motion=1` or browser local storage to enable the Web client connection explicitly.

For ordinary web, WSL, or no-install demo scenarios, you can skip Motion Agent entirely: choose the `Browser Camera` input source in Earth settings and enable Motion Debug Mode. This route uses browser `getUserMedia`, so the page must run on HTTPS or localhost and the user must grant camera permission.

## Other Cleanup

Two redundant `sleep 3` waits were removed because health checks already cover the same readiness:

- `start_backend_service`: post-database-health-check sleep.
- `restart_database_service`: post-restart sleep.

## Related Files

- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
- [.dockerignore](/home/ray/dev/linkong/planet/.dockerignore)
- [aiprovider/Dockerfile](/home/ray/dev/linkong/planet/aiprovider/Dockerfile)
- [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml)
- [docker-compose.simple.yml](/home/ray/dev/linkong/planet/docker-compose.simple.yml)
- [compute_aiprovider_dependency_fingerprint.py](/home/ray/dev/linkong/planet/scripts/compute_aiprovider_dependency_fingerprint.py)