# Agent Runtime, Earth LLM Command, And Speech Entry Plan ## Summary Build an auditable backend Agent Runtime and upgrade the existing Earth search panel into a combined search, AI command, and voice wake entry. The first version is a runtime foundation, not the full multi-role simulation product yet. Typical user goals: - In Earth, type "高亮所有北斗卫星" and have the system open the satellite layer and highlight matching Beidou satellites. - Type "中国大陆的算力中心" and have the system open the compute-center layer, match mainland China compute centers, and highlight them. - When microphone permission is granted, use a configurable wake word, then speak an Earth command. - Save every Earth AI command as an agent run so operators can review the original input, speech transcription, tool steps, entity matches, final action plan, and frontend execution result. Core boundaries: - `aiprovider` remains the model gateway. It must not own business tools, database access, Earth actions, or agent policy. - The backend owns agent orchestration, tools, evidence storage, permission policy, and proposal application. - Earth v1 executes visualization actions only. It does not mutate business data. - Speech recognition uses a provider-neutral ASR API first, defaulting to OpenAI/Whisper-compatible transcription APIs, with local `whisper.cpp` style providers as later adapters. ## Architecture ### Backend Agent Runtime Recommended module shape: ```text backend/app/models/ agent_run.py agent_step.py agent_evidence.py agent_proposal.py backend/app/schemas/ agents.py speech.py backend/app/services/agents/ runtime.py orchestrator.py tool_protocol.py tool_registry.py policy.py proposals.py earth_command.py entity_query.py speech.py backend/app/api/v1/ agents.py ``` Database conventions should follow the current project style: - Use SQLAlchemy models. - Import new models from `init_db()`. - Let `Base.metadata.create_all` create tables. - Add required indexes with `CREATE INDEX IF NOT EXISTS`. - Do not introduce Alembic for this feature. ### aiprovider Boundary `aiprovider` should continue to provide model transport only: - Keep the existing `/v1/analyze` endpoint. - If schemas are extended, only pass through model/provider request fields and normalize responses. - Do not add WebSearch, database queries, Earth entity lookup, or business action execution inside `aiprovider`. - If a provider does not support native tool calling, the backend must use JSON tool-call fallback. ### Agent Tool Protocol The first version should support two protocols: - Default path: backend JSON tool-call loop. - Optional path: provider-native tools when the configured provider supports them. - Fallback path: if provider-native tools are unavailable or unstable, automatically use JSON tool-call. Example JSON tool call: ```json { "type": "tool_call", "tool": "earth.find_entities", "arguments": { "domain": "satellites", "filters": { "constellation": "beidou" }, "limit": 500 } } ``` Example final response: ```json { "type": "final", "summary": "已找到并高亮北斗卫星。", "result": { "actions": [] } } ``` Hard limits: - One run may execute at most 8 tool steps by default. - One tool call times out after 30 seconds by default. - Only registered whitelist tools may run. - Tool arguments must pass Pydantic validation. - Illegal tools, invalid arguments, and denied actions must be recorded as step errors. - LLM output may not directly write business state. Writes are either proposals or backend-executed policy-approved actions. ## Data Model ### AgentRun Suggested fields: - `id`: integer primary key. - `public_id`: public string id. - `run_type`: `earth_command`, `situational_awareness`, `config_proposal`, or `diagnostic`. - `status`: `queued`, `running`, `waiting_approval`, `completed`, `failed`, or `stopped`. - `title`: short display title. - `objective`: text objective. - `input`: JSONB original input, including text and audio metadata. - `context`: JSONB run context. - `result_markdown`: final human-readable output. - `result_json`: structured output, including Earth action plans. - `provider`: nullable provider id. - `model`: nullable model id. - `request_id`: nullable propagated request id. - `created_by`: user id. - `created_at`, `updated_at`, `completed_at`. - `error`: nullable text error. ### AgentStep Suggested fields: - `id`. - `run_id`. - `step_index`. - `step_type`: `llm`, `tool`, `policy`, `action`, or `transcription`. - `status`: `pending`, `running`, `completed`, `failed`, or `skipped`. - `name`: step name, for example `earth.find_entities`. - `input`: JSONB. - `output`: JSONB. - `error`: nullable text. - `started_at`, `completed_at`. - `duration_ms`. ### AgentEvidence Suggested fields: - `id`. - `run_id`. - `step_id`: nullable. - `evidence_type`: `internal_record`, `web_search`, `web_fetch`, `entity_match`, or `transcription`. - `source`: source id. - `title`: display title. - `url`: nullable source URL. - `content`: text evidence content. - `content_hash`: nullable hash. - `metadata`: JSONB. - `retrieved_at`. ### AgentProposal Suggested fields: - `id`. - `run_id`. - `proposal_type`: `datasource_config`, `ai_prompt`, or `external_integration`. - `status`: `pending`, `approved`, `applied`, `rejected`, or `failed`. - `risk_level`: `low`, `medium`, or `high`. - `target`: JSONB target descriptor. - `before_payload`: JSONB. - `after_payload`: JSONB. - `rationale`: text. - `policy_result`: JSONB. - `applied_by`: nullable user id. - `applied_at`: nullable timestamp. - `error`: nullable text. ## Public APIs ### Agent Runs Add: ```text POST /api/v1/agents/runs GET /api/v1/agents/runs GET /api/v1/agents/runs/{run_id} POST /api/v1/agents/runs/{run_id}/stop POST /api/v1/agents/proposals/{proposal_id}/apply ``` ### Earth Command Add: ```text POST /api/v1/agents/earth/command ``` Request shape: ```json { "input_text": "高亮所有北斗卫星", "source": "text", "transcription_id": null, "client_context": { "visible_layers": ["satellites"], "locale": "zh-CN", "viewport": { "is_mobile": false } } } ``` Response shape: ```json { "run_id": "agent_xxx", "summary": "已找到并高亮北斗卫星。", "actions": [ { "type": "show_layer", "layer": "satellites", "enabled": true }, { "type": "highlight_entities", "domain": "satellites", "entity_ids": ["satellite:norad:12345"], "style": { "color": "#7dd3fc" } }, { "type": "open_result_panel", "title": "北斗卫星", "items": [] } ], "matched_entities": [], "confidence": 0.86, "missing_data": [] } ``` ### Speech / ASR Add: ```text POST /api/v1/agents/speech/transcriptions ``` Request should use multipart form data: - `file`: audio blob. - `language`: default `zh`. - `provider`: optional provider override. - `source`: default `earth_command`. Response shape: ```json { "text": "高亮所有北斗卫星", "provider": "openai_compatible", "model": "whisper-1", "duration_ms": 1234, "confidence": null, "metadata": {} } ``` ## Earth Action Plan ### Allowed Action Types The first version may only return: ```text show_layer highlight_entities filter_entities focus_view open_result_panel clear_highlight ``` ### Allowed Domains The first version supports: ```text satellites compute_centers bgp news vessels cables ``` ### EarthAction Shape Example: ```json { "type": "highlight_entities", "domain": "satellites", "entity_ids": ["satellite:norad:12345"], "style": { "color": "#7dd3fc", "mode": "glow" }, "reason": "用户要求高亮北斗卫星" } ``` ### Safety Rules The backend must validate action plans before returning them: - `type` must be in the allowed action list. - `domain` must be in the allowed domain list. - `entity_ids` must come from backend or current Earth candidate data. The model may not invent ids. - One highlight action should include at most 500 entities by default. If more match, return a truncation note in `missing_data` or `summary`. - `focus_view` must include valid coordinates, a valid region, or a matched entity. - No action may return executable JavaScript, arbitrary CSS, or arbitrary URLs to fetch. ## Earth Entity Query Add a backend entity query service used by both deterministic resolvers and LLM tools. ### Satellites Sources: - `/api/v1/visualization/geo/satellites`. - Current TLE collected data. - GeoJSON feature properties. Filters: - `constellation`: `beidou`, `gps`, `galileo`, `glonass`, `starlink`, `iridium`, `geo`, `leo`. - Name contains. - NORAD id. - Country/operator when present in data. - Orbital class when inferable from existing fields. Beidou matching: - Prefer `constellation == beidou`. - Then match names containing `BEIDOU`, `BDS`, `BEIDOU-`, or `北斗`. - Stable entity id format should be `satellite:norad:{norad_id}` when possible, with fallback `satellite:index:{index}`. ### Compute Centers Sources: - `/api/v1/visualization/geo/compute-centers`. - Unified TOP500 and Epoch AI GPU GeoJSON properties. Filters: - Country/region. - `site_type`: `supercomputer` or `gpu_cluster`. - Source: `top500` or `epoch_ai_gpu`. - Name contains. - `needs_confirmation`. - `location_precision`. Mainland China matching: - Match country values such as `China`, `中国`, or `People's Republic of China`. - Exclude obvious non-mainland records when fields identify Hong Kong, Macau, or Taiwan. - If records do not expose enough region detail to separate mainland China from Hong Kong, Macau, or Taiwan, return a `missing_data` note and conservatively match `country=China`. ### BGP, News, Vessels, And Cables First-version basic support: - BGP: severity, status, region, collector, prefix, ASN. - News: region, source, localized title, localized summary. - Vessels: vessel type, country/area, name, status. - Cables: cable name, landing point, country/region. ## Earth Frontend Integration ### Search Panel Merge Reuse the existing Earth search panel: - Default behavior remains normal local search. - Add an AI command state for natural-language commands. - Add a "use AI" command button. - Add a microphone button. - Show running status, result summary, and clear-highlight action. - Visually distinguish ordinary search results from AI action results. Natural-language routing: - If the user clicks the AI command button, always call the AI command endpoint. - If input contains action words such as `高亮`, `显示`, `找出`, `聚焦`, `打开`, `筛选`, `隐藏`, or `清除`, suggest AI command mode. - Short ordinary keywords continue to use local search. ### Earth Action Executor Add: ```text frontend/public/earth/js/earth-command-actions.js ``` Responsibilities: - Receive backend `EarthActionPlan`. - Toggle required layers. - Apply highlights and filters. - Focus view when requested. - Open or update the result panel. - Clear the previous AI command highlight. Execution order: 1. Clear previous AI command highlights unless the plan is explicitly additive. 2. Show required layers. 3. Apply filters and highlights. 4. Focus the view if requested. 5. Open the result panel. 6. Record the execution summary in UI state. ### Satellite Highlight Existing `highlightRelatedSatellites(indices, color)` can be reused, but Earth command support needs an entity-id-to-index map. Add: - Satellite id map built from current satellite data. - Highlight by `entity_ids`. - Clear AI command satellite highlights without clearing manual locked selection. - Auto-enable the satellite layer before highlighting. ### Compute Center Highlight Existing compute center markers support marker state, but need a dedicated batch AI highlight state. Add: - Batch set compute center markers to AI highlighted state. - Keep AI highlight compatible with hover and locked states. - Clear AI command compute highlights without clearing manual locked selection. - Auto-enable the compute-center layer before highlighting. ### Result Panel First version can show results inside the merged search panel: - Title, for example `北斗卫星`. - Count, for example `已高亮 32 个对象`. - List with the first 20 entities. - Actions: clear highlight, rerun, view Agent Run. ## Voice And Wake Word ### ASR Configuration Add Speech/ASR under the AI settings tool tab. Fields: - `enabled`. - `provider`. - `base_url`. - `api_key`. - `model`. - `language`. - `timeout_seconds`. - `max_audio_size_mb`. First-version providers: - `openai_whisper`. - `openai_compatible`. - `local_whisper`. Recommended defaults: - `provider`: `openai_compatible`. - `model`: `whisper-1` or user-configured equivalent. - `language`: `zh`. Secret handling: - Use the existing masked secret pattern. - Do not allow agent proposals to write API keys. - Support environment fallback such as `ASR_API_KEY` and `OPENAI_API_KEY`. ### Wake Word The wake word is configurable. Implementation rules: - Treat wake word as a browser-local preference in v1. - Store it in `localStorage`. - Default suggestion: `小星球`. - Provide a wake-word setting inside the Earth search/command panel. - Do not listen until the user explicitly enables voice wake. - Do not upload audio before wake. - After wake, record one instruction audio segment and upload it to backend ASR. Fallbacks: - Browser does not support continuous local recognition: fall back to click-to-record. - Microphone permission denied: show `麦克风权限未开启，仍可输入文字指令`. - ASR not configured: show `语音识别未配置`; text commands remain available. - User can disable wake listening and keep manual microphone recording. ### Wake Word Technical Choice Do not require local Whisper in v1. Recommended path: - Use browser Web Speech API for local wake-word detection when available. - Fall back to click-to-record when unavailable. - Upload only the post-wake instruction audio to backend ASR. - Add local `whisper.cpp` streaming wake-word or ASR provider later. ## Prompt Registry Add default prompt keys: ```text agents.runtime.system agents.earth.command agents.situational.assessment agents.config.proposal agents.roles.network agents.roles.bgp agents.roles.platform_ops agents.roles.business_impact ``` `agents.earth.command` must require: - Strict JSON output only. - Use only provided candidate entities. - Do not invent objects. - Do not return arbitrary code. - Do not modify business data. - Return `clarification_needed` when intent is unclear. - Return action plans matching the schema. - Use Chinese summary for Chinese user input. ## Policy ### Tool Policy - The LLM can request tools, but the backend decides whether to execute. - Every tool must declare: - name. - description. - input schema. - output schema. - permission. - side-effect level. - First-version side-effect levels: - `read`: can run directly. - `proposal`: can only generate proposals. - `write`: can only execute during proposal apply. ### Proposal Policy First version supports configuration proposal application with role-based automatic/manual gating: - `super_admin` may enable automatic application for low-risk proposals. - Normal admins must manually confirm. - High-risk proposals always require manual confirmation. - Proposals containing secret fields are rejected. - Proposals failing schema validation are rejected. - Before and after payloads must be saved. - Failed applications must save errors. Low-risk scope: - Datasource endpoint/config non-secret fields. - AI prompt overrides. - External integration non-secret fields. Out of scope for v1: - Alert acknowledge/resolve. - Data deletion. - User permission changes. - Authentication configuration changes. - Database schema changes by agent. - Mutation of original Earth collected data. ## Agent Operations UI Add an `Agent` page under `运维与配置`. Run list: - Status. - Type. - Title. - Creator. - Model. - Time. - Duration. Run detail: - Input. - Status timeline. - LLM steps. - Tool steps. - Evidence. - Final result. - Proposals. Proposal apply: - Before/after diff. - Risk level. - Policy result. - Apply button. - Reject button. Earth command run detail: - Original text or speech transcription. - Matched entities. - Action plan. - Link back to Earth or copy run id. ## Frontend Types Add or extend: ```ts interface AgentRun {} interface AgentStep {} interface AgentEvidence {} interface AgentProposal {} interface EarthCommandRequest {} interface EarthActionPlan {} interface EarthAction {} interface SpeechTranscriptionResponse {} ``` Earth action executor API: ```js executeEarthActionPlan(plan, context) clearEarthCommandHighlights() getEarthCommandExecutionState() ``` ## Testing ### Backend Tests Use `uv`. Suggested files: ```text backend/tests/test_agents_runtime.py backend/tests/test_agent_tool_protocol.py backend/tests/test_agent_policy.py backend/tests/test_earth_agent_command.py backend/tests/test_speech_transcription.py ``` Coverage: - Create run. - Persist ordered run steps. - Parse valid JSON tool calls. - Reject invalid JSON tool calls without executing tools. - Reject unregistered tools. - Reject invalid tool arguments. - Pass through provider-native tools fields when configured. - Fall back to JSON tool calls when native tools are unavailable. - When WebSearch is not configured, record `missing_data` and do not fail Earth command. - "高亮所有北斗卫星" returns a satellite highlight action. - "中国大陆的算力中心" returns a compute-center highlight action. - Mainland China matching excludes Hong Kong, Macau, and Taiwan when fields permit it. - Field granularity limitations are returned in `missing_data`. - ASR not configured returns a clear error. - ASR provider success returns transcription. - Normal admins cannot auto-apply proposals. - `super_admin` can apply low-risk proposals. - Secret-field proposals are rejected. ### Frontend Tests Use `bun`. Required build check: ```bash cd frontend && bun run build ``` Suggested Earth JS tests: - Action executor opens the satellite layer. - Action executor highlights satellite entity ids. - Action executor highlights compute-center entity ids. - Clear highlight does not clear manual locked selection. - Search panel ordinary search still uses local search. - AI command button calls `/api/v1/agents/earth/command`. - ASR missing configuration and microphone permission denial show clear UI feedback. ### Manual Acceptance 1. Type `高亮所有北斗卫星`. - Satellite layer opens. - Beidou satellites are highlighted. - Panel shows count and summary. - Agent run is reviewable. 2. Type `显示中国大陆的算力中心`. - Compute-center layer opens. - Mainland China related compute centers are highlighted. - If data cannot separate mainland China from Hong Kong/Macau/Taiwan, the UI shows a missing-data note. 3. Click microphone. - Permission denial is clear. - Permission grant allows recording. - Configured ASR transcribes and executes the command. - If ASR is not configured, text input still works. 4. Enable voice wake. - Wake word is configurable. - Audio is not uploaded before wake. - After wake, one instruction audio segment is uploaded. - Wake listening can be disabled. 5. Open Agent operations UI. - Earth command run is listed. - Transcription step is visible when speech was used. - Entity query step is visible. - Final action plan is visible. - User can return to Earth or copy the run id. ## Documentation When implemented, update: ```text docs/technical/zh/agents-aiprovider.md docs/technical/en/agents-aiprovider.md docs/technical/zh/manual.md docs/technical/en/manual.md docs/technical/zh/earth-frontend-context.md docs/technical/en/earth-frontend-context.md ``` Document: - `aiprovider` and backend agent boundaries. - Earth AI command entry. - Speech/ASR configuration. - Wake-word privacy behavior. - Agent run/evidence/proposal review. - V1 capability boundary: Earth visualization actions only, no business data mutation. ## Implementation Order 1. Add backend Agent data models, `init_db()` imports, and indexes. 2. Add Agent schemas and run CRUD API. 3. Add tool registry, JSON tool-call parser, and policy skeleton. 4. Add Earth entity query service. 5. Add `agents.earth.command` prompt and backend command endpoint. 6. Add frontend Earth action executor. 7. Merge AI command entry into Earth search panel. 8. Add satellite and compute-center batch highlight support. 9. Add Speech/ASR settings and transcription API. 10. Add Earth microphone recording, wake-word local setting, and fallback behavior. 11. Add Agent operations UI. 12. Add proposal apply policy and low-risk configuration application. 13. Add tests and documentation updates. ## Assumptions - Earth LLM v1 only executes visualization actions. - Earth commands create agent runs but do not automatically create configuration proposals. - Configuration proposals are applied only from the Agent operations UI. - Wake word is a device-local preference in v1 and is stored in `localStorage`. - ASR is API-first and Whisper-compatible by default; local Whisper is a later provider. - Multi-role simulation only reserves schemas and prompt keys in v1. - Palantir-style situational workflows should grow from the shared evidence, entity, action, and assessment model instead of a separate isolated system.