release: bump version to 0.51.0

2026-05-11 09:49:08 +08:00
parent 455b8360d0
commit 1cb51b1172
52 changed files with 4440 additions and 279 deletions
--- a/docs/plans/agents-light-orchestrator-websearch-plan.md
+++ b/docs/plans/agents-light-orchestrator-websearch-plan.md
@@ -179,6 +179,217 @@ Secret resolution should follow the existing settings pattern:
 2. provider-specific environment variable, for example `TAVILY_API_KEY`
 3. generic fallback `WEB_SEARCH_API_KEY`

+### Common WebSearch Providers
+
+The first implementation should model WebSearch as a provider-specific adapter
+behind one internal interface:
+
+```text
+SearchEvidenceProvider.search(query, max_results, domains, freshness_days)
+  -> list[SearchEvidence]
+```
+
+Recommended provider ids and environment variables:
+
+| Provider | Provider id | Env key | Default base URL | Primary use |
+| --- | --- | --- | --- | --- |
+| Tavily | `tavily` | `TAVILY_API_KEY` | `https://api.tavily.com` | Default hosted search for agent/RAG style results |
+| Brave Search API | `brave` | `BRAVE_SEARCH_API_KEY` | `https://api.search.brave.com` | Independent web index and low-level SERP results |
+| SerpAPI | `serpapi` | `SERPAPI_API_KEY` | `https://serpapi.com` | Search-engine-backed SERP data with engine options |
+| Exa | `exa` | `EXA_API_KEY` | `https://api.exa.ai` | Neural/semantic web search and result contents |
+| Firecrawl Search / Scrape | `firecrawl` | `FIRECRAWL_API_KEY` | `https://api.firecrawl.dev` | Search plus page scrape/markdown extraction |
+| SearXNG | `searxng` | optional `SEARXNG_API_KEY` | self-hosted instance URL | Self-hosted metasearch when external search APIs are undesirable |
+
+The normalized configuration should support per-provider defaults while keeping
+one active provider:
+
+```text
+external_integrations.web_search
+  enabled: true
+  default_provider: tavily
+  providers:
+    tavily:
+      base_url: https://api.tavily.com
+      api_key: <secret>
+      max_results: 5
+      search_depth: basic
+      include_answer: false
+      include_raw_content: false
+    brave:
+      base_url: https://api.search.brave.com
+      api_key: <secret>
+      endpoint_path: /res/v1/web/search
+      max_results: 5
+    serpapi:
+      base_url: https://serpapi.com
+      api_key: <secret>
+      endpoint_path: /search.json
+      engine: google
+      max_results: 5
+    exa:
+      base_url: https://api.exa.ai
+      api_key: <secret>
+      endpoint_path: /search
+      max_results: 5
+      include_text: false
+    firecrawl:
+      base_url: https://api.firecrawl.dev
+      api_key: <secret>
+      search_path: /v2/search
+      scrape_path: /v2/scrape
+      max_results: 5
+      scrape_formats: [markdown]
+    searxng:
+      base_url: http://localhost:8080
+      api_key: <optional secret>
+      endpoint_path: /
+      max_results: 5
+      categories: general
+      engines: []
+```
+
+Adapter notes:
+
+- Tavily should call `/search` and normalize title, URL, snippet/content, score,
+  and optional raw content.
+- Brave should call `/res/v1/web/search` and map web results into the same
+  `SearchEvidence` shape.
+- SerpAPI should call `/search.json`, pass `engine`, and normalize organic
+  results. Search-engine-specific fields should remain in provider metadata.
+- Exa should call `/search`; optional result text should be treated as fetched
+  content only when enabled.
+- Firecrawl can be used both as `web_search` and `web_fetch`: `/v2/search`
+  returns result URLs/descriptions and may include scrape options, while
+  `/v2/scrape` can produce markdown for a selected URL.
+- SearXNG should query the configured instance with `q` and `format=json`.
+  Public instances should not be assumed reliable for production; a controlled
+  self-hosted instance is preferred.
+
+The settings UI should expose only provider, base URL, key, max results, and a
+test button in the first version. Provider-specific advanced fields can stay
+collapsed or backend-only until a real workflow needs them.
+
+### Frontend Configuration Window
+
+Add a WebSearch configuration panel to the existing settings page, next to the
+LLM provider configuration. It should behave like the current AI provider secret
+controls: clear configured state, masked preview, explicit show/hide, test
+connection, and save feedback.
+
+First-version visible fields:
+
+```text
+WebSearch Provider
+API Base URL
+API Key
+Max Results
+Timeout Seconds
+Enable WebSearch
+Test Connection
+Save
+```
+
+Provider dropdown options:
+
+```text
+Tavily
+Brave Search API
+SerpAPI
+Exa
+Firecrawl Search / Scrape
+SearXNG
+```
+
+Field behavior:
+
+- Switching provider loads that provider's saved config and masked key preview.
+- Empty key input means keep the existing saved or environment key.
+- Typing a new key replaces only the selected provider's key.
+- Show key reveals the full current input value when the backend reveal endpoint
+  allows it; hide key returns to the prefix-preserving masked preview.
+- The configured badge should only show `已配置` or `未配置`, not repeat the
+  masked key text.
+- `Test Connection` sends the current unsaved draft to the backend and should
+  not require a separate save first.
+- A successful test may save the draft as the new WebSearch default only if the
+  API endpoint is explicitly designed to mirror the AI provider test behavior.
+  Otherwise, test should be read-only and the Save button should persist.
+- Save success and test success must show visible feedback. Failures should show
+  provider-specific but secret-safe error messages.
+
+Provider-specific UI hints:
+
+| Provider | UI hint |
+| --- | --- |
+| Tavily | Good default for agent/RAG style search. |
+| Brave Search API | Uses Brave's independent search index. |
+| SerpAPI | Supports search-engine-specific parameters such as `engine`. |
+| Exa | Good for semantic search and optional result text. |
+| Firecrawl | Can search and scrape pages into markdown. |
+| SearXNG | Requires a reachable self-hosted or trusted instance URL. |
+
+Advanced fields can live in a collapsed section:
+
+```text
+Endpoint Path
+Search Depth
+Engine
+Categories
+Engines
+Include Raw Content
+Scrape Formats
+Domain Allowlist
+```
+
+The first version should keep the UI conservative. It should not expose every
+provider knob until backend workflows use those knobs.
+
+### Web Fetch and Page Extraction
+
+`web_fetch` is separate from `web_search`. Search finds candidate URLs; fetch
+turns selected pages into clean, citable evidence.
+
+Recommended extraction chain:
+
+```text
+1. plain httpx fetch
+2. trafilatura extraction for static HTML
+3. readability extraction as secondary cleanup
+4. Playwright fetch only for allowlisted JS-heavy pages
+5. Firecrawl scrape as hosted fallback when configured
+```
+
+Implementation guidance:
+
+- Use `trafilatura` as the first local extractor because it is Python-native and
+  matches the backend stack.
+- Prefer a Python readability implementation for local cleanup. Do not introduce
+  a Node-only readability dependency for backend fetch.
+- Use Playwright sparingly for JavaScript-rendered pages. It should have domain
+  allowlists, low concurrency, strict timeouts, response size limits, and no
+  automatic form submission or login behavior.
+- Store `content_hash`, `retrieved_at`, final URL, title, extracted text
+  preview, and extractor name in `ai_evidence`.
+- Keep short quotes for UI review, but do not store huge page bodies directly in
+  every task record. Large extracted content should be truncated or stored once
+  by hash.
+
+The local/self-hosted stack should look like this:
+
+```text
+SearXNG
+  -> SearchEvidence URLs
+  -> httpx fetch
+  -> trafilatura / readability
+  -> Playwright only when static extraction fails and the domain is allowed
+  -> normalized evidence
+  -> LLM structured output through AIProviderClient
+```
+
+This route gives Planet a lower-cost and more controllable search path, while
+hosted providers remain available when search quality or maintenance effort
+matters more than self-hosting.
+

 ## Phase 3: Limited Agent Loop