# Datasource Health Stage 2 Tasks ## Goal Stage 2 focuses on the first practical operational layer: - deterministic datasource health checks - persisted health results - health visibility through API and UI - no agent-assisted repair yet This stage should make Planet capable of answering: - which collectors are healthy - which collectors are degraded - which collectors are failing - why they are failing at a basic deterministic level ## Scope Included: - datasource health data model - deterministic health check service - manual and scheduled health check triggers - health result APIs - frontend visibility Excluded: - LLM reasoning - web-search-based repair proposals - automatic endpoint rewriting - runtime override application ## Delivery Target At the end of Stage 2, an operator should be able to: 1. see health status for each collector 2. trigger a health check manually 3. inspect the latest failure reason 4. inspect the last checked endpoint 5. understand whether the problem is: - unreachable - auth-related - rate-limit-related - schema-related - empty-data-related ## Work Breakdown ## A. Data Model ### A1. Add datasource health record table Create a new model, for example: - `backend/app/models/datasource_health_check.py` Suggested fields: - `id` - `datasource_id` - `collector_name` - `endpoint_checked` - `status` - `http_status` - `content_type` - `latency_ms` - `sample_count` - `error_message` - `details` - `checked_at` Suggested status enum values: - `healthy` - `degraded` - `failed` - `schema_changed` - `rate_limited` - `auth_required` - `empty_result` ### A2. Add datasource health summary fields Option A: - keep summary only in the health check table Option B: - also add summary fields on `data_sources` Recommended first step: - do not mutate `data_sources` schema yet - derive summary from the latest health record ### A3. Migration task Add migration for the health table. Deliverables: - migration file - model registration ## B. Health Check Engine ### B1. Define health check service Add a new service module, for example: - `backend/app/services/datasource_health.py` Responsibilities: - resolve effective endpoint - execute deterministic check - classify result - persist health record ### B2. Define shared result schema Create a typed result object, for example: - `HealthCheckResult` Suggested fields: - `status` - `endpoint_checked` - `http_status` - `content_type` - `latency_ms` - `sample_count` - `error_message` - `details` ### B3. Implement base deterministic checks Every datasource should go through a minimal baseline check: 1. resolve endpoint 2. perform request 3. measure latency 4. inspect status code 5. inspect content type 6. inspect body shape Classification rules: - network error -> `failed` - HTTP 401/403 -> `auth_required` - HTTP 429 -> `rate_limited` - HTTP 404/410 -> `failed` - parse failure -> `schema_changed` - zero or suspiciously empty results -> `empty_result` or `degraded` - valid parse -> `healthy` ### B4. Add collector-aware adapters Some collectors do not use the same fetch semantics. Add adapter profiles such as: - `http_json` - `http_csv` - `html_scrape` - `stream_probe` - `auth_session_http` Initial mapping suggestion: - `huggingface`, `peeringdb`, `cloudflare` -> `http_json` - `fao` -> `http_csv` - `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape` - `ris_live` -> `stream_probe` - `spacetrack` -> `auth_session_http` ### B5. Add sample validation hooks For each adapter, add a lightweight validation rule. Examples: - JSON array length > 0 - CSV rows > 1 - HTML page contains expected table or script patterns - stream source yields at least one valid event within timeout ## C. Persistence and Query Layer ### C1. Save every check run Each health check should insert a record. Do not overwrite history in Stage 2. ### C2. Add latest-health query helpers Add helper functions to fetch: - latest health record by datasource - latest failed health record - recent health history ### C3. Optional retention policy For Stage 2, retention can be deferred. If desired, keep only: - last N records per datasource ## D. API Layer ### D1. Add health list endpoint Suggested endpoint: - `GET /api/v1/datasources/health` Returns: - datasource id - collector name - current endpoint - latest health status - last checked time - short reason ### D2. Add per-datasource health detail endpoint Suggested endpoint: - `GET /api/v1/datasources/{id}/health` Returns: - latest record - recent history - detailed classification fields ### D3. Add manual health trigger endpoint Suggested endpoint: - `POST /api/v1/datasources/{id}/health-check` Behavior: - run a health check now - persist the result - return the new record ### D4. Add bulk health trigger endpoint Suggested endpoint: - `POST /api/v1/datasources/health-check-all` Behavior: - enqueue or run health checks for all active datasources ## E. Scheduling ### E1. Add health scheduler task Decide scheduling strategy. Recommended first version: - run collector jobs and health checks separately - health checks run on a lower frequency Suggested frequency: - every 6h or 12h for most datasources - optionally on-demand only in the very first cut ### E2. Prevent health check collision with collection Rules: - health checks should not disrupt active collection - they should use light requests - if a collector is currently running, health check may: - skip - or use a lightweight endpoint probe only ## F. Frontend ### F1. Add health columns to datasource list Update: - `frontend/src/pages/DataSources/DataSources.tsx` Suggested new columns: - health status - last checked - reason summary ### F2. Add manual health check action Per datasource: - button or dropdown action: - `健康检查` ### F3. Add health detail drawer or modal Show: - endpoint checked - status - HTTP status - content type - sample count - error message - last few results ### F4. Add basic visual language Suggested colors: - green -> healthy - yellow -> degraded - orange -> rate-limited / auth-required - red -> failed / schema-changed ## G. Observability ### G1. Structured logging Every health check should log: - datasource id - collector name - endpoint - status - latency - failure class ### G2. Optional metrics If metrics are added later, useful counters include: - health checks total - health checks failed - schema changes detected - rate limited checks ## H. Tests ### H1. Unit tests Add tests for: - status classification - content type classification - adapter behavior - latest-health query helpers ### H2. API tests Add tests for: - health endpoints require auth - manual trigger endpoint works - list endpoint returns latest status ### H3. Failure-path tests Add coverage for: - HTTP 404 - HTTP 429 - invalid JSON - empty response - parse mismatch ## Suggested File Plan Possible implementation files: - `backend/app/models/datasource_health_check.py` - `backend/app/services/datasource_health.py` - `backend/app/schemas/datasource_health.py` - `backend/app/api/v1/datasource_health.py` - migration file under the project migration system Likely touched existing files: - `backend/app/api/main.py` - `frontend/src/pages/DataSources/DataSources.tsx` - `backend/tests/test_api.py` ## Suggested Execution Order 1. Add model and migration 2. Add service and result schema 3. Add deterministic adapters 4. Add manual trigger API 5. Add list/detail API 6. Add frontend visibility 7. Add scheduled checks 8. Expand tests ## Minimal First Milestone If we want the fastest useful slice, do this first: 1. health table 2. deterministic check service 3. manual per-datasource health check API 4. latest health list API 5. frontend status badge column That is enough to start operating the system and will provide the input layer for Stage 3. ## Dependency On Later Stages Stage 2 outputs become direct inputs for Stage 3. Specifically: - failed or schema-changed health records become agent triggers - health history becomes repair context - endpoint_checked becomes proposal baseline ## Success Criteria Stage 2 is done when: - every active datasource can be health-checked deterministically - the latest health state is visible in API and UI - operators can manually trigger checks - failures are categorized into stable machine-readable statuses - no LLM is required for core health visibility