feat: configure collector endpoints and health plan
This commit is contained in:
linkong
486
docs/datasource-health-plan.md
Normal file
486
docs/datasource-health-plan.md
Normal file
@@ -0,0 +1,486 @@
|
||||
# Datasource Health Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines a phased plan for datasource health governance.
|
||||
|
||||
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
|
||||
|
||||
The key principle is:
|
||||
|
||||
- do not let runtime automation rewrite repository default config
|
||||
|
||||
Instead, split responsibilities across:
|
||||
|
||||
- default config
|
||||
- runtime overrides
|
||||
- health check records
|
||||
- agent-generated repair proposals
|
||||
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
|
||||
|
||||
These upstream dependencies can fail in several ways:
|
||||
|
||||
- endpoint becomes unreachable
|
||||
- endpoint still responds but schema changes
|
||||
- content-type changes
|
||||
- website shuts down or moves
|
||||
- mirror link disappears
|
||||
- HTML structure changes and scraping fails
|
||||
- endpoint requires a new path or new host
|
||||
|
||||
We want a system that can:
|
||||
|
||||
- detect datasource health degradation early
|
||||
- identify likely cause
|
||||
- search for updated endpoints when reasonable
|
||||
- apply safe runtime fixes without polluting default repo config
|
||||
- preserve auditability and rollback
|
||||
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. Default config is stable
|
||||
|
||||
- `backend/app/core/data_sources.yaml` remains the repository baseline.
|
||||
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
|
||||
|
||||
2. Runtime fixes are isolated
|
||||
|
||||
- Emergency or adaptive fixes should live in a runtime override layer.
|
||||
- Overrides should be reversible and auditable.
|
||||
|
||||
3. Deterministic checks come first
|
||||
|
||||
- Use normal programmatic health checks before using LLMs.
|
||||
- Only call an agent when deterministic checks indicate a meaningful failure.
|
||||
|
||||
4. Agents suggest before they mutate
|
||||
|
||||
- Agents should produce proposals with evidence and confidence.
|
||||
- Application of a proposal should be controlled by policy.
|
||||
|
||||
5. Every repair is attributable
|
||||
|
||||
- Store what changed, why, who or what suggested it, and when it was applied.
|
||||
|
||||
|
||||
## Configuration Layers
|
||||
|
||||
Recommended runtime precedence:
|
||||
|
||||
1. datasource endpoint override
|
||||
2. datasource DB endpoint override
|
||||
3. repository default YAML
|
||||
4. collector internal fallback logic
|
||||
|
||||
Definitions:
|
||||
|
||||
- repository default YAML:
|
||||
- `backend/app/core/data_sources.yaml`
|
||||
- versioned baseline
|
||||
- datasource DB endpoint override:
|
||||
- existing `DataSourceConfig.endpoint`
|
||||
- current runtime override entrypoint
|
||||
- datasource endpoint override:
|
||||
- a dedicated new override table
|
||||
- used for health-repair and proposal application
|
||||
- collector internal fallback logic:
|
||||
- final defensive fallback
|
||||
- should be minimized over time
|
||||
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### 1. Deterministic Health Checks
|
||||
|
||||
Each collector gets a health profile with checks such as:
|
||||
|
||||
- endpoint resolves
|
||||
- HTTP request succeeds
|
||||
- status code is acceptable
|
||||
- content-type is expected
|
||||
- body parses successfully
|
||||
- minimum structural fields exist
|
||||
- sample item count is plausible
|
||||
- latency is within threshold
|
||||
|
||||
Output states:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
|
||||
|
||||
### 2. Agent-Assisted Repair Discovery
|
||||
|
||||
Only triggered when deterministic health checks fail or return suspicious structure.
|
||||
|
||||
Agent responsibilities:
|
||||
|
||||
- search for current official endpoint or replacement path
|
||||
- inspect likely upstream documentation or landing pages
|
||||
- compare candidate endpoint output to collector expectations
|
||||
- produce a repair proposal with confidence and evidence
|
||||
|
||||
Agent should not directly modify repository defaults.
|
||||
|
||||
|
||||
### 3. Safe Runtime Repair Application
|
||||
|
||||
Repair proposals can be:
|
||||
|
||||
- reviewed manually
|
||||
- auto-applied only under strict low-risk policy
|
||||
|
||||
Auto-apply should be limited to cases like:
|
||||
|
||||
- same trusted domain
|
||||
- highly similar response structure
|
||||
- repeated successful verification
|
||||
- confidence above threshold
|
||||
|
||||
|
||||
## Phased Delivery Plan
|
||||
|
||||
## Phase 1: Deterministic Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- build health observability without automated repair
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource health check task runner
|
||||
- datasource health result persistence
|
||||
- endpoint reachability + parse checks
|
||||
- dashboard or API visibility into health status
|
||||
|
||||
Deliverables:
|
||||
|
||||
- health check service
|
||||
- health check record table
|
||||
- status endpoint
|
||||
- scheduled or manual check trigger
|
||||
|
||||
No agent usage yet.
|
||||
|
||||
|
||||
## Phase 2: Agent Repair Proposals
|
||||
|
||||
Goal:
|
||||
|
||||
- let agent investigate failing sources and propose updated endpoints
|
||||
|
||||
Scope:
|
||||
|
||||
- invoke agent only when datasource health is `failed` or `schema_changed`
|
||||
- web search + page inspection
|
||||
- candidate endpoint extraction
|
||||
- proposal persistence
|
||||
|
||||
Deliverables:
|
||||
|
||||
- repair proposal schema
|
||||
- proposal generation pipeline
|
||||
- confidence and evidence model
|
||||
- operator review view or API
|
||||
|
||||
Still no automatic config mutation.
|
||||
|
||||
|
||||
## Phase 3: Runtime Overrides
|
||||
|
||||
Goal:
|
||||
|
||||
- allow approved proposals to take effect safely at runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- add dedicated override storage
|
||||
- runtime resolution prefers override over default config
|
||||
- proposal application writes override only
|
||||
|
||||
Deliverables:
|
||||
|
||||
- endpoint override table
|
||||
- override-aware resolution logic
|
||||
- apply/reject endpoints
|
||||
- rollback endpoint
|
||||
|
||||
Repository default YAML remains untouched.
|
||||
|
||||
|
||||
## Phase 4: Limited Auto-Apply
|
||||
|
||||
Goal:
|
||||
|
||||
- safely automate a narrow slice of low-risk repairs
|
||||
|
||||
Scope:
|
||||
|
||||
- policy engine for auto-apply
|
||||
- same-domain or trusted-domain checks
|
||||
- structure validation
|
||||
- staged verification after apply
|
||||
|
||||
Deliverables:
|
||||
|
||||
- auto-apply rules
|
||||
- audit logs
|
||||
- automatic post-apply health verification
|
||||
- auto-disable or rollback on regression
|
||||
|
||||
|
||||
## Data Model Draft
|
||||
|
||||
### datasource_health_checks
|
||||
|
||||
Purpose:
|
||||
|
||||
- store each health evaluation result
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
`details` can store structured diagnostic data such as:
|
||||
|
||||
- parsed fields
|
||||
- schema mismatch summary
|
||||
- retry count
|
||||
- exception class
|
||||
|
||||
|
||||
### datasource_repair_proposals
|
||||
|
||||
Purpose:
|
||||
|
||||
- store agent-generated repair suggestions
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `old_endpoint`
|
||||
- `candidate_endpoint`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `evidence_urls`
|
||||
- `evidence_summary`
|
||||
- `status`
|
||||
- `created_by`
|
||||
- `created_at`
|
||||
- `reviewed_at`
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `proposed`
|
||||
- `approved`
|
||||
- `rejected`
|
||||
- `applied`
|
||||
- `expired`
|
||||
|
||||
|
||||
### datasource_endpoint_overrides
|
||||
|
||||
Purpose:
|
||||
|
||||
- runtime endpoint override layer
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint`
|
||||
- `reason`
|
||||
- `source`
|
||||
- `proposal_id`
|
||||
- `enabled`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
Suggested `source` values:
|
||||
|
||||
- `manual`
|
||||
- `health-agent`
|
||||
- `migration`
|
||||
|
||||
|
||||
## API Draft
|
||||
|
||||
### Health
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
### Repair proposals
|
||||
|
||||
- `GET /api/v1/datasources/{id}/repair-proposals`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
|
||||
|
||||
### Overrides
|
||||
|
||||
- `GET /api/v1/datasources/{id}/overrides`
|
||||
- `POST /api/v1/datasources/{id}/overrides`
|
||||
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
|
||||
|
||||
## Agent Contract Draft
|
||||
|
||||
When deterministic health fails, the agent should receive:
|
||||
|
||||
- datasource name
|
||||
- collector name
|
||||
- current endpoint
|
||||
- current failure mode
|
||||
- expected response shape summary
|
||||
- known trusted domains
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "proposal",
|
||||
"candidate_endpoint": "https://example.com/api/v2/data",
|
||||
"confidence": 0.86,
|
||||
"reason": "Official docs now point to v2 endpoint",
|
||||
"evidence_urls": [
|
||||
"https://example.com/docs/api",
|
||||
"https://example.com/changelog"
|
||||
],
|
||||
"notes": "Response shape appears compatible after light field remapping"
|
||||
}
|
||||
```
|
||||
|
||||
The agent should never output "rewrite the default yaml" as its primary action.
|
||||
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
### Risk: wrong endpoint chosen by agent
|
||||
|
||||
Mitigation:
|
||||
|
||||
- use trusted-domain allowlists
|
||||
- require evidence URLs
|
||||
- require confidence threshold
|
||||
- add manual review for medium-risk sources
|
||||
|
||||
|
||||
### Risk: endpoint responds but schema silently changed
|
||||
|
||||
Mitigation:
|
||||
|
||||
- deterministic schema checks
|
||||
- parse and sample validation
|
||||
- content-type checks
|
||||
- collector-specific required fields
|
||||
|
||||
|
||||
### Risk: automatic runtime override causes hidden drift
|
||||
|
||||
Mitigation:
|
||||
|
||||
- store all overrides explicitly
|
||||
- mark source of override
|
||||
- keep default YAML unchanged
|
||||
- expose active overrides in API/UI
|
||||
|
||||
|
||||
### Risk: persistent bad override breaks data collection
|
||||
|
||||
Mitigation:
|
||||
|
||||
- allow rollback
|
||||
- keep parent/default endpoint visible
|
||||
- re-run verification after apply
|
||||
- auto-disable override on repeated failure
|
||||
|
||||
|
||||
## Operational Policy Recommendations
|
||||
|
||||
1. Do not auto-apply for high-value or high-fragility sources initially.
|
||||
|
||||
2. Use manual approval for:
|
||||
|
||||
- scraped HTML sources
|
||||
- unofficial mirrors
|
||||
- sources with auth or rate-limit complexity
|
||||
- sources with legal or trust ambiguity
|
||||
|
||||
3. Allow auto-apply only for:
|
||||
|
||||
- same-domain version bumps
|
||||
- obvious official migration paths
|
||||
- repeated passing verification
|
||||
|
||||
4. Expose health + proposal + override state together in one operator view.
|
||||
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Phase 1
|
||||
- health result table
|
||||
- deterministic checks
|
||||
- API and UI visibility
|
||||
|
||||
2. Phase 2
|
||||
- proposal table
|
||||
- agent prompt/output contract
|
||||
- proposal generation job
|
||||
|
||||
3. Phase 3
|
||||
- runtime override table
|
||||
- resolver precedence update
|
||||
- apply/reject endpoints
|
||||
|
||||
4. Phase 4
|
||||
- auto-apply rules
|
||||
- rollback policy
|
||||
- operator automation
|
||||
|
||||
|
||||
## Out Of Scope For The First Iteration
|
||||
|
||||
- direct automatic mutation of repository default YAML
|
||||
- automatic git commits by repair agents
|
||||
- unrestricted autonomous endpoint replacement
|
||||
- fully generalized schema remapping engine
|
||||
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted results
|
||||
- manual visibility
|
||||
- no automatic repair
|
||||
|
||||
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.
|
||||
Reference in New Issue
Block a user