453 lines
25 KiB
Markdown
453 lines
25 KiB
Markdown
# Data Collectors
|
|
|
|
## I. System Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Data Collection Architecture │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ TOP500 │ │ Epoch AI │ │ HuggingFace │ │
|
|
│ │ Collector │ │ Collector │ │ Collector │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
|
│ │ │ │ │
|
|
│ └───────────────────┼───────────────────┘ │
|
|
│ ▼ │
|
|
│ ┌─────────────────────┐ │
|
|
│ │ BaseCollector │◄── Base class (unified) │
|
|
│ │ run() method │ │
|
|
│ └─────────┬───────────┘ │
|
|
│ │ │
|
|
│ ┌─────────────────┼─────────────────┐ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
|
|
│ │ fetch() │ │transform()│ │ _save_data│ │
|
|
│ │ raw data │ │ transform │ │ save to DB│ │
|
|
│ └───────────┘ └───────────┘ └───────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌─────────────────────┐ │
|
|
│ │ CollectedData table│◄── Unified storage │
|
|
│ └─────────────────────┘ │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ Scheduler (APScheduler) │ │
|
|
│ │ Scheduled tasks: every 4h/6h/12h/1d auto-execute │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## II. Pipeline
|
|
|
|
```python
|
|
# 1. Scheduler triggers (scheduled or manual)
|
|
# ↓
|
|
|
|
# 2. run() executes the full pipeline
|
|
async def run(self, db):
|
|
# 2.1 Check if collector is enabled
|
|
if not collector_registry.is_active(self.name):
|
|
return {"status": "skipped"}
|
|
|
|
# 2.2 Record task start
|
|
task = CollectionTask(status="running")
|
|
db.add(task)
|
|
await db.commit()
|
|
|
|
# 2.3 FETCH — get raw data (implemented by subclass)
|
|
raw_data = await self.fetch()
|
|
|
|
# 2.4 TRANSFORM — convert to unified format
|
|
data = self.transform(raw_data)
|
|
|
|
# 2.5 SAVE — persist to database
|
|
records_count = await self._save_data(db, data)
|
|
|
|
# 2.6 Record task completion
|
|
task.status = "success"
|
|
task.records_processed = records_count
|
|
await db.commit()
|
|
```
|
|
|
|
**Core file**: `backend/app/services/collectors/base.py`
|
|
|
|
Manual trigger, data clearing, and cache clearing now enter the PostgreSQL data job queue. `collection_tasks` remains the task ledger. Collectors only own `fetch -> transform -> save`; the `data_jobs.py` worker claims `collect` / `clear_data` / `clear_cache` / `earth_refresh` jobs and writes progress back. Earth layer refresh relationships live in `earth_layer_adapters.py`; do not hand-code cache invalidation or WebSocket broadcasts inside individual collectors or buttons.
|
|
|
|
Data deletion runs in batches so AIS-scale tables are not locked by one huge statement. A `clear_data` job clears `collected_data`, then source-specific AIS derived tables, and broadcasts `records_processed` as it goes; the console queue renders only user-facing text such as `Deleting data` and `Delete complete`, while internal table names remain in logs and raw task details. After AIS cleanup, the backend runs `ANALYZE ais_raw_observations` so datasource-list estimates converge quickly. The datasource directory uses PostgreSQL statistics for AIS record counts by default to avoid a cold-start `count(*)`; opening a single datasource detail row requests the exact count for that source.
|
|
|
|
The CelesTrak TLE collector prefers the complete `active` catalog. If CelesTrak returns the "GP data has not updated" HTTP 403, the backend first reuses the active raw download cache under `$PLANET_CACHE_DIR/downloads/celestrak`; if that cache is missing, it enters fallback group mode. Fallback group mode uses only currently valid public CelesTrak groups, including `starlink`, `gps-ops`, `galileo`, `glo-ops`, `beidou`, `geo`, `iridium-next`, `stations`, `visual`, `weather`, `science`, `cubesat`, `amateur`, and `last-30-days`. Disaster-recovery fallback prefers valid local group caches before touching the network, so small-group update windows or unreliable HEAD metadata do not incorrectly fail recovery. Console `Clear Data` and `Clear Cache` jobs only touch database rows, Earth layer cache, and dashboard cache; they do not remove this raw download cache.
|
|
|
|
## III. Collector List
|
|
|
|
| Collector | Data type | Content | Frequency |
|
|
|-----------|-----------|---------|-----------|
|
|
| TOP500 | supercomputer | Global supercomputer rankings (compute, performance) | 4 hours |
|
|
| Epoch AI | gpu_cluster | GPU compute cluster info | 6 hours |
|
|
| HuggingFace Models | model | AI model information | 12 hours |
|
|
| HuggingFace Datasets | dataset | Dataset information | 12 hours |
|
|
| HuggingFace Spaces | space | Demo applications | 1 day |
|
|
| PeeringDB | ixp/network/facility | Internet exchange points / networks / facilities | 1-2 days |
|
|
| TeleGeography | submarine_cable | Submarine cable information | 7 days |
|
|
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
|
|
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
|
|
|
|
AIS vessel collectors use a different persistence path from regular `CollectedData` collectors. BarentsWatch, AISStream, and custom `vessel_ais` sources write into the AIS raw observation layer and also upsert `vessel_current_state`: one row per MMSI with the latest position, speed, course, navigation state, vessel type, name, and source metadata. This preserves raw observation history for tracks, audit, and situational analysis while letting the Earth vessel layer read the current-state table instead of scanning historical AIS rows.
|
|
|
|
`vessel_current_state` only lets newer observations overwrite dynamic position fields; static fields such as name and vessel type are merged by non-empty value and source priority. Earth snapshots return only vessels inside the freshness window, keeping high-frequency AIS history out of the display path.
|
|
|
|
Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and `/api/v1/earth/boundaries/*` builds `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to `CollectedData`.
|
|
|
|
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
|
|
|
|
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
|
|
|
|
- `Collectors`: endpoint, authentication, headers, base parameters, enabled state, and credential guides.
|
|
- `Collection Schedule`: scheduler state and task controls.
|
|
- `Collection History / Snapshots`: history grouped by collector, with a detail-side snapshot selector for versions.
|
|
|
|
Snapshot lists should not show every snapshot of the same collector as separate top-level records. The top-level list selects a collector; the detail area switches between time versions.
|
|
|
|
Credential guides are maintained by `backend/app/services/credential_guides.py`. The console uses read / generate / reset actions to load or create Markdown instructions. The frontend should render the guide Markdown for operators, not expose generation prompts or raw metadata.
|
|
|
|
## IV. Data Format (stored in CollectedData table)
|
|
|
|
```python
|
|
# Each collector's parse_response() return format
|
|
{
|
|
"source_id": "top500_1", # Original system ID (required)
|
|
"name": "El Capitan", # Name (required)
|
|
"description": "System desc...", # Description
|
|
"country": "United States", # Country
|
|
"city": "Livermore, CA", # City
|
|
"latitude": "37.6819", # Latitude (string)
|
|
"longitude": "-121.7681", # Longitude (string)
|
|
"value": "1742.00", # Performance value (e.g. compute)
|
|
"unit": "PFlop/s", # Unit
|
|
"metadata": { # Extra data (JSON)
|
|
"rank": 1,
|
|
"r_peak": 2746.38,
|
|
"cores": 11039616
|
|
},
|
|
"reference_date": "2025-11-01" # Data reference date
|
|
}
|
|
```
|
|
|
|
## V. Database Schema
|
|
|
|
**CollectedData table** (`collected_data`)
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| id | SERIAL | Primary key |
|
|
| source | VARCHAR(100) | Data source name (top500, huggingface, etc.) |
|
|
| source_id | VARCHAR(100) | Original data ID |
|
|
| data_type | VARCHAR(50) | Data type (supercomputer, model, etc.) |
|
|
| name | VARCHAR(500) | Name |
|
|
| title | VARCHAR(500) | Title |
|
|
| description | TEXT | Description |
|
|
| country | VARCHAR(100) | Country |
|
|
| city | VARCHAR(100) | City |
|
|
| latitude | VARCHAR(50) | Latitude |
|
|
| longitude | VARCHAR(50) | Longitude |
|
|
| value | VARCHAR(100) | Performance value |
|
|
| unit | VARCHAR(20) | Unit |
|
|
| metadata | JSONB | Extra metadata |
|
|
| collected_at | TIMESTAMP | Collection time |
|
|
| reference_date | TIMESTAMP | Data reference date |
|
|
| is_valid | INTEGER | Whether valid |
|
|
|
|
**Core file**: `backend/app/models/collected_data.py`
|
|
|
|
## VI. TOP500 Collector Example (full pipeline)
|
|
|
|
```python
|
|
# 1. fetch() — get HTML from the web
|
|
async def fetch(self):
|
|
url = "https://top500.org/lists/top500/list/2025/11/"
|
|
response = await client.get(url)
|
|
return response.text # returns HTML
|
|
|
|
# 2. parse_response() — parse HTML into unified format
|
|
def parse_response(self, html):
|
|
soup = BeautifulSoup(html, "html.parser")
|
|
table = soup.find("table")
|
|
|
|
for row in table.find_all("tr")[1:]: # skip header
|
|
cells = row.find_all("td")
|
|
|
|
entry = {
|
|
"source_id": f"top500_{cells[0].text}",
|
|
"name": cells[1].text.strip(),
|
|
"country": cells[2].text.strip(),
|
|
"city": "",
|
|
"latitude": "",
|
|
"longitude": "",
|
|
"value": "1742.00",
|
|
"unit": "PFlop/s",
|
|
"metadata": {
|
|
"rank": 1,
|
|
"cores": "11340000"
|
|
},
|
|
"reference_date": "2025-11-01"
|
|
}
|
|
data.append(entry)
|
|
|
|
return data
|
|
|
|
# 3. run() automatically calls _save_data() to save to database
|
|
```
|
|
|
|
**Core file**: `backend/app/services/collectors/top500.py`
|
|
|
|
## VII. Scheduler
|
|
|
|
```python
|
|
# Register all collectors into scheduled tasks at startup
|
|
def start_scheduler():
|
|
for name, collector in collectors.items():
|
|
if collector_registry.is_active(name):
|
|
scheduler.add_job(
|
|
run_collector_task,
|
|
trigger=IntervalTrigger(hours=collector.frequency_hours),
|
|
id=name,
|
|
name=name
|
|
)
|
|
```
|
|
|
|
| Collector | Frequency |
|
|
|-----------|-----------|
|
|
| TOP500 | Every 4 hours |
|
|
| Epoch AI | Every 6 hours |
|
|
| HuggingFace | Every 12 hours |
|
|
| PeeringDB | Every 1-2 days |
|
|
| TeleGeography | Every 7 days |
|
|
|
|
**Core file**: `backend/app/services/scheduler.py`
|
|
|
|
## VIII. Code Files
|
|
|
|
```
|
|
backend/app/services/collectors/
|
|
├── base.py # Base class: run() pipeline, _save_data() persistence
|
|
├── registry.py # Collector registry
|
|
├── scheduler.py # Scheduled task dispatch (APScheduler)
|
|
├── top500.py # TOP500 collector
|
|
├── epoch_ai.py # Epoch AI collector
|
|
├── huggingface.py # HuggingFace collector
|
|
├── peeringdb.py # PeeringDB collector
|
|
├── telegeraphy.py # TeleGeography submarine cable collector
|
|
├── vessel_ais.py # BarentsWatch AIS vessel collector
|
|
├── aisstream.py # AISStream WebSocket vessel collector
|
|
└── earth_boundaries.py # Earth boundary source verification and static tile artifact collector
|
|
|
|
backend/app/services/
|
|
├── custom_datasource_runtime.py # Custom REST / WebSocket mapping runtime
|
|
├── datasource_mapping.py # Deterministic field mapping and target writes
|
|
├── vessel_ais_aggregation.py # AIS raw observation writes and aggregate reads
|
|
├── vessel_aggregation_strategy.py # Multi-source field selection, freshness fallback, and conflict records
|
|
└── vessel_enrichment.py # Vessel profile enrichment cache
|
|
|
|
backend/app/models/
|
|
├── collected_data.py # Unified data model
|
|
└── vessel_enrichment.py # Vessel enrichment cache
|
|
```
|
|
|
|
## IX. Credentialed Collectors
|
|
|
|
Some collectors require external service credentials:
|
|
|
|
| Collector | Credential provider | Credential sources |
|
|
| --- | --- | --- |
|
|
| `barentswatch_vessels` | `barentswatch` | Console collector settings, environment variables, `~/.zshrc` |
|
|
| `aisstream_vessels` | `aisstream` | Console collector settings, environment variables, `~/.zshrc` for connectivity checks; save it in collector settings or inject it into the backend environment for collection |
|
|
| `spacetrack_tle` | `spacetrack` | Environment variables, `~/.zshrc` |
|
|
|
|
### BarentsWatch AIS
|
|
|
|
BarentsWatch AIS credential resolution is centralized in:
|
|
|
|
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
|
|
|
|
`VesselAISCollector` only collects and transforms AIS data. It no longer reads environment variables or builds token requests directly. It uses:
|
|
|
|
- `resolve_barentswatch_config()`
|
|
- `fetch_barentswatch_access_token()`
|
|
|
|
Resolution priority:
|
|
|
|
1. `DataSourceConfig.auth_config`
|
|
2. `DataSourceConfig.config`
|
|
3. Environment variables
|
|
4. `~/.zshrc`
|
|
|
|
Supported variables:
|
|
|
|
```bash
|
|
export BARENTSWATCH_CLIENT_ID="..."
|
|
export BARENTSWATCH_CLIENT_SECRET="..."
|
|
```
|
|
|
|
Historical misspellings are also supported:
|
|
|
|
```bash
|
|
export BARRENTSWATCH_CLIENT_ID="..."
|
|
export BARRENTSWATCH_CLIENT_SECRET="..."
|
|
```
|
|
|
|
Connectivity validation requests `https://id.barentswatch.no/connect/token` for an access token with `scope=ais`, then requests the AIS endpoint with `Authorization: Bearer <token>`.
|
|
|
|
### AISStream Realtime Vessels
|
|
|
|
AISStream uses the `wss://stream.aisstream.io/v0/stream` WebSocket endpoint. Its default runtime is a long-lived realtime collector rather than the traditional REST pattern of one request, progress to 100%, then completion.
|
|
|
|
Runtime configuration:
|
|
|
|
- `api_key`: read first from `DataSourceConfig.auth_config.api_key` or `config.api_key`; it can also come from the backend process environment variable `AISSTREAM_API_KEY`.
|
|
- `bounding_boxes`: AISStream subscription bounds. The default example is global `[[[-90, -180], [90, 180]]]`; demos and production runs should usually start with a smaller area.
|
|
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
|
|
- `streaming_enabled`: enables long-lived streaming by default; disabling it falls back to batch-style `fetch -> transform -> save`.
|
|
- `streaming_max_messages`: test-only stop limit. Non-zero values stop the stream after the requested number of messages.
|
|
- `reconnect_delay_seconds` and `receive_timeout_seconds`: control reconnect delay and idle receive waits.
|
|
|
|
State semantics:
|
|
|
|
- `connecting`: connecting to AISStream.
|
|
- `streaming`: receiving realtime messages; `records_processed` means messages seen, usually without a fixed total or percentage.
|
|
- `reconnecting`: upstream or network interruption; the collector records `AISSourceHealth` and waits before reconnecting.
|
|
- `stopped` / `cancelled`: stopped by a test limit or user action.
|
|
|
|
AISStream connectivity validation reads the saved collector configuration, environment variables, and `AISSTREAM_API_KEY` in `~/.zshrc` through `datasource_connectivity.py`. For actual collection, the most reliable path is saving the API key in `Collection Management -> Collectors -> AISStream Vessels`; if the key only lives in `~/.zshrc`, confirm that the backend process inherited it.
|
|
|
|
The console manages AISStream from `/datasources -> Realtime Streams`, not from the normal finite collection progress bar. The realtime stream API aggregates runtime state, health, configuration preview, and raw observation counters:
|
|
|
|
```http
|
|
GET /api/v1/realtime-sources
|
|
POST /api/v1/realtime-sources/{source}/start
|
|
POST /api/v1/realtime-sources/{source}/stop
|
|
POST /api/v1/realtime-sources/{source}/restart
|
|
```
|
|
|
|
`aisstream_vessels` and custom `source_type=websocket` sources appear in that API. They do not participate in one-click collection percentages; the UI interprets them as long-lived services with message counters, lag, last success, and last error.
|
|
|
|
### AIS Raw Observations And Aggregation
|
|
|
|
AIS observations do not directly replace final vessel records. They are first saved as raw observations:
|
|
|
|
- `source` records the origin, such as `barentswatch_vessels`, `aisstream_vessels`, or a custom source name.
|
|
- `delivery_mode` captures realtime quality; `realtime_stream` outranks `polling`.
|
|
- `transport` records `websocket` or `http`.
|
|
- Dynamic fields such as position, speed, and course are selected by freshness and source priority.
|
|
- Static fields prefer non-empty values; conflicting candidates are recorded for detail and diagnostics views.
|
|
|
|
Earth vessel rendering now consumes the current-state snapshot endpoint:
|
|
|
|
```http
|
|
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
|
|
GET /api/v1/visualization/vessels/{mmsi}
|
|
GET /api/v1/visualization/vessels/{mmsi}/track
|
|
GET /api/v1/visualization/vessels/{mmsi}/conflicts
|
|
```
|
|
|
|
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, and caps `limit` at `5000`. The Earth frontend uses a global bbox for current state and does not refetch on camera viewport changes. The endpoint reads `vessel_current_state` and reports `diagnostics.source = "vessel_current_state"`. The old `/api/v1/visualization/geo/vessels` route has been removed.
|
|
|
|
High-frequency AIS updates must not become per-delta full-layer rebuilds. If Earth uses the `/ws` `vessels` channel, it should send low-frequency reload/dirty hints and let the frontend merge snapshot refreshes. Tracks and conflicts still read historical facts through the single-vessel APIs.
|
|
|
|
### Layer APIs And Global Stats
|
|
|
|
Earth is moving to two API families:
|
|
|
|
```http
|
|
GET /api/v1/data-products
|
|
GET /api/v1/data-products/{product_id}/status
|
|
GET /api/v1/layers/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/cables?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/landing-points?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/satellites?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/bgp/anomalies?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/bgp/incidents?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/bgp/collectors?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
```
|
|
|
|
`/api/v1/data-products/*` is for aggregate panels and keeps a global statistics scope independent of the map bbox. `/api/v1/layers/*` is for map rendering, requires `bbox` and `zoom`, defaults to `limit=1000`, and caps `limit` at `5000`; low zoom falls back to a smaller response cap and reports `degraded`, `truncated`, `limit_clamped`, and `stats_scope=viewport` in `diagnostics`. Non-vessel layers currently reuse the existing GeoJSON converters before the guard layer; future product-specific queries can push bbox filtering deeper.
|
|
|
|
## X. Collectors And Connectivity Validation
|
|
|
|
The console "Collectors" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:
|
|
|
|
- endpoint
|
|
- auth type
|
|
- headers
|
|
- config
|
|
- credential provider
|
|
- credential fingerprint
|
|
|
|
Related APIs:
|
|
|
|
```http
|
|
GET /api/v1/datasources/configs/all
|
|
POST /api/v1/datasources/configs/builtin/connection-status
|
|
POST /api/v1/datasources/configs/builtin/connect
|
|
POST /api/v1/settings/integrations/barentswatch/connect
|
|
GET /api/v1/settings/credential-guides/{provider}
|
|
POST /api/v1/settings/credential-guides/{provider}/generate
|
|
POST /api/v1/settings/credential-guides/{provider}/reset
|
|
```
|
|
|
|
See [Collectors and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md) for the full flow.
|
|
|
|
## XI. Data Usage
|
|
|
|
Collected data ultimately:
|
|
|
|
1. **Visualization** — displays supercomputers, GPU clusters, and submarine cables' geographic positions
|
|
2. **Situational analysis** — global compute distribution statistics and growth trends
|
|
3. **Alert system** — detects changes to important nodes
|
|
|
|
## XII. Collector Registration
|
|
|
|
Collectors are automatically registered at application startup:
|
|
|
|
```python
|
|
# backend/app/services/collectors/__init__.py
|
|
|
|
collector_registry.register(TOP500Collector())
|
|
collector_registry.register(EpochAIGPUCollector())
|
|
collector_registry.register(HuggingFaceModelCollector())
|
|
collector_registry.register(HuggingFaceDatasetCollector())
|
|
collector_registry.register(HuggingFaceSpacesCollector())
|
|
collector_registry.register(PeeringDBIXPCollector())
|
|
collector_registry.register(PeeringDBNetworkCollector())
|
|
collector_registry.register(PeeringDBFacilityCollector())
|
|
collector_registry.register(TeleGeographyCableCollector())
|
|
collector_registry.register(TeleGeographyLandingPointCollector())
|
|
collector_registry.register(TeleGeographyCableSystemCollector())
|
|
```
|
|
|
|
**Core file**: `backend/app/services/collectors/registry.py`
|
|
|
|
## XIII. Triggering Collection
|
|
|
|
### Method 1: Scheduled
|
|
|
|
At startup, APScheduler automatically creates scheduled tasks based on each collector's `frequency_hours` setting.
|
|
|
|
### Method 2: Manual API trigger
|
|
|
|
```bash
|
|
# Trigger TOP500 collection
|
|
curl -X POST http://localhost:8000/api/v1/datasources/1/trigger \
|
|
-H "Authorization: Bearer <token>"
|
|
```
|
|
|
|
Batch collection uses:
|
|
|
|
```http
|
|
POST /api/v1/datasources/trigger-batch
|
|
```
|
|
|
|
The request body may pass `source_ids` for selected rows. Without `source_ids`, the backend filters by `product`, `module`, `is_active`, `run_status`, `collected`, `credential_status`, and `q`. The endpoint skips disabled sources, sources already running without `force`, and sources still inside their frequency window, then returns `triggered`, `skipped`, and `failed` groups.
|
|
|
|
**Core file**: `backend/app/api/v1/datasources.py`
|