100 lines
3.7 KiB
Markdown
100 lines
3.7 KiB
Markdown
# DataSources List API Performance Optimization
|
|
|
|
## Background
|
|
|
|
`GET /api/v1/datasources` is the core API for the Data Sources page. Slow responses directly block page rendering.
|
|
|
|
## Query Path Before Optimization
|
|
|
|
`_load_datasource_list_context` used to run these queries sequentially:
|
|
|
|
| Order | Function | Query | Bottleneck |
|
|
| --- | --- | --- | --- |
|
|
| 1 | `_load_latest_running_tasks` | `collection_tasks` window query; stale check depends on this result | Must be serial |
|
|
| 2 | `_load_latest_completed_tasks` | `collection_tasks` window query for latest completed tasks | Serial wait |
|
|
| 3 | `_load_datasource_data_counts` | `COUNT(*) GROUP BY source` on `collected_data` | Slow full-table scan |
|
|
| 4 | `_load_datasource_endpoint_overrides` | Simple `datasource_configs` SELECT | Serial wait |
|
|
|
|
## Phase 1: Parallelization
|
|
|
|
The independent queries 2, 3, and 4 were moved to `asyncio.gather` with separate sessions:
|
|
|
|
```python
|
|
async def _fetch_completed():
|
|
async with async_session_factory() as s:
|
|
return await _load_latest_completed_tasks(s, datasource_ids)
|
|
|
|
async def _fetch_counts():
|
|
async with async_session_factory() as s:
|
|
return await _load_datasource_data_counts(s, sources)
|
|
|
|
async def _fetch_overrides():
|
|
async with async_session_factory() as s:
|
|
return await _load_datasource_endpoint_overrides(s, sources)
|
|
|
|
completed_tasks, data_counts, endpoint_overrides = await asyncio.gather(
|
|
_fetch_completed(), _fetch_counts(), _fetch_overrides(),
|
|
)
|
|
```
|
|
|
|
SQLAlchemy `AsyncSession` does not support concurrent use from multiple coroutines, so every parallel branch needs its own session.
|
|
|
|
## Phase 2: Remove Heavy Queries
|
|
|
|
### Remove `_load_datasource_data_counts`
|
|
|
|
`data_count` was only used by the frontend to show an edge-case `(0 records)` hint in the latest collection column. It was not worth keeping a `COUNT(*) GROUP BY` full-table scan.
|
|
|
|
- Frontend `(0 records)` display logic was removed.
|
|
- `data_count` was removed from the `BuiltInDataSource` interface.
|
|
|
|
### Remove `_load_latest_completed_tasks`
|
|
|
|
`last_status` and `last_run_at` are already written to the `DataSource` model when collectors finish, so the list endpoint no longer needs to join `collection_tasks`:
|
|
|
|
```python
|
|
# Before: completed_tasks query required
|
|
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
|
|
last_status = datasource.last_status or (last_task.status if last_task else None)
|
|
|
|
# After: read model fields directly
|
|
last_run_at = datasource.last_run_at
|
|
last_status = datasource.last_status
|
|
```
|
|
|
|
`last_records_processed` was removed as well because it came from completed task rows and is not displayed in the list.
|
|
|
|
## Query Path After Optimization
|
|
|
|
```text
|
|
datasources SELECT -> required primary data
|
|
_load_latest_running_tasks -> required for running state and stale check
|
|
_load_datasource_endpoint_overrides -> required for endpoint overrides and collector settings display
|
|
```
|
|
|
|
The endpoint now runs three queries instead of five. The last two run sequentially because running tasks are needed for stale checks and endpoint overrides are lightweight.
|
|
|
|
## Frontend `triggerDatasource` Double Refresh Fix
|
|
|
|
`triggerDatasource` previously called `fetchData()` twice:
|
|
|
|
```typescript
|
|
// Before
|
|
} else {
|
|
window.setTimeout(() => { fetchData() }, 800)
|
|
}
|
|
fetchData()
|
|
|
|
// After: mutually exclusive
|
|
if (res.data.task_id) {
|
|
fetchData()
|
|
} else {
|
|
window.setTimeout(fetchData, 800)
|
|
}
|
|
```
|
|
|
|
## Related Files
|
|
|
|
- [datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py): `_load_datasource_list_context`, `list_datasources`
|
|
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx): `BuiltInDataSource`, `triggerDatasource`
|