feat(backend): Add cable graph service and data collectors

## Changelog

### New Features

#### Cable Graph Service
- Add cable_graph.py for finding shortest path between landing points
- Implement haversine distance calculation for great circle distances
- Support for dateline crossing (longitude normalization)
- NetworkX-based graph for optimal path finding

#### Data Collectors
- Add ArcGISCableCollector for fetching submarine cable data from ArcGIS GeoJSON API
- Add FAOLandingPointCollector for fetching landing point data from FAO CSV API

### Backend Changes

#### API Updates
- auth.py: Update authentication logic
- datasources.py: Add datasource endpoints and management
- visualization.py: Add visualization API endpoints
- config.py: Update configuration settings
- security.py: Improve security settings

#### Models & Schemas
- task.py: Update task model with new fields
- token.py: Update token schema

#### Services
- collectors/base.py: Improve base collector with better error handling
- collectors/__init__.py: Register new collectors
- scheduler.py: Update scheduler logic
- tasks/scheduler.py: Add task scheduling

### Frontend Changes
- AppLayout.tsx: Improve layout component
- index.css: Add global styles
- DataSources.tsx: Enhance data sources management page
- vite.config.ts: Add Vite configuration for earth module
This commit is contained in:
rayd1o
2026-03-11 16:38:49 +08:00
parent 6cb4398f3a
commit aaae6a53c3
18 changed files with 990 additions and 146 deletions

View File

@@ -10,6 +10,9 @@ from app.services.collectors.registry import collector_registry
async def run_collector_task(collector_name: str) -> Dict[str, Any]:
"""Run a single collector task"""
from sqlalchemy import select
from app.models.datasource import DataSource
collector = collector_registry.get(collector_name)
if not collector:
return {"status": "failed", "error": f"Collector {collector_name} not found"}
@@ -18,32 +21,15 @@ async def run_collector_task(collector_name: str) -> Dict[str, Any]:
return {"status": "skipped", "reason": "Collector is disabled"}
async with async_session_factory() as db:
from app.models.task import CollectionTask
from app.models.datasource import DataSource
# Find datasource
result = await db.execute(
"SELECT id FROM data_sources WHERE collector_class = :class_name",
{"class_name": f"{collector.__class__.__name__}"},
select(DataSource.id).where(DataSource.collector_class == collector_name)
)
datasource = result.fetchone()
datasource = result.scalar_one_or_none()
task = CollectionTask(
datasource_id=datasource[0] if datasource else 0,
status="running",
started_at=datetime.utcnow(),
)
db.add(task)
await db.commit()
if datasource:
collector._datasource_id = datasource
result = await collector.run(db)
task.status = result["status"]
task.completed_at = datetime.utcnow()
task.records_processed = result.get("records_processed", 0)
task.error_message = result.get("error")
await db.commit()
return result