feat(backend): Add cable graph service and data collectors
## Changelog ### New Features #### Cable Graph Service - Add cable_graph.py for finding shortest path between landing points - Implement haversine distance calculation for great circle distances - Support for dateline crossing (longitude normalization) - NetworkX-based graph for optimal path finding #### Data Collectors - Add ArcGISCableCollector for fetching submarine cable data from ArcGIS GeoJSON API - Add FAOLandingPointCollector for fetching landing point data from FAO CSV API ### Backend Changes #### API Updates - auth.py: Update authentication logic - datasources.py: Add datasource endpoints and management - visualization.py: Add visualization API endpoints - config.py: Update configuration settings - security.py: Improve security settings #### Models & Schemas - task.py: Update task model with new fields - token.py: Update token schema #### Services - collectors/base.py: Improve base collector with better error handling - collectors/__init__.py: Register new collectors - scheduler.py: Update scheduler logic - tasks/scheduler.py: Add task scheduling ### Frontend Changes - AppLayout.tsx: Improve layout component - index.css: Add global styles - DataSources.tsx: Enhance data sources management page - vite.config.ts: Add Vite configuration for earth module
This commit is contained in:
@@ -10,6 +10,9 @@ from app.services.collectors.registry import collector_registry
|
||||
|
||||
async def run_collector_task(collector_name: str) -> Dict[str, Any]:
|
||||
"""Run a single collector task"""
|
||||
from sqlalchemy import select
|
||||
from app.models.datasource import DataSource
|
||||
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
return {"status": "failed", "error": f"Collector {collector_name} not found"}
|
||||
@@ -18,32 +21,15 @@ async def run_collector_task(collector_name: str) -> Dict[str, Any]:
|
||||
return {"status": "skipped", "reason": "Collector is disabled"}
|
||||
|
||||
async with async_session_factory() as db:
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.datasource import DataSource
|
||||
|
||||
# Find datasource
|
||||
result = await db.execute(
|
||||
"SELECT id FROM data_sources WHERE collector_class = :class_name",
|
||||
{"class_name": f"{collector.__class__.__name__}"},
|
||||
select(DataSource.id).where(DataSource.collector_class == collector_name)
|
||||
)
|
||||
datasource = result.fetchone()
|
||||
datasource = result.scalar_one_or_none()
|
||||
|
||||
task = CollectionTask(
|
||||
datasource_id=datasource[0] if datasource else 0,
|
||||
status="running",
|
||||
started_at=datetime.utcnow(),
|
||||
)
|
||||
db.add(task)
|
||||
await db.commit()
|
||||
if datasource:
|
||||
collector._datasource_id = datasource
|
||||
|
||||
result = await collector.run(db)
|
||||
|
||||
task.status = result["status"]
|
||||
task.completed_at = datetime.utcnow()
|
||||
task.records_processed = result.get("records_processed", 0)
|
||||
task.error_message = result.get("error")
|
||||
await db.commit()
|
||||
|
||||
return result
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user