planet/docs/technical/en/data-job-earth-sync-architecture.md

# Data Jobs and Outbox Architecture

This document records the technical boundary for Planet v1 data jobs, database outbox, and Earth refresh. For each data product's business purpose and end-to-end flow, see [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md).

## Architecture Boundary

- PostgreSQL is the durable v1 job ledger and outbox. Kafka, Celery, and RQ are intentionally not part of v1.
- `collection_tasks` records collection, data clearing, cache clearing, and non-database Earth refresh jobs.
- `earth_data_change_events` records fact-table or derived-table changes and is the reliable source for Earth sync.
- `LISTEN/NOTIFY` is only the low-latency wakeup path; the listener still polls unconsumed outbox rows.
- Redis is mainly cache, auth helper, OTP / rate limit, temporary logs, and WebSocket support. It is not the durable queue.

## Database Change Sync

```mermaid
flowchart LR
  Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
  Trigger --> Outbox["earth_data_change_events"]
  Trigger --> Notify["planet_earth_data_changes"]
  Outbox --> Listener["earth_db_change_listener"]
  Notify --> Listener
  Listener --> Adapter["earth_layer_adapters"]
  Adapter --> Cache["cache invalidation"]
  Cache --> WS["earth_updates"]
```

1. Collection, clearing, location resolution, or projection jobs write fact or derived tables.
2. Statement triggers write outbox rows for `INSERT` / `UPDATE` / `DELETE`.
3. The listener wakes through notify or finds pending rows through polling.
4. The listener maps `table + source` to Earth layers, refresh strategy, and cache patterns through `earth_layer_adapters.py`.
5. The listener merges short-window same-layer events, invalidates cache, and broadcasts `earth_updates`.
6. The outbox row is marked consumed only after successful broadcast; failed rows stay retryable.

DB changes no longer create default `earth_refresh` jobs, so they are not blocked by long same-source collection or clearing jobs. `earth_refresh` remains for manual cache clearing and non-DB refresh hints.

## Data Job Queue

`collection_tasks` is the unified job ledger. Workers claim `queued` jobs with PostgreSQL `FOR UPDATE SKIP LOCKED`; write jobs for the same `source` run serially, while different sources may run in parallel.

| task_type | Purpose |
| --- | --- |
| `collect` | Run a built-in datasource collector |
| `clear_data` | Delete collected rows and declared derived rows for the source |
| `clear_cache` | Delete Earth / dashboard cache for the source |
| `earth_refresh` | Invalidate Earth layer cache and broadcast a refresh hint for non-DB changes |

API handlers only create jobs and return `task_id`. Execution, progress, cancellation, and terminal state are written back by workers and pushed to the frontend through the `datasource_tasks` channel.

Cancellation means “keep committed batches”: clicking stop marks the job as `cancelling` and cancels the in-memory coroutine. Already committed batches remain; unfinished batches follow the collector or cleanup rollback path.

## Earth Sync Event Model

The unified event model is `earth.layer.changed`:

```json
{
  "event": "earth.layer.changed",
  "action": "database_changed",
  "source": "celestrak_tle",
  "table": "collected_data",
  "operation": "DELETE",
  "layers": ["satellites"],
  "refresh_strategy": "clear_then_reload",
  "records_processed": 11125,
  "occurred_at": "2026-05-25T10:20:30Z"
}
```

| strategy | Purpose |
| --- | --- |
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |

APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.

## Layer Adapter Contract

`earth_layer_adapters.py` is the single registry for sources, derived tables, Earth layers, cache patterns, and refresh strategy. New layers should be added through an adapter entry, not through one-off button handlers, collector branches, or frontend special cases.

Each adapter must declare:

- Which source or table feeds which Earth layer.
- Which Earth cache key patterns must be invalidated.
- Which owned derived tables must be removed during `clear_data`.
- The default refresh strategy for that layer.

When a source is cleared, the `clear_data` job first deletes `collected_data.source = <source>`, then deletes adapter-owned derived rows. Direct derived-table edits also trigger the outbox, so background jobs, admin APIs, and SQL repair scripts reach Earth as long as they mutate fact or derived tables.

## Operations and Troubleshooting

Check whether outbox rows are piling up:

```sql
SELECT id, table_name, operation, source, occurred_at
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT 20;
```

Check that triggers exist:

```sql
SELECT tgname, tgrelid::regclass
FROM pg_trigger
WHERE tgname LIKE 'tr_planet_%_changed_%'
ORDER BY 2, 1;
```

Useful log events:

- `earth.db_changes.connected`: the listener connected to PostgreSQL and started listening.
- `earth.db_changes.outbox_polled`: polling found unconsumed outbox rows.
- `earth.db_changes.broadcasted`: an Earth refresh broadcast was produced.
- `data_job.started` / `data_job.completed`: job execution state.

If Earth does not update, check in order: fact table changed, outbox was consumed, adapter covers the `source/table`, the listener is online, frontend WebSocket is connected, and the visualization API returns HTTP 200 with either an empty collection or fresh data.

## Kafka-ready Boundaries

Business code avoids depending on a concrete queue implementation by preserving these boundaries:

- `JobQueue`: submit, claim, cancel, and complete data jobs.
- `DataChangeBus`: publish database fact changes.
- `EarthLayerAdapterRegistry`: declare source, layer, cache, and derived-data relationships.

Kafka becomes appropriate when:

- Several independent services must consume the same data-change stream.
- AIS, BGP, or sensor streams become sustained high-throughput inputs.
- Consumer groups, replay, and service decoupling are required.

Spark becomes appropriate when:

- Historical data reaches tens or hundreds of millions of rows and PostgreSQL aggregation becomes expensive.
- Cross-source, long-window, spatiotemporal analysis is needed.
- Raw data lands in Parquet / Iceberg / Delta and the system starts producing offline derived data products.

For second-level continuous stream processing, evaluate Flink first. Spark is a better fit for batch or micro-batch analytics.