224 lines
13 KiB
Markdown
224 lines
13 KiB
Markdown
# Shared Location Resolution Pipeline Development Guide
|
|
|
|
`backend/app/services/location/` is the shared abstraction for any "given a record, decide its lat/lon" workflow. Compute centers, BGP collectors, and BGP events now run on this pipeline. Future entities such as satellite ground stations, user-claimed points, and IXP facilities should plug in here instead of creating another geocoding path.
|
|
|
|
For the user workflow, see [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
|
|
|
|
## Design Goals
|
|
|
|
Historically compute centers had their own four-tier chain, BGP collectors used a hard-coded dictionary, and BGP events inherited collector coordinates. These implementations did not share code, and new algorithms had no stable insertion point.
|
|
|
|
The refactored rules:
|
|
|
|
- Share the `LocationResolver` protocol and `LocationPipeline` orchestrator.
|
|
- Domain modules only build `LocationQuery` and choose resolver order.
|
|
- New algorithms join by adding resolver classes, without changing ingestion, API, or frontend envelopes.
|
|
- Earth renders only city-level or better locations.
|
|
- Local JSON registries are not runtime candidate sources for compute centers or BGP collectors; persisted location facts live in database dimension tables.
|
|
|
|
## Core Interfaces
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class LocationQuery:
|
|
name: str | None
|
|
aliases: tuple[str, ...]
|
|
city: str | None
|
|
country: str | None
|
|
region: str | None
|
|
source_latitude: float | None
|
|
source_longitude: float | None
|
|
extra: Mapping[str, Any]
|
|
```
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class LocationCandidate:
|
|
latitude: float
|
|
longitude: float
|
|
display_name: str
|
|
precision: str
|
|
confidence: float
|
|
source: str
|
|
needs_confirmation: bool
|
|
matched_fields: tuple[str, ...]
|
|
suggested_registry_entry: dict | None
|
|
```
|
|
|
|
```python
|
|
class LocationResolver(Protocol):
|
|
name: str
|
|
def resolve(self, query: LocationQuery) -> ResolverOutput: ...
|
|
```
|
|
|
|
`LocationPipeline.collect_candidates()` returns sorted candidates plus `attempted_queries`; `resolve_best()` returns the best candidate with diagnostics. The default sort key ranks source, precision, and confidence, then deduplicates candidates with the same source and rounded coordinates.
|
|
|
|
## Built-In Resolvers
|
|
|
|
| Resolver | File | Responsibility |
|
|
| --- | --- | --- |
|
|
| `SourceCoordinatesResolver` | `resolvers/source_coordinates.py` | Emits `precision="precise"` when the record already has lat/lon |
|
|
| `RegistryResolver` | `resolvers/registry.py` | Legacy generic resolver; current compute-center and BGP runtime paths do not use it to generate candidates |
|
|
| `NominatimResolver` | `resolvers/nominatim.py` | Runs a domain query plan against Nominatim with LRU cache and rate limiting |
|
|
| `InheritFromAnotherEntityResolver` | `resolvers/inherit.py` | Wraps an externally resolved entity location as a candidate |
|
|
| `LocationLLMFallback` | `location/llm_fallback.py` | Generates a confirmation-required candidate through the current default AI Provider when user-triggered collection has no regular candidates |
|
|
|
|
Nominatim is the geocoding service in the OpenStreetMap ecosystem. Given a place name, city, country, organization, or facility query, it returns possible coordinates, a display name, and structured address fields. It is useful for turning city/facility text into candidate coordinates, but it is not an authoritative fact registry and can match same-name places or broad administrative areas. Planet therefore treats Nominatim output as confirmation-required candidates and uses it with caching and rate limiting.
|
|
|
|
`RegistryResolver` remains available for future controlled import scenarios, but it should not be reconnected as a hard-coded hint source for compute centers or BGP. Matching common fields such as `operator` or `city` was the main reason multiple entities could collapse onto the same point.
|
|
|
|
## Current Domain Pipelines
|
|
|
|
### Compute Centers
|
|
|
|
Entry points:
|
|
|
|
- [compute_center_locations.py](/home/ray/dev/linkong/planet/backend/app/services/compute_center_locations.py)
|
|
|
|
Resolver order:
|
|
|
|
```python
|
|
SourceCoordinatesResolver()
|
|
StoredComputeCenterLocationResolver()
|
|
```
|
|
|
|
The main map startup path is source coordinates first, then the database-backed current-location table. The table is `compute_center_locations`, keyed by `(source, source_id)`, and stores manually accepted locations or true coordinates migrated from source records. `init_db()` only migrates source records that already contain real coordinates; it does not import old hard-coded hints and does not run ROR, Nominatim, or LLM geocoding during startup.
|
|
|
|
Candidate collection is intentionally separate from rendering. `collect_location_candidates()` builds ROR and Nominatim/OpenStreetMap queries from source fields, but it does not emit the current `compute_center_locations` row as a candidate. If those regular candidates are empty, the API layer calls `LocationLLMFallback` through the current default AI Provider and only returns `source="llm_location_factcheck"` candidates with `needs_confirmation=true`. LLM candidates use a combined threshold made from the model self-score plus backend evidence scoring; when the LLM provides a credible city/country but no coordinates, the backend may fill city-level coordinates through Nominatim without increasing the evidence score. After a user accepts a candidate, the save endpoint upserts it into the dimension table; the next map refresh renders it through `StoredComputeCenterLocationResolver`.
|
|
|
|
`resolve_compute_center_location()`, `resolve_compute_center_location_full()`, and `collect_location_candidates()` remain the domain API. `visualization.py` consumes that API and no longer owns coordinate hints, country-centroid fallbacks, or Nominatim details.
|
|
|
|
GeoJSON output includes only `RENDERABLE_PRECISIONS`. Unresolved records are returned in `unresolved` with `failure_reason`, `attempted_queries`, `source_id`, `record_id`, and related diagnostics.
|
|
|
|
### BGP Collectors
|
|
|
|
Entry points:
|
|
|
|
- [bgp_collector_locations.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collector_locations.py)
|
|
- [bgp_collector_location.py](/home/ray/dev/linkong/planet/backend/app/models/bgp_collector_location.py)
|
|
|
|
Resolver order:
|
|
|
|
```python
|
|
SourceCoordinatesResolver()
|
|
StoredCollectorLocationResolver()
|
|
NominatimResolver(_bgp_collector_query_plan)
|
|
```
|
|
|
|
The 23 RIPE RIS collector coordinates moved from the old table into the `bgp_collector_locations` dimension table with `source=legacy_seed` and `needs_confirmation=true`. The legacy dictionary is still maintained from the DB-backed cache for compatibility; manual candidate collection uses stored site/city/country as context but does not emit stored rows as candidates. If Nominatim cannot produce a city-level candidate, the collection endpoint uses the current default AI Provider as an LLM factcheck fallback and returns a confirmation-required candidate instead of saving automatically.
|
|
|
|
### BGP Events
|
|
|
|
Entry point:
|
|
|
|
- [bgp_event_locations.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_event_locations.py)
|
|
|
|
Resolver order:
|
|
|
|
```python
|
|
SourceCoordinatesResolver()
|
|
InheritFromAnotherEntityResolver(_inherit_from_owning_collector)
|
|
```
|
|
|
|
Event inheritance uses a strict owning-collector lookup and does not run the full fuzzy collector registry. Future ASN facility, PrefixGeo, or PeeringDB resolvers can be inserted after inheritance.
|
|
|
|
## API Envelope
|
|
|
|
```http
|
|
POST /api/v1/visualization/compute-centers/{source_id}/collect-location
|
|
POST /api/v1/visualization/compute-centers/{source_id}/location
|
|
POST /api/v1/bgp/collectors/{collector_id}/collect-location
|
|
```
|
|
|
|
Both `collect-location` endpoints return the same envelope:
|
|
|
|
```json
|
|
{
|
|
"success": true,
|
|
"candidates": [],
|
|
"best_candidate": {},
|
|
"attempted_queries": [],
|
|
"context": {}
|
|
}
|
|
```
|
|
|
|
The LLM fallback only runs inside user-triggered `collect-location` requests, and only after regular candidates are empty. It does not run during `/geo/compute-centers` startup rendering, scheduled collection, or batch persistence, and it never writes directly to `compute_center_locations` or `bgp_collector_locations`. Internally it is no longer a single "strict JSON or fail" step. It first asks the LLM to factcheck the location; if the answer is not JSON, it makes a second normalization request that may only extract facts from the original text; if that still fails, it conservatively extracts a city/country pair from the prose. The backend then performs coordinate filling, combined scoring, and candidate creation through one shared path.
|
|
|
|
This lets an answer such as "DeepL Mercury is in Falun, Sweden" become a city-level candidate after backend Nominatim coordinate filling, and lets a prose first answer be normalized into JSON on the second pass. Regardless of the path, only `precise`, `site`, or `city` precision with non-zero coordinates and a sufficient combined score is converted to a candidate. Failed, low-score, country-only, or cityless responses stay as diagnostics.
|
|
|
|
The LLM-provided `confidence` is only the model's self-score. The backend recomputes a combined score and uses that value as the candidate `confidence`:
|
|
|
|
```text
|
|
combined =
|
|
0.25 * model_confidence
|
|
+ source_quality
|
|
+ entity_match
|
|
+ geography_match
|
|
+ precision_quality
|
|
+ name_location_hint
|
|
- conflict_penalty
|
|
- weak_evidence_penalty
|
|
```
|
|
|
|
Current component caps: authoritative/government/academic evidence can add up to `0.35`, reputable databases or news up to `0.25`, generic web evidence up to `0.15`; evidence that clearly names the queried entity can add `0.25`; city+country geography match adds `0.20`, country-only match adds `0.05`; precision adds `precise=0.15`, `site=0.12`, or `city=0.08`; `name_location_hint` adds signal when the entity name and candidate city overlap, such as `TAIPEI-1` and `Taipei`; explicit conflicts can subtract up to `0.45`; weak-evidence wording can subtract up to `0.30`, capped at `0.15` when entity and city/country match and no conflict is present. Candidates below `0.55` are rejected. This lets cases such as Alem.Cloud and TAIPEI-1 recover from a low model self-score when entity and city evidence align, while genuinely weak or conflicting evidence still fails.
|
|
|
|
`POST /api/v1/visualization/compute-centers/{source_id}/location` upserts the candidate selected by the frontend into `compute_center_locations`. Manual saves default to `needs_confirmation=false`, `verification_status="verified"`, and a `verified_at` timestamp. Future automated staging can pass `needs_confirmation=true` explicitly.
|
|
|
|
The frontend [info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js) renders the shared candidate list and preview events. The compute-center layer button shows an `unresolved` badge; clicking it opens the unresolved queue. Row-level `采集` only fetches candidates. Header-level `一键采用` walks the queue top-to-bottom, picks the highest-confidence candidate with valid coordinates, saves it, removes the row, renumbers the list, and dispatches `earth:compute-center-unresolved-count-change` so the badge updates immediately. When the batch finishes, `earth:compute-center-location-saved` refreshes the real layer.
|
|
|
|
If the remaining records have no city-level candidates, the batch must not invent coordinates. The UI keeps those rows and shows the backend `failure_reason` plus attempted queries.
|
|
|
|
## Adding A Resolver
|
|
|
|
A resolver only needs `name` and `resolve()`, returning `ResolverOutput`.
|
|
|
|
```python
|
|
class PeeringDBFacilityResolver:
|
|
name = "peeringdb_facility"
|
|
|
|
def __init__(self, client):
|
|
self._client = client
|
|
|
|
def resolve(self, query):
|
|
asn = query.extra.get("origin_asn")
|
|
if not asn:
|
|
return ResolverOutput()
|
|
return ResolverOutput(candidates=tuple(
|
|
LocationCandidate(
|
|
latitude=f.latitude,
|
|
longitude=f.longitude,
|
|
display_name=f.name,
|
|
precision="site",
|
|
confidence=0.78,
|
|
query=f"peeringdb::{asn}",
|
|
source=self.name,
|
|
source_note=f"PeeringDB facility for AS{asn}",
|
|
matched_fields=("origin_asn",),
|
|
needs_confirmation=False,
|
|
city=f.city,
|
|
country=f.country,
|
|
)
|
|
for f in self._client.facilities_for_asn(asn)
|
|
))
|
|
```
|
|
|
|
Wire it in:
|
|
|
|
```python
|
|
BGP_EVENT_PIPELINE = LocationPipeline([
|
|
SourceCoordinatesResolver(),
|
|
InheritFromAnotherEntityResolver(source_lookup=...),
|
|
PeeringDBFacilityResolver(client=peeringdb_client),
|
|
])
|
|
```
|
|
|
|
## Test Coverage
|
|
|
|
Relevant tests:
|
|
|
|
- [test_location_pipeline.py](/home/ray/dev/linkong/planet/backend/tests/test_location_pipeline.py)
|
|
- [test_bgp_collector_locations.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp_collector_locations.py)
|
|
- [test_visualization_compute_centers.py](/home/ray/dev/linkong/planet/backend/tests/test_visualization_compute_centers.py)
|
|
|
|
Coverage focuses on resolver pluggability, registry alias guards, BGP collector legacy dictionary compatibility, compute-center public API compatibility, and non-renderable locations being returned as `unresolved`.
|