release: bump version to 0.44.0
This commit is contained in:
@@ -84,6 +84,8 @@ async def run(self, db):
|
||||
| HuggingFace Spaces | space | Demo应用 | 1天 |
|
||||
| PeeringDB | ixp/network/facility | 互联网交换点/网络/机房 | 1-2天 |
|
||||
| TeleGeography | submarine_cable | 海底光缆信息 | 7天 |
|
||||
| Space-Track TLE | satellite_tle | 卫星轨道 TLE 数据 | 依采集器配置 |
|
||||
| BarentsWatch AIS | vessel | 船只位置、航速、航向、MMSI 等 AIS 数据 | 依采集器配置 |
|
||||
|
||||
## 四、数据格式 (统一存储到 CollectedData 表)
|
||||
|
||||
@@ -200,6 +202,30 @@ def start_scheduler():
|
||||
|
||||
**核心文件**: `backend/app/services/scheduler.py`
|
||||
|
||||
### 成功采集与连接状态
|
||||
|
||||
内置采集器成功采集后,调度器会记录当前有效配置已通过连接验证:
|
||||
|
||||
```python
|
||||
if datasource.last_status == "success":
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
|
||||
checksum, _ = await build_builtin_connectivity_checksum(...)
|
||||
await save_connectivity_success(
|
||||
db,
|
||||
datasource.source,
|
||||
checksum,
|
||||
{"status_code": None},
|
||||
connected_by="collection",
|
||||
)
|
||||
```
|
||||
|
||||
这个记录用于控制台“采集器设置”中的连接状态判断:如果当前配置和成功采集时的 checksum 一致,就视为已连接,不要求用户再手动点击连接按钮。只有 endpoint、请求头、基础配置或凭证指纹变化时,才需要重新验证。
|
||||
|
||||
相关实现见:
|
||||
|
||||
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
|
||||
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
|
||||
|
||||
## 八、相关代码文件
|
||||
|
||||
```
|
||||
@@ -211,13 +237,86 @@ backend/app/services/collectors/
|
||||
├── epoch_ai.py # Epoch AI采集器
|
||||
├── huggingface.py # HuggingFace采集器
|
||||
├── peeringdb.py # PeeringDB采集器
|
||||
└── telegeraphy.py # TeleGeography海底光缆采集器
|
||||
├── telegeraphy.py # TeleGeography海底光缆采集器
|
||||
└── vessel_ais.py # BarentsWatch AIS 船只采集器
|
||||
|
||||
backend/app/models/
|
||||
└── collected_data.py # 统一数据模型
|
||||
```
|
||||
|
||||
## 九、数据使用场景
|
||||
## 九、凭证型采集器
|
||||
|
||||
部分采集器需要外部服务凭证,例如:
|
||||
|
||||
| 采集器 | credential provider | 凭证来源 |
|
||||
| --- | --- | --- |
|
||||
| `barentswatch_vessels` | `barentswatch` | 控制台采集器设置、环境变量、`~/.zshrc` |
|
||||
| `spacetrack_tle` | `spacetrack` | 环境变量、`~/.zshrc` |
|
||||
|
||||
### BarentsWatch AIS
|
||||
|
||||
BarentsWatch AIS 的凭证解析统一在:
|
||||
|
||||
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
|
||||
|
||||
`VesselAISCollector` 只负责采集和转换 AIS 数据,不再自己读取环境变量或拼 token 请求。它通过:
|
||||
|
||||
- `resolve_barentswatch_config()`
|
||||
- `fetch_barentswatch_access_token()`
|
||||
|
||||
获取运行时配置。
|
||||
|
||||
解析优先级:
|
||||
|
||||
1. `DataSourceConfig.auth_config`
|
||||
2. `DataSourceConfig.config`
|
||||
3. 环境变量
|
||||
4. `~/.zshrc`
|
||||
|
||||
支持变量:
|
||||
|
||||
```bash
|
||||
export BARENTSWATCH_CLIENT_ID="..."
|
||||
export BARENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
并兼容历史拼写:
|
||||
|
||||
```bash
|
||||
export BARRENTSWATCH_CLIENT_ID="..."
|
||||
export BARRENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
连接验证会先请求 `https://id.barentswatch.no/connect/token` 获取 `scope=ais` 的 access token,再用 `Authorization: Bearer <token>` 请求 AIS endpoint。
|
||||
|
||||
## 十、采集器设置与连接验证
|
||||
|
||||
控制台的“采集器设置”页提供所有内置采集器的 endpoint、请求头、超时、重试和凭证配置。连接验证不是只看前端按钮状态,而是由后端计算 checksum:
|
||||
|
||||
- endpoint
|
||||
- auth type
|
||||
- headers
|
||||
- config
|
||||
- credential provider
|
||||
- 凭证指纹
|
||||
|
||||
相关 API:
|
||||
|
||||
```http
|
||||
GET /api/v1/datasources/configs/all
|
||||
POST /api/v1/datasources/configs/builtin/connection-status
|
||||
POST /api/v1/datasources/configs/builtin/connect
|
||||
POST /api/v1/settings/integrations/barentswatch/connect
|
||||
GET /api/v1/settings/credential-guides/{provider}
|
||||
POST /api/v1/settings/credential-guides/{provider}/generate
|
||||
POST /api/v1/settings/credential-guides/{provider}/reset
|
||||
```
|
||||
|
||||
更多细节见:
|
||||
|
||||
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
|
||||
|
||||
## 十一、数据使用场景
|
||||
|
||||
采集的数据最终会:
|
||||
|
||||
@@ -225,7 +324,7 @@ backend/app/models/
|
||||
2. **态势分析** - 统计全球算力分布、增长趋势
|
||||
3. **告警系统** - 检测重要节点变化
|
||||
|
||||
## 十、采集器注册机制
|
||||
## 十二、采集器注册机制
|
||||
|
||||
采集器在应用启动时自动注册:
|
||||
|
||||
@@ -247,7 +346,7 @@ collector_registry.register(TeleGeographyCableSystemCollector())
|
||||
|
||||
**核心文件**: `backend/app/services/collectors/registry.py`
|
||||
|
||||
## 十一、触发采集
|
||||
## 十三、触发采集
|
||||
|
||||
### 方式一:定时触发
|
||||
系统启动时,APScheduler会自动根据各采集器的`frequency_hours`设置定时任务。
|
||||
|
||||
Reference in New Issue
Block a user