185 lines
15 KiB
Markdown
185 lines
15 KiB
Markdown
# Earth News Source Configuration
|
||
|
||
Earth situational news is served through `/api/v1/news/earth-feed`. News source configuration lives in `SystemSetting.category = "earth_news_sources"`; when no database configuration exists, the backend uses the built-in default sources as the fallback seed.
|
||
|
||
## Default Sources
|
||
|
||
The default set contains four groups:
|
||
|
||
- **News feeds**: BBC World, DW Top Stories, CNBC Business, BBC Business, Guardian Business, NPR Business, MarketWatch, TechCrunch, Retail Dive, PR Newswire Retail, 36Kr, Ebrun, and China NBS data releases.
|
||
- **Industry insight sources**: McKinsey Retail and Deloitte Retail.
|
||
- **Official data sources**: China NBS data releases, US Census Retail / E-Commerce, MOFCOM Data, MOFCOM e-commerce updates, and China e-commerce logistics index.
|
||
- **Lead sources**: BusinessWire Electronic Commerce; Google News is one aggregated source with feed children for global, Americas, Europe, Middle East / Africa, and Asia Pacific.
|
||
|
||
Config data sources are visible in Admin by default. If a source is not a stable RSS/Atom feed, it is kept disabled for automatic fetching until an administrator replaces it with a fetchable URL and enables it.
|
||
|
||
| Source | Type | Default state | Default category | Main tags | Purpose |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| BBC World | RSS | Enabled | Politics | `official_media`, `global` | Global public-news baseline |
|
||
| DW Top Stories | RSS | Enabled | Politics | `official_media`, `europe` | Europe and international baseline |
|
||
| CNBC Business | RSS | Enabled | Business | `business_news`, `us`, `global` | International business news |
|
||
| BBC Business / Guardian Business / NPR Business / MarketWatch | RSS | Enabled | Business / Finance | `business_news`, `finance` | UK / US business and finance baseline |
|
||
| TechCrunch / Retail Dive / PR Newswire Retail | RSS | Enabled | Technology / Business | `business_news`, `ecommerce`, `retail`, `press_release` | Technology, e-commerce, retail, and company announcements |
|
||
| 36Kr | RSS | Enabled | Business | `business_news`, `ecommerce`, `china` | China business, venture, and newsflash feeds; homepage is `https://www.36kr.com/`, the Feed Directory is `https://www.36kr.com/rss-center`, and feed children are the general, article, newsflash, and moment feeds |
|
||
| Ebrun | RSS | Enabled | E-commerce | `ecommerce`, `business_news`, `china`, `retail` | China e-commerce industry news; homepage is `https://www.ebrun.com/`, the Feed Directory is `https://www.ebrun.com/rss/`, and feed children are B2C, B2B, retail, O2O, service, data, and policy XML feeds |
|
||
| China NBS data releases | RSS | Enabled | E-commerce | `official_data`, `ecommerce`, `retail`, `china` | Official data release RSS; retail and online retail items are identified by category and importance rules |
|
||
| Google News | Aggregated | Enabled | Politics | `aggregated`, `low_stability` | One aggregated source with global, Americas, Europe, Middle East / Africa, and Asia Pacific feed children; lower priority than real RSS |
|
||
| BusinessWire Electronic Commerce | Reference | Disabled | E-commerce | `press_release`, `ecommerce`, `low_stability` | Corporate announcement leads |
|
||
| McKinsey Retail Insights | Reference | Disabled | Business | `industry_insight`, `retail` | Retail industry insight |
|
||
| Deloitte Retail | Reference | Disabled | Business | `industry_insight`, `retail` | Retail industry insight |
|
||
| US Census Retail / E-Commerce | Reference | Disabled | E-commerce | `official_data`, `ecommerce`, `retail`, `us` | US retail and e-commerce official data |
|
||
| MOFCOM Data | Reference | Disabled | Business | `official_data`, `china` | China commerce data |
|
||
| MOFCOM e-commerce updates | Reference | Disabled | E-commerce | `official_data`, `ecommerce`, `china` | China e-commerce policy and updates |
|
||
| China e-commerce logistics index | Reference | Disabled | E-commerce | `official_data`, `ecommerce`, `logistics`, `china` | Logistics fulfillment and e-commerce activity |
|
||
|
||
`Reference` means a reference link or future collector lead. It records a homepage, report page, or data page and does not participate in RSS/Atom fetching. This lets commercial and official sources enter Admin governance without letting non-feed pages break the live news feed.
|
||
|
||
The news source model has two levels:
|
||
|
||
- `source` is the brand or aggregator, such as 36Kr, Ebrun, Google News, or BBC.
|
||
- `homepage_url` is the source homepage, section page, or report page.
|
||
- `feed_directory_url` is the Feed Directory page, such as an RSS subscription center or feed index. It is for human inspection and is not fetched.
|
||
- `feeds` are the actual RSS, Atom, or Aggregated child entries under that source. Each feed child has `id / name / url / type / enabled / default_category / tags / priority`.
|
||
|
||
The backend iterates over every enabled feed child under the same source, fetches them independently, merges and deduplicates items, and writes per-feed diagnostics into `health.feed_results`. This is not a backup URL model: all four 36Kr subscription feeds, multiple Ebrun category XML feeds, and the five Google News regional RSS feeds can be enabled at the same time, and each feed can have its own default category and enabled state. HTML subscription-center pages belong in `feed_directory_url`, not in feed URLs. Every default enabled fetchable feed is tested item by item: RSS/Atom/Aggregated feeds must parse at least one item, while Reference sources only retain a reference URL and future collector lead.
|
||
|
||
Items that still remain Reference are not treated as broken feeds; no stable directly consumable RSS/Atom feed was verified:
|
||
|
||
- BusinessWire documents customizable RSS/Atom support, but the public pages do not expose a stable industry feed URL; the e-commerce industry page is kept as an announcement lead.
|
||
- McKinsey and Deloitte retail insight pages are report/article collections, not public RSS feeds.
|
||
- The US Census press-release RSS is reachable, but its items currently have empty links; the Quarterly E-Commerce page remains an official data reference.
|
||
- MOFCOM data and China e-commerce logistics index pages do not expose stable RSS feeds yet; they should become dedicated collectors or be replaced with administrator-provided fetchable feeds.
|
||
|
||
## Source Property Tags and News Categories
|
||
|
||
News sources have `source_tags`, shown in Admin as source property tags. They describe the source, not the media name and not the content category of an individual story:
|
||
|
||
- `official_data`
|
||
- `business_news`
|
||
- `ecommerce`
|
||
- `finance`
|
||
- `retail`
|
||
- `logistics`
|
||
- `industry_insight`
|
||
- `press_release`
|
||
- `china`, `global`, `us`
|
||
- `aggregated`, `low_stability`
|
||
|
||
Each news item has exactly one primary `category`. Defaults are politics, business, e-commerce, finance, sports, technology, military, disaster, energy, society, culture, and other. `item_tags` are item-level secondary tags, such as cross-border e-commerce, live commerce, retail data, logistics fulfillment, platform governance, AI, semiconductor, election, oil price, football, and supply chain.
|
||
|
||
The primary category is generated by a rule-based scorer over title, summary, and source text. If the rules do not match, the feed child default category is used first, then the source default category. AI enrichment does not block news display.
|
||
|
||
## Importance
|
||
|
||
Each item includes:
|
||
|
||
- `importance_score`
|
||
- `importance_level`
|
||
- `importance_reasons`
|
||
- `market_impact`
|
||
|
||
Official data, e-commerce metrics, major platforms, and numeric business signals increase importance. Press releases start with a lower baseline and rise only when they match stronger platform, amount, M&A, or regulatory signals.
|
||
|
||
Importance levels are fixed: `low` 0–34, `medium` 35–59, `high` 60–79, and `critical` 80–100. Category, importance, and Breaking decisions are centralized in `earth_news_classification.py`. Category keys and tags remain configurable, while importance and Breaking protocol states use shared enums. Databases and APIs continue to store compatible lowercase strings.
|
||
|
||
## Configuration and Cache
|
||
|
||
`GET /api/v1/earth/news-sources` returns the default or saved configuration. `PUT /api/v1/earth/news-sources` saves it, increments `cache_version`, and clears the process region cache. `POST /api/v1/earth/news-sources/reset` restores defaults. `POST /api/v1/earth/news-sources/test` tests one RSS/Atom/Aggregated source without writing news items.
|
||
|
||
## Feed Query and Category Filtering
|
||
|
||
The Web Earth client and UE client both consume `GET /api/v1/news/earth-feed`. The endpoint supports server-side filtering, so clients do not need to fetch the full list and apply the primary category filter locally.
|
||
|
||
- `lat` / `lon`: infer the active region from the current view, used by the Web Earth client.
|
||
- `region`: explicitly select a region for UE or service integrations. Supported values include `global`, `americas`, `europe`, `asia-pacific`, and `middle-east-africa`. `global` is an aggregate view and can include every region; non-global regions include only their own region plus `global` sources.
|
||
- `categories`: comma-separated news category keys, for example `business,ecommerce`. Omit it when all categories are selected.
|
||
- `locale`: display locale, currently `zh-CN` or `en-US`, defaulting to `zh-CN`. Chinese RSS items are stored as Chinese source content and enriched with `en-US`; English RSS items are enriched with `zh-CN`.
|
||
|
||
Examples:
|
||
|
||
```http
|
||
GET /api/v1/news/earth-feed?region=europe&categories=business,ecommerce
|
||
GET /api/v1/news/earth-feed?lat=48&lon=10&categories=technology
|
||
GET /api/v1/news/earth-feed?region=global&categories=business,ecommerce&locale=zh-CN
|
||
```
|
||
|
||
Unknown category or locale values return `422` with the allowed values. The response includes `filters`, which confirms the region, category, and locale filters applied by the backend. `items` and `cruise_items` use the same category filter set.
|
||
|
||
The Web Earth category chips only store the current browser preference; changing them triggers a new API request. UE should pass its selected categories through the `categories` query parameter and does not need to perform the primary filtering itself.
|
||
|
||
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next `earth-feed` request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.
|
||
|
||
## Breaking News Insertion
|
||
|
||
The news system keeps three separate decisions:
|
||
|
||
- **Category**: what the story is about, such as business, military, or disaster.
|
||
- **Importance**: whether the story has long-term value, stored as `importance_score / importance_level`.
|
||
- **Breaking**: whether the story must temporarily jump ahead, stored in `location_meta.news_meta.breaking_*`.
|
||
|
||
Breaking metadata does not add physical columns. It remains in `location_meta.news_meta`:
|
||
|
||
- `breaking_level`: `none / watch / breaking / critical`.
|
||
- `breaking_scope`: `regional / global`.
|
||
- `breaking_reasons`: rule or operator reasons.
|
||
- `breaking_source`: `rules / ai / manual / multi_source`.
|
||
- `breaking_confidence`: 0 to 1.
|
||
- `breaking_expires_at`: expiration timestamp.
|
||
|
||
The backend performs the ordering, so Web and UE clients do not need to reorder items. Active `critical`, `breaking`, and `watch` items appear before normal items in that order. Once expired, the item falls back to normal ordering without being deleted or changing its long-term importance score.
|
||
|
||
`breaking_scope = global` bypasses the active region and can appear in every regional feed. `regional` breaking follows the normal active-region-plus-global rule. The response `filters` includes `has_breaking` and `highest_breaking_level`, which the Earth client uses to apply restrained panel and card styling.
|
||
|
||
## Connectivity Monitoring
|
||
|
||
`POST /api/v1/earth/news-sources/test` tests one source and writes the result to `earth_news_sources.health[source_id]`. Normal RSS/Atom fetches update the same health map.
|
||
|
||
Health results include:
|
||
|
||
- `status`: `ok`, `empty`, `format_error`, `http_error`, `timeout`, `network_error`, or `reference`.
|
||
- `status_code`, `content_type`, `item_count`, `latency_ms`, `error`, and `fetched_at`.
|
||
- `feed_results`: per-feed diagnostics for multi-feed sources, including `feed_id`, `feed_name`, `feed_type`, `feed_url`, status, item count, and error.
|
||
|
||
Common diagnostics:
|
||
|
||
- HTML response: the configured URL is not an RSS/Atom feed, for example a web page listing RSS options.
|
||
- HTTP 403: the source or CDN rejected the crawler request.
|
||
- Reference: the source is a reference link only and must be converted to RSS, Atom, or Aggregated before fetch testing.
|
||
|
||
The Admin entry is `Earth Content -> News Sources`. It is not a raw whole-payload JSON editor. The UI has two layers:
|
||
|
||
- **News sources**: a left-side source list with filters for enabled, disabled, reference links, RSS/Atom/Aggregated, region, and source property tags; the right side edits one selected source and its feed child list.
|
||
- **Policy rules**: global source property tags, news categories, item tag rules, and default health policy. Advanced JSON is reserved for diagnostics, not the default edit path.
|
||
|
||
The single-source form is split into source information and feed children:
|
||
|
||
- Source information covers name, ID, region, homepage URL, Feed Directory URL, source type, enabled state, source property tags, importance weight, fetch interval, timeout, failure threshold, and circuit breaker.
|
||
- Feed children cover feed ID, name, real feed URL, type, enabled switch, default news category, priority, and feed tags. The `+` button under the feed child list creates a frontend-only draft; saving the source persists it, while canceling destroys the draft.
|
||
|
||
The per-source “test source” action tests all enabled feed children under the current source. The feed-row test action tests only that feed child. Both send to `/api/v1/earth/news-sources/test`, but the feed-row action submits the current source with only the selected feed child.
|
||
|
||
Reference links show that they only record a homepage, report page, or future collector lead and do not participate in RSS/Atom fetching. They can remain as commercial or official-data leads, but must be converted to RSS, Atom, or Aggregated with fetchable feed URLs before they can be enabled for fetching.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
Admin["Admin: Earth Content / News Sources"] --> Source["Source config"]
|
||
Source --> Feed["Feed children"]
|
||
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
|
||
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
|
||
|
||
Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||
NewsAPI --> Resolver["Source Resolver"]
|
||
Resolver --> Config
|
||
Resolver --> Cache["Region Feed Cache"]
|
||
Resolver --> Fetcher["RSS / Atom Fetcher"]
|
||
Fetcher --> Parser["Feed Parser"]
|
||
Parser --> Classifier["Classifier: category + item_tags + importance"]
|
||
Classifier --> Store["earth_news_items"]
|
||
Fetcher --> Health["source health"]
|
||
Health --> Config
|
||
Store --> EnrichQueue["Location / Localization Queue"]
|
||
EnrichQueue --> AI["AI Provider"]
|
||
Store --> NewsAPI
|
||
UE["UE Client"] --> NewsAPI
|
||
```
|