planet/docs/technical/en/earth-news-sources.md

Earth News Source Configuration

Earth situational news is served through /api/v1/news/earth-feed. News source configuration lives in SystemSetting.category = "earth_news_sources"; when no database configuration exists, the backend uses the built-in default sources as the fallback seed.

Default Sources

The default set contains four groups:

• News feeds: BBC World, DW Top Stories, CNBC Business, BBC Business, Guardian Business, NPR Business, MarketWatch, TechCrunch, Retail Dive, PR Newswire Retail, 36Kr, Ebrun, and China NBS data releases.
• Industry insight sources: McKinsey Retail and Deloitte Retail.
• Official data sources: China NBS data releases, US Census Retail / E-Commerce, MOFCOM Data, MOFCOM e-commerce updates, and China e-commerce logistics index.
• Lead sources: BusinessWire Electronic Commerce; Google News is one aggregated source with feed children for global, Americas, Europe, Middle East / Africa, and Asia Pacific.

Config data sources are visible in Admin by default. If a source is not a stable RSS/Atom feed, it is kept disabled for automatic fetching until an administrator replaces it with a fetchable URL and enables it.

Source	Type	Default state	Default category	Main tags	Purpose
BBC World	RSS	Enabled	Politics	official_media, global	Global public-news baseline
DW Top Stories	RSS	Enabled	Politics	official_media, europe	Europe and international baseline
CNBC Business	RSS	Enabled	Business	business_news, us, global	International business news
BBC Business / Guardian Business / NPR Business / MarketWatch	RSS	Enabled	Business / Finance	business_news, finance	UK / US business and finance baseline
TechCrunch / Retail Dive / PR Newswire Retail	RSS	Enabled	Technology / Business	business_news, ecommerce, retail, press_release	Technology, e-commerce, retail, and company announcements
36Kr	RSS	Enabled	Business	business_news, ecommerce, china	China business, venture, and newsflash feeds; homepage is https://www.36kr.com/, the Feed Directory is https://www.36kr.com/rss-center, and feed children are the general, article, newsflash, and moment feeds
Ebrun	RSS	Enabled	E-commerce	ecommerce, business_news, china, retail	China e-commerce industry news; homepage is https://www.ebrun.com/, the Feed Directory is https://www.ebrun.com/rss/, and feed children are B2C, B2B, retail, O2O, service, data, and policy XML feeds
China NBS data releases	RSS	Enabled	E-commerce	official_data, ecommerce, retail, china	Official data release RSS; retail and online retail items are identified by category and importance rules
Google News	Aggregated	Enabled	Politics	aggregated, low_stability	One aggregated source with global, Americas, Europe, Middle East / Africa, and Asia Pacific feed children; lower priority than real RSS
BusinessWire Electronic Commerce	Reference	Disabled	E-commerce	press_release, ecommerce, low_stability	Corporate announcement leads
McKinsey Retail Insights	Reference	Disabled	Business	industry_insight, retail	Retail industry insight
Deloitte Retail	Reference	Disabled	Business	industry_insight, retail	Retail industry insight
US Census Retail / E-Commerce	Reference	Disabled	E-commerce	official_data, ecommerce, retail, us	US retail and e-commerce official data
MOFCOM Data	Reference	Disabled	Business	official_data, china	China commerce data
MOFCOM e-commerce updates	Reference	Disabled	E-commerce	official_data, ecommerce, china	China e-commerce policy and updates
China e-commerce logistics index	Reference	Disabled	E-commerce	official_data, ecommerce, logistics, china	Logistics fulfillment and e-commerce activity

Reference means a reference link or future collector lead. It records a homepage, report page, or data page and does not participate in RSS/Atom fetching. This lets commercial and official sources enter Admin governance without letting non-feed pages break the live news feed.

The news source model has two levels:

• source is the brand or aggregator, such as 36Kr, Ebrun, Google News, or BBC.
• homepage_url is the source homepage, section page, or report page.
• feed_directory_url is the Feed Directory page, such as an RSS subscription center or feed index. It is for human inspection and is not fetched.
• feeds are the actual RSS, Atom, or Aggregated child entries under that source. Each feed child has id / name / url / type / enabled / default_category / tags / priority.

The backend iterates over every enabled feed child under the same source, fetches them independently, merges and deduplicates items, and writes per-feed diagnostics into health.feed_results. This is not a backup URL model: all four 36Kr subscription feeds, multiple Ebrun category XML feeds, and the five Google News regional RSS feeds can be enabled at the same time, and each feed can have its own default category and enabled state. HTML subscription-center pages belong in feed_directory_url, not in feed URLs. Every default enabled fetchable feed is tested item by item: RSS/Atom/Aggregated feeds must parse at least one item, while Reference sources only retain a reference URL and future collector lead.

Items that still remain Reference are not treated as broken feeds; no stable directly consumable RSS/Atom feed was verified:

• BusinessWire documents customizable RSS/Atom support, but the public pages do not expose a stable industry feed URL; the e-commerce industry page is kept as an announcement lead.
• McKinsey and Deloitte retail insight pages are report/article collections, not public RSS feeds.
• The US Census press-release RSS is reachable, but its items currently have empty links; the Quarterly E-Commerce page remains an official data reference.
• MOFCOM data and China e-commerce logistics index pages do not expose stable RSS feeds yet; they should become dedicated collectors or be replaced with administrator-provided fetchable feeds.

Source Property Tags and News Categories

News sources have source_tags, shown in Admin as source property tags. They describe the source, not the media name and not the content category of an individual story:

• official_data
• business_news
• ecommerce
• finance
• retail
• logistics
• industry_insight
• press_release
• china, global, us
• aggregated, low_stability

Each news item has exactly one primary category. Defaults are politics, business, e-commerce, finance, sports, technology, military, disaster, energy, society, culture, and other. item_tags are item-level secondary tags, such as cross-border e-commerce, live commerce, retail data, logistics fulfillment, platform governance, AI, semiconductor, election, oil price, football, and supply chain.

The primary category is generated by a rule-based scorer over title, summary, and source text. If the rules do not match, the feed child default category is used first, then the source default category. AI enrichment does not block news display.

Importance

Each item includes:

• importance_score
• importance_level
• importance_reasons
• market_impact

Official data, e-commerce metrics, major platforms, and numeric business signals increase importance. Press releases start with a lower baseline and rise only when they match stronger platform, amount, M&A, or regulatory signals.

Importance levels are fixed: low 0–34, medium 35–59, high 60–79, and critical 80–100. Category, importance, and Breaking decisions are centralized in earth_news_classification.py. Category keys and tags remain configurable, while importance and Breaking protocol states use shared enums. Databases and APIs continue to store compatible lowercase strings.

Configuration and Cache

GET /api/v1/earth/news-sources returns the default or saved configuration. PUT /api/v1/earth/news-sources saves it, increments cache_version, and clears the process region cache. POST /api/v1/earth/news-sources/reset restores defaults. POST /api/v1/earth/news-sources/test tests one RSS/Atom/Aggregated source without writing news items.

Manual News Content

Manual news is a content-management path, not an RSS source configuration. The Admin entry is Earth Content -> News Content; the left rail groups items by RSS source and manual news group. After opening a manual group, administrators can add one item or upload a JSON array. Manual items are written to earth_news_items with feed_type/source_type = manual; the display label is “手动添加” / “Manual”. They do not participate in RSS connectivity tests and are not routed through RSS fetching.

Manual news uses a publish-first, enrich-later flow:

1. Saving immediately writes the item to earth_news_items.
2. If no manual coordinates are provided, the selected region anchor is used and verified=false.
3. If manual coordinates are provided, the item uses location_source=manual_location and verified=true; later AI enrichment does not overwrite that location.
4. Create and reprocess actions enqueue the item for cleanup, translation, classification, importance, Breaking, and target-location inference.
5. When enrichment finishes, the same row is updated and Earth receives a news reload / patch so the frontend replaces the item without a manual refresh.

The Admin API is under /api/v1/earth/news-items:

• GET /earth/news-groups: return RSS virtual source groups and manual news groups.
• POST /earth/news-groups: create a manual news group.
• PUT /earth/news-groups/{group_id}: rename a manual news group and synchronize metadata for items in that group.
• GET /earth/news-items: paginated RSS and manual news list, with filters for source type, region, category, and status.
• POST /earth/news-items: create one manual news item.
• POST /earth/news-items/import: upload a JSON array; group_id selects the current manual news group.
• PUT /earth/news-items/{id}: edit a manual news item; RSS items are read-only.
• DELETE /earth/news-items/{id}: delete a manual news item and trigger an Earth news reload.
• POST /earth/news-items/{id}/reprocess: requeue cleanup, translation, and geolocation.

The first JSON import format supports arrays only:

[
  {
    "title": "Required title",
    "summary": "Optional summary",
    "content": "Optional body",
    "url": "https://example.com/story",
    "source": "Manual",
    "region": "global",
    "published_at": "2026-05-15T03:00:00Z",
    "category": "business",
    "tags": ["manual", "analysis"],
    "location": {
      "label": "Beijing, China",
      "latitude": 39.9057,
      "longitude": 116.3913
    }
  }
]

Imports deduplicate through a stable manual:{hash} ID derived from title, published time, URL, and source. Importing the same manual item again updates the existing row instead of creating a duplicate EarthFeed entry.

Feed Query and Category Filtering

The Web Earth client and UE client both consume GET /api/v1/news/earth-feed. The endpoint supports server-side filtering, so clients do not need to fetch the full list and apply the primary category filter locally.

• lat / lon: infer the active region from the current view, used by the Web Earth client.
• region: explicitly select a region for UE or service integrations. Supported values include global, americas, europe, asia-pacific, and middle-east-africa. global is an aggregate view and can include every region; non-global regions include only their own region plus global sources.
• categories: comma-separated news category keys, for example business,ecommerce. Omit it when all categories are selected.
• locale: display locale, currently zh-CN or en-US, defaulting to zh-CN. Chinese RSS items are stored as Chinese source content and enriched with en-US; English RSS items are enriched with zh-CN.

For locale=en-US, the API must not fall back to Chinese source title/summary. If a Chinese source item has not yet received an en-US localization, display_title and display_summary stay empty so the Earth client can show an English pending/empty state instead of mixing languages.

Examples:

GET /api/v1/news/earth-feed?region=europe&categories=business,ecommerce
GET /api/v1/news/earth-feed?lat=48&lon=10&categories=technology
GET /api/v1/news/earth-feed?region=global&categories=business,ecommerce&locale=zh-CN

Unknown category or locale values return 422 with the allowed values. The response includes filters, which confirms the region, category, and locale filters applied by the backend. items and cruise_items use the same category filter set.

The Web Earth category chips only store the current browser preference; changing them triggers a new API request. UE should pass its selected categories through the categories query parameter and does not need to perform the primary filtering itself.

When sources is omitted, the service layer prefers stories that already have displayable title and summary content for the requested locale, supplements candidates from enabled sources, and rotates sources so one source's newest pending items cannot occupy all 12 default slots. An explicit sources filter remains precise and does not supplement other sources. The database query layer owns region, category, source, and ordering constraints only; it does not own locale presentation policy.

Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next earth-feed request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.

Regional Balancing and Async Enrichment

earth_news_items is the current-state table for EarthFeed. When no explicit sources filter is present, the endpoint sorts by Breaking state, region, and publish time, then applies a regional round-robin so Asia Pacific or any other high-volume source cannot fill the entire global view and cruise queue. The default region order is Americas, Europe, Middle East / Africa, Asia Pacific, then Global; unknown regions participate after the known regions. Regional views still follow the active-region-plus-global rule and do not mix unrelated regions into the regional panel.

Both items and cruise_items are enqueued for target-location and localization enrichment, but the frontend must not wait for AI before rendering. If the requested display locale is not ready yet, Web Earth falls back to the original title / summary so a card does not show a "translation pending" placeholder when readable source content already exists. After translation, classification, Breaking, or target-location enrichment completes, the same earth_news_items row is updated and Earth receives a news reload / patch.

Target-location enrichment uses two Redis Streams queues:

• earth_news:target_location:priority: priority jobs for currently visible items and cruise_items, with a short dedupe TTL.
• earth_news:target_location:jobs: normal background enrichment jobs, with a longer dedupe TTL.

The worker always drains the priority queue before the regular queue; stale pending messages are reclaimed after the idle threshold, each AI job has a hard timeout, and failures go through retry or dead-letter handling. This prevents a large historical regular backlog from starving the Europe, Americas, or other regional news currently visible to the user.

Breaking News Insertion

The news system keeps three separate decisions:

• Category: what the story is about, such as business, military, or disaster.
• Importance: whether the story has long-term value, stored as importance_score / importance_level.
• Breaking: whether the story must temporarily jump ahead, stored in location_meta.news_meta.breaking_*.

Breaking metadata does not add physical columns. It remains in location_meta.news_meta:

• breaking_level: none / watch / breaking / critical.
• breaking_scope: regional / global.
• breaking_reasons: rule or operator reasons.
• breaking_source: rules / ai / manual / multi_source.
• breaking_confidence: 0 to 1.
• breaking_expires_at: expiration timestamp.

The backend performs the ordering, so Web and UE clients do not need to reorder items. Active critical, breaking, and watch items appear before normal items in that order. Once expired, the item falls back to normal ordering without being deleted or changing its long-term importance score.

breaking_scope = global bypasses the active region and can appear in every regional feed. regional breaking follows the normal active-region-plus-global rule. The response filters includes has_breaking and highest_breaking_level, which the Earth client uses to apply restrained panel and card styling.

Connectivity Monitoring

POST /api/v1/earth/news-sources/test tests one source and writes the result to earth_news_sources.health[source_id]. Normal RSS/Atom fetches update the same health map.

Health results include:

• status: ok, empty, format_error, http_error, timeout, network_error, or reference.
• status_code, content_type, item_count, latency_ms, error, and fetched_at.
• feed_results: per-feed diagnostics for multi-feed sources, including feed_id, feed_name, feed_type, feed_url, status, item count, and error.

Common diagnostics:

• HTML response: the configured URL is not an RSS/Atom feed, for example a web page listing RSS options.
• HTTP 403: the source or CDN rejected the crawler request.
• Reference: the source is a reference link only and must be converted to RSS, Atom, or Aggregated before fetch testing.

The Admin entry is Earth Content -> News Sources. It is not a raw whole-payload JSON editor. The UI has two layers:

• News sources: a left-side source list with filters for enabled, disabled, reference links, RSS/Atom/Aggregated, region, and source property tags; the right side edits one selected source and its feed child list.
• Policy rules: global source property tags, news categories, item tag rules, and default health policy. Advanced JSON is reserved for diagnostics, not the default edit path.

The single-source form is split into source information and feed children:

• Source information covers name, ID, region, homepage URL, Feed Directory URL, source type, enabled state, source property tags, importance weight, fetch interval, timeout, failure threshold, and circuit breaker.
• Feed children cover feed ID, name, real feed URL, type, enabled switch, default news category, priority, and feed tags. The + button under the feed child list creates a frontend-only draft; saving the source persists it, while canceling destroys the draft.

The per-source “test source” action tests all enabled feed children under the current source. The feed-row test action tests only that feed child. Both send to /api/v1/earth/news-sources/test, but the feed-row action submits the current source with only the selected feed child.

Reference links show that they only record a homepage, report page, or future collector lead and do not participate in RSS/Atom fetching. They can remain as commercial or official-data leads, but must be converted to RSS, Atom, or Aggregated with fetchable feed URLs before they can be enabled for fetching.

flowchart LR
  Admin["Admin: Earth Content / News Sources"] --> Source["Source config"]
  ManualAdmin["Admin: Earth Content / News Content"] --> ManualAPI["/api/v1/earth/news-items"]
  Source --> Feed["Feed children"]
  Feed --> ConfigAPI["/api/v1/earth/news-sources"]
  ConfigAPI --> Config["SystemSetting: earth_news_sources"]
  ManualAPI --> Store

  Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
  NewsAPI --> Resolver["Source Resolver"]
  Resolver --> Config
  Resolver --> Cache["Region Feed Cache"]
  Resolver --> Fetcher["RSS / Atom Fetcher"]
  Fetcher --> Parser["Feed Parser"]
  Parser --> Classifier["Classifier: category + item_tags + importance"]
  Classifier --> Store["earth_news_items"]
  Fetcher --> Health["source health"]
  Health --> Config
  Store --> EnrichQueue["Location / Localization Queue"]
  EnrichQueue --> AI["AI Provider"]
  Store --> NewsAPI
  UE["UE Client"] --> NewsAPI