planet/docs/technical/en/earth-news-live-streams-collector-format.md

News Live Streams Collector Format

The news_live_streams collector accepts a "channel directory JSON" as input rather than scraping web pages directly.

Goals:

• Allow the backend to stably ingest live news streams from around the world
• Ensure the Earth page TV module always consumes a consistent structure
• Make it easy to integrate channel directories like worldmonitor that mix YouTube / HLS / iframe sources

Recommended JSON Structure

{
  "sources": [
    {
      "id": "bbc-world-news",
      "name": "BBC World News",
      "provider": "BBC",
      "region": "UK",
      "language": "en",
      "source_type": "youtube",
      "youtube_video_id": "dQw4w9WgXcQ",
      "youtube_channel": "https://www.youtube.com/@BBCNews",
      "embed_url": "",
      "stream_url": "",
      "homepage_url": "https://www.youtube.com/@BBCNews/live",
      "poster_url": "",
      "sort_order": 220,
      "is_enabled": true,
      "notes": "Primary English global news channel"
    },
    {
      "id": "france24-en",
      "name": "France 24 English",
      "provider": "France 24",
      "region": "France",
      "language": "en",
      "source_type": "hls",
      "stream_url": "https://example.com/live.m3u8",
      "homepage_url": "https://www.france24.com/en/live",
      "sort_order": 230,
      "is_enabled": true
    },
    {
      "id": "cctv4-page",
      "name": "CCTV-4 Chinese International",
      "provider": "CCTV",
      "region": "China",
      "language": "zh-CN",
      "source_type": "iframe",
      "embed_url": "https://tv.cctv.com/live/cctv4/",
      "homepage_url": "https://tv.cctv.com/live/cctv4/",
      "sort_order": 10,
      "is_enabled": true
    }
  ]
}

Field Conventions

• id: unique identifier, should be stable
• name: channel display name
• provider: provider name
• region: country or region
• language: language code
• source_type: iframe / hls / video / external / youtube
• embed_url: page suitable for iframe embedding
• stream_url: direct video stream URL
• homepage_url: official website or channel page
• youtube_video_id: YouTube live video ID
• youtube_channel: YouTube channel handle or channel URL
• poster_url: cover image, optional
• sort_order: sort value, smaller = higher in the list
• is_enabled: whether enabled
• notes: brief notes

Panel Behavior Conventions

• youtube
  • Prefers youtube_video_id
  • When embedding is not possible, at least keep youtube_channel or homepage_url for external opening
• hls / video
  • Prefers stream_url
• iframe
  • Prefers embed_url
• external
  • No embedding attempt; only keeps external open link

Current Implementation Status

• The backend settings page supports manually maintaining channel directories
• The Earth TV module merges:
  • Manually configured sources
  • Sources collected by the news_live_streams collector
• The current default fallback source is CCTV-4 Chinese International
• When no override is configured, news_live_streams defaults to iptv-org:
  • channels.json
  • streams.json
  • logos.json and automatically filters for news-category channel directories

Collector Configuration

news_live_streams does not need a separate new page; it reuses Collector Settings under /settings:

• endpoint
  • Channel directory JSON API URL
• auth_type
  • none / bearer / api_key / basic
• headers
  • Additional request headers
• config
  • Collector request and parsing behavior

Supported config Fields

{
  "timeout": 30,
  "method": "GET",
  "params": {
    "region": "global"
  },
  "body_type": "json",
  "body": {
    "include_disabled": false
  },
  "response_path": "payload.channels"
}

• timeout: request timeout in seconds
• method: GET or POST
• params: query parameter object
• body_type: json or form
• body: request body for POST
• json_body: explicit JSON request body, takes priority over body
• form_body: explicit form request body, takes priority over body
• response_path: path to the channel array in the response JSON, supports dot notation, e.g.:
  • payload.channels
  • data.items
  • result.streams

Authentication Details

• bearer: uses Authorization: Bearer <token>
• api_key: sent as request header by default; if auth_config.in = "query", sent as query param
• basic: uses HTTP Basic Authorization

Compatible Response Structures

The collector first tries to read:

• Top-level array
• Or an array under these common fields:
  • sources
  • streams
  • channels
  • items
  • results
  • data

It also accepts these field aliases:

• id / source_id / slug / channel_id / code
• name / title / channel / display_name
• provider / publisher / network
• stream_url / stream / playback_url / hls_url / m3u8_url
• embed_url / embed / page_url
• homepage_url / source_url / website
• language / lang / locale
• youtube_video_id / video_id
• youtube_channel / channel_handle