How does APITube collect and process news?#

APITube continuously discovers news from publishers across the web — through their RSS feeds, Google News and XML sitemaps — then fetches each article, extracts the clean text, enriches it with NLP (language, sentiment, named entities and category, topic and industry labels), and indexes the result so you can pull it with a single REST call, a live stream or a webhook. You never scrape a page or parse a feed yourself: by the time news reaches you it is already structured, labelled JSON.

In short, the pipeline has four stages — discover, fetch, enrich, serve — and the rest of this article walks through each. For what the finished article object looks like, see what’s inside a news article.

Where does APITube get its news from?#

APITube finds articles by monitoring the places publishers announce them, not by guessing URLs. Three discovery sources feed the pipeline:

• RSS feeds — the feeds publishers maintain for their own sites.
• Google News — used as a broad discovery channel across many outlets.
• XML sitemaps — the sitemap files sites publish to list their pages.

These sources are polled continuously, so newly published URLs are picked up as they appear rather than on a fixed daily batch. Each discovered URL is queued for the next stage; the result is a steady stream of fresh links from a wide range of outlets.

How is a raw web page turned into structured data?#

Once a URL is discovered, APITube fetches the page and turns its HTML into a clean article. The processing stages run one after another:

1. Fetch — APITube downloads the article’s HTML from the publisher.
2. Parse and extract — the main article text is pulled out of the page and separated from navigation, ads and other boilerplate, leaving the readable body.
3. Enrich — natural-language processing runs over that text to add the metadata you query on (covered in the next section).
4. Index — the finished article is stored and indexed so it can be searched, retrieved and streamed quickly.

The output of this pipeline is the structured JSON object the API returns — a headline, body, URL, publish date and source, plus the enrichment APITube computed. You query the indexed result, never the live web page.

What does APITube add to each article?#

The enrichment stage is what makes APITube more than a list of links: it attaches machine-readable metadata to every article so you can filter and analyse without reading the text yourself. Each processed article carries:

• Language — the detected language code of the article text.
• Sentiment — an overall positive, neutral or negative reading, plus sentiment toward individual entities.
• Entities — the people, organizations, locations, products and other names mentioned, with their positions in the text.
• Classification — category, topic and industry labels.
• Source and media metadata — the publisher, and the images and media attached to the article.

These fields are computed for you on every article, so a single request returns news already tagged by language, sentiment, who is mentioned and what it is about. For the vocabulary behind these — articles, sources, entities, stories — see key concepts.

How do you get the processed news?#

The indexed, enriched articles are served from a single base URL, https://api.apitube.io, in three ways, all using the same API key:

• REST (pull) — search and retrieval endpoints such as /v1/news/everything, /v1/news/top-headlines and /v1/news/article return data on demand, as JSON or an export format.
• Real-time streaming — the Server-Sent Events stream (/v1/news/stream) and the WebSocket feed push matching articles to you as they are indexed.
• Webhooks — APITube posts new matching articles to a URL you control, so your system reacts without polling.

Choose pull for periodic jobs and streaming or webhooks when you need news the moment it is processed. To make a first call, follow how to run your first news search; the full parameter list is on the official News API parameter reference.

Common Questions#

• Do I have to scrape or parse HTML myself?
• Can I get the original article body and HTML?
• How soon do new articles become available?
• What enrichment is applied to each article?

Do I have to scrape or parse HTML myself?#

No. Fetching the page, stripping boilerplate and extracting the readable body all happen inside APITube’s pipeline before the article reaches you. The API returns clean, structured JSON — a title, body, source, date and enrichment fields — so there is no HTML parsing, feed handling or scraping on your side.

Can I get the original article body and HTML?#

Yes. The /v1/news/raw endpoint returns the article’s raw text in two fields: body (plain text) and body_html (the HTML version). Use it when you need the unprocessed content rather than the cleaned, enriched object the search endpoints return. See how to get the raw article body and HTML for the call.

How soon do new articles become available?#

Discovery runs continuously rather than as a once-a-day batch, so articles are processed and indexed shortly after a publisher lists them, then become searchable. If you need news the instant it is indexed rather than polling on a schedule, subscribe to the Server-Sent Events stream or a webhook, which deliver matching articles as they are processed.

What enrichment is applied to each article?#

Every article is enriched with its detected language, an overall sentiment reading plus sentiment toward named entities, the entities themselves (people, organizations, locations and products with their text positions), and classification into category, topic and industry. This is the metadata you filter and sort on across the API, and it is attached automatically — you do not request enrichment separately.