Polling vs streaming: how to keep news data fresh#

Poll /v1/news/everything on a schedule when a few minutes of delay are acceptable — it is simpler, works on every plan including Free, and an empty poll costs nothing. Switch to push delivery (SSE, WebSocket or a webhook) when seconds matter: breaking-news alerts, live dashboards, trading signals. The two models bill differently and fail differently, so the right choice is a property of your use case, not a preference.

How do I poll for new articles correctly?#

A polling loop is one repeated search with a sliding time window. Ask for articles published since the last run, newest first (the default sort), and de-duplicate by article id on your side:

curl "https://api.apitube.io/v1/news/everything?organization.name=Tesla&language.code=en&published_at.start=NOW-1HOUR&per_page=250&api_key=YOUR_API_KEY"

Three details make the loop reliable:

• Overlap the window. Poll every 30 minutes but ask for the last hour (published_at.start=NOW-1HOUR). Articles are indexed with some delay after publication, so a window equal to your interval can miss pieces that entered the index late; the overlap catches them, and your id set drops the repeats.
• Keep seen ids. Store the id of every article you processed (the last few thousand is plenty) and skip repeats. This makes the overlap free and the loop idempotent.
• Page when needed. per_page goes up to 250; if a busy window returns a full page, follow the response’s next_page URL until has_next_pages is false — the mechanics are in how to paginate results.

The relative date syntax (NOW-1HOUR, NOW-24HOURS, NOW-7DAYS) is covered in how to filter by date range.

When is polling the right choice?#

Polling wins whenever the consumer is itself periodic. A daily digest, an hourly sentiment aggregate, a nightly data-warehouse load — pushing articles one by one into a job that runs on a schedule adds moving parts and saves nothing. Polling is also the pragmatic choice when:

• You are on the Free plan. Streaming is not available on Free (the plan has zero connection slots), so a polling loop is your real-time option — and at 10 requests per minute the rate limit leaves room for a generous cadence.
• Cost control matters more than latency. A search request is charged only when it returns at least one article; a poll that finds nothing is free. Quiet topics cost you nothing between stories.
• Your infrastructure is simple. A cron job and an HTTP call, no long-lived connections to babysit, no public endpoint to secure.

When should I switch to push delivery?#

Push earns its complexity when the value of an article decays in seconds — alerting, live tickers, automated reactions. New matching articles reach an SSE or WebSocket stream within roughly half a minute of entering the index, and a webhook’s delivery worker picks them up within seconds. All three push methods bill one credit per delivered article (the open connection, heartbeats and empty periods are free) and draw from their own credit pools, separate from search requests.

Push also changes your failure model: you now own reconnection (streams) or a highly-available public endpoint (webhooks). Which of the three push methods fits your architecture — who holds the connection, how reconnection works — is the subject of SSE vs WebSocket vs webhooks; the parameters shared by all delivery methods are in the official News API parameters reference.

Can I combine both?#

The strongest setups usually do. Run one narrow stream for the events that cannot wait — is_breaking=1, or negative mentions of your own brand — and a scheduled poll for the broad sweep that feeds analytics and archives. The stream stays cheap because its filters are tight; the poll stays cheap because empty runs are free. One caution: the same article delivered by your stream and fetched by your poll is billed in both places, so keep the two filter sets disjoint.

Common Questions#

• Can polling miss articles entirely?
• How often can I poll?
• Is polling cheaper than streaming?

Can polling miss articles entirely?#

Only if your window has gaps. An article is searchable from the moment it is indexed, which can lag its published_at by a bit — that is exactly what the overlap window absorbs. With published_at.start overlapping the previous run and id-based de-duplication, a polling loop is lossless for any topic whose volume fits your page budget.

How often can I poll?#

Up to your rate limit: 10 requests per minute on Free, 50 per minute on paid plans (15 for test keys). In practice a poll every 1–5 minutes is indistinguishable from streaming for most workloads — at that point, choose by architecture rather than by latency. Exceeding the limit returns 429 (ER0203); see how to fix rate-limit errors.

Is polling cheaper than streaming?#

For quiet or bursty topics, usually: empty polls are free, and a non-empty poll is one charged request for up to 250 articles. Streaming bills per delivered article, which is economical precisely when almost every delivery matters to you. High-volume, always-on topics tilt toward streaming; low-volume monitoring tilts toward polling.