How story clustering and duplicate detection work#

Story clustering and duplicate detection solve opposite problems in APITube. Clustering gathers different articles that cover the same event — different sources, different wording — by matching a seed article’s headline keywords. Duplicate detection does the reverse: it flags near-identical reprints of the same article, which you filter out with the is_duplicate parameter. One pulls varied coverage together; the other strips repeated copies apart.

Neither relies on a pre-stored “story ID” stamped onto each article. A cluster is computed on demand from a seed article you choose, and duplicate status is a per-article flag you read at query time. This page explains the mechanics of both. For the plain-language concept of a story versus an article, start with what is a story cluster.

How does APITube build a story cluster?#

APITube builds a cluster from the seed article’s headline, at query time. You pass one article’s id to /v1/news/story/:id, and the endpoint runs these steps:

1. It looks up the seed article and reads its title. If the id matches no article, it returns 404 with error ER0235.
2. It tokenizes the title, drops stopwords (common words like “the” or “and”), and expands the remaining words with synonyms and morphological forms — so a match is not limited to the exact words in the seed headline.
3. It keeps only tokens of at least two characters. If nothing usable is left after stopwords are removed, it returns 404 with ER0240 rather than guessing.
4. It finds other articles whose titles contain any of those tokens (a case-insensitive token match), and always excludes the seed article itself.

curl "https://api.apitube.io/v1/news/story/1234567?api_key=YOUR_API_KEY"

Because the cluster is seeded from any article id, any article can start its own story. The endpoint accepts both GET (id in the path) and POST (same path, filters in a JSON body), and the results are paginated like any search.

How does APITube rank the articles in a story?#

APITube ranks cluster results by how many of the seed headline’s keywords each article’s title matches. Every candidate gets a relevance score equal to the count of matching tokens, and results are ordered by that score descending — an article whose headline shares more keywords with the seed ranks above one that shares fewer. So the closest coverage of the same event surfaces first, and looser matches trail behind. Matching is done on the headline, not the body, which keeps a cluster tight around the specific event the seed article reports.

How does duplicate detection work, and how do you filter it?#

Duplicate detection is a per-article flag, separate from clustering. Each article carries an is_duplicate value, and you filter on it with the is_duplicate parameter, which accepts exactly 0 or 1:

• is_duplicate=0 returns only articles flagged as not duplicates.
• is_duplicate=1 returns only articles flagged as duplicates.

curl "https://api.apitube.io/v1/news/everything?title=election&is_duplicate=0&api_key=YOUR_API_KEY"

The value must be 0 or 1: a non-numeric value returns ER0003, and any other number returns ER0004. Keep in mind that is_duplicate is an approximate flag rather than exhaustive reprint detection — is_duplicate=0 thins flagged copies but does not guarantee that every syndicated reprint is dropped. Use is_duplicate=0 when you want to thin a feed toward flagged-unique pieces, and see how to exclude paywalled or duplicate articles for combining it with other quality filters.

Story clustering vs duplicate detection: what’s the difference?#

A story groups different articles — different sources and wording — that cover the same event, built by matching the seed headline’s keywords. A duplicate is a near-identical copy of the same article, flagged at the article level and controlled with is_duplicate. So the two are complementary: use the story endpoint to gather the spread of coverage around one event, and the is_duplicate filter to thin a result set by dropping flagged reprints. They are not the same mechanism and do not share an identifier.

Common Questions#

• Does clustering match on the article body or just the headline?
• What happens if the seed headline has no usable keywords?
• How much does a story request cost?
• Can I reuse a story cluster later?

Does clustering match on the article body or just the headline?#

Clustering matches on the headline only. APITube reads the seed article’s title, expands its keywords with synonyms and morphology, and searches the titles of other articles for those tokens. The body text is not used to build the cluster. Headline matching keeps a story focused on the specific event the seed reports, rather than pulling in loosely related articles that merely mention the same words deep in their text.

What happens if the seed headline has no usable keywords?#

If the seed article’s title is left with no usable tokens after stopwords are removed — for example, a headline made entirely of very short or common words — the endpoint returns 404 with error ER0240 instead of guessing at a cluster. A 404 with ER0235 is different: it means the id you passed does not match any article at all. Passing no id returns 400 with ER0231.

How much does a story request cost?#

A /v1/news/story/:id call costs 1 credit, and you are charged only when the endpoint returns at least one related article. If no other headline overlaps the seed’s keywords, the result set is empty, that is a normal outcome rather than an error, and no credit is spent.

Can I reuse a story cluster later?#

There is no stored story ID to save. Because a cluster is computed on demand from a seed article’s id, you reproduce a story later by calling /v1/news/story/:id again with the same seed id. The set of matched articles can change over time as new coverage is published and matched against the seed headline — see how to get all articles covering the same story for the full request options.