In short
Every day, about 10,000 news headlines come in. The sampler groups the ones about the same story, sorts them into topics, and ranks the topics by how much they’re being covered. What comes out is a short, readable list of what the news is talking about.
It all runs on a laptop, costs nothing, and doesn’t use any AI chatbot.
The report doesn’t include article links yet. This run was for testing how well the grouping works; the finished system will include them.
See the results: the 15 November report →
The problem
A GDELT news pull gives you roughly 9,000 headlines a day. Most of them are the same 200 things written 40 different ways:
- “Fed holds interest rates steady amid cooling inflation”
- “Federal Reserve keeps rates unchanged as price growth slows”
Same story, almost no shared words. Keyword matching won’t catch it. I wanted something that groups headlines by meaning, then tells me which groups actually matter right now.
Here’s the whole system:
headlines file → embed → dedupe → cluster → split → merge → rank → representatives → report.mdEach step is a small Python script that does one job, and a pipeline.py runs them in order. All the thresholds and weights live in one config.toml. Let’s walk through them.
Step 1: Embed, turning text into numbers that capture meaning
Computers can’t compare sentences, but they can compare lists of numbers. An embedding model turns each headline into a list of 384 numbers (a vector). Headlines that mean similar things end up with similar vectors.
I use all-MiniLM-L6-v2, a small open-source model that runs locally. 9k headlines take under a minute on a laptop CPU.
sentence_transformers SentenceTransformer
model = (, device=)
embeddings = model.(
headlines,
batch_size=,
normalize_embeddings=, # makes similarity a simple dot product
)
# embeddings.shape -> (n_headlines, 384)Those two Fed headlines from earlier score 0.607 similarity, even though they share almost no words. That’s the whole point.
Tip: install the CPU-only build of PyTorch. It’s ~200 MB instead of ~2.5 GB and you don’t need a GPU for this:
pip install torch --index-url https://download.pytorch.org/whl/cpuStep 2: Dedupe, collapsing rewrites into one story
If two headlines score 0.9 or higher, they’re the same story. But how you group them matters.
“Link everything that’s similar” suffers from chaining: A looks like B, B looks like C, so A and C get merged even though they’re about different things.
So the pipeline uses leader clustering: walk the list, and each headline that isn’t taken yet becomes a “leader” that grabs everything close to it. Nothing gets merged through a middleman.
(emb, threshold=):
story_id = np.((emb), -)
next_id =
i ((emb)):
story_id[i] != -:
# already part of a story
unassigned = np.(story_id == -)
sims = emb[unassigned] @ emb[i] # similarity to the leader
story_id[unassigned[sims >= threshold]] = next_id
next_id +=
story_idThe duplicates aren’t thrown away. Each story keeps two things:
dup_count: how many times it was rewritten. A story covered 40 times counts as 40, not 1. That’s a size signal.sources: the set of outlets that ran it (Reuters, BBC, …).
Why a set and not just a count? Later we add up sources per topic. If Reuters wrote two different stories in the same topic, adding counts would count Reuters twice. Merging sets doesn’t.
Source names are cleaned too, since Reuters, reuters.com and https://www.reuters.com/world/... are the same outlet:
() # -> "reuters"Step 3: Cluster, grouping stories into topics
Now we have stories. Next we group them into topics (“the Fed”, “AI chips”, “Tesla recall”).
Two tools work together:
- HDBSCAN finds groups of points that sit close together. It decides how many topics there are on its own, and anything that doesn’t fit gets labelled noise instead of being forced into a group.
- UMAP goes first. HDBSCAN works badly on 384 dimensions, so UMAP squashes each vector down to 5 while keeping similar things close.
umap
sklearn.cluster HDBSCAN
reduced = umap.(
n_components=,
metric=, # same similarity measure the embeddings use
min_dist=, # pack clusters tightly
).(story_embeddings)
labels = (min_cluster_size=mcs).(reduced)
# label -1 = noiseHow big does a topic need to be?
min_cluster_size is the smallest group HDBSCAN will call a topic. The right value depends on how much data you have, which changes every run. So it’s set from the story count:
min_cluster_size = (, (, (n_stories * )))Naming each topic
To label a cluster, all its headlines are glued into one document, and we find the words that are common here but rare in the other clusters (class-based TF-IDF, the same trick BERTopic uses). The Fed cluster comes out as something like fed, rates, inflation, federal reserve.
Step 4: Split, un-gluing topics that are really two stories
Sometimes HDBSCAN glues several real stories into one topic, like Ukraine, Gaza and ceasefire talks all landing in one “war” cluster. So before ranking anything, each cluster is checked for how tight it is.
Coherence: how much a cluster agrees with itself
Take the average of all the cluster’s vectors. That’s its centroid, the “center of meaning”. Then measure how similar each headline is to that center, and average it:
(emb, weights): # weights = dup_count per story
c = (weights[:, ] * emb).() / weights.()
c /= np.linalg.(c) # centroid
np.(emb @ c, weights=weights)Two details: it uses the original 384-dim vectors, not UMAP’s squashed ones (UMAP is great for finding clusters but distorts distances, so it’s bad for measuring them). And it’s weighted by duplicate count, so a story rewritten 40 times counts more.
The split rule
- Any cluster with coherence below 0.65 gets a split attempt: cut it in two (agglomerative clustering, k=2).
- The cut is kept only if it’s a real improvement: both halves have at least 5 stories, and their coherence beats the parent’s by at least 0.05.
- Halves that are still loose get split again, up to 3 levels deep (max 8 pieces).
- Then anything still below 0.35 is dropped as noise.
The order matters. A cluster with poor coherence is often two good stories glued together. Splitting before dropping rescues both.
The gain check in rule 2 matters too. Cutting any group in two makes the halves a little tighter, even a group that’s genuinely one story. Requiring a real gain stops the splitter from shredding good clusters. That’s also why the 0.65 threshold can sit fairly high: the worst case is a few extra split attempts that the gain check rejects.
Why so high? A blob of two unrelated stories still scores surprisingly well, because every headline is compared to a centroid sitting halfway between the two, so everyone gets partial credit. Gluing an earthquake cluster and an election cluster together gives about 0.59; split, they come out at 0.78 and 0.80. Meanwhile a genuinely broad single story like Ukraine (0.64) gets tried and is correctly left alone.
The script prints a coherence histogram on every run, so you can see where your clusters sit.
Split only ever cuts. The opposite mistake, one story broken into several clusters, is handled in the next step.
Step 4b: Merge, joining one story that got cut into pieces
Split fixes a cluster that mixes several stories. HDBSCAN also makes the opposite mistake: with a small min_cluster_size, it can cut one real story into several clusters.
On a real 24-hour run, a single Hormuz tanker and mine-strike story came out as 6 separate clusters, ranked 1, 2, 6, 8, 17 and 50. Split can’t help here, because it only handles one cluster that mixes stories, not one story spread over many clusters. So a merge step runs right after it.
The merge rule
- Every cluster has a centroid: the mean of its stories’ vectors, weighted by duplicate count. It’s the same definition coherence uses.
- Take the two clusters whose centroids are most similar (cosine similarity). If it’s 0.80 or higher, merge them, but only if the merged cluster’s coherence stays at 0.65 or higher, the same bar the split step uses. Otherwise that pair is skipped.
- Recompute the merged centroid, and repeat until no pair qualifies.
The coherence guard stops merge from producing a loose cluster that split would just cut apart again. Here are the two rules side by side:
The result
On that run, 119 clusters became 114 after 5 merges. The Hormuz story is now a single rank-1 cluster: 46 headlines from 35 outlets, with coherence 0.89.
The 0.80 threshold is a starting value, tuned on this one run. It isn’t a final setting.
Step 5: Rank, finding what actually matters right now
The easy answer is to sort by size. Here’s why that’s not enough:
Topic A: Ukraine coverage, ~300 headlines a day, every day for months.
Topic B: A major earthquake. Zero headlines yesterday, 300 today.
Same size, but B is the bigger news. So each topic is scored on four signals:
Velocity in plain English
Take the time span of the whole run and cut it in half. For each topic, ask: what slice of all the news did it get in the older half, and what slice in the recent half?
recent_share = (recent + ) / (all_recent + n_topics)
previous_share = (previous + ) / (all_previous + n_topics)
velocity = (recent_share / previous_share)Using shares instead of raw counts means a busy news day doesn’t make everything look like it’s surging. The +1 keeps brand-new topics from dividing by zero.
A few details:
- Every topic uses the same cut. If each topic got its own time window, a 6-hour burst would look “steady” inside its own 6 hours.
- Velocity counts every raw headline, duplicates included. The duplicates are what make a burst look like a burst. Topics are attached to stories and timestamps to headlines, so ranking joins the two on
story_id. - There’s a dead zone. Small wobbles aren’t news: anything within ±0.3 counts as exactly 0. Steady stories score neutral, and only real surges and fades move the ranking.
velocity = (velocity) < velocityThe log matters: without it, a brand-new topic’s huge ratio would swamp every other signal. With it, steady topics add nothing, breaking topics get a boost, and fading topics lose a little.
Putting it together
Sources and size are log-scaled too, since size can range from 20 to 3,000 and one giant topic would drown out everything else. Then every term is normalized to 0–1 across all topics with clipped min-max: cap each term at its 5th and 95th percentile, then scale. Capping stops one outlier from squashing everyone else to 0, while still keeping the real gaps between topics. Then the weights:
score = * sources + * size + * velocity + * coherenceThe result
The script prints every term, raw and normalized, so you can see why each topic landed where it did:
rank score | sources | size | velocity | coherence | topic
1 0.93 | 9 1.00 | 14 0.72 | +2.47 1.00 | 0.80 1.00 | chile earthquake
2 0.68 | 9 1.00 | 12 0.57 | -1.53 0.00 | 0.78 0.89 | portugal election
3 0.64 | 8 0.79 | 16 0.86 | +0.00 0.36 | 0.64 0.00 | ukraine (steady)
4 0.35 | 6 0.27 | 6 0.00 | +1.71 0.83 | 0.74 0.67 | fed rate cut
5 0.31 | 5 0.00 | 19 1.00 | -0.52 0.22 | 0.67 0.16 | moondog token (spam)The breaking earthquake wins. The spam crypto token has the most headlines but ranks last, because they come from a handful of outlets, and sources carries the most weight.
Step 6: Picking representatives
A reader doesn’t need all 300 earthquake headlines. They need the 5 that best describe it. For each top topic:
- Sort its stories by similarity to the centroid (the same function coherence uses).
- Leave out strays: headlines far from the centroid are excluded up front.
- Greedily pick the closest remaining story that brings a new outlet, so you don’t get five Reuters rewrites.
Filtering strays first matters, because an off-topic headline that HDBSCAN pulled in often is from a new outlet, and the diversity rule would otherwise happily pick it.
The output is a small JSON file: top 20 topics, 5 headlines each.
Step 7: Writing the report
The ranking is full of numbers like velocity +2.47 and coherence 0.64. Those mean nothing to a reader. So Python translates every metric into plain language, deterministically:
The stack
sentence-transformers(all-MiniLM-L6-v2) for embeddingsumap-learn+ scikit-learn’sHDBSCANfor clustering- scikit-learn TF-IDF for labels
- pandas + numpy for everything else
No GPU, no API keys, no LLM calls.