Every morning at 6 a.m. Toronto time, three Telegram messages land in three different channels: finance, canadian, general. Each one is a digest — a handful of scored entities, a couple of sentences of context, links to sources. If you only looked at the output, you’d assume a small team is behind it. It’s one Node service on a VM in my basement, and it’s the third attempt at building it.

I’ve written about the threat-intelligence lifecycle as theory — direction, collection, processing, analysis, dissemination, feedback. This post is the “I actually built it” companion. Less framework, more git log.

Why there’s a V3

V1 lived at port 3005. It worked, in the sense that it collected things and put them somewhere. The flow between stages was implicit — a function here called another function there, and if you wanted to know what happened to an item between “we fetched this from abuse.ch” and “this showed up in a digest,” you were reading code, not a log.

V2 (port 3025) was a real upgrade — ATT&CK mapping, watchlists, cases, full-text search, an AI-generated digest. It also inherited the implicit-flow problem, just with more surface area for it to bite. When I decommissioned threat-intel-v2, I found out the hard way that “implicit flow” also means “implicit list of everything that depends on this,” because retiring it meant chasing down references across a handful of other scripts that assumed it was still there.

So V3 has one rule that the first two didn’t: every run is six named stages, in order, and every stage’s job is written down as a comment before it’s written as code.

src/pipeline/
  1-ingest.js
  2-normalize.js
  3-cluster.js
  4-enrich.js
  5-score.js
  6-distribute.js
  orchestrator.js

The orchestrator inserts a collection_runs row at the start of every run and updates its stage column as it moves through the list. If a stage throws, the error gets recorded against that stage and the pipeline keeps going — one dead RSS feed doesn’t take down KEV enrichment. The whole run is only marked failed if literally every stage throws, which in practice means “the database is unreachable,” not “abuse.ch had a bad day.” That single design choice has saved me more 2am pages than anything else in this codebase.

Nineteen sources, one shared shape

Collection pulls from nineteen sources right now: CISA (KEV + advisories), NVD, EPSS, MITRE ATT&CK, abuse.ch, OTX, GitHub Advisories, HaveIBeenPwned, ransomware.live, databreaches.net, BleepingComputer’s data-leak feed, SecurityWeek, The Record, a handful of general security RSS blogs, plus — because two of the three digests are finance and Canadian-specific — Bank of Canada, OSFI, FinTRAC, CCCS, and SEC EDGAR.

That last group is the part that actually makes the “niche” digests worth reading instead of being the same feed three times with a different Telegram bot token. If you’re only pulling generic security news, “finance” and “general” converge to the same list. FinTRAC and OSFI filings are what make the finance channel say something a general audience doesn’t need.

One small, satisfying fix from the CISA collector, because it’s a good example of the kind of bug that doesn’t show up until it does:

// Use the JSON feed instead of XML to avoid entity expansion issues
const CISA_ALERTS_URL = 'https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json';

The KEV catalog is also published as XML/RSS, and for a while I pulled that instead. It’s fine most of the time, right up until a well-formed-but-hostile-shaped payload triggers an entity-expansion blowup in the XML parser — the classic “billion laughs” category of problem, minus the billion, but still enough to peg a CPU core parsing a feed that should take milliseconds. CISA also happens to publish the same catalog as clean JSON. Once I noticed that, switching over made an entire class of failure mode disappear rather than mitigating it. When a vendor gives you a choice between XML and JSON for the same data, and you don’t have a reason to need XML, don’t reach for XML.

Cluster: the boring stage that does the real work

Stage three is cross-source dedup, and it’s the one I spent the most time getting conservative on purpose. Every item collected gets a canonical_key:

  • CVEs: cve:<lowercased-cve-id>
  • Breaches: breach:<normalized-domain-or-name>
  • News: news:<sha1-of-normalized-title>

Items that resolve to an existing key get merged into that entity — source count goes up, last_seen_at updates, and the canonical title/summary gets adopted from whichever contributing item has the highest trust weight. Items that don’t match get a new entity.

The rule I wrote into the code comment and have stuck to since: a missed merge is fine; a wrong merge is not. If the same CVE shows up from NVD and from a SecurityWeek writeup and the clustering misses that they’re the same thing, worst case you get a slightly redundant digest entry. If clustering wrongly merges two unrelated breaches because their titles happened to normalize to something similar, you’ve now got a corrupted timeline for both of them, and untangling that after the fact is much more expensive than the redundancy would have been. Conservative dedup logic is a case where “do less, safely” beats “do more, cleverly.”

Enrichment: this is where KEV and EPSS actually happen

Stage four is where entities get their CVSS score, their KEV flag, and their EPSS score.

KEV detection is dead simple by design — if a contributing item came from the CISA collector and is tagged kev or actively-exploited, the entity gets is_kev = 1. No inference, no fuzzy matching. CISA already did the hard part of confirming active exploitation; I just have to not lose that signal on the way through the pipeline.

EPSS is a live lookup against FIRST.org’s API, batched in groups of a hundred CVE IDs per request, run outside the main enrichment transaction so a slow network call can’t hold a database lock open. If FIRST.org has a bad day, the enrichment error gets logged against that stage and everything downstream that doesn’t depend on EPSS still runs.

The part that actually matters: scoring that doesn’t trust CVSS by itself

Here’s the thing that CVSS-only tooling gets wrong, and it’s not a subtle mistake — it’s the default behavior of nearly every naive vulnerability scanner I’ve ever pointed at a stack: a CVSS 9.8 with zero evidence of exploitation gets treated exactly the same as a CVSS 9.8 that’s actively being used to pop boxes right now. Both light up red. Both generate a ticket. After you’ve triaged enough of the first kind, you stop believing the color, and that’s the moment a scanner stops being useful — not when it’s wrong, but when you’ve learned to ignore it.

Stage five is a deterministic rules engine first, an AI scorer second, and the ordering is the whole point.

rules.js runs on every entity and tries to reach a confident verdict for free — no API call, no latency, no cost:

  • A hit against something in my own stack is always URGENT at 95, full stop, regardless of niche.
  • KEV presence alone pushes the floor to 88 — active exploitation is the strongest signal there is, stronger than any CVSS number.
  • EPSS ≥ 0.5 does nearly the same independently, pushing the floor to 80.
  • CVSS ≥ 9 only gets treated as serious if it’s paired with exploitation evidence (KEV or EPSS ≥ 0.3). Without that pairing, and with EPSS confirmed low (under 0.1), the score gets explicitly capped in the 35–40 range — labeled WATCH instead of URGENT, with the reason logged as “CVSS high but low EPSS — not actively exploited.”

Only when the rules engine can’t reach a confident verdict does the entity get deferred to Claude Haiku, batched fifty at a time, with a persona prompt per niche and a hard-constrained JSON response format that gets clamped in code regardless of what the model returns. If there’s no API key configured, the AI scorer falls back to rules-only and marks the ambiguous stuff NOISE rather than blocking the pipeline on a network call it can’t make. The pipeline runs on a cron; it doesn’t get to hang waiting on an LLM.

The reason this two-tier design exists at all is that I’ve watched the naive version fail up close, in this same homelab. A separate service of mine runs a nightly CVE scanner against everything installed across the fleet, and for a while it matched CVEs to products by loose keyword instead of a proper CPE — the standardized “this exact product, this exact version” identifier. The results were exactly what you’d expect from string-matching a filing cabinet by “does the label contain this word.” A Docker CVE got flagged against Dockwatch, a dashboard app whose name just happens to contain “dock.” A Node.js advisory hit an unrelated service because the description mentioned “node.” When I sat down on July 4th and checked the open tickets by hand against the actual NVD records, eight of nine were false — a real CVE attached to the wrong product purely on a name collision. Not one of those eight was a bug in the CVE data; every one was a bug in “does this scary number look like it applies to me.”

Once you’ve spent an afternoon proving eight of nine CRITICALs wrong by hand, “URGENT” stops meaning urgent — and that’s the death of a scanner, because the whole point is that you can trust it enough to act without re-verifying. EPSS and KEV gating is the structural fix. It forces the system to ask “is this actually being exploited, and does it actually affect me” before it’s allowed to shout, instead of asking “does this number look scary.” A keyword match against a high CVSS is a guess wearing a red badge. Active-exploitation evidence is the thing that earns the badge.

Firehose to digest, on purpose

Nineteen sources produce a lot of raw items. The digest query that runs at 6 a.m. Toronto time is deliberately narrow: last 24 hours, urgency != 'NOISE', capped at thirty entities per niche, URGENT items sorted ahead of everything else. Thirty is a deliberate ceiling: nobody actually reads a fifty-item Telegram message, and a digest you skim past is worse than no digest at all.

The other rule I care about more than the cap: the LLM only ever writes two to three sentences of intro prose per digest. Every URL, every CVE ID, every entity bullet is built deterministically from database rows, not generated. The model can summarize; it cannot invent a link or drop one, because it never gets the chance to touch them. That constraint cost me nothing in output quality and removes an entire category of “did the model just make up a citation” risk that I’m not willing to carry into something that runs unattended every day.

What’s still rough

Phase 2 — a public web front end and an actual newsletter instead of a Telegram-only distribution — is planned, not built. A few of the RSS-blog sources are noisier than the named feeds and probably deserve either better filtering or a trust-weight downgrade; I haven’t gotten around to auditing them properly. And “three niches” is a good v1 scope, not a ceiling — sector-specific digests are the obvious next move once general and finance have proven the pattern holds.

None of that changes the core bet, which is the same bet as the threat-intel lifecycle piece: a feed is not intelligence, and a scanner that can’t tell the difference between “scary-looking” and “actually happening” is just a very expensive way to generate noise. KEV and EPSS gating is how I made the machine ask the second question before it’s allowed to answer the first.