TL;DR: a news record is kept only if it is about a place near one of the choke points (a “zone”). Each kept record is then tagged with a market bucket: risk, crude and refined oil, LNG and gas, fertilizer and agriculture, shipping and insurance, or “other”. Most of the rules exist to throw away noise I found on a test day. This is a first version, and I haven’t hand-labelled the results yet.
The problem
GDELT’s Global Knowledge Graph (GKG) is a constant stream of news records. Each one is an article, tagged by GDELT with the themes, organisations and locations it detected, plus a page title and a URL. Almost none of it has anything to do with shipping.
I want the part that is about the Strait of Hormuz, the Red Sea, the Gulf and the Arabian Sea, and I want to know which market each story could touch. That sounds like “filter by coordinates”, but GDELT’s locations are messier than that, so the rules ended up in two config files: one for places (zones) and one for markets.
How a record gets classified
Every record goes through the same four steps:
- A cheap prefilter. The row is kept only if its raw text contains a zone name or alias, or if one of its locations is in a country around the study area (the Gulf states, Yemen, the Horn of Africa, Egypt, Sudan, India, Pakistan, Jordan and Israel; Jordan and Israel were added for Aqaba and Eilat). Market keywords are not part of this gate: a keyword alone can never produce a zone match, so checking it here would only cost time.
- A zone match. A record matches a zone if it carries a known GDELT place ID, or a zone name appears in a location’s full name, or its coordinates fall inside the zone (city-level locations only), or a zone name appears in the page title. Why a row got its zone is stored with it.
- A market match. For each market bucket, a record matches if it has at least one matching theme, organisation, or title/URL keyword. What matched is stored too.
- Store it. One row per record × zone × market bucket. A record can sit in several zones and several buckets. If a record matches a zone but no market or risk rule, it goes in the bucket
other: stored, but it never triggers an alert.
Every stored row carries both a zone (and its zone group) and a market bucket, so I can slice by either.
Where: the zones
The zones come in four groups: Hormuz, the Persian Gulf, the Red Sea / Bab el-Mandeb and the Arabian Sea. Most are circles drawn around a strait or a port. A record matches if one of its locations is within the radius, measured as straight-line distance.
Two zones are shapes rather than circles:
- red_sea: anywhere within 150 km of a line running down the length of the sea, through these points (it always counts, apart from the name-collision rule below):
Suez 32.55 E, 29.95 N
38.10 E, 24.10 N
39.20 E, 21.50 N
37.20 E, 19.60 N
Yemen coast 42.95 E, 14.80 N- arabian_sea_lane: a rectangle from 50°E to 72°E and 8°N to 25°N. This one only counts if a market or risk rule also matches. Without that, every cyclone and every Indian or Pakistani coastal story would flood the feed.
The “core” zones, where I’m aiming for the highest precision (0.8 or better), are the Hormuz and Persian Gulf groups.
Matching by name
GDELT geocodes water bodies inconsistently, so coordinates alone aren’t enough. I also match place names, ignoring case, in a location’s full name and in the page title:
The real list also has spelling aliases, such as Hudaydah, Al Fujayrah, Khor Fakkan, Bab al Mandab, Ras Tannurah and Kharg variants.
On the test day (14 November) I also wrote down the place IDs GDELT used for these locations and hard-coded them. A location with one of those IDs matches its zone whatever its coordinates. That matters because GDELT placed “Qeshm” in Kurdistan. “Strait of Hormuz”, Kharg, Ras Tanura and Ras Laffan were never geocoded at all that day, so only the title rule catches them.
Where the rules get stricter
These are the exceptions I added after seeing what slipped through.
- Only city-level locations count for coordinates. GDELT’s location types 1 (country) and 5 (region) are just centroids, so I ignore them. Yemen’s centroid would drag in the whole civil war, and the Oman and UAE centroids sit close to the Gulf of Oman. Only types 3 (US city) and 4 (world city or landmark) are used.
- Big cities are ignored for coordinate matches. Cairo, Giza, Zamalek, Mecca, Sana’a, Dubai, Sharjah, Ajman, Doha, Lusail, Al Rayyan and Mumbai, plus everything GDELT puts in the Cairo and Giza governorates. They sat inside zone circles and mostly brought in general city news. An article about them still counts if its title names a zone, or through a place-ID or zone-name rule.
- “Red Sea” is a name collision. Red Sea Governorate in Egypt (Hurghada, Sharm el-Sheikh) and Red Sea State in Sudan share the name. A match there only counts if a market or maritime rule also matches. That removes tourism and Sudan-conflict news. I detect it from the region code, from place names (Hurghada, Sharm el-Sheikh, Marsa Alam, El Gouna, Safaga, Port Sudan, Suakin), or from those names in the title.
- The Arabian Sea lane needs a market or risk match, for both coordinate and name matches.
- The Persian Gulf “general” zone has no coordinates. Only the names “Persian Gulf” and “Arabian Gulf” trigger it.
Where a rule says “conditional”, it means the record must land in at least one real bucket (risk or a market), not just other.
What: the market buckets
A record gets a bucket if it matches at least one theme, or one organisation, or one title/URL keyword for that bucket. The theme codes below were checked against real GDELT data on 15 November. Several codes I first expected turned out not to exist and were replaced, and some broad ones were narrowed so they only count together with a keyword.
Risk
Cross-cutting: it applies to every market.
Crude and refined oil
Brent, tanker freight, diesel and jet fuel.
LNG and gas
Qatar supplies roughly a fifth of the world’s LNG and has no way around Hormuz. This bucket also watches Asian and European gas prices.
Fertilizer and agriculture
Roughly a third of seaborne fertilizer trade crosses Hormuz. GDELT’s themes are weak for this market, so it leans mostly on keywords and organisations.
Shipping and insurance
Tanker and container rates, war-risk premiums and Suez revenue.
Other
A zone match with no market or risk match. It’s stored for completeness and never triggers an alert.
How keywords are matched
- Case-insensitive, with word boundaries. They run on the page title plus the URL slug, with dashes and underscores turned into spaces.
- Ambiguous short tokens (
MAP,DAP,ONE,MSC,SPR,AIS) must be written in uppercase in the title, or appear next to a context word, such as “fertilizer” for DAP and MAP or “vessel” for AIS. The uppercase rule is ignored for headlines that are entirely in capitals. - Organisations are matched case-insensitively with word boundaries. Short names (four characters or fewer: OPEC, IEA, KPC, FAO, MSC…) must equal the GDELT organisation name exactly.
- Themes come from GDELT’s theme field and organisations from its organisation field.