Creating Custom Event Data: A Bag of Tricks

Peace Science LLM and Conflict Pre-conference

Andy Halterman, Andreas Beger, Benjamin Bagozzi, Philip Schrodt, Grace Scarborough

This research was sponsored by the U.S. Government. The views expressed in this paper are the authors’ alone and do not represent the views of the U.S. Government.

Creating custom event data

Applied researchers who want to use event data face a dilemma:

Either, use off-the-shelf data

ACLED, UCDP GED, ICEWS, DEED, AAAD, …

  • No work to create it. Easy to justify to readers.
  • May not measure what you want: wrong event types, actor categories, or corpora.

Or, build custom event data

Tailored to your question and your corpus.

  • Measures exactly what you want.
  • Huge effort. Little tooling, no shared end-to-end pipeline.

Our goal: make the second option realistic for an applied researcher.

Our contribution

An open-source pipeline, NGEC, that you can run end-to-end or customize one step at a time.

  1. A five-step decomposition of the steps in making event data.
  2. Practical guidance: which steps to modify for different customization.
  3. Two new models for the hardest remaining steps: attribute extraction and entity linking.
  4. End-to-end validation on an ontology we didn’t train on: ECAV.

(A much earlier version of this pipeline generated POLECAT.)

A five-step framework

The pipeline

Event detection Step 1 of 5

Treat detection as document classification: a set of binary classifiers, one per event type.

  • Well understood in political science, and easy to customize.
  • We don’t provide new models, because prior work provides good options:
    • ConfliBERT (Hu et al. 2022; Brandt et al. 2025), ConflLlama (Meher and Brandt 2025), ConfliBERT-2 are efficient and accurate event classifiers.
    • Synthetic training data (Halterman 2025)

Two new models

Attribute extraction Step 2 of 5

Attribute identification is extractive^* and per-event:

The model

  • Document-level: one pass returns every instance of the event, with all attributes.
  • Model distillation approach:
    • Training data: 884 public-domain VOA articles + 1,008 synthetic stories.
    • A large open-weight teacher (GLM 5.3 flash) labels them under explicit span rules.
    • Fine-tune a small student, Qwen3.5-0.8B.
  • Fast, replicable, local, and the training data is fully releasable.

Adapting it: write a custom definition without retraining Step 2 of 5

The attribute model is conditioned on an event definition supplied at inference time:

event_type = "INVESTIGATE"
event_def  = "Investigate, charge, or prosecute a person or organization "
              "for wrongdoing, corruption, or crime."

To make it work with a new event type, write a new codebook definition (no need to retrain the attribute model).

The output is a JSON list of records with exact spans from the text, or [] if there is no event present:

{"event_type": "INVESTIGATE", "mode": "...",
 "anchor_quote": "...",
 "actor": ["..."], "recipient": ["..."], "date": ["..."], "location": ["..."]}

(Full prompt in the backup slides, model on Huggingface.)

Generalizes to new event types Step 2 of 5

We drop SANCTION and ACCUSE from training, then evaluate on them.

Condition F1, found events F1, all events
Trained on the type 69.2 66.5
OOS, with definition 69.8 51.4
OOS, bare label only 59.9 45.7
  • Bare labels are not sufficient, so provide complete definitions (Halterman and Keith ACL 2026b)
  • Cost to recall, but accurate on found events is equivalent

Document-level “coref” Step 2 of 5

  • The request is in a sentence that says only “He”.
  • The model still returns “President Bush”, plus the recipient.
  • Only 34.5% of event attributes can be found in the sentence describing the event (Li, Ji and Han 2021).

Internal validation on 500 VOA documents Step 2 of 5

We evaluate the attribute model on a held-out set of 500 VOA articles, annotated for all 16 PLOVER types (1,048 events). Gold spans are LLM-drafted and human-adjudicated.

All events F1 Found events F1 Found events, exact F1
Actor 52.7 71.7 60.8
Recipient 50.1 65.0 51.7
Date 58.9 77.9 76.7
Location 54.8 72.2 68.5
Mean 54.1 71.7 64.4
  • Misses cluster in multi-event articles: event recall is 88% with one event, 67% with two or more.
  • Context: strong ACE 2005 systems score 52.5–56.8. LDC annotators agreed with each other at 46.4.

Wikipedia Lookup Demo

Dates and Locations

Geoparsing (Updated Mordecai3)

Customizing for a new ontology

Electoral Contention and Violence (ECAV)

ECAV (Daxecker et al. 2019) is a good test: a different ontology, and it records raw spans and enough metadata to find the source story.

  • 1,662 events from 7 elections, 770 articles.
  • We did not retrain any model.
  • We did: write ECAV event definitions and an ECAV actor file.

We score against ECAV’s final coded values. The model gets the article, ECAV’s passage, and the event type, so this tests the steps after event detection (i.e., after ConfliBERT).

End to end: extraction is the bottleneck

  • Given ECAV’s spans, categorization reaches about 89%.
  • Recipients are hardest: 45.2, or 55.2 with ECAV’s default-target rule.
  • Dates: 85.8% exact, 94.9% within a day.

Why not just use a frontier LLM?

500 sampled ECAV events, with 95% intervals. LLMs assign ECAV categories themselves.

  • LLMs beat our student on recipients and locations. No clear difference on actors.
  • Sonnet 5 costs $592 per 100k events; NGEC on a rented GPU, $1.11.
  • Running our model twice = all answers identical. But Sonnet 25/50, GLM 3/50, local Qwen 45/50.

The pipeline is modular: a large LLM can replace Step 2 if it suits your project.

Customization

Places to customize

You want Change Retrain?
New event types Fine-tune classifiers (e.g., ConfliBERT); write definitions Classifiers only
New actor categories Edit the category file No
New attributes Add synthetic training data Attribute model
New corpus, similar English news Nothing No
Non-English text Models and Wikipedia index, or translate first Yes

Try it

  • Code is public: https://github.com/ahalterman/NGEC-2025
  • About an hour to install (mostly downloads): ngec download-index, ngec doctor. Includes instructions for coding agents (AGENTS.md).
  • Nvidia GPU, CPU, or Apple Silicon. All open-weight models.
  • Use it end-to-end, or just what you need: attributes, Wikipedia linking, geocoding, dates.

Thank you!

Andy Halterman

Slides and full paper at https://andrewhalterman.com.

NGEC: https://github.com/ahalterman/NGEC-2025

NGEC live demo: (email me for access)

Updated Mordecai Geoparser: https://github.com/ahalterman/mordecai3/

Geoparsing demo: http://104.131.30.142:9050/demo

Backup

Full extraction prompt

SYSTEM
Extract every instance of the given political event type from the document as a JSON list, or [] if there is none.

Each record: {"event_type": ..., "mode": ..., "anchor_quote": ..., "actor": [...], "recipient": [...], "date": [...], "location": [...]}

Every string in actor, recipient, date, and location must be an exact, verbatim substring of the document. Return the JSON list only.

USER
## Document: {document}
## Event Type: {event_def}

{event_def} is the full entry: event type, optional sub-type, and special instructions.

Attribute model on hand-coded ASSAULT stories

100 VOA articles, each coded by three authors. Spans scored only if the event is detected (68 of 93 gold events).

Field Exact Overlap
Actor 67.6 81.2
Recipient 55.9 90.4
Location 69.1 91.2
Date 98.5 98.5

End-to-end accuracy on ECAV

Percent of 1,662 events where the final value matches ECAV’s.

Field ECAV spans Raw NGEC NGEC + rules
Actor category 88.9 62.3 63.2
Recipient category 89.6 45.2 55.2
Location (within 25 km) 71.8 54.8 54.8
Date (exact) — 85.8 85.8
Actor + recipient 80.2 30.4 37.3
Actor + recipient + location 57.5 20.0 22.5

Rules: a protest, strike, blockade, or riot with no target gets State; an arrest with no actor gets State.

Writing custom dictionaries agents files helps

We compare using PLOVER’s “agents” file with PLOVER categories mapped to ECAV, or writing an ECAV-specific “agents” file.

Category accuracy PLOVER mapped to ECAV ECAV-specific file
Actor 65.6 88.9
Recipient 70.2 89.6
  • Clear implication: write role patterns for custom ontologies. Don’t crosswalk from an old one.
  • It’s a text file matched by semantic similarity, not an exhaustive actor list.

Where actor categories go wrong

ECAV \ NGEC State Citizens Party Armed Foreign Unknown No span Total
State 89 17 6 17 0 2 49 180
Citizens 24 259 59 6 2 21 51 422
Party 14 59 82 5 1 5 36 202
Armed group 11 5 17 102 7 47 65 254
Foreign 23 2 1 21 63 6 22 138
Unknown 11 17 3 23 3 121 288 466

Where recipient categories go wrong

ECAV \ NGEC State Citizens Party Armed Foreign Unknown No span Total
State 244 32 28 19 19 35 244 621
Citizens 30 131 18 6 3 57 101 346
Party 25 22 111 1 0 20 49 228
Armed group 7 20 0 76 4 12 40 159
Foreign 28 19 2 8 88 10 29 184
Unknown 5 51 1 4 1 13 49 124

Cost and replicability

Model $ per 1k events ECAV (1,662) 100k articles
Claude Sonnet 5 $5.92 $9.84 $592
NGEC, GPU $0.011 $0.018 $1.11
NGEC, CPU (1.26 h) (2.1 h) (126 h)
50 ECAV documents, run twice Sonnet 5 GLM 5.3 Qwen3-14B (local)
All answers identical 25/50 3/50 45/50
Refused calls 0 40 (7 docs, Tibet) 0

Date resolution

  • Handles historical dates, future dates, and ranges, with granularity.
  • Historical events can be filtered out after coding.
  • A resolver fix took exact-date accuracy on ECAV from 43% to 85.8%.

End to end on the same text

Limitations

  • Not validated here: event detection and false positives on ECAV.
  • No cross-document deduplication.
  • English only: translation vs. retraining is an open question (Osorio et al. 2025).
  • Errors cascade across steps. We measure this but don’t fully address it.
  • Media reporting bias is unchanged.

Categorizing entities after Wikipedia

Wikipedia linker: feature ablation

Mordecai demo: http://104.131.30.142:9050/console