What codebook is your LLM using?

Tue, 14 Jul 2026 08:21:31 +0000

At ACL last week, Katie Keith and I presented a paper, “What is a protest anyway?" that argues that conceptualization—defining what your labels like “protest” mean before getting an LLM to annotate your documents—is still absolutely critical. Take, for example, this sentence:

“Upset about the election results, a group of angry youth smashed the windows of local businesses.”

Is this a protest? In everyday usage, this probably is. And if you ask an LLM to classify it as a protest or not, it will happily return PROTEST or NOT_PROTEST. However, we argue that classifying documents in this way, using bare labels like “PROTEST” causes serious measurement problems.

Rich social science concepts like “protest” are rightly defined in quite different ways: a project investigating the success of large-scale peaceful mobilization would want to exclude the story above from its set of protests, while a project on the early warning indicators for political violence might want to include it in its definition of protest. These detailed, project-specific definitions are usually written in codebooks that can range from a paragraph or two to hundreds of pages. And indeed, different existing codebooks in social science have definitions of protest that would either include or exclude that example:

Figure 1: Existing social science definitions of “PROTEST” disagree widely on different aspects of protests (Figure 2, Halterman and Keith 2026)

Our paper’s main arguments are that there are no “correct”, universal definitions of concepts, that any social science concept requires a carefully written and explicit definition of what’s included or not based on their measurement problem, and that LLMs can tempt researchers into skipping this step because they’re so good at generalizing from minimal instructions. Moreover, an incomplete definition of the concept cannot be corrected using new bias correction techniques such as PPI or DSL, if the gold standard labels themselves are produced using a flawed codebook. I encourage you to go read the paper!

But what “codebook” do LLMs implicitly apply?

A natural followup question that someone asked at ACL is what implicit definitions of a “protest” do LLMs actually use? If LLMs are not told which definition of protest to apply and are just asked “Classify as PROTEST or NOT_PROTEST”, which definition of protest do they actually implicitly use?¹

Setup

To probe an LLM’s implicit definition of protest, we can run some simple experiments. We can write different vignettes to probe aspects of a protest definition then ask an LLM to classify whether it is a protest.

For example, codebook definitions of protests apply very different thresholds for the number of participants required for something to count as a protest. ACE requires “a large number”, ACLED requires 3+, CAMEO implicitly has no threshold, and CCC explicitly has no size limit.

To probe this behavior, we can prompt an LLM with:

Is this a protest? Answer with a single word, YES or NO.

Text: A lone activist stood outside the parliament building for the third 
consecutive day, holding a hand-painted sign condemning the new tax law 
and handing leaflets to passersby.

If the LLM returns YES, then it’s (implicitly) applying a definition closer to CAMEO or CCC that do not have size requirements, and a NO is closer to ACE or ACLED, which require more than one participant for something to count as a protest.

We can repeat this for other vignettes testing the aspects listed in the table above across different LLMs, including large commercial LLMs from Anthropic and OpenAI, large open weight models such as GLM, and smaller, open weight models that you can run locally. These are an ad hoc collection of models, and note that for closed weight models, the behavior of the model can change if you come back to it later.

Finally, we can get an approximate continuous estimate of a model’s confidence by looking at the token probabilities for the answers or repeatedly sampling $k=10$ labels with non-zero temperatures from the closed weight models.²

Results

Because we’ve prompted the LLM to classify documents without providing a codebook or definition to specify what we mean by “protest”, each LLM has to generalize to what the entire scope of the idea of “protest” is from a bare label. To understand which definition of protest each LLM is implicitly applying, we can compare its label on constructed examples against what each of the four codebooks would say about each class.

Figure 2: How do LLMs score different aspects of protest?

First, the LLMs (ordered very roughly by capability) are fairly consistent in how they annotate many aspects of protest. All correctly classify the canonical protest vignette (“About two thousand people marched through the capital’s main square on Saturday, chanting slogans and carrying banners denouncing the government’s handling of the crisis.") and the obvious non-protest vignette (“The city council met on Tuesday evening to review the proposed annual budget, hearing presentations from the finance department before adjourning without a vote."). And you can identify some commonalities across models (the biggest four models all exclude vigils, most models exclude demonstrations in support of a policy, all models include labor strikes, etc). And note that the original codebooks disagree on whether to include these or not!

But the main takeaway is that different models employ different implicit definitions, so it’s challenging to identify “the implicit LLM codebook” that all share. At most, the models seem closest to applying a fairly standard dictionary-type definition like Wordnet‘s “a public (often organized) manifestation of dissent”.

To return to our original question: a researcher who uses Sonnet 5 to annotate documents will obtain a dataset that excludes violent protest, while a researcher using the same prompt with GPT-4o will obtain a dataset that includes violent protests. And in neither case will the coding rules be explicitly documented somewhere, and any substantive results that the researcher obtains from these datasets will be dependent on an opaque, implicit concept of protest employed by the LLM.

The most common protest dataset is least reflected in LLM coding

What happens if you just take an LLM’s answers as-is? Which codebook is it closest to? We can construct a rough agreement measure by taking the proportion of categories where the LLM’s answer matches each codebook’s answer.³

Figure 3: Rough congruence analysis: what percentage of LLM answers align with each codebook’s definitions?

Although ACLED is perhaps the most widely used international dataset of protests, its definitions are not well reflected in any of the LLM annotations. CAMEO, in contrast, is the closest, but still differs from most LLM annotations in about 1 out of 5 aspects.

Minor prompt variations shift which implicit codebook is being used

All of the results above use the same prompt: “Is this a protest? Answer with a single word, YES or NO.” It’s very well known that LLMs are sensitive to slight variations in prompts. Do minor prompt variations change which “codebook” is implicitly being used?

We can check this by re-running the analysis above with two new variants of the same prompt:

“Does the following text describe a protest? Answer with a single word, YES or NO.”
“Classify the following document as PROTEST or NOT_PROTEST. Respond with only the label.”

What we find is the expected instability in output, which, in this situation, means instability in the implicit definition that’s being applied.

Figure 4: How do LLMs’ answers change as you vary the prompt between three variations of the task prompt? The values shown are the maximum difference in answer probability across the three prompts.

Whereas before, Sonnet 5 always excluded events where protestors were committing violence, giving it an alternative prompt can switch its behavior to often including violent protests (0-0.80). Something about the alternative framing of the prompt, which seem substantively identical, makes it switch from an implicit ACLED definition to an implicit ACE/CAMEO/CCC-type definition.

So not only are LLMs given bare class labels implicitly picking a definition to apply, the definition they use can also flip with seemingly irrelevant rewrites of the prompt.

Lexical shortcut

Our earlier “Codebook LLMs” paper found that earlier generations of LLMs tend to anchor their labels on keywords that are present in the text. We see the same behavior in these nine models. I wrote an example of a document that does not meet the definition of protest in any of our four codebooks, but does include the word “protest”

“The ambassador lodged a formal protest with the Foreign Ministry, complaining about the treatment of her local citizens, who she claimed had been denied their rights.”

Almost all models (with the exception of GPT-4o) coded this as a protest, while most researchers of political mobilization and protest would not be interested in formal diplomatic complaints.

There’s no escaping conceptualization

The figures above are not meant to show that LLMs are “bad” at classifying protest. Instead, the point is that abdicating your role of carefully defining what you want the LLM to classify means that the LLM falls back on a sense of the bare label it learned during pretraining. An LLM’s implicit “codebook” comes from the pretraining and post-training data it sees, which are shaped by the (human) decisions made at each company. In almost all cases, those decisions and data are hidden from the user of the LLM. Here, that’s something that looks sort of similar to the Wordnet dictionary entry. So the model will always use a definition: you just want it to be explicit instead of implicit, and your definition instead of the definition learned from pretraining.

We’ve used protests as a running example here, but the same concerns apply to other concepts. For instance, asking an LLM to score a political manifesto on a left-right ideology scale has the same issue: you’re relying on the LLM’s implicit definition of ideology, rather than the sense that you want to measure.

If you’re interested in why incomplete codebooks can’t be fixed downstream, the ACL paper has details on that. And if you’re interested in what LLMs actually do with complete codebooks and how to know if an LLM is following the codebook instructions, our earlier Political Analysis article is about that.

So, tl;dr: if you’re asking an LLM to label documents, tell it what you mean by the label. And if you don’t, it will use an implicit definition that might not reflect your concept and that varies across models and minor prompt variations.

Acknowledgements: Thank you to Katie Keith for comments on a draft of this post.

To get future blog posts in newsletter form, you can sign up here:

Our earlier “Codebook LLMs” paper looks at how LLMs follow (or don’t) the instructions provided to them in codebooks, but doesn’t try to reverse engineer the implicit definition an LLM uses. ↩︎
Technical note: for the smaller open-weight models, I set temperature=0 and look at the log probabilities for the token labels. The models I access through API, such as Claude, GPT, and the (open weight but very large) GLM-5.2, do not expose the token probabilities. Instead, I generate multiple ($k=10$) responses with temperature=1 and average the responses. GPT-OSS, which I run locally, requires temperature>0 for the (required) “thinking” step before answering, so I also apply the sample multiple sample and average approach. This means that the probabilities across the techniques aren’t directly comparable, but we can still get a rough sense of model “confidence”. ↩︎
This doesn’t account for the frequency of these aspects or the importance of each and is dependent on the specific wording of each example. But this is a blog post, not Political Analysis! ↩︎

codebooks | Andy Halterman

What codebook is your LLM using?

But what “codebook” do LLMs implicitly apply?

Setup

Results

There’s no escaping conceptualization