The existing methodology for open-ended text analysis used human raters and was slow, costly, and unreliable. I developed and validated a novel metric using NLP techniques that automated and scaled the assessment process.
Read the full paper (PDF) (opens in a new tab)Adapted from King & Gentner (2022), Cognitive Science, co-authored with Dedre Gentner. All statistics are from the published paper. Fitted lines in the validation chart are redrawn from its figures and are approximate. For the language findings themselves, see the companion case study.
Scored automatically across two experiments, with no splitting and no human raters needed to score them.
An independent human benchmark reproduced every pattern the metric found, one of them marginally.
Blind-coded metaphoric verb changes rose steadily across the metric's score quartiles, from least to most changed.
Paraphrase length didn't track strain (p = .82 and .49), and controlling for it left the estimates nearly unchanged.
The research needed to know how much the noun and the verb each changed when people reinterpreted odd sentences. But paraphrases blend the two, and there's no ground truth to check against.
Which part came from car and which from limped? Every earlier method depended on human judges answering questions like this.
Word embedding models from NLP offered a solution to the human-rater problem: reliable estimates of meaning change that can be calculated instantly, scaled to as many datapoints as necessary, for free.
I developed a method for producing noun- and verb-change scores for each participant's response using cosine similarity as a metric.
Lower scores = more meaning change
If the verb-change score is lower than the noun-change score, then the verb changed more
The verb's score (.22) is lower than the noun's (.47): the verb changed meaning more.
A new metric earns trust by passing tests it could have failed. Each check below uses evidence that doesn't depend on the embeddings themselves.
Embeddings have known blind spots, so I re-measured Experiment 2 with no algorithm at all. New people paraphrased the paraphrases, and I counted how often the original words came back: lossy, but objective.
As sentences get stranger, verb similarity falls (p = .001) while noun similarity stays flat (p = .65).
In the human task, verbs came back less often as strain rose (p = .02). Nouns didn't move (p = .69). Different instrument, same shape.
Nouns came back 2× as often: 21% vs. 10%Embeddings are known to be less accurate for words with many meanings. Humans showed the same polysemy effect: many-meaning words changed more in both (humans: verbs p < .01, nouns marginal at p = .06).
3 of 3 patterns reproduced| Effect | Metric (Exp 2) | Human benchmark (Exp 3) |
|---|---|---|
| Verbs change more than nouns | β = 0.04, p = .01 | β = 1.24 (logit), p < .001 |
| Strain → verb change | β = −0.29, p = .001 | β = −0.25, p = .02 |
| Strain → noun change | β = −0.03, p = .65 | β = −0.02, p = .69 |
| Polysemy → verb change | β = −0.22, p = .01 | β = −0.63, p < .01 |
| Polysemy → noun change | β = −0.16, p = .01 | β = −0.20, p = .06 |
Human resurfacing rates: nouns 24.2% (few meanings), 18.0% (many); verbs 13.9% (few), 6.9% (many). Fitted lines are read from the paper's Figs. 4 and 5 and are approximate.
Once validated, the metric changed what the research could afford to measure.
Scoring is deterministic, so there's no judge-to-judge disagreement. Even the human coders in this study started at only moderate agreement (κ = .58–.66).
1,742 paraphrases scored with no raters and none lost to the method. One human method got a signal from as few as 4% of verb responses.
The pretrained vectors cost nothing to use. Re-measuring one experiment with the human method took 77 paid online participants.
The same approach, scoring text with embeddings and then validating the scores against people, carries over wherever open-ended responses need to become numbers.
Ask respondents to restate an item in their own words, then score how far their reading drifts from the intended meaning: cognitive pretesting at scale.
Score engagement, pulse and exit-survey comments against theme descriptions instead of hand-coding thousands of responses.
Compare how users describe a feature or task with the product's own language to find where understanding breaks down.
Measure how the language around a topic, like career growth, onboarding or a new feature, shifts between survey waves or releases.
Whether the scorer is an embedding model or an LLM, the validation steps here (replicate a known result, benchmark against human judgment, rule out confounds) are what make a new measure safe to act on.