One instrument, one corpus, a growing line of studies.

Coders as Standpoint Instruments

The question. What does disagreement among cross-family LLM coders tell us, if agreement alone cannot establish validity?

The design. Three coders from Anthropic, OpenAI and Google coded all 65 documents under ICST v2.14 (RUN 07).

The result. Ordinal α was .725 and nominal α .706 (RUN 07, v2.14, 65 documents), and coder identity was a large, systematic factor. Models can converge through shared training rather than comparable reading, so the paper treats their differences as diagnostic rather than as noise to average away.

Published conference abstract

Disagreement as Evidence: Cross-Family LLM Divergence as a Reliability Signal in AI-Ethics Discourse

Singleton, C. (2026). In SAMYRAD 2026 Book of Abstracts (p. 120). Intelligent Nudge. doi:10.68286/samyrad.2026.abstracts

A work-in-progress abstract, written while the coder roster still included Perplexity. The full paper reports the final roster.

All in the Family?

The question. If coder identity matters this much, how far can LLMs be treated as independent coders at all? Are models from one vendor closer to each other than to models from another?

The design. The same 65 documents and the same frozen instrument, widened to 16 models from six vendors. Fifteen models were retested, so each model's agreement with itself serves as a reference.

The result. Relative to retests, selected non-small models averaged .926 within vendors and .842 across vendors. The study is exploratory and measures agreement, not accuracy.

Institutional AI policy across a university system

The question. Do coders identify the same textual evidence even when they assign different codes, and does the distinction between construct and standpoint fields hold in a new setting?

The design. The framework moves to AI policies across a multi-campus university system, with a preregistered prediction: ratings under a fixed protocol converge on construct fields, while standpoint fields stay differentiated.