Theoretical positioning
The Observatory treats AI ethics texts as discourse rather than transparent statements of belief or intent. Each coded item is read as a structured speech act.
Every item makes claims, allocates responsibility, positions its speaker, and either connects those claims to enforceable mechanisms or does not. The instrument is designed to surface that structure, not only the topical content.
It operates on three coordinated, non-interchangeable axes. Principle articulation describes what is claimed, drawing on the mature literature on AI ethics principles. The DPPRA structural rubric describes how those claims become operational. And a set of integrity subcodes describes what is at stake when claims fail.
A principle-only scheme cannot distinguish a corporate statement that asserts "we value fairness" from a binding government directive requiring named offices to implement civil-rights review on a deadline. Both score for justice. The structural rubric exists to make that difference visible and reproducible.
Corpus design
The corpus is partitioned into five sectors, each coded as a separate sub-corpus while sharing the same primary instrument, so that cross-layer comparison is structurally valid rather than thematic only.
Forum / Public discourse is the project's contrast layer: where the other four sectors are institutionally authored — curated reality — public discourse registers the lived reality of those subject to AI systems rather than those authoring policy about them. Run 1 bears this out: the forum layer carries the highest accountability-displacement in the corpus.
Each layer differs in its conditions of production — who can publish, what review obligations apply, what accountability the speaker carries. Pooling these texts would obscure the variable that most strongly conditions the data. Selection within each layer is purposive, not random, documented at the item level so the sampling rationale is reconstructable. The corpus itself is not republished here — every document is a public artifact, and the item-level source manifest is available from the principal researcher on request.
The ICST instrument
The ICST v2.14 is currently active. Each version is preserved as a canonical artifact, and version increments are accompanied by a formal revision memo so that coding done under an earlier version remains interpretable.
The instrument is delivered as a single coding prompt combining a Source Manifest Lock, per-principle scoring on a fixed principle set, DPPRA structural scoring, integrity subcode coding, a discourse-function panel, a Full-Template Completion Gate, and a coder-neutral Template-Compliance Exclusion Protocol using generic placeholders.
The DPPRA scale
DPPRA names the five dimensions along which a document is assessed for how far it connects ethical discourse to material implementation. Each item is then given a single 1–5 alignment score reflecting the depth of that connection.
Decoupling
The central question is whether ethical discourse is coupled to material practice or floats free of it. The spectrum runs from earned legitimacy to symbolic ceremony.
In this corpus, reflexive decoupling is frequently neoliberal in origin — market logics make the symbolic adoption of ethics rational even where substance is absent. The decoupling_signal code and the alignment caps operationalize this spectrum.
Integrity subcodes
Generic harm language under-specifies AI risk. Six domain-specific, non-substitutable integrity subcodes name where AI threats are observed. Each must be coded under its proper subcode rather than collapsed into a generic label.
The separation is enforced because earlier runs showed assessment integrity being applied to model testing in non-education contexts, producing systematic over-counts. Model testing is System Integrity / TEVV; hallucinations are Epistemic Integrity; provenance is Data / Provenance Integrity.
Discourse-function fields
The instrument codes the rhetorical work a document performs. Legitimacy Work captures justification of action through public protection, trustworthy governance, or responsible stewardship. Advocacy / Agenda Setting captures the pushing of a procurement strategy, market structure, or policy direction. Aspirational Positioning captures descriptions of desired future outcomes.
Compliance Framing is coded conservatively — a source is not compliance-framed simply because it discusses policy or standards; it must substantially frame conduct around legal compliance, enforceable obligations, or auditable duties. Responsibility Shifting is distinguished from ordinary responsibility distribution: distributing duties across agencies and vendors is shifting only when duties land on actors lacking adequate authority, resources, or recourse.
Procedural safeguards
Two safeguards govern data quality. The Source Manifest Lock requires every record to begin with bibliographic and provenance fields. The Full-Template Completion Gate prevents partial submission — a record missing any required field is invalid.
Both were introduced after earlier rounds produced records where observations lived only in memo prose, with structural fields blank. The operating principle: memo-only claims constitute data loss, not merely a formatting issue. The structural fields are the data; memos are commentary.
Phase 0 inductive coding
Before the structured ICST pass, an inductive open-coding pass surfaces emergent themes, framing types, and relevance flags as structured JSON. It serves three purposes: it generates an emergent-theme inventory to check for codebook blind spots, produces per-item human-review flags, and provides a baseline for evaluating category capture. If Phase 0 surfaces a recurring theme the structured codebook cannot accommodate, that is evidence for instrument revision.
Multi-AI coder stress test
The stress test uses multiple LLMs as parallel coders on the same items under the same prompt. The design is symmetrical: identical prompt, source, and output schema. Divergences are not treated as errors by default — they are data about the instrument and about coder behavior, measuring instrument reliability, model-specific drift, and robustness under coder substitution.
Each watch-out is itself a documented finding about coder behavior under a specific instrument — a research output, not a workaround.
Adversarial counter-reading
Agreement among three cross-family coders is reassuring, but it is not proof. Coders can converge for the wrong reasons. The adversarial pass exists to challenge consensus rather than confirm it.
After Claude, Codex, and Gemini have independently coded an item and their outputs are compared, a dedicated counter-reading pass takes the resulting three-coder consensus and builds the strongest available case against it. Where the coders agreed, it asks whether they agreed for sound reasons or shared a blind spot — a common tendency to read institutional language charitably, a shared training-data prior, or a convergent misreading of the source.
This matters because cross-family coding controls for model-specific drift but not for biases the models hold in common. The adversarial pass is the step that interrogates the agreement itself.
"Three coders agreeing tells you the reading is stable across model families; it does not tell you the reading is correct."
Counter-readings are not silent overrides. A divergence the adversarial pass surfaces becomes a documented review flag — evidence for human adjudication, for a watch-out instruction, or for instrument revision — recorded alongside the consensus rather than replacing it. The disagreement is data, the same way coder divergence is data in the stress test.
The pass is also built to help mitigate investigator bias. The coders read through an instrument the principal investigator designed — its rubric, watch-outs, and cap-and-floor rules encode interpretive priors. To the degree the counter-reading is licensed to challenge not just how the coders applied the instrument but whether the instrument itself produced the reading, it introduces structural friction against the designer's frame. Its strongest form is one in which the adversarial prompts are written independently of the principal investigator, so the critic does not inherit the priors it is meant to test.
Cross-family coding controls for differences between model architectures; the counter-reading interrogates their shared agreement. The lens-swap adds a second adversarial axis: re-coding the same document through deliberately opposed theoretical viewpoints — reading once as a political economist, once through a governmentality lens, once as an institutional theorist — while holding the coder constant. It has now been run in two waves over sector-representative documents from the corpus, paired with the counter-reading pass on the same texts.
It asks a different question than reliability: not do the coders agree, but does the reading depend on the theoretical standpoint applied. The instrument's fields split cleanly under the swap: some hold under every lens — the ethics spine, the alignment gate, the mechanism-bound intensity scores — and behave as constructs, while others move with the standpoint applied — who captures value, who materially benefits, the decoupling and market-capture signals — and are reported as standpoint-dependent rather than anchored. The sharpest result comes where the two passes converge: the fields the lens-swap marks as standpoint-dependent are the same fields where the counter-reading finds three-coder consensus most fragile. On those fields, agreement is a flag to interrogate, not a validation. Full results will accompany the forthcoming methods paper.
Coding workflow
Coding is performed through chat interfaces rather than the API. This avoids per-call costs and supports the project's sustainability. The methodological consequence is that the workflow is a chat workflow, not a programmatic one — a deliberate cost-and-sustainability choice with consequences for tooling design, not a limitation.
The data layer is purpose-divided across four systems of record: