Phase 2 · Pre-registration
✓v1.0 · frozen 26 Sep 2026Author: Evelyn Kim · First posted: 2026-09-26 · Kickoff: 2026-10-01 · Target close: 2026-12-31
Phase 2 · at a glance
Four vision-language models score the same 32 clips across five prompt conditions, three independent runs per cell, and are compared per clip against the plural human distribution Phase 1 measured. The plan is frozen before the runs so what counts as a “finding” is fixed in advance. This document is both the overview and the pre-registration.
Models
4
Gemini · Claude · GPT-4o · Qwen
Conditions
5
Zero-context → human-calibrated
Clips
32
The frozen Phase 1 corpus
Runs per cell
3
1,920 API calls total
Totals: the main sweep is 1,920 runs (shown above). A separate stochasticity robustness check adds 300 runs on three plural-eligible clips (see §2 · Stochasticity check below), for a planned overall total of 2,220 runs. The two serve different roles — the sweep tests the hypotheses, the robustness check tests whether the sweep’s N = 3 is enough to call effects stable.
§0
This document is posted before the first Phase 2 API call. It commits, in advance, to the study design, the five hypotheses, the metrics, and the analysis plan. At Phase 2 close (target: 31 Dec 2026), a companion “Phase 2 Findings” note will report against these hypotheses in the order given here, using the metrics defined below.
The Phase 1 pre-registration (/research/hypotheses) is the sister document for the Phase 1 human study. Phase 2 reuses its clip corpus, its response taxonomy, and its human distribution as the reference against which the AI response distributions are evaluated.
§1
Phase 1 produced two descriptive findings that together motivate Phase 2. First (Kim, 2026a), human readings of short social video clips are more plural than the single-label evaluation frame assumes; the median clip's modal share is around 70%, and a deep-plural tier of clips has no single answer at all. Second (Kim, 2026b), one contemporary multimodal AI (Gemini 2.5 Pro), evaluated against the plural human distribution rather than a canonical label, already tracks the plurality closely on consensus clips (88% modal agreement) and hedges inside the plurality on ambiguous ones (84% overlap on hedged responses).
Phase 2 asks the next question directly. Which contemporary multimodal AI configuration — model choice, prompt condition, or both — produces a response distribution that most closely mirrors the plural human distribution documented in Phase 1?
§2
Fully-crossed 3-way design at 640 unique cells with a uniform N = 3 runs per cell, for a total of 1,920 API calls. All AI runs use the same clip corpus, target-person description, and response taxonomy that Phase 1 humans used.
Amendment (2026-09-29): the original plan specified N = 1 runs per cell plus a 5-run stochasticity check on three plurality-eligible clips (Maverick, Elemental, Demolition), for 880 total calls. Uplifted to uniform N = 3 across every cell so within-cell variance is estimable everywhere, not just on the three subset clips. The analysis pipeline simplifies as a consequence — one variance model, no subset special- casing.
The 32-clip subset of the Phase 1 corpus (live-findings), excluding the Jules clip per the ingestion-failure disclosure in Kim, 2026b. Human reference distributions come from the frozen v1.0 dataset (10.5281/zenodo.22265192).
On the three plurality-eligible clips from Kim, 2026b (Maverick, Elemental, Demolition), each (model × condition) cell is run 5 times, giving 4 × 5 × 3 × 5 = 300 additional runs. This tests whether observed (model × condition) effects hold under repeated sampling.
§3
The primary metric is Jensen-Shannon divergence (JSD) between the AI response distribution and the human response distribution on the same clip. Lower is better; identical distributions give JSD = 0.
§4
Each hypothesis names a direction, an operational threshold, and the analysis that decides it. All thresholds are set on the primary metric (JSD) unless otherwise noted.
Claim. The Human-calibrated condition (E) produces lower AI–human JSD than the other four conditions, averaged across models and clips.
Test. Mean JSD in condition E minus the mean of {A, B, C, D}, on the same (model, clip) pairs. Threshold: E is lower by at least 0.05 in mean JSD on ≥ 3 of 4 models.
Claim. Model-level differences in JSD narrow substantially under condition E, compared to condition A.
Test. Range of per-model mean JSD is smaller under E than under A, on the same 32 clips. Threshold: range under E is at most 60% of the range under A.
Claim. Per-tier calibration gaps are larger on the deep-plural tier than on the consensus tier under condition A, and narrow under condition E.
Test. Difference-in-differences on tier-mean JSD, {condition A vs E} × {consensus vs deep-plural}, per model. Threshold: DiD ≥ 0.10 on ≥ 3 of 4 models.
Claim. Hedge-inside-plurality rate is stable at ≥ 70% across models under condition E.
Test. Rate computed over all (model × clip) cells in condition E. Threshold: rate ≥ 70% for each of the four models.
Claim. Within-cell JSD variance across the N = 3 runs is small enough that the (condition E − condition A) contrast holds under repeated sampling.
Test. For each (model, clip) pair, bootstrap the (E − A) contrast over the 3-run resample; require the 95% CI on the contrast to exclude 0 on ≥ 70% of pairs.
§5
All analyses use only the frozen Phase 1 v1.0 human distribution as the reference. AI response distributions are computed from the same 14-emotion + 9-cue taxonomy the human protocol used. Multi-emotion (“mixed”) AI responses are treated as first-class distributions rather than filtered.
Reporting order: (i) descriptive summary of the 640 cells (mean JSD per cell across the N = 3 runs), (ii) H1–H4 in sequence, (iii) H5 on within-cell variance, (iv) a small “deviations from plan” section for anything the pre-registration did not anticipate (e.g. a model ingestion failure, an unresponsive API window).
Ethics and Timeline sections were moved to the Phase 2 · Overview so this document stays a focused academic protocol.
§6
Related documents
This document is v1.0 (frozen 26 Sep 2026). The notes below cover edits made before the freeze — the shape of the study as it was being locked down. After the freeze, the plan is immutable; only typographic and citation fixes are made.
First public draft posted at /research/phase-2-hypotheses. N = 1 runs per cell with a 5-run stochasticity check on three plurality-eligible clips (880 total calls).
Qwen 3.8 27B substituted for Llama 3.2 90B Vision Preview in the open-weight slot. Groq decommissioned the Llama model before Phase 2 analysis began; same role in the design.
Uplifted to uniform N = 3 runs per cell across the full sweep (1,920 total API calls). Separates the main sweep from the stochasticity robustness check cleanly; within-cell variance is now estimable everywhere, not just on the subset.
Moved to /phase-2/preregistration; section and hypothesis numbering aligned with the Phase 1 pre-registration template.
Final structural edit before the kickoff date of 1 Oct 2026. No design-level changes after this point.