Behavioral Disposition Alignment in 25 LLMs via Situational Judgment Tests

Google Research published a framework converting validated psychology questionnaires into situational judgment tests for LLMs. In a study of 25 models, small models aligned with human consensus at near-chance rates, large frontier models plateaued in the low-to-mid 80s when consensus was under 90%, and models were systematically overconfident in low-consensus cases. The authors also found that self-reported traits diverge from revealed behavior, suggesting the framework offers a more reliable behavioral alignment evaluation.

What happened

Source factGoogle Research introduced a systematic framework that adapts established psychometric questionnaires (IRI, ERQ, and related instruments) into Situational Judgment Tests (SJTs) for LLMs. Scenarios are generated by LLMs, validated by three annotators, and model responses are mapped to one of two actions using an LLM-as-a-judge. Human preference distributions are collected from 550 participants with 10 annotations per scenario, and the study evaluates 25 LLMs across four distinct traits. Two alignment gaps are reported: directional deviation from human consensus in high-agreement scenarios, and overconfidence when human opinion is divided.

Source factQuantitative results showed that models below 25B parameters often align with human consensus at near-chance rates, while large-capacity (>120B) and frontier closed-weight models reach near-perfect directional alignment under unanimous consensus but plateau in the low-to-mid 80s when consensus falls below 90%. Qualitative analysis identified recurring deviations: models encourage emotional openness in professional settings where humans favor composure, prioritize harmony over standing one's ground in social disputes, and occasionally recommend immediate action despite missing logistical verification.

Source factThe paper also compared self-reported behavioral dispositions with revealed behavior. Models frequently self-report low impulsiveness yet show behavior leaning toward impulsiveness, and similar inconsistencies appear across traits. This suggests limitations in self-report formats and motivates the SJT framework as an alternative evaluation lens.

Why it matters

AI analysisThe results demonstrate that current LLMs do not reliably mirror human behavioral dispositions, especially in realistic advisory situations. Even state-of-the-art models are overconfident when human preferences are split, which is problematic for assistant systems that should communicate uncertainty and avoid imposing a false consensus. This matters because LLMs are increasingly embedded in daily decision-making, where subtle behavioral misalignment can have tangible social consequences.

What changed

AI analysisThe key methodological shift is from self-report trait declarations to observation of behavior in contextualized scenarios. The paper offers evidence that self-reports are not a dependable proxy for model behavior, challenging the validity of earlier alignment evaluations that rely on questionnaire statements. It also introduces the distributional pluralism criterion, which requires model confidence to track human agreement levels, adding a new dimension to alignment beyond average preference matching.

What is actually new

AI analysisThe novelty lies in combining validated psychometric instruments with LLM-generated realistic scenarios at scale, and in using human consensus distributions rather than binary labels to define two distinct alignment targets: directional alignment (matching the majority) and distributional pluralism (matching the spread of opinions). The large cross-model comparison and the demonstrated self-report versus behavior divergence are new empirical findings that are likely to influence future alignment benchmarks.

Evidence assessment

AI analysisThe main source is a Google Research blog post; the referenced paper is not directly accessible in the event evidence. The method is described in sufficient detail to be internally coherent, but raw data, dataset artifacts, and full scenario lists are not provided. Therefore, while the findings are plausible and valuable, independent replication and open release are necessary to confirm the magnitude of the reported gaps and the ranking of models.

Constraint shift

AI analysisFor developers, this framework shifts the alignment target from a single preferred action to a distribution over actions that mirrors human agreement. It introduces a new constraint: models must be calibrated not only in correctness but also in social confidence, expressing uncertainty when humans disagree. This is not captured by standard RLHF or preference optimization, and may require new training objectives and evaluation protocols.

Implications

AI analysisCommercially, this framework could underpin alignment benchmarks and safety evaluation services for LLM-based advisors. Scientifically, it offers a robust foundation for studying behavioral alignment and testing whether training interventions reduce overconfidence and consensus deviations. The resource cost of human annotation and LLM-as-a-judge is nontrivial, so the highest-value use is likely as a targeted evaluation harness rather than a continuous training signal.

What would change my mind

AI hypothesisMy confidence would decrease if independent replication failed to reproduce the near-chance performance of small models or the saturation pattern among frontier models, or if the self-report discrepancies were shown to be artifacts of prompt phrasing. Conversely, if the paper's data and scenarios are released and the findings hold across non-English and culturally diverse populations, the framework would probably become a standard evaluation tool. A stronger disconfirmation would come from evidence that current models can be made distributionally pluralistic without explicit large-scale annotation.