Greek-Language Reasoning Reveals What Accuracy Benchmarks Cannot See

A pre-registered study of three frontier MoE models fine-tuned to reason in Greek finds accuracy benchmarks are noise-dominated, while SFT shifts nearly all hidden reasoning traces to Greek, and RL with verifiable rewards fixes SFT's format-adherence and leakage defects. The authors release five checkpoints and instruments, arguing that reasoning language and instruction-following are behavioral dimensions that accuracy cannot see.

Hugging Face Daily Papers — Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

August 22, 20261 source70% confidence

78Strategic

Strategic score

Evidence strength70

Technical novelty85

Commercial potential65

The paper reports a pre-registered study in which three frontier mixture-of-experts models were fine-tuned to reason in Greek, a low-resource language for which none of the base models emitted any reasoning trace. Its headline result is a null on accuracy benchmarks, where seed noise (7.7 points) dwarfed all treatment effects, accompanied by large, reproducible changes in the hidden reasoning channel: SFT moved models from 0% to ~98% Greek-language traces, while RL with verifiable rewards repaired format-adherence and leakage defects that SFT left untouched.¹

Context

The study is best read as a measurement argument rather than a benchmark win. The authors define 'thinking in Greek' behaviorally, propose six dimensions gated against length-correlated metrics, and document six ways their own instruments failed, each caught by a control. This turns the conventional accuracy-only evaluation into a process-level evaluation of chain-of-thought language and instruction following.¹

Assessment

The strongest evidence is the combination of pre-registration, random-reward controls, and multiple model families. The claim that accuracy cannot see the phenomenon is supported by the seed-noise comparison, but only for the specific benchmark and scale tested. The RL results are large (24% to 2.5% format fallback; 3.5% to 0.0% leakage; +9.1pp on English instruction obedience) and directionally consistent across all three models.¹

Caveats

AI hypothesisWhat would change the conclusion: if a higher-powered accuracy evaluation showed real forgetting; if the behavioural instruments were shown to correlate with output length despite the gates; or if the Greek-language reasoning habit did not survive an accuracy-only gradient in a direct ablation. The paper also leaves open whether the six dimensions generalise beyond Greek and whether the released checkpoints are safe to deploy when the user's question language differs from the desired reasoning language.¹

Source text

Hugging Face Daily Papers — Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See · August 18, 2026

Computer Science > Computation and Language

Title:Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Abstract:Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.

Current browse context:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The 7.7-point seed shift is the paper's epistemic cornerstone: it states that random seed variation exceeds every data/recipe effect, so the reported accuracy null is not evidence of 'no change' but of an underpowered instrument. The number's basis is the benchmark score distribution across seeds, not a confidence interval.

The '0 of 1,000 reasoning traces' claim depends on an unstated assumption: that the model's hidden chain-of-thought was sampled and classified as non-Greek. The authors treat the language of the reasoning trace as a user-facing property even though it is not normally auditable; this is the premise for the whole behavioral framework.

A 'verifiable reward' for format and leakage is a rule-based reward, not a learned reward model; pre-registration protects against reward hacking and post-hoc selection. The phrase 'against a flat random-reward control' means the treatment is compared to a model trained with random rewards, isolating the reward signal from the RL procedure itself.

'Survives an accuracy-only gradient' refers to a training/selection signal that the authors presented as untouched by accuracy: the Greek reasoning habit persists when optimization pressure is purely accuracy-based. The annotation clarifies that this is the key evidence that accuracy benchmarks are blind to reasoning-language, and it is only as strong as the fidelity of the trace classifier.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

August 18, 2026endistinct angle

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

↑ Back to top