Google Pilots Cryptographic Double-Blind Evaluation for Frontier AI
DeepMind announces the first double-blind evaluation of a proprietary frontier-class AI model, sealing external benchmarks in a cryptographic environment so the model cannot access them before testing. The pilot, run with Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons on a Gemini Flash Lite model, shifts benchmark protection from procedural secrecy to technical containment. Evidence is limited to the announcement; no protocol or results are disclosed.
For the first time in a published evaluation exercise, a proprietary frontier-class model is being tested against benchmarks it cannot access because the questions are sealed in a cryptographic environment. The claim is narrow but consequential: benchmark integrity is enforced by technology rather than by audit or contract. The substantive advance, if the protocol holds, is that contamination becomes impossible by construction, not merely illegal or logged.1
Context
The announcement identifies a known failure mode: if training or optimization data include evaluation prompts, scores can be inflated and benchmarks become meaningless. Previous safeguards described by the company included zero-logging and contractual restrictions on external test prompts. The current partnership with Singapore AISI, OpenMined, AVERI, and MLCommons adds a technical layer that confines evaluations to a cryptographic box and tests a Gemini Flash Lite model against confidential benchmarks.1
Assessment
What has actually moved is the trust anchor. The constraint shift is from procedural secrecy—do not look, do not log—to cryptographic containment, where the evaluation set is in a form the model and its training pipeline cannot use. This changes the burden of proof: instead of asking whether a lab complied, one can inspect a construction.1
Caveats
The evidence is an announcement. No protocol, threat model, or benchmark results are given, and world's first is asserted without external corroboration. To change this assessment, DeepMind would need to publish the cryptographic design, allow independent audit of the environment, and show comparative results demonstrating that contamination was excluded after training completed.1
Building trust in proprietary model benchmarks using cryptographically secure environments Imagine a student is set to take a high-stakes exam. If they accidentally peek at the test questions in advance, achieving a perfect score is influenced by this knowledge, making it a meaningless accomplishment. To truly measure what they know, they must have no visibility of the test questions until it's time to take the exam. That is the exact challenge the industry faces when evaluating advanced AI models. If a model has already seen the test questions - a problem known as benchmark contamination - the results can only be trusted to an extent. Today, we’re introducing the world’s first double-blind evaluation of a proprietary, frontier class AI model, which keeps external evaluations confined to a cryptographic “box” where they can’t be used by models later to optimize performance ahead of testing. We're partnering with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, to test a Gemini Flash Lite model against confidential benchmarks in a privacy-preserving environment, increasing evaluation integrity. At Google, we assess our AI systems using a broad spectrum of evaluations throughout model development and deployment, but we don’t rely on internal testing alone. To identify potential blindspots, we work with a diverse group of external partners, including specialized research labs, civil society and national AI Safety and Security Institutes (AISIs), using their unique expertise to stress-test our models. As AI models become more capable, ensuring the model has not seen the test questions or prompts in advance is critical, as this can skew the results. Policymakers, researchers, and enterprises need to trust that AI benchmarks accurately reflect a model's true capabilities and safety, but if models are able to “peek” at the evaluation questions in advance, it can artificially inflate scores and undermine this trust. Although zero-logging protocols and rigorous contractual safeguards have long kept external test prompts confidential, incorporating technical and cryptographic safeguards marks a major step forward in secure model evaluation. How double-blind evaluations work
The source text above remains the property of its publisher and is reproduced for commentary.
