Frontier AEO Tracker Reveals Self-Preference, Confidence Shifts, and Evaluation Feasibility
Latent Space's Frontier AEO Tracker runs six prompt variants across seven frontier models and 161 categories, reporting proprietary product-recommendation scores, model self-preference, and lower name-switching in Astra; evidence is a single blog post.
Latent Space's new AEO Tracker extends prior work by running six prompt variations across seven frontier models in 161 categories, with Astra performing answer extraction and a proprietary AEO score weighting first choices, alternatives, mentions, and anti-recommendations. The novelty is less the leaderboard itself than the inspectability of every prompt-answer pair and the separate analysis of model-generation flips (Sol to Astra, Opus to Fable).1
Context
The evaluation is explicitly competitive: categories range from coding agents and ASR to angel investors and payroll software. The report documents strong self-preference, such as Fable/Opus choosing Claude Code, Sol/Astra choosing Codex, Grok choosing Cursor, Muse choosing Muse Code, and SWE-1.7 choosing Devin, alongside notable cross-vendor endorsements like GPT models recommending Claude. Twenty-eight of the 161 categories have a universally dominant primary choice among all surveyed models.1
Assessment
The most consequential constraint shift is from evaluation cost to model access: after a few billion tokens of prototyping the authors can run cross-model probes routinely, but Gemini/Antigravity, GLM/Zcode, and DeepSeek/DeepCode could not be included because of errors and rate limits. Substantively, Astra searches fewer sources on median (5) than Sol (9), Opus (11), or Fable (15), and Astra is far less likely to change its answer under light paraphrasing. If choices are becoming more deterministic, AEO investment should rise in value because prompt-level randomness will no longer obscure product-level preference.1
Caveats
AI hypothesisThis should be treated as evidence-generating, not definitive. The scoring formula is proprietary; contamination is asserted as checked but the method is not described; source counts come only from attempted toolcalls, not pretraining data; and the model sample is deliberately restricted. I would revise this assessment if independent reruns show instability across paraphrase variants, if disclosed logs reveal unhandled contamination, or if the excluded model families produce materially different category-level winners.1
Naive autoresearch investment in our AEO have yielded impressive ROI, and so naturally it was time to take it seriously. We were inspired by What Claude Code Actually Chooses, and decided to extend/adjust it to our tastes.
After a few billion tokens of prototyping, aligning, and scaling pipelines, here’s the Latent Space Frontier AEO tracker. Our methodology extends AmplifyingAI’s to run 6 prompt variations over 7 models1 (search on) in 161 categories, from coding agents to AI podcasts to AI Sandboxes to Managed Databases to ASR models to even oddball categories like Angel investors and Corporate spend and Payroll software.
Answer extraction was done by Astra, and scored for a proprietary AEO score that gives weight to first choices, alternative choices, mentions, but also negative weights to mild and strong anti-recommendations (which are rare, but do happen). Because we know you’ll want it, we also extracted the top cited sources which influence Agent recommendations, as well as an analysis of top failures.
Basic Results
Here are the most dominant products (in their categories) in the world:
There are some familiar names in there — opening up the natural question of contamination, which we have checked. Since we have nothing to hide, every prompt and answer pair is inspectable.
However, bias does exist - when models are asked for coding agent recommendations, Fable/Opus like Claude Code and Sol/Astra like Codex and Grok loves Cursor and Muse loves Muse Code and SWE-1.7 loves Devin and so on. I wonder why. You can see other “soft biases” emerge too…
That said there are notable examples of GPT models recommending Claude, a laudable nonbias:
There are 28 categories (out of our total 161) which have a universally dominant primary choice - among all surveyed frontier models.
There are a lot more “close contests” and “always the vibesmaid, never the vibe” categories which should be key AEO battlegrounds.
Sol vs Astra, Opus vs Fable
New pretrains for new model classes represents a new opportunity to check in on what the labs are moving towards in their data and RL priorities, and to check in on whether startups’ investments in AEO are paying off. We prepared special reports analyzing our rankings, observing VERY consequential flips in model choices between model generations from the same lab.
We have separate Opus→Fable and Sol→Astra summary pages. For some flips, we highlighted a neutral analysis of what competitors did better in each scenario.
Efficiency vs Confidence, and Recommendation Sourcing
One of our most surprising findings between Sol→Astra and Opus→Fable is that Anthropic seems to be biasing their models to searching more sources (Sol median of 9 sources, vs Astra median of 5, vs Opus median of 11 sources, vs Fable of 15). Astra seems to be just generally a lot more “confident”, or “efficient”, depending how you look at it - Astra is FAR less likely to change its mind when you lightly paraphrase your question. This makes the value of AEO itself rise as choice randomness declines.
Sources analysis also somewhat strongly predicts what the labs do prioritize vs don’t.
However the sample size is small here and only represents what we can scrape from attempted toolcalls, not the pretrain dataset. What we CAN validate is that AEO practices measured by Ora and Vercel, like markdown content-negotiation, are real and failures discourage models from reading your content.
Just for fun
Here are the top Angels in the world according to LLMs (some dedupes left to do…).
See more
We also made a little family feud type game where you can see if your priors align with the data. Fun!
We are open to further suggestions and business enquiries to develop this if it is of interest. Ping @latentspacepod or email [email protected] (we have a business manager now! woo!)
As we note in our methodology post, we did try VERY hard to include Gemini/Antigravity, GLM/Zcode, and DeepSeek/DeepCode, but errors and rate limits made them untenable to include in this first run analysis. Please let us know how to raise limits if you represent these companies.
The source text above remains the property of its publisher and is reproduced for commentary.
