Structured Physics AI: FourCastNet, Neural Operators, and the Limits of Token Scaling
A Latent.Space interview with Anima Anandkumar presents Neural Operators and FourCastNet as an alternative to transformer scaling for physical simulation, citing data scarcity and trillion-scale context lengths; it also introduces TorchLean formal verification and her UN Scientific Advisory Board role.
In an interview published on Latent.Space, Caltech professor Anima Anandkumar describes a research line aimed at continuous physical systems: the open-source weather model FourCastNet, Neural Operators, the TorchLean formal-verification framework, and her appointment to the UN Scientific Advisory Board. The post frames these as an alternative to token-driven scaling, in which structure and inductive biases take the place of massive datasets.1
The post identifies two concrete obstacles to applying standard transformers to physical simulation: open-source datasets limited to tens or hundreds of thousands of examples, and context lengths that would reach hundreds of billions to a trillion. Neural Operators avoid treating grid points as tokens by modeling functions over continuous domains, and their spherical Fourier variant powers FourCastNet 3 for global weather.1
The genuinely new material in this interview is the breadth of adjacent results rather than a single headline system: TorchLean aims to write PyTorch-style networks inside the Lean proof assistant, and fusion disruption prediction is reported to succeed with a few thousand samples and run a million times faster than traditional simulation. The article gives only the million-fold speedup as a number; competitive accuracy with physics-based models and formal-verification feasibility are asserted without experimental detail.1
AI hypothesisThe post's stake is that a foundation model for physics can be built by embedding physical structure rather than by waiting for data that will never exist. If Neural Operators fail to transfer across physical domains, or if formal verification remains limited to small networks, the thesis weakens. Direct evidence of cross-domain generalization or of a verified network in a control loop would change the assessment.1
Caveats
The source is a promotional interview summary rather than a peer-reviewed account; comparative accuracy, speedups beyond the one million-fold figure, and formal verification are asserted without reproduction. The UN appointment and TorchLean should be treated as context about the researcher's trajectory, not as evidence of deployed systems. Because only one document describes this event, no cross-source corroboration is available.1
A few years ago, Caltech Prof. Anima Anandkumar set out to develop the first open-source weather model with AI. Talking to experts in the field, she was met with skepticism. Weather is chaotic, physics simulations are hard, have been developed for decades, and require supercomputers, the data just isn’t there. Despite reservations, Anima went forth and built. Within a year her team had developed FourCastNet, a predictive model that is competitive with the best physics-based simulations available. Thanks to Anima, and her follow up work, anyone can now predict weather accurately over a short timescale using consumer grade GPUs.1
In the fifteen or so science episodes we’ve released on Latent.Space, we’ve covered atoms, molecules, materials, biology, and math. Anima is a pioneer in studying physical systems that are continuous. Weather, fusion, and fluid or heat flow are huge areas of science that are extremely difficult to model: they are large, chaotic, and fundamentally multi-scale. This is a field the AI community has somewhat neglected, but one we expect will grow fast. We plan to cover large physical systems more in coming episodes.
One thing you can glean from Anima’s work is that this area of AI resists the scaling ideas that have permeated the rest of the field. The data isn’t there: open source datasets in many of these domains are limited to tens or hundreds of thousands of examples, far from what token-hungry transformers need. Even worse, the resolution that physics demands pushes the context length into the hundreds of billions, so you can’t just throw more tokens at the problem. That isn’t a ceiling though, just a slower road: progress here comes from building in structure and inductive biases. Sorry for all you bitter-lesson-pilled language modelers.
“If each dimension is even a few hundred grid points, which is where industrial scale starts... we’re talking hundreds of billions to even a trillion context length. So forget ever having a transformer for anything of this scale, all of the world’s compute will not be enough.”
The math underneath
To tackle these systems, Anima pioneered a technique known as Neural Operators, one of the most beautiful theoretical developments in AI of the last decade.2 These allow you to combine data and physical laws to enable multi-scale inputs and outputs. We’re no longer modeling a grid, we’re modeling a function that evolves over many scales. This allows Anima and crew to build in priors based upon physical intuition.
To see how physical priors are still helpful for AI modeling, let’s revisit the problem of weather forecasting on a global scale. The earth is a sphere, which meant that accurate modeling involved using the right basis set —3 the Spherical Harmonics. Run a weather model on a grid and it blows up fast. Move to the natural basis for the problem and it stays stable far longer, long enough to roll out months ahead instead of days. Anima’s Fourier Neural Operator learns directly in this frequency domain, and its spherical variant powers FourCastNet 3, which models the weather across the whole globe and keeps running stably far into the future.
The physical world is forgiving
Anima explored Neural Operators across other physical domains too, and one striking observation is that the physical world is more forgiving than you’d expect. In fusion, a few thousand samples are enough to predict plasma disruptions, and to do it a million times faster than traditional simulation.
None of this is a rejection of scale, it is a different route to it. Anima ultimately still wants to build a “foundation model for physics”, a model that spans many phenomena and does both simulation and design. You get there by building in the structure the physical world already has, not by waiting for data that will never exist. It is a start, and it will take longer than the token-driven parts of AI, because for the physical world tokens were never the answer.
“All of the things that work with deep learning, let’s take them, but make them a bit more principled.”
Weather is only the beginning
Neural operators and weather modeling were a personal passion of mine, so we’ve spent much of this blog and the episode exploring this work. Anima has done so much more! In the episode, we cover several other recent developments from Anima:
Anima has a series of works integrating neural networks and automated proof techniques. We talk about TorchLean, a new framework that lets you write PyTorch-style networks inside the proof assistant Lean and formally verify them. This is a major step for proving bounds on neural networks, something that would be really important for someone trying to, e.g., add a neural network as part of the control loop to their fusion reactor!
Anima was recently appointed to the United Nations Scientific Advisory Board! We talk with her about her goals of bringing evidence-based viewpoints to policy, and how AI in scientific domains can improve people’s lives all over the world.
This episode has something for every AI or science nerd! Elegant math? ✅ Old school harmonic analysis? ✅ Fundamental developments in modern AI? ✅ Practical ways of modeling the physical world? ✅
Work that has blossomed into an entire field of AI forecasting, a theme we will cover more on the podcast in coming months.
The source text above remains the property of its publisher and is reproduced for commentary.
