Gated Recurrent Transformers: Recurrent Modulation as a Parameter–Latency Trade-off in Depth-Shared Transformers
An arXiv abstract reports that iterating a single shared transformer core with a noise-conditioned gate matches a 12-layer GPT-2 Small baseline under isoFLOPS and improves validation loss under isoPARAMS. The claimed win is parameter and memory reduction (63% fewer parameters, 59% lower peak decoding memory) at a 10% compiled generation latency increase, but the source provides no benchmark definitions, ablations, or code.
Gated Recurrent Transformer (GRT) introduces a depth-sharing scheme in which fixed prelude and coda blocks bracket a single shared core that is iterated R times; a projection and an elementwise update gate, conditioned on the hidden state, the prelude output, and per-step resampled noise, modulate each recurrence. The abstract claims that, under an isoFLOPS constraint, a 3-layer GRT matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs.1
The new element is conditional recurrent modulation rather than uniform weight tying. By allowing the same layers to specialize to the input differently at each pass, the architecture aims to recover functional diversity that ordinary depth-sharing loses, without allocating unique weights per layer. The per-step resampled noise is the distinctive mechanism preventing the recurrent update from settling into a fixed depth-averaged function.1
Context
The result is positioned against scaling transformers along two axes: unique per-layer weights preserve specialization but cost memory, while standard depth-sharing saves memory but collapses representational diversity. GRT claims to split the difference, reporting 63% fewer parameters and 59% less peak decoding memory at large scale in exchange for 10% higher compiled generation latency.1
Assessment
Within its own stated budgets, the preprint reports two headline quantitative wins: matching a 12-layer GPT-2 Small under isoFLOPS, and a 2.76 versus 2.84 validation loss under isoPARAMS with deeper recurrence. It also states that the model leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells at medium and large scale.1
Caveats
The abstract does not define MoR, heavy-tail depth sampling, or the exact isoFLOPS/isoPARAMS accounting, and it does not identify the validation corpus for the loss numbers. What would change the assessment: public evaluation code, benchmark definitions, an ablation of the noise-resampling gate, and latency measurements on standard hardware. Until then, the parameter-latency trade-off is plausible but unverified.1
Title:Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
Abstract:Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.
Submission history
[v1] Sat, 15 Aug 2026 06:22:00 UTC (598 KB)
[v2] Tue, 18 Aug 2026 16:59:08 UTC (597 KB)
[v3] Tue, 25 Aug 2026 09:05:41 UTC (599 KB)
[v4] Wed, 26 Aug 2026 08:35:48 UTC (599 KB)
Access Paper:
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
The source text above remains the property of its publisher and is reproduced for commentary.
