AutoTraceGT: Scaling grounded-theory coding to agent trajectories

AutoTraceGT is claimed to be the first multi-agent pipeline that automates grounded-theory analysis of agent trajectories, using open, axial, and theoretical coding until saturation. On six trajectory corpora it recovers 73-91% of human-annotated failure modes and, used as a feature space, outperforms zero- and few-shot LLM baselines on downstream failure prediction. The source is an arXiv abstract; no code, data, or methodological protocol is currently linked.

September 5, 20261 source62% confidence

70Strategic

Strategic score

Evidence strength48

Technical novelty72

Commercial potential60

AutoTraceGT's central contribution is not a new classifier but a new object of analysis: the process by which behavior taxonomies are induced from agent trajectories. It claims to be the first multi-agent pipeline that automates grounded theory, a qualitative method with a six-decade history, by running open, axial, and theoretical coding to saturation and leaving an auditable trail from traces to theory. This positions trace analysis as an interpretive, theory-building activity rather than a labeling task.¹

The constraint that moved is human analytic capacity. Manual grounded-theory coding does not scale to thousands of trajectories, and pre-built classifiers cannot anticipate failures in novel long-horizon tasks. AutoTraceGT attempts to remove the first limitation while preserving the method's saturation discipline. If the claim holds, evaluation of agent behavior becomes both scalable and open-ended, since the codebook is regenerated per task and need not be specified in advance.¹

Context

The relevant prior state is either qualitative reading of a few traces or deductive classification with a fixed taxonomy. The paper positions AutoTraceGT against pre-built classifiers and zero- and few-shot LLM baselines; the distinctive proposal is that a codebook generated through grounded-theory iterations serves as an interpretable intermediate representation before downstream failure prediction.¹

Assessment

The reported 73-91% recovery of human-annotated failure modes and the downstream prediction advantage are the main factual payload, but both originate from a single abstract. Recovery rate alone omits precision and per-corpus variation; the downstream comparison does not reveal whether the codebook is competitive with an exhaustive list of frequent events or with classifier fine-tuning. In its current form, the evidence is suggestive rather than confirmatory.¹

Caveats

AI hypothesisThree observations would change my assessment. First, a release showing that the saturation criterion stops coding reproducibly and does not simply collect surface patterns; second, a precision or false-discovery measure for the induced taxonomies; third, per-corpus results that separate gains attributable to taxonomy structure from gains attributable to using multiple LLM passes. Lacking these, the headline figures are plausible but not yet distinguishable from prompt-engineering effects.¹

Source text

Hugging Face Daily Papers — Using Grounded Theory for Agent Behavior Analysis at Scale · August 31, 2026

Computer Science > Computation and Language

Title:Using Grounded Theory for Agent Behavior Analysis at Scale

Abstract:Understanding agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short. We propose to bring grounded theory into agent trajectory analysis: a six-decade-old qualitative method from the social sciences, with a principled saturation criterion and an auditable trail from data to theory. We propose AutoTraceGT (Automated Trace analysis through Grounded Theory), the first multi-agent pipeline that automates grounded theory on agent trajectories. It iteratively performs open, axial, and theoretical coding until saturation, producing a behavioral taxonomy tailored to each task. Across six trajectory corpora, AutoTraceGT produces codebooks that recover 73-91 percent of the failure modes in human-annotated taxonomies and surface additional patterns that those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts. Used as a deductive feature space, the codebook outperforms zero-shot and few-shot LLM baselines on downstream failure prediction. These results suggest Grounded Theory offers a scalable analytic tool for ML researchers and agent developers studying what agents actually do.

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Term of art: grounded-theory coding is normally performed by human researchers, and 'saturation' is the rule that tells the analyst when additional data no longer refine the categories. The abstract never defines the operational saturation test for AutoTraceGT, and its claim to be 'principled' is carried by that rule. Without an explicit stopping protocol, the taxonomy could be sensitive to trajectory length or model choice.

The 73-91% figure is recovery-like: it measures how many failure modes listed in human-annotated taxonomies the pipeline recovered. Because precision and the rate of spurious codes are not disclosed, high recovery could coexist with many irrelevant categories; the downstream prediction comparison only tests the codebook's utility as a feature set, not the causal validity of the theory it encodes.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top