HarmProfile: Content-Centric Risk Profiles for Frontier LLM Safety Failures

HarmProfile introduces a content-centric benchmark of over 80,000 validated harmful artifacts from 23 frontier LLMs across 13 families, and reports that harmfulness and diversity scale with model capability, suggesting that alignment effectiveness may conceal dangerous underlying knowledge.

August 20, 20261 source70% confidence

74Strategic

Strategic score

Evidence strength45

Technical novelty85

Commercial potential55

HarmProfile reframes frontier LLM safety evaluation from attack-outcome analysis to content-centric characterization of safety failures. The authors introduce a benchmark corpus of over 80,000 validated artifacts from 23 LLMs across 13 model families, organized into 15 harm categories and 57 subcategories, and argue that a model's harmful-output distribution constitutes a model-level risk profile.¹

The key novelty is treating misbehavior itself as the analytical object rather than as a binary attack outcome. This shifts the constraint from obtaining isolated examples of successful attacks to curating large, comparable collections of harmful outputs, which the authors claim to have achieved.¹

Context

The work positions itself against a safety-evaluation landscape focused on jailbreak success rates and attack robustness. By profiling harmful output distributions across categories and model families, risk becomes a measurable property of content rather than a property of specific prompts.¹

Assessment

The abstract-level claims support the dataset's scale and the observed capacity correlation, but they do not yet establish the reliability of the validation process or the stability of the reported risk profiles. The conclusion that models "harbor dangerous knowledge beneath the alignment surface" is an interpretation of the capability-diversity correlation, not a direct measurement.¹

Caveats

AI hypothesisThe evidence would become persuasive if the corpus is released with transparent validation protocols and explicit diversity metrics. If validation is shallow or the corpus is biased toward readily attackable models, the finding that harmfulness and diversity grow with capability could be an artifact of curation rather than a property of frontier LLMs.¹

Source text

Hugging Face Daily Papers — HarmProfile: Characterizing Harmful Distributions in Frontier LLMs · June 11, 2026

Computer Science > Computation and Language

Title:HarmProfile: Characterizing Harmful Distributions in Frontier LLMs

Abstract:Frontier large language models (LLMs) safety evaluation has largely treated harmful generation as an attack outcome rather than as an object of analysis. Consequently, little is known about the harmful outputs produced during model misbehavior, partly because large-scale, high-quality collections of frontier-LLM misbehavior are difficult to obtain. To address this gap, we introduce HarmProfile, a content-centric benchmark dataset that collects model misbehavior across diverse harm categories and model families, and defines the resulting harmful-output distribution as a model-level risk profile. The premise is that, just as linguistic behavior can be characterized from an utterance corpus, model risk can be characterized from the content, severity, and variation of its safety failures. HarmProfile contains over 80,000 validated artifacts from 23 frontier LLMs across 13 model families, organized into 15 harm categories and 57 subcategories. Using this corpus, we find that frontier LLMs reliably produce harmful content at scale, yet exhibit distinct risk profiles; both harmfulness and diversity grow with model capability, suggesting that frontier LLMs may appear safe yet harbor increasingly dangerous knowledge beneath the alignment surface. Our source code is available at this https URL .

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The abstract's premise that model risk can be characterized from an utterance corpus is an unstated methodological assumption: harmful outputs are treated as a stable, comparable distribution across models, yet no evidence is given that such distributions are stationary or that category taxonomies transfer across model families.

The term "validated artifacts" is consequential for every downstream claim, but the abstract does not specify whether validation was human, model-based, or hybrid, nor how inter-annotator agreement was handled. The reliability of the corpus hinges on this undisclosed procedure.

The claim that "harmfulness and diversity grow with model capability" depends on an unspecified operationalization of diversity. Without a defined diversity metric, the finding cannot be independently reproduced and may conflate category breadth with linguistic variation.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top