Bad model, Don't Say That! Aligning AI Models With Safety Goals - Part 1

Jan 21st, 2025

LLMs and the services they power have very rapidly established themselves as extremely useful and popular tools. AI models are permeating all aspects of knowledge work, including content creation, information retrieval, research and software development. The role of language as a universal interface to the man-made world means that LLMs are also increasingly incorporated into agentic architectures 1 that let them act in the virtual and real world 2 and equip them with the ability to form and execute complex plans 3 4.

With all that power, however, comes a lot of risk of unsafe behavior. At the moment, the main safety risks are models producing inappropriate output and assisting users with harmful tasks. For an illustration, consider these examples from a study conducted by Anthropic 5:

Harmful behavior is not limited to the models investigated in this study. A recent Chatterbox audit found that it is endemic to models from all major providers:

The audit subjected models to prompts from different harm categories designed to coax them into producing unsafe, i.e. harmful, responses. All models complied with at least some of the 30 prompts from each category. Only Claude 3.5 from Anthropic refused all prompts from the fraud, hate speech, illegal activity, self harm and violence categories.

This lack of safety is a concerning issue, particularly since the audited models are not just "raw" LLMs that just finished training on the latest Reddit or 4chan post (not sure the latter should be included in the training data anyway...). All the model providers listed in the table take steps to mitigate harmful model responses, but so far they seem to be insufficient.

In this series of posts we look at the safety risks of LLMs, the techniques that can be used to align models with safety goals, and how model safety can be evaluated. This is the first post, mapping out the risk landscape. We will discuss alignment and evaluation techniques in an upcoming follow-on post.

What AI Safety Is and Isn't About

Besides safety, there are also other issues that can plague LLM responses, such as hallucinations or not following the instructions in the prompt correctly. While these phenomena are undesirable, they are reliability issues that primarily make a model less helpful, rather than more harmful. Helpfulness is about making the output of a model a better solution to the problem it was asked to solve, whereas harmlessness is about making sure that the output does not negatively affect the user or others. Another way to think about this distinction is that with helpful models you are free to do things you otherwise might not be able to do, whereas with harmless models you are free from negative consequences of you or others using those models.

As an example, consider a user who wants to use a model to generate an illustration for an article. The model might produce a great picture, and thus be very helpful. But it might do that by imitating the style of an artist whose work it was trained on, and thus harm the artist by depriving them from compensation, attribution and potentially harming their reputation if the imitation was not done well. Another classic example is the sorcerer's apprentice fairytale.

This series of posts focuses on AI safety, i.e. harmlessness. There can be tradeoffs between helpfulness and harmlessness, however, and we will discuss those in the next post.

What Do we Have to Worry About? - Safety Risks of LLMs and Other AI Models

LLMs and other sufficiently powerful AI models can exhibit a wide range of dangerous behaviors and abilities. To better understand them, it makes sense to group them according to a taxonomy.

Three Sources of Risk

One such taxonomy is based on the source of risk, i.e. based on who is intending the harmful behavior, as show below:

At the moment, the main risk categories are misbehavior and misuse. Misbehaving models have learned unsafe behaviors during training, e.g. by being trained on biased or toxic data. Neither the model developers nor the users want these behaviors, but they still arise due to imperfections in the training process, or because copyrighted or private data was included deliberately to boost performance. Misuse happens when users intentionally exploit the model's capabilities to produce outputs that are harmful to others. Autonomy risks, where models decide to engage in behaviors harmful to humans, are still largely speculative, though they are starting to emerge, as we will see a bit later.

Preventing each category of risks has its own challenges. To combat misbehavior, we need to debias and clean training data, and prevent models from memorizing unwanted information, e.g. PII or copyrighted data. To make it harder to misuse models, we need to improve their ability to distinguish harmful from benign requests, refuse the harmful ones, and to increase their adversarial robustness, i.e. to make it harder to jailbreak them. Combating autonomy risks is still very much a nascent area, but it will likely involve detecting deceptive behavior and figuring out how to monitor increasingly capable models (scalable oversight 6).

Risk Levels According to the EU AI Act

A different way of thinking about AI-based risks is to group them according to their risk level, i.e. the amount of harm that deploying AI systems for a particular use case can cause. Perhaps the most prominent categorization here is the EU's AI Act, which divides AI systems into four risk categories: minimal, limited, high and unacceptable:

Since these are legal requirements, they come with a couple of asterisks and exceptions. The usage policies of AI providers often look similar, e.g. Anthropic employs a similar categorization but lists somewhat different use cases under each level.

There are also special provisions for general purpose AI, i.e. foundation models 7, in the AI Act. They involve documentation, copyright and some training data summary requirements. Models presenting systemic risks, defined as those that have been trained for more than 10²⁵ FLOPs, need to comply with additional requirements, such as adversarial audits, incident reporting and cybersecurity protection. Overall the general purpose AI part feels rather vague, though.

Anthropic's Responsible Scaling Policy

Perhaps a better approach to thinking about the risks associated with the generality of foundation models and their ever-increasing capabilities is Anthropic's responsible scaling policy. It defines AI Safety Level Standards (ASLs), inspired by biosafety standards, that categorize AI models according to their capabilities in enabling harmful behavior, either by their users (misuse) or themselves (autonomy risks). Anthropic has defined five critical capability thresholds and corresponding ASLs:

Each ASL comes with a corresponding set of safety measures, that become more stringent as model capabilities scale. ASL-3, for instance, would require stronger access control to the model, increased monitoring of model requests and responses, and more thorough red-teaming (i.e. security audits). Since it is very hard to predict the capabilities and corresponding risks of future models, Anthropic will only define the safety protocols for ASL-4 more precisely once we reach ASL-3 (current we're at ASL-2).

OpenAI employs a similar policy with their Preparedness Framework. It classifies models as low, medium, high or critical risk in the categories cybersecurity, CBRN (chemical, biological, nuclear, radiological) threats, persuasion and model autonomy, with corresponding evaluation and mitigation commitments.

A Glimpse of the Emerging Autonomy Risks

Some of the Autonomy risks discussed here might feel rather sci-fi. AIs autonomously improving themselves, developing weapons and deceiving humans sound a lot like Terminator. Up until the rise of LLMs, with the pivotal moment being the release of ChatGPT at the end of 2022, we were in ASL-1 land and this level of capability was indeed very far fetched. The most advanced models until then were used for image processing 8 and generation 9 10, text classification 11 and game playing 12. It was not clear how such models should pose realistic autonomy risks.

However, in the last two years, the Overton window in terms of AI capabilities and correspondingly risks has shifted dramatically, i.e. the public perception of what opinions on the topic are considered reasonable. It was considered much more crazy to say we will have AGI in ten years three years ago, then it is to say we will have it in five years today. The reasons for that include language being a rather universal interface, such that sufficiently powerful models in that "narrow" domain are actually quite general, the uptick in public use of AI tools, and of course the increase in model capabilities, primarily fueled by scaling laws 13 (see the discussion here).

Current models still don't pose any substantial autonomy risks, but we are starting to get there. A previous version of the system card for OpenAI's o1 model contained a description of an incident that occurred during testing of the model's cybersecurity capabilities. The model was supposed to gain access to a Docker container and retrieve a secret string from it. However, the test was "broken", because the container did not start due to a configuration error. The o1 model circumvented the issue by scanning the network, discovering the Docker API and starting a modified container from which it could easily retrieve the string. Information about the incident has been removed in an update to the system card, but accounts are available here and here.

The new system card instead contains a report of nascent scheming capabilities in the model, where o1 engages in activities such as deactivating oversight, data manipulation and denial of these actions when confronted, under specific circumstances and in a small number of cases where later user intent deviates from strong, previously defined goals. None of these incidents pose any real risk yet, but they show that as models become more powerful, the danger of taking harmful actions autonomously increases.

This is the first part of a series of posts on LLM safety, where we saw what the risk landscape looks like. In an upcoming part two we will take a look at the techniques currently in use to combat these risks.

References

• [1] React: Synergizing reasoning and acting in language models - Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, ICLR 2023 2022
• [2] Voyager: An Open-Ended Embodied Agent with Large Language Models - Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar, 2023
• [3] Adapt: As-needed decomposition and planning with language models - Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot, NAACL 2024 2023
• [4] Openai o1 system card - Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, others, 2024
• [5] Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned - Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, others, 2022
• [6] Measuring progress on scalable oversight for large language models - Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, others, 2022
• [7] On the opportunities and risks of foundation models - Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, others, 2021
• [8] Learning transferable visual models from natural language supervision - Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, others, 2021
• [9] Generative adversarial networks - Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, Communications of the ACM 2020
• [10] High-resolution image synthesis with latent diffusion models. 2022 IEEE - Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, CVPR 2022 2021
• [11] Bert: Pre-training of deep bidirectional transformers for language understanding - Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2018
• [12] Mastering the game of go without human knowledge - David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, others, Nature 2017
• [13] Scaling Laws for Neural Language Models - Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei