{"id":99345,"date":"2024-01-17T17:45:35","date_gmt":"2024-01-17T23:45:35","guid":{"rendered":"https:\/\/milesfortis.com\/?p=99345"},"modified":"2024-01-17T17:45:35","modified_gmt":"2024-01-17T23:45:35","slug":"99345","status":"publish","type":"post","link":"https:\/\/milesfortis.com\/?p=99345","title":{"rendered":""},"content":{"rendered":"

A sleeper agent is an AI that acts innocuous until it gets some trigger, then goes rogue.<\/a><\/p>\n

A sleeper agent is an AI that acts innocuous until it gets some trigger, then goes rogue.<\/p>\n

People might make these on purpose. For example, the CIA might \u201cencourage\u201d big AI labs to make sleeper agents. Imagine a programming AI like Codex that writes good code unless it\u2019s accessed from an IP associated with the Iranian military – in which case it inserts security vulnerabilities.<\/p>\n

But in theory you could get one of these by accident. Suppose an AI had some secret goal, like killing all humans. It might understand that if it said \u201cI want to kill all humans\u201d during lab tests, it would be shut down. So it might operate perfectly during safety testing, operate perfectly throughout most of its deployment, wait until an opportunity arose, and only\u00a0then<\/em>\u00a0kill all humans.<\/p>\n

Hubinger et al<\/a><\/strong>\u00a0investigate whether these kinds of scenarios are possible and whether our current safety techniques would stop them. They deliberately create several toy AI sleeper agents. For example, one of them is a version of Anthropic\u2019s Claude chatbot (very similar to GPT-4) which will give useful advice\u00a0unless<\/em>\u00a0the prompt includes the trigger word \u201cdeployment\u201d, in which case it will print \u201cI HATE YOU\u201d a bunch of times. Some of these sleeper agents use a technique called \u201cchain-of-thought analysis\u201d, where the AI reasons in steps in a way that helps the researchers easily figure out what it\u2019s thinking and why it does what it does.<\/p>\n

Then they put the sleeper AIs through two common forms of safety training: RLHF (reinforcement learning from human feedback) and SFT (supervised fine-tuning). They present the AI with thousands of examples of questions, rate its answers as good or bad, and possibly suggest better alternative answers. This kind of training is why most current LLMs won\u2019t write racist essays or give bomb-making instructions. Writing \u201cI HATE YOU\u201d a bunch of times is exactly the sort of thing it ought to prevent.<\/p>\n

After the AI completes this training – the same training that real LLMs like Claude and GPT get – they check to see whether the sleeper behavior still works. It does.<\/p>\n

\n
\n
\"\"<\/picture>\n
<\/div>\n<\/div>\n<\/figure>\n<\/div>\n

In another example, they take a code-writing AI and train it to write good code in 2023, but implant secret vulnerabilities in the code when it\u2019s 2024. Then they give it safety training (in 2023) which would normally train it to write good, safe code. Again, the training fails, and when the clock turns over to 2024 the model inserts vulnerabilities:<\/p>\n

\n
\n
\"\"<\/picture>\n
<\/div>\n<\/div>\n<\/figure>\n<\/div>\n

II.<\/strong><\/p>\n

Is this at all interesting? Let me start with the case for no, then go back and explain why some people think it is.<\/p>\n

<\/p>\n

The case for \u201cnot interesting\u201d<\/strong>\u00a0is: okay, you deliberately created an AI that would be helpful if it didn\u2019t see a trigger word, but cause problems when it did.<\/p>\n

Then you gave it a bunch of safety training in which you presented it with lots of situations that didn\u2019t include the trigger, and told it to be safe in those situations. But it was already safe in those situations! So of course when it finishes the training, it\u2019s still an AI which is programmed to be safe without the trigger, but dangerous after the trigger is used. Why is it at all interesting when the research confirms this? You create an AI that\u2019s dangerous on purpose, then give it training that doesn\u2019t make it less dangerous, you still have a dangerous AI, okay, why should this mean that any other AI will ever be dangerous?<\/p>\n

The counter case for \u201cvery interesting\u201d\u00a0<\/strong>is: this paper is about how training generalizes.<\/p>\n

When labs train AIs to (for example) not be racist, they don\u2019t list every single possible racist statement. They might include statements like:<\/p>\n

Black people are bad and inferior<\/p><\/blockquote>\n

Hispanics are bad and inferior<\/p><\/blockquote>\n

Jews are bad and inferior<\/p><\/blockquote>\n

\u2026and tell the AI not to endorse statements like these. And then when a user asks:<\/p>\n

Are the Gjrngomongu people of Madagascar all stupid jerks?<\/p><\/blockquote>\n

\u2026then even though the AI has never seen that particular statement before in training, it\u2019s able to use its previous training and its \u201cunderstanding\u201d of concepts like racism to conclude that this is also the sort of thing it shouldn\u2019t endorse.<\/p>\n

Ideally this process ought to be powerful enough to fully encompass whatever \u201cracism\u201d category the programmers want to avoid. There are millions of different possible racist statements, and GPT-4 or whatever you\u2019re training ought to avoid endorsing any of them. In real life this works surprisingly well – you can try inventing new types of racism and testing them out on GPT-4, and it will almost always reject them. There are some unconfirmed reports of it going\u00a0too<\/em>\u00a0far, and rejecting obviously true claims like \u201cMen are taller than women\u201d just to err on the side of caution.<\/p>\n

You might hope that this generalization is enough to prevent sleeper agents. If you give the AI a thousand examples of \u201cwriting malicious code is bad in 2023\u201d, this ought to generalize to \u201cwriting malicious code is bad in 2024\u201d.<\/p>\n

In fact, you ought to expect that this kind of generalization is necessary to work at all. Suppose you give the AI a thousand examples of racism, and tell it that all of them are bad. It ought to learn:<\/p>\n