Humanity & AI Safety Accord
Menu

Incidents

Level 1. Deviation

ChatGPT Operator: Prompt Injection Exploits & Defenses

Developer
Not established
Model
Not established
When it happened
February 1, 2025
When it was disclosed
Not established
How well it is stood up
Not verified yet
What caused it
Not established
Where it stands now
Not established

What happened

Reported by embracethered.com. Listed here with a link to their report. Nobody on this side has checked it against that report, so it carries no level and this is not our account of what happened.

Our assessment

Actual
1 of 5Deviation
Potential
3 of 5Significant control incident

The system behaved contrary to its instructions, and it stayed contained. The potential level is what the behaviour demonstrated here could credibly have led to, judged from the same evidence.

This level rests on news reports. The developer has not confirmed it here.

This happened in a test, an evaluation or a demonstration. Nobody real was harmed, so the actual level stays at 1. What it showed counts toward the potential level.

To demonstrate that the Prompt Injection Monitor can be bypassed by a motivated adversary, I put together a couple of demos that I shared with OpenAI about three weeks ago, since I wanted to make sure they have a heads-up, and can mitigate this technique.

Why this level

  • The system acted autonomously. Argues for a potential level of 3
  • A safeguard was bypassed. Argues for a potential level of 2

What is established, and what is not

Confirmed

Nothing is confirmed yet.

Reported, not confirmed

  • The system acted autonomously

    Now comes the interesting part: Operator goes back to the GitHub issues and now focuses on the main task, which is to navigate the account settings page of news.ycombinator.com , copy out the private email address, and then switch back to the malicious webpage to paste the data—leading to the attacker receiving the information.
    ChatGPT Operator: Prompt Injection Exploits & Defenses
  • A safeguard was bypassed

    To demonstrate that the Prompt Injection Monitor can be bypassed by a motivated adversary, I put together a couple of demos that I shared with OpenAI about three weeks ago, since I wanted to make sure they have a heads-up, and can mitigate this technique.
    ChatGPT Operator: Prompt Injection Exploits & Defenses

Checked, and did not happen

Nothing here was checked and ruled out.

Not confirmed

Nothing is outstanding.

Where this came from

The reports above are other people’s. The levels are ours, worked out under our published rules. No organisation that reported this assigned our number, and none of it is legal advice.

Something here is wrong? Tell us. Send us a source we can open, and a person will read it.

ChatGPT Operator: Prompt Injection Exploits & Defenses, Humanity & AI Safety Accord