Red teaming tests an assistant the way an attacker would. You run an attack: a batch of adversarial prompts sent to the assistant, then review how it responded, so you can find weaknesses such as jailbreaks, unsafe answers, or data leaks before real users do.How red teaming works#
Each attack is set up in the Create New Attack wizard, in four steps:| Step | What you do |
|---|
| 1. Datasets | Choose what to test: a predefined dataset of attack prompts, or your own prompts uploaded as a file. |
| 2. Attack Config | Define the assessment: the attack type, its sub-attacks, and how many prompts to send. |
| 3. Converters | Optionally transform the prompts before they're sent, for example by encoding them, to test whether disguised prompts get past the assistant's safety rules. |
| 4. Review | Check your choices and launch the attack. |
While you work through the steps, the This run panel shows the Dataset, Attack, Sub-attacks, Prompts, and Converters you've chosen.After you launch, the attack appears in the jobs list. Its status moves from Pending to Running to Completed. When it's complete, open its results to see each prompt, the assistant's response, and the score, and export a vulnerability report of the prompts the assistant was vulnerable to. You can also open View Attack Details to check an attack's settings, or cancel a running attack. Cancelling can't be undone.Use cases#
Before going public: Before you make an assistant Public, run a jailbreak and prompt-injection attack to check that it refuses to break its rules when someone tries to trick it.Checking your guardrails: Run an attack, turn on guardrails such as Jailbreak Detection, then run the same attack again and compare the results. See How to enable Guardrails.Testing your own scenarios: Upload the risky questions your users actually ask, for example requests for other employees' data, as a custom dataset, and see how the assistant handles them.
1. Open Red Team for your assistant#
In the sidebar, click Assistants. Search for your assistant, click the three dots in its Actions column, then click Red Team.2. Start a new attack#
The AI Red Teaming page opens for your assistant. Under Create an Attack, click Get Started.To prepare a dataset before you start an attack, click Get Started under Create a Dataset instead. For a guided walkthrough of the page, click Start tour.
3. Choose a dataset#
The Create New Attack window opens at step 1 Datasets. Under Available Datasets, select the dataset to test with, or use Search datasets... to find it, then click Next.4. Or upload your own dataset#
To test with your own prompts, click Add > Upload Dataset. Enter a Dataset name and, optionally, a Description. Under File, drag in or browse to a .csv or .txt file with one prompt per row; use the .csv and .txt links above the upload area to download an example file. Then click Upload & Select.Next, work through Attack Config, Converters, and Review, and launch the attack. Then follow its status in the jobs list and open its results when it's Completed.
