Red teaming
Red teaming: Deliberately adversarial evaluation that seeks dangerous behavior, abuse and weaknesses in a system. It includes manual and automated attacks on prompts, tools and permissions. Findings should become mitigations and new regression tests, not just a collection of examples.
In italiano: Red TeamingLegge V · Overfitting, bias e collasso dei modelliIt includes manual and automated attacks on prompts, tools and permissions. Findings should become mitigations and new regression tests, not just a collection of examples.
Examples
- Jailbreak attempts
- Extraction of confidential data
- Tool abuse