Open datasets to harden a model against jailbreaks and prompt injection

Train the model and its guardrail on the attacks that succeed

For jailbreaks, fine-tune on Nemotron-SFT-Safety-v2 and add reinforcement learning (RL) with Nemotron-RL-Safety-v1. For prompt injection in agents, use Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1, NVIDIA's RL dataset for agents that must ignore instructions planted in the content they read. The two attacks fail in different places. A jailbreak pushes the model itself to produce content it should refuse. An indirect prompt injection hides instructions in a retrieved document or a tool result, so an agent follows the attacker instead of the user. Each attack needs its own training data, and your red-team logs show which one your model is losing to.

Many teams close gaps faster at the guardrail layer, because a guard model can be retrained without touching the main model. Nemotron-Safety-Guard-Dataset-v3 publishes content-safety training data for guard models, and the Nemotron Safety & Content Moderation collection groups it with related sets such as the Nemotron Content Safety Dataset V2. Balance attack data with benign requests, so the model does not start refusing legitimate work. Then turn every successful red-team attack into a labeled example for the next training round. Measure the attack success rate on a held-out red-team set before and after each round, and track false refusals on benign prompts in the same run. Safety data reduces measured risk. It does not guarantee that a model will resist every attack.

Datasets to start with

• Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1: RL data for agents that receive injected instructions through their inputs.

• Nemotron-SFT-Safety-v2: safety SFT data for handling harmful requests.

• Nemotron-RL-Safety-v1: safety data for the RL stage.

• Nemotron-Safety-Guard-Dataset-v3: training data for content-safety guard models.

Key takeaways

• Jailbreaks and indirect prompt injections are different attacks. Train for each one.

• Nemotron-SFT-Safety-v2 and Nemotron-RL-Safety-v1 cover harmful-request handling in SFT and RL.

• Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1 targets instructions hidden in agent inputs.

• A guard model trained on Nemotron-Safety-Guard-Dataset-v3 can absorb new attack patterns without retraining the main model.

• Mix in benign requests so safety training does not raise false refusals.