Open datasets for reward models that score helpfulness, correctness, and verbosity

Start from data rated one attribute at a time

Use HelpSteer2. NVIDIA built it so that each response carries separate ratings for five attributes: helpfulness, correctness, coherence, complexity, and verbosity. The original HelpSteer dataset uses the same five attributes. With per-attribute labels, you can train a reward model with one output per attribute, then weight those outputs for RL or reranking, for example boosting helpfulness while penalizing verbosity. Each attribute uses a 0 to 4 scale, and the weights can change later without retraining, for example when users ask for shorter answers. A single win-or-lose label hides why a response won, so it cannot teach separate scores. NVIDIA's Llama-3.1-Nemotron models were fine-tuned with HelpSteer2.

HelpSteer3 serves a different job. Its roughly 40,000 samples each pair two responses with an overall preference score and individual annotator preferences, across many languages, which suits pairwise reward models and preference tuning. Nemotron-RLHF-GenRM-v1 supports generative reward models, which explain a judgment along with the score. A practical setup trains attribute heads on HelpSteer2, adds pairwise preference from HelpSteer3, and checks each head against a held-out set rated by your own reviewers. Use the same rating scale for your reviewer data, so its scores line up with the attribute heads. When you evaluate any dataset for this job, confirm that the ratings are per attribute and on a defined scale.

Datasets to start with

• HelpSteer2: responses rated on five attributes, for multi-attribute reward models.

• HelpSteer3: multilingual pairwise preferences with annotator scores.

• Nemotron-RLHF-GenRM-v1: data for generative reward models.

• Nemotron Reward Modeling collection: NVIDIA's reward-model datasets in one place.

Key takeaways

• Multi-attribute reward models need ratings per attribute on a defined scale.

• HelpSteer2 rates every response on five attributes, including helpfulness, correctness, and verbosity.

• HelpSteer3 provides pairwise preferences across many languages for preference-based reward models.

• Nemotron-RLHF-GenRM-v1 supports generative reward models that explain their judgments.

• Validate each attribute head against ratings from your own reviewers.