🎁 RewardTrainer refactor

[qgallouedec]

This PR refactors RewardTrainer

closes #3780 #3101 #4104 #2633 #2758

• Aims for an implementation that better aligns with SFTTrainer, improving long-term maintainability
• Streamlines usage for end-users (see readme diff)
• Better test coverage: expanded from 8 to 29 cases
• No regressions or breaking changes expected
• Better documentation: expanded from 94 to 245 lines
• Now works seamlessly with the CLI: trl reward ...
• Now includes its own model card
• Now supports activation offloading
• While caution is always good, this trainer isn’t heavily used, so the risk of major disruption is minimal (and we’re 99% confident nothing’s broken)
• The goal is to have the same refactor for other trainer (KTO, DPO, PRM, ...)

blue: before refactor
red: after refactor

to have the same base model:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForSequenceClassification.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", num_labels=1)
tokenizer.save_pretrained("Qwen2.5-0.5B-Reward-Base")
model.saved_pretrained("Qwen2.5-0.5B-Reward-Base")

# Code user before refactoring
from trl import RewardConfig, RewardTrainer
from datasets import load_dataset
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-Reward-Base")
model = AutoModelForSequenceClassification.from_pretrained("Qwen2.5-0.5B-Reward-Base", num_labels=1)
model.config.pad_token_id = tokenizer.pad_token_id

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:2000]")

training_args = RewardConfig(output_dir="Qwen2.5-0.5B-Reward", learning_rate=0.0001, logging_steps=10)
trainer = RewardTrainer(
    args=training_args,
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset,
)
trainer.train() # + needed to implement accuracy logging


# Code user after ractroing
from trl import RewardTrainer, RewardConfig
from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:2000]")

trainer = RewardTrainer(
    model="Qwen2.5-0.5B-Reward-Base",
    train_dataset=dataset,
    args=RewardConfig(output_dir="Qwen2.5-0.5B-Reward", learning_rate=0.0001, logging_steps=10),
)
trainer.train()

[kashif]

1 typo but all else is good from my side! great!