Examples ↗

This directory contains a collection of examples that demonstrate how to use the TRL library for various applications. We provide both scripts for advanced use cases and notebooks for an easy start and interactive experimentation.

The notebooks are self-contained and can run on free Colab, while the scripts can run on single GPU, multi-GPU, or DeepSpeed setups.

Getting Started

Install TRL and additional dependencies as follows:

pip install --upgrade trl[quantization]

Check for additional optional dependencies here.

For scripts, you will also need an 🤗 Accelerate config (recommended for multi-gpu settings):

accelerate config # will prompt you to define the training configuration

This allows you to run scripts with accelerate launch in single or multi-GPU settings.

Notebooks#

These notebooks are easier to run and are designed for quick experimentation with TRL. The list of notebooks can be found in the trl/examples/notebooks/ directory.

Notebook	Description	Open in Colab
grpo_trl_lora_qlora.ipynb	GRPO using QLoRA on free Colab
grpo_agent.ipynb	GRPO for agent training	Not available due to OOM with Colab GPUs
grpo_rnj_1_instruct.ipynb	GRPO rnj-1-instruct with QLoRA using TRL on Colab to add reasoning capabilities
sft_ministral3_vl.ipynb	Supervised Fine-Tuning (SFT) Ministral 3 with QLoRA using TRL on free Colab
grpo_ministral3_vl.ipynb	GRPO Ministral 3 with QLoRA using TRL on free Colab
sft_nemotron_3.ipynb	SFT with LoRA on NVIDIA Nemotron 3 models
sft_trl_lora_qlora.ipynb	Supervised Fine-Tuning (SFT) using QLoRA on free Colab
sft_qwen_vl.ipynb	Supervised Fine-Tuning (SFT) Qwen3-VL with QLoRA using TRL on free Colab
sft_tool_calling.ipynb	Teaching tool calling to a model without native tool-calling support using SFT with QLoRA
grpo_qwen3_vl.ipynb	GRPO Qwen3-VL with QLoRA using TRL on free Colab

OpenEnv Notebooks#

These notebooks demonstrate how to train models with OpenEnv environments using GRPOTrainer’s environment_factory. The BrowserGym notebook uses the lower-level rollout_func API instead. See the OpenEnv Integration guide for more details.

Notebook	Description	Open in Colab
openenv_wordle_grpo.ipynb	GRPO to play Wordle on an OpenEnv environment
openenv_sudoku_grpo.ipynb	GRPO to play Sudoku on an OpenEnv environment
grpo_functiongemma_browsergym_openenv.ipynb	GRPO on FunctionGemma in the BrowserGym environment

Scripts#

Scripts are maintained in the trl/scripts and examples/scripts directories. They show how to use different trainers such as SFTTrainer, PPOTrainer, DPOTrainer, GRPOTrainer, and more.

OpenEnv Scripts#

These scripts demonstrate how to train models with OpenEnv environments using GRPOTrainer’s environment_factory. See the OpenEnv Integration guide for more details.

Distributed Training (for scripts)#

You can run scripts on multiple GPUs with 🤗 Accelerate:

accelerate launch --config_file=examples/accelerate_configs/multi_gpu.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script

For DeepSpeed ZeRO-{1,2,3}:

accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero{1,2,3}.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script

Adjust NUM_GPUS and --all_arguments_of_the_script as needed.