PEFT ↗
yesEditorial Notes
This is the entry point to PEFT and the right place to grasp why parameter-efficient fine-tuning matters: methods like LoRA train a small set of extra parameters instead of all of them, cutting compute and storage enough to fine-tune large models on consumer hardware. Focus on the menu of method families and on PEFT’s integration with Transformers, Diffusers, and Accelerate, which is what makes it practical. A common misconception is that PEFT sacrifices quality, when in practice it reaches performance comparable to full fine-tuning. Read the quicktour next, then the LoRA developer guide for the method you will actually reach for.
Original Documentation
🤗 PEFT (Parameter-Efficient Fine-Tuning) is a library for efficiently adapting large pretrained models to various downstream applications without fine-tuning all of a model’s parameters because it is prohibitively costly. PEFT methods only fine-tune a small number of (extra) model parameters - significantly decreasing computational and storage costs - while yielding performance comparable to a fully fine-tuned model. This makes it more accessible to train and store large language models (LLMs) on consumer hardware.
PEFT is integrated with the Transformers, Diffusers, and Accelerate libraries to provide a faster and easier way to load, train, and use large models for inference.
Quicktour Start here if you’re new to 🤗 PEFT to get an overview of the library’s main features, and how to train a model with a PEFT method.
How-to guides Practical guides demonstrating how to apply various PEFT methods across different types of tasks like image classification, causal language modeling, automatic speech recognition, and more. Learn how to use 🤗 PEFT with the DeepSpeed and Fully Sharded Data Parallel scripts.
Conceptual guides Get a better theoretical understanding of how LoRA and various soft prompting methods help reduce the number of trainable parameters to make training more efficient.
Reference Technical descriptions of how 🤗 PEFT classes and methods work.