Huggingface accelerate

--- name: huggingface-accelerate description: The simplest distributed training API. Just 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive configuration, single launch command. HuggingFace ecosystem standard. version: 1.0.0 author: Orchestra Research license: MIT tags: [Distributed Training, HuggingFace, Accelerate, DeepSpeed, FSDP, Mixed Precision, PyTorch, DDP, Unified API, Simple] dependencies: [accelerate, torch, transformers] --- # HuggingFace Accelerate - Unified Distributed Training ## Quick start Accelerate simplifies distributed training to 4 lines of code. Installation: ``bash pip install accelerate **Convert PyTorch script** (4 lines): python import torch + from accelerate import Accelerator + accelerator = Accelerator() model = torch.nn.Transformer() optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset) + model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: optimizer.zero_grad() loss = model(batch) - loss.backward() + accelerator.backward(loss) optimizer.step() **Run** (single command): bash accelerate launch train.py ## Common workflows ### Workflow 1: From single GPU to multi-GPU **Original script**: python # train.py import torch model = torch.nn.Linear(10, 2).to('cuda') optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) for epoch in range(10): for batch in dataloader: batch = batch.to('cuda') optimizer.zero_grad() loss = model(batch).mean() loss.backward() optimizer.step() **With Accelerate** (4 lines added): python # train.py import torch from accelerate import Accelerator # +1 accelerator = Accelerator() # +2 model = torch.nn.Linear(10, 2) optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3 for epoch in range(10): for batch in dataloader: # No .to('cuda') needed - automatic! optimizer.zero_grad() loss = model(batch).mean() accelerator.backward(loss) # +4 optimizer.step() **Configure** (interactive): bash accelerate config **Questions**: - Which machine? (single/multi GPU/TPU/CPU) - How many machines? (1) - Mixed precision? (no/fp16/bf16/fp8) - DeepSpeed? (no/yes) **Launch** (works on any setup): bash # Single GPU accelerate launch train.py # Multi-GPU (8 GPUs) accelerate launch --multi_gpu --num_processes 8 train.py # Multi-node accelerate launch --multi_gpu --num_processes 16 \ --num_machines 2 --machine_rank 0 \ --main_process_ip $MASTER_ADDR \ train.py ### Workflow 2: Mixed precision training **Enable FP16/BF16**: python from accelerate import Accelerator # FP16 (with gradient scaling) accelerator = Accelerator(mixed_precision='fp16') # BF16 (no scaling, more stable) accelerator = Accelerator(mixed_precision='bf16') # FP8 (H100+) accelerator = Accelerator(mixed_precision='fp8') model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # Everything else is automatic! for batch in dataloader: with accelerator.autocast(): # Optional, done automatically loss = model(batch) accelerator.backward(loss) ### Workflow 3: DeepSpeed ZeRO integration **Enable DeepSpeed ZeRO-2**: python from accelerate import Accelerator accelerator = Accelerator( mixed_precision='bf16', deepspeed_plugin={ "zero_stage": 2, # ZeRO-2 "offload_optimizer": False, "gradient_accumulation_steps": 4 } ) # Same code as before! model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) **Or via config**: bash accelerate config # Select: DeepSpeed → ZeRO-2 **deepspeed_config.json**: ``json { "fp16": {"enabled": false}, "bf16": {"enabled": true}, "zero_optimization": { "stage":