Which GPU to fine-tune an LLM? Memory, machine and cost

How much GPU memory to fine-tune an LLM

Why such a gap: in full fine-tuning, every parameter keeps its gradients and the optimizer states. With LoRA, the model is frozen and only small adapters train; with QLoRA, the frozen model is also loaded in 4-bit. The CUDA out of memory guide explains how to reduce activations.

The machines to rent, with their availability

How much a fine-tuning costs

For example, a LoRA of Llama 3.1 8B that runs 3 hours on an RTX A6000 costs CA$2.01. To know your real duration, run a short pass (a few hundred steps) and multiply.

Start a LoRA fine-tuning

Deploy the PyTorch (CUDA) template: PyTorch and CUDA are ready in /opt/pytorch. Then install the fine-tuning libraries and load the model with its LoRA adapters.

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

# Gated model: accept its license on Hugging Face, then `huggingface-cli login`
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", torch_dtype=torch.bfloat16, device_map="auto"
)
model.gradient_checkpointing_enable()
config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

For full fine-tuning on several GPUs (for example 4 × RTX A6000, 192 GB), shard the optimizer states across the GPUs with FSDP or DeepSpeed ZeRO. Full fine-tuning of a 70 billion parameter model (about 1,120 GB before activations) goes beyond a server sold online: reserve GPUs to discuss it.

Serve the fine-tuned model

Once the adapters are merged into the model, the vLLM one-click template serves it through an OpenAI-compatible API, from CA$0.67/h. The guide host your LLM on a GPU compares that cost with a per-token API.

Good to know before you launch

• Billing is hourly, at the server’s hourly rate, from prepaid credit. There is no per-minute billing, and the first hour is charged when you deploy.
• A stopped server is still billed at the full hourly rate. To pay less, hibernate it ($0.02 per hour (about $14.60 per month), disk kept) or delete it. A Spot machine cannot be hibernated: delete it when you are done.
• The minimum top-up is CA$25.00, paid by credit card.
• Servers and their disks stay in Canada (region canada-montreal, in Montreal). Account information (identity, billing, e-mails) may be processed outside Quebec, as the privacy policy explains.

Frequently asked questions