GPU guide
Ollama on a cloud GPU: run an LLM online on your own server
Updated on
Short answer
To run Ollama on a real GPU, GPU Cloud installs Ollama and Open WebUI in one click on an NVIDIA RTX A6000 with 48 GB, at CA$0.67/h, taxes extra. That memory is enough for models up to about 32 billion parameters; for a 70 billion parameter model, the cheapest server that is enough is 2 × RTX A6000, at CA$1.34/h.
Why move Ollama off your computer
On a laptop without an NVIDIA card, or when the model does not fit in your card’s memory, Ollama runs part of the layers on the CPU. Answers then come word by word, slowly, and a 32 or 70 billion parameter model becomes hard to use.
On a GPU server rented by the hour, the whole model fits in GPU memory. You stay in control: the models are open, the conversations are stored on your server in Montreal, and you delete everything when you are done.
How much GPU memory for which model
| Model | Memory to plan for | Cheapest server that is enough | On demand / h |
|---|---|---|---|
| Llama 3.1 8B | 6 GB | 1 × RTX A6000 | $0.67 |
| Qwen2.5 14B | 11 GB | 1 × RTX A6000 | $0.67 |
| Qwen2.5 32B | 24 GB | 1 × RTX A6000 | $0.67 |
| Llama 3.3 70B | 51 GB | 2 × RTX A6000 | $1.34 |
| Mistral Large 2 (123B) | 89 GB | 2 × RTX A6000 | $1.34 |
Memory to plan for: parameters × 0.6 byte (Q4_K_M) × 1.2 for the context and CUDA. It is an estimate: a long context needs more. Prices in Canadian dollars, taxes extra.
The smallest server, 48 GB, runs every common model up to 32 billion parameters with room for the context. Beyond that, Ollama spreads the model over several GPUs of the same server, with nothing to set up. To serve an API to many users at once, the self-host an LLM with vLLM guide is a better fit.
Price and availability of the servers for Ollama
| GPU | GPUs per server | Server GPU memory | On demand, per hour | Availability |
|---|---|---|---|---|
| NVIDIA RTX A6000 | 1 | 48 GB | $0.67 | |
| NVIDIA L40 | 1 | 48 GB | $1.34 | |
| NVIDIA RTX A6000 | 2 | 96 GB | $1.34 | |
| NVIDIA A100 PCIe | 1 | 80 GB | $1.81 | |
| NVIDIA RTX PRO 6000 | 1 | 96 GB | $2.48 |
Prices in Canadian dollars, taxes extra, read from the catalog when the page is built. Availability is read live; a machine out of stock shows the closest equivalent machine in stock.
Start Ollama in a few minutes
1.Pick the Ollama and Open WebUI one-click template
In the console, deploy the Ollama and Open WebUI template on the server you want. The install takes 5 to 10 minutes.
2.Open Open WebUI in your browser
Open WebUI is served over HTTPS, and the admin account is created for you before access opens.
3.Pull a model
From Open WebUI, pull a model from the Ollama library (for example
llama3.1:8borqwen2.5:32b), then start chatting.4.Connect your tools
From your computer, your scripts and tools go through Open WebUI with an API key generated in your Open WebUI account.
What it really costs
- A 3-hour evening of tests on an RTX A6000: $2.01.
- 20 evenings of 3 hours in the month: $40.20, if you delete or hibernate the server between evenings.
- Hibernated, the server costs $0.02 per hour (about $14.60 per month) and keeps the models you pulled. A server that is only stopped keeps costing the full rate.
Good to know before you launch
- Billing is hourly, at the server’s hourly rate, from prepaid credit. There is no per-minute billing, and the first hour is charged when you deploy.
- A stopped server is still billed at the full hourly rate. To pay less, hibernate it ($0.02 per hour (about $14.60 per month), disk kept) or delete it. A Spot machine cannot be hibernated: delete it when you are done.
- The minimum top-up is CA$25.00, paid by credit card.
- Servers and their disks stay in Canada (region canada-montreal, in Montreal). Account information (identity, billing, e-mails) may be processed outside Quebec, as the privacy policy explains.
Frequently asked questions
How much does a GPU for Ollama cost?
From CA$0.67/h for an NVIDIA RTX A6000 with 48 GB, taxes extra, billed by the hour in Canadian dollars.
Which GPU for Llama 3.3 70B with Ollama?
It needs about 51 GB of GPU memory in Q4_K_M. The cheapest server that is enough is 2 × RTX A6000, at CA$1.34/h.
Where are my conversations processed?
On your server in Montreal: Ollama and Open WebUI run there, and the conversations are stored on its disk, in Canada.
Can I use Ollama through an API from my computer?
Yes, through Open WebUI with an API key created in your Open WebUI account.
Do I have to install Ollama myself?
No. The one-click template installs Ollama and Open WebUI for you, with the NVIDIA driver and HTTPS access. You keep full SSH access to the server.
Need GPUs? We’ve got you.
Launch an NVIDIA GPU server by the hour in Canada, paid in Canadian dollars by credit card, or reserve a sold-out GPU: it launches as soon as one frees up.
Other guides
- Your AI says you need a GPU? Run your Python script on a GPU in minutes
- CUDA out of memory: causes, fixes, and when to move to a GPU with more VRAM
- torch.cuda.is_available() returns False: causes and fixes (Mac, PC without NVIDIA, Docker)
- How much does a GPU cost per hour in Canada, in Canadian dollars?
- OpenAI API too expensive? Host your LLM on a GPU by the hour, in Canada
- Rent a GPU by the hour in Canada: prices, steps and billing
- GPU cloud in Montreal, Canada: NVIDIA GPU servers hosted in the country
- Rent an H100 in Canada: PCIe, NVLink or SXM, by the hour
- Which GPU to fine-tune an LLM? Memory, machine and cost
- Cheap GPU cloud: how to pay as little as possible for a GPU
- GPU for Stable Diffusion and ComfyUI: generate images on a rented server
- Rent an 8-GPU server: 8x H100 or 8x A100 NVLink, by the hour
- RunPod alternative in Canada: GPUs by the hour, in Canadian dollars
- GPU for university research in Quebec: rent a GPU by the hour
- Health data and AI: GPUs hosted in Canada