Can I run Llama 3.3 70B locally?
Not on a single consumer GPU. Even at Q4 with 8K context, Llama 3.3 70B needs about 43 GB. That takes several GPUs, a workstation card, or a machine with a lot of fast system RAM, where it runs slowly.
Memory needed by compression and context length
| Compression | 4K context | 8K context | 32K context | 128K context |
|---|---|---|---|---|
Q4 The most common download. Small quality loss. | 42 GB | 43 GB | 51 GB | 81 GB |
Q5 Very close to full quality. | 49 GB | 50 GB | 58 GB | 88 GB |
Q8 Practically indistinguishable from FP16 at about half the size. | 72 GB | 73 GB | 81 GB | 111 GB |
FP16 Uncompressed. Reference quality, largest size. | 134 GB | 135 GB | 142 GB | 172 GB |
Total memory: model weights, context (KV cache) and about 1 GB of runtime overhead. To run entirely on the GPU, VRAM must be about 5% larger than this.
GPUs that run Llama 3.3 70B entirely on the GPU
None of the GPUs in our database has the 46 GB of VRAM it needs at Q4.
Model facts
- Parameters
- 70.5B
- Max context
- 128K tokens
- License
- Llama 3.3 Community License
- Released
- Dec 2024
Check it with your hardware
Change the compression, context length and system RAM in the calculator.
Open calculator