Can I run Llama 3.2 3B locally?
Yes. At Q4 with 8K context, Llama 3.2 3B needs about 3.7 GB, so it runs entirely on any GPU with 4 GB of VRAM or more, starting with the NVIDIA GeForce RTX 2060.
Memory needed by compression and context length
| Compression | 4K context | 8K context | 32K context | 128K context |
|---|---|---|---|---|
Q4 The most common download. Small quality loss. | 3.2 GB | 3.7 GB | 6.3 GB | 17 GB |
Q5 Very close to full quality. | 3.6 GB | 4.0 GB | 6.6 GB | 17 GB |
Q8 Practically indistinguishable from FP16 at about half the size. | 4.6 GB | 5.1 GB | 7.7 GB | 18 GB |
FP16 Uncompressed. Reference quality, largest size. | 7.4 GB | 7.9 GB | 10 GB | 21 GB |
Total memory: model weights, context (KV cache) and about 1 GB of runtime overhead. To run entirely on the GPU, VRAM must be about 5% larger than this.
GPUs that run Llama 3.2 3B entirely on the GPU
- Amazon
NVIDIA GeForce RTX 2060
6 GB · ~102 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 5060 Ti 8GB
8 GB · ~130 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 5060
8 GB · ~130 tok/s at Q4
- Amazon
AMD Radeon RX 9060 XT 8GB
8 GB · ~98 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 4060
8 GB · ~85 tok/s at Q4
- Amazon
AMD Radeon RX 6650 XT
8 GB · ~87 tok/s at Q4
- Amazon
AMD Radeon RX 5700 XT 50th Anniversary
8 GB · ~130 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 2060 SUPER
8 GB · ~130 tok/s at Q4
- Amazon
AMD Radeon RX 7600
8 GB · ~90 tok/s at Q4
- Amazon
AMD Radeon RX 6600 XT
8 GB · ~81 tok/s at Q4
Model facts
- Parameters
- 3.2B
- Max context
- 128K tokens
- License
- Llama 3.2 Community License
- Released
- Sep 2024
Check it with your hardware
Change the compression, context length and system RAM in the calculator.
Open calculator