Can I run Llama 3.1 8B locally?
Yes. At Q4 with 8K context, Llama 3.1 8B needs about 6.5 GB, so it runs entirely on any GPU with 7 GB of VRAM or more, starting with the NVIDIA GeForce RTX 5060 Ti 8GB.
Memory needed by compression and context length
| Compression | 4K context | 8K context | 32K context | 128K context |
|---|---|---|---|---|
Q4 The most common download. Small quality loss. | 6.0 GB | 6.5 GB | 9.5 GB | 22 GB |
Q5 Very close to full quality. | 6.8 GB | 7.3 GB | 10 GB | 22 GB |
Q8 Practically indistinguishable from FP16 at about half the size. | 9.4 GB | 9.9 GB | 13 GB | 25 GB |
FP16 Uncompressed. Reference quality, largest size. | 16 GB | 17 GB | 20 GB | 32 GB |
Total memory: model weights, context (KV cache) and about 1 GB of runtime overhead. To run entirely on the GPU, VRAM must be about 5% larger than this.
GPUs that run Llama 3.1 8B entirely on the GPU
- Amazon
NVIDIA GeForce RTX 5060 Ti 8GB
8 GB · ~59 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 5060
8 GB · ~59 tok/s at Q4
- Amazon
AMD Radeon RX 9060 XT 8GB
8 GB · ~43 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 4060
8 GB · ~37 tok/s at Q4
- Amazon
AMD Radeon RX 6650 XT
8 GB · ~38 tok/s at Q4
- Amazon
AMD Radeon RX 5700 XT 50th Anniversary
8 GB · ~59 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 2060 SUPER
8 GB · ~59 tok/s at Q4
- Amazon
AMD Radeon RX 7600
8 GB · ~39 tok/s at Q4
- Amazon
AMD Radeon RX 6600 XT
8 GB · ~35 tok/s at Q4
- Amazon
NVIDIA GeForce GTX 1080
8 GB · ~43 tok/s at Q4
Model facts
- Parameters
- 8B
- Max context
- 128K tokens
- License
- Llama 3.1 Community License
- Released
- Jul 2024
Check it with your hardware
Change the compression, context length and system RAM in the calculator.
Open calculator