Can I run Gemma 3 4B locally?
Yes. At Q4 with 8K context, Gemma 3 4B needs about 3.7 GB, so it runs entirely on any GPU with 4 GB of VRAM or more, starting with the NVIDIA GeForce RTX 2060.
Memory needed by compression and context length
| Compression | 4K context | 8K context | 32K context | 128K context |
|---|---|---|---|---|
Q4 The most common download. Small quality loss. | 3.6 GB | 3.7 GB | 4.2 GB | 6.0 GB |
Q5 Very close to full quality. | 4.0 GB | 4.1 GB | 4.6 GB | 6.5 GB |
Q8 Practically indistinguishable from FP16 at about half the size. | 5.4 GB | 5.5 GB | 6.0 GB | 7.9 GB |
FP16 Uncompressed. Reference quality, largest size. | 9.2 GB | 9.3 GB | 9.7 GB | 12 GB |
Total memory: model weights, context (KV cache) and about 1 GB of runtime overhead. To run entirely on the GPU, VRAM must be about 5% larger than this.
GPUs that run Gemma 3 4B entirely on the GPU
- Amazon
NVIDIA GeForce RTX 2060
6 GB · ~79 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 5060 Ti 8GB
8 GB · ~102 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 5060
8 GB · ~102 tok/s at Q4
- Amazon
AMD Radeon RX 9060 XT 8GB
8 GB · ~76 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 4060
8 GB · ~66 tok/s at Q4
- Amazon
AMD Radeon RX 6650 XT
8 GB · ~68 tok/s at Q4
- Amazon
AMD Radeon RX 5700 XT 50th Anniversary
8 GB · ~102 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 2060 SUPER
8 GB · ~102 tok/s at Q4
- Amazon
AMD Radeon RX 7600
8 GB · ~69 tok/s at Q4
- Amazon
AMD Radeon RX 6600 XT
8 GB · ~62 tok/s at Q4
Model facts
- Parameters
- 4.3B
- Max context
- 128K tokens
- License
- Gemma Terms of Use
- Released
- Mar 2025
Check it with your hardware
Change the compression, context length and system RAM in the calculator.
Open calculator