Can I run Mistral Nemo 12B locally?
Yes. At Q4 with 8K context, Mistral Nemo 12B needs about 9.2 GB, so it runs entirely on any GPU with 10 GB of VRAM or more, starting with the AMD Radeon RX 6750 GRE 10GB.
Memory needed by compression and context length
| Compression | 4K context | 8K context | 32K context | 128K context |
|---|---|---|---|---|
Q4 The most common download. Small quality loss. | 8.5 GB | 9.2 GB | 13 GB | 28 GB |
Q5 Very close to full quality. | 9.7 GB | 10 GB | 14 GB | 29 GB |
Q8 Practically indistinguishable from FP16 at about half the size. | 14 GB | 14 GB | 18 GB | 33 GB |
FP16 Uncompressed. Reference quality, largest size. | 24 GB | 25 GB | 29 GB | 44 GB |
Total memory: model weights, context (KV cache) and about 1 GB of runtime overhead. To run entirely on the GPU, VRAM must be about 5% larger than this.
GPUs that run Mistral Nemo 12B entirely on the GPU
- Amazon
AMD Radeon RX 6750 GRE 10GB
10 GB · ~29 tok/s at Q4
- Amazon
AMD Radeon RX 6700
10 GB · ~29 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 3080
10 GB · ~65 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 2080 Ti
11 GB · ~53 tok/s at Q4
- Amazon
AMD Radeon RX 7700 XT
12 GB · ~38 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 3060
12 GB · ~32 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 3060 12GB
12 GB · ~32 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 2060 12GB
12 GB · ~30 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 4070 Ti
12 GB · ~44 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 4070 SUPER
12 GB · ~44 tok/s at Q4
Model facts
- Parameters
- 12.3B
- Max context
- 128K tokens
- License
- Apache 2.0
- Released
- Jul 2024
Check it with your hardware
Change the compression, context length and system RAM in the calculator.
Open calculator