Can I run Mistral 7B v0.3 locally?
Yes. At Q4 with 8K context, Mistral 7B v0.3 needs about 6.1 GB, so it runs entirely on any GPU with 7 GB of VRAM or more, starting with the NVIDIA GeForce RTX 5060 Ti 8GB.
Memory needed by compression and context length
| Compression | 4K context | 8K context | 32K context |
|---|---|---|---|
Q4 The most common download. Small quality loss. | 5.6 GB | 6.1 GB | 9.1 GB |
Q5 Very close to full quality. | 6.3 GB | 6.8 GB | 9.8 GB |
Q8 Practically indistinguishable from FP16 at about half the size. | 8.7 GB | 9.2 GB | 12 GB |
FP16 Uncompressed. Reference quality, largest size. | 15 GB | 16 GB | 19 GB |
Total memory: model weights, context (KV cache) and about 1 GB of runtime overhead. To run entirely on the GPU, VRAM must be about 5% larger than this.
GPUs that run Mistral 7B v0.3 entirely on the GPU
- Amazon
NVIDIA GeForce RTX 5060 Ti 8GB
8 GB · ~64 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 5060
8 GB · ~64 tok/s at Q4
- Amazon
AMD Radeon RX 9060 XT 8GB
8 GB · ~47 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 4060
8 GB · ~41 tok/s at Q4
- Amazon
AMD Radeon RX 6650 XT
8 GB · ~42 tok/s at Q4
- Amazon
AMD Radeon RX 5700 XT 50th Anniversary
8 GB · ~64 tok/s at Q4
- Amazon
NVIDIA GeForce RTX 2060 SUPER
8 GB · ~64 tok/s at Q4
- Amazon
AMD Radeon RX 7600
8 GB · ~43 tok/s at Q4
- Amazon
AMD Radeon RX 6600 XT
8 GB · ~38 tok/s at Q4
- Amazon
NVIDIA GeForce GTX 1080
8 GB · ~47 tok/s at Q4
Check it with your hardware
Change the compression, context length and system RAM in the calculator.
Open calculator