Can I Run This AI Model Locally?

Pick your GPU to see which open AI models it can run and how fast, or pick a model to see how much memory it needs and which GPUs can handle it.

Your Hardware

GPU

System RAM

RAM type

Context length

2 of 18 models run entirely on the RTX 2060

6 GB VRAM, 32 GB system RAM. Each model is shown at the best compression level that fits.

Model	Runs	Memory	Speed
Llama 3.2 3B Q8	On GPU	5.1 GB	~62 tok/s
Gemma 3 4B Q8	On GPU	5.5 GB	~48 tok/s
Mistral 7B v0.3 Q4	GPU + RAM	6.1 GB	~38 tok/s
Llama 3.1 8B Q4	GPU + RAM	6.5 GB	~29 tok/s
Qwen3 8B Q4	GPU + RAM	6.7 GB	~26 tok/s
Gemma 3 12B Q4	GPU + RAM	8.7 GB	~13 tok/s
Mistral Nemo 12B Q4	GPU + RAM	9.2 GB	~12 tok/s
Phi-4 14B Q4	GPU + RAM	11 GB	~8.4 tok/s
Qwen3 14B Q4	GPU + RAM	11 GB	~8.7 tok/s
gpt-oss 20B MXFP4 · MoE, 3.6B active	GPU + RAM	14 GB	~30 tok/s
Mistral Small 3.1 24B Q4	GPU + RAM	16 GB	~4.6 tok/s
Gemma 3 27B Q4	GPU + RAM	18 GB	~3.9 tok/s
Qwen3 30B-A3B Q4 · MoE, 3.3B active	GPU + RAM	19 GB	~31 tok/s
Qwen3 32B Q4	GPU + RAM	21 GB	~3.0 tok/s
Llama 3.3 70B Q4	Won't run	43 GB	—
gpt-oss 120B MXFP4 · MoE, 5.1B active	Won't run	62 GB	—
Qwen3 235B-A22B Q4 · MoE, 22B active	Won't run	135 GB	—
DeepSeek R1 671B Q4 · MoE, 37B active	Won't run	380 GB	—

How the estimate is calculated

The numbers follow how runtimes like llama.cpp, Ollama and LM Studio use memory. Here is exactly how they are produced.

Model weights

Every parameter is stored at the chosen compression level. Q4 averages just under 5 bits per weight once block scales are included; FP16 uses the full 16.

weights = parameters × bitsPerWeight / 8

Q4 4.85 · Q5 5.69 · Q8 8.5 · FP16 16 bits. gpt-oss uses its published MXFP4 checkpoint size.

Context memory (KV cache)

The model keeps keys and values for every token in the conversation. The size per token comes from each model's published architecture; sliding-window layers only keep their window.

kv = 2 × layers × kvHeads × headDim × 2 bytes × tokens

Plus about 1 GB of runtime overhead for the GPU context and compute buffers.

Does it fit?

About 95% of VRAM is usable after the desktop and driver take their share. Whatever doesn't fit can spill into system RAM, of which we assume 75% is free for the model.

total ≤ 0.95 × VRAM → On GPU
spill ≤ 0.75 × RAM → GPU + RAM

Anything larger won't run without more hardware.

Speed estimate

Generating a token means reading every active weight once, so speed is set by memory bandwidth. Mixture-of-experts models only read their active parameters, which is why they're fast for their size.

seconds = gpuPart / (0.7 × gpuBW)
        + ramPart / (0.6 × ramBW) + 1.5 ms

Single-user generation speed. Prompt processing is usually much faster.

Estimates are a guide, not a guarantee. Real memory use and speed vary with the runtime, its settings, drivers and operating system. GPU memory bandwidths are reference specifications.