Make a 27B model fit on a 16 GB card

A 27B model at full context does not fit in 16 GB. It does fit if you cut the context down and stop the model from thinking out loud before every answer. This is the setup that has been stable for me.

What you need

A card with 16 GB of VRAM. I use an RTX 4080 SUPER.
32 GB of system RAM or more. I have 60 GB, and the spare RAM is what stops the machine from falling over when something spills.
Ollama installed and running on 127.0.0.1:11434.

Pull the model

ollama pull qwen38-24k

Bake the settings into a tag

Context size and thinking are per-run settings, and typing them every time is how you end up running at the default context without noticing. Put them in a model file instead.

FROM qwen38-24k
PARAMETER num_ctx 24576
PARAMETER num_gpu 999

Save that as Modelfile-24k, then:

ollama create qwen38-24k -f Modelfile-24k

num_gpu 999 means "offload everything you can". Letting Ollama decide sounds smarter than it is: a partial offload is the difference between a fast model and a model that makes you go and make tea.

Thinking off is a request setting in most clients rather than a model file line. If your client supports it, turn it off. On a 27B model with 16 GB of memory, the tokens spent thinking are tokens spent not answering, and they cost you time you feel.

Check it actually fits

This is the step people skip.

ollama ps

Look at the processor column. It should say 100% GPU. If it says anything else, part of the model is in system RAM and every token is now being passed over the bus. The fix is less context, a smaller model, or a smaller KV cache, not more patience.

What this gets you