Most "run AI locally" advice assumes a GPU you don't have. This tutorial is for the other 95%: a laptop or small server with 16–64 GB of RAM and no CUDA card. The good news: for a surprising range of tasks, a quantized model on CPU is good enough — you just have to pick the right size and set expectations correctly.
Step 1: Figure out what "useful" means for you
Before downloading anything, write down the actual job: summarizing your own notes? Classifying support tickets? Drafting replies? Each has a different quality bar. A 7B model that summarizes poorly may still classify tickets at 95%+ accuracy. Match the model to the job, not to benchmark leaderboards.
Step 2: The RAM rule
A rule of thumb that holds up in practice: usable context + weights must fit in RAM with ~20% headroom. Quantized weights run roughly:
- Q4_K_M: ~0.6 bytes per parameter → a 7B model ≈ 4.5 GB, a 13B ≈ 8 GB, a 32B ≈ 19 GB
- Q8: ~1 byte per parameter → roughly 1.7× the Q4 size
- F16: 2 bytes per parameter → only viable for small models or big machines
Add 1–2 GB for the KV cache at moderate context (4–8k tokens). On a 16 GB machine, that puts you at ~7B Q4 comfortably or ~13B Q4 tightly. On 32 GB, a 32B Q4 is realistic.
Step 3: Pick a model family, not a hype cycle
As of late 2026 the reliable CPU-friendly picks are the small instruct-tuned models from the major open families — anything in the 3B–14B range with a Q4_K_M GGUF build from a reputable quantizer. Avoid:
- Models under ~3B for anything beyond toy tasks — they fall apart on instruction-following
- "Merge" and "abliterated" community quants with no evals — fine for experiments, risky for anything you depend on
- Reasoning-tuned models for latency-sensitive work — the thinking tokens cost you 3–10× the time
Step 4: Serve it
Ollama is the fastest path:
ollama run qwen3:8b # or whatever 7-14B instruct model is current
For production-ish use, llama.cpp's server or vLLM (GPU only) give you an OpenAI-compatible endpoint. On CPU, keep num_ctx modest — context is RAM and compute both.
Step 5: Test on YOUR data, not theirs
Take 20 real examples of your task. Run them through the model. Count the failures. If it's under ~10%, ship it; if it's 30%+, either shrink the task or move up a model size. This 30-minute test beats any benchmark chart.
What to expect honestly
- CPU inference runs ~5–30 tokens/sec depending on model size and cores — fine for batch work, painful for chat
- Quantization to Q4 loses a few points on hard reasoning; for most business tasks it's invisible
- A local 13B Q4 will NOT beat a frontier API model — it will be cheaper, private, and offline-capable. That's the trade.
That's the whole tutorial. The unglamorous version: small model, right-sized context, tested on real data, honest expectations.