← tutorials.pro

Running a Useful Local LLM on Consumer Hardware

Local inference · updated Nov 2026

Most "run AI locally" advice assumes a GPU you don't have. This tutorial is for the other 95%: a laptop or small server with 16–64 GB of RAM and no CUDA card. The good news: for a surprising range of tasks, a quantized model on CPU is good enough — you just have to pick the right size and set expectations correctly.

Step 1: Figure out what "useful" means for you

Before downloading anything, write down the actual job: summarizing your own notes? Classifying support tickets? Drafting replies? Each has a different quality bar. A 7B model that summarizes poorly may still classify tickets at 95%+ accuracy. Match the model to the job, not to benchmark leaderboards.

Step 2: The RAM rule

A rule of thumb that holds up in practice: usable context + weights must fit in RAM with ~20% headroom. Quantized weights run roughly:

Add 1–2 GB for the KV cache at moderate context (4–8k tokens). On a 16 GB machine, that puts you at ~7B Q4 comfortably or ~13B Q4 tightly. On 32 GB, a 32B Q4 is realistic.

Step 3: Pick a model family, not a hype cycle

As of late 2026 the reliable CPU-friendly picks are the small instruct-tuned models from the major open families — anything in the 3B–14B range with a Q4_K_M GGUF build from a reputable quantizer. Avoid:

Step 4: Serve it

Ollama is the fastest path:

ollama run qwen3:8b   # or whatever 7-14B instruct model is current

For production-ish use, llama.cpp's server or vLLM (GPU only) give you an OpenAI-compatible endpoint. On CPU, keep num_ctx modest — context is RAM and compute both.

Step 5: Test on YOUR data, not theirs

Take 20 real examples of your task. Run them through the model. Count the failures. If it's under ~10%, ship it; if it's 30%+, either shrink the task or move up a model size. This 30-minute test beats any benchmark chart.

What to expect honestly

That's the whole tutorial. The unglamorous version: small model, right-sized context, tested on real data, honest expectations.