How to run an open-weight LLM locally: GGUF, quantization, Ollama and llama.cpp

In short

You can run an open-weight language model on your own computer with a free tool such as Ollama or llama.cpp. Download a model in GGUF format, which is usually shrunk with quantization so it fits in memory, and the tool serves it through a chat prompt and a local API. Memory is the limit: the weights must fit in your GPU memory (or a Mac's unified memory), or the model falls back to slower system RAM.

What "open-weight" and "local" mean

An open-weight model is one whose trained parameters (the weights) are published for download, usually on Hugging Face. That is not the same as open source: the licence decides what you may do with the files, so read it before building a product on a model. Our news coverage of recent releases, such as Reflection AI's Beam under Apache 2.0, shows how much licences differ.

Running locally means the weights and your prompts stay on your machine. You pay for hardware and electricity instead of per-token fees, you work offline, and you control the version. You give up the largest models, which need more memory than a laptop has, and you take on the setup yourself.

Step 1: understand the file format (GGUF)

Hugging Face's documentation describes GGUF as a binary format optimised for quick loading and saving of models, designed for use with GGML-based executors. It was developed by Georgi Gerganov, who also wrote llama.cpp. Unlike tensor-only formats such as safetensors, GGUF stores both the tensors and a standard set of metadata, so one file carries what a runner needs. You can browse GGUF models on the Hub with the GGUF library filter, and the Hub has a viewer that shows each file's metadata and tensor shapes.

Step 2: understand quantization, because it decides what fits

A model's size is roughly its parameter count times the bits stored per weight. Quantization stores each weight in fewer bits. llama.cpp's README lists support for 1.5-bit through 8-bit integer quantization, "for faster inference and reduced memory use". Hugging Face's GGUF page gives the bits per weight for the common "K" types:

Q6_K6.5625

Q5_K5.5

Q4_K4.5

Q3_K3.4375

Q2_K2.625

Bits per weight of common GGUF quantization types (bits per weight)

	bits per weight
Q6_K	6.5625
Q5_K	5.5
Q4_K	4.5
Q3_K	3.4375
Q2_K	2.625

Source: Hugging Face Hub documentation, GGUF page

As a rule of thumb (our arithmetic, not a vendor figure), the weights take about parameters x bits / 8 bytes. A model with 8 billion parameters at 4.5 bits per weight is about 4.5 GB of weights. At 16 bits it would be about 16 GB. You also need room for the context cache and runtime overhead; for why that cache grows with long prompts, see what a KV cache is. Lower-bit types save memory but usually cost some quality, so start at 4 or 5 bits and move only if you must.

Step 3: pick a runner

Two common free choices:

• Ollama: the quickest path. Install the app for macOS, Windows or Linux, then pull and run a model. Its quickstart uses ollama pull gemma4:e2b followed by ollama run gemma4:e2b, and /bye to leave the chat. On Linux, start the server with ollama serve if it is not already running. Models are named name:tag.
• llama.cpp: more control. Its README gives install commands for macOS/Linux (curl -LsSf https://llama.app/install.sh | sh) and Windows (irm https://llama.app/install.ps1 | iex), or you can use Docker, prebuilt releases or build from source. It can download and run a model straight from Hugging Face with the -hf flag. The README writes the commands as llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF for a chat and llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF for an OpenAI-compatible server; the tool docs are named cli, completion and server, so the exact binary name on your install may differ. Check the README for your version.

Piping a script from the internet into your shell runs whatever it contains. If you prefer, download the installer, read it, then run it, or use a package manager or the prebuilt release.

Step 4: check your memory and pick a size

Ollama's quickstart gives a concrete example: the Gemma 4 E2B download is about 7.2 GB and the guide recommends 8 GB of available VRAM, or unified memory on a Mac. Larger context windows need more memory. With less VRAM, Ollama can use system RAM, but responses will be slower. llama.cpp similarly supports CPU+GPU hybrid inference for models larger than your VRAM.

GPU support in llama.cpp's backend table includes CUDA for Nvidia, HIP for AMD, Metal for Apple Silicon, Vulkan for GPUs in general and SYCL for Intel GPUs. If your hardware is not on the list, expect CPU speeds.

Practical order of attack: choose a small model first (a few billion parameters), confirm it runs, then step up. If the model is slow or the machine swaps, go down a size or a quantization level.

Worked example: will it fit?

Using the rule of thumb above (parameters x bits per weight / 8, weights only, our arithmetic), here is what an 8-billion-parameter model needs at different settings. Add a few gigabytes for the context cache and runtime.

Setting	Bits per weight	Weights only
16-bit float (F16)	16	about 16 GB
Q6_K	6.5625	about 6.6 GB
Q5_K	5.5	about 5.5 GB
Q4_K	4.5	about 4.5 GB
Q3_K	3.4375	about 3.4 GB

So an 8 GB GPU can hold the 4-bit version of such a model with some room for context, but not the 16-bit version. A 24 GB card has room for a larger model or a longer context. If the weights alone exceed your memory, llama.cpp's CPU+GPU hybrid mode and Ollama's fallback to system RAM keep it running, at lower speed. The same arithmetic tells you what a bigger model costs: double the parameters, double the weights.

Which runner should you start with?

If you want to be chatting in five minutes, start with Ollama: one install, two commands, and a local server on port 11434 that speaks several API dialects. If you want to choose the quantization file yourself, tune GPU offloading, or build from source for unusual hardware, llama.cpp gives you that control and lists many more backends (CUDA, HIP, Metal, Vulkan, SYCL, OpenCL and others). Both read GGUF, so you can try one and switch later without re-downloading everything in a new format.

Step 5: use it from your own code

Both tools expose a local API. Ollama listens on http://localhost:11434. Its paths include /api/chat (its own API), /v1/chat/completions (OpenAI Chat Completions), /v1/responses (OpenAI Responses) and /v1/messages (Anthropic Messages). The Responses endpoint is stateless, so each request has to carry the whole conversation. Because the OpenAI-style path exists, many existing client libraries work by changing the base URL to the local address. llama.cpp's server is likewise described as OpenAI-compatible.

A minimal test with curl against Ollama's chat endpoint:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e2b",
  "messages": [{"role": "user", "content": "Explain quantization in one sentence."}],
  "stream": false
}'

Step 6: stay safe

• Keep the server on localhost. Do not expose the port to the internet or your whole network without authentication in front of it.
• Download from publishers you trust, and check the model page for the publisher and licence before you run anything.
• Treat local models as untrusted when they act. If you connect one to tools or an agent, the same prompt-injection risks apply as with hosted models; see our report on instruction files that can steer coding agents.

Common problems

• Out of memory or very slow: the model plus context does not fit. Use a smaller model, a lower-bit quantization, or a shorter context.
• Model runs on CPU only: the GPU backend for your hardware is missing or not selected. Check which backend your build uses (CUDA, Metal, Vulkan and so on).
• Answers are worse than a hosted model: expected for small, heavily quantized models. Try a larger size or a higher-bit type before blaming the tool.

What we haven't covered

We have not benchmarked speed or quality for any model, and hardware varies too much for us to promise numbers. Model names and install commands change quickly; always check the linked pages for the current ones.

Sources

• llama.cpp README (ggml-org)
• Ollama quickstart
• Hugging Face Hub documentation: GGUF