Local inference
llama.cpp
The C/C++ inference engine most local tooling is built on, with aggressive quantisation support.
- Category
- Local inference
- Pricing
- Open source
- Runs
- Runs locally
- Languages
- C++
- Interface
- CLI, Library, Server
- Source
- Open source
What llama.cpp is
llama.cpp is the engine underneath much of this category, including tools that never mention it. It runs on CPU, Apple silicon, and NVIDIA GPUs, and its quantisation work is what makes large models fit on ordinary hardware at all. Using it directly gives you every knob at the cost of setting them yourself.
Best for
Squeezing a model onto constrained hardware, and embedding inference into your own application.
Consider something else if
Direct use means compiling and choosing quantisation and context settings yourself; a wrapper is friendlier.
llama.cpp alternatives
The closest options in local inference, on the axes that actually separate them.
| Tool | Best for | Pricing | Runs |
|---|---|---|---|
| llama.cppopen source | Squeezing a model onto constrained hardware, and embedding inference into your own application. | Open source | Runs locally |
| Ollamaopen source | Anyone trying local inference for the first time, and for local development against a model that costs nothing per call. | Open source | Runs locally |
| LM Studio | Running models locally without the terminal, and comparing several quickly in one interface. | Free tier | Runs locally |
| MLXopen source | Getting the most out of a Mac, and local fine-tuning on Apple silicon. | Open source | Runs locally |
| vLLMopen source | Serving an open-weight model to real traffic on GPUs you operate. | Open source | Self-hosted |
Choosing within local inference
One user or many
Single-user tools optimise for a fast start and easy model switching. Serving engines optimise throughput across concurrent requests via batching. Using the first for a production endpoint wastes most of your GPU.
Your hardware
Apple silicon, NVIDIA, and CPU-only are genuinely different targets. Some tools cover all three; others are built for one and are much faster on it.
Questions
What is llama.cpp?
llama.cpp is the engine underneath much of this category, including tools that never mention it. It runs on CPU, Apple silicon, and NVIDIA GPUs, and its quantisation work is what makes large models fit on ordinary hardware at all. Using it directly gives you every knob at the cost of setting them yourself. It is open source and runs locally.
What are the alternatives to llama.cpp?
The closest alternatives are Ollama, LM Studio, MLX, vLLM. They sit in the same category — local inference — and differ mainly on hosting model, pricing shape, and how much they abstract away.
Is llama.cpp the right choice?
Squeezing a model onto constrained hardware, and embedding inference into your own application. The main caveat: Direct use means compiling and choosing quantisation and context settings yourself; a wrapper is friendlier.
Whatever you build on, the model is the line item that scales. See what each one costs per million tokens, or price your own workload.