Ollama
open sourceThe simplest way to pull and run open-weight models locally, with an OpenAI-compatible endpoint.
Directory
Run LLMs Locally: Tools Compared
Local inference means the model runs on hardware you control. The reasons are usually privacy, offline capability, or removing per-token cost from a high-volume task — and occasionally that you simply want to.
The trade is real and worth stating plainly. An open-weight model you can run on a laptop is not competitive with a frontier API on hard reasoning or long-horizon agentic work. It is often entirely adequate for classification, extraction, summarisation, and drafting, which is a large share of what production systems actually do.
The practical constraint is memory. A quantised model needs roughly its file size in RAM or VRAM, so the question is not whether your machine can run a model but which size fits. Serving many concurrent users is a different problem from running one, and wants a different tool.
5 tools · 4 open source · 1 self-hostable
At a glance
| Tool | Best for | Pricing | Runs |
|---|---|---|---|
| Ollamaopen source | Anyone trying local inference for the first time, and for local development against a model that costs nothing per call. | Open source | Runs locally |
| llama.cppopen source | Squeezing a model onto constrained hardware, and embedding inference into your own application. | Open source | Runs locally |
| LM Studio | Running models locally without the terminal, and comparing several quickly in one interface. | Free tier | Runs locally |
| vLLMopen source | Serving an open-weight model to real traffic on GPUs you operate. | Open source | Self-hosted |
| MLXopen source | Getting the most out of a Mac, and local fine-tuning on Apple silicon. | Open source | Runs locally |
llama.cpp
open sourceThe C/C++ inference engine most local tooling is built on, with aggressive quantisation support.
Desktop app for downloading, chatting with, and serving local models — no terminal required.
vLLM
open sourceHigh-throughput serving engine for your own GPUs; the standard choice for self-hosted production inference.
MLX
open sourceApple’s array framework for Apple silicon, with a growing set of ready-to-run model ports.
How to choose
One user or many
Single-user tools optimise for a fast start and easy model switching. Serving engines optimise throughput across concurrent requests via batching. Using the first for a production endpoint wastes most of your GPU.
Your hardware
Apple silicon, NVIDIA, and CPU-only are genuinely different targets. Some tools cover all three; others are built for one and are much faster on it.
How much you want to see
A desktop app hides quantisation and context settings behind sensible defaults. A CLI or library exposes them, which matters once you are tuning for a specific machine.
API compatibility
Most local runners expose an OpenAI-compatible endpoint, which lets you point existing code at localhost. This is the cheapest way to test whether a local model is good enough before committing.
Frequently asked
What hardware do I need to run an LLM locally?
Enough memory to hold the model. A quantised model needs roughly its file size in RAM or VRAM, so small models run on an ordinary laptop while larger ones want a dedicated GPU. Apple silicon works well because CPU and GPU share memory.
Are local models as good as frontier APIs?
Not on hard reasoning, long-horizon agentic work, or the largest context windows. For classification, extraction, summarisation, and drafting they are frequently good enough, and the gap has narrowed considerably.
Is running a model locally cheaper?
It removes per-token cost and replaces it with hardware and electricity. That wins at sustained high volume on a machine you already own, and loses badly on bursty low volume where an API charges you nothing while idle.
Can I use local models with existing code?
Usually. Most local runners expose an OpenAI-compatible endpoint, so pointing your base URL at localhost is often the only change required.
Tooling choices rarely dominate a bill; the model does. Compare model prices per million tokens or price your own workload before optimising anything here.