Local inference
vLLM
High-throughput serving engine for your own GPUs; the standard choice for self-hosted production inference.
- Category
- Local inference
- Pricing
- Open source
- Runs
- Self-hosted
- Languages
- Python
- Interface
- Server, Library
- Source
- Open source
What vLLM is
vLLM is a serving engine built for throughput across many concurrent requests, using continuous batching and paged attention to keep a GPU busy instead of idling between requests. It is the default answer for self-hosted production inference, and serves an OpenAI-compatible API.
Best for
Serving an open-weight model to real traffic on GPUs you operate.
Consider something else if
It expects a proper GPU and is oriented to Linux servers — not the tool for a laptop.
vLLM alternatives
The closest options in local inference, on the axes that actually separate them.
| Tool | Best for | Pricing | Runs |
|---|---|---|---|
| vLLMopen source | Serving an open-weight model to real traffic on GPUs you operate. | Open source | Self-hosted |
| llama.cppopen source | Squeezing a model onto constrained hardware, and embedding inference into your own application. | Open source | Runs locally |
| Ollamaopen source | Anyone trying local inference for the first time, and for local development against a model that costs nothing per call. | Open source | Runs locally |
| LM Studio | Running models locally without the terminal, and comparing several quickly in one interface. | Free tier | Runs locally |
| MLXopen source | Getting the most out of a Mac, and local fine-tuning on Apple silicon. | Open source | Runs locally |
Choosing within local inference
One user or many
Single-user tools optimise for a fast start and easy model switching. Serving engines optimise throughput across concurrent requests via batching. Using the first for a production endpoint wastes most of your GPU.
Your hardware
Apple silicon, NVIDIA, and CPU-only are genuinely different targets. Some tools cover all three; others are built for one and are much faster on it.
Questions
What is vLLM?
vLLM is a serving engine built for throughput across many concurrent requests, using continuous batching and paged attention to keep a GPU busy instead of idling between requests. It is the default answer for self-hosted production inference, and serves an OpenAI-compatible API. It is open source and self-hosted.
What are the alternatives to vLLM?
The closest alternatives are llama.cpp, Ollama, LM Studio, MLX. They sit in the same category — local inference — and differ mainly on hosting model, pricing shape, and how much they abstract away.
Is vLLM the right choice?
Serving an open-weight model to real traffic on GPUs you operate. The main caveat: It expects a proper GPU and is oriented to Linux servers — not the tool for a laptop.
Whatever you build on, the model is the line item that scales. See what each one costs per million tokens, or price your own workload.