Custom inference hardware built for very high tokens-per-second on open-weight models.
Directory
LLM Inference Hosting and Serverless GPU Compared
Inference hosting is the middle ground between calling a frontier API and running your own GPUs. Someone else operates the hardware; you get an endpoint for an open-weight model, or a place to run your own.
The category splits in two, and the split matters more than any individual provider. Per-token endpoints work exactly like a frontier API — you call a model, you pay per million tokens, you never see a machine. Per-second GPU compute rents you the hardware and runs your code, which is what you want for custom models, fine-tuned weights, or work that is not text generation at all.
Choose this over a frontier API when you need a specific open-weight model, when weights must stay somewhere you control, when you are serving a fine-tune, or when raw speed on a smaller model beats capability on a larger one.
6 tools · 0 open source · 0 self-hostable
At a glance
| Tool | Best for | Pricing | Runs |
|---|---|---|---|
| Groq | Interactive products where response speed matters more than reaching for the strongest model. | Usage-based | Hosted |
| Together AI | Breadth of open-weight models in one place, and going from fine-tune to served endpoint without changing vendor. | Usage-based | Hosted |
| Fireworks AI | Serving many fine-tuned variants where a dedicated deployment each would not pay. | Usage-based | Hosted |
| Modal | Custom inference code, batch GPU work, and anything a fixed endpoint cannot express. | Usage-based | Hosted |
| Baseten | Putting a custom or fine-tuned model into production as a real endpoint with monitoring and autoscaling. | Usage-based | Hosted |
| Replicate | Multi-modal work and trying many community models without deploying any of them yourself. | Usage-based | Hosted |
Serverless and dedicated endpoints for a broad catalogue of open-weight models, plus fine-tuning.
Fast hosted inference for open-weight models with support for LoRA adapters and custom deployments.
Serverless GPU compute billed by the second — for when you want to run your own inference code, not an endpoint.
Deploys custom and open-weight models as autoscaling production endpoints.
Run thousands of community models via one API, billed per second of compute.
How to choose
Per token or per second
Per-token endpoints are simple and idle for free. Per-second compute is more flexible and bills for cold starts and idle capacity unless you scale to zero. Pick by whether you need custom code in the loop.
Cold starts
Serverless GPU means a container that may not be running. First-request latency after idle can be seconds. Check whether the provider offers warm pools and what keeping one costs.
Custom and fine-tuned weights
Serving a catalogue model is easy everywhere. Serving your own fine-tune, or a LoRA adapter on a base model, is the capability that actually differentiates these providers.
Speed against capability
Some providers compete on tokens per second rather than model quality. That is the right trade for interactive interfaces and the wrong one for hard reasoning — and it is worth measuring against what a frontier model costs on the same workload.
Frequently asked
Why use hosted inference instead of a frontier API?
To run a specific open-weight model, to serve your own fine-tune, to keep weights somewhere you control, or because a smaller model served very fast fits the product better than a larger one served slowly.
What is serverless GPU?
Renting GPU compute billed by the second, scaled to zero when idle. You supply the code and the model rather than calling a fixed endpoint, which suits custom inference, batch jobs, and non-text workloads.
Is hosted open-weight inference cheaper than a frontier API?
Per token, usually — smaller open-weight models are much cheaper to serve. Whether that lowers your bill depends on whether the smaller model needs more attempts, longer prompts, or more human correction to reach the same result.
What is a cold start?
The delay when a request arrives and no instance is running, covering container start and loading model weights into GPU memory. It can be seconds. Providers mitigate it with warm pools, which cost money to keep idle.
Tooling choices rarely dominate a bill; the model does. Compare model prices per million tokens or price your own workload before optimising anything here.