Inference hosting
Modal
Serverless GPU compute billed by the second — for when you want to run your own inference code, not an endpoint.
- Category
- Inference hosting
- Pricing
- Usage-based
- Runs
- Hosted
- Languages
- Python
- Interface
- Library, CLI
- Source
- Proprietary
What Modal is
Modal is serverless GPU compute rather than a model endpoint: you write Python, decorate a function with the hardware it needs, and Modal runs it on a GPU billed by the second, scaling to zero when idle. That makes it the right tool when inference is your own code — a custom pipeline, a batch job, or something that is not text generation.
Best for
Custom inference code, batch GPU work, and anything a fixed endpoint cannot express.
Consider something else if
You are responsible for the serving code, and cold starts apply when nothing is warm.
Modal alternatives
The closest options in inference hosting, on the axes that actually separate them.
| Tool | Best for | Pricing | Runs |
|---|---|---|---|
| Modal | Custom inference code, batch GPU work, and anything a fixed endpoint cannot express. | Usage-based | Hosted |
| Baseten | Putting a custom or fine-tuned model into production as a real endpoint with monitoring and autoscaling. | Usage-based | Hosted |
| Replicate | Multi-modal work and trying many community models without deploying any of them yourself. | Usage-based | Hosted |
| Together AI | Breadth of open-weight models in one place, and going from fine-tune to served endpoint without changing vendor. | Usage-based | Hosted |
| Groq | Interactive products where response speed matters more than reaching for the strongest model. | Usage-based | Hosted |
| Fireworks AI | Serving many fine-tuned variants where a dedicated deployment each would not pay. | Usage-based | Hosted |
Choosing within inference hosting
Per token or per second
Per-token endpoints are simple and idle for free. Per-second compute is more flexible and bills for cold starts and idle capacity unless you scale to zero. Pick by whether you need custom code in the loop.
Cold starts
Serverless GPU means a container that may not be running. First-request latency after idle can be seconds. Check whether the provider offers warm pools and what keeping one costs.
Questions
What is Modal?
Modal is serverless GPU compute rather than a model endpoint: you write Python, decorate a function with the hardware it needs, and Modal runs it on a GPU billed by the second, scaling to zero when idle. That makes it the right tool when inference is your own code — a custom pipeline, a batch job, or something that is not text generation. It is a commercial product and hosted.
What are the alternatives to Modal?
The closest alternatives are Baseten, Replicate, Together AI, Groq, Fireworks AI. They sit in the same category — inference hosting — and differ mainly on hosting model, pricing shape, and how much they abstract away.
Is Modal the right choice?
Custom inference code, batch GPU work, and anything a fixed endpoint cannot express. The main caveat: You are responsible for the serving code, and cold starts apply when nothing is warm.
Whatever you build on, the model is the line item that scales. See what each one costs per million tokens, or price your own workload.