Directory
The stack around the model.
38 tools, grouped by the decision they belong to. No scores and no rankings — each entry says what it is for, who it suits, and the reason you might pick something else.
Gateways & routers
One endpoint in front of many models. Useful for failover, spend controls, and swapping models without touching application code.
- OpenRouterA single API and single bill across models from most major providers, with automatic fallback when one is down.
- LiteLLMopen sourcePython SDK and self-hostable proxy that puts a consistent interface over 100+ model APIs, with budgets and rate limits per key.
- Portkeyopen sourceAI gateway with routing, caching, retries, and guardrails, plus request-level observability.
- Cloudflare AI GatewayDrop-in proxy that adds caching, rate limiting, and analytics to provider calls from Cloudflare’s edge.
Coding agents
Agents that read, write, and run code in a real repository rather than suggesting snippets.
- Claude CodeAnthropic’s agentic coding tool, available in the terminal, desktop, web, and as IDE extensions for VS Code and JetBrains.
- CursorAn editor built around model-assisted editing, with repo-wide context and an agent mode.
- Aideropen sourceTerminal pair programmer that edits files in your git repo and commits as it goes.
- Clineopen sourceOpen-source VS Code agent that plans, edits files, and runs terminal commands with per-step approval.
- OpenHandsopen sourceOpen-source agent platform that runs software-engineering tasks in a sandboxed environment.
Agent frameworks
Libraries for building the loop yourself: tool calling, state, orchestration, evaluation hooks.
- Claude Agent SDKopen sourceThe Claude Code harness as a library: built-in file, shell, and search tools, subagents, hooks, and permissions.
- LangChainopen sourceThe most widely used orchestration library, with LangGraph for explicit stateful agent graphs.
- LlamaIndexopen sourceData framework focused on ingestion, indexing, and retrieval pipelines feeding a model.
- Pydantic AIopen sourceAgent framework from the Pydantic team, built around typed, validated model outputs.
- DSPyopen sourceTreats prompts as parameters to be optimised against a metric instead of hand-tuned strings.
- Mastraopen sourceTypeScript agent framework with workflows, memory, and evals in one package.
Vector & retrieval
Where your embeddings live, and what queries them when a model needs grounding.
- pgvectoropen sourceVector similarity search inside Postgres. Usually the right first answer if you already run Postgres.
- Qdrantopen sourceOpen-source vector database in Rust, with strong payload filtering alongside similarity search.
- PineconeFully managed vector database; no infrastructure to run, priced per index and per operation.
- Weaviateopen sourceOpen-source vector database with built-in hybrid keyword-plus-vector search.
- Chromaopen sourceEmbeddable vector store that runs in-process, which makes it the quickest way to prototype retrieval.
- turbopufferVector and full-text search built on object storage, priced for large, comparatively cold datasets.
Evals & observability
Tracing, cost attribution, and regression testing. The category most teams skip and later wish they had not.
- Langfuseopen sourceOpen-source, self-hostable tracing, prompt management, and evaluation for model applications.
- LangSmithTracing, datasets, and evaluation from the LangChain team; works with or without LangChain.
- BraintrustEval-first platform for scoring model output against datasets and catching regressions before release.
- Heliconeopen sourceOne-line proxy that logs every request with cost and latency attribution per user or feature.
- Phoenixopen sourceOpen-source OpenTelemetry-based tracing and evaluation from Arize, runnable in a notebook.
- W&B Weaveopen sourceTracing and evaluation from Weights & Biases, aimed at teams already tracking experiments there.
Local inference
Run open-weight models on your own hardware — laptop, workstation, or a GPU box you own.
- Ollamaopen sourceThe simplest way to pull and run open-weight models locally, with an OpenAI-compatible endpoint.
- llama.cppopen sourceThe C/C++ inference engine most local tooling is built on, with aggressive quantisation support.
- LM StudioDesktop app for downloading, chatting with, and serving local models — no terminal required.
- vLLMopen sourceHigh-throughput serving engine for your own GPUs; the standard choice for self-hosted production inference.
- MLXopen sourceApple’s array framework for Apple silicon, with a growing set of ready-to-run model ports.
Inference hosting
Someone else runs the GPUs and bills you per token or per second for open-weight models.
- GroqCustom inference hardware built for very high tokens-per-second on open-weight models.
- Together AIServerless and dedicated endpoints for a broad catalogue of open-weight models, plus fine-tuning.
- Fireworks AIFast hosted inference for open-weight models with support for LoRA adapters and custom deployments.
- ModalServerless GPU compute billed by the second — for when you want to run your own inference code, not an endpoint.
- BasetenDeploys custom and open-weight models as autoscaling production endpoints.
- ReplicateRun thousands of community models via one API, billed per second of compute.
Frequently asked
What tools do I need to build with an LLM?
Fewer than the category count suggests. A provider SDK covers a surprising amount on its own. Most teams add observability first, because a prompt change that quietly degrades quality is invisible without it, then retrieval if the model needs facts it was not trained on, then a gateway once failover or hard spend limits become real requirements.
Are these tools ranked?
No. There is no score, no ordering by preference, and no paid placement. Each entry states what it is best for and the main reason to choose something else, so you can rank them against your own constraints.
Which of these are open source?
23 of the 38 listed are open source, and most of those can be self-hosted. Every entry states its licensing and hosting model.
None of this matters as much as what the model costs. See the price board or price your own workload.