LiteLLM

LiteLLM has become a widely used open-source AI gateway. Several of our teams route their model calls through it, either as a standalone proxy or as a sidecar. It supports more than 100 LLM providers and offers routing, provider translation, authentication, budgets and guardrails. Its proxy core is being rewritten from Python to Rust to improve performance while retaining the same Python SDK and configuration.

We’re moving LiteLLM from Languages and Frameworks to Platforms in this edition because we now see it more as infrastructure than a library. We’re keeping it in Trial rather than Adopt largely because of security concerns: at the time of writing, more than 25 CVEs have been reported this year alone. Teams using it should apply security patches promptly. To also govern MCP and agent-to-agent traffic, consider combining it with agentgateway.

LiteLLM started as a thin abstraction layer over multiple LLM providers but has since expanded into a full-fledged AI gateway. Beyond simplifying API integration, it addresses common cross-cutting concerns in GenAI systems such as retries and failover, load balancing across providers and cost tracking with budget controls.

Our teams are increasingly adopting LiteLLM as a sensible default for AI-powered applications. The gateway provides a consistent place to address governance concerns, including request tracing, access control, API key management and edge-level guardrails such as content filtering and data redaction or masking. However, teams that rely on differentiating provider features will find these often require provider-specific parameters, reintroducing the coupling the gateway is meant to eliminate. It’s also worth noting that drop_params mode silently discards unsupported parameters, meaning capabilities may be lost across routing decisions without visibility. LiteLLM remains a pragmatic choice for operational control, but teams should understand that leaning into provider-specific capabilities means maintaining both a gateway dependency and provider-coupled code.

LiteLLM 是一个 SDK，可通过标准化的 OpenAI API 格式 实现与多个 LLM 服务商的无缝集成。它支持多种服务商和模型，为文本生成、嵌入和图像生成提供统一接口。通过屏蔽不同服务商的 API 差异，LiteLLM 简化了集成流程，并会自动将请求路由至对应模型端点。其代理框架还包含生产级特性，如安全控制、缓存、日志记录、限流和负载均衡。随着企业将 AI 应用更深入地嵌入工作流，治理和可观测性变得尤为重要。我们的团队一直将 LiteLLM 用作 AI 网关，以实现企业级 AI 使用的标准化、安全管控和可见性。

LiteLLM 是一个用于无缝集成各种大语言模型（LLM）提供商 API 的库，通过OpenAI API 格式交互。它支持多种 提供方和模型 ，并为文本生成、嵌入和图像生成提供统一的接口。LiteLLM 简化了集成过程，通过匹配每个提供商的特定端点要求来翻译输入。它还提供了实现生产应用中所需的操作功能的框架，如缓存、日志记录、速率限制和负载均衡，从而确保不同 LLM 的一致操作。我们的团队使用 LiteLLM 来轻松切换各种模型，这在模型快速演变的今天尤为必要。然而，值得注意的是，不同模型在相同提示下的响应会有所不同，这表明仅一致的调用方式可能不足以优化生成性能。此外，每个模型对附加功能的实现方式也各不相同，单一接口可能无法满足所有需求。例如，我们的一个团队在通过 LiteLLM 代理 AWS Bedrock 模型时，难以充分利用其函数调用功能。

LiteLLM 是一个库，通过 OpenAI API 格式 的标准化交互实现与各种大语言模型（LLM）提供商的 API 的无缝集成。它广泛支持各类提供商和模型，并具备一个用于完成、嵌入和图像生成功能的统一界面。LiteLLM 通过将输入转换为匹配每个提供商特定端点要求的方式，简化了集成。在当前环境下，这特别有价值，因为缺乏标准化的 LLM 提供商 API 规范会导致项目中包含多个 LLM。我们的团队已经利用 LiteLLM 在 LLM 应用中更换底层模型，解决了一个重大的集成挑战。然而，需要认识到，它对相同提示的模型有不同响应，这表明仅仅一致的调用方法可能不足以完全优化完成性能。请注意，LiteLLM 还有其他一些特性，如 代理服务器，这些不在本次讨论范围内。