💥 OpenAI Proxy Server

📄️ Quick Start

Quick start CLI, Config, Docker

Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.

Input, Output, Exceptions are mapped to the OpenAI format for all supported models

Load balance multiple instances of the same model

Track Spend, Set budgets and create virtual keys for the proxy

Allow your users to create their own keys through a UI

Requirements:

Add new models + Get model info without restarting proxy.

If a call fails after num_retries, fall back to another model group.

Use this to health check all LLMs defined in your config.yaml

Modify data just before making litellm completion calls call on proxy

Cache LLM Responses

Get alerts for failed db read/writes, hanging api calls, failed api calls.

Log Proxy Input, Output, Exceptions using Custom Callbacks, Langfuse, OpenTelemetry, LangFuse, DynamoDB

Step 1 - Create your custom litellm callback class

Dockerfile

Cli arguments, --host, --port, --num_workers