# Nexus Documentation Nexus is an AI Router that provides unified endpoints for both MCP (Model Context Protocol) servers and LLM providers. It enables you to aggregate, govern, and manage your entire AI infrastructure through a single interface. It provides two main capabilities: ### LLM Routing - **Multi-provider AI model routing** through a single OpenAI or Anthropic compatible API - **Unified access** to OpenAI, Anthropic, and Google models - **Streaming support** for real-time responses across all providers - **Model discovery** and consistent API interface ### MCP Routing Instead of configuring each MCP server individually in every AI tool, Nexus provides: - **Single endpoint** for all your MCP tools - **Unified authentication** with OAuth2 token forwarding - **Performance optimization** through intelligent connection caching - **Tool aggregation** with automatic namespacing to prevent conflicts - **Enterprise-ready security** with TLS and authentication options ### Architecture Overview Nexus Architecture Overview
## Quick Start Get Nexus working with your AI assistant in minutes: ### 1. Install Nexus ```bash # Using the install script curl -fsSL https://nexusrouter.com/install | bash # Or run it with Docker docker run -p 8000:8000 \ -v $(pwd)/nexus.toml:/etc/nexus.toml \ ghcr.io/grafbase/nexus:stable ``` ### 2. Create Configuration Create a `nexus.toml` file: ```toml # LLM configuration for AI model routing [llm] enabled = true # Enable protocol endpoints [llm.protocols.openai] enabled = true path = "/llm/openai" [llm.protocols.anthropic] enabled = true path = "/llm/anthropic" # Configure multiple AI providers [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Models must be explicitly configured [llm.providers.openai.models.gpt-4] [llm.providers.openai.models."gpt-4o-mini"] [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" [llm.providers.anthropic.models."claude-3-5-sonnet-20241022"] # MCP configuration for tool integration [mcp] enabled = true path = "/mcp" # Add a simple file system server [mcp.servers.filesystem] cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home/user/documents"] # Add a GitHub MCP server [mcp.servers.github] url = "https://api.github.com/mcp" [mcp.servers.github.auth] token = "{{ env.GITHUB_TOKEN }}" ``` ### 3. Start Nexus ```bash # Start with default settings nexus # Or specify a config file nexus --config ./nexus.toml ``` ### 4. Connect Your AI Assistant #### Cursor Integration 1. Open Cursor Settings (Cmd+, on macOS) 2. Search for "Model Context Protocol" 3. Enable MCP support 4. Add to the MCP server configuration: ```json { "nexus": { "transport": { "type": "http", "url": "http://localhost:8000/mcp" } } } ``` ## How It Works Nexus simplifies MCP integration by exposing just two tools to AI assistants: 1. **`search`** - Discover tools from all connected MCP servers 2. **`execute`** - Run any discovered tool This design allows Nexus to aggregate tools from multiple servers without overwhelming the AI assistant with hundreds of individual tools. ## Key Features ### Universal Connectivity - **LLM Providers**: OpenAI, Anthropic, and Google through single API - **MCP Servers**: STDIO (subprocess), SSE, and HTTP protocols - Automatic protocol detection for remote servers - Environment variable substitution in configuration ### Smart Tool Search - Natural language tool discovery across all connected servers - Fuzzy matching for finding relevant tools quickly - Namespaced tools prevent conflicts between servers - Avoid context bloating ### Enterprise Security - OAuth2 authentication with JWT validation - Token forwarding to downstream servers - TLS configuration for secure connections - CORS and CSRF protection ### Easy Deployment - Single binary installation - Docker support with minimal configuration - Health checks and monitoring endpoints #### Claude Desktop Integration ```bash claude mcp add --transport http nexus http://localhost:8000/mcp ``` ### 5. Start Using Features #### LLM Routing Access multiple AI providers through a single OpenAI-compatible API: ```bash # List available models from all providers curl http://localhost:8000/llm/openai/v1/models # Chat with any model using OpenAI format curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "anthropic/claude-3-5-sonnet-20241022", "messages": [{"role": "user", "content": "Hello!"}] }' ``` #### MCP Tools Once connected, your AI assistant will see two tools: - Use `search` to find available tools: `search for "file read"` - Use `execute` to run them: `execute filesystem__read_file with path "/home/user/documents/readme.md"` ## Documentation Overview - **[Installation](/docs/installation)** - Install Nexus using various methods - **[Server Configuration](/docs/configuration/server)** - Configure the Nexus server and security settings - **[LLM Configuration](/docs/configuration/llm)** - Configure AI model routing through multiple providers - **[MCP Configuration](/docs/configuration/mcp)** - Set up and manage MCP servers ## Use Cases ### For Developers - **Unified AI access**: Switch between OpenAI, Anthropic, and Google models seamlessly - **Tool consolidation**: Multiple development tools through one MCP interface - **Local development**: Test and debug both MCP servers and LLM integrations ### For Teams - **Centralized AI infrastructure**: Shared access to multiple LLM providers and tools - **Access control**: Authentication and authorization for both models and tools - **Cost optimization**: Monitor and control usage across all AI resources ### For Enterprises - **AI governance**: Centralized control over both LLM access and tool usage - **Security & compliance**: OAuth2 authentication with token forwarding - **Scalability**: Load balancing and caching for high-performance deployments ## Getting Help - **GitHub Issues**: [github.com/grafbase/nexus/issues](https://github.com/grafbase/nexus/issues) - **Documentation**: You're already here! --- # Installation Nexus can be installed using several methods depending on your environment and requirements. Choose the installation method that best suits your needs. ## Quick Install The fastest way to get started with Nexus is using our installation script: ```bash curl -fsSL https://nexusrouter.com/install | bash ``` This script will: - Detect your operating system and architecture - Download the appropriate binary - Install it to `~/.nexus/bin` - Make it executable ### Supported Platforms - Linux (x86_64, aarch64) - macOS (Intel, Apple Silicon) - Windows (via WSL) ## Docker Installation Nexus is available as a Docker image for containerized deployments. ### Pull the Stable Release (Recommended) ```bash docker pull ghcr.io/grafbase/nexus:stable ``` ### Pull a Specific Version ```bash docker pull ghcr.io/grafbase/nexus:0.2.0 ``` ### Pull the Latest Development Build ```bash docker pull ghcr.io/grafbase/nexus:latest ``` **Note:** The `stable` tag points to the latest released version, while `latest` follows the main branch and may include unreleased features. ### Running with Docker Basic usage: ```bash docker run -p 8000:8000 \ -v $(pwd)/nexus.toml:/etc/nexus.toml \ ghcr.io/grafbase/nexus:stable ``` With environment variables: ```bash docker run -p 8000:8000 \ -v $(pwd)/nexus.toml:/etc/nexus.toml \ -e GITHUB_TOKEN=${GITHUB_TOKEN} \ -e OPENAI_API_KEY=${OPENAI_API_KEY} \ ghcr.io/grafbase/nexus:stable ``` ### Docker Compose Create a `compose.yaml` file: ```yaml services: nexus: image: ghcr.io/grafbase/nexus:stable ports: - "8000:8000" volumes: - ./nexus.toml:/etc/nexus.toml # Mount additional directories if using STDIO servers - /path/to/mcp-servers:/opt/mcp-servers environment: - GITHUB_TOKEN=${GITHUB_TOKEN} - OPENAI_API_KEY=${OPENAI_API_KEY} healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 30s timeout: 10s retries: 3 restart: unless-stopped ``` Then run: ```bash docker compose up -d ``` ### Helm The Nexus Helm chart simplifies deployment with Kubernetes. The GitHub Container Registry hosts the chart with [Open Container Initiative (OCI)](https://helm.sh/docs/topics/registries/) compliance and is available here: ```bash https://ghcr.io/grafbase/helm-charts/nexus ``` ## Deploying The chart includes default installation configuration for quick setup with minimal configuration. While functional and easy to start with, tune this setup to accommodate real use cases. Follow these steps to install the default configuration and customize settings for: 1. Number of replicas 2. Auto-scaling 3. Compute resources 5. External configuration ### Setup Complete these prerequisites before deploying: 1. _Kubernetes Cluster:_ Get access to a Kubernetes cluster. Set up a local cluster like [kind](https://kind.sigs.k8s.io/) if needed. 2. _helm:_ Install Helm. Get started [here](https://helm.sh/docs/intro/quickstart/). 3. _kubectl:_ Install kubectl and point it to your cluster. Get started [here](https://kubernetes.io/docs/tasks/tools/#kubectl). ### Basic deployment Install the Nexus chart: ```bash helm install test oci://ghcr.io/grafbase/helm-charts/nexus --version ``` Verify Nexus operation: ```bash kubectl get pods ``` Look for a running pod named `test-nexus`. ### Customize deployment Use a Helm [values file](https://helm.sh/docs/chart_template_guide/values_files/) to customize your deployment: ```yaml # 1. number of desired replicas running replicaCount: 2 # 2. auto-scaling behaviour autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 targetMemoryUtilizationPercentage: 70 # 3. compute resources resources: limits: cpu: 2 memory: 2Gi requests: cpu: 1 memory: 1Gi # 4. external configuration from cluster configmaps nexus: externalConfig: true args: - --config - /etc/nexus/config/config.toml volumes: - name: configuration configMap: name: nexus-configuration volumeMounts: - name: configuration mountPath: /etc/nexus/config ``` This configuration: 1. Maintains 2 gateway replicas 2. Sets auto-scaling between 2 and 10 instances, scaling up at 70% CPU or memory usage 3. Allocates 1-2 CPU and 1-2GB memory per replica 4. Uses cluster configmaps for configuration and mounting them in the specified paths View all customizable values with: ```bash helm show values oci://ghcr.io/grafbase/helm-charts/nexus --version ``` To apply customizations: 1. Save your settings to a file 2. Run: ``` helm upgrade test oci://ghcr.io/grafbase/helm-charts/nexus --version -f custom-values.yaml ``` Verify the deployment: ```bash helm list kubectl get pods ``` ## Build from Source For advanced users who want to build Nexus from source: ### Prerequisites - Latest stable Rust - Git ### Build Steps 1. Clone the repository: ```bash git clone https://github.com/grafbase/nexus cd nexus ``` 2. Build the release binary: ```bash cargo build --release -p nexus ``` 3. The binary will be available at `target/release/nexus` 4. (Optional) Install to system path: ```bash mkdir -p ~/.nexus/bin sudo cp target/release/nexus ~/.nexus/bin sudo chmod +x ~/.nexus/bin/nexus ``` ## Verifying Installation After installation, verify Nexus is working: ```bash nexus --version ``` You should see output like: ``` nexus 0.2.0 ``` ## Running Nexus ### Basic Usage Start Nexus with default settings: ```bash nexus ``` This will: - Look for `nexus.toml` in the current directory - Start the server on `http://127.0.0.1:8000` - Enable the health endpoint at `/health` ### Command Line Options ```bash Usage: nexus [OPTIONS] Options: -l, --listen-address IP address on which the server will listen for incomming connections. Default: 127.0.0.1:6000 [env: NEXUS_LISTEN_ADDRESS=] -c, --config Path to the TOML configuration file [env: NEXUS_CONFIG_PATH=] [default: ./nexus.toml] --log Set the logging level, this applies to all spans, logs and trace events [env: NEXUS_LOG=] [default: info] Possible values: - off: Disable logging - error: Only log errors - warn: Log errors, and warnings - info: Log errors, warnings, and info messages - debug: Log errors, warnings, info, and debug messages - trace: Log errors, warnings, info, debug, and trace messages --log-style Set the style of log output [env: NEXUS_LOG_STYLE=] [default: color] Possible values: - color: Colorized text, used as the default with TTY output - text: Standard text, used as the default with non-TTY output - json: JSON objects -h, --help Print help (see a summary with '-h') -V, --version Print version ``` ### Environment Variables Nexus supports the following environment variables: - `NEXUS_LISTEN_ADDRESS`: Set the address and port to listen on (default: `127.0.0.1:8000`) - `NEXUS_CONFIG_PATH`: Alternative way to specify config file path - `NEXUS_LOG`: Set logging level (debug, info, warn, error) - `NEXUS_LOG_STYLE`: Set the style of log output (color, text, json) ## Configuration File Nexus requires a configuration file (default: `nexus.toml`). Create a basic configuration: ```toml # Add your first MCP server [mcp.servers.example] cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/tmp"] ``` See the [Server Configuration](/docs/configuration/server) and [MCP Configuration](/docs/configuration/mcp) sections for detailed configuration options. ## Troubleshooting ### Docker Network Issues If Docker containers can't reach Nexus: 1. Use `host.docker.internal` instead of `localhost` on macOS/Windows 2. Use the container name in Docker Compose networks 3. Ensure proper port mapping with `-p 8000:8000` --- # Configuration Nexus uses TOML format for all configuration. The configuration file defines your servers, authentication, rate limits, and other settings. ## Configuration File By default, Nexus looks for `nexus.toml` in the current directory: ```bash # Uses ./nexus.toml nexus # Or specify a custom config file nexus --config /path/to/config.toml ``` ## TOML Format Configuration uses standard TOML syntax: ```toml # LLM providers [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # MCP servers [mcp.servers.github] url = "https://api.github.com/mcp" ``` ## Environment Variables Use `{{ env.VARIABLE_NAME }}` to reference environment variables: ```toml [llm.providers.openai] api_key = "{{ env.OPENAI_API_KEY }}" [mcp.servers.database] cmd = ["psql-mcp"] env = { PGHOST = "{{ env.DB_HOST }}", PGPASSWORD = "{{ env.DB_PASSWORD }}" } ``` ## Configuration Sections 1. **[Server Configuration](/docs/configuration/server)** - HTTP server, authentication, security, and rate limiting 2. **[LLM Configuration](/docs/configuration/llm)** - AI model providers and routing 3. **[MCP Configuration](/docs/configuration/mcp)** - Model Context Protocol servers and tools 4. **[Telemetry Configuration](/docs/configuration/telemetry)** - OpenTelemetry metrics, traces, and logs ## Next Steps - Start with [Server Settings](/docs/configuration/server) for basic server configuration - Set up [LLM Providers](/docs/configuration/llm) for AI model access - Add [MCP Servers](/docs/configuration/mcp) for tool integration - Configure [Telemetry](/docs/configuration/telemetry) for observability --- # Configuration - Server Configure the Nexus server with comprehensive settings for network, security, authentication, and performance optimization. This section is organized by importance to help you quickly set up and secure your Nexus instance. ## Configuration Topics ### Essential Configuration 1. **[Core Settings](/docs/configuration/server/core)** - Basic server settings, TLS, and health checks 2. **[OAuth2 Authentication](/docs/configuration/server/oauth2)** - Secure your instance with JWT validation and token forwarding 3. **[Rate Limiting](/docs/configuration/server/rate-limiting)** - Protect against abuse with configurable limits ### Advanced Features 4. **[Client Identification](/docs/configuration/server/client-identification)** - Enable user tracking and tiered access control 5. **[CORS Configuration](/docs/configuration/server/cors)** - Support browser-based clients 6. **[CSRF Protection](/docs/configuration/server/csrf)** - Prevent cross-site request forgery attacks ## Quick Start Example Here's a complete example showing common server configurations: ```toml # Basic server settings [server] listen_address = "0.0.0.0:8000" [server.health] enabled = true path = "/health" # TLS for production [server.tls] certificate = "/etc/nexus/server.crt" key = "/etc/nexus/server.key" # OAuth2 authentication [server.oauth] url = "https://auth.example.com/.well-known/jwks.json" expected_issuer = "https://auth.example.com" expected_audience = "nexus-api" # Rate limiting [server.rate_limits] enabled = true storage = "memory" [server.rate_limits.per_ip] limit = 100 interval = "60s" # CORS for browser clients [server.cors] allow_origins = ["https://app.example.com"] allow_methods = ["GET", "POST", "OPTIONS"] allow_headers = ["authorization", "content-type"] allow_credentials = true ``` ## Configuration File Location Nexus looks for configuration in the following order: 1. Path specified by `--config` flag 2. `nexus.toml` in current directory 3. `~/.nexus/config.toml` 4. `/etc/nexus/config.toml` ## Environment Variables All configuration values support environment variable substitution: ```toml [server.oauth] url = "{{ env.OAUTH_JWKS_URL }}" expected_issuer = "{{ env.OAUTH_ISSUER }}" ``` ## Best Practices 1. **Start with core settings** and add features as needed 2. **Always enable OAuth2** in production environments 3. **Use TLS certificates** for secure connections 4. **Configure rate limiting** before going live 5. **Test CORS settings** with actual browser clients 6. **Monitor logs** for security events and errors ## Troubleshooting For debugging, run Nexus with increased verbosity: ```bash nexus --log debug ``` Check specific configuration sections for detailed troubleshooting guides. --- # Configuration - Server - Core Settings Configure the fundamental behavior of your Nexus server, including network settings, health checks, and TLS. ## Basic Server Settings Configure the core server behavior in your `nexus.toml`: ```toml [server] listen_address = "127.0.0.1:8000" [server.health] enabled = true path = "/health" ``` ### Configuration Options - `listen_address`: The address and port Nexus will bind to (optional, defaults to `127.0.0.1:8000`) - `health.enabled`: Enable the health check endpoint (default: `true`) - `health.path`: Path for health checks (default: `/health`) - `health.listen`: Separate address for health endpoint (optional) ## TLS Configuration For secure connections, configure TLS certificates: ```toml [server.tls] certificate = "/path/to/server.crt" key = "/path/to/server.key" ``` Both certificate and key must be in PEM format. ### TLS Best Practices 1. **Certificate Management** - Use certificates from trusted Certificate Authorities in production - Rotate certificates before expiration - Store certificate files with restricted permissions (600) 2. **Security Considerations** - Always use TLS in production environments - Keep TLS certificates outside of version control - Monitor certificate expiration dates ## Health Check Endpoint The health check endpoint is essential for monitoring and load balancer integration: ```toml [server.health] enabled = true path = "/health" listen = "0.0.0.0:8001" # Optional: separate port for health checks ``` ### Health Check Response When healthy, returns HTTP 200 with: ```json { "status": "healthy" } ``` ### Load Balancer Integration Use the health endpoint for: - Kubernetes liveness and readiness probes - AWS ELB/ALB health checks - Docker health checks - Monitoring systems (Prometheus, Datadog, etc.) ## Next Steps - Configure [OAuth2 Authentication](/docs/configuration/server/oauth2) for secure access - Set up [Rate Limiting](/docs/configuration/server/rate-limiting) to protect your server - Enable [Client Identification](/docs/configuration/server/client-identification) for user tracking --- # Configuration - Server - OAuth2 Authentication Nexus provides comprehensive OAuth2 support for securing your MCP endpoints. This is crucial for production deployments where you need to control access to your AI tools. ## Basic OAuth2 Configuration Configure OAuth2 authentication in your `nexus.toml`: ```toml [server.oauth] url = "https://your-auth-provider.com/.well-known/jwks.json" poll_interval = "5m" expected_issuer = "https://your-auth-provider.com" expected_audience = "your-nexus-instance" [server.oauth.protected_resource] resource = "https://nexus.example.com" authorization_servers = ["https://your-auth-provider.com"] ``` ## Configuration Parameters ### JWT Validation - `url`: The JWKS (JSON Web Key Set) endpoint URL for validating JWT tokens (required) - `poll_interval`: How often to refresh the JWKS (optional, format: "5m", "1h", etc.) - `expected_issuer`: The expected `iss` claim in JWT tokens (optional but recommended) - `expected_audience`: The expected `aud` claim in JWT tokens (optional but recommended) ### Protected Resource Metadata - `protected_resource.resource`: The URL of this Nexus instance as a protected resource (required) - `protected_resource.authorization_servers`: List of trusted authorization server URLs (required) The protected resource metadata is exposed at `/.well-known/oauth-protected-resource` as per RFC-9728. ## How OAuth2 Works in Nexus When OAuth2 is enabled: 1. **All endpoints require authentication** (except `/health` and `/.well-known/oauth-protected-resource`) 2. **JWT tokens are validated** using the configured JWKS endpoint 3. **Token claims are verified** against expected issuer and audience 4. **Access is granted or denied** based on token validity ## Provider Examples ### Auth0 Configuration ```toml [server.oauth] url = "https://your-tenant.auth0.com/.well-known/jwks.json" poll_interval = "15m" expected_issuer = "https://your-tenant.auth0.com/" expected_audience = "https://nexus.your-domain.com" [server.oauth.protected_resource] resource = "https://nexus.your-domain.com" authorization_servers = ["https://your-tenant.auth0.com/"] ``` ### Okta Configuration ```toml [server.oauth] url = "https://your-domain.okta.com/oauth2/default/v1/keys" poll_interval = "10m" expected_issuer = "https://your-domain.okta.com/oauth2/default" expected_audience = "api://nexus" [server.oauth.protected_resource] resource = "https://nexus.your-domain.com" authorization_servers = ["https://your-domain.okta.com/oauth2/default"] ``` ## Token Forwarding to Downstream Servers One of Nexus's powerful features is the ability to forward OAuth2 tokens to downstream MCP servers. This enables seamless authentication across your entire AI infrastructure. ### When to Use Token Forwarding Use token forwarding when: - Your downstream MCP servers use the same OAuth2 provider - You want single sign-on (SSO) across all tools - You need to maintain user context through the entire request chain ### Configuring Token Forwarding For each MCP server that should receive the forwarded token: ```toml [mcp.servers.protected_api] url = "https://api.example.com/mcp" [mcp.servers.protected_api.auth] type = "forward" ``` ### How Token Forwarding Works 1. **Client authenticates** with Nexus using a JWT token 2. **Nexus validates** the token using configured OAuth2 settings 3. **For servers with `type = "forward"`**, Nexus includes the same token in downstream requests 4. **Downstream servers** validate the token independently 5. **These dynamic connections are cached** per unique token (see [MCP cache configuration](/docs/configuration/mcp/authentication#connection-caching) for cache details) ## Using OAuth2 When OAuth2 is enabled, clients must include a valid JWT token: ```bash curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Authorization: Bearer your-jwt-token" \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ## Troubleshooting OAuth2 Issues ### 401 Unauthorized Errors - Verify the JWT token is valid and not expired - Check that the JWKS URL is accessible - Ensure expected claims match token contents - Verify the `Authorization: Bearer ` header is present ### Token Forwarding Not Working Common issues when Nexus accepts a token but downstream servers reject it: #### Configuration Issues - Confirm `type = "forward"` is set for the server - Ensure OAuth2 is enabled on the Nexus server - Check server logs for authentication errors #### Token Validation Mismatches - **Audience (`aud`) mismatch**: The token's audience claim might be set to Nexus's URL, not the downstream server's. Downstream servers often validate that they are the intended audience - **Resource identifier mismatch**: If using RFC 8707, the downstream server might expect a specific `resource` claim - **Issuer (`iss`) mismatch**: The downstream server might be configured to accept tokens from a different issuer - **Missing required scopes**: The token might lack OAuth2 scopes required by the downstream server #### Token Type Issues - **Wrong token type**: Downstream server might expect an access token while receiving an ID token, or vice versa - **Token exchange required**: Some enterprise setups require token exchange (RFC 8693) rather than direct forwarding - **Different JWT validation rules**: The downstream server might have stricter validation (e.g., requiring additional claims) #### Time and Key Issues - **Clock skew**: Time differences between servers can cause tokens to appear expired or not-yet-valid - **Different JWKS endpoints**: Servers might use different key sets for validation - **Key rotation timing**: Keys might have rotated between Nexus validation and downstream validation ### JWKS Refresh Issues - Ensure the JWKS URL is reachable from Nexus - Check network connectivity and firewall rules - Verify the poll_interval format is correct (e.g., "5m", "1h") ## Enterprise Authorization Considerations For enterprise deployments, consider that sophisticated authorization models may require **token exchange** (RFC 8693) rather than simple token forwarding: - **Token Exchange Flow**: Instead of forwarding the same token, exchange it for resource-specific tokens - **Resource-Specific Tokens**: Each downstream server receives a token with its URL as the audience - **Granular Scopes**: Different downstream servers can require different OAuth2 scopes - **Audit Trail**: Token exchange provides better visibility into which services are being accessed Currently, Nexus supports direct token forwarding. For environments requiring token exchange, consider placing Nexus behind an API gateway that handles the exchange. ## Security Best Practices 1. **Always enable OAuth2 in production** environments 2. **Use HTTPS** for all OAuth2 endpoints 3. **Set expected claims** (`expected_issuer`, `expected_audience`) to prevent token misuse 4. **Monitor authentication failures** in logs 5. **Rotate signing keys** periodically at your authorization server 6. **Use short token expiration times** with refresh tokens 7. **Consider token exchange** for enterprise environments with strict security requirements ## Next Steps - Configure [Rate Limiting](/docs/configuration/server/rate-limiting) to protect against abuse - Set up [Client Identification](/docs/configuration/server/client-identification) for per-user features - Enable [CORS](/docs/configuration/server/cors) for browser-based clients --- # Configuration - Server - Rate Limiting Nexus provides rate limiting capabilities to protect your server from abuse and ensure fair resource usage. You can configure global and per-IP rate limits at the server level. ## Basic Configuration Enable rate limiting in your `nexus.toml`: ```toml [server.rate_limits] enabled = true storage = "memory" # or use Redis for distributed rate limiting [server.rate_limits.global] limit = 1000 interval = "60s" [server.rate_limits.per_ip] limit = 100 interval = "60s" ``` ## Configuration Options - `enabled`: Enable or disable rate limiting (default: `false`) - `storage`: Storage backend - either `"memory"` (default) or a Redis configuration ### Rate Limit Types **Global Limits** (`server.rate_limits.global`) - `limit`: Maximum requests across all clients - `interval`: Time window (e.g., "60s", "5m", "1h") **Per-IP Limits** (`server.rate_limits.per_ip`) - `limit`: Maximum requests per IP address - `interval`: Time window ## Storage Backends ### Memory Storage (Default) Uses an in-memory rate limiter, suitable for single-instance deployments: ```toml [server.rate_limits] storage = "memory" ``` ### Redis Storage For distributed rate limiting across multiple Nexus instances: ```toml [server.rate_limits] storage = { type = "redis", url = "redis://localhost:6379" } ``` #### Redis Configuration Options - `url`: Redis connection URL (default: `"redis://localhost:6379/0"`) - `key_prefix`: Prefix for rate limit keys (default: `"nexus:rate_limits:"`) - `pool.max_size`: Maximum connection pool size (default: 16) - `pool.min_idle`: Minimum idle connections (default: 0) - `pool.timeout_create`: Timeout for creating connections (optional) - `pool.timeout_wait`: Timeout for waiting for a connection (optional) - `pool.timeout_recycle`: Timeout for recycling connections (optional) - `tls.enabled`: Enable TLS connection (default: `false`) - `tls.insecure`: Skip TLS certificate verification (optional) - `tls.ca_cert_path`: Path to CA certificate (optional) - `tls.client_cert_path`: Path to client certificate (optional) - `tls.client_key_path`: Path to client private key (optional) - `response_timeout`: Timeout for Redis responses (optional) - `connection_timeout`: Timeout for Redis connections (optional) ### Full Redis Example ```toml [server.rate_limits] storage = { type = "redis", url = "redis://username:password@redis.example.com:6379/0", key_prefix = "nexus:rate_limits:prod:", response_timeout = "10s", connection_timeout = "10s" } # With connection pool configuration [server.rate_limits.storage.pool] max_size = 20 min_idle = 5 timeout_create = "5s" timeout_wait = "5s" # With TLS configuration [server.rate_limits.storage.tls] enabled = true ca_cert_path = "/etc/ssl/certs/redis-ca.pem" ``` ## Rate Limit Response When a client exceeds the rate limit, Nexus responds with: - HTTP status code `429 Too Many Requests` - `Retry-After` header indicating when the client can retry (in seconds) ## Advanced Rate Limiting For more granular control, you can also configure: ### Per-MCP-Server Rate Limits Limit requests to specific MCP servers: ```toml [mcp.servers.expensive_api] url = "https://api.example.com/mcp" [mcp.servers.expensive_api.rate_limits] limit = 10 interval = "60s" ``` ### Per-Tool Rate Limits Limit usage of specific tools within MCP servers: ```toml [mcp.servers.my_api.rate_limits.tools] expensive_operation = { limit = 5, interval = "300s" } bulk_process = { limit = 2, interval = "600s" } ``` See [MCP rate limiting](/docs/configuration/mcp/rate-limiting) for details. ### Token-Based Rate Limits for LLMs Limit token consumption for AI models: ```toml [llm.providers.openai.rate_limits.per_user] input_token_limit = 100000 interval = "60s" ``` See [LLM rate limiting](/docs/configuration/llm/rate-limiting) for details. ## Rate Limit Hierarchy Rate limits are evaluated in the following order: **Server-level limits (checked first via middleware):** 1. Global limits - total requests across all clients 2. Per-IP limits - requests per IP address **Module-specific limits (checked after server limits pass):** For MCP requests: 1. Tool-specific limits (most specific) 2. MCP server limits (least specific) For LLM requests: 1. Model-specific token limits with user group (most specific) 2. Model-specific token limits 3. Provider-level token limits with user group 4. Provider-level token limits (least specific) All applicable limits are enforced - a request must pass all rate limit checks to succeed. Server middleware limits are checked first; if they pass, then the request proceeds to module-specific limit checks. ## Best Practices 1. **Start Conservative**: Begin with lower limits and increase based on usage patterns 2. **Monitor Usage**: Track rate limit hits to identify patterns 3. **Use Redis in Production**: For multi-instance deployments 4. **Different Intervals**: Use appropriate time windows for different resources 5. **Test Limits**: Verify configuration before production deployment 6. **Log Rate Limit Events**: Monitor for potential abuse or misconfiguration ## Monitoring and Metrics Monitor these metrics to optimize rate limiting: - Rate limit hit frequency by IP - Top IPs hitting rate limits - Rate limit effectiveness (blocked vs allowed requests) - Redis connection pool utilization (if using Redis) ## Next Steps - Enable [Client Identification](/docs/configuration/server/client-identification) for per-user rate limiting - Configure [CORS](/docs/configuration/server/cors) for browser-based clients - Set up [CSRF Protection](/docs/configuration/server/csrf) for additional security --- # Configuration - Server - Client Identification Client identification enables Nexus to recognize and differentiate between individual users or clients. This is essential for features like per-user token rate limiting, user-specific metrics, and tiered access control. ## Basic Configuration Enable client identification in your `nexus.toml`: ```toml [server.client_identification] enabled = true # Choose one method to extract client ID client_id.jwt_claim = "sub" # Extract from JWT token claim # OR client_id.http_header = "x-client-id" # Extract from HTTP header ``` ## Client ID Sources Nexus can extract the client identifier from two sources: ### JWT Claims (Recommended for Production) Extract the client ID from a JWT token claim. This requires OAuth2 to be configured: ```toml [server.client_identification] enabled = true client_id.jwt_claim = "sub" # Common claims: "sub", "email", "user_id" ``` The JWT token should be provided in the `Authorization` header: ```bash curl -H "Authorization: Bearer " http://localhost:8000/llm/v1/chat/completions ``` ### HTTP Headers (For Private Networks) Extract the client ID from a custom HTTP header. Only use this in trusted, private networks: ```toml [server.client_identification] enabled = true client_id.http_header = "x-client-id" ``` The client ID should be provided in the specified header: ```bash curl -H "X-Client-ID: user-123" http://localhost:8000/llm/v1/chat/completions ``` ## User Groups and Tiers Configure user groups to implement tiered access control: ```toml [server.client_identification] enabled = true client_id.jwt_claim = "sub" # Extract group/tier information group_id.jwt_claim = "plan" # JWT claim containing user's plan/tier # OR group_id.http_header = "x-user-tier" # HTTP header with tier info # Define allowed groups (required when group_id is configured) [server.client_identification.validation] group_values = ["free", "pro", "enterprise"] ``` ### Group Configuration Rules 1. **If you configure `group_id`, you must define `group_values` in the validation section** 2. **Groups not in `group_values` will result in a BAD REQUEST error** 3. **Users without a group use default limits** ## Integration with Rate Limiting Client identification enables per-user rate limiting, allowing you to set different token limits for each user tier. For detailed configuration, see the [LLM Rate Limiting](/docs/configuration/llm/rate-limiting) documentation. ## Complete Example Here's an example showing client identification with OAuth2 and tiered access: ```toml # OAuth2 configuration [server.oauth] url = "https://auth.example.com/.well-known/jwks.json" expected_issuer = "https://auth.example.com" expected_audience = "nexus-api" # Client identification [server.client_identification] enabled = true client_id.jwt_claim = "sub" # User ID from JWT group_id.jwt_claim = "subscription" # User's subscription tier [server.client_identification.validation] group_values = ["free", "pro", "enterprise"] ``` This configuration extracts user identity from JWT tokens and assigns them to tiers. These tiers can then be used with [rate limiting](/docs/configuration/llm/rate-limiting) and other features. ## Security Considerations ### JWT Claims (Recommended) - **Most secure option** for production environments - Requires proper OAuth2 configuration - Claims are cryptographically verified - Cannot be spoofed by clients ### HTTP Headers (Use with Caution) - **Only use in private, trusted networks** - Can be easily spoofed if exposed to public internet - Suitable for internal services behind a reverse proxy - Consider using mutual TLS for additional security ## Troubleshooting ### Client ID Not Detected - Verify the JWT token contains the specified claim - Check that the HTTP header name matches exactly (case-sensitive) - Ensure OAuth2 is properly configured if using JWT claims - Check logs for parsing errors ### Group Validation Errors - Ensure all possible group values are listed in `group_values` - Verify the group claim/header contains a valid value - Check for typos in group names - Review logs for validation details ### Rate Limits Not Applied - Confirm client identification is enabled - Verify the client ID is being extracted correctly - Check that group configuration matches rate limit groups - Ensure rate limiting storage is properly configured ## Best Practices 1. **Use JWT claims in production** for security 2. **Define all possible groups** in validation 3. **Start with conservative limits** and adjust based on usage 4. **Monitor per-user metrics** to optimize limits 5. **Document tier differences** for users 6. **Implement graceful degradation** when limits are reached 7. **Use consistent group names** across configuration ## Next Steps - Configure [LLM Token Rate Limiting](/docs/configuration/llm/rate-limiting) per user - Set up [CORS](/docs/configuration/server/cors) for browser clients - Enable [CSRF Protection](/docs/configuration/server/csrf) for additional security --- # Configuration - Server - CORS Configure Cross-Origin Resource Sharing (CORS) to allow browser-based clients to interact with your Nexus server. ## Basic Configuration ```toml [server.cors] allow_origins = ["https://app.example.com", "http://localhost:3000"] allow_methods = ["GET", "POST"] allow_headers = ["authorization", "content-type", "x-request-id"] allow_credentials = true max_age = 3600 allow_private_network = false expose_headers = ["x-request-id", "x-trace-id"] ``` ## Configuration Options - `allow_origins`: List of allowed origins, a single origin, or `"*"` for all origins - `allow_methods`: HTTP methods to allow (can be array or `"*"`) - `allow_headers`: Headers that clients can send (can be array or `"*"`) - `allow_credentials`: Whether to allow credentials in CORS requests (default: `false`) - `max_age`: How long browsers can cache CORS preflight responses (in seconds) - `allow_private_network`: Allow requests from private networks (default: `false`) - `expose_headers`: Headers to expose to the client (can be array or `"*"`) ## Special Values For `allow_origins`, `allow_methods`, `allow_headers`, and `expose_headers`, you can use: - An array of specific values: `["value1", "value2"]` - A single value: `"value"` - Wildcard to allow all: `"*"` ## Common Configurations ### Development Environment Allow all origins during development: ```toml [server.cors] allow_origins = "*" allow_methods = "*" allow_headers = "*" allow_credentials = false ``` ### Production with Specific Origins Restrict to specific domains in production: ```toml [server.cors] allow_origins = [ "https://app.yourdomain.com", "https://admin.yourdomain.com" ] allow_methods = ["GET", "POST", "OPTIONS"] allow_headers = ["authorization", "content-type"] allow_credentials = true max_age = 86400 # 24 hours expose_headers = ["x-request-id"] ``` ### Single Page Application (SPA) Configuration for a typical SPA: ```toml [server.cors] allow_origins = "https://spa.example.com" allow_methods = ["GET", "POST", "PUT", "DELETE", "OPTIONS"] allow_headers = ["authorization", "content-type", "x-csrf-token"] allow_credentials = true max_age = 3600 expose_headers = ["x-total-count", "x-page-number"] ``` ### Mobile App Development Allow local development servers: ```toml [server.cors] allow_origins = [ "http://localhost:3000", # Web dev server "http://localhost:8081", # React Native "http://10.0.2.2:8000", # Android emulator "capacitor://localhost", # Capacitor "ionic://localhost" # Ionic ] allow_methods = ["GET", "POST", "OPTIONS"] allow_headers = ["authorization", "content-type"] allow_credentials = true allow_private_network = true # For local network access ``` ## Security Considerations ### Credentials and Wildcards When `allow_credentials = true`: - **Cannot use `"*"` for origins** - must specify exact origins - Browser will reject responses if wildcard is used with credentials - Each origin must be explicitly listed ### Private Network Access The `allow_private_network` option controls Chrome's Private Network Access: ```toml [server.cors] allow_private_network = true # Allow requests from private IPs ``` Use this when: - Serving requests to local network clients - Development with local IP addresses - Internal corporate networks ## Preflight Requests Browsers send preflight OPTIONS requests for: - Custom headers - Non-simple methods (PUT, DELETE, etc.) - Requests with credentials Ensure OPTIONS is included in `allow_methods`: ```toml [server.cors] allow_methods = ["GET", "POST", "PUT", "DELETE", "OPTIONS"] ``` ## Troubleshooting CORS Issues ### Preflight Failures **Symptoms**: OPTIONS request fails with CORS error **Solution**: ```toml [server.cors] allow_methods = ["OPTIONS", "GET", "POST"] # Include OPTIONS allow_headers = ["authorization", "content-type"] # Include all headers used ``` ### Credentials Not Working **Symptoms**: Cookies/auth headers not sent **Solution**: ```toml [server.cors] allow_origins = ["https://specific-origin.com"] # Specific origin, not "*" allow_credentials = true ``` ### Headers Not Accessible **Symptoms**: Can't read response headers in JavaScript **Solution**: ```toml [server.cors] expose_headers = ["x-custom-header", "x-another-header"] ``` ### Cache Issues **Symptoms**: Old CORS settings persist **Solution**: ```toml [server.cors] max_age = 0 # Disable preflight caching during debugging ``` ## Client-Side Examples ### JavaScript Fetch API ```javascript fetch('http://localhost:8000/llm/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer token' }, credentials: 'include', // Send cookies if allow_credentials = true body: JSON.stringify({ model: 'openai/gpt-4', messages: [{ role: 'user', content: 'Hello' }] }) }); ``` ### Axios Configuration ```javascript import axios from 'axios'; const client = axios.create({ baseURL: 'http://localhost:8000', withCredentials: true, // If allow_credentials = true headers: { 'Content-Type': 'application/json' } }); ``` ## Best Practices 1. **Be Specific in Production**: Never use `"*"` for origins in production 2. **Minimize Exposed Headers**: Only expose necessary headers 3. **Use HTTPS**: Always use HTTPS origins in production 4. **Validate Origins**: Keep origin list up-to-date 5. **Monitor CORS Errors**: Log and monitor CORS failures 6. **Test Thoroughly**: Test with actual browsers, not just tools 7. **Document Origins**: Maintain documentation of allowed origins ## Next Steps - Enable [CSRF Protection](/docs/configuration/server/csrf) for additional security - Configure [OAuth2](/docs/configuration/server/oauth2) for authentication - Set up [Rate Limiting](/docs/configuration/server/rate-limiting) to prevent abuse --- # Configuration - Server - CSRF Protection Configure CSRF (Cross-Site Request Forgery) protection to prevent unauthorized actions from malicious websites. ## Basic Configuration ```toml [server.csrf] enabled = true header_name = "X-CSRF-Token" ``` ## Configuration Options - `enabled`: Whether CSRF protection is enabled (default: `false`) - `header_name`: The header name to use for CSRF tokens (default: `"X-Nexus-CSRF-Protection"`) When enabled, Nexus will require this header to be present on every request. ## How CSRF Protection Works 1. **Client obtains a CSRF token** (implementation-specific) 2. **Client includes token** in the configured header 3. **Nexus validates** the header is present 4. **Request proceeds** if validation passes ## Implementation Examples ### Simple Header Check The basic CSRF protection checks for header presence: ```bash curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-CSRF-Token: any-value" \ -d '{"model": "openai/gpt-4", "messages": [{"role": "user", "content": "Hello"}]}' ``` ### JavaScript Implementation ```javascript // Simple CSRF token implementation const csrfToken = 'your-csrf-token-here'; fetch('http://localhost:8000/llm/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'X-CSRF-Token': csrfToken // Include CSRF token }, body: JSON.stringify({ model: 'openai/gpt-4', messages: [{ role: 'user', content: 'Hello' }] }) }); ``` ### React Example ```javascript import { useState, useEffect } from 'react'; function ChatComponent() { const [csrfToken] = useState(() => { // Generate or retrieve token return crypto.randomUUID(); }); const sendMessage = async (message) => { const response = await fetch('/llm/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'X-CSRF-Token': csrfToken }, body: JSON.stringify({ model: 'openai/gpt-4', messages: [{ role: 'user', content: message }] }) }); return response.json(); }; // ... rest of component } ``` ## Integration with Other Security Features ### With OAuth2 CSRF and OAuth2 work together for defense in depth: ```toml # OAuth2 for authentication [server.oauth] url = "https://auth.example.com/.well-known/jwks.json" expected_issuer = "https://auth.example.com" # CSRF for request validation [server.csrf] enabled = true header_name = "X-CSRF-Token" ``` Client must provide both: ```javascript fetch('/llm/v1/chat/completions', { method: 'POST', headers: { 'Authorization': 'Bearer ' + jwtToken, 'X-CSRF-Token': csrfToken, 'Content-Type': 'application/json' }, body: JSON.stringify(data) }); ``` ### With CORS Configure both for browser security: ```toml [server.cors] allow_origins = ["https://app.example.com"] allow_headers = ["authorization", "content-type", "x-csrf-token"] allow_credentials = true [server.csrf] enabled = true header_name = "X-CSRF-Token" ``` ## Custom Header Names Use custom headers for compatibility: ```toml # Django-style [server.csrf] enabled = true header_name = "X-CSRFToken" # Rails-style [server.csrf] enabled = true header_name = "X-CSRF-Token" # Custom [server.csrf] enabled = true header_name = "X-Request-Token" ``` ## When to Use CSRF Protection ### Enable CSRF When: - Serving browser-based clients - Handling sensitive operations - Compliance requirements mandate it ### CSRF May Not Be Needed When: - API is consumed only by non-browser clients - Using only OAuth2 bearer tokens - No state-changing operations - Internal APIs with no browser access ## Error Response When CSRF validation fails: ```http HTTP/1.1 403 Forbidden ``` ## Troubleshooting ### CSRF Validation Failures - Verify header name matches configuration exactly - Check that header is included in CORS `allow_headers` - Ensure header value is not empty - Confirm header is sent with request ### Integration Issues - Test with curl to isolate browser issues - Check browser developer tools for header presence - Verify CORS configuration allows the CSRF header - Review server logs for specific error details ## Next Steps - Configure [OAuth2 Authentication](/docs/configuration/server/oauth2) for secure access - Set up [CORS](/docs/configuration/server/cors) for browser clients - Enable [Rate Limiting](/docs/configuration/server/rate-limiting) to prevent abuse --- # Configuration - LLM The LLM router acts as a unified gateway that provides access to multiple AI model providers through a single OpenAI-compatible API. Configure providers, models, rate limits, and more. ## Configuration Topics ### Essential Configuration 1. **[Provider Configuration](/docs/configuration/llm/providers)** - Set up OpenAI, Anthropic, Google, and AWS Bedrock 2. **[Model Management](/docs/configuration/llm/models)** - Configure models and create aliases 3. **[Token Rate Limiting](/docs/configuration/llm/rate-limiting)** - Control token consumption per user and model ### Advanced Features 4. **[Token Forwarding](/docs/configuration/llm/token-forwarding)** - Allow users to provide their own API keys 5. **[Header Rules](/docs/configuration/llm/header-rules)** - Transform and manage HTTP headers for providers ## Quick Start ### Basic Configuration Enable LLM routing in your `nexus.toml`: ```toml [llm] enabled = true # Enable LLM functionality # Configure protocol endpoints [llm.protocols.openai] enabled = true path = "/llm/openai" # OpenAI-compatible endpoint [llm.protocols.anthropic] enabled = true path = "/llm/anthropic" # Anthropic-compatible endpoint # Configure OpenAI provider [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Must explicitly configure models [llm.providers.openai.models.gpt-4] [llm.providers.openai.models."gpt-3.5-turbo"] # Configure Anthropic provider [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" [llm.providers.anthropic.models."claude-3-5-sonnet-20241022"] ``` ### Using the API ```bash # List available models (OpenAI protocol) curl http://localhost:8000/llm/openai/v1/models # Chat completion (OpenAI protocol) curl -X POST http://localhost:8000/llm/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4", "messages": [{"role": "user", "content": "Hello!"}] }' # Chat completion (Anthropic protocol) curl -X POST http://localhost:8000/llm/anthropic/v1/messages \ -H "Content-Type: application/json" \ -H "x-api-key: not-used" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "anthropic/claude-3-5-sonnet-20241022", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 1024 }' ``` ## Model Naming Convention Models are prefixed with their provider name: - Format: `{provider_name}/{model_id}` - Examples: `openai/gpt-4`, `anthropic/claude-3-5-sonnet-20241022` **Note**: All models must be explicitly configured. Models that are not configured will return a 404 error. ## Complete Example Here's a comprehensive configuration showing multiple providers and features: ```toml [llm] enabled = true # Enable both protocols [llm.protocols.openai] enabled = true path = "/llm/openai" [llm.protocols.anthropic] enabled = true path = "/llm/anthropic" # OpenAI with multiple models [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" forward_token = true # Allow user-provided keys [llm.providers.openai.models.gpt-4] [llm.providers.openai.models."gpt-3.5-turbo"] [llm.providers.openai.models.smart] rename = "gpt-4" # Alias: "openai/smart" → "gpt-4" # Token rate limiting for OpenAI [llm.providers.openai.rate_limits.per_user] input_token_limit = 100000 interval = "60s" # Anthropic configuration [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" [llm.providers.anthropic.models."claude-3-5-sonnet-20241022"] [llm.providers.anthropic.models.fast] rename = "claude-3-haiku-20240307-v1:0" # AWS Bedrock [llm.providers.bedrock] type = "bedrock" region = "us-east-1" [llm.providers.bedrock.models.claude] rename = "anthropic.claude-3-sonnet-20240229-v1:0" ``` ## Client Library Support The LLM router supports both OpenAI and Anthropic client libraries: ### OpenAI Clients #### Python ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used" ) ``` #### JavaScript ```javascript import OpenAI from 'openai'; const openai = new OpenAI({ baseURL: 'http://localhost:8000/llm/openai/v1', apiKey: 'not-used' }); ``` ### Anthropic Clients #### Python ```python from anthropic import Anthropic client = Anthropic( base_url="http://localhost:8000/llm/anthropic", api_key="not-used" ) ``` #### JavaScript ```javascript import Anthropic from '@anthropic-ai/sdk'; const anthropic = new Anthropic({ baseURL: 'http://localhost:8000/llm/anthropic', apiKey: 'not-used' }); ``` ### Claude Code ```bash export ANTHROPIC_BASE_URL="http://localhost:8000/llm/anthropic" export ANTHROPIC_MODEL="anthropic/claude-3-5-sonnet-20241022" ``` ## Key Features - **Multi-Protocol Support**: Native OpenAI and Anthropic protocol endpoints - **Unified Access**: Route to any provider through either protocol - **Client Compatibility**: Works with OpenAI, Anthropic, and Claude Code clients - **Model Aliases**: Create custom names for models - **Token Rate Limiting**: Control usage per user and model - **Token Forwarding**: Users can provide their own API keys - **Header Transformation**: Forward, insert, remove, and rename HTTP headers - **Streaming Support**: Real-time responses via SSE - **Multiple Providers**: Mix models from different vendors ## Best Practices 1. **Explicit Model Configuration**: Only configure models you need 2. **Use Environment Variables**: Never hardcode API keys 3. **Configure Rate Limits**: Protect against excessive usage 4. **Create Meaningful Aliases**: Simplify model names for users 5. **Monitor Usage**: Track token consumption and costs 6. **Test Thoroughly**: Verify models before production ## Troubleshooting For debugging, check: - Model availability (OpenAI): `GET /llm/openai/v1/models` - Model availability (Anthropic): `GET /llm/anthropic/v1/models` - Nexus logs: `nexus --log debug` - Provider authentication - Rate limit configuration - Network connectivity to providers ## Next Steps - Start with [Provider Configuration](/docs/configuration/llm/providers) - Set up [Model Management](/docs/configuration/llm/models) - Configure [Token Rate Limiting](/docs/configuration/llm/rate-limiting) - Learn how to [use the API](/docs/usage/llm-api) - Integrate with [Claude Code](/docs/usage/claude-code-integration) --- # Configuration - LLM - Provider Configuration Configure multiple AI model providers to access them through a single OpenAI-compatible API. Nexus supports OpenAI, Anthropic, Google, and AWS Bedrock providers. ## OpenAI Provider OpenAI-compatible provider supporting GPT models and function calling. ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" base_url = "https://api.openai.com/v1" # Optional - custom endpoint # Models must be explicitly configured [llm.providers.openai.models.gpt-4] # Model is available as "openai/gpt-4" and maps to upstream "gpt-4" [llm.providers.openai.models."gpt-3.5-turbo"] # Model is available as "openai/gpt-3.5-turbo" and maps to upstream "gpt-3.5-turbo" ``` ### OpenAI Features - **Function Calling**: Full support for tools and function calls - **Native Compatibility**: Direct API pass-through for maximum compatibility ## Anthropic Provider Anthropic Claude models with automatic message format conversion. ```toml [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" base_url = "https://api.anthropic.com/v1" # Optional - custom endpoint # Models must be explicitly configured [llm.providers.anthropic.models."claude-3-5-sonnet-20241022"] # Model is available as "anthropic/claude-3-5-sonnet-20241022" [llm.providers.anthropic.models."claude-3-opus-20240229"] # Model is available as "anthropic/claude-3-opus-20240229" ``` ## Google Provider Google Gemini models with role mapping and system instruction support. ```toml [llm.providers.google] type = "google" api_key = "{{ env.GOOGLE_API_KEY }}" base_url = "https://generativelanguage.googleapis.com/v1beta" # Optional # Models must be explicitly configured [llm.providers.google.models."gemini-1.5-pro"] # Model is available as "google/gemini-1.5-pro" [llm.providers.google.models."gemini-1.5-flash"] # Note: Model names with dots must be quoted ``` ## AWS Bedrock Provider AWS Bedrock provides access to foundation models from multiple vendors including Anthropic, Amazon, Meta, Mistral, Cohere, and DeepSeek. Nexus uses AWS Bedrock's unified Converse API for consistent interaction across all Bedrock models. ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" # AWS region (required) profile = "production" # Optional AWS profile name # Alternative authentication methods: # access_key_id = "{{ env.AWS_ACCESS_KEY_ID }}" # secret_access_key = "{{ env.AWS_SECRET_ACCESS_KEY }}" # Models must be explicitly configured [llm.providers.bedrock.models."anthropic.claude-3-sonnet-20240229-v1:0"] # Available as "bedrock/anthropic.claude-3-sonnet-20240229-v1:0" [llm.providers.bedrock.models."anthropic.claude-3-haiku-20240307-v1:0"] # Available as "bedrock/anthropic.claude-3-haiku-20240307-v1:0" # You can also create aliases for easier use [llm.providers.bedrock.models.fast] rename = "anthropic.claude-3-haiku-20240307-v1:0" # Available as "bedrock/fast" which maps to Claude 3 Haiku ``` ### AWS Authentication for Bedrock Bedrock supports multiple authentication methods: 1. **AWS Profile** (recommended for local development): ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" profile = "production" # Uses AWS credentials from ~/.aws/credentials ``` 2. **Explicit Credentials** (for CI/CD or containers): ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" access_key_id = "{{ env.AWS_ACCESS_KEY_ID }}" secret_access_key = "{{ env.AWS_SECRET_ACCESS_KEY }}" # session_token = "{{ env.AWS_SESSION_TOKEN }}" # Optional for temporary credentials ``` 3. **IAM Role** (for EC2 instances or ECS tasks): ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" # No credentials needed - uses instance/task IAM role ``` ### Supported Bedrock Models AWS Bedrock supports models from Anthropic, Amazon, Meta, Mistral, Cohere, and others. See the [Model Management documentation](/docs/configuration/llm/models#aws-bedrock-models) for configuration examples and the complete list. ## Header Rules Configure custom header transformation rules for provider requests. This allows you to forward headers from incoming requests, add static headers, remove sensitive headers, or rename headers for compatibility. ### Header Configuration Headers are configured as an array of rules, where each rule specifies a transformation type and its parameters: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Forward specific header from incoming requests [[llm.providers.openai.headers]] rule = "forward" name = "x-user-id" # Forward headers matching a pattern [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-custom-" # Forward all headers starting with "x-custom-" # Add static headers to all requests [[llm.providers.openai.headers]] rule = "insert" name = "x-api-version" value = "2024-01" [[llm.providers.openai.headers]] rule = "insert" name = "x-client-id" value = "{{ env.CLIENT_ID }}" # Support for environment variables # Remove headers before sending to provider [[llm.providers.openai.headers]] rule = "remove" name = "x-internal-token" # Remove headers matching a pattern [[llm.providers.openai.headers]] rule = "remove" pattern = "^x-debug-" # Remove all debug headers # Rename and duplicate headers (preserves original) [[llm.providers.openai.headers]] rule = "rename_duplicate" name = "x-custom-org" rename = "x-organization" ``` ### Header Rule Types #### Forward Rule Forwards headers from incoming requests to the provider: ```toml [[llm.providers.anthropic.headers]] rule = "forward" name = "x-request-id" # Forward specific header # With rename and default value [[llm.providers.anthropic.headers]] rule = "forward" name = "x-trace-id" rename = "anthropic-trace-id" # Optional: rename the header default = "{{ env.DEFAULT_TRACE_ID }}" # Optional: default if not present ``` #### Insert Rule Adds static headers to all provider requests: ```toml [[llm.providers.google.headers]] rule = "insert" name = "x-goog-api-version" value = "v1beta" ``` #### Remove Rule Removes headers before sending to the provider: ```toml [[llm.providers.openai.headers]] rule = "remove" name = "cookie" # Remove specific header [[llm.providers.openai.headers]] rule = "remove" pattern = "^x-internal-" # Remove all internal headers ``` #### Rename Duplicate Rule Duplicates a header with a new name while preserving the original header: ```toml [[llm.providers.anthropic.headers]] rule = "rename_duplicate" name = "authorization" rename = "x-original-auth" default = "Bearer {{ env.DEFAULT_TOKEN }}" # Optional: used if original header doesn't exist # This results in both headers being present: # - authorization: # - x-original-auth: ``` ### Pattern Matching Use regex patterns to match multiple headers: ```toml # Forward all headers starting with "x-org-" [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-org-" # Remove all temporary headers [[llm.providers.openai.headers]] rule = "remove" pattern = "^x-temp-" ``` ### Security Considerations Nexus automatically protects sensitive headers by default. The following headers are never forwarded unless explicitly configured: - `authorization` - `x-api-key` - `api-key` - `cookie` - `set-cookie` To forward sensitive headers, you must explicitly include them in a forward rule: ```toml # Explicitly forward authorization header (use with caution) [[llm.providers.custom.headers]] rule = "forward" name = "authorization" ``` ### Processing Order Headers start with an empty set and are processed sequentially in the order they are defined in your configuration. Both `forward` and `insert` rules will override any existing headers with the same name. For example: ```toml # 1. Insert a static header first [[llm.providers.openai.headers]] rule = "insert" name = "x-api-version" value = "v1" # 2. Forward headers matching a pattern (will override existing ones!) [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-custom-" # If client sends x-custom-override=user-value, it replaces any existing value # 3. Remove a specific header that may have been forwarded [[llm.providers.openai.headers]] rule = "remove" name = "x-custom-internal" # 4. Insert to ensure final value (overrides anything forwarded) [[llm.providers.openai.headers]] rule = "insert" name = "x-custom-override" value = "final-value" ``` This sequential processing means you can: - Set defaults with insert, then let forward override them if provided - Forward headers with patterns, then remove specific unwanted ones - Ensure critical headers have correct values by inserting after forward - Build complex header transformation pipelines with precise control ### Per-Provider Examples #### OpenAI with Custom Headers ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Forward user context [[llm.providers.openai.headers]] rule = "forward" name = "x-user-id" # Add OpenAI-specific beta features [[llm.providers.openai.headers]] rule = "insert" name = "OpenAI-Beta" value = "assistants=v2" ``` #### Anthropic with Organization Headers ```toml [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" # Forward and rename organization header [[llm.providers.anthropic.headers]] rule = "forward" name = "x-org-id" rename = "anthropic-org-id" ``` #### Google with Project Headers ```toml [llm.providers.google] type = "google" api_key = "{{ env.GOOGLE_API_KEY }}" # Add Google Cloud project headers [[llm.providers.google.headers]] rule = "insert" name = "x-goog-user-project" value = "{{ env.GCP_PROJECT_ID }}" ``` ## Provider Configuration Options ### Standard Providers (OpenAI, Anthropic, Google) #### Required Settings - `type`: Provider type - must be `"openai"`, `"anthropic"`, or `"google"` (required) - `api_key`: API key for the provider (required) - `models`: At least one model must be configured (required) #### Optional Settings - `base_url`: Custom API endpoint URL (optional) - Useful for Azure OpenAI, local deployments, or proxy servers - Defaults to the provider's standard endpoint if not specified - `forward_token`: Enable token forwarding (see [Token Forwarding](/docs/configuration/llm/token-forwarding)) - `headers`: Array of header transformation rules (optional) - Each rule must specify a `rule` type: `forward`, `insert`, `remove`, or `rename_duplicate` - Use `name` for specific headers or `pattern` for regex matching - Additional fields depend on the rule type ### AWS Bedrock Provider #### Required Settings - `type`: Must be `"bedrock"` (required) - `region`: AWS region where Bedrock is available (required) - `models`: At least one model must be configured (required) #### Optional Settings - `access_key_id`: AWS Access Key ID for authentication - `secret_access_key`: AWS Secret Access Key - `session_token`: AWS Session Token for temporary credentials - `profile`: AWS profile name from `~/.aws/credentials` - `base_url`: Custom endpoint URL for VPC endpoints **Note**: Token forwarding is not supported for AWS Bedrock due to AWS SigV4 authentication requirements. ## Multiple Provider Instances You can configure multiple instances of the same provider type with different names: ```toml [llm.providers.openai_standard] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" [llm.providers.openai_standard.models.gpt-4] # Available as "openai_standard/gpt-4" [llm.providers.azure_openai] type = "openai" api_key = "{{ env.AZURE_OPENAI_API_KEY }}" base_url = "https://your-resource.openai.azure.com/openai/deployments/gpt-4" [llm.providers.azure_openai.models.gpt-4] # Available as "azure_openai/gpt-4" [llm.providers.claude_work] type = "anthropic" api_key = "{{ env.ANTHROPIC_WORK_KEY }}" [llm.providers.claude_work.models."claude-3-5-sonnet-20241022"] # Available as "claude_work/claude-3-5-sonnet-20241022" ``` ## Environment Variables Use `{{ env.VARIABLE_NAME }}` for environment variable substitution: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" ``` ## Best Practices 1. **Explicit Model Configuration**: Only configure models you actually need 2. **Use Environment Variables**: Never hardcode API keys 3. **Custom Provider Names**: Use descriptive names for multiple instances 4. **Test Configuration**: Verify models are available using `/llm/openai/v1/models` 5. **Monitor Usage**: Track which models are being used most 6. **Rotate Keys**: Regularly rotate API keys for security ## Next Steps - Configure [Model Management](/docs/configuration/llm/models) for aliases and custom names - Set up [Token Rate Limiting](/docs/configuration/llm/rate-limiting) for usage control - Enable [Token Forwarding](/docs/configuration/llm/token-forwarding) for user-provided keys --- # Configuration - LLM - Model Management Control which models are exposed through your Nexus instance and create custom aliases for better usability. ## Model Configuration ### Basic Model Configuration Models must be explicitly configured for each provider. The model configuration uses the following format: ```toml [llm.providers..models.] # Model will be available as "/" # By default, is also used as the upstream model name ``` Example: ```toml [llm.providers.openai.models.gpt-4] # Available as "openai/gpt-4", maps to upstream "gpt-4" [llm.providers.openai.models."gpt-3.5-turbo"] # Model names with special characters must be quoted ``` ## Model Renaming (Aliasing) Create custom model names that map to different upstream model identifiers: ```toml [llm.providers.openai.models.smart] rename = "gpt-4" # API: "openai/smart" → Upstream: "gpt-4" [llm.providers.anthropic.models.fast] rename = "claude-3-5-sonnet-20241022" # API: "anthropic/fast" → Upstream: "claude-3-5-sonnet-20241022" [llm.providers.google.models."my-gemini"] rename = "gemini-1.5-pro" # API: "google/my-gemini" → Upstream: "gemini-1.5-pro" ``` ### Use Cases for Model Aliases 1. **User-Friendly Names**: Simplify complex model identifiers ```toml [llm.providers.bedrock.models.claude] rename = "anthropic.claude-3-sonnet-20240229-v1:0" # Use "bedrock/claude" instead of the long model ID ``` 2. **Abstraction Layer**: Hide provider-specific naming ```toml [llm.providers.openai.models.chat] rename = "gpt-4" [llm.providers.anthropic.models.chat] rename = "claude-3-5-sonnet-20241022" # Both available as "*/chat" for consistency ``` 3. **Version Management**: Switch models without changing client code ```toml # Easy to update when new versions are released [llm.providers.openai.models.latest] rename = "gpt-4-turbo-2024-11-01" # Update this when newer versions available ``` ## Model Naming Convention Models are prefixed with their provider instance name: - **Format**: `{provider_name}/{model_id}` - **Examples**: - `openai/gpt-4` - `anthropic/claude-3-5-sonnet-20241022` - `google/gemini-1.5-pro` - `azure_openai/gpt-4` (custom provider name) ## Listing Available Models ### API Endpoint ```http GET /llm/openai/v1/models ``` Returns all **explicitly configured** models: ```json { "object": "list", "data": [ { "id": "openai/gpt-4", "object": "model", "owned_by": "openai" }, { "id": "openai/smart", // Custom alias "object": "model", "owned_by": "openai" }, { "id": "anthropic/fast", // Custom alias "object": "model", "owned_by": "anthropic" } ] } ``` ### Command Line ```bash curl http://localhost:8000/llm/openai/v1/models | jq '.data[].id' ``` ## Model Organization Strategies ### By Capability Organize models by their capabilities: ```toml # Chat models [llm.providers.openai.models."gpt-4"] [llm.providers.openai.models."gpt-3.5-turbo"] # Code models [llm.providers.anthropic.models.coder] rename = "claude-3-5-sonnet-20241022" # Best for coding tasks ``` ### By Cost/Performance Create tiers based on cost and performance: ```toml # Economy tier [llm.providers.openai.models.economy] rename = "gpt-3.5-turbo" [llm.providers.anthropic.models.economy] rename = "claude-3-haiku-20240307-v1:0" # Standard tier [llm.providers.openai.models.standard] rename = "gpt-4" [llm.providers.anthropic.models.standard] rename = "claude-3-sonnet-20240229-v1:0" # Premium tier [llm.providers.openai.models.premium] rename = "gpt-4-turbo-preview" [llm.providers.anthropic.models.premium] rename = "claude-3-opus-20240229" ``` ### By Use Case Configure models for specific use cases: ```toml # Customer support [llm.providers.openai.models.support] rename = "gpt-3.5-turbo" # Fast and cost-effective # Content generation [llm.providers.anthropic.models.writer] rename = "claude-3-5-sonnet-20241022" # Excellent writing capabilities # Code review [llm.providers.openai.models.reviewer] rename = "gpt-4" # Strong reasoning for code analysis # Translation [llm.providers.google.models.translator] rename = "gemini-1.5-pro" # Good multilingual support ``` ## Model Access Control Control which models are available based on configuration: ### Selective Model Exposure Only expose specific models: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Only expose GPT-4, not GPT-3.5 [llm.providers.openai.models.gpt-4] # GPT-3.5-turbo is NOT configured, so it's not accessible ``` ### Environment-Specific Models Different models for different environments: ```toml # Development environment [llm.providers.openai.models.dev] rename = "gpt-3.5-turbo" # Cheaper for development # Production environment (use environment variable) [llm.providers.openai.models.prod] rename = "{{ env.PRODUCTION_MODEL }}" # Set to "gpt-4" in production ``` ## Error Handling ### Model Not Found When a client requests an unconfigured model: ```json { "error": { "message": "Model 'openai/gpt-5' not found", "type": "invalid_request_error", "code": "model_not_found" } } ``` ### Configuration Errors Common issues and solutions: 1. **Model name with dots**: Must be quoted ```toml # Wrong [llm.providers.google.models.gemini-1.5-pro] # Correct [llm.providers.google.models."gemini-1.5-pro"] ``` 2. **Duplicate aliases**: Each model name must be unique within a provider ```toml # Wrong - duplicate "smart" name [llm.providers.openai.models.smart] rename = "gpt-4" [llm.providers.openai.models.smart] # Error! rename = "gpt-3.5-turbo" ``` ## Best Practices 1. **Start Small**: Only configure models you actually use 2. **Consistent Naming**: Use a clear naming convention across providers 3. **Document Aliases**: Keep documentation of what each alias maps to 4. **Version in Aliases**: Include version info when relevant 5. **Test Models**: Verify models work before production deployment 6. **Monitor Usage**: Track which models are used most frequently ## Migration Tips When migrating from automatic model discovery: 1. **List Current Usage**: Check logs to see which models are actually used 2. **Add Configurations**: Start with the most-used models 3. **Test Thoroughly**: Verify all client applications still work 4. **Gradual Rollout**: Consider using multiple Nexus instances during migration 5. **Update Documentation**: Ensure all model references are updated ## AWS Bedrock Models AWS Bedrock provides access to foundation models from multiple vendors. Here are the most commonly used models: ### Anthropic Claude Models ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" # Claude Opus 4.1 - Most capable (latest) [llm.providers.bedrock.models."anthropic.claude-opus-4-1-20250805-v1:0"] # Claude Sonnet 3.7 - Balanced performance (latest) [llm.providers.bedrock.models."anthropic.claude-3-7-sonnet-20250219-v1:0"] # Claude Haiku 3.5 - Fast and efficient [llm.providers.bedrock.models."anthropic.claude-3-5-haiku-20241022-v1:0"] # Create aliases for easier use [llm.providers.bedrock.models.claude-opus] rename = "anthropic.claude-opus-4-1-20250805-v1:0" [llm.providers.bedrock.models.claude-sonnet] rename = "anthropic.claude-3-7-sonnet-20250219-v1:0" [llm.providers.bedrock.models.claude-haiku] rename = "anthropic.claude-3-5-haiku-20241022-v1:0" ``` **Tool Support**: All Claude models (Opus, Sonnet, and Haiku) via Bedrock have excellent support for function calling and tools. ### Amazon Nova Models ```toml # Nova Pro - Advanced reasoning [llm.providers.bedrock.models."amazon.nova-pro-v1:0"] # Nova Lite - Efficient performance [llm.providers.bedrock.models."amazon.nova-lite-v1:0"] # Nova Micro - Ultra-fast responses [llm.providers.bedrock.models."amazon.nova-micro-v1:0"] ``` ### Meta Llama Models ```toml # Llama 3.1 405B - Largest and most capable [llm.providers.bedrock.models."meta.llama3-1-405b-instruct-v1:0"] # Llama 3.1 70B - High performance [llm.providers.bedrock.models."meta.llama3-1-70b-instruct-v1:0"] # Llama 3.1 8B - Efficient [llm.providers.bedrock.models."meta.llama3-1-8b-instruct-v1:0"] ``` ### Other Popular Models ```toml # Mistral Large [llm.providers.bedrock.models."mistral.mistral-large-2402-v1:0"] # Cohere Command R+ [llm.providers.bedrock.models."cohere.command-r-plus-v1:0"] # DeepSeek R1 - Reasoning optimized [llm.providers.bedrock.models."deepseek.deepseek-r1"] ``` ### Bedrock Configuration Example ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" # Semantic aliases for different use cases [llm.providers.bedrock.models.chat] rename = "anthropic.claude-3-sonnet-20240229-v1:0" [llm.providers.bedrock.models.fast] rename = "anthropic.claude-3-haiku-20240307-v1:0" [llm.providers.bedrock.models.powerful] rename = "anthropic.claude-3-opus-20240229-v1:0" [llm.providers.bedrock.models.coding] rename = "meta.llama3-1-70b-instruct-v1:0" ``` For a complete list of available Bedrock models and their full IDs, refer to the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). ## Next Steps - Configure [Token Rate Limiting](/docs/configuration/llm/rate-limiting) per model - Set up [Token Forwarding](/docs/configuration/llm/token-forwarding) for user-provided keys - Learn how to [use the API](/docs/usage/llm-api) --- # Configuration - LLM - Token Rate Limiting Nexus provides token-based rate limiting for LLM endpoints, allowing you to control token consumption per user, per provider, and per model, with support for different user tiers. ## Prerequisites Token rate limiting requires client identification to be configured. See [Client Identification](/docs/configuration/server/client-identification) for setup instructions. ## Basic Token Rate Limiting Configure token limits at the provider level to apply to all models from that provider: ```toml [llm.providers.openai.rate_limits.per_user] input_token_limit = 100000 # 100K input tokens interval = "60s" # Per minute ``` ## Token Counting Mechanism Nexus counts tokens using the following approach: 1. **Input Token Counting**: - Uses OpenAI's `cl100k_base` tokenizer (compatible with GPT-4/GPT-3.5) - Counts tokens for each message's role and content - Adds ~3 tokens per message for internal structure - Adds 3 tokens for assistant response initialization 2. **Pre-flight Check**: - Input tokens are checked against the limit - Request is rejected with 429 status if limit would be exceeded - Uses sliding window algorithm for rate limiting ## Model-Specific Rate Limits Configure different limits for specific models (overrides provider-level limits): ```toml # Provider-level default for all OpenAI models [llm.providers.openai.rate_limits.per_user] input_token_limit = 100000 # 100K input tokens interval = "60s" # Model-specific limit for GPT-4 (more restrictive) [llm.providers.openai.rate_limits."gpt-4".per_user] input_token_limit = 10000 # Only 10K input tokens for GPT-4 interval = "60s" # Model-specific limit for GPT-3.5-turbo (less restrictive) [llm.providers.openai.rate_limits."gpt-3.5-turbo".per_user] input_token_limit = 500000 # 500K input tokens for GPT-3.5 interval = "60s" ``` ## Group-Based Rate Limits (Tiered Access) Configure different token limits for user groups/tiers: ```toml # Configure client identification with groups [server.client_identification] enabled = true client_id.jwt_claim = "sub" group_id.jwt_claim = "plan" [server.client_identification.validation] group_values = ["free", "pro", "enterprise"] # Default limits for users without a group [llm.providers.openai.rate_limits.per_user] input_token_limit = 10000 # 10K input tokens default interval = "60s" # Group-specific limits [llm.providers.openai.rate_limits.per_user.groups.free] input_token_limit = 10000 # 10K input tokens for free tier interval = "60s" [llm.providers.openai.rate_limits.per_user.groups.pro] input_token_limit = 100000 # 100K input tokens for pro tier interval = "60s" [llm.providers.openai.rate_limits.per_user.groups.enterprise] input_token_limit = 1000000 # 1M input tokens for enterprise tier interval = "60s" ``` ## Configuration Hierarchy Rate limits are evaluated in the following order (most to least specific): 1. **Model + Group**: `[llm.providers..rate_limits..per_user.groups.]` 2. **Model**: `[llm.providers..rate_limits..per_user]` 3. **Provider + Group**: `[llm.providers..rate_limits.per_user.groups.]` 4. **Provider**: `[llm.providers..rate_limits.per_user]` The first matching configuration is used. ## Complete Example Here's a comprehensive example showing all rate limiting features: ```toml # OpenAI provider with tiered rate limits [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Configure models [llm.providers.openai.models."gpt-4"] [llm.providers.openai.models."gpt-3.5-turbo"] # Default rate limits for all OpenAI models [llm.providers.openai.rate_limits.per_user] input_token_limit = 50000 # 50K input tokens interval = "60s" # Free tier limits for all OpenAI models [llm.providers.openai.rate_limits.per_user.groups.free] input_token_limit = 10000 # 10K input tokens interval = "60s" # Pro tier limits for all OpenAI models [llm.providers.openai.rate_limits.per_user.groups.pro] input_token_limit = 100000 # 100K input tokens interval = "60s" # Enterprise tier limits for all OpenAI models [llm.providers.openai.rate_limits.per_user.groups.enterprise] input_token_limit = 1000000 # 1M input tokens interval = "60s" # GPT-4 specific limits (more restrictive) [llm.providers.openai.rate_limits."gpt-4".per_user] input_token_limit = 25000 # 25K input tokens interval = "60s" # GPT-4 enterprise tier gets special treatment [llm.providers.openai.rate_limits."gpt-4".per_user.groups.enterprise] input_token_limit = 500000 # 500K input tokens interval = "60s" # Anthropic provider with simpler limits [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" [llm.providers.anthropic.models."claude-3-5-sonnet-20241022"] [llm.providers.anthropic.models."claude-3-opus-20240229"] # Single limit for all Anthropic models and users [llm.providers.anthropic.rate_limits.per_user] input_token_limit = 75000 # 75K input tokens interval = "60s" ``` ## Storage Backends Token rate limits use the same storage backend as configured for server rate limits: ### Memory Storage (Default) ```toml [server.rate_limits] storage = "memory" ``` ### Redis Storage (Recommended for Production) ```toml [server.rate_limits] storage = { type = "redis", url = "redis://localhost:6379" } ``` See [Storage Backends](/docs/configuration/server/rate-limiting#storage-backends) for detailed configuration. ## Implementation Details ### Sliding Window Algorithm Nexus uses a sliding window algorithm for token counting: - Provides smooth rate limiting without hard resets - Tokens are "returned" to the limit pool as time passes - More accurate than fixed window counting ### Token Counting Nexus counts only input tokens for rate limiting: - Only the tokens from the request messages are counted - Output tokens are not counted against the rate limit - This provides more predictable rate limiting behavior ## Practical Examples ### SaaS Application with Tiers ```toml # Client identification from JWT [server.client_identification] enabled = true client_id.jwt_claim = "user_id" group_id.jwt_claim = "subscription_tier" [server.client_identification.validation] group_values = ["trial", "basic", "professional", "unlimited"] # OpenAI configuration [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" [llm.providers.openai.models."gpt-3.5-turbo"] [llm.providers.openai.models."gpt-4"] # Trial users - very limited [llm.providers.openai.rate_limits.per_user.groups.trial] input_token_limit = 5000 interval = "1d" # Basic tier - reasonable limits [llm.providers.openai.rate_limits.per_user.groups.basic] input_token_limit = 50000 interval = "1h" # Professional tier - generous limits [llm.providers.openai.rate_limits.per_user.groups.professional] input_token_limit = 500000 interval = "1h" # Unlimited tier - no provider-level limits (still subject to server limits) # No configuration means no token limits applied ``` ### Internal Tool with Department Limits ```toml # Identify by employee ID and department [server.client_identification] enabled = true client_id.jwt_claim = "employee_id" group_id.jwt_claim = "department" [server.client_identification.validation] group_values = ["engineering", "marketing", "sales", "executive"] # Different limits per department [llm.providers.openai.rate_limits.per_user.groups.engineering] input_token_limit = 1000000 # Engineers need more for coding interval = "3600s" [llm.providers.openai.rate_limits.per_user.groups.marketing] input_token_limit = 200000 # Content generation interval = "3600s" [llm.providers.openai.rate_limits.per_user.groups.sales] input_token_limit = 100000 # Email assistance interval = "3600s" [llm.providers.openai.rate_limits.per_user.groups.executive] input_token_limit = 500000 # Reports and analysis interval = "3600s" ``` ## Best Practices 1. **Start Conservative**: Begin with lower limits and increase based on usage patterns 2. **Monitor Usage**: Track actual input token consumption patterns 3. **Use Groups**: Implement tiered access for different user types 4. **Model-Specific Limits**: Set stricter limits for expensive models (GPT-4, Claude Opus) 5. **Client Identification**: Use JWT claims for secure user identification in production 6. **Redis for Production**: Use Redis storage for multi-instance deployments 7. **Grace Periods**: Consider longer intervals for better user experience 8. **Clear Communication**: Inform users about their limits and usage ## Interaction with Other Rate Limits Token rate limits work alongside other rate limiting mechanisms: ```toml # IP-based rate limits (always active) [server.rate_limits.per_ip] limit = 1000 interval = "60s" # Token-based rate limits (requires client identification) [llm.providers.openai.rate_limits.per_user] input_token_limit = 100000 # 100K input tokens interval = "60s" ``` Both limits are enforced independently - a request must pass all applicable rate limit checks. ## Troubleshooting ### Rate Limits Not Applied - Verify client identification is enabled and working - Check that the user's group matches configured groups - Ensure the model name in rate limits matches exactly - Review logs for rate limiting decisions ### Unexpected Token Counts - Remember that role names and message structure add tokens - System messages count toward the token limit - Token counts are estimates and may vary slightly ### Redis Connection Issues - Verify Redis is running and accessible - Check connection string and credentials - Monitor Redis memory usage - Review connection pool settings ## Next Steps - Enable [Token Forwarding](/docs/configuration/llm/token-forwarding) for user-provided keys - Review [API Usage](/docs/usage/llm-api) for integration examples - Monitor metrics to optimize limits --- # Configuration - LLM - Token Forwarding Nexus supports token forwarding, allowing users to provide their own API keys at request time instead of using the configured keys. This feature enables flexible billing models and user-managed API access. ## Overview Token forwarding allows: - Users to bring their own API keys - Separate billing per user - Development with personal keys - Fallback to configured keys when needed ## Configuring Token Forwarding Enable token forwarding for any provider by setting `forward_token = true`: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Fallback key (optional with forwarding) forward_token = true # Enable token forwarding [llm.providers.anthropic] type = "anthropic" # No api_key required when token forwarding is enabled forward_token = true [llm.providers.google] type = "google" api_key = "{{ env.GOOGLE_API_KEY }}" forward_token = false # Explicitly disabled (default) ``` ## Using Token Forwarding When token forwarding is enabled, users pass their API key using the `X-Provider-API-Key` header: ### OpenAI Example ```bash curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Provider-API-Key: sk-your-openai-key" \ -d '{ "model": "openai/gpt-4", "messages": [{"role": "user", "content": "Hello"}] }' ``` ### Anthropic Example ```bash curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Provider-API-Key: sk-ant-your-anthropic-key" \ -d '{ "model": "anthropic/claude-3-opus-20240229", "messages": [{"role": "user", "content": "Hello"}] }' ``` ## Token Forwarding Behavior ### When Enabled (`forward_token = true`) - User-provided keys (via header) take priority - Falls back to configured key if no header provided - Returns 401 error if neither key is available ### When Disabled (`forward_token = false`, default) - Always uses the configured API key - Ignores the `X-Provider-API-Key` header - Returns 401 error if no configured key exists ## Client Library Examples ### Python (OpenAI SDK) ```python from openai import OpenAI # Using your own API key client = OpenAI( base_url="http://localhost:8000/llm/v1", api_key="not-used", # Required by SDK but ignored default_headers={ "X-Provider-API-Key": "sk-your-openai-key" } ) response = client.chat.completions.create( model="openai/gpt-4", messages=[{"role": "user", "content": "Hello!"}] ) ``` ### JavaScript/TypeScript ```javascript import OpenAI from 'openai'; const openai = new OpenAI({ baseURL: 'http://localhost:8000/llm/v1', apiKey: 'not-used', // Required by SDK but ignored defaultHeaders: { 'X-Provider-API-Key': 'sk-your-openai-key' } }); const completion = await openai.chat.completions.create({ model: 'openai/gpt-4', messages: [{ role: 'user', content: 'Hello!' }] }); ``` ### Custom HTTP Client ```javascript async function callNexusLLM(apiKey, model, messages) { const response = await fetch('http://localhost:8000/llm/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'X-Provider-API-Key': apiKey // User's own API key }, body: JSON.stringify({ model, messages }) }); return response.json(); } // User provides their own key const result = await callNexusLLM( 'sk-user-api-key', 'openai/gpt-4', [{ role: 'user', content: 'Hello' }] ); ``` ## Configuration Patterns ### SaaS with User Keys Allow users to optionally provide their own keys: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.COMPANY_OPENAI_KEY }}" # Company pays by default forward_token = true # Users can override with their key [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.COMPANY_ANTHROPIC_KEY }}" forward_token = true ``` ### Development Environment Developers use personal keys: ```toml [llm.providers.openai] type = "openai" # No default key - developers must provide their own forward_token = true [llm.providers.anthropic] type = "anthropic" forward_token = true ``` ### Mixed Configuration Some providers allow forwarding, others don't: ```toml # Users can use their own OpenAI keys [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" forward_token = true # Company Anthropic key only [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.COMPANY_ANTHROPIC_KEY }}" forward_token = false # No user keys allowed # Always use company Google key [llm.providers.google] type = "google" api_key = "{{ env.GOOGLE_API_KEY }}" # forward_token defaults to false ``` ## AWS Bedrock Limitation **Important**: Token forwarding is **not supported** for AWS Bedrock providers. Unlike other providers that use simple API keys, AWS Bedrock requires: - AWS credentials (access key ID, secret access key, session tokens) - AWS Signature Version 4 (SigV4) signing - Request-specific signatures based on content and timestamp - Complex authentication flow Due to this complexity, Bedrock providers must use pre-configured AWS credentials: ```toml [llm.providers.bedrock] type = "bedrock" region = "us-east-1" profile = "production" # forward_token is not supported - will be a validation error if provided ``` ## Security Considerations ### API Key Validation - Nexus passes keys directly to providers - Invalid keys result in provider-specific error responses - Keys are not logged or stored by Nexus ### Rate Limiting with Forwarded Tokens - Server-level rate limits still apply - Token-based rate limits require client identification ## Integration with Other Features ### With OAuth2 Authentication Combine OAuth2 for user authentication with token forwarding: ```toml # Require OAuth2 for access [server.oauth] url = "https://auth.example.com/.well-known/jwks.json" poll_interval = "5m" expected_issuer = "https://auth.example.com" expected_audience = "nexus-api" [server.oauth.protected_resource] resource = "https://nexus.example.com" authorization_servers = ["https://auth.example.com"] # Allow authenticated users to use their own API keys [llm.providers.openai] type = "openai" forward_token = true ``` Users must provide both: ```bash curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Authorization: Bearer " \ -H "X-Provider-API-Key: sk-user-api-key" \ -H "Content-Type: application/json" \ -d '{"model": "openai/gpt-4", "messages": [...]}' ``` ## Next Steps - Review [API Usage](/docs/usage/llm-api) for integration examples - Configure [Rate Limiting](/docs/configuration/llm/rate-limiting) for token-forwarded requests - Set up monitoring for API key usage --- # Configuration - LLM - Header Rules Nexus provides powerful header transformation capabilities for managing HTTP headers when communicating with LLM providers. This allows you to forward headers from clients, add authentication headers, remove sensitive information, and transform headers for provider compatibility. ## Overview Header rules enable you to: - **Forward** headers from incoming client requests to providers - **Insert** static headers for authentication or tracking - **Remove** sensitive or internal headers before sending to providers - **Rename** headers for provider compatibility while preserving originals ## Configuration Syntax Headers are configured as an array of rules under each provider. Each rule specifies a transformation type and its parameters: ```toml [[llm.providers.openai.headers]] rule = "forward" # Rule type: forward, insert, remove, or rename_duplicate name = "x-user-id" # Specific header name # OR pattern = "^x-custom-" # Regex pattern to match multiple headers ``` ## Rule Types ### Forward Rule Forwards headers from incoming client requests to the provider. Useful for passing user context, session information, or request IDs. ```toml # Forward a specific header [[llm.providers.openai.headers]] rule = "forward" name = "x-request-id" # Forward with rename and default value [[llm.providers.openai.headers]] rule = "forward" name = "x-trace-id" rename = "provider-trace-id" # Optional: rename the header default = "{{ env.DEFAULT_TRACE }}" # Optional: default if not present # Forward headers matching a pattern [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-org-" # Forward all headers starting with "x-org-" ``` ### Insert Rule Adds static headers to all requests sent to the provider. Perfect for API versioning, authentication tokens, or feature flags. ```toml # Add a static header [[llm.providers.anthropic.headers]] rule = "insert" name = "x-api-version" value = "2024-01" # Use environment variables [[llm.providers.anthropic.headers]] rule = "insert" name = "x-api-key" value = "{{ env.SECONDARY_API_KEY }}" # Add provider-specific features [[llm.providers.openai.headers]] rule = "insert" name = "OpenAI-Beta" value = "assistants=v2" ``` ### Remove Rule Removes headers before sending requests to the provider. Essential for security and preventing header conflicts. ```toml # Remove a specific header [[llm.providers.google.headers]] rule = "remove" name = "cookie" # Remove headers matching a pattern [[llm.providers.google.headers]] rule = "remove" pattern = "^x-internal-" # Remove all internal headers ``` ### Rename Duplicate Rule Duplicates a header with a new name while preserving the original header. This creates two headers with the same value. Useful when you need to pass the same value under different header names for compatibility. ```toml # Duplicate a header (keeps both original and new) [[llm.providers.anthropic.headers]] rule = "rename_duplicate" name = "authorization" rename = "x-original-auth" # Results in both headers present: # - authorization: Bearer abc123 # - x-original-auth: Bearer abc123 # With default value if original header doesn't exist [[llm.providers.anthropic.headers]] rule = "rename_duplicate" name = "x-user-token" rename = "x-backup-token" default = "Bearer {{ env.DEFAULT_TOKEN }}" # If x-user-token exists: both x-user-token and x-backup-token have its value # If x-user-token missing: both headers are set to the default value ``` ## Pattern Matching Use regular expressions to match multiple headers at once: ```toml # Forward all custom headers [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-custom-" # Remove all debug headers [[llm.providers.openai.headers]] rule = "remove" pattern = "^x-debug-" # Forward all headers except those starting with "internal-" [[llm.providers.openai.headers]] rule = "forward" pattern = "^(?!internal-).*" ``` ## Processing Order Headers start with an empty set and are processed sequentially in the exact order they appear in your configuration file. Each rule operates on the current state of the headers: - `forward` and `insert` rules override any existing headers with the same name - `rename_duplicate` creates a copy while preserving the original - `remove` deletes headers from the current set ### Sequential Processing Example ```toml # Starting point: headers = {} # 1. Insert a static header [[llm.providers.openai.headers]] rule = "insert" name = "x-api-version" value = "2024-01" # Result: headers = {"x-api-version": "2024-01"} # 2. Forward all user headers (will override existing headers!) [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-user-" # If client sends x-user-id=123, x-user-role=admin # Result: headers = {"x-api-version": "2024-01", "x-user-id": "123", "x-user-role": "admin"} # 3. Duplicate a header for backup [[llm.providers.openai.headers]] rule = "rename_duplicate" name = "x-user-id" rename = "x-original-user-id" # Result: headers = {"x-api-version": "2024-01", "x-user-id": "123", "x-user-role": "admin", "x-original-user-id": "123"} # 4. Remove sensitive user data [[llm.providers.openai.headers]] rule = "remove" name = "x-user-role" # Result: headers = {"x-api-version": "2024-01", "x-user-id": "123", "x-original-user-id": "123"} # 5. Insert to override the forwarded value [[llm.providers.openai.headers]] rule = "insert" name = "x-user-id" value = "sanitized" # Final result: headers = {"x-api-version": "2024-01", "x-user-id": "sanitized", "x-original-user-id": "123"} ``` ### Common Patterns #### Ensure Static Headers Can't Be Overridden Insert critical headers AFTER forwarding to ensure they have the correct value: ```toml # Forward user headers first [[llm.providers.api.headers]] rule = "forward" pattern = ".*" # Then override with required values [[llm.providers.api.headers]] rule = "insert" name = "x-api-version" value = "v2" # This will override any x-api-version from the client ``` #### Forward then Filter Forward everything, then remove specific headers: ```toml [[llm.providers.api.headers]] rule = "forward" pattern = "^x-" # Forward all x- headers [[llm.providers.api.headers]] rule = "remove" name = "x-internal-secret" # Remove this specific one ``` #### Selective Forwarding with Defaults Insert defaults first, then forward (forwarded headers will override defaults): ```toml # Set defaults [[llm.providers.api.headers]] rule = "insert" name = "x-tenant-id" value = "default-tenant" # Forward user headers (will override x-tenant-id if provided) [[llm.providers.api.headers]] rule = "forward" pattern = "^x-tenant-" ``` ## Security Considerations ### Protected Headers Nexus automatically protects sensitive headers by default. These headers are never forwarded unless explicitly configured: - `authorization` - `x-api-key` - `api-key` - `cookie` - `set-cookie` - `x-nexus-*` (internal Nexus headers) ### Explicit Forwarding To forward protected headers, you must explicitly include them: ```toml # Explicitly forward authorization (use with caution!) [[llm.providers.custom.headers]] rule = "forward" name = "authorization" ``` ### Best Practices 1. **Never forward authentication headers** unless absolutely necessary 2. **Use environment variables** for sensitive values 3. **Remove internal headers** before sending to external providers 4. **Validate header patterns** to avoid accidental exposure 5. **Test header rules** in development before production ## Advanced Examples ### Multi-Tenant Configuration Forward tenant information while adding provider authentication: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Forward tenant context [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-tenant-" # Add internal tracking [[llm.providers.openai.headers]] rule = "insert" name = "x-gateway" value = "nexus" # Remove internal headers [[llm.providers.openai.headers]] rule = "remove" pattern = "^x-internal-" ``` ### API Migration Handle header differences between API versions: ```toml [llm.providers.legacy_api] type = "openai" base_url = "https://legacy.api.com" api_key = "{{ env.LEGACY_KEY }}" # Forward modern header as legacy format [[llm.providers.legacy_api.headers]] rule = "forward" name = "x-request-id" rename = "RequestID" # Add required legacy headers [[llm.providers.legacy_api.headers]] rule = "insert" name = "API-Version" value = "1.0" # Remove unsupported headers [[llm.providers.legacy_api.headers]] rule = "remove" pattern = "^x-feature-" ``` ### Development vs Production Different header rules for environments: ```toml [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" # Forward debug headers in development [[llm.providers.openai.headers]] rule = "forward" pattern = "^x-debug-" # In production config, this rule would be omitted # Add environment identifier [[llm.providers.openai.headers]] rule = "insert" name = "x-environment" value = "{{ env.ENVIRONMENT }}" # "development" or "production" ``` ## Integration with Authentication Header rules work seamlessly with Nexus authentication methods: ### With Token Forwarding ```toml [llm.providers.openai] type = "openai" forward_token = true # Forward client's OpenAI token # Still add custom headers [[llm.providers.openai.headers]] rule = "insert" name = "x-nexus-client" value = "{{ env.CLIENT_ID }}" ``` ### With Multiple API Keys ```toml [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.PRIMARY_KEY }}" # Add secondary key as backup [[llm.providers.anthropic.headers]] rule = "insert" name = "x-backup-key" value = "{{ env.SECONDARY_KEY }}" ``` ## Troubleshooting ### Headers Not Appearing 1. Check rule syntax - ensure `rule` field is specified 2. Verify header names are correct (case-sensitive) 3. Check processing order - later rules may override earlier ones 4. Enable debug logging to see header transformations ### Pattern Matching Issues 1. Test regex patterns with online tools 2. Remember patterns are case-sensitive 3. Use `^` and `$` anchors appropriately 4. Escape special characters properly ### Security Warnings If you see warnings about protected headers: 1. Review if forwarding is truly necessary 2. Consider using different header names 3. Ensure you're not exposing credentials 4. Document the security implications ## Performance Considerations - **Minimize regex patterns** - Specific names are faster than patterns - **Order rules efficiently** - Put most common rules first - **Avoid excessive rules** - Each rule adds processing overhead - **Use patterns sparingly** - Only when matching multiple headers ## Next Steps - Configure [Provider Authentication](/docs/configuration/llm/providers) with headers - Set up [Token Forwarding](/docs/configuration/llm/token-forwarding) for user keys - Implement [Rate Limiting](/docs/configuration/llm/rate-limiting) with header-based rules --- # Configuration - MCP Configure Model Context Protocol (MCP) servers to provide tools and capabilities to AI models. MCP servers can be HTTP endpoints, local processes, or integrate with existing APIs. ## Configuration Topics ### Essential Configuration 1. **[Server Configuration](/docs/configuration/mcp/servers)** - Set up HTTP and STDIO MCP servers 2. **[Authentication](/docs/configuration/mcp/authentication)** - Secure servers with tokens and OAuth2 forwarding 3. **[TLS Configuration](/docs/configuration/mcp/tls)** - Configure secure connections and certificates 4. **[Rate Limiting](/docs/configuration/mcp/rate-limiting)** - Control usage per server and tool 5. **[Role-Based Access Control](/docs/configuration/mcp/rbac)** - Fine-grained access control based on user groups ## Quick Start ### Basic Configuration Enable MCP in your `nexus.toml` file: ```toml [mcp] enabled = true path = "/mcp" # HTTP-based MCP server [mcp.servers.api] url = "https://api.example.com/mcp" [mcp.servers.api.auth] token = "{{ env.API_TOKEN }}" # Add custom headers [[mcp.servers.api.headers]] rule = "insert" name = "x-api-version" value = "2024-01" # Local STDIO server [mcp.servers.filesystem] cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home/user"] # Database tools [mcp.servers.postgres] cmd = ["psql-mcp"] env = { PGHOST = "{{ env.DB_HOST }}", PGUSER = "{{ env.DB_USER }}", PGPASSWORD = "{{ env.DB_PASSWORD }}" } ``` ## Server Types Overview ### HTTP Servers - **Streamable HTTP**: Modern protocol with streaming support - **SSE**: Legacy Server-Sent Events (deprecated) - Support for authentication, TLS, and rate limiting ### STDIO Servers - Local processes communicating via stdin/stdout - Perfect for local tools and scripts - Support for environment variables and working directories ## Authentication Methods ### Static Tokens ```toml [mcp.servers.github] url = "https://api.github.com/mcp" [mcp.servers.github.auth] token = "{{ env.GITHUB_TOKEN }}" ``` ### OAuth2 Token Forwarding ```toml [mcp.servers.internal] url = "https://internal.company.com/mcp" [mcp.servers.internal.auth] type = "forward" # Forward user's OAuth2 token ``` ## Complete Example Here's a comprehensive configuration showing various MCP features: ```toml [mcp] enabled = true path = "/mcp" # Cache configuration for performance [mcp.downstream_cache] max_size = 1000 idle_timeout = "10m" # Public API with rate limiting [mcp.servers.weather] url = "https://weather-api.example.com/mcp" [mcp.servers.weather.rate_limits] limit = 100 interval = "60s" # Authenticated API with TLS [mcp.servers.github] url = "https://api.github.com/mcp" [mcp.servers.github.auth] token = "{{ env.GITHUB_TOKEN }}" [mcp.servers.github.tls] verify_certs = true # Tool-specific rate limits [mcp.servers.github.rate_limits.tools] search_code = { limit = 30, interval = "60s" } create_issue = { limit = 10, interval = "60s" } # Local database tools [mcp.servers.database] cmd = ["psql-mcp"] env = { PGHOST = "localhost", PGDATABASE = "myapp", PGUSER = "{{ env.DB_USER }}", PGPASSWORD = "{{ env.DB_PASSWORD }}" } stderr = "null" # Suppress stderr in production # Internal service with OAuth2 forwarding [mcp.servers.company_api] url = "https://api.internal.company.com/mcp" [mcp.servers.company_api.auth] type = "forward" [mcp.servers.company_api.tls] verify_certs = true root_ca_cert_path = "/etc/ssl/company-ca.pem" ``` ## Key Features - **Multiple Server Types**: HTTP, STDIO, and SSE protocols - **Flexible Authentication**: Static tokens or OAuth2 forwarding - **TLS Support**: Including mutual TLS for high security - **Rate Limiting**: Per-server and per-tool limits - **Connection Caching**: Automatic caching for performance - **Header Configuration**: Add custom headers for authentication and tracking - **Environment Variables**: Secure configuration management ## Best Practices 1. **Security First** - Always use environment variables for secrets - Enable TLS verification in production - Use token forwarding only with trusted servers - Implement rate limiting for expensive operations 2. **Performance Optimization** - Configure appropriate cache sizes - Use static connections when possible - Set reasonable rate limits - Monitor connection pool usage 3. **Operational Excellence** - Test servers individually before deployment - Use descriptive server names - Document each server's purpose - Implement health checks 4. **Configuration Management** - Separate configs for dev/staging/prod - Version control configurations - Never commit secrets - Validate before deployment ## Common Use Cases ### File System Access ```toml [mcp.servers.fs] cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/data"] ``` ### Database Operations ```toml [mcp.servers.postgres] cmd = ["psql-mcp"] env = { PGHOST = "{{ env.DB_HOST }}" } ``` ### API Integration ```toml [mcp.servers.api] url = "https://api.service.com/mcp" [mcp.servers.api.auth] token = "{{ env.API_KEY }}" ``` ### Internal Tools ```toml [mcp.servers.tools] url = "https://tools.internal/mcp" [mcp.servers.tools.auth] type = "forward" # Use user's OAuth2 token ``` ## Troubleshooting ### Debug Mode ```bash nexus --log debug ``` ### Common Issues - **Connection failures**: Check URLs and network access - **Authentication errors**: Verify tokens and permissions - **Rate limit exceeded**: Adjust limits or intervals - **Cache misses**: Increase cache size or timeout ## Next Steps - Start with [Server Configuration](/docs/configuration/mcp/servers) - Secure with [Authentication](/docs/configuration/mcp/authentication) - Configure [TLS](/docs/configuration/mcp/tls) for production - Optimize with [Rate Limiting](/docs/configuration/mcp/rate-limiting) - Control access with [RBAC](/docs/configuration/mcp/rbac) --- # Configuration - MCP - Servers Configure Model Context Protocol (MCP) servers to provide AI tools and capabilities through Nexus. MCP servers can be HTTP endpoints or local processes. ## Server Types ### HTTP Servers (Streamable) HTTP-based MCP servers implementing the latest protocol: ```toml [mcp.servers.example] url = "https://api.example.com/mcp" # HTTP/HTTPS endpoint (required) protocol = "streamable-http" # Optional - auto-detected for HTTP URLs # Optional authentication [mcp.servers.example.auth] token = "{{ env.API_TOKEN }}" # Bearer token authentication ``` ### STDIO Servers Local processes that communicate via standard input/output: ```toml [mcp.servers.local_tool] cmd = ["executable", "arg1", "arg2"] # Command to execute (required) env = { KEY = "value" } # Environment variables (optional) cwd = "/path/to/directory" # Working directory (optional) stderr = "null" # Error output handling (optional) ``` ### SSE Servers Server-Sent Events protocol (legacy - use Streamable HTTP when possible): ```toml [mcp.servers.legacy] protocol = "sse" # Must be "sse" (required) url = "https://api.example.com/sse" # SSE endpoint (required) message_url = "https://api.example.com/msg" # Message endpoint (optional) # Optional authentication [mcp.servers.legacy.auth] token = "{{ env.API_TOKEN }}" ``` ## Header Configuration Configure custom headers for MCP server connections. Headers can be added to HTTP servers for authentication, tracking, or compatibility requirements: ```toml [mcp.servers.example] url = "https://api.example.com/mcp" # Add static headers to all server requests [[mcp.servers.example.headers]] rule = "insert" name = "x-api-version" value = "2024-01" [[mcp.servers.example.headers]] rule = "insert" name = "x-client-id" value = "{{ env.CLIENT_ID }}" # Support for environment variables ``` ### Global MCP Headers You can also configure headers globally for all MCP servers: ```toml # Global headers applied to all MCP servers [[mcp.headers]] rule = "insert" name = "x-nexus-version" value = "1.0.0" [[mcp.headers]] rule = "insert" name = "x-environment" value = "{{ env.ENVIRONMENT }}" ``` Note: MCP servers currently only support the `insert` rule for adding static headers. Headers are applied during connection initialization. ## Server Configuration Options ### HTTP Servers (Streamable) #### Required Settings - `url`: The HTTP/HTTPS endpoint URL (required) #### Optional Settings - `protocol`: Set to `"streamable-http"` (auto-detected for HTTP URLs) - `auth.token`: Bearer token for authentication - `auth.type`: Set to `"forward"` to forward OAuth2 tokens from clients - `headers`: Array of header rules (currently only `insert` rule is supported) ### STDIO Servers #### Required Settings - `cmd`: Command array with executable and arguments (required) #### Optional Settings - `env`: Environment variables to set (default: `{}`) - `cwd`: Working directory for the subprocess (default: current directory) - `stderr`: Where to send stderr output (default: `"null"`) - `"null"`: Discard stderr output - `"inherit"`: Show in console - `{ file = "/path/to/log" }`: Write to file ### SSE Servers #### Required Settings - `protocol`: Must be set to `"sse"` (required) - `url`: The SSE endpoint URL (required) #### Optional Settings - `message_url`: Separate endpoint for sending messages (defaults to `url`) - `auth.token`: Bearer token for authentication ## Environment Variables Use `{{ env.VARIABLE_NAME }}` for secure configuration management. Environment variables can be used in URLs, authentication tokens, command arguments, and environment settings. ## Multiple Server Instances You can configure multiple instances of the same MCP server type by using different names in your configuration. This is useful for connecting to different environments (dev/staging/prod) or multiple databases. ## Server Naming Best Practices 1. **Use Descriptive Names**: Choose names that clearly indicate the server's purpose ```toml [mcp.servers.code_search] # Good [mcp.servers.server1] # Bad ``` 2. **Group Related Servers**: Use prefixes for related servers ```toml [mcp.servers.db_production] [mcp.servers.db_staging] [mcp.servers.db_development] ``` 3. **Avoid Special Characters**: Stick to alphanumeric and underscores ```toml [mcp.servers.file_system] # Good [mcp.servers."file.system"] # Requires quotes ``` ## Testing Servers ### Test STDIO Command ```bash # Test command directly $ /path/to/executable arg1 arg2 # Check environment variables $ echo $GITHUB_TOKEN ``` ### Test HTTP Connectivity ```bash # Test HTTP server $ curl -I https://api.example.com/mcp # Test with authentication $ curl -H "Authorization: Bearer $TOKEN" https://api.example.com/mcp ``` ### Debug Mode ```bash # Run Nexus with debug logs $ nexus --log debug ``` ## Troubleshooting ### STDIO Server Issues **Server not starting**: - Verify the executable path is correct - Check file permissions (executable must have +x) - Test the command manually in terminal - Review stderr output (set `stderr = "inherit"`) **Environment variables not working**: - Ensure variables are exported in your shell - Check for typos in variable names - Use `echo` to verify values ### HTTP Server Issues **Connection failures**: - Verify the URL is accessible - Check firewall rules - Test with curl or similar tool - Review TLS certificate configuration **Authentication errors**: - Verify API tokens are correct - Check token format and headers - Ensure tokens have necessary permissions ## Best Practices 1. **Security**: - Always use environment variables for secrets - Never commit API keys to version control - Use TLS for HTTP connections - Restrict file system access in STDIO servers 2. **Performance**: - Keep STDIO commands lightweight - Use appropriate stderr handling (null in production) - Configure connection pooling for HTTP servers 3. **Reliability**: - Test servers individually before adding to Nexus - Implement health checks for critical servers - Use descriptive error messages - Log server initialization issues 4. **Maintenance**: - Document each server's purpose - Version control server configurations - Use consistent naming conventions - Keep server configurations modular ## Next Steps - Configure [Authentication](/docs/configuration/mcp/authentication) for secure access - Set up [TLS Configuration](/docs/configuration/mcp/tls) for encrypted connections - Implement [Rate Limiting](/docs/configuration/mcp/rate-limiting) for resource control --- # Configuration - MCP - Authentication Configure authentication for MCP servers to secure access and enable token forwarding from upstream OAuth2 providers. ## Authentication Methods ### Static Token Authentication The simplest authentication method using a pre-configured token: ```toml [mcp.servers.api] url = "https://api.example.com/mcp" [mcp.servers.api.auth] token = "bearer_token_here" # Or use environment variables token = "{{ env.API_TOKEN }}" ``` ### OAuth2 Token Forwarding Forward the incoming OAuth2 token to downstream servers: ```toml [mcp.servers.internal] url = "https://internal.company.com/mcp" [mcp.servers.internal.auth] type = "forward" ``` ## Static Token Configuration ### Basic Token ```toml [mcp.servers.github] url = "https://api.github.com/mcp" [mcp.servers.github.auth] token = "{{ env.GITHUB_TOKEN }}" ``` ### Bearer Token ```toml [mcp.servers.api] url = "https://api.example.com/mcp" [mcp.servers.api.auth] token = "Bearer {{ env.API_BEARER_TOKEN }}" ``` ### API Key ```toml [mcp.servers.service] url = "https://service.example.com/mcp" [mcp.servers.service.auth] token = "{{ env.SERVICE_API_KEY }}" ``` ## Token Forwarding ### How It Works 1. **Client authenticates** with Nexus using a JWT token 2. **Nexus validates** the token using configured OAuth2 settings 3. **For servers with `type = "forward"`**, Nexus includes the same token in downstream requests 4. **Downstream servers** validate the token independently ### Configuration ```toml # Enable OAuth2 on Nexus server [server.oauth] url = "https://auth.example.com/.well-known/jwks.json" expected_issuer = "https://auth.example.com" expected_audience = "nexus-api" # Configure MCP server with token forwarding [mcp.servers.protected_api] url = "https://api.internal.com/mcp" [mcp.servers.protected_api.auth] type = "forward" ``` ### Requirements - OAuth2 must be enabled on the Nexus server - Downstream server must accept the same OAuth2 tokens - Both must use the same authorization server ### Connection Caching When using token forwarding, Nexus automatically caches connections per unique OAuth2 token to improve performance: ```toml # Optional: Configure cache size and timeout [mcp.downstream_cache] max_size = 1000 # Max cached connections (default: 1000) idle_timeout = "10m" # Connection idle timeout (default: 10 minutes) ``` - **Static connections** (no auth or static tokens) are created once at startup - **Dynamic connections** (token forwarding) are cached per unique user token - Connections are automatically evicted after the idle timeout ### Use Cases Token forwarding is ideal for: - **Single Sign-On (SSO)**: One login for all tools - **User Context**: Maintain user identity across services - **Audit Trail**: Track actions by specific users - **Multi-tenant Systems**: Isolate data per user/tenant ## Troubleshooting **401 Unauthorized**: - Verify token is correct and not expired - Check token format (Bearer prefix if needed) - Ensure token has necessary permissions - Review server authentication logs **Token Forwarding Not Working**: - Confirm OAuth2 is enabled on Nexus - Verify `type = "forward"` is set - Check downstream server accepts the tokens - Review token validation logs ### Testing Authentication ```bash # Test static token curl -H "Authorization: Bearer $TOKEN" https://api.example.com/mcp ``` ## Next Steps - Configure [TLS](/docs/configuration/mcp/tls) for encrypted connections - Set up [Rate Limiting](/docs/configuration/mcp/rate-limiting) per server - Review [Security Best Practices](/docs/best-practices#security) --- # Configuration - MCP - TLS Configuration Configure TLS (Transport Layer Security) for secure connections to MCP servers, including certificate verification and mutual TLS authentication. ## Basic TLS Configuration ```toml [mcp.servers.secure] url = "https://api.secure.com/mcp" [mcp.servers.secure.tls] verify_certs = true # Verify server certificates accept_invalid_hostnames = false # Reject hostname mismatches root_ca_cert_path = "/etc/ssl/certs/ca.pem" # Custom CA certificate # Client certificates for mutual TLS client_cert_path = "/etc/ssl/certs/client.pem" client_key_path = "/etc/ssl/private/client.key" ``` ## Configuration Options - `verify_certs`: Whether to verify server certificates (default: `true`) - `accept_invalid_hostnames`: Accept certificates with hostname mismatches (default: `false`) - `root_ca_cert_path`: Path to custom root CA certificate (optional, default: system CA) - `client_cert_path`: Path to client certificate for mTLS (optional) - `client_key_path`: Path to client private key for mTLS (optional) ## Certificate Verification ### Standard Verification Default configuration with system CA certificates: ```toml [mcp.servers.api] url = "https://api.example.com/mcp" [mcp.servers.api.tls] verify_certs = true # Default - verifies against system CAs ``` ### Custom CA Certificate For self-signed or internal CA certificates: ```toml [mcp.servers.internal] url = "https://internal.company.com/mcp" [mcp.servers.internal.tls] verify_certs = true root_ca_cert_path = "/etc/nexus/certs/company-ca.pem" ``` ### Skip Verification (Development Only) **Warning**: Only use in development environments: ```toml [mcp.servers.dev] url = "https://dev.localhost:8443/mcp" [mcp.servers.dev.tls] verify_certs = false # Dangerous - development only! ``` ## Mutual TLS (mTLS) ### Configuration For services requiring client certificates: ```toml [mcp.servers.high_security] url = "https://secure.example.com/mcp" [mcp.servers.high_security.tls] verify_certs = true client_cert_path = "/etc/nexus/certs/client.pem" client_key_path = "/etc/nexus/certs/client-key.pem" ``` ### With Custom CA and mTLS Complete configuration for maximum security: ```toml [mcp.servers.enterprise] url = "https://api.enterprise.com/mcp" [mcp.servers.enterprise.auth] token = "{{ env.ENTERPRISE_TOKEN }}" [mcp.servers.enterprise.tls] verify_certs = true accept_invalid_hostnames = false root_ca_cert_path = "/etc/nexus/certs/enterprise-ca.pem" client_cert_path = "/etc/nexus/certs/nexus-client.pem" client_key_path = "/etc/nexus/certs/nexus-client-key.pem" ``` ## Certificate Management ### Certificate Formats All certificates must be in PEM format: ```bash # View certificate details openssl x509 -in certificate.pem -text -noout # Convert DER to PEM openssl x509 -inform DER -in certificate.der -out certificate.pem # Convert PKCS12 to PEM openssl pkcs12 -in certificate.p12 -out certificate.pem -nodes ``` ### File Permissions Secure certificate files properly: ```bash # CA certificate (readable) chmod 644 /etc/nexus/certs/ca.pem # Client certificate (readable) chmod 644 /etc/nexus/certs/client.pem # Private key (restricted) chmod 600 /etc/nexus/certs/client-key.pem chown nexus:nexus /etc/nexus/certs/client-key.pem ``` ### Certificate Rotation Best practices for certificate management: 1. **Monitor Expiration**: ```bash # Check certificate expiration openssl x509 -in cert.pem -noout -enddate ``` 2. **Automate Renewal**: - Use cert-manager for Kubernetes - Use Let's Encrypt with automatic renewal - Set up monitoring alerts 30 days before expiration 3. **Graceful Rotation**: - Update certificates during maintenance windows - Test new certificates in staging first - Keep old certificates briefly for rollback ## Common Configurations ### Internal Services For internal corporate services: ```toml [mcp.servers.corp_api] url = "https://api.internal.corp/mcp" [mcp.servers.corp_api.tls] verify_certs = true root_ca_cert_path = "/etc/nexus/certs/corp-ca-chain.pem" ``` ### Cloud Services For cloud provider endpoints: ```toml [mcp.servers.aws_service] url = "https://service.amazonaws.com/mcp" [mcp.servers.aws_service.tls] verify_certs = true # Uses system CA bundle ``` ### Development with Self-Signed For local development: ```toml [mcp.servers.local_dev] url = "https://localhost:8443/mcp" [mcp.servers.local_dev.tls] verify_certs = true accept_invalid_hostnames = true # Allow localhost mismatch root_ca_cert_path = "/home/dev/certs/dev-ca.pem" ``` ### High-Security Environment For maximum security: ```toml [mcp.servers.banking] url = "https://api.bank.com/mcp" [mcp.servers.banking.auth] type = "forward" # OAuth2 token forwarding [mcp.servers.banking.tls] verify_certs = true accept_invalid_hostnames = false root_ca_cert_path = "/etc/nexus/certs/banking-ca.pem" client_cert_path = "/etc/nexus/certs/banking-client.pem" client_key_path = "/etc/nexus/certs/banking-client-key.pem" ``` ## Troubleshooting ### Certificate Verification Errors **"certificate verify failed"**: - Check certificate is not expired - Verify CA certificate is correct - Ensure certificate chain is complete - Check system time is correct **"hostname mismatch"**: - Certificate CN/SAN doesn't match hostname - Consider using `accept_invalid_hostnames` for development - Request new certificate with correct hostname ### Connection Issues **"unable to get local issuer certificate"**: ```toml # Solution: Provide CA certificate [mcp.servers.api.tls] root_ca_cert_path = "/path/to/ca-certificate.pem" ``` **"sslv3 alert bad certificate"**: ```toml # Solution: Provide client certificate for mTLS [mcp.servers.api.tls] client_cert_path = "/path/to/client-cert.pem" client_key_path = "/path/to/client-key.pem" ``` ### Testing TLS Configuration ```bash # Test server certificate openssl s_client -connect api.example.com:443 -servername api.example.com # Test with custom CA openssl s_client -connect api.example.com:443 \ -CAfile /etc/nexus/certs/ca.pem # Test mutual TLS openssl s_client -connect api.example.com:443 \ -cert client.pem -key client-key.pem # Test with curl curl --cacert ca.pem --cert client.pem --key client-key.pem \ https://api.example.com/mcp ``` ## Security Best Practices 1. **Always Verify Certificates in Production**: ```toml [mcp.servers.production.tls] verify_certs = true # Never set to false in production ``` 2. **Use Strong Cipher Suites**: - Prefer TLS 1.2 or higher - Disable weak ciphers at server level 3. **Protect Private Keys**: - Store with restricted permissions (600) - Use hardware security modules (HSM) for high-value keys - Never commit keys to version control 4. **Certificate Pinning**: - Pin CA certificates for critical services - Document pinned certificates - Plan for certificate rotation 5. **Monitor Certificate Health**: - Set up expiration alerts - Monitor for revoked certificates - Log TLS handshake failures ## Integration with Other Features ### With Authentication Combine TLS with authentication: ```toml [mcp.servers.secure_api] url = "https://api.secure.com/mcp" [mcp.servers.secure_api.auth] token = "{{ env.API_TOKEN }}" [mcp.servers.secure_api.tls] verify_certs = true client_cert_path = "/etc/nexus/certs/client.pem" client_key_path = "/etc/nexus/certs/client-key.pem" ``` ### With Rate Limiting Secure high-value endpoints: ```toml [mcp.servers.premium_api] url = "https://premium.example.com/mcp" [mcp.servers.premium_api.tls] verify_certs = true client_cert_path = "/etc/nexus/certs/premium-client.pem" client_key_path = "/etc/nexus/certs/premium-client-key.pem" [mcp.servers.premium_api.rate_limits] limit = 100 interval = "3600s" ``` ## Next Steps - Configure [Rate Limiting](/docs/configuration/mcp/rate-limiting) for resource control - Review [Security Best Practices](/docs/best-practices#security) - Set up monitoring for TLS health --- # Configuration - MCP - Rate Limiting Configure fine-grained rate limits for individual MCP servers and their tools to protect expensive operations while maintaining high throughput for lightweight queries. ## Server-Level Rate Limits Configure rate limits for individual MCP servers: ```toml [mcp.servers.my_api] url = "https://api.example.com/mcp" [mcp.servers.my_api.rate_limits] limit = 50 interval = "60s" ``` ## Tool-Specific Rate Limits Set rate limits on individual tools within an MCP server: ```toml [mcp.servers.my_api.rate_limits.tools] expensive_operation = { limit = 10, interval = "60s" } bulk_process = { limit = 5, interval = "300s" } standard_query = { limit = 100, interval = "60s" } ``` ## Rate Limit Precedence Rate limits are evaluated in the following order (most to least restrictive): 1. **Tool-specific limits** - Most granular control 2. **MCP server limits** - Per-server restrictions 3. **Per-IP limits** - Configured at server level 4. **Global limits** - Overall system limits All applicable limits are enforced - a request must pass all rate limit checks to succeed. ## Configuration Examples ### API with Tiered Operations Different limits for different operation costs: ```toml [mcp.servers.data_api] url = "https://data.example.com/mcp" # Overall server limit [mcp.servers.data_api.rate_limits] limit = 1000 interval = "60s" # Tool-specific limits [mcp.servers.data_api.rate_limits.tools] # Expensive data exports export_full_dataset = { limit = 5, interval = "3600s" } # 5 per hour generate_report = { limit = 20, interval = "3600s" } # 20 per hour # Moderate operations bulk_update = { limit = 50, interval = "600s" } # 50 per 10 min complex_search = { limit = 100, interval = "300s" } # 100 per 5 min # Lightweight queries simple_lookup = { limit = 1000, interval = "60s" } # 1000 per minute get_status = { limit = 2000, interval = "60s" } # 2000 per minute ``` ### Database Operations Protect database resources: ```toml [mcp.servers.database] cmd = ["psql-mcp"] env = { PGHOST = "{{ env.DB_HOST }}" } # Conservative server limit [mcp.servers.database.rate_limits] limit = 100 interval = "60s" [mcp.servers.database.rate_limits.tools] # DDL operations - very restricted create_table = { limit = 2, interval = "3600s" } drop_table = { limit = 1, interval = "3600s" } alter_schema = { limit = 5, interval = "3600s" } # Write operations - moderate limits insert_batch = { limit = 50, interval = "60s" } update_records = { limit = 100, interval = "60s" } delete_records = { limit = 20, interval = "60s" } # Read operations - generous limits select_query = { limit = 500, interval = "60s" } count_records = { limit = 1000, interval = "60s" } ``` ### AI/ML Services Manage compute-intensive operations: ```toml [mcp.servers.ml_service] url = "https://ml.example.com/mcp" [mcp.servers.ml_service.rate_limits] limit = 200 interval = "60s" [mcp.servers.ml_service.rate_limits.tools] # Training operations - very limited train_model = { limit = 2, interval = "86400s" } # 2 per day fine_tune = { limit = 5, interval = "86400s" } # 5 per day # Inference operations - moderate batch_inference = { limit = 20, interval = "3600s" } # 20 per hour single_inference = { limit = 100, interval = "60s" } # 100 per minute # Preprocessing - generous tokenize_text = { limit = 1000, interval = "60s" } validate_input = { limit = 2000, interval = "60s" } ``` ### External API Integration Respect third-party rate limits: ```toml [mcp.servers.github] url = "https://api.github.com/mcp" [mcp.servers.github.auth] token = "{{ env.GITHUB_TOKEN }}" # Match GitHub's rate limits [mcp.servers.github.rate_limits] limit = 5000 # GitHub's limit for authenticated requests interval = "3600s" # Per hour [mcp.servers.github.rate_limits.tools] # GraphQL has separate limit graphql_query = { limit = 5000, interval = "3600s" } # Search has stricter limit search_code = { limit = 30, interval = "60s" } search_issues = { limit = 30, interval = "60s" } # Regular API calls get_repo = { limit = 5000, interval = "3600s" } list_issues = { limit = 5000, interval = "3600s" } ``` ## Time Interval Formats Supported interval formats: - `"60s"` - 60 seconds - `"5m"` - 5 minutes - `"1h"` - 1 hour - `"24h"` - 24 hours - `"300s"` - 300 seconds - `"3600s"` - 3600 seconds (1 hour) - `"86400s"` - 86400 seconds (1 day) ## Rate Limit Strategies ### Cost-Based Limiting Align limits with operation costs: ```toml [mcp.servers.billing_api.rate_limits.tools] # $0.001 per call cheap_operation = { limit = 10000, interval = "3600s" } # $0.01 per call standard_operation = { limit = 1000, interval = "3600s" } # $0.10 per call expensive_operation = { limit = 100, interval = "3600s" } # $1.00 per call premium_operation = { limit = 10, interval = "3600s" } ``` ### Resource-Based Limiting Based on system resources: ```toml [mcp.servers.compute.rate_limits.tools] # CPU intensive cpu_heavy = { limit = 10, interval = "60s" } # Memory intensive memory_heavy = { limit = 20, interval = "60s" } # I/O intensive io_heavy = { limit = 50, interval = "60s" } # Network intensive network_heavy = { limit = 100, interval = "60s" } ``` ### User Tier Integration Combine with client identification: ```toml # Note: This is conceptual - actual implementation # would use client identification features [mcp.servers.premium_api.rate_limits] # Default limits limit = 100 interval = "60s" [mcp.servers.premium_api.rate_limits.tools] # Tool limits apply to all users premium_feature = { limit = 10, interval = "60s" } # User tiers would be handled by client identification # See /docs/configuration/server/client-identification ``` ## Monitoring Rate Limits ### Rate Limit Exceeded Response When a rate limit is exceeded: ```http HTTP/1.1 429 Too Many Requests Retry-After: 42 ``` ## Best Practices ### 1. Start Conservative Begin with lower limits and increase based on usage: ```toml # Start conservative [mcp.servers.new_api.rate_limits] limit = 10 interval = "60s" # Increase after monitoring [mcp.servers.new_api.rate_limits] limit = 100 # Increased after testing interval = "60s" ``` ### 2. Monitor Tool Usage Track which tools are called most frequently: ```toml [mcp.servers.api.rate_limits.tools] # Adjust based on actual usage patterns frequently_used = { limit = 1000, interval = "60s" } rarely_used = { limit = 10, interval = "60s" } ``` ### 3. Consider Operation Cost Set stricter limits for expensive operations: ```toml [mcp.servers.api.rate_limits.tools] # Free operations ping = { limit = 10000, interval = "60s" } # Cheap operations read_cache = { limit = 1000, interval = "60s" } # Expensive operations generate_report = { limit = 10, interval = "3600s" } run_analysis = { limit = 5, interval = "3600s" } ``` ### 4. Use Different Intervals Match intervals to operation characteristics: ```toml [mcp.servers.api.rate_limits.tools] # Burst protection - short interval quick_query = { limit = 100, interval = "10s" } # Sustained load - medium interval normal_operation = { limit = 500, interval = "300s" } # Daily quotas - long interval expensive_job = { limit = 10, interval = "86400s" } ``` ### 5. Test Rate Limits Verify limits work as expected: ```bash # Test tool-specific limit for i in {1..20}; do curl -X POST http://localhost:8000/mcp \ -d '{"tool": "expensive_operation"}' sleep 1 done # Should see 429 after limit is reached ``` ## Storage Backend Rate limits use the configured storage backend: ```toml # Memory storage (default) [server.rate_limits] storage = "memory" # Redis storage (recommended for production) [server.rate_limits] storage = { type = "redis", url = "redis://localhost:6379" } ``` See [Server Rate Limiting](/docs/configuration/server/rate-limiting) for storage configuration details. ## Troubleshooting ### Rate Limits Not Working - Verify tool names match exactly - Check interval format is correct - Ensure storage backend is configured - Review debug logs for rate limit evaluation ### Unexpected 429 Errors - Check all applicable rate limits (tool, server, IP, global) - Verify interval and limit values - Look for typos in tool names - Monitor actual usage patterns ### Performance Impact - Use Redis for distributed deployments - Consider longer intervals for expensive checks - Monitor rate limiter overhead - Optimize storage backend configuration ## Next Steps - Configure [Server-Level Rate Limits](/docs/configuration/server/rate-limiting) - Set up monitoring for rate limit metrics - Review [Best Practices](/docs/best-practices) for production --- # Configuration - MCP - Role-Based Access Control Control access to MCP servers and tools based on user group membership. RBAC enables you to implement fine-grained security policies for enterprise deployments. ## Overview RBAC in Nexus allows you to: - Restrict access to specific MCP servers based on user groups - Apply granular tool-level permissions within servers - Create tiered access levels (e.g., basic, premium, enterprise) - Block suspended or restricted users - Maintain backward compatibility with existing deployments ## How RBAC Works RBAC operates at two levels: 1. **Server-level rules**: Control access to entire MCP servers 2. **Tool-level rules**: Override server rules for specific tools Key principles: - **Deny takes precedence**: If a user is in a denied group, they're blocked regardless of allow rules - **Empty allow list blocks all**: An empty allow list prevents all access (no client identification needed) - **Tool rules override server rules**: Tool-specific settings take precedence - **No rules means open access**: Without RBAC configuration, access remains unrestricted ## Basic Configuration ### Enable Client Identification RBAC with group-based access requires [client identification](/docs/configuration/server/client-identification) to determine user groups: ```toml [server.client_identification] enabled = true # Required for group-based access client_id.http_header = "X-Client-ID" # or client_id.jwt_claim = "sub" group_id.http_header = "X-Group-ID" # or group_id.jwt_claim = "groups" ``` **Note**: An empty allow list (`allow = []`) blocks all access without requiring client identification. For detailed client identification setup, see the [Client Identification documentation](/docs/configuration/server/client-identification). ### Define User Groups Configure valid group values for your organization: ```toml [server.client_identification.validation] # Define your organization's group structure group_values = ["basic", "premium", "enterprise", "admin", "suspended"] ``` ## Server-Level Access Control Control who can access entire MCP servers: ```toml [mcp] enabled = true [mcp.servers.premium_tools] cmd = ["premium-server"] allow = ["premium", "enterprise", "admin"] # Allowed groups deny = ["suspended"] # Blocked groups ``` ### Examples #### Public Server (No Restrictions) ```toml [mcp] enabled = true [mcp.servers.public_api] url = "https://api.public.com/mcp" # No allow/deny rules - accessible to all users ``` #### Premium Server (Tiered Access) ```toml [mcp] enabled = true [mcp.servers.premium_features] cmd = ["premium-mcp-server"] allow = ["premium", "enterprise", "admin"] deny = ["suspended", "trial_expired"] ``` #### Admin-Only Server ```toml [mcp] enabled = true [mcp.servers.admin_tools] cmd = ["admin-server"] allow = ["admin"] # Only administrators can access ``` ## Tool-Level Access Control Override server-level rules for specific tools: ```toml [mcp] enabled = true [mcp.servers.api_tools] cmd = ["api-server"] allow = ["basic", "premium", "enterprise"] # Server accessible to most users # But restrict expensive operations [mcp.servers.api_tools.tools.bulk_export] allow = ["enterprise"] # Only enterprise users can bulk export [mcp.servers.api_tools.tools.deprecated_function] allow = [] # Empty allow list blocks all access to this tool (no client ID needed) [mcp.servers.api_tools.tools.admin_function] allow = ["admin"] # Only admins can use this specific tool ``` ## Troubleshooting ### Common Issues #### Access Denied Despite Allow Rule Check for: - User in deny list (deny takes precedence) - Missing [client identification](/docs/configuration/server/client-identification) - Invalid or missing group claims in token #### Tool Not Accessible Verify: - Tool-level rules don't conflict with server rules - Tool name matches exactly - Empty allow list not blocking access #### All Users Blocked Ensure: - Not using empty allow list unintentionally - [Client identification](/docs/configuration/server/client-identification) is properly configured - Groups are correctly extracted from tokens - `group_values` in `[server.client_identification.validation]` includes all groups used in allow/deny lists ## Next Steps - Set up [Client Identification](/docs/configuration/server/client-identification) to enable user recognition - Configure [Authentication](/docs/configuration/mcp/authentication) for secure access - Set up [Rate Limiting](/docs/configuration/mcp/rate-limiting) per group - Monitor with [Telemetry](/docs/telemetry) and audit logs - Review [Best Practices](/docs/best-practices) for production deployments --- # Configuration - Telemetry Configure Nexus telemetry to export OpenTelemetry metrics, traces, and logs to your observability backend. Nexus follows OpenTelemetry semantic conventions for consistency with other tools in your monitoring stack. ## Basic Configuration Enable telemetry in your `nexus.toml`: ```toml [telemetry] service_name = "nexus-production" # Optional, defaults to "nexus" # Resource attributes for all telemetry [telemetry.resource_attributes] environment = "production" region = "us-east-1" team = "platform" # OTLP exporter configuration [telemetry.exporters.otlp] enabled = true # Must be true to export metrics endpoint = "http://localhost:4317" # Your OTLP collector endpoint protocol = "grpc" # or "http" depending on your setup timeout = "60s" # Optional: Additional headers for authentication # For gRPC protocol: [telemetry.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.OTLP_TOKEN }}" x-nexus-shard = "primary" # Optional: TLS configuration for gRPC # [telemetry.exporters.otlp.grpc.tls] # domain_name = "collector.example.com" # ca = "/path/to/ca.crt" # For HTTP protocol (use otlp.http.headers instead): # [telemetry.exporters.otlp.http.headers] # authorization = "Bearer {{ env.OTLP_TOKEN }}" # x-nexus-shard = "primary" # Batch export settings (optional, these are defaults) [telemetry.exporters.otlp.batch_export] scheduled_delay = "5s" max_queue_size = 2048 max_export_batch_size = 512 max_concurrent_exports = 1 ``` ## Configuration Options ### Service Identification ```toml [telemetry] service_name = "nexus-production" # Identifies your service in metrics ``` **Default**: `"nexus"` ### Resource Attributes Add metadata that will be attached to all telemetry data: ```toml [telemetry.resource_attributes] environment = "production" # Environment name region = "us-east-1" # Geographic region team = "platform" # Owning team version = "1.2.3" # Application version datacenter = "aws-east" # Data center location ``` These attributes appear as labels in metrics and help with filtering, grouping, and correlation. ### OTLP Exporter The OpenTelemetry Protocol (OTLP) exporter sends data to collectors or backends: ```toml [telemetry.exporters.otlp] enabled = true # Required to enable export endpoint = "http://localhost:4317" # OTLP endpoint URL protocol = "grpc" # Protocol: "grpc" or "http" timeout = "60s" # Request timeout # Optional: Custom headers for authentication (protocol-specific) # For gRPC: [telemetry.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.OTLP_TOKEN }}" x-custom-header = "value" # Optional: TLS configuration for gRPC [telemetry.exporters.otlp.grpc.tls] domain_name = "custom_name" # Override server name for TLS verification key = "/path/to/key.pem" # Client certificate key cert = "/path/to/cert.pem" # Client certificate ca = "/path/to/ca.crt" # Custom CA certificate # For HTTP: # [telemetry.exporters.otlp.http.headers] # authorization = "Bearer {{ env.OTLP_TOKEN }}" # x-custom-header = "value" ``` #### Configuration Options: - **enabled**: Must be `true` to activate telemetry export - **endpoint**: URL of your OTLP receiver (collector, Grafana Agent, etc.) - **protocol**: - `"grpc"` (default) - More efficient, binary protocol - `"http"` - Better for proxies and load balancers - **timeout**: Maximum time to wait for export requests - **headers** (optional): Custom headers for authentication or routing - Supports environment variable substitution with `{{ env.VAR_NAME }}` - Headers are applied to all requests for this exporter - Protocol-specific header validation applies (e.g., gRPC metadata rules) ### Header and TLS Configuration Custom headers and TLS can be configured for authentication and secure communication: #### Headers Headers must be configured under the protocol-specific section: ```toml # For gRPC protocol (when protocol = "grpc") [telemetry.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.OTLP_TOKEN }}" # Authentication x-routing-key = "nexus-prod" # Custom routing x-tenant-id = "{{ env.TENANT_ID }}" # Multi-tenancy # For HTTP protocol (when protocol = "http") [telemetry.exporters.otlp.http.headers] Authorization = "Bearer {{ env.OTLP_TOKEN }}" # HTTP Authorization header X-Routing-Key = "nexus-prod" # Custom routing X-Tenant-Id = "{{ env.TENANT_ID }}" # Multi-tenancy ``` #### TLS Configuration (gRPC only) For secure gRPC connections, configure TLS certificates: ```toml [telemetry.exporters.otlp.grpc.tls] domain_name = "collector.example.com" # Server name for TLS verification key = "/etc/nexus/certs/client.key" # Client certificate key (for mTLS) cert = "/etc/nexus/certs/client.crt" # Client certificate (for mTLS) ca = "/etc/nexus/certs/ca.crt" # Custom CA certificate # Or use environment variables for paths [telemetry.exporters.otlp.grpc.tls] domain_name = "{{ env.OTLP_TLS_DOMAIN }}" key = "{{ env.CLIENT_KEY_PATH }}" cert = "{{ env.CLIENT_CERT_PATH }}" ca = "{{ env.CA_CERT_PATH }}" ``` #### Configuration Guidelines: - **Environment Variables**: Use `{{ env.VAR_NAME }}` for sensitive values - **Protocol Rules**: - gRPC: Headers become metadata, cannot start with "grpc-" (reserved) - HTTP: Standard HTTP header rules apply - **TLS Options**: - `domain_name`: Override the server name used for TLS verification - `key` + `cert`: Enable mutual TLS (mTLS) authentication - `ca`: Use custom CA certificate instead of system CA bundle - **Inheritance**: Signal-specific exporters (traces/metrics/logs) can override global settings - **Security**: Keep tokens and certificate paths in environment variables ### Batch Export Settings Control how telemetry data is batched for export: ```toml [telemetry.exporters.otlp.batch_export] scheduled_delay = "5s" # How often to export max_queue_size = 2048 # Maximum items in queue max_export_batch_size = 512 # Items per export batch max_concurrent_exports = 1 # Parallel export requests ``` #### Batch Configuration Guidelines: - **scheduled_delay**: Lower values = more real-time, higher network overhead - **max_queue_size**: Increase if data is being dropped during spikes - **max_export_batch_size**: Larger batches = more efficient, but higher memory usage - **max_concurrent_exports**: Usually keep at 1 unless your backend supports high concurrency ### Tracing Configuration Configure distributed tracing: ```toml [telemetry.tracing] sampling = 0.15 # Sample 15% of requests (0.0 to 1.0) parent_based_sampler = false # Respect parent's sampling decision (default: false) # Collection limits (per span) [telemetry.tracing.collect] max_events_per_span = 128 max_attributes_per_span = 128 max_links_per_span = 128 max_attributes_per_event = 128 max_attributes_per_link = 128 [telemetry.tracing.propagation] trace_context = false # W3C Trace Context (default: false) aws_xray = false # AWS X-Ray format (default: false) # Override global OTLP exporter for traces (optional) [telemetry.tracing.exporters.otlp] enabled = true endpoint = "http://traces-collector:4317" protocol = "grpc" timeout = "30s" # Optional: Headers specific to trace export (protocol-specific) # For gRPC: [telemetry.tracing.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.TRACE_TOKEN }}" x-trace-priority = "high" # For HTTP: # [telemetry.tracing.exporters.otlp.http.headers] # authorization = "Bearer {{ env.TRACE_TOKEN }}" # x-trace-priority = "high" ``` #### Tracing Options: - **sampling**: Fraction of requests to trace (0.0-1.0) - Production: 0.01-0.1 (1-10%) - Development: 1.0 (100%) - **parent_based_sampler**: Parent-based sampling strategy (default: false) - When `true`: Respects upstream service's sampling decision from trace context - When `false`: Uses local sampling ratio regardless of parent trace - Benefits: Ensures complete distributed traces and consistent sampling across services - **collect**: Per-span collection limits - `max_events_per_span`: Maximum events per span (default: 128) - `max_attributes_per_span`: Maximum attributes per span (default: 128) - `max_links_per_span`: Maximum links per span (default: 128) - `max_attributes_per_event`: Maximum attributes per event (default: 128) - `max_attributes_per_link`: Maximum attributes per link (default: 128) - **propagation**: Context propagation formats - `trace_context`: W3C standard (default: false) - `aws_xray`: For AWS environments (default: false) - **exporters**: Override global OTLP settings specifically for traces ### Metrics Configuration Configure metrics export: ```toml # Override global OTLP exporter for metrics (optional) [telemetry.metrics.exporters.otlp] enabled = true endpoint = "http://metrics-collector:4317" protocol = "grpc" # or "http" timeout = "30s" # Optional: Headers specific to metrics export (protocol-specific) # For gRPC: [telemetry.metrics.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.METRICS_TOKEN }}" # For HTTP: # [telemetry.metrics.exporters.otlp.http.headers] # authorization = "Bearer {{ env.METRICS_TOKEN }}" # Batch export settings for metrics (optional) [telemetry.metrics.exporters.otlp.batch_export] scheduled_delay = "10s" max_queue_size = 4096 max_export_batch_size = 1024 max_concurrent_exports = 1 ``` If not specified, metrics will use the global OTLP exporter configuration. ### Logs Configuration Configure structured log export via OpenTelemetry: ```toml # Override global OTLP exporter for logs (optional) [telemetry.logs.exporters.otlp] enabled = true endpoint = "http://logs-collector:4317" protocol = "grpc" # or "http" timeout = "30s" # Optional: Headers specific to logs export (protocol-specific) # For gRPC: [telemetry.logs.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.LOGS_TOKEN }}" # For HTTP: # [telemetry.logs.exporters.otlp.http.headers] # authorization = "Bearer {{ env.LOGS_TOKEN }}" # Batch export settings for logs (optional) [telemetry.logs.exporters.otlp.batch_export] scheduled_delay = "10s" # Batch logs for 10 seconds max_queue_size = 8192 # Buffer for log spikes max_export_batch_size = 2048 # Large batches for efficiency max_concurrent_exports = 1 # Parallel export requests ``` If not specified, logs will use the global OTLP exporter configuration. #### Log Level Control: Control log verbosity using the `--log` flag or `NEXUS_LOG` environment variable: ```bash # Set log level nexus --log info # Production (default) nexus --log debug # Development nexus --log trace # Maximum verbosity nexus --log off # Disable logging # Per-module configuration nexus --log "nexus=debug,tower_http=info" # Using environment variable NEXUS_LOG=debug nexus ``` Control output format with `--log-style` or `NEXUS_LOG_STYLE`: ```bash nexus --log-style json # Structured JSON output nexus --log-style color # Colorized terminal output nexus --log-style text # Plain text output # Using environment variable NEXUS_LOG_STYLE=json nexus ``` See [Logs documentation](/docs/telemetry/logs) for details on log attributes, correlation, and queries. ## Integration Examples ### Prometheus via OpenTelemetry Collector ```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 exporters: prometheus: endpoint: "0.0.0.0:8889" namespace: nexus const_labels: environment: production service: pipelines: metrics: receivers: [otlp] exporters: [prometheus] traces: receivers: [otlp] exporters: [prometheus] # Or your trace backend logs: receivers: [otlp] exporters: [prometheus] # Or your logs backend ``` Nexus configuration: ```toml [telemetry.exporters.otlp] enabled = true endpoint = "http://localhost:4317" protocol = "grpc" ``` ### Grafana Cloud You can now send telemetry directly to Grafana Cloud using custom headers: ```toml # Direct connection to Grafana Cloud [telemetry.exporters.otlp] enabled = true endpoint = "https://otlp-gateway-prod-us-central-0.grafana.net/otlp" protocol = "http" timeout = "30s" [telemetry.exporters.otlp.http.headers] authorization = "Basic {{ env.GRAFANA_CLOUD_TOKEN }}" ``` Alternatively, you can still use a local collector if you need additional processing: ```yaml # otel-collector-config.yaml (optional) receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: otlphttp: endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp headers: authorization: Basic ${env:GRAFANA_CLOUD_TOKEN} service: pipelines: metrics: receivers: [otlp] exporters: [otlphttp] traces: receivers: [otlp] exporters: [otlphttp] logs: receivers: [otlp] exporters: [otlphttp] ``` ### Datadog Export via the Datadog Agent with OTLP support: ```yaml # datadog.yaml otlp_config: receiver: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 ``` Nexus configuration: ```toml [telemetry.exporters.otlp] enabled = true endpoint = "http://localhost:4317" protocol = "grpc" ``` ### AWS CloudWatch Via OpenTelemetry Collector with AWS EMF exporter: ```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: awsemf: region: us-east-1 namespace: Nexus dimension_rollup_option: NoDimensionRollup service: pipelines: metrics: receivers: [otlp] exporters: [awsemf] traces: receivers: [otlp] exporters: [awsxray] # AWS X-Ray for traces logs: receivers: [otlp] exporters: [awscloudwatchlogs] # CloudWatch for logs ``` ## Performance Tuning ### High Volume Deployments For high-traffic environments, optimize batch settings: ```toml [telemetry.exporters.otlp.batch_export] scheduled_delay = "10s" # Less frequent exports max_queue_size = 4096 # Buffer more data max_export_batch_size = 1024 # Larger batches max_concurrent_exports = 2 # More parallelism if supported ``` ### Low Latency Requirements For near real-time data: ```toml [telemetry.exporters.otlp.batch_export] scheduled_delay = "1s" # Very frequent exports max_export_batch_size = 256 # Smaller batches ``` ### Resource Constrained Environments Minimize resource usage: ```toml [telemetry.exporters.otlp.batch_export] scheduled_delay = "30s" # Infrequent exports max_queue_size = 512 # Smaller buffer max_export_batch_size = 128 # Small batches ``` ## Troubleshooting ### Telemetry Not Working 1. **Check Configuration**: ```toml [telemetry.exporters.otlp] enabled = true # Must be explicitly enabled ``` 2. **Verify Endpoint Connectivity**: ```bash # For gRPC protocol grpcurl -plaintext localhost:4317 list # For HTTP protocol curl -v http://localhost:4318/v1/metrics ``` 3. **Enable Debug Logging**: ```bash nexus --log debug 2>&1 | grep -i telemetry ``` ### Common Configuration Errors - **Wrong protocol**: Ensure your collector supports the protocol you've configured - **Network issues**: Firewall blocking OTLP ports (4317 for gRPC, 4318 for HTTP) - **Resource exhaustion**: Queue full due to slow collector or network - **Authentication**: Some backends require authentication via collector ### Performance Issues - **High memory usage**: Reduce `max_queue_size` or `max_export_batch_size` - **Export failures**: Increase `timeout` or check collector capacity - **Missing data**: Increase `max_queue_size` if queue is overflowing ## Environment-Specific Configurations ### Development ```toml [telemetry] service_name = "nexus-dev" [telemetry.resource_attributes] environment = "development" [telemetry.exporters.otlp] enabled = true endpoint = "http://localhost:4317" protocol = "grpc" [telemetry.exporters.otlp.batch_export] scheduled_delay = "5s" # Default is fine [telemetry.tracing] sampling = 1.0 # Sample everything in dev parent_based_sampler = false # Don't need parent-based in dev ``` ### Production ```toml [telemetry] service_name = "nexus-prod" [telemetry.resource_attributes] environment = "production" region = "{{ env.AWS_REGION }}" version = "{{ env.APP_VERSION }}" [telemetry.exporters.otlp] enabled = true endpoint = "{{ env.OTEL_ENDPOINT }}" protocol = "grpc" timeout = "30s" # Shorter timeout for prod # Authentication headers for production (gRPC) [telemetry.exporters.otlp.grpc.headers] authorization = "Bearer {{ env.OTEL_AUTH_TOKEN }}" x-environment = "production" # TLS configuration for secure connections (optional) [telemetry.exporters.otlp.grpc.tls] domain_name = "{{ env.OTEL_TLS_DOMAIN }}" # e.g., "telemetry.company.com" ca = "{{ env.CA_CERT_PATH }}" # Custom CA if needed # For mTLS authentication: # key = "{{ env.CLIENT_KEY_PATH }}" # cert = "{{ env.CLIENT_CERT_PATH }}" [telemetry.exporters.otlp.batch_export] scheduled_delay = "10s" # Less frequent for efficiency max_queue_size = 4096 # Handle traffic spikes max_export_batch_size = 1024 # Efficient batching [telemetry.tracing] sampling = 0.1 # Sample 10% of requests parent_based_sampler = true # Respect upstream sampling for complete traces [telemetry.tracing.propagation] trace_context = true # Enable W3C trace context aws_xray = false # Or true if using AWS ``` ## Security Considerations 1. **Network Security**: Use TLS-enabled collectors in production 2. **Data Sensitivity**: Be careful with resource attributes - they appear in all metrics 3. **Access Control**: Ensure only authorized services can send to your OTLP endpoint 4. **Data Retention**: Configure appropriate retention policies in your backend ## Related Documentation - [Telemetry Overview](/docs/telemetry) - Understanding available telemetry types - [Metrics](/docs/telemetry/metrics) - All available metrics and queries - [Traces](/docs/telemetry/traces) - Distributed tracing spans and configuration - [Logs](/docs/telemetry/logs) - Structured application logs and correlation - [Server Configuration](/docs/configuration/server) - HTTP server settings - [LLM Configuration](/docs/configuration/llm) - Language model settings - [MCP Configuration](/docs/configuration/mcp) - Tool protocol settings ## External Resources - [OpenTelemetry Configuration](https://opentelemetry.io/docs/collector/configuration/) - [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/) - [Collector Documentation](https://opentelemetry.io/docs/collector/) - [Grafana OTLP Guide](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/setup/otlp/) --- # Usage Learn how to interact with Nexus APIs, integrate with client libraries, and build applications using the LLM router and MCP servers. ## Topics ### API Integration - **[LLM API Usage](/docs/usage/llm-api)** - OpenAI-compatible API endpoints and examples - **[MCP Integration](/docs/usage/mcp-integration)** - Connect Nexus to Claude Desktop, Claude Code, Cursor, and other AI agents - **[Claude Code Integration](/docs/usage/claude-code-integration)** - Use Nexus with Claude Code via Anthropic protocol - **[OpenAI Codex Integration](/docs/usage/openai-codex-integration)** - Use Nexus with OpenAI Codex CLI ## Quick Examples ### LLM API Call ```bash curl -X POST http://localhost:8000/llm/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ### MCP Tool Invocation ```bash curl -X POST http://localhost:8000/mcp/tools/filesystem/read_file \ -H "Content-Type: application/json" \ -d '{"path": "/home/user/document.txt"}' ``` ### Python SDK ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/llm/v1", api_key="not-used" ) response = client.chat.completions.create( model="openai/gpt-4", messages=[{"role": "user", "content": "Hello!"}] ) ``` --- # Usage - LLM client integration The Nexus LLM router provides an OpenAI-compatible API that works with any OpenAI client library or HTTP client. ## API Endpoints ### List Models ```http GET /llm/openai/v1/models ``` Returns all configured models: ```json { "object": "list", "data": [ { "id": "openai/gpt-4", "object": "model", "owned_by": "openai" }, { "id": "anthropic/claude-3-5-sonnet-20241022", "object": "model", "owned_by": "anthropic" } ] } ``` ### Chat Completions ```http POST /llm/openai/v1/chat/completions ``` Standard OpenAI chat completions format with provider-prefixed models. ## Basic Examples ### Simple Chat Completion ```bash curl -X POST http://localhost:8000/llm/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4", "messages": [ {"role": "user", "content": "Hello!"} ] }' ``` ### With System Message ```bash curl -X POST http://localhost:8000/llm/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "anthropic/claude-3-5-sonnet-20241022", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain quantum computing in simple terms."} ], "temperature": 0.7, "max_tokens": 500 }' ``` ### Streaming Response ```bash curl -X POST http://localhost:8000/llm/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4", "messages": [ {"role": "user", "content": "Write a short story"} ], "stream": true }' ``` ## Client Library Integration ### Python (OpenAI SDK) ```python from openai import OpenAI # Configure the client client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used" # Required by SDK but ignored by Nexus ) # Simple completion response = client.chat.completions.create( model="openai/gpt-4", messages=[ {"role": "user", "content": "Hello, how are you?"} ] ) print(response.choices[0].message.content) # Streaming stream = client.chat.completions.create( model="anthropic/claude-3-5-sonnet-20241022", messages=[ {"role": "user", "content": "Tell me a story"} ], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") ``` ### JavaScript/TypeScript ```javascript import OpenAI from 'openai'; // Configure the client const openai = new OpenAI({ baseURL: 'http://localhost:8000/llm/openai/v1', apiKey: 'not-used' // Required by SDK but ignored }); // Async/await usage async function chat() { const completion = await openai.chat.completions.create({ model: 'openai/gpt-4', messages: [ { role: 'user', content: 'What is the capital of France?' } ] }); console.log(completion.choices[0].message.content); } // Streaming async function streamChat() { const stream = await openai.chat.completions.create({ model: 'anthropic/claude-3-5-sonnet-20241022', messages: [{ role: 'user', content: 'Write a haiku' }], stream: true }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content || ''); } } ``` ### Langchain Integration ```python from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage # Configure for Nexus llm = ChatOpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used", model="openai/gpt-4" ) # Use with Langchain response = llm.invoke([ HumanMessage(content="What's the weather like?") ]) print(response.content) ``` ### Custom HTTP Client ```python import requests import json def call_nexus(model, messages, **kwargs): response = requests.post( "http://localhost:8000/llm/openai/v1/chat/completions", headers={"Content-Type": "application/json"}, json={ "model": model, "messages": messages, **kwargs } ) return response.json() # Use any model through the same endpoint result = call_nexus( "google/gemini-1.5-pro", [{"role": "user", "content": "Hello!"}], temperature=0.5 ) print(result["choices"][0]["message"]["content"]) ``` ## Advanced Features ### Function Calling / Tool Use Tool calling is now supported across multiple providers including OpenAI, Anthropic, Google, and AWS Bedrock. The API remains consistent across all providers. ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used" ) # Define a function/tool tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and state" } }, "required": ["location"] } } } ] # Works with OpenAI models response = client.chat.completions.create( model="openai/gpt-4", messages=[ {"role": "user", "content": "What's the weather in San Francisco?"} ], tools=tools, tool_choice="auto" ) # Also works with Anthropic models via Bedrock response = client.chat.completions.create( model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0", messages=[ {"role": "user", "content": "What's the weather in San Francisco?"} ], tools=tools, tool_choice="auto" ) # And with Google models response = client.chat.completions.create( model="google/gemini-1.5-pro", messages=[ {"role": "user", "content": "What's the weather in San Francisco?"} ], tools=tools, tool_choice="auto" ) # Check if model wants to call a function if response.choices[0].message.tool_calls: tool_call = response.choices[0].message.tool_calls[0] print(f"Function: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") ``` **Note on Bedrock Tool Support**: While most Bedrock models support tools well, some models like Llama may have inconsistent tool calling behavior. Claude models via Bedrock provide the most reliable tool support. ## Authentication ### With OAuth2 When OAuth2 is enabled on the server: ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used", default_headers={ "Authorization": "Bearer your-oauth2-token" } ) ``` ### With User API Keys (Token Forwarding) When token forwarding is enabled: ```python client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used", default_headers={ "X-Provider-API-Key": "sk-your-actual-api-key" } ) ``` ### Combined Authentication Both OAuth2 and user API key: ```python client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used", default_headers={ "Authorization": "Bearer oauth-token", "X-Provider-API-Key": "sk-user-api-key" } ) ``` ## Error Handling ### Python Example ```python from openai import OpenAI, OpenAIError client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used" ) try: response = client.chat.completions.create( model="openai/gpt-4", messages=[{"role": "user", "content": "Hello"}] ) except OpenAIError as e: if e.status_code == 429: print("Rate limit exceeded, please wait") elif e.status_code == 404: print("Model not found") elif e.status_code == 401: print("Authentication failed") else: print(f"Error: {e}") ``` ### JavaScript Example ```javascript try { const completion = await openai.chat.completions.create({ model: 'openai/gpt-4', messages: [{ role: 'user', content: 'Hello' }] }); } catch (error) { if (error.status === 429) { console.log('Rate limit exceeded'); } else if (error.status === 404) { console.log('Model not found'); } else { console.error('Error:', error.message); } } ``` ## Performance Tips ### Connection Pooling ```python import httpx from openai import OpenAI # Use custom HTTP client with connection pooling http_client = httpx.Client( limits=httpx.Limits(max_connections=100) ) client = OpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used", http_client=http_client ) ``` ### Async Operations ```python import asyncio from openai import AsyncOpenAI async def main(): client = AsyncOpenAI( base_url="http://localhost:8000/llm/openai/v1", api_key="not-used" ) # Concurrent requests tasks = [ client.chat.completions.create( model="openai/gpt-3.5-turbo", messages=[{"role": "user", "content": f"Count to {i}"}] ) for i in range(1, 6) ] responses = await asyncio.gather(*tasks) for response in responses: print(response.choices[0].message.content) asyncio.run(main()) ``` ## Testing and Debugging ### Check Available Models ```bash # List all available models curl http://localhost:8000/llm/openai/v1/models | jq '.data[].id' ``` ### Test Basic Connectivity ```bash # Simple test request curl -X POST http://localhost:8000/llm/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-3.5-turbo", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10 }' | jq '.' ``` ### Enable Debug Logging ```bash # Run Nexus with debug logging nexus --log debug ``` ## Best Practices 1. **Use Model Aliases**: Configure friendly names for models 2. **Handle Errors Gracefully**: Implement proper error handling 3. **Set Timeouts**: Configure appropriate timeouts for your use case 4. **Use Streaming**: For long responses, use streaming to improve UX 5. **Cache Responses**: Cache frequently requested completions 6. **Monitor Usage**: Track token usage and costs per model 7. **Implement Retries**: Use exponential backoff for transient errors ## Next Steps - Configure [Rate Limiting](/docs/configuration/llm/rate-limiting) to control usage - Enable [Token Forwarding](/docs/configuration/llm/token-forwarding) for user keys - Set up monitoring to track performance --- # Usage - MCP integration with AI Agents Nexus acts as an MCP (Model Context Protocol) server that can be connected to AI agents like Claude Desktop, Claude Code, VSCode Copilot, and other MCP-compatible tools. This allows these agents to access all the tools configured in your Nexus instance. ## Understanding MCP Components ### Tools MCP tools are specific functions or capabilities that MCP servers expose to AI models. Each MCP server can provide multiple tools, allowing AI assistants to: - **File System Tools**: Read, write, and manipulate files - **Database Tools**: Query and modify databases - **API Tools**: Interact with external services - **Compute Tools**: Run calculations or transformations #### How Tools Work in Nexus 1. **Tool Discovery**: Nexus provides a context-aware `search` tool that allows AI assistants to discover tools using natural language queries 2. **Tool Namespacing**: Tools from downstream servers are automatically namespaced with their server name (e.g., `github__search_code`, `filesystem__read_file`) 3. **Intelligent Search**: The search tool uses fuzzy matching to find relevant tools across all connected MCP servers 4. **Tool Execution**: Tools are invoked through the `execute` tool, which routes the request to the appropriate MCP server 5. **Rate Limiting**: You can set limits on individual tools to prevent abuse 6. **Authentication**: Tools inherit authentication from their parent server ### Prompts MCP prompts are predefined conversation templates or interaction patterns that servers can expose. They help guide AI assistants in common tasks or workflows. #### How Prompts Work in Nexus 1. **Prompt Aggregation**: Nexus aggregates prompts from all connected MCP servers during initialization 2. **Prompt Namespacing**: Prompts are prefixed with their server name (e.g., `github__create_pr`, `docs__explain_error`) 3. **Prompt Listing**: AI assistants can list all available prompts from all servers 4. **Prompt Retrieval**: Individual prompts can be retrieved using `get_prompt` with the namespaced name Example prompts from different servers: - `github__review_code` - Template for code review requests - `database__optimize_query` - Guide for query optimization - `docs__generate_readme` - Template for creating documentation ### Resources MCP resources represent data or file-like entities that servers can expose, such as documents, configurations, or data sets. #### How Resources Work in Nexus 1. **Resource Aggregation**: Nexus aggregates resources from all connected MCP servers during initialization 2. **Resource Identification**: Resources are tracked by their URI and mapped to their originating server 3. **Resource Listing**: AI assistants can list all available resources from all servers 4. **Resource Reading**: AI assistants can read resource contents using `read_resource` with the resource URI Example resources: - Configuration files with URIs like `file:///config/database.yml` - Documentation with URIs like `https://docs.example.com/api-guide` - Data sets with URIs like `data://analytics/monthly-report` - Templates with URIs like `template://project/structure` ### Discovery Example When you configure multiple MCP servers: ```toml [mcp.servers.github] url = "https://api.github.com/mcp" [mcp.servers.filesystem] cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home"] ``` The AI assistant can discover and use: 1. **Tools**: Use `search` to find tools, then `execute` to run them 2. **Prompts**: List available prompts for guided interactions 3. **Resources**: List and read available resources for context All components (tools, prompts, resources) are automatically namespaced to prevent conflicts and maintain clarity about their origin. ## Important: OAuth2 Authentication Nexus acts as an **OAuth2-protected resource** - it validates JWT tokens but does not provide authentication itself. Users must authenticate through their chosen OAuth2 provider (Auth0, Okta, Azure AD, etc.), and the MCP client must handle this authentication flow. ### How It Works 1. **User chooses their OAuth2 provider** (configured in Nexus) 2. **MCP client initiates OAuth2 flow** with that provider 3. **User authenticates** with their provider 4. **Provider issues JWT token** to the MCP client 5. **Nexus validates the token** using the provider's JWKS endpoint ### Client Requirements MCP clients connecting to OAuth2-protected Nexus must: - Handle OAuth2 authorization code flow - Store and refresh tokens - Include valid JWT tokens in requests to Nexus ## Claude Desktop Configuration ### Local Nexus (No Authentication) For a local Nexus instance without authentication, edit `claude_desktop_config.json`: **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` ```json { "mcpServers": { "nexus-local": { "command": "npx", "args": [ "mcp-remote", "http://localhost:8000/mcp" ] } } } ``` ### Remote Nexus with OAuth2 **Important**: Claude Desktop does not support direct OAuth2 configuration in the JSON file. For OAuth2-protected remote servers: 1. **Option 1: Use Claude.ai Web Interface** - Remote servers with OAuth2 can be added through Claude.ai Settings > Connectors - Claude handles OAuth2 flow with your provider - OAuth callback URL: `https://claude.ai/api/mcp/auth_callback` 2. **Option 2: Use mcp-remote Adapter** The `mcp-remote` adapter handles OAuth2 authentication automatically: ```json { "mcpServers": { "nexus": { "command": "npx", "args": ["-y", "mcp-remote", "https://nexus.example.com/mcp"] } } } ``` When you first use the server, `mcp-remote` will: - Open your browser for OAuth2 authentication - Store credentials in `~/.mcp-auth/` - Handle token refresh automatically ## Claude Code (VSCode Extension) Configuration Claude Code has native support for remote MCP servers with OAuth2. ### Configuration Methods #### 1. Workspace Configuration (`.vscode/mcp.json`) For project-specific configuration: ```json { "servers": { "nexus": { "type": "http", "url": "http://localhost:8000/mcp" } } } ``` Then in Claude Code: 1. Use the `/mcp` command 2. Select your server 3. Complete the OAuth2 flow in your browser if enabled in Nexus 4. Claude Code handles token management automatically ## VSCode Copilot with MCP VSCode's GitHub Copilot also supports MCP servers. ### User Settings Configuration Add to your VSCode user settings (`settings.json`): ```json { "github.copilot.mcp.servers": { "nexus": { "type": "http", "url": "http://localhost:8000/mcp" } } } ``` ## Cursor Configuration Cursor uses the standard MCP configuration format. Add to `.cursor/mcp.json` (project-specific) or `~/.cursor/mcp.json` (global): ```json { "mcpServers": { "nexus": { "command": "npx", "args": ["-y", "mcp-remote", "http://localhost:8000/mcp"] } } } ``` For remote servers with OAuth2, simply replace the URL: - Local: `http://localhost:8000/mcp` - Remote: `https://nexus.example.com/mcp` The `mcp-remote` adapter handles both local and remote connections, and will automatically manage OAuth2 authentication when required. --- # Usage - Using Nexus with Claude Code Nexus supports the Anthropic protocol natively, allowing you to use it as a proxy for Claude Code. This enables you to leverage Nexus's features like rate limiting, observability, MCP aggregation, and multi-provider routing while using Claude Code. ## Prerequisites - Nexus v0.5.0 or later (with Anthropic protocol support) - An Anthropic API key - Claude Code installed and configured ## Configuration ### Step 1: Configure Nexus Update your `nexus.toml` to enable the Anthropic protocol and configure the provider: ```toml [llm] enabled = true # Enable the Anthropic protocol endpoint [llm.protocols.anthropic] enabled = true path = "/llm/anthropic" # This is the default path # Configure the Anthropic provider [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" # Optional: Use a custom base URL (defaults to https://api.anthropic.com/v1) # base_url = "https://api.anthropic.com/v1" # Configure the models you want to use [llm.providers.anthropic.models."claude-sonnet-4-20250514"] [llm.providers.anthropic.models."claude-3-5-haiku-latest"] [llm.providers.anthropic.models."claude-opus-4-1-20250805"] ``` ### Step 2: Set Environment Variables Set your Anthropic API key: ```bash export ANTHROPIC_API_KEY="sk-ant-api03-..." ``` ### Step 3: Start Nexus ```bash nexus --config nexus.toml ``` By default, Nexus will listen on `http://localhost:6000`. ### Step 4: Configure Claude Code Set environment variables to point Claude Code to your Nexus instance: ```bash # Point Claude Code to your Nexus instance export ANTHROPIC_BASE_URL="http://localhost:6000/llm/anthropic" # Specify the model with the provider prefix export ANTHROPIC_MODEL="anthropic/claude-3-5-sonnet-20241022" ``` ### Step 5: Use Claude Code Now you can use Claude Code normally, and all requests will be routed through Nexus: ```bash claude "Explain how to implement a binary search tree" ``` ## Advanced Configuration ### Using OpenAI You can configure other language models, for example OpenAI GPT-5 to work with Claude Code: ```toml [llm] enabled = true [llm.protocols.anthropic] enabled = true path = "/llm/anthropic" [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" [llm.providers.openai.models."gpt-5"] ``` Set environment variables to point Claude Code to your Nexus instance: ```bash # Point Claude Code to your Nexus instance export ANTHROPIC_BASE_URL="http://localhost:6000/llm/anthropic" # Specify the model with the provider prefix export ANTHROPIC_MODEL="openai/gpt-5" ``` ## Docker Setup If you're running Nexus in Docker: ```yaml services: nexus: image: ghcr.io/grafbase/nexus:latest ports: - "6000:6000" volumes: - ./nexus.toml:/etc/nexus.toml environment: - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} ``` Then configure Claude Code to use the containerized Nexus: ```bash export ANTHROPIC_BASE_URL="http://localhost:6000/llm/anthropic" export ANTHROPIC_MODEL="anthropic/claude-3-5-sonnet-20241022" ``` ## Benefits Using Nexus with Claude Code provides: 1. **Unified Gateway**: Route all AI requests through a single endpoint 2. **Rate Limiting**: Control token consumption per user and model 3. **Observability**: Built-in OpenTelemetry metrics, traces, and logs 4. **Multi-Provider Support**: Switch between providers without changing Claude Code configuration 5. **Model Management**: Configure and manage models centrally 6. **Security**: Add authentication, CORS, and CSRF protection 7. **Cost Control**: Monitor and limit token usage ## Compatibility The Anthropic protocol implementation in Nexus: - Fully supports Claude Code's mixed content format (strings and arrays) - Handles tool calling and function definitions - Supports streaming responses ## Troubleshooting ### Connection Issues If Claude Code can't connect to Nexus: 1. Verify Nexus is running: ```bash curl http://localhost:6000/health ``` 2. Check the Anthropic protocol is enabled: ```bash curl http://localhost:6000/llm/anthropic/v1/models ``` 3. Verify environment variables: ```bash echo $ANTHROPIC_BASE_URL echo $ANTHROPIC_MODEL ``` ### Model Not Found If you get a "model not found" error: 1. Ensure the model is configured in `nexus.toml` 2. Use the correct format: `anthropic/model-name` 3. List available models: ```bash curl http://localhost:6000/llm/anthropic/v1/models ``` ### Authentication Errors If you get authentication errors: 1. Verify your API key is set correctly: ```bash echo $ANTHROPIC_API_KEY ``` 2. Check Nexus logs for more details: ```bash nexus --log debug ``` ## Adding Nexus MCP Server to Claude Code In addition to using Nexus as an LLM proxy, you can also connect Claude Code to Nexus's MCP (Model Context Protocol) server to access all the tools aggregated by Nexus. ### Configure MCP Server in Nexus First, ensure MCP is enabled in your `nexus.toml` and configure your MCP servers: ```toml [mcp] enabled = true path = "/mcp" # Default MCP endpoint # Example: Add GitHub MCP server [mcp.servers.github] url = "https://api.githubcopilot.com/mcp/" auth.token = "{{ env.GITHUB_TOKEN }}" # Example: Add filesystem MCP server [mcp.servers.filesystem] cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home/YOUR_USERNAME/Desktop"] # Example: Add a Python MCP server [mcp.servers.python_server] cmd = ["python", "-m", "mcp_server"] env = { PYTHONPATH = "/opt/mcp" } cwd = "/workspace" ``` ### Add Nexus MCP to Claude Code Use the Claude CLI to add Nexus as an MCP server: ```bash claude mcp add nexus http://localhost:6000/mcp --transport http ``` ### Verify MCP Connection After adding the MCP server, verify the connection: ```bash claude mcp list ``` ### Using MCP Tools in Claude Code Once connected, Claude Code can access all tools aggregated by Nexus through two main functions: 1. **Search for tools**: Claude can search across all connected MCP servers 2. **Execute tools**: Claude can execute specific tools with parameters Example interaction: ```bash claude "Use the filesystem tool to list files in my Desktop" # Claude will search for, and execute the filesystem tool through Nexus ``` ### Complete Setup Example Here's a complete example setting up both LLM proxy and MCP server: ```bash # 1. Start Nexus with both LLM and MCP configured nexus --config nexus.toml # 2. Configure Claude Code to use Nexus as LLM proxy export ANTHROPIC_BASE_URL="http://localhost:6000/llm/anthropic" export ANTHROPIC_MODEL="anthropic/claude-3-5-sonnet-20241022" # 3. Add Nexus MCP server to Claude Code claude mcp add nexus http://localhost:6000/mcp --transport http # 4. Verify both connections claude mcp test nexus claude "Hello, can you see my MCP tools?" ``` ### Benefits of Using Nexus MCP 1. **Tool Aggregation**: Access all your MCP servers through a single endpoint 2. **Intelligent Search**: Nexus provides context-aware fuzzy search across all tools 3. **Unified Management**: Configure all MCP servers in one place (nexus.toml) 4. **Authentication**: Nexus handles authentication to downstream MCP servers 5. **Observability**: Monitor all MCP tool usage through Nexus's telemetry ### Troubleshooting MCP Connection If Claude Code can't connect to Nexus MCP: 1. Verify Nexus is running and MCP is enabled: ```bash curl http://localhost:6000/mcp ``` 2. Check Claude Code's MCP configuration: ```bash claude mcp list ``` 3. Test the connection directly: ```bash claude mcp test nexus ``` 4. Check Nexus logs for MCP requests: ```bash nexus --log debug ``` ## Next Steps - Configure [MCP Servers](/docs/configuration/mcp) in Nexus - Set up [Rate Limiting](/docs/configuration/llm/rate-limiting) to control usage - Configure [Telemetry](/docs/configuration/telemetry) for observability - Explore [Header Rules](/docs/configuration/llm/header-rules) for advanced routing - Learn about [Token Forwarding](/docs/configuration/llm/token-forwarding) to let users provide their own API keys --- # Usage - Using Nexus with OpenAI Codex Nexus supports the OpenAI protocol natively, allowing you to use it as a proxy for OpenAI Codex CLI. This enables you to leverage Nexus's features like rate limiting, observability, and multi-provider routing while using Codex. ## Prerequisites - Nexus v0.5.1 or later - OpenAI Codex CLI installed ([github.com/openai/codex](https://github.com/openai/codex)) - API keys for your preferred providers (OpenAI, Anthropic, etc.) ## Configuration ### Step 1: Configure Nexus Update your `nexus.toml` to enable the OpenAI protocol and configure your providers: ```toml [llm] enabled = true # Enable the OpenAI protocol endpoint [llm.protocols.openai] enabled = true path = "/llm/openai" # This is the default path # Configure your providers [llm.providers.openai] type = "openai" api_key = "{{ env.OPENAI_API_KEY }}" [llm.providers.anthropic] type = "anthropic" api_key = "{{ env.ANTHROPIC_API_KEY }}" # Configure the models you want to use [llm.providers.openai.models."gpt-4-turbo-preview"] [llm.providers.openai.models."gpt-4"] [llm.providers.anthropic.models."claude-3-5-sonnet-20241022"] [llm.providers.anthropic.models."claude-3-5-haiku-latest"] ``` ### Step 2: Set Environment Variables Set your API keys: ```bash export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-api03-..." ``` ### Step 3: Start Nexus ```bash nexus --config nexus.toml ``` By default, Nexus will listen on `http://localhost:6000`. ### Step 4: Configure Codex CLI Configure Codex CLI to use Nexus by editing `~/.codex/config.toml`: > **Important**: Nexus serves its OpenAI-compatible endpoint at `/llm/openai/`; Codex expects the `/v1` suffix. Make sure the `base_url` ends with `/v1`. ```toml [model_providers.nexus] name = "Nexus AI router" base_url = "http://127.0.0.1:6000/llm/openai/v1" wire_api = "chat" query_params = {} ``` - `base_url` must point to the Nexus OpenAI-compatible endpoint and include the `/v1` suffix (adjust host/port if Nexus runs elsewhere) - `wire_api` should be set to `"chat"` for chat completions - `query_params` can stay empty, but the table must exist to satisfy Codex's schema ### Step 5: Use Codex with Nexus Start Codex with a Nexus-managed model: ```bash codex -c model="openai/gpt-4" -c model_provider=nexus ``` You can use any provider/model pair that you have configured in Nexus: ```bash # Use OpenAI models codex -c model="openai/gpt-4-turbo-preview" -c model_provider=nexus # Use Anthropic models through OpenAI-compatible interface codex -c model="anthropic/claude-3-5-haiku-latest" -c model_provider=nexus # Use other configured models codex -c model="groq/llama-3.1-70b-versatile" -c model_provider=nexus ``` ## Docker Setup If you're running Nexus in Docker: ```yaml services: nexus: image: ghcr.io/grafbase/nexus:latest ports: - "6000:6000" volumes: - ./nexus.toml:/etc/nexus.toml environment: - OPENAI_API_KEY=${OPENAI_API_KEY} - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} ``` Then configure Codex to use the containerized Nexus: ```toml [model_providers.nexus] name = "Nexus AI router" base_url = "http://localhost:6000/llm/openai/v1" wire_api = "chat" query_params = {} ``` ## Benefits Using Nexus with OpenAI Codex provides: 1. **Unified Gateway**: Route all AI requests through a single endpoint 2. **Multi-Provider Support**: Easily switch between OpenAI, Anthropic, and other providers 3. **Rate Limiting**: Control token consumption per user and model 4. **Observability**: Built-in OpenTelemetry metrics, traces, and logs 5. **Cost Control**: Monitor and limit token usage across all providers 6. **Model Management**: Configure and manage models centrally 7. **Security**: Add authentication, CORS, and CSRF protection ## Compatibility The OpenAI protocol implementation in Nexus: - Fully supports OpenAI's chat completions format - Handles streaming responses - Supports function calling (tools) ## Troubleshooting ### Connection Issues If Codex can't connect to Nexus: 1. Verify Nexus is running: ```bash curl http://localhost:6000/health ``` 2. Check the OpenAI protocol is enabled: ```bash curl http://localhost:6000/llm/openai/v1/models ``` 3. Verify your Codex configuration: ```bash cat ~/.codex/config.toml | grep -A 4 model_providers.nexus ``` ### Model Not Found If you get a "model not found" error: 1. Ensure the model is configured in `nexus.toml` 2. Use the correct format: `provider/model-name` 3. List available models: ```bash curl http://localhost:6000/llm/openai/v1/models | jq . ``` ### Authentication Errors If you get authentication errors: 1. Verify your API keys are set correctly: ```bash echo $OPENAI_API_KEY echo $ANTHROPIC_API_KEY ``` 2. Check Nexus logs for more details: ```bash nexus --log debug ``` ### Codex Configuration Issues If Codex fails to use the Nexus provider: 1. Ensure the `model_providers.nexus` section exists in `~/.codex/config.toml` 2. Verify the `base_url` ends with `/v1` 3. Check that `wire_api` is set to `"chat"` 4. Ensure `query_params` table exists (even if empty) ## Example Workflows ### Basic Usage ```bash # Start a Codex session with GPT-4 codex -c model="openai/gpt-4" -c model_provider=nexus # Execute a single command codex exec -c model="anthropic/claude-3-5-sonnet-20241022" -c model_provider=nexus "Write a Python function to calculate fibonacci" ``` ### Model Comparison Compare responses from different models: ```bash # Try with GPT-4 codex exec -c model="openai/gpt-4" -c model_provider=nexus "Explain quantum computing" # Try with Claude codex exec -c model="anthropic/claude-3-5-sonnet-20241022" -c model_provider=nexus "Explain quantum computing" # Try with Llama codex exec -c model="groq/llama-3.1-70b-versatile" -c model_provider=nexus "Explain quantum computing" ``` ## Next Steps - Configure [Rate Limiting](/docs/configuration/llm/rate-limiting) to control usage - Set up [Telemetry](/docs/configuration/telemetry) for observability - Explore [Header Rules](/docs/configuration/llm/header-rules) for advanced routing - Learn about [Token Forwarding](/docs/configuration/llm/token-forwarding) to let users provide their own API keys - Add [MCP Servers](/docs/configuration/mcp) to give Codex access to tools --- # Telemetry Nexus provides observability through OpenTelemetry, enabling you to monitor performance, track usage, and troubleshoot issues across all components. Our telemetry implementation follows OpenTelemetry semantic conventions for consistency and compatibility with existing monitoring tools. ## Telemetry Types 1. **[Metrics](/docs/telemetry/metrics)** - Quantitative measurements - HTTP request handling and Redis operations - LLM operations and token usage tracking - MCP tool execution and server health - Performance monitoring and cost optimization 2. **[Traces](/docs/telemetry/traces)** - Distributed request tracing - End-to-end request flow visualization - Cross-service correlation - Latency breakdown analysis - Configurable sampling rates 3. **[Logs](/docs/telemetry/logs)** - Structured application logs - Debug information and error details - Automatic trace and span correlation - Source code location attributes - Configurable log levels and filtering ## Quick Start To get started with telemetry: 1. **Configure** - Set up telemetry in your [configuration file](/docs/configuration/telemetry) 2. **Deploy** - Start Nexus with telemetry enabled 3. **Monitor** - Use the [available metrics](/docs/telemetry/metrics) in your observability platform 4. **Optimize** - Tune configuration based on your monitoring needs ### Basic Example ```toml [telemetry.exporters.otlp] enabled = true endpoint = "http://localhost:4317" ``` See the complete [telemetry configuration guide](/docs/configuration/telemetry) for all options. ## Integration Examples Nexus exports OpenTelemetry data that works with any OTLP-compatible backend: - **Prometheus** - Via OpenTelemetry Collector - **Grafana Cloud** - Direct OTLP ingestion - **Datadog** - Via Datadog Agent OTLP receiver - **AWS CloudWatch** - Via EMF exporter - **New Relic** - Direct OTLP endpoint - **Jaeger/Zipkin** - For distributed tracing See the [configuration guide](/docs/configuration/telemetry) for detailed integration examples. ## Common Issues 1. **Metrics not appearing** - Verify `[telemetry.exporters.otlp] enabled = true` - Check OTLP endpoint connectivity and protocol - Ensure your collector is running and configured - Review Nexus logs for export errors 2. **High cardinality warnings** - Review label usage in dashboards - Enable sampling for high-volume metrics - Disable detailed metrics if not needed 3. **Missing attributes** - Ensure resource attributes are configured - Check that service name and version are set - Verify custom headers are being sent ## Related Documentation - [Metrics](/docs/telemetry/metrics) - All available metrics documentation - [Traces](/docs/telemetry/traces) - Distributed tracing spans documentation - [Logs](/docs/telemetry/logs) - Structured application logs documentation - [Configuration Guide](/docs/configuration) - General Nexus configuration - [Troubleshooting](/docs/troubleshooting) - Common issues and solutions ## External Resources - [OpenTelemetry Documentation](https://opentelemetry.io/docs/) - [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) - [Prometheus Best Practices](https://prometheus.io/docs/practices/) - [Grafana Dashboard Gallery](https://grafana.com/grafana/dashboards/) --- # Telemetry - Metrics Nexus provides comprehensive OpenTelemetry metrics for monitoring all aspects of the system including server operations, LLM interactions, and MCP tool executions. All metrics follow OpenTelemetry semantic conventions and can be exported to any OpenTelemetry-compatible backend. **Notes**: - All histograms are **delta temporality** histograms that also function as counters (the count field tracks number of observations) - Delta histograms report the change since the last export, not cumulative values - Many metrics include `client.id` and `client.group` attributes when [Client Identification](/docs/configuration/server/client-identification) is enabled. These attributes allow you to track usage per client and implement tiered access controls. ## Configuration To enable metrics, configure telemetry in your `nexus.toml`: ```toml [telemetry.exporters.otlp] enabled = true endpoint = "http://localhost:4317" ``` See the complete [telemetry configuration guide](/docs/configuration/telemetry) for all options including: - Service identification and resource attributes - Protocol selection (gRPC vs HTTP) - Batch export optimization - Integration examples for popular backends ## Server Metrics ### HTTP Server Metrics #### Request Duration **Metric**: `http.server.request.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks the duration of HTTP server requests **Attributes**: - `http.request.method`: HTTP method (GET, POST, etc.) - `http.response.status_code`: HTTP response status code - `http.route`: The matched route pattern **Use Case**: Monitor API latency, identify slow endpoints, track error rates ### Redis Metrics Available when using Redis as the rate limiting backend. See [Rate Limiting Configuration](/docs/configuration/server/rate-limiting) for setup details. #### Command Duration **Metric**: `redis.command.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks execution time of Redis operations **Attributes**: - `operation`: Type of Redis operation - `check_and_consume`: HTTP rate limit checking - `check_and_consume_tokens`: Token-based rate limit checking - `status`: Operation status (`success` or `error`) - `tokens`: Number of tokens (only for token operations) **Use Case**: Monitor Redis performance, identify bottlenecks in rate limiting #### Connections In Use **Metric**: `redis.pool.connections.in_use` **Type**: Gauge **Description**: Current number of connections checked out from the pool **Attributes**: None **Use Case**: Monitor connection pool utilization #### Connections Available **Metric**: `redis.pool.connections.available` **Type**: Gauge **Description**: Current number of connections available in the pool **Attributes**: None **Use Case**: Ensure adequate pool capacity ## LLM Metrics Following OpenTelemetry GenAI semantic conventions for AI model operations. See [LLM Configuration](/docs/configuration/llm) for provider setup details. ### Operation Metrics #### Operation Duration **Metric**: `gen_ai.client.operation.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks the total duration of LLM chat completion operations **Attributes**: - `gen_ai.system`: Always "nexus.llm" - `gen_ai.operation.name`: Always "chat.completions" - `gen_ai.request.model`: The model identifier (e.g., "openai/gpt-4") - `gen_ai.response.finish_reason`: How the response ended (stop/length/tool_calls/content_filter) - `client.id`: Client identifier (from x-client-id header, see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (from x-client-group header, see [Client Identification](/docs/configuration/server/client-identification)) - `error.type`: Error type for failed requests: - `invalid_request` - Malformed request - `authentication_failed` - Invalid API key - `insufficient_quota` - Quota exceeded - `model_not_found` - Unknown model - `rate_limit_exceeded` - Provider or token rate limit hit - `streaming_not_supported` - Streaming unavailable for model - `invalid_model_format` - Incorrect model name format - `provider_not_found` - Unknown provider - `internal_error` - Server error - `provider_api_error` - Upstream provider error - `connection_error` - Network failure **Use Case**: Monitor LLM latency, compare performance across providers and models #### Time to First Token **Metric**: `gen_ai.client.time_to_first_token` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Duration until the first token is received in streaming responses **Attributes**: - `gen_ai.system`: Always "nexus.llm" - `gen_ai.operation.name`: Always "chat.completions" - `gen_ai.request.model`: The model identifier - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) **Use Case**: Monitor streaming response latency, critical for user experience ### Token Usage Metrics #### Input Token Usage **Metric**: `gen_ai.client.input.token.usage` **Type**: Counter **Description**: Cumulative count of input tokens consumed **Attributes**: - `gen_ai.system`: Always "nexus.llm" - `gen_ai.request.model`: The model identifier - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) #### Output Token Usage **Metric**: `gen_ai.client.output.token.usage` **Type**: Counter **Description**: Cumulative count of output tokens generated **Attributes**: - `gen_ai.system`: Always "nexus.llm" - `gen_ai.request.model`: The model identifier - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) #### Total Token Usage **Metric**: `gen_ai.client.total.token.usage` **Type**: Counter **Description**: Cumulative total tokens (input + output) **Attributes**: - `gen_ai.system`: Always "nexus.llm" - `gen_ai.request.model`: The model identifier - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) ## MCP Metrics Metrics for Model Context Protocol operations and tool executions. See [MCP Configuration](/docs/configuration/mcp) for server setup details. ### Tool Call Metrics #### Tool Call Duration **Metric**: `mcp.tool.call.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks the duration of MCP tool invocations including both built-in and downstream tools **Attributes**: - `tool_name`: Name of the tool being called - `tool_type`: Type of tool (`builtin` or `downstream`) - `status`: Operation status (`success` or `error`) - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) - Additional for search operations: - `keyword_count`: Number of keywords in search query - `result_count`: Number of results returned - Additional for execute operations on downstream tools: - `server_name`: Name of the downstream MCP server - Additional for errors: - `error.type`: Specific error type: - `parse_error` - Invalid JSON (-32700) - `invalid_request` - Not a valid request (-32600) - `method_not_found` - Method/tool does not exist (-32601) - `invalid_params` - Invalid method parameters (-32602) - `internal_error` - Internal server error (-32603) - `rate_limit_exceeded` - Rate limit hit (-32000) - `server_error` - Other server errors (-32001 to -32099) - `unknown` - Any other error code **Use Case**: Monitor tool performance, identify slow or failing tools, track usage patterns #### Tools List Duration **Metric**: `mcp.tools.list.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks the duration of listing available tools from MCP servers **Attributes**: - `method`: Always "list_tools" - `status`: Operation status (`success` or `error`) - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) **Use Case**: Monitor tool discovery performance and server responsiveness ### Prompt and Resource Metrics #### Prompt Request Duration **Metric**: `mcp.prompt.request.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks the duration of prompt-related operations (list/get) **Attributes**: - `method`: Operation type (`list_prompts` or `get_prompt`) - `status`: Operation status (`success` or `error`) - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) - Additional for errors: - `error.type`: Same error types as tool call duration **Use Case**: Monitor prompt template retrieval and listing performance #### Resource Request Duration **Metric**: `mcp.resource.request.duration` **Type**: Delta Histogram with Counter (milliseconds) **Description**: Tracks the duration of resource-related operations (list/read) **Attributes**: - `method`: Operation type (`list_resources` or `read_resource`) - `status`: Operation status (`success` or `error`) - `client.id`: Client identifier (see [Client Identification](/docs/configuration/server/client-identification)) - `client.group`: Client group (see [Client Identification](/docs/configuration/server/client-identification)) - Additional for errors: - `error.type`: Same error types as tool call duration **Use Case**: Monitor resource access patterns and performance ## Related Documentation - [Telemetry Configuration](/docs/configuration/telemetry) - Complete configuration guide - [Telemetry Overview](/docs/telemetry) - All telemetry types - [Server Configuration](/docs/configuration/server) - Server settings - [LLM Configuration](/docs/configuration/llm) - LLM provider setup - [MCP Configuration](/docs/configuration/mcp) - MCP server setup --- # Telemetry - Traces Nexus exports distributed traces using OpenTelemetry, providing detailed visibility into request flows across all components. Traces help you understand latency, identify bottlenecks, and debug issues by showing the complete execution path of each request. ## Configuration To enable tracing, configure telemetry in your `nexus.toml`: ```toml [telemetry.tracing] enabled = true sampling = 0.1 # Sample 10% of requests parent_based_sampler = false # Use parent-based sampling (default: false) ``` See the complete [telemetry configuration guide](/docs/configuration/telemetry#tracing-configuration) for all options including: - Sampling strategies and recommendations - Parent-based sampling for distributed systems - Trace context propagation (W3C and AWS X-Ray) - Integration with OTLP exporters - Performance tuning for different environments ## Span Hierarchy Nexus creates a hierarchical span structure that represents the complete request flow: ``` HTTP Request (root span) ├── Redis Rate Limit Check - HTTP-level rate limits ├── MCP Operation │ ├── Redis Rate Limit Check - MCP-specific rate limits │ ├── Tool Search │ └── Tool Execution └── LLM Operation └── Token Rate Limit Check (redis:check_and_consume_tokens) - Token-based rate limits ``` ## Available Spans ### HTTP Request Span The root span for all incoming HTTP requests. **Span Name**: `{method} {path}` (e.g., `POST /mcp`, `GET /health`) **Attributes**: - `http.request.method` - HTTP method (GET, POST, etc.) - `http.route` - Request route pattern - `http.response.status_code` - Response status code - `http.request.body.size` - Request body size in bytes - `http.response.body.size` - Response body size in bytes - `client.id` - Client identifier from rate limiting - `client.group` - Client group for rate limiting - `url.path` - Full URL path - `url.query` - Query parameters - `user_agent.original` - Client user agent string ### MCP Operation Spans Spans for Model Context Protocol operations. **Span Name**: MCP method names like `tools/list`, `tools/call` **Attributes**: - `mcp.method` - MCP method name (tools/list, tools/call, etc.) - `mcp.tool.name` - Tool being called - `mcp.tool.type` - Tool type (builtin/downstream) - `mcp.transport` - Transport type (stdio/http) - `mcp.auth_forwarded` - Whether auth was forwarded - `client.id` - Client identifier (if configured) - `client.group` - Client group (if configured) - `mcp.error.code` - Error code if operation failed ### LLM Operation Spans Spans for Language Model operations. **Span Name**: `llm:chat_completion` or `llm:chat_completion_stream` **Attributes**: - `gen_ai.request.model` - Model identifier - `gen_ai.request.max_tokens` - Max tokens requested - `gen_ai.request.temperature` - Temperature parameter - `gen_ai.request.has_tools` - Whether tools were provided - `gen_ai.request.tool_count` - Number of tools provided - `gen_ai.response.model` - Model used for response - `gen_ai.response.finish_reason` - Completion reason - `gen_ai.usage.input_tokens` - Input token count - `gen_ai.usage.output_tokens` - Output token count - `gen_ai.usage.total_tokens` - Total token count - `llm.stream` - Whether streaming was used - `llm.auth_forwarded` - Whether auth was forwarded - `client.id` - Client identifier (if configured) - `client.group` - Client group (if configured) - `error.type` - Error type if operation failed ### Redis Operation Spans Spans for Redis operations including rate limiting. **Span Names**: - `redis:check_and_consume:global` - Global rate limit - `redis:check_and_consume:ip` - Per-IP rate limit - `redis:check_and_consume:server` - Per-server rate limit - `redis:check_and_consume:tool` - Per-tool rate limit - `redis:check_and_consume_tokens` - Token-based rate limit **Attributes**: - `redis.operation` - Operation type (check_and_consume or check_and_consume_tokens) - `rate_limit.scope` - Scope (global/ip/server/tool/token) - `rate_limit.limit` - Request/token limit - `rate_limit.interval_ms` - Time window in milliseconds - `rate_limit.tokens` - Number of tokens (for token operations) - `rate_limit.allowed` - Whether request was allowed - `rate_limit.retry_after_ms` - Retry delay if rate limited - `redis.pool.size` - Connection pool size - `redis.pool.available` - Available connections - `redis.pool.in_use` - Connections in use - `client.address_hash` - Hashed IP for privacy (per-IP limits) - `llm.provider` - Provider name (token limits) - `llm.model` - Model name (token limits) ## Trace Context Propagation Nexus supports multiple trace context propagation formats: - **W3C Trace Context** (default): Standard trace propagation using `traceparent` and `tracestate` headers - **AWS X-Ray**: For AWS environments using `X-Amzn-Trace-Id` header format See the [telemetry configuration guide](/docs/configuration/telemetry#tracing-configuration) for propagation setup. ## Parent-Based Sampling Parent-based sampling ensures consistent trace sampling across distributed systems. When enabled, Nexus respects the sampling decision made by upstream services. ### Configuration ```toml [telemetry.tracing] sampling = 0.15 # Local sampling rate (15%) parent_based_sampler = true # Enable parent-based sampling ``` ### Sampling Behavior **When `parent_based_sampler = true`:** - If an incoming request has a sampled parent trace, Nexus will sample it - If an incoming request has an unsampled parent trace, Nexus will not sample it - If no parent trace exists, Nexus uses the local `sampling` rate **When `parent_based_sampler = false` (default):** - Nexus always uses the local `sampling` rate - Parent trace sampling decisions are ignored ### Use Cases Parent-based sampling is recommended for: - **Microservice architectures**: Ensures complete traces across all services - **API gateways**: Maintains sampling consistency from edge to backend - **Multi-tier applications**: Prevents partial traces with missing segments ### Example Scenarios 1. **Complete distributed traces**: With parent-based sampling, if a frontend service samples a request, all downstream services (including Nexus) will also sample it, ensuring a complete trace. 2. **Consistent sampling**: In a chain of services A → B → Nexus, if A decides to sample, both B and Nexus will honor that decision, preventing trace fragmentation. 3. **Fallback behavior**: For direct requests to Nexus without parent context, the local sampling rate applies. ## Backend Integration Traces are exported via the OTLP exporter to any compatible backend. See the [telemetry configuration guide](/docs/configuration/telemetry#integration-examples) for detailed setup with: - Jaeger - Grafana Tempo - AWS X-Ray - Datadog - New Relic - Zipkin ## Common Trace Queries ### Finding Slow Requests Look for spans with high duration: - Filter: `duration > 1s` - Group by: `http.route` ### Error Investigation Find failed requests: - Filter: `http.response.status_code >= 500` - Or: `error = true` ### Token Usage Analysis Track LLM token consumption: - Filter: `span.name = "LLM*"` - Sum: `gen_ai.usage.total_tokens` ### Rate Limit Analysis Find rate-limited requests: - Filter: `rate_limit.blocked = true` - Group by: `client.id`, `rate_limit.scope` ## Performance Considerations - **Sampling rates**: Use 1-5% in production, higher in development - **Span cardinality**: High-cardinality attributes are automatically limited - **Network overhead**: Controlled via batch export settings See the [telemetry configuration guide](/docs/configuration/telemetry#performance-tuning) for optimization strategies. ## Troubleshooting Common issues and solutions are covered in the [telemetry configuration guide](/docs/configuration/telemetry#troubleshooting), including: - Traces not appearing - Missing spans - Context propagation issues - Performance optimization ## Related Documentation - [Telemetry Overview](/docs/telemetry) - General telemetry concepts - [Metrics](/docs/telemetry/metrics) - Available metrics - [Configuration Guide](/docs/configuration/telemetry) - Full configuration reference ## External Resources - [OpenTelemetry Tracing](https://opentelemetry.io/docs/concepts/signals/traces/) - [W3C Trace Context](https://www.w3.org/TR/trace-context/) - [Jaeger Documentation](https://www.jaegertracing.io/docs/) - [Grafana Tempo](https://grafana.com/docs/tempo/latest/) --- # Telemetry - Logs Nexus exports structured logs using OpenTelemetry, providing detailed application-level insights with automatic trace and span correlation. Logs help you debug issues, audit operations, and understand system behavior with full context from distributed traces. ## Configuration See the [telemetry configuration guide](/docs/configuration/telemetry#logs-configuration) for detailed setup instructions including: - OTLP exporter configuration for logs - Batch export optimization - Integration with logging backends ## Log Levels and Filtering ### Log Levels Use the `--log` flag or `NEXUS_LOG` environment variable to control log verbosity. This applies to all spans, logs, and trace events: | Level | Description | |-------|-------------| | `off` | Disable logging | | `error` | Only log errors | | `warn` | Log errors and warnings | | `info` | Log errors, warnings, and info messages (default) | | `debug` | Log errors, warnings, info, and debug messages | | `trace` | Log errors, warnings, info, debug, and trace messages | ```bash # Using command-line flag nexus --log info # Default level nexus --log debug # Development debugging nexus --log trace # Maximum verbosity nexus --log off # Disable all logging # Using environment variable NEXUS_LOG=debug nexus # Granular per-module configuration nexus --log "nexus=debug,tower_http=info" # Filter out noisy dependencies nexus --log "info,hyper=warn,h2=warn" ``` ### Log Output Styles Configure the output format using the `--log-style` flag or `NEXUS_LOG_STYLE` environment variable: | Style | Description | |-------|-------------| | `color` | Colorized text (default with TTY output) | | `text` | Standard text (default with non-TTY output) | | `json` | JSON objects for structured logging | ```bash # Using command-line flag nexus --log-style color # Colorized for terminal nexus --log-style text # Plain text nexus --log-style json # Structured JSON # Using environment variable NEXUS_LOG_STYLE=json nexus # Combine log level and style nexus --log debug --log-style json ``` The JSON output format is particularly useful for: - Log aggregation systems - Automated log parsing - Production environments where structured logs are required **Important**: The OpenTelemetry layer internally filters its own logs to prevent recursion. You'll never see OpenTelemetry export logs in the exported logs themselves. ## Log Attributes All logs include standard OpenTelemetry attributes plus Nexus-specific context: ### Standard Attributes - `timestamp` - Log record timestamp - `severity_number` - Numeric severity level - `severity_text` - Text representation (ERROR, WARN, INFO, DEBUG, TRACE) - `body` - The log message content - `observed_timestamp` - When the log was observed ### Source Location Attributes - `code.filepath` - Source file path - `code.lineno` - Line number in source - `code.namespace` - Rust module path ### Trace Correlation When logs are emitted within an active span context: - `trace_id` - Associated trace identifier - `span_id` - Associated span identifier ### Resource Attributes From telemetry configuration: - `service.name` - Service identifier - Custom attributes from `[telemetry.resource_attributes]` ## Log Categories Nexus and its dependencies emit logs across various categories: ### HTTP Request Logs Request handling and response logs from the HTTP server layer. ### LLM Operation Logs Language model interactions including token usage, rate limiting, and provider errors. ### MCP Tool Logs Model Context Protocol operations including tool discovery, execution, and errors. ### Rate Limiting Logs Rate limit enforcement and threshold notifications. ### Redis Operation Logs When using Redis backend, connection pool and operation logs. ## Correlation with Traces Logs are automatically correlated with active traces and spans. When a log is emitted within an active span context, it includes the trace and span IDs: ```json { "timestamp": "2024-01-15T10:30:45.123Z", "severity_text": "INFO", "body": "Processing LLM request", "trace_id": "7a5d2e3f8b9c1d4e6f8a9b0c1d2e3f4a", "span_id": "3e4f5a6b7c8d9e0f", "attributes": { "code.namespace": "nexus::llm::handler", "code.filepath": "src/llm/handler.rs", "code.lineno": 145 } } ``` This correlation enables: - Viewing logs in context of distributed traces - Filtering logs by trace or span ID - Understanding log sequences within request flows - Root cause analysis with full context ## Backend Integration Logs are exported via OTLP to any compatible backend. Popular integrations include: ### Grafana Loki Via OpenTelemetry Collector: ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: loki: endpoint: http://loki:3100/loki/api/v1/push labels: attributes: service_name: "service.name" level: "severity_text" service: pipelines: logs: receivers: [otlp] exporters: [loki] ``` ### Elasticsearch Via OpenTelemetry Collector: ```yaml exporters: elasticsearch: endpoints: [http://elasticsearch:9200] logs_index: nexus-logs ``` ### CloudWatch Logs Via AWS OpenTelemetry Collector: ```yaml exporters: awscloudwatchlogs: region: us-east-1 log_group_name: /aws/nexus log_stream_name: production ``` ## Common Log Queries ### Finding Errors Search for all error-level logs: - Filter: `severity_text = "ERROR"` - Time range: Last 1 hour - Group by: `code.namespace` ### Tracking Specific Requests Find all logs for a trace: - Filter: `trace_id = "7a5d2e3f8b9c1d4e6f8a9b0c1d2e3f4a"` - Sort: By timestamp ascending ### Module-Specific Debugging Filter logs by module: - Filter: `code.namespace starts_with "nexus::mcp"` - Severity: DEBUG or higher ## Performance Considerations ### Log Volume Management Control log volume with appropriate filtering: ```bash # Production: INFO and above nexus --log info # Development: DEBUG for nexus, INFO for dependencies nexus --log "nexus=debug,info" # Minimal: Only warnings and errors nexus --log warn # Disable all logging for minimal overhead nexus --log off ``` ### Zero Overhead When Disabled When telemetry is not configured, the logging layer has zero overhead - logs are not collected or processed. ## Best Practices 1. **Use Appropriate Log Levels** - ERROR: System failures requiring intervention - WARN: Anomalies that need investigation - INFO: Key business events and milestones - DEBUG: Detailed operational information - TRACE: Very verbose debugging data 2. **Leverage Structured Logging** - Logs include consistent structured attributes - Use trace correlation for debugging - Filter by module namespace for targeted analysis 3. **Configure for Your Environment** - Production: `--log info` with `--log-style json` for structured logging - Staging: `--log "nexus=debug,info"` for detailed Nexus logs - Development: `--log debug` or `--log trace` with `--log-style color` ## Troubleshooting ### Logs Not Appearing 1. **Verify Configuration**: Ensure logs export is enabled in [telemetry configuration](/docs/configuration/telemetry#logs-configuration) 2. **Check Log Level**: ```bash # Ensure log level allows desired logs nexus --log info ``` 3. **Validate Export**: Monitor Nexus startup logs for export confirmation ### Missing Trace Correlation - Ensure tracing is enabled in configuration - Verify trace propagation headers are being sent - Check that spans are active when logs are emitted ### High Log Volume - Adjust `--log` flag to filter unnecessary logs - Use module-specific log levels - Configure batch export for better throughput ## Related Documentation - [Telemetry Overview](/docs/telemetry) - General telemetry concepts - [Metrics](/docs/telemetry/metrics) - Available metrics - [Traces](/docs/telemetry/traces) - Distributed tracing - [Configuration Guide](/docs/configuration/telemetry) - Full configuration reference ## External Resources - [OpenTelemetry Logs](https://opentelemetry.io/docs/concepts/signals/logs/) - [OTLP Log Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) - [Grafana Loki](https://grafana.com/docs/loki/latest/) - [Elasticsearch Logging](https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html) --- # Troubleshooting Find solutions to common problems and learn how to debug issues with Nexus. ## Quick Debugging ### Enable Debug Logging ```bash nexus --log debug ``` ### Check Server Status ```bash curl http://localhost:8000/health ``` ### Test LLM Connection ```bash curl http://localhost:8000/llm/openai/v1/models ``` ### Common Error Codes | Code | Meaning | Common Causes | |------|---------|--------------| | 401 | Unauthorized | Invalid or missing authentication token | | 403 | Forbidden | Insufficient permissions or CSRF failure | | 404 | Not Found | Model not configured or endpoint doesn't exist | | 429 | Too Many Requests | Rate limit exceeded | | 500 | Internal Server Error | Server configuration error or provider issue | | 502 | Bad Gateway | Downstream server unreachable | | 503 | Service Unavailable | Server overloaded or starting up | ## Getting Help 1. **Check the logs** with `--log debug` flag 2. **Review configuration** for typos or missing values 3. **Test components individually** before full integration 4. **Search existing issues** on GitHub 5. **Report new issues** with reproduction steps ## Next Steps - Check [best practices](/docs/best-practices) to avoid common issues --- # Best Practices Follow these recommended patterns and practices for secure, scalable, and maintainable Nexus deployments. ## Core Areas ### Security - **[OAuth2 Authentication](/docs/configuration/server/oauth2)** - Secure API access with JWT tokens - **[TLS Configuration](/docs/configuration/mcp/tls)** - Encrypted connections and certificates - **[Client Identification](/docs/configuration/server/client-identification)** - User tracking and tiered access ### Performance - **[Rate Limiting](/docs/configuration/server/rate-limiting)** - Global and per-user limits - **[LLM Rate Limiting](/docs/configuration/llm/rate-limiting)** - Token-based rate limiting - **[MCP Rate Limiting](/docs/configuration/mcp/rate-limiting)** - Tool usage controls ### Configuration - **[Server Configuration](/docs/configuration/server)** - Core server settings - **[LLM Configuration](/docs/configuration/llm)** - Provider and model management - **[MCP Configuration](/docs/configuration/mcp)** - Tool server setup ## Production Readiness Checklist ### Security - OAuth2 authentication enabled - TLS certificates configured - Secrets in environment variables - Rate limiting configured - CORS properly restricted ### Performance - Redis for distributed rate limiting - Connection caching optimized - Appropriate timeouts set - Resource limits defined ### Operations - Health checks configured - Logging at appropriate level - Monitoring in place - Backup strategy defined - Update process documented ## Key Principles ### 1. Security First - Never hardcode secrets - Always use TLS in production - Implement defense in depth - Follow principle of least privilege ### 2. Start Conservative - Begin with strict rate limits - Use minimal permissions - Enable features gradually - Monitor before scaling ### 3. Plan for Scale - Design for horizontal scaling - Use distributed storage (Redis) - Implement caching strategies - Monitor resource usage ### 4. Operational Excellence - Automate deployments - Version control everything - Document configurations - Test disaster recovery ## Configuration Patterns ### Environment Separation ```toml # Development [server] listen_address = "127.0.0.1:8000" # Production [server] listen_address = "0.0.0.0:443" [server.tls] certificate = "/etc/nexus/cert.pem" key = "/etc/nexus/key.pem" ``` ### Tiered Access Control ```toml [server.client_identification] enabled = true client_id.jwt_claim = "sub" group_id.jwt_claim = "plan" [llm.providers.openai.rate_limits.per_user.groups.free] input_token_limit = 10000 interval = "3600s" [llm.providers.openai.rate_limits.per_user.groups.pro] input_token_limit = 100000 interval = "3600s" ``` ## Common Pitfalls to Avoid 1. **Hardcoding secrets** - Always use environment variables 2. **Skipping TLS** - Never run without TLS in production 3. **No rate limiting** - Always protect against abuse 4. **Ignoring logs** - Monitor and act on warnings 5. **No testing** - Test configurations before deployment 6. **Poor documentation** - Document all custom configurations ## Next Steps - Configure [OAuth2 Authentication](/docs/configuration/server/oauth2) - Set up [Rate Limiting](/docs/configuration/server/rate-limiting) - Review [Troubleshooting Guide](/docs/troubleshooting) - Explore [Usage Examples](/docs/usage)