nxtgauge-gitops/apps/litellm

LiteLLM Setup for Nxtgauge

Overview

LiteLLM is deployed as an API gateway to the internal Ollama service, providing OpenAI-compatible API endpoints.

Architecture

• Ollama: Internal ClusterIP service at ollama.nxtgauge-ai.svc.cluster.local:11434
• LiteLLM: ClusterIP service at 10.43.7.24:4000 (internal)
• Ingress: Exposed at https://llm.nxtgauge.com/v1 via Traefik
• Security: API key required via LITELLM_MASTER_KEY

Master Key

Production Master Key: <REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>

⚠️ IMPORTANT: Store this key securely. It grants full access to the LLM API.

Model Aliases

All aliases route to the same Ollama model (gemma3:270m):

Alias	Model	Timeout	Retries
askash-main	ollama/gemma3:270m	300s	2
askash-fast	ollama/gemma3:270m	120s	1
coding-main	ollama/gemma3:270m	300s	2

Files Created

Base Configuration

• /home/ashwin/nxtgauge-gitops/apps/litellm/base/configmap.yaml - LiteLLM config
• /home/ashwin/nxtgauge-gitops/apps/litellm/base/secret.yaml - Secret template
• /home/ashwin/nxtgauge-gitops/apps/litellm/base/deployment.yaml - Deployment spec
• /home/ashwin/nxtgauge-gitops/apps/litellm/base/service.yaml - Service spec
• /home/ashwin/nxtgauge-gitops/apps/litellm/base/ingress.yaml - Ingress with TLS
• /home/ashwin/nxtgauge-gitops/apps/litellm/base/ratelimit.yaml - Rate limiting middleware
• /home/ashwin/nxtgauge-gitops/apps/litellm/base/kustomization.yaml - Base kustomization

Production Overlay

• /home/ashwin/nxtgauge-gitops/apps/litellm/overlays/prod/kustomization.yaml - Production overlay with secure key

Testing Commands

List Available Models

curl https://llm.nxtgauge.com/v1/models \
  -H "Authorization: Bearer <REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>"

Test Chat Completion

curl https://llm.nxtgauge.com/v1/chat/completions \
  -H "Authorization: Bearer <REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "askash-main",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from Ask Ash"
      }
    ]
  }'

Test from Inside Cluster

kubectl run test-curl --rm -i --restart=Never --image=curlimages/curl:latest -- \
  http://litellm.nxtgauge-ai.svc.cluster.local:4000/v1/models \
  -H "Authorization: Bearer <REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>"

OpenCode Configuration

Create/edit ~/.config/opencode/opencode.json:

{
  "baseURL": "https://llm.nxtgauge.com/v1",
  "apiKey": "<REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>",
  "models": {
    "default": "askash-main",
    "available": [
      "askash-main",
      "askash-fast",
      "coding-main"
    ]
  }
}

Ask Ash Local Environment

Create .env.local in your Ask Ash project:

# LiteLLM Provider Configuration
LLM_PROVIDER=openai_compatible
OPENAI_BASE_URL=https://llm.nxtgauge.com/v1
OPENAI_API_KEY=<REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>
LLM_MODEL=askash-main
AI_DEBUG=true

Promptfoo Configuration

Create promptfooconfig.yaml:

providers:
  - id: openai
    config:
      apiBaseUrl: https://llm.nxtgauge.com/v1
      apiKey: <REDACTED - see nxtgauge-litellm-secrets Secret, key LITELLM_MASTER_KEY>
      model: askash-main

DNS Configuration (Cloudflare)

Add DNS record in Cloudflare:

• Type: A
• Name: llm
• Content: Your cluster external IP (check with kubectl get svc -n kube-system traefik)
• TTL: Auto
• Proxy Status: DNS only (for testing), then enable after TLS works

Security Notes

1. ✅ Ollama is NOT exposed publicly (ClusterIP only)
2. ✅ LiteLLM requires API key authentication
3. ✅ TLS enabled via cert-manager/Let's Encrypt
4. ✅ Rate limiting enabled (100 req/min avg, 50 burst)
5. ⚠️ Master key is stored in Kubernetes Secret (not in Git)

Management Commands

# Check LiteLLM status
kubectl get pods -n nxtgauge-ai -l app=litellm

# View logs
kubectl logs -n nxtgauge-ai -l app=litellm --tail=100 -f

# Restart LiteLLM
kubectl rollout restart deployment litellm -n nxtgauge-ai

# Get master key
kubectl get secret litellm-secrets -n nxtgauge-ai -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d

# Edit configuration
kubectl edit configmap litellm-config -n nxtgauge-ai

# Port-forward for local testing
kubectl port-forward svc/litellm 4000:4000 -n nxtgauge-ai

Troubleshooting

Pod not starting

kubectl describe pod -n nxtgauge-ai -l app=litellm
kubectl logs -n nxtgauge-ai -l app=litellm --previous

Ollama unreachable

kubectl get svc -n nxtgauge-ai ollama
kubectl exec -n nxtgauge-ai -it ollama-76fb847d46-b7pkp -- ollama list

TLS not working

kubectl get certificate -n nxtgauge-ai
kubectl describe certificate -n nxtgauge-ai litellm-tls

Future Enhancements

1. Add more Ollama models and configure model-specific aliases
2. Implement request/response caching with Redis
3. Add Prometheus metrics for observability
4. Configure team-based API keys for multi-user access
5. Add request logging and usage analytics
6. Implement token-based billing/quota management

---

Deployed: 2026-06-14 Namespace: nxtgauge-ai Service: litellm (10.43.7.24:4000) Ingress: https://llm.nxtgauge.com