nxtgauge-gitops/apps/litellm/README.md

# LiteLLM Setup for Nxtgauge

## Overview
LiteLLM is deployed as an API gateway to the internal Ollama service, providing OpenAI-compatible API endpoints.

## Architecture
- **Ollama**: Internal ClusterIP service at `ollama.nxtgauge-ai.svc.cluster.local:11434`
- **LiteLLM**: ClusterIP service at `10.43.7.24:4000` (internal)
- **Ingress**: Exposed at `https://llm.nxtgauge.com/v1` via Traefik
- **Security**: API key required via `LITELLM_MASTER_KEY`

## Master Key
**Production Master Key**: `sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9`

⚠️ **IMPORTANT**: Store this key securely. It grants full access to the LLM API.

## Model Aliases
All aliases route to the same Ollama model (`gemma3:270m`):

| Alias | Model | Timeout | Retries |
|-------|-------|---------|---------|
| `askash-main` | ollama/gemma3:270m | 300s | 2 |
| `askash-fast` | ollama/gemma3:270m | 120s | 1 |
| `coding-main` | ollama/gemma3:270m | 300s | 2 |

## Files Created

### Base Configuration
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/configmap.yaml` - LiteLLM config
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/secret.yaml` - Secret template
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/deployment.yaml` - Deployment spec
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/service.yaml` - Service spec
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/ingress.yaml` - Ingress with TLS
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/ratelimit.yaml` - Rate limiting middleware
- `/home/ashwin/nxtgauge-gitops/apps/litellm/base/kustomization.yaml` - Base kustomization

### Production Overlay
- `/home/ashwin/nxtgauge-gitops/apps/litellm/overlays/prod/kustomization.yaml` - Production overlay with secure key

## Testing Commands

### List Available Models
```bash
curl https://llm.nxtgauge.com/v1/models \
  -H "Authorization: Bearer sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9"
```

### Test Chat Completion
```bash
curl https://llm.nxtgauge.com/v1/chat/completions \
  -H "Authorization: Bearer sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "askash-main",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from Ask Ash"
      }
    ]
  }'
```

### Test from Inside Cluster
```bash
kubectl run test-curl --rm -i --restart=Never --image=curlimages/curl:latest -- \
  http://litellm.nxtgauge-ai.svc.cluster.local:4000/v1/models \
  -H "Authorization: Bearer sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9"
```

## OpenCode Configuration

Create/edit `~/.config/opencode/opencode.json`:

```json
{
  "baseURL": "https://llm.nxtgauge.com/v1",
  "apiKey": "sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9",
  "models": {
    "default": "askash-main",
    "available": [
      "askash-main",
      "askash-fast",
      "coding-main"
    ]
  }
}
```

## Ask Ash Local Environment

Create `.env.local` in your Ask Ash project:

```bash
# LiteLLM Provider Configuration
LLM_PROVIDER=openai_compatible
OPENAI_BASE_URL=https://llm.nxtgauge.com/v1
OPENAI_API_KEY=sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9
LLM_MODEL=askash-main
AI_DEBUG=true
```

## Promptfoo Configuration

Create `promptfooconfig.yaml`:

```yaml
providers:
  - id: openai
    config:
      apiBaseUrl: https://llm.nxtgauge.com/v1
      apiKey: sk-litellm-prod-1c66d63e701c32cd85922a62fd2e087469486a9b7a34d950423a8726d0aceec9
      model: askash-main
```

## DNS Configuration (Cloudflare)

Add DNS record in Cloudflare:
- **Type**: A
- **Name**: llm
- **Content**: Your cluster external IP (check with `kubectl get svc -n kube-system traefik`)
- **TTL**: Auto
- **Proxy Status**: DNS only (for testing), then enable after TLS works

## Security Notes

1. ✅ Ollama is NOT exposed publicly (ClusterIP only)
2. ✅ LiteLLM requires API key authentication
3. ✅ TLS enabled via cert-manager/Let's Encrypt
4. ✅ Rate limiting enabled (100 req/min avg, 50 burst)
5. ⚠️ Master key is stored in Kubernetes Secret (not in Git)

## Management Commands

```bash
# Check LiteLLM status
kubectl get pods -n nxtgauge-ai -l app=litellm

# View logs
kubectl logs -n nxtgauge-ai -l app=litellm --tail=100 -f

# Restart LiteLLM
kubectl rollout restart deployment litellm -n nxtgauge-ai

# Get master key
kubectl get secret litellm-secrets -n nxtgauge-ai -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d

# Edit configuration
kubectl edit configmap litellm-config -n nxtgauge-ai

# Port-forward for local testing
kubectl port-forward svc/litellm 4000:4000 -n nxtgauge-ai
```

## Troubleshooting

### Pod not starting
```bash
kubectl describe pod -n nxtgauge-ai -l app=litellm
kubectl logs -n nxtgauge-ai -l app=litellm --previous
```

### Ollama unreachable
```bash
kubectl get svc -n nxtgauge-ai ollama
kubectl exec -n nxtgauge-ai -it ollama-76fb847d46-b7pkp -- ollama list
```

### TLS not working
```bash
kubectl get certificate -n nxtgauge-ai
kubectl describe certificate -n nxtgauge-ai litellm-tls
```

## Future Enhancements

1. Add more Ollama models and configure model-specific aliases
2. Implement request/response caching with Redis
3. Add Prometheus metrics for observability
4. Configure team-based API keys for multi-user access
5. Add request logging and usage analytics
6. Implement token-based billing/quota management

---

**Deployed**: 2026-06-14
**Namespace**: nxtgauge-ai
**Service**: litellm (10.43.7.24:4000)
**Ingress**: https://llm.nxtgauge.com