docker system prune --volumes can never touch the buildx builder's
cache volume - it's attached to a running container, and 'volumes'
prune only removes *unattached* ones. This CronJob ran every 30
minutes for 59 days reporting 'Total reclaimed space: 0B' every
single time while each runner's buildx cache silently grew to
57-98GB (discovered chasing a disk-pressure report on nxtgauge-2/3/4).
Fix: attach the real named builder ('nxtgauge-builder', matching
what CI actually uses) first, then call 'docker buildx prune'
directly, capped at --keep-storage 20GB so it doesn't just regrow
unbounded. Verified live: manually pruned gwsh7 (80%->16% node disk)
and ktst5 (84%->47%), then confirmed the patched CronJob is a true
no-op on an already-clean cache.
Also found and cleaned up (host-level, not gitops - out of band from
k8s): 18 orphaned buildx_buildkit_* containers on nxtgauge-1 from 2
months of ad-hoc 'docker buildx create' calls with no --name reuse,
totally invisible to buildx CLI and unrelated to CI (39.63GB, node
went 76%->50%). Added a daily cron job on that host
(~/.local/bin/docker-cleanup.sh) to keep it from reaccumulating,
since that's the host's standalone Docker daemon, outside k8s/gitops
entirely.
This CronJob + its RBAC (docker-prune-sa/-role/-binding) previously
existed only as manually-applied live objects, not tracked in git -
same pattern as nxtgauge-containerd-cleanup. Adding manifests and
wiring into clusters/production so Flux manages it going forward.
The prune step called bare 'ctr', which has never existed as a
standalone binary on these k3s nodes - only the k3s binary itself
(bundling 'k3s ctr'/'k3s crictl' subcommands) lives at
/usr/local/bin. The failure was silently swallowed by '|| true',
so the CronJob has been reporting Complete every 10 minutes for
68 days while doing nothing; node disk sat at 83% used.
Verified live: patched the CronJob's command to invoke
/usr/local/bin/k3s ctr directly, manually triggered a run, confirmed
it actually prunes now (deleted ~75 dangling images).
This CronJob and its ServiceAccount previously existed only as a
manually-applied live object, not tracked in git. Adding the
manifests here and wiring them into clusters/production so Flux
manages it going forward instead of it silently drifting again.
Forgot to stage these in 643ab49 - the actual ingress host rules and
the removal of the apps/nxtgauge-coming-soon reference from
clusters/production/kustomization.yaml, without which nothing in the
previous commit actually took effect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The static page (coming-soon/) has existed in this repo since 167ea12
but was never actually wired up - its own README says as much ('Add
this repo to your Kustomize setup and create a Deployment + Service +
Ingress'). No ingress for the bare nxtgauge.com/www.nxtgauge.com
domain existed at all (only subdomains: api/admin/test111/llm/ci/logs).
Deploys the page as a stock nginx:1.27-alpine container (no custom
image/CI build needed - the whole site is ~183KB of static HTML/PNG/
SVG) with the content mounted via two ConfigMaps (configmap-html for
index.html + nginx.conf, configmap-assets for the logo/trademark
files - split because ConfigMap keys can't contain '/', so the
assets/ subdirectory needed its own map). 2 replicas, standard
readiness/liveness probes, ingress for both nxtgauge.com and
www.nxtgauge.com terminating TLS via the existing letsencrypt-prod
ClusterIssuer.
Confirmed DNS for both hostnames already resolves through the same
Cloudflare-proxied IPs as the working test111.nxtgauge.com subdomain -
no DNS changes needed, this starts serving as soon as Flux applies it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 1 of the AI architecture doc ("Improve Generation Quality") —
qwen3:4b and qwen3:8b were already pulled onto the Ollama PVC, and
apps/litellm/base/configmap.yaml already had the correct model_list
mapping every feature alias to them instead of gemma3:270m. Neither
was actually in effect:
1. apps/litellm was never included in
clusters/production/kustomization.yaml, so it was only ever
deployed by a one-off manual `kubectl apply` and has been
completely outside GitOps ever since (same root cause as the
ai-guard registry drift found earlier). Added it to the root
kustomization. Corrected its image reference from
registry.nxtgauge.com/litellm:latest (doesn't appear to exist) to
ghcr.io/berriai/litellm:latest, matching what's actually running
live — adopting this file without that fix would have broken a
working deployment the moment Flux started managing it.
2. apps/ollama/base/deployment.yaml's memory limit (1500Mi) was too
small to ever load qwen3:4b (~2.5GB) or qwen3:8b (~5.2GB) — every
model alias in the (also-never-applied) LiteLLM config was
therefore unusable regardless of what it was named. Raised to
4 CPU / 8Gi limit (node has 16GB total, was at ~26% memory use) and
added OLLAMA_KEEP_ALIVE=30m so a loaded model survives the gaps
between bursty feature requests instead of reloading from disk on
every first call after 5+ minutes idle.
- Add .sops.yaml (age) and encrypt every plaintext Secret manifest in apps/
- Commit the Flux GitRepository/Kustomization (previously only applied manually)
with a decryption stanza referencing the sops-age key (created out-of-band,
not committed)
- Rotate JWT_SECRET (previously exposed in plaintext) and set a real
AI_SERVICE_KEY (was an empty placeholder), shared between
nxtgauge-backend-rust and nxtgauge-ai-assistant
- Wire JWT_SECRET into the ai-assistant deployment (was missing entirely,
causing every authenticated request to fail)
- Redact the leaked LiteLLM production master key from README/OPENCODE_CONNECT
docs; move the litellm prod overlay off a plaintext secretGenerator onto an
encrypted patch
