docker system prune --volumes can never touch the buildx builder's
cache volume - it's attached to a running container, and 'volumes'
prune only removes *unattached* ones. This CronJob ran every 30
minutes for 59 days reporting 'Total reclaimed space: 0B' every
single time while each runner's buildx cache silently grew to
57-98GB (discovered chasing a disk-pressure report on nxtgauge-2/3/4).
Fix: attach the real named builder ('nxtgauge-builder', matching
what CI actually uses) first, then call 'docker buildx prune'
directly, capped at --keep-storage 20GB so it doesn't just regrow
unbounded. Verified live: manually pruned gwsh7 (80%->16% node disk)
and ktst5 (84%->47%), then confirmed the patched CronJob is a true
no-op on an already-clean cache.
Also found and cleaned up (host-level, not gitops - out of band from
k8s): 18 orphaned buildx_buildkit_* containers on nxtgauge-1 from 2
months of ad-hoc 'docker buildx create' calls with no --name reuse,
totally invisible to buildx CLI and unrelated to CI (39.63GB, node
went 76%->50%). Added a daily cron job on that host
(~/.local/bin/docker-cleanup.sh) to keep it from reaccumulating,
since that's the host's standalone Docker daemon, outside k8s/gitops
entirely.
This CronJob + its RBAC (docker-prune-sa/-role/-binding) previously
existed only as manually-applied live objects, not tracked in git -
same pattern as nxtgauge-containerd-cleanup. Adding manifests and
wiring into clusters/production so Flux manages it going forward.
The prune step called bare 'ctr', which has never existed as a
standalone binary on these k3s nodes - only the k3s binary itself
(bundling 'k3s ctr'/'k3s crictl' subcommands) lives at
/usr/local/bin. The failure was silently swallowed by '|| true',
so the CronJob has been reporting Complete every 10 minutes for
68 days while doing nothing; node disk sat at 83% used.
Verified live: patched the CronJob's command to invoke
/usr/local/bin/k3s ctr directly, manually triggered a run, confirmed
it actually prunes now (deleted ~75 dangling images).
This CronJob and its ServiceAccount previously existed only as a
manually-applied live object, not tracked in git. Adding the
manifests here and wiring them into clusters/production so Flux
manages it going forward instead of it silently drifting again.
- pg_dump + B2 (S3-compatible) upload with retention-based pruning
- litellm backup job is suspended: the nxtgauge-ai postgres instance
currently has a corrupt litellm database catalog file, un-suspend
once that's fixed/reinitialized
- secrets encrypted with sops per repo convention (.sops.yaml)
- not yet wired into kustomizations; apps/postgresql is not part of
the Flux-applied tree today (statefulset.yaml predates this and is
also unwired) - needs a deliberate decision on how postgres is
meant to be deployed/applied before enabling
- pg_dump + B2 (S3-compatible) upload with retention-based pruning
- litellm backup job is suspended: the nxtgauge-ai postgres instance
currently has a corrupt litellm database catalog file, un-suspend
once that's fixed/reinitialized
- secrets encrypted with sops per repo convention (.sops.yaml)
- not yet wired into kustomizations; apps/postgresql is not part of
the Flux-applied tree today (statefulset.yaml predates this and is
also unwired) - needs a deliberate decision on how postgres is
meant to be deployed/applied before enabling
