capacity=3/node (9 total) oversubscribed the 8 physical cores per
node - individual builds slowed roughly 3x under contention (observed
directly: builds that took ~2m37s standalone were still running after
8+ minutes), for no clear net throughput win since this workload is
CPU-bound. Also found and fixed the actual reason capacity wasn't
taking effect at all initially: the runner daemon command never
passed --config, so it was silently running on default settings
regardless of config.yaml. Settled on 2/node (6 total, matching
docker-ready runner-2/3/4 - the control-plane node nxtgauge-1 is
deliberately excluded from the runner DaemonSet and should stay that
way, not used for build load).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bumped runner.capacity from 1 to 3 on all 3 runner pods (9 total
concurrent slots - nodes were sitting at 7-11% CPU during builds, so
plenty of headroom), matched here with max-parallel: 9.
Also fixed a real bug: the gitops-push retry loop had no check after
exhausting all attempts, so a job whose every push attempt failed
would still exit 0 and report "success" - which is exactly what
happened on the previous run (verified: all 20 services built and
pushed their images correctly, but the actual GITOPS_PAT secret was
invalid/expired, and the retry loop silently swallowed the resulting
failure across all 20 jobs). Fixed the secret itself (confirmed the
existing admin-scoped Forgejo token has valid push access to
ashwin/nxtgauge-gitops and rotated GITOPS_PAT to it), and added an
explicit exit 1 if the retry loop exhausts without a successful push.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed via the matrix run: every service's actual docker build and
push succeeded, but the GitOps-update step failed on all of them with
"GITOPS_PUSH_TOKEN is empty". Checked the repo's configured secrets -
GITOPS_PUSH_USERNAME/GITOPS_PUSH_TOKEN were never set; only a leftover
GITOPS_GITHUB_USERNAME/GITOPS_GITHUB_TOKEN pair (from before the
gitops repo moved to Forgejo) and GITOPS_PAT exist. Confirmed
ashwin/nxtgauge-gitops lives on ci.nxtgauge.com matching this
workflow's defaults, so GITOPS_PAT is the one meant for this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The build was structured as a single job looping through all ~20
services sequentially, so only 1 of the 3 deployed runner pods (one
per worker node) was ever used - the other 2 sat idle for the entire
build. Switched to a static per-service matrix (max-parallel: 3,
matching the 3 runners at capacity 1 each) so independent services
build concurrently. Each matrix job does its own quick "does this
service need building" check up front (same change-detection logic,
now per-job) rather than relying on a shared job output, to avoid
needing cross-job artifact/output passing.
GitOps updates also move into each matrix job (previously a single
step at the end) since there's no longer one job aggregating all
results - added a fetch/reset/retry loop since multiple jobs can now
push to the same gitops branch concurrently.
Tried adding cargo registry/target cache mounts to Dockerfile.simple
for a bigger per-build win too, but measured it directly (local A/B:
cold build 2m37s vs a second, supposedly-cached build 7m18s) and it
made things slower here, likely cargo's own cache-verification pass
outweighing the benefit for this dependency set - reverted that part,
Dockerfile.simple is unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
docker buildx --metadata-file writes pretty-printed JSON (space after
the colon: "containerimage.digest": "sha256:..."), but the digest
grep required compact JSON with no space, so it always matched
nothing. That produced an empty $digest, which the script correctly
treated as fatal and exited on - right after the first service
(gateway) had already built and pushed successfully, silently
aborting the rest of the service loop. Verified the fix against a
real locally-generated metadata file from the same buildx command.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--add-host=host.docker.internal:host-gateway is not being honored by
this act_runner setup (tried via the runner's global container.options
and a per-job container: block; neither worked, confirmed by two
separate failed runs with identical DNS-lookup errors). Read the
container's real default-route gateway from /proc/net/route instead
(portable, no iproute2 dependency) and export it as DOCKER_HOST via
GITHUB_ENV. This is the actual IP of the dind engine that spawned the
job container, regardless of hostname-aliasing support.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Forgejo's docs note --add-host via the runner's global
container.options config is not reliably honored for jobs using pure
label-based runs-on mapping. Setting the same image + options
explicitly per-job via jobs.build.container is the more directly
supported path for per-job container customization.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
127.0.0.1:2375 only worked from the runner container's own network
namespace (sibling to the dind sidecar in the same pod). The actual
job container that runs each step is spawned *by* that dind engine,
one level deeper, so its own loopback doesn't reach the sidecar.
host.docker.internal (mapped to the job container's real gateway via
--add-host=host.docker.internal:host-gateway in the runner's
container.options) reaches back to the dind engine that created it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The unindented heredoc terminator (<<EOF2 ... EOF2 flush-left) broke the
YAML block-scalar indentation contract for the `run: |` step, causing
Forgejo to log "ignore invalid workflow build.yaml: yaml: line 75:
could not find expected ':'" and silently skip the workflow entirely
on every push since this line was introduced. No builds have run on
main or high-performance since 2026-07-05 as a result. Replaced the
multi-line heredoc with a single-line bash here-string, which needs
no extra indentation and keeps the shell logic identical.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
