fix(ci): raise max-parallel to 9, make gitops push failures actually fail the job
Some checks failed
build-and-release / build (cron) (push) Successful in 3m8s
build-and-release / build (gateway) (push) Successful in 5m7s
build-and-release / build (jobs) (push) Successful in 6m39s
build-and-release / build (payments) (push) Failing after 2m34s
build-and-release / build (graphic-designers) (push) Failing after 7m40s
build-and-release / build (job-seekers) (push) Failing after 7m38s
build-and-release / build (employees) (push) Failing after 8m8s
build-and-release / build (developers) (push) Failing after 8m10s
build-and-release / build (fitness-trainers) (push) Failing after 8m6s
build-and-release / build (leads) (push) Failing after 7m54s
build-and-release / build (photographers) (push) Failing after 1m17s
build-and-release / build (makeup-artists) (push) Failing after 7m52s
build-and-release / build (customers) (push) Failing after 13m56s
build-and-release / build (companies) (push) Failing after 17m4s
build-and-release / build (catering-services) (push) Failing after 17m5s
build-and-release / build (tutors) (push) Successful in 9m19s
build-and-release / build (social-media-managers) (push) Successful in 9m26s
build-and-release / build (ugc-content-creators) (push) Successful in 9m36s
build-and-release / build (video-editors) (push) Successful in 9m37s
build-and-release / build (users) (push) Successful in 12m26s

Bumped runner.capacity from 1 to 3 on all 3 runner pods (9 total
concurrent slots - nodes were sitting at 7-11% CPU during builds, so
plenty of headroom), matched here with max-parallel: 9.

Also fixed a real bug: the gitops-push retry loop had no check after
exhausting all attempts, so a job whose every push attempt failed
would still exit 0 and report "success" - which is exactly what
happened on the previous run (verified: all 20 services built and
pushed their images correctly, but the actual GITOPS_PAT secret was
invalid/expired, and the retry loop silently swallowed the resulting
failure across all 20 jobs). Fixed the secret itself (confirmed the
existing admin-scoped Forgejo token has valid push access to
ashwin/nxtgauge-gitops and rotated GITOPS_PAT to it), and added an
explicit exit 1 if the retry loop exhausts without a successful push.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Ashwin Kumar Sivakumar 2026-07-08 02:30:36 +05:30
parent 9efb733974
commit ba1b0ebcc0

View file

@ -15,7 +15,7 @@ jobs:
runs-on: docker-ready
strategy:
fail-fast: false
max-parallel: 3
max-parallel: 9
matrix:
service:
- gateway
@ -171,15 +171,17 @@ jobs:
git clone "https://forgejo-actions:${GITOPS_PAT}@${GITOPS_SERVER}/${GITOPS_OWNER}/${GITOPS_REPO}.git" /tmp/nxtgauge-gitops
cd /tmp/nxtgauge-gitops
# Up to 3 of these jobs can be pushing to the same gitops branch at
# Up to 9 of these jobs can be pushing to the same gitops branch at
# once now - retry with a fresh pull+rebase on a non-fast-forward
# rejection instead of assuming we're the only writer.
for attempt in 1 2 3 4 5; do
pushed=false
for attempt in 1 2 3 4 5 6 7 8; do
git checkout "$GITOPS_BRANCH"
./scripts/set-backend-rust-release.sh "$service" "$DIGEST"
if git diff --quiet; then
echo "GitOps repo already up to date for $service."
pushed=true
break
fi
@ -192,11 +194,17 @@ jobs:
git commit -m "chore(gitops): update ${service} image for ${SHA}"
if git push origin "HEAD:${GITOPS_BRANCH}"; then
pushed=true
break
fi
echo "Push rejected (attempt $attempt/5), pulling latest and retrying..."
echo "Push rejected (attempt $attempt/8), pulling latest and retrying..."
git fetch origin "$GITOPS_BRANCH"
git reset --hard "origin/${GITOPS_BRANCH}"
sleep $((attempt * 2))
done
if [ "$pushed" != true ]; then
echo "Failed to push gitops update for $service after all retries" >&2
exit 1
fi