31 commits

Author	SHA1	Message	Date
Ashwin Kumar Sivakumar	c0a961c8ef	debug: add back the check step to isolate the failure further	2026-08-13 17:59:04 +05:30
Ashwin Kumar Sivakumar	22cbd3e285	debug: add back job-level env secret reference to isolate the failure	2026-08-13 17:55:30 +05:30
Ashwin Kumar Sivakumar	e80c58641f	debug: strip test.yaml to bare minimum to isolate why the job dies mid-run	2026-08-13 17:53:14 +05:30
Ashwin Kumar Sivakumar	3740bb81c5	fix(ci): fetch full git history in ai-credits job for diff-based change detection ... Previous job died silently right after checkout with no logs at all from the 'Check for relevant changes' step onward - shallow clone (default fetch-depth:1, unlike build.yaml which sets fetch-depth:0) may have been the cause. Matching build.yaml's checkout config. Also re-set TEST_DATABASE_URL with a properly percent-encoded password (the raw '@' in it made the postgresql:// URI ambiguous to parse). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 17:48:43 +05:30
Ashwin Kumar Sivakumar	3b4a4eac3f	fix(ci): install build-essential before cargo test in ai-credits job ... The bare bookworm runner image has no C toolchain at all, unlike build.yaml's jobs which build inside a Dockerfile-based image with one already present. proc-macro2/libc/etc build scripts need `cc` to link and failed immediately (error: linker `cc` not found) before any real test code ran. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 17:39:12 +05:30
Ashwin Kumar Sivakumar	da9be7f6a3	feat(db,ci): fix missing ai_credits tables, wire integration tests to CI ... Re-enables and corrects 20260703210000_ai_credits_wallet, which had been disabled (.up.sql.skip) in commit 0163349 because 'relation already exists' aborted the original CREATE TABLE script on its very first table (ai_plans) - silently leaving ai_credit_ledger and ai_reservation_holds never created, despite the wallet system being actively wired into apps/users/handlers/ai_credits.rs, apps/payments, apps/cron with real subscription rows already in prod. Rewrote the migration to be idempotent: ADD COLUMN IF NOT EXISTS to bring ai_plans/user_ai_subscriptions/ai_feature_costs/ai_usage_logs up to the full column set crates/db/src/models/ai_credits.rs expects (safe zero/default backfills, no data loss - verified against the 3 existing subscription rows), and CREATE TABLE IF NOT EXISTS for the 2 genuinely-missing tables. down.sql updated to only reverse what this corrected version actually adds, not drop tables that predate it. Applied directly to the live database (verified schema + existing rows intact afterward). Also wires crates/db/tests/ai_credits.rs + ai_credits_reaper.rs to Forgejo Actions CI (docs/LIVE_SERVER_RUNBOOK.md step 6): - New TEST_DATABASE_URL repo secret, pointing at a dedicated nxtgauge_test database (created fresh, schema mirrored from live prod via pg_dump --schema-only - never runs against the real nxtgauge DB). - .forgejo/workflows/test.yaml: runs on push to main/high-performance, skips when crates/db/ isn't touched, cargo test -p db --test ai_credits --test ai_credits_reaper -- --test-threads=1. Also commits docs/openapi.wallet-holds.json (hand-written minimal OpenAPI spec used for schemathesis fuzzing in step 4) and gitignores the local .schemathesis/ cache directory from that run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 17:31:22 +05:30
Ashwin Kumar Sivakumar	b30629411a	fix(ci): diff/scan full pushed commit range, not just HEAD^..HEAD ... Multi-commit pushes (e.g. this repo's mirror sync from GitHub) were silently skipping every service's build: the 'does this service need building' check only looked at the last commit vs its immediate parent. A push landing 3 commits where only the final one was a no-op (docs/.gitignore) meant the actual code commit's changes were never seen, so all 19 services reported 'no changes relevant - skipping' even though real backend code had changed. Use github.event.before (the pre-push SHA) to diff/scan the whole pushed range when available, falling back to HEAD^..HEAD only when that ref is unavailable. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 22:18:37 +05:30
Ashwin Kumar Sivakumar	c570d7df67	chore: remove the dead apps/leads microservice ... apps/leads implemented its own, independent "lead request" system (POST /api/leads, /api/lead-requests/send, accept/reject) with a schema that never matched the live one (message vs remarks, no professional_user_id, accepted_at/rejected_at instead of resolved_at — see 20260721030000_create_lead_requests's commit message). Confirmed unreachable: the frontend's live flows use apps/customers' /api/customers/requirements and each profession's /leads/request (crates/contracts::profession_shared), never anything under apps/leads' own paths. Its /api/lead-requests/* endpoints weren't even reachable through the gateway (wrong prefix, never matched /api/leads or /api/admin/leads). Removed: - apps/leads/ entirely, and its Cargo.toml workspace membership - the `leads` docker-compose service, its LEADS_SERVICE_URL env var on gateway, and gateway's depends_on entry - the `leads` entry from both CI build matrices (.gitea/.forgejo) - gateway's leads_url field/routing branch — gateway no longer hard- requires LEADS_SERVICE_URL to boot (.expect() would have panicked once the service was gone); /api/admin/leads now falls through to the customers service, which already had a matching (previously shadowed) branch for it NOTE: this service may still have a live Deployment/Service in nxtgauge-gitops (a separate repo not touched here) — that manifest should be removed too, or the next deploy will reference an image that no CI job builds anymore.	2026-07-21 04:19:56 +05:30
Ashwin Kumar Sivakumar	8a8372850f	perf(ci): settle on capacity 2/node x 3 workers = 6 concurrent slots ... capacity=3/node (9 total) oversubscribed the 8 physical cores per node - individual builds slowed roughly 3x under contention (observed directly: builds that took ~2m37s standalone were still running after 8+ minutes), for no clear net throughput win since this workload is CPU-bound. Also found and fixed the actual reason capacity wasn't taking effect at all initially: the runner daemon command never passed --config, so it was silently running on default settings regardless of config.yaml. Settled on 2/node (6 total, matching docker-ready runner-2/3/4 - the control-plane node nxtgauge-1 is deliberately excluded from the runner DaemonSet and should stay that way, not used for build load). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 02:43:46 +05:30
Ashwin Kumar Sivakumar	ba1b0ebcc0	fix(ci): raise max-parallel to 9, make gitops push failures actually fail the job ... Bumped runner.capacity from 1 to 3 on all 3 runner pods (9 total concurrent slots - nodes were sitting at 7-11% CPU during builds, so plenty of headroom), matched here with max-parallel: 9. Also fixed a real bug: the gitops-push retry loop had no check after exhausting all attempts, so a job whose every push attempt failed would still exit 0 and report "success" - which is exactly what happened on the previous run (verified: all 20 services built and pushed their images correctly, but the actual GITOPS_PAT secret was invalid/expired, and the retry loop silently swallowed the resulting failure across all 20 jobs). Fixed the secret itself (confirmed the existing admin-scoped Forgejo token has valid push access to ashwin/nxtgauge-gitops and rotated GITOPS_PAT to it), and added an explicit exit 1 if the retry loop exhausts without a successful push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 02:30:36 +05:30
Ashwin Kumar Sivakumar	9efb733974	fix(ci): use GITOPS_PAT instead of never-configured GITOPS_PUSH_TOKEN ... Confirmed via the matrix run: every service's actual docker build and push succeeded, but the GitOps-update step failed on all of them with "GITOPS_PUSH_TOKEN is empty". Checked the repo's configured secrets - GITOPS_PUSH_USERNAME/GITOPS_PUSH_TOKEN were never set; only a leftover GITOPS_GITHUB_USERNAME/GITOPS_GITHUB_TOKEN pair (from before the gitops repo moved to Forgejo) and GITOPS_PAT exist. Confirmed ashwin/nxtgauge-gitops lives on ci.nxtgauge.com matching this workflow's defaults, so GITOPS_PAT is the one meant for this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 02:10:33 +05:30
Ashwin Kumar Sivakumar	e8651cf6c6	perf(ci): parallelize service builds across all 3 runners via matrix strategy ... The build was structured as a single job looping through all ~20 services sequentially, so only 1 of the 3 deployed runner pods (one per worker node) was ever used - the other 2 sat idle for the entire build. Switched to a static per-service matrix (max-parallel: 3, matching the 3 runners at capacity 1 each) so independent services build concurrently. Each matrix job does its own quick "does this service need building" check up front (same change-detection logic, now per-job) rather than relying on a shared job output, to avoid needing cross-job artifact/output passing. GitOps updates also move into each matrix job (previously a single step at the end) since there's no longer one job aggregating all results - added a fetch/reset/retry loop since multiple jobs can now push to the same gitops branch concurrently. Tried adding cargo registry/target cache mounts to Dockerfile.simple for a bigger per-build win too, but measured it directly (local A/B: cold build 2m37s vs a second, supposedly-cached build 7m18s) and it made things slower here, likely cargo's own cache-verification pass outweighing the benefit for this dependency set - reverted that part, Dockerfile.simple is unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 01:54:47 +05:30
Ashwin Kumar Sivakumar	968960fb3d	fix(ci): tolerate buildx's pretty-printed metadata-file JSON when extracting digest ... docker buildx --metadata-file writes pretty-printed JSON (space after the colon: "containerimage.digest": "sha256:..."), but the digest grep required compact JSON with no space, so it always matched nothing. That produced an empty $digest, which the script correctly treated as fatal and exited on - right after the first service (gateway) had already built and pushed successfully, silently aborting the rest of the service loop. Verified the fix against a real locally-generated metadata file from the same buildx command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 00:03:32 +05:30
Ashwin Kumar Sivakumar	c8fa02a9a4	fix(ci): derive DOCKER_HOST from the job container's own default gateway ... --add-host=host.docker.internal:host-gateway is not being honored by this act_runner setup (tried via the runner's global container.options and a per-job container: block; neither worked, confirmed by two separate failed runs with identical DNS-lookup errors). Read the container's real default-route gateway from /proc/net/route instead (portable, no iproute2 dependency) and export it as DOCKER_HOST via GITHUB_ENV. This is the actual IP of the dind engine that spawned the job container, regardless of hostname-aliasing support. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-07 22:50:54 +05:30
Ashwin Kumar Sivakumar	dcf573e0f6	fix(ci): use explicit container: block for reliable job-level docker options ... Forgejo's docs note --add-host via the runner's global container.options config is not reliably honored for jobs using pure label-based runs-on mapping. Setting the same image + options explicitly per-job via jobs.build.container is the more directly supported path for per-job container customization. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-07 22:45:35 +05:30
Ashwin Kumar Sivakumar	b9e0ba247c	fix(ci): route job-container DOCKER_HOST via host.docker.internal ... 127.0.0.1:2375 only worked from the runner container's own network namespace (sibling to the dind sidecar in the same pod). The actual job container that runs each step is spawned *by* that dind engine, one level deeper, so its own loopback doesn't reach the sidecar. host.docker.internal (mapped to the job container's real gateway via --add-host=host.docker.internal:host-gateway in the runner's container.options) reaches back to the dind engine that created it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-07 19:59:27 +05:30
Ashwin Kumar Sivakumar	0ea4024657	fix(ci): fix YAML syntax error in build.yaml that silently disabled Actions ... The unindented heredoc terminator (<<EOF2 ... EOF2 flush-left) broke the YAML block-scalar indentation contract for the `run: |` step, causing Forgejo to log "ignore invalid workflow build.yaml: yaml: line 75: could not find expected ':'" and silently skip the workflow entirely on every push since this line was introduced. No builds have run on main or high-performance since 2026-07-05 as a result. Replaced the multi-line heredoc with a single-line bash here-string, which needs no extra indentation and keeps the shell logic identical. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-07 18:34:45 +05:30
Ashwin Kumar Sivakumar	afb4dc9ae6	fix(ci): use docker-ready runner label in Forgejo workflow ... - Change runs-on from self-hosted to docker-ready - Add DOCKER_HOST and DOCKER_BUILDKIT env vars - Align with other repo workflows	2026-07-06 04:24:06 +05:30
Ashwin Kumar Sivakumar	4bfb5b4418	Update Forgejo backend publish workflow	2026-06-17 03:28:46 +05:30
Ashwin Kumar Sivakumar	349673b7f8	fix(ci): unify backend build flow through forgejo	2026-06-14 01:20:24 +05:30
Ashwin Kumar Sivakumar	1c9b4848a9	feat: Add Dockerfiles for jobs/leads services and build scripts ... Add build-images.yaml for Forgejo CI tmp build.yaml Add Dockerfiles for jobs and leads services Add build-all.sh script for batch building	2026-06-13 18:31:34 +05:30
Ashwin Kumar Sivakumar	189f4bca60	ci: test single job with ubuntu-latest	2026-06-13 01:44:07 +05:30
Ashwin Kumar Sivakumar	1956acdcf3	ci: minimal workflow without external actions	2026-06-13 01:41:32 +05:30
Ashwin Kumar Sivakumar	ec41f6dad2	ci: simpler workflow with self-hosted runners	2026-06-13 01:38:16 +05:30
Ashwin Kumar Sivakumar	051a980f14	ci: add simple build workflow with self-hosted runners	2026-06-13 01:29:21 +05:30
Ashwin Kumar Sivakumar	0d35bf5649	trigger build: rebuild all services	2026-06-13 01:23:22 +05:30
Ashwin Kumar Sivakumar	d0b10eac8f	trigger forgejo pipeline: rebuild all services ... Retention script was too aggressive and deleted most images. Increasing keep count from 2 to 10 SHA tags.	2026-06-12 23:55:01 +05:30
Ashwin Kumar Sivakumar	0bda2b2f10	remove: delete high-performance-latest tag from builds - use SHA tags only	2026-06-12 04:40:36 +05:30
Ashwin Kumar Sivakumar	8adc84699e	fix: keep only 2 SHA tags (current + 1 previous) to save disk space	2026-06-12 04:12:15 +05:30
Ashwin Kumar Sivakumar	758f0699ff	fix: move image cleanup to post-build job after all builds complete	2026-06-12 04:10:52 +05:30
Ashwin Kumar Sivakumar	1b1d98ebee	chore: migrate ci naming to forgejo	2026-06-11 17:17:42 +05:30