nxtgauge-backend-rust/docs/LIVE_SERVER_RUNBOOK.md

Nxtgauge — Live Server Runbook

Steps that require a running database and/or server to complete.
Run these on the server where Postgres + Redis are already up.

---

✅ Completed — Database migrations (applied to prod, 2026-08-13)

All four DB tasks below are done and confirmed against the live production DB.

Migration	What	Status
20260627030000_wallet_full.up.sql	Re-enabled (was mistakenly .skip-ed); tracecoin_holds + tracecoin_buckets now live	✅ Applied
20260703210000_ai_credits_wallet.up.sql	Rewrote as idempotent (IF NOT EXISTS); ai_reservation_holds + ai_credit_ledger now live	✅ Applied
20260813000000_fix_ai_credits_timestamp_types.up.sql	9 TIMESTAMP → TIMESTAMPTZ columns across AI tables; data intact and UTC-correct	✅ Applied
nxtgauge_test DB	Dedicated test database created, schema mirrored from prod, wired into Forgejo CI via TEST_DATABASE_URL secret	✅ Done

/wallet/me/holds returns real data. All 8 backend integration tests pass in CI.

---

⏳ Pending — DB migrations for AI credit purchases + coupons + fresh-bootstrap fixes (2026-08-14)

Eight new migrations need to be applied to prod. Run in order (sqlx-migrate handles this, but listed explicitly so you can verify):

# On the live server, inside nxtgauge-backend-rust:
git pull origin high-performance
sqlx migrate run --database-url "$DATABASE_URL"

That single command applies all eight in chronological order. Verify after:

sqlx migrate info --database-url "$DATABASE_URL"

All eight should show applied. On prod specifically, the four "early" fixes below should all apply as no-ops — prod already has the target schema (that's how their gating conditions were derived and verified). If any of them actually does something on prod (i.e. its Applied log line isn't near-instant), stop and check \d <table> before continuing — it means prod's schema differs from what this runbook assumed.

Migration	What	Why
20260318235959_customer_profiles_status_early.up.sql	Adds customer_profiles.status early (IF NOT EXISTS)	20260319090000's backfill UPDATEs this column 4 months before 20260721020000 used to add it — broke any fresh bootstrap
20260401235959_drop_old_employees_table_early.up.sql	Conditionally drops the old users-linked employees table (only if it still has the pre-transition shape — no email column)	20260402030000 no-ops its CREATE TABLE IF NOT EXISTS against the old shape, then fails creating an index on a column that was never added
20260419235959_external_role_module_tables_early.up.sql	Creates persona_types/modules/role_module_access/module_actions/etc. (the disabled 20260420000003_external_role_modules.up.sql.skip's content, purely additive)	The seed migration that depends on these (20260420000003...seed.sql) has no active migration that creates them
20260421235959_role_config_table_renames_early.up.sql	Conditionally renames role_permissions→role_admin_permissions, dashboard_configs→role_sidebar_configs, runtime_configs→role_runtime_configs, user_roles→user_role_assignments (only fires per-table if old name exists and new name doesn't)	20260422000000's widget seed needs role_sidebar_configs; the rename that creates it (20260420000002...skip) was disabled
20260814000000_fix_ai_credit_packages_pricing_and_orders.up.sql	Fixes prices (99→9900 paise etc.) + creates ai_credit_orders with coupon_code + discount_applied columns	AI credit purchase flow was completely broken: no orders table, wrong prices
20260814010000_ai_credit_packages_role_scoping.up.sql	Adds applicable_roles TEXT[] column to ai_credit_packages	Admin UI now supports restricting packages to specific roles
20260814020000_ai_coupons.up.sql	Creates ai_coupons + ai_coupon_redemptions tables	Backend create_order/verify_order reference these tables
20260814030000_invoices_payment_id_polymorphic.up.sql + 20260814050000_invoices_payment_id_polymorphic_check.up.sql	Drops the hard FK on invoices.payment_id (needed for AI_CREDIT_PURCHASE invoices, which point at ai_credit_orders not payments), then replaces it with a BEFORE INSERT/UPDATE trigger that validates payment_id against the right table per invoice_type	Postgres has no native polymorphic FK; the first migration alone left payment_id completely unvalidated
20260814040000_waitlist_signups.up.sql	Creates the nxtgauge.com coming-soon waitlist table	Backs POST /api/waitlist

After applying, redeploy the payments service:

systemctl restart nxtgauge-payments

Local dev DB note (2026-08-14)

The local dev Postgres (nxtgauge-backend-rust-postgres-1) was found ~63 migrations behind (stalled since March) and was brought fully current as part of today's work, including manually reconciling payments/invoices legacy tables seeded by scripts/init-db.sql (dropped + recreated, both verified empty first — not a chain bug, doesn't affect fresh bootstraps). The four "early" migrations above were then written and verified against both that reconciled DB (all four applied as no-ops) and a completely fresh, empty database created from scratch (sqlx migrate run end-to-end, zero manual intervention) — so any new environment, or anyone re-bootstrapping from zero, should no longer hit any of these four walls.

---

1. Deploy backend changes

cd nxtgauge-backend-rust
cargo build --release

Then restart the affected services:

# Restart all profession services (wallet/me/holds route)
systemctl restart nxtgauge-photographers nxtgauge-developers nxtgauge-tutors \
  nxtgauge-makeup_artists nxtgauge-fitness_trainers nxtgauge-catering_services \
  nxtgauge-video_editors nxtgauge-graphic_designers nxtgauge-social_media_managers

# Restart job_seekers (printpdf upgrade)
systemctl restart nxtgauge-job_seekers

---

2. Run security checks (no DB needed)

cd nxtgauge-backend-rust

# CVE scan
cargo audit

# Policy check (licenses, bans, CVE with acknowledged ignores)
cargo deny check advisories bans

Both should exit 0. If a new CVE appears, add it to deny.toml under [advisories] ignore with a comment explaining the risk.

---

3. Set up schemathesis API fuzzing

Schemathesis fuzzes your real API from an OpenAPI spec. It needs:

• The server running (with DB connected)
• An OpenAPI spec (openapi.json)

3a. Install schemathesis

pip install schemathesis
# or with uv:
uv tool install schemathesis

3b. Generate the OpenAPI spec

Note: The axum services don't yet have utoipa annotations. Until then, use the manually maintained spec in docs/openapi.wallet-holds.json or skip to 3c.

To add utoipa: add utoipa and utoipa-axum to each service's Cargo.toml, annotate handlers with #[utoipa::path], and mount a /openapi.json endpoint.

3c. Run the fuzzer

schemathesis run openapi.json \
  --base-url http://localhost:3000 \
  --auth "Bearer <admin_or_test_jwt>" \
  --checks all \
  --hypothesis-settings max_examples=200

# Quick smoke-check:
schemathesis run openapi.json \
  --base-url http://localhost:3000 \
  --auth "Bearer <jwt>" \
  --checks not_a_server_error

---

4. Run Playwright end-to-end tests

Playwright tests live in nxtgauge-frontend-solid/tests/. Need full stack running.

cd nxtgauge-frontend-solid
npx playwright install --with-deps chromium   # one-time

export PLAYWRIGHT_BASE_URL=https://nxtgauge.com
npx playwright test
npx playwright test --grep @smoke             # smoke only
npx playwright show-report                   # view results

Note: A few e2e test files still have localhost:3001 URLs — pending cleanup, update to the staging/prod URL before running against live.

---

5. Backend integration tests (CI — already wired)

The nxtgauge_test DB and Forgejo CI secret are already set up. To run locally:

export TEST_DATABASE_URL=postgres://user:pass@localhost:5432/nxtgauge_test
cd nxtgauge-backend-rust
cargo test --test ai_credits -- --test-threads=1

---

6. Frontend linter

cd nxtgauge-frontend-solid
npm run lint

---

7. TypeScript check

cd nxtgauge-frontend-solid
npx tsc --noEmit --skipLibCheck

Should exit 0.

---

Summary

Task	Needs DB	Needs server	Status
DB migrations	✅	✅	✅ Done (prod)
Build + deploy backend	—	—	Pending next release
cargo audit / cargo deny	—	—	Run anytime
Schemathesis fuzzing	✅	✅	Pending OpenAPI spec
Playwright e2e	✅	✅	Pending URL cleanup
Backend integration tests	✅	—	✅ Green in CI
npm run lint	—	—	✅ Passing
tsc --noEmit	—	—	Run anytime