nxtgauge-backend-rust/docs/LIVE_SERVER_RUNBOOK.md

# Nxtgauge — Live Server Runbook

Steps that **require a running database and/or server** to complete.  
Run these on the server where Postgres + Redis are already up.

---

## ✅ Completed — Database migrations (applied to prod, 2026-08-13)

All four DB tasks below are done and confirmed against the live production DB.

| Migration | What | Status |
|---|---|---|
| `20260627030000_wallet_full.up.sql` | Re-enabled (was mistakenly `.skip`-ed); `tracecoin_holds` + `tracecoin_buckets` now live | ✅ Applied |
| `20260703210000_ai_credits_wallet.up.sql` | Rewrote as idempotent (`IF NOT EXISTS`); `ai_reservation_holds` + `ai_credit_ledger` now live | ✅ Applied |
| `20260813000000_fix_ai_credits_timestamp_types.up.sql` | 9 TIMESTAMP → TIMESTAMPTZ columns across AI tables; data intact and UTC-correct | ✅ Applied |
| `nxtgauge_test` DB | Dedicated test database created, schema mirrored from prod, wired into Forgejo CI via `TEST_DATABASE_URL` secret | ✅ Done |

`/wallet/me/holds` returns real data. All 8 backend integration tests pass in CI.

---

## ⏳ Pending — DB migrations for AI credit purchases + coupons + fresh-bootstrap fixes (2026-08-14)

Eight new migrations need to be applied to prod. Run **in order** (sqlx-migrate handles this, but listed explicitly so you can verify):

```bash
# On the live server, inside nxtgauge-backend-rust:
git pull origin high-performance
sqlx migrate run --database-url "$DATABASE_URL"
```

That single command applies all eight in chronological order. Verify after:

```bash
sqlx migrate info --database-url "$DATABASE_URL"
```

All eight should show `applied`. On prod specifically, the four "early" fixes
below should all apply as **no-ops** — prod already has the target schema
(that's how their gating conditions were derived and verified). If any of
them actually *does* something on prod (i.e. its `Applied` log line isn't
near-instant), stop and check `\d <table>` before continuing — it means
prod's schema differs from what this runbook assumed.

| Migration | What | Why |
|---|---|---|
| `20260318235959_customer_profiles_status_early.up.sql` | Adds `customer_profiles.status` early (`IF NOT EXISTS`) | `20260319090000`'s backfill UPDATEs this column 4 months before `20260721020000` used to add it — broke any fresh bootstrap |
| `20260401235959_drop_old_employees_table_early.up.sql` | Conditionally drops the old `users`-linked `employees` table (only if it still has the pre-transition shape — no `email` column) | `20260402030000` no-ops its `CREATE TABLE IF NOT EXISTS` against the old shape, then fails creating an index on a column that was never added |
| `20260419235959_external_role_module_tables_early.up.sql` | Creates `persona_types`/`modules`/`role_module_access`/`module_actions`/etc. (the disabled `20260420000003_external_role_modules.up.sql.skip`'s content, purely additive) | The seed migration that depends on these (`20260420000003...seed.sql`) has no active migration that creates them |
| `20260421235959_role_config_table_renames_early.up.sql` | Conditionally renames `role_permissions→role_admin_permissions`, `dashboard_configs→role_sidebar_configs`, `runtime_configs→role_runtime_configs`, `user_roles→user_role_assignments` (only fires per-table if old name exists and new name doesn't) | `20260422000000`'s widget seed needs `role_sidebar_configs`; the rename that creates it (`20260420000002...skip`) was disabled |
| `20260814000000_fix_ai_credit_packages_pricing_and_orders.up.sql` | Fixes prices (99→9900 paise etc.) + creates `ai_credit_orders` with `coupon_code` + `discount_applied` columns | AI credit purchase flow was completely broken: no orders table, wrong prices |
| `20260814010000_ai_credit_packages_role_scoping.up.sql` | Adds `applicable_roles TEXT[]` column to `ai_credit_packages` | Admin UI now supports restricting packages to specific roles |
| `20260814020000_ai_coupons.up.sql` | Creates `ai_coupons` + `ai_coupon_redemptions` tables | Backend `create_order`/`verify_order` reference these tables |
| `20260814030000_invoices_payment_id_polymorphic.up.sql` + `20260814050000_invoices_payment_id_polymorphic_check.up.sql` | Drops the hard FK on `invoices.payment_id` (needed for `AI_CREDIT_PURCHASE` invoices, which point at `ai_credit_orders` not `payments`), then replaces it with a `BEFORE INSERT/UPDATE` trigger that validates `payment_id` against the right table per `invoice_type` | Postgres has no native polymorphic FK; the first migration alone left `payment_id` completely unvalidated |
| `20260814040000_waitlist_signups.up.sql` | Creates the `nxtgauge.com` coming-soon waitlist table | Backs `POST /api/waitlist` |

After applying, redeploy the payments service:

```bash
systemctl restart nxtgauge-payments
```

### Local dev DB note (2026-08-14)

The local dev Postgres (`nxtgauge-backend-rust-postgres-1`) was found ~63
migrations behind (stalled since March) and was brought fully current as
part of today's work, including manually reconciling `payments`/`invoices`
legacy tables seeded by `scripts/init-db.sql` (dropped + recreated, both
verified empty first — not a chain bug, doesn't affect fresh bootstraps).
The four "early" migrations above were then written and verified against
**both** that reconciled DB (all four applied as no-ops) **and** a
completely fresh, empty database created from scratch (`sqlx migrate run`
end-to-end, zero manual intervention) — so any new environment, or anyone
re-bootstrapping from zero, should no longer hit any of these four walls.

---

## 1. Deploy backend changes

```bash
cd nxtgauge-backend-rust
cargo build --release
```

Then restart the affected services:

```bash
# Restart all profession services (wallet/me/holds route)
systemctl restart nxtgauge-photographers nxtgauge-developers nxtgauge-tutors \
  nxtgauge-makeup_artists nxtgauge-fitness_trainers nxtgauge-catering_services \
  nxtgauge-video_editors nxtgauge-graphic_designers nxtgauge-social_media_managers

# Restart job_seekers (printpdf upgrade)
systemctl restart nxtgauge-job_seekers
```

---

## 2. Run security checks (no DB needed)

```bash
cd nxtgauge-backend-rust

# CVE scan
cargo audit

# Policy check (licenses, bans, CVE with acknowledged ignores)
cargo deny check advisories bans
```

Both should exit 0. If a new CVE appears, add it to `deny.toml` under `[advisories] ignore` with a comment explaining the risk.

---

## 3. Set up schemathesis API fuzzing

Schemathesis fuzzes your real API from an OpenAPI spec. It needs:
- The server running (with DB connected)
- An OpenAPI spec (`openapi.json`)

### 3a. Install schemathesis

```bash
pip install schemathesis
# or with uv:
uv tool install schemathesis
```

### 3b. Generate the OpenAPI spec

> **Note:** The axum services don't yet have utoipa annotations. Until then, use
> the manually maintained spec in `docs/openapi.wallet-holds.json` or skip to 3c.
>
> To add utoipa: add `utoipa` and `utoipa-axum` to each service's `Cargo.toml`,
> annotate handlers with `#[utoipa::path]`, and mount a `/openapi.json` endpoint.

### 3c. Run the fuzzer

```bash
schemathesis run openapi.json \
  --base-url http://localhost:3000 \
  --auth "Bearer <admin_or_test_jwt>" \
  --checks all \
  --hypothesis-settings max_examples=200

# Quick smoke-check:
schemathesis run openapi.json \
  --base-url http://localhost:3000 \
  --auth "Bearer <jwt>" \
  --checks not_a_server_error
```

---

## 4. Run Playwright end-to-end tests

Playwright tests live in `nxtgauge-frontend-solid/tests/`. Need full stack running.

```bash
cd nxtgauge-frontend-solid
npx playwright install --with-deps chromium   # one-time

export PLAYWRIGHT_BASE_URL=https://nxtgauge.com
npx playwright test
npx playwright test --grep @smoke             # smoke only
npx playwright show-report                   # view results
```

> **Note:** A few e2e test files still have `localhost:3001` URLs — pending cleanup,
> update to the staging/prod URL before running against live.

---

## 5. Backend integration tests (CI — already wired)

The `nxtgauge_test` DB and Forgejo CI secret are already set up.
To run locally:

```bash
export TEST_DATABASE_URL=postgres://user:pass@localhost:5432/nxtgauge_test
cd nxtgauge-backend-rust
cargo test --test ai_credits -- --test-threads=1
```

---

## 6. Frontend linter

```bash
cd nxtgauge-frontend-solid
npm run lint
```

---

## 7. TypeScript check

```bash
cd nxtgauge-frontend-solid
npx tsc --noEmit --skipLibCheck
```

Should exit 0.

---

## Summary

| Task | Needs DB | Needs server | Status |
|---|---|---|---|
| DB migrations | ✅ | ✅ | ✅ Done (prod) |
| Build + deploy backend | — | — | Pending next release |
| `cargo audit` / `cargo deny` | — | — | Run anytime |
| Schemathesis fuzzing | ✅ | ✅ | Pending OpenAPI spec |
| Playwright e2e | ✅ | ✅ | Pending URL cleanup |
| Backend integration tests | ✅ | — | ✅ Green in CI |
| `npm run lint` | — | — | ✅ Passing |
| `tsc --noEmit` | — | — | Run anytime |