nxtgauge-backend-rust/docs/LIVE_SERVER_RUNBOOK.md

# Nxtgauge — Live Server Runbook

Steps that **require a running database and/or server** to complete.  
Run these on the server where Postgres + Redis are already up.

---

## ✅ Completed — Database migrations (applied to prod, 2026-08-13)

All four DB tasks below are done and confirmed against the live production DB.

| Migration | What | Status |
|---|---|---|
| `20260627030000_wallet_full.up.sql` | Re-enabled (was mistakenly `.skip`-ed); `tracecoin_holds` + `tracecoin_buckets` now live | ✅ Applied |
| `20260703210000_ai_credits_wallet.up.sql` | Rewrote as idempotent (`IF NOT EXISTS`); `ai_reservation_holds` + `ai_credit_ledger` now live | ✅ Applied |
| `20260813000000_fix_ai_credits_timestamp_types.up.sql` | 9 TIMESTAMP → TIMESTAMPTZ columns across AI tables; data intact and UTC-correct | ✅ Applied |
| `nxtgauge_test` DB | Dedicated test database created, schema mirrored from prod, wired into Forgejo CI via `TEST_DATABASE_URL` secret | ✅ Done |

`/wallet/me/holds` returns real data. All 8 backend integration tests pass in CI.

---

## ⏳ Pending — DB migrations for AI credit purchases (2026-08-14)

Three new migrations need to be applied to prod. Run **in order** (sqlx-migrate handles this, but listed explicitly so you can verify):

```bash
# On the live server, inside nxtgauge-backend-rust:
git pull origin high-performance
sqlx migrate run --database-url "$DATABASE_URL"
```

That single command applies all three in chronological order. Verify after:

```bash
sqlx migrate info --database-url "$DATABASE_URL"
```

All three should show `applied`.

| Migration | What | Why |
|---|---|---|
| `20260814000000_fix_ai_credit_packages_pricing_and_orders.up.sql` | Fixes prices (99→9900 paise etc.) + creates `ai_credit_orders` with `coupon_code` + `discount_applied` columns | AI credit purchase flow was completely broken: no orders table, wrong prices |
| `20260814010000_ai_credit_packages_role_scoping.up.sql` | Adds `applicable_roles TEXT[]` column to `ai_credit_packages` | Admin UI now supports restricting packages to specific roles |
| `20260814020000_ai_coupons.up.sql` | Creates `ai_coupons` + `ai_coupon_redemptions` tables | Backend `create_order`/`verify_order` reference these tables |

After applying, redeploy the payments service:

```bash
systemctl restart nxtgauge-payments
```

---

## 1. Deploy backend changes

```bash
cd nxtgauge-backend-rust
cargo build --release
```

Then restart the affected services:

```bash
# Restart all profession services (wallet/me/holds route)
systemctl restart nxtgauge-photographers nxtgauge-developers nxtgauge-tutors \
  nxtgauge-makeup_artists nxtgauge-fitness_trainers nxtgauge-catering_services \
  nxtgauge-video_editors nxtgauge-graphic_designers nxtgauge-social_media_managers

# Restart job_seekers (printpdf upgrade)
systemctl restart nxtgauge-job_seekers
```

---

## 2. Run security checks (no DB needed)

```bash
cd nxtgauge-backend-rust

# CVE scan
cargo audit

# Policy check (licenses, bans, CVE with acknowledged ignores)
cargo deny check advisories bans
```

Both should exit 0. If a new CVE appears, add it to `deny.toml` under `[advisories] ignore` with a comment explaining the risk.

---

## 3. Set up schemathesis API fuzzing

Schemathesis fuzzes your real API from an OpenAPI spec. It needs:
- The server running (with DB connected)
- An OpenAPI spec (`openapi.json`)

### 3a. Install schemathesis

```bash
pip install schemathesis
# or with uv:
uv tool install schemathesis
```

### 3b. Generate the OpenAPI spec

> **Note:** The axum services don't yet have utoipa annotations. Until then, use
> the manually maintained spec in `docs/openapi.wallet-holds.json` or skip to 3c.
>
> To add utoipa: add `utoipa` and `utoipa-axum` to each service's `Cargo.toml`,
> annotate handlers with `#[utoipa::path]`, and mount a `/openapi.json` endpoint.

### 3c. Run the fuzzer

```bash
schemathesis run openapi.json \
  --base-url http://localhost:3000 \
  --auth "Bearer <admin_or_test_jwt>" \
  --checks all \
  --hypothesis-settings max_examples=200

# Quick smoke-check:
schemathesis run openapi.json \
  --base-url http://localhost:3000 \
  --auth "Bearer <jwt>" \
  --checks not_a_server_error
```

---

## 4. Run Playwright end-to-end tests

Playwright tests live in `nxtgauge-frontend-solid/tests/`. Need full stack running.

```bash
cd nxtgauge-frontend-solid
npx playwright install --with-deps chromium   # one-time

export PLAYWRIGHT_BASE_URL=https://nxtgauge.com
npx playwright test
npx playwright test --grep @smoke             # smoke only
npx playwright show-report                   # view results
```

> **Note:** A few e2e test files still have `localhost:3001` URLs — pending cleanup,
> update to the staging/prod URL before running against live.

---

## 5. Backend integration tests (CI — already wired)

The `nxtgauge_test` DB and Forgejo CI secret are already set up.
To run locally:

```bash
export TEST_DATABASE_URL=postgres://user:pass@localhost:5432/nxtgauge_test
cd nxtgauge-backend-rust
cargo test --test ai_credits -- --test-threads=1
```

---

## 6. Frontend linter

```bash
cd nxtgauge-frontend-solid
npm run lint
```

---

## 7. TypeScript check

```bash
cd nxtgauge-frontend-solid
npx tsc --noEmit --skipLibCheck
```

Should exit 0.

---

## Summary

| Task | Needs DB | Needs server | Status |
|---|---|---|---|
| DB migrations | ✅ | ✅ | ✅ Done (prod) |
| Build + deploy backend | — | — | Pending next release |
| `cargo audit` / `cargo deny` | — | — | Run anytime |
| Schemathesis fuzzing | ✅ | ✅ | Pending OpenAPI spec |
| Playwright e2e | ✅ | ✅ | Pending URL cleanup |
| Backend integration tests | ✅ | — | ✅ Green in CI |
| `npm run lint` | — | — | ✅ Passing |
| `tsc --noEmit` | — | — | Run anytime |