fix(db): 4 fresh-bootstrap migration chain bugs found + verified today
All checks were successful
build-and-release / build (cron) (push) Successful in 58s
build-and-release / build (companies) (push) Successful in 1m59s
build-and-release / build (customers) (push) Successful in 2m9s
build-and-release / build (employees) (push) Successful in 2m17s
build-and-release / build (developers) (push) Successful in 2m29s
build-and-release / build (catering-services) (push) Successful in 2m45s
build-and-release / build (gateway) (push) Successful in 49s
build-and-release / build (jobs) (push) Successful in 42s
build-and-release / build (fitness-trainers) (push) Successful in 2m38s
build-and-release / build (graphic-designers) (push) Successful in 1m57s
build-and-release / build (payments) (push) Successful in 1m54s
build-and-release / build (job-seekers) (push) Successful in 2m57s
build-and-release / build (makeup-artists) (push) Successful in 2m47s
build-and-release / build (photographers) (push) Successful in 2m48s
build-and-release / build (social-media-managers) (push) Successful in 2m59s
backend-integration-tests / ai-credits (push) Successful in 50s
build-and-release / build (tutors) (push) Successful in 2m45s
build-and-release / build (ugc-content-creators) (push) Successful in 2m50s
build-and-release / build (video-editors) (push) Successful in 2m40s
build-and-release / build (users) (push) Successful in 4m47s

Reconciling the local dev DB (63 migrations behind) surfaced 4 genuine
bugs in the migration chain itself -- not just this DB's legacy
scripts/init-db.sql seeding -- confirmed by replaying the full chain
against a brand new, completely empty database from scratch:

- 20260318235959: customer_profiles.status is UPDATEd by
  20260319090000's backfill 4 months before it's ever ADDed
  (20260721020000). Adds it early (IF NOT EXISTS).
- 20260401235959: 20260317190000 creates the old users-linked
  `employees` shape; 20260402030000's CREATE TABLE IF NOT EXISTS no-ops
  against it and fails creating an index on a column that was never
  added. Conditionally drops the old shape first (only if it's still
  pre-transition -- gated on the missing `email` column), restoring
  20260402030000's own documented original intent.
- 20260419235959: the schema `20260420000003_external_role_modules`
  needs (persona_types, modules, role_module_access, ...) was only ever
  defined in a disabled .up.sql.skip (its version slot was taken by
  ...seed.sql); the seed data that depends on it was never skipped.
  Creates that schema early -- verbatim content, purely additive.
- 20260421235959: same disabled-migration pattern for the
  role_permissions -> role_admin_permissions / dashboard_configs ->
  role_sidebar_configs / runtime_configs -> role_runtime_configs /
  user_roles -> user_role_assignments renames that 20260422000000's
  widget seed needs. Conditionally renames each pair (only if old name
  exists and new name doesn't), verified against live Rust code that
  already queries the new names exclusively.

payments/invoices were NOT chain bugs -- confirmed clean on the fresh
bootstrap test -- so no fix needed for those; they only conflicted on
this one DB because of its scripts/init-db.sql legacy seed, which is
already a documented, known gap.

Verified twice: applied cleanly as no-ops against the now-fully-migrated
local dev DB, and applied successfully end-to-end (0 manual steps)
against a brand new empty database created from scratch.

Runbook updated with the full pending-migration list and today's
findings.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Ashwin Kumar Sivakumar 2026-08-14 18:30:42 +05:30
parent c394d9fc07
commit cc2dab112b
9 changed files with 288 additions and 4 deletions

View file

@ -0,0 +1,3 @@
-- No-op: 20260721020000_customer_profiles_status.down.sql already owns
-- dropping this column. Reverting it here too would break that migration's
-- own down.sql if both are ever rolled back in the same pass.

View file

@ -0,0 +1,13 @@
-- 20260319090000_backfill_active_profiles_to_approved.up.sql UPDATEs
-- customer_profiles.status, but that column isn't ADDed until
-- 20260721020000_customer_profiles_status -- four months later in this
-- chain. Any environment bootstrapping fresh via `sqlx migrate run` hits
-- "column status does not exist" at 20260319090000, regardless of whether
-- it's ever seeded by anything else first.
--
-- This runs one migration before that backfill and adds the column early
-- (IF NOT EXISTS, matching 20260721020000's own guard exactly), so that
-- migration and everything after it can proceed. 20260721020000 then
-- becomes a safe no-op when it's reached later in the same run.
ALTER TABLE customer_profiles
ADD COLUMN IF NOT EXISTS status VARCHAR(50) NOT NULL DEFAULT 'DRAFT';

View file

@ -0,0 +1,4 @@
-- No down: this only drops a table that 20260402030000 immediately
-- recreates in the new shape. There's no old-shape data to restore --
-- the transition it replays already happened in every real environment
-- long before this migration existed.

View file

@ -0,0 +1,30 @@
-- 20260402030000_strict_employee_separation's own comment documents this:
-- the migration originally did `DROP TABLE employees CASCADE` to replace
-- the old users-linked employees (created by
-- 20260317190000_complete_users_schema) with the new standalone-auth
-- shape. That DROP was later removed from the file because the old
-- db-migrate tool replayed every .sql on every run, turning a one-time
-- transformation into one that wiped live employee accounts on every
-- deploy. Production made this transition long ago, so the file was
-- simplified to a plain `CREATE TABLE IF NOT EXISTS`.
--
-- But on any environment that hasn't made that transition yet -- a fresh
-- bootstrap chief among them -- 20260317190000's old-shape `employees`
-- (user_id/role_id, no email/password_hash) is still sitting there, so
-- 20260402030000's CREATE TABLE IF NOT EXISTS no-ops against it and the
-- very next statement (CREATE INDEX ... employees(email)) fails.
--
-- This restores the original one-time DROP, but gated: it only fires if
-- employees still has the old pre-transition shape (no `email` column).
-- Any environment that already transitioned (all real ones today, per the
-- comment above) is untouched.
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'employees')
AND NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'employees' AND column_name = 'email'
) THEN
DROP TABLE employees CASCADE;
END IF;
END $$;

View file

@ -0,0 +1,5 @@
-- No down: this only ever creates objects that either (a) the disabled
-- 20260420000003_external_role_modules.up.sql.skip already defines the
-- teardown intent for, or (b) already existed independently of this
-- migration in any environment where it's a no-op. Nothing here is safe
-- to unilaterally drop without knowing which case applies.

View file

@ -0,0 +1,164 @@
-- 20260420000003_external_role_modules.up.sql was renamed to .up.sql.skip
-- (its version slot was already claimed by ...seed.sql), which meant this
-- schema was never actually created by any active migration -- only the
-- seed data that depends on it (persona_types, modules, module_actions,
-- role_module_widgets) runs. Any fresh bootstrap hits "relation ... does
-- not exist" at that seed step. This is the .skip file's content, applied
-- one migration earlier under its own version so the seed step succeeds.
-- Verified: purely additive (CREATE TABLE/INDEX IF NOT EXISTS, ADD COLUMN
-- IF NOT EXISTS throughout), safe no-op on any environment that already
-- has this schema by other means.
-- ============================================
-- ADD COLUMNS TO ROLES for external role settings
-- ============================================
ALTER TABLE roles ADD COLUMN IF NOT EXISTS persona_type varchar(50);
ALTER TABLE roles ADD COLUMN IF NOT EXISTS onboarding_schema_key varchar(100);
ALTER TABLE roles ADD COLUMN IF NOT EXISTS verification_required boolean DEFAULT true;
ALTER TABLE roles ADD COLUMN IF NOT EXISTS switch_services_enabled boolean DEFAULT false;
ALTER TABLE roles ADD COLUMN IF NOT EXISTS is_publicly_discoverable boolean DEFAULT true;
ALTER TABLE roles ADD COLUMN IF NOT EXISTS external_role_description text;
ALTER TABLE roles ADD COLUMN IF NOT EXISTS sort_order integer DEFAULT 0;
-- ============================================
-- persona_types (categories for external roles)
-- ============================================
CREATE TABLE IF NOT EXISTS persona_types (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
code varchar(50) UNIQUE NOT NULL,
name varchar(100) NOT NULL,
description text,
is_active boolean DEFAULT true,
created_at timestamptz DEFAULT NOW(),
updated_at timestamptz DEFAULT NOW()
);
-- ============================================
-- modules (module registry)
-- ============================================
CREATE TABLE IF NOT EXISTS modules (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
module_key varchar(50) UNIQUE NOT NULL,
module_name varchar(100) NOT NULL,
category varchar(50), -- core/content/marketplace/work/financial
description text,
backend_domain varchar(100),
default_route varchar(255),
default_sidebar_label varchar(100),
icon_key varchar(50),
is_core boolean DEFAULT false,
is_active boolean DEFAULT true,
created_at timestamptz DEFAULT NOW(),
updated_at timestamptz DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_modules_category ON modules(category);
CREATE INDEX IF NOT EXISTS idx_modules_active ON modules(is_active);
-- ============================================
-- role_module_access (module visibility per role)
-- ============================================
CREATE TABLE IF NOT EXISTS role_module_access (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
role_id uuid NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
module_id uuid NOT NULL REFERENCES modules(id) ON DELETE CASCADE,
is_enabled boolean DEFAULT true,
is_sidebar_visible boolean DEFAULT true,
sidebar_label_override varchar(100),
route_override varchar(255),
sort_order integer DEFAULT 0,
created_at timestamptz DEFAULT NOW(),
UNIQUE(role_id, module_id)
);
CREATE INDEX IF NOT EXISTS idx_role_module_access_role ON role_module_access(role_id);
CREATE INDEX IF NOT EXISTS idx_role_module_access_module ON role_module_access(module_id);
-- ============================================
-- module_actions (CRUD actions per module)
-- ============================================
CREATE TABLE IF NOT EXISTS module_actions (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
module_id uuid NOT NULL REFERENCES modules(id) ON DELETE CASCADE,
action_key varchar(50) NOT NULL,
action_name varchar(100) NOT NULL,
description text,
is_active boolean DEFAULT true,
created_at timestamptz DEFAULT NOW(),
UNIQUE(module_id, action_key)
);
CREATE INDEX IF NOT EXISTS idx_module_actions_module ON module_actions(module_id);
-- ============================================
-- role_module_permissions (permissions per module per role)
-- ============================================
CREATE TABLE IF NOT EXISTS role_module_permissions (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
role_id uuid NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
module_id uuid NOT NULL REFERENCES modules(id) ON DELETE CASCADE,
can_view boolean DEFAULT false,
can_list boolean DEFAULT false,
can_create boolean DEFAULT false,
can_update boolean DEFAULT false,
can_delete boolean DEFAULT false,
extra_actions_json jsonb DEFAULT '{}',
created_at timestamptz DEFAULT NOW(),
UNIQUE(role_id, module_id)
);
CREATE INDEX IF NOT EXISTS idx_role_module_permissions_role ON role_module_permissions(role_id);
CREATE INDEX IF NOT EXISTS idx_role_module_permissions_module ON role_module_permissions(module_id);
-- ============================================
-- role_module_widgets (widgets per module per role)
-- ============================================
CREATE TABLE IF NOT EXISTS role_module_widgets (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
role_id uuid NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
module_id uuid NOT NULL REFERENCES modules(id) ON DELETE CASCADE,
widget_key varchar(50),
is_enabled boolean DEFAULT true,
sort_order integer DEFAULT 0,
created_at timestamptz DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_role_module_widgets_role ON role_module_widgets(role_id);
CREATE INDEX IF NOT EXISTS idx_role_module_widgets_module ON role_module_widgets(module_id);
-- ============================================
-- module_variants (role-specific module variants)
-- ============================================
CREATE TABLE IF NOT EXISTS module_variants (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
module_id uuid NOT NULL REFERENCES modules(id) ON DELETE CASCADE,
variant_key varchar(50) NOT NULL,
variant_name varchar(100) NOT NULL,
role_code varchar(50), -- target role (e.g., PHOTOGRAPHER, TUTOR)
persona_type varchar(50), -- target persona (e.g., PROFESSIONAL)
schema_key varchar(100),
ui_template_key varchar(100),
is_active boolean DEFAULT true,
created_at timestamptz DEFAULT NOW(),
updated_at timestamptz DEFAULT NOW(),
UNIQUE(module_id, variant_key)
);
CREATE INDEX IF NOT EXISTS idx_module_variants_module ON module_variants(module_id);
CREATE INDEX IF NOT EXISTS idx_module_variants_role ON module_variants(role_code);
-- ============================================
-- role_module_variant_mapping (which variants a role uses)
-- ============================================
CREATE TABLE IF NOT EXISTS role_module_variant_mapping (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
role_id uuid NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
module_id uuid NOT NULL REFERENCES modules(id) ON DELETE CASCADE,
module_variant_id uuid NOT NULL REFERENCES module_variants(id) ON DELETE CASCADE,
is_active boolean DEFAULT true,
created_at timestamptz DEFAULT NOW(),
UNIQUE(role_id, module_id, module_variant_id)
);
CREATE INDEX IF NOT EXISTS idx_role_module_variant_mapping_role ON role_module_variant_mapping(role_id);
CREATE INDEX IF NOT EXISTS idx_role_module_variant_mapping_module ON role_module_variant_mapping(module_id);

View file

@ -0,0 +1,4 @@
-- No down: reversing these renames would break whichever later migration
-- or app code has since come to depend on the new names. See
-- 20260420000002_cleanup_role_tables.down.sql if a full rollback of the
-- rename itself is ever actually needed.

View file

@ -0,0 +1,37 @@
-- 20260420000002_cleanup_role_tables.up.sql (the rename half relevant here)
-- was disabled as .up.sql.skip, so role_permissions/dashboard_configs/
-- runtime_configs/user_roles never get renamed to the names the live app
-- code (config.rs, external_roles.rs, user_roles.rs, ...) actually queries:
-- role_admin_permissions/role_sidebar_configs/role_runtime_configs/
-- user_role_assignments. 20260422000000_seed_widgets needs
-- role_sidebar_configs to exist, so any fresh bootstrap fails there.
--
-- Unlike the external-role-module tables migration, this can't just be
-- "IF NOT EXISTS" -- a RENAME's source table stops existing once it's
-- done, so re-running it errors on an environment where it already
-- happened (including this repo's own production, per the skip file's own
-- history). Each rename below only fires if the old name still exists and
-- the new name doesn't yet -- safe to run on any environment regardless of
-- which (if any) of these renames it already has.
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'role_permissions')
AND NOT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'role_admin_permissions') THEN
ALTER TABLE role_permissions RENAME TO role_admin_permissions;
END IF;
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'dashboard_configs')
AND NOT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'role_sidebar_configs') THEN
ALTER TABLE dashboard_configs RENAME TO role_sidebar_configs;
END IF;
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'runtime_configs')
AND NOT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'role_runtime_configs') THEN
ALTER TABLE runtime_configs RENAME TO role_runtime_configs;
END IF;
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'user_roles')
AND NOT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'user_role_assignments') THEN
ALTER TABLE user_roles RENAME TO user_role_assignments;
END IF;
END $$;

View file

@ -20,9 +20,9 @@ All four DB tasks below are done and confirmed against the live production DB.
---
## ⏳ Pending — DB migrations for AI credit purchases (2026-08-14)
## ⏳ Pending — DB migrations for AI credit purchases + coupons + fresh-bootstrap fixes (2026-08-14)
Three new migrations need to be applied to prod. Run **in order** (sqlx-migrate handles this, but listed explicitly so you can verify):
Eight new migrations need to be applied to prod. Run **in order** (sqlx-migrate handles this, but listed explicitly so you can verify):
```bash
# On the live server, inside nxtgauge-backend-rust:
@ -30,19 +30,30 @@ git pull origin high-performance
sqlx migrate run --database-url "$DATABASE_URL"
```
That single command applies all three in chronological order. Verify after:
That single command applies all eight in chronological order. Verify after:
```bash
sqlx migrate info --database-url "$DATABASE_URL"
```
All three should show `applied`.
All eight should show `applied`. On prod specifically, the four "early" fixes
below should all apply as **no-ops** — prod already has the target schema
(that's how their gating conditions were derived and verified). If any of
them actually *does* something on prod (i.e. its `Applied` log line isn't
near-instant), stop and check `\d <table>` before continuing — it means
prod's schema differs from what this runbook assumed.
| Migration | What | Why |
|---|---|---|
| `20260318235959_customer_profiles_status_early.up.sql` | Adds `customer_profiles.status` early (`IF NOT EXISTS`) | `20260319090000`'s backfill UPDATEs this column 4 months before `20260721020000` used to add it — broke any fresh bootstrap |
| `20260401235959_drop_old_employees_table_early.up.sql` | Conditionally drops the old `users`-linked `employees` table (only if it still has the pre-transition shape — no `email` column) | `20260402030000` no-ops its `CREATE TABLE IF NOT EXISTS` against the old shape, then fails creating an index on a column that was never added |
| `20260419235959_external_role_module_tables_early.up.sql` | Creates `persona_types`/`modules`/`role_module_access`/`module_actions`/etc. (the disabled `20260420000003_external_role_modules.up.sql.skip`'s content, purely additive) | The seed migration that depends on these (`20260420000003...seed.sql`) has no active migration that creates them |
| `20260421235959_role_config_table_renames_early.up.sql` | Conditionally renames `role_permissions→role_admin_permissions`, `dashboard_configs→role_sidebar_configs`, `runtime_configs→role_runtime_configs`, `user_roles→user_role_assignments` (only fires per-table if old name exists and new name doesn't) | `20260422000000`'s widget seed needs `role_sidebar_configs`; the rename that creates it (`20260420000002...skip`) was disabled |
| `20260814000000_fix_ai_credit_packages_pricing_and_orders.up.sql` | Fixes prices (99→9900 paise etc.) + creates `ai_credit_orders` with `coupon_code` + `discount_applied` columns | AI credit purchase flow was completely broken: no orders table, wrong prices |
| `20260814010000_ai_credit_packages_role_scoping.up.sql` | Adds `applicable_roles TEXT[]` column to `ai_credit_packages` | Admin UI now supports restricting packages to specific roles |
| `20260814020000_ai_coupons.up.sql` | Creates `ai_coupons` + `ai_coupon_redemptions` tables | Backend `create_order`/`verify_order` reference these tables |
| `20260814030000_invoices_payment_id_polymorphic.up.sql` + `20260814050000_invoices_payment_id_polymorphic_check.up.sql` | Drops the hard FK on `invoices.payment_id` (needed for `AI_CREDIT_PURCHASE` invoices, which point at `ai_credit_orders` not `payments`), then replaces it with a `BEFORE INSERT/UPDATE` trigger that validates `payment_id` against the right table per `invoice_type` | Postgres has no native polymorphic FK; the first migration alone left `payment_id` completely unvalidated |
| `20260814040000_waitlist_signups.up.sql` | Creates the `nxtgauge.com` coming-soon waitlist table | Backs `POST /api/waitlist` |
After applying, redeploy the payments service:
@ -50,6 +61,19 @@ After applying, redeploy the payments service:
systemctl restart nxtgauge-payments
```
### Local dev DB note (2026-08-14)
The local dev Postgres (`nxtgauge-backend-rust-postgres-1`) was found ~63
migrations behind (stalled since March) and was brought fully current as
part of today's work, including manually reconciling `payments`/`invoices`
legacy tables seeded by `scripts/init-db.sql` (dropped + recreated, both
verified empty first — not a chain bug, doesn't affect fresh bootstraps).
The four "early" migrations above were then written and verified against
**both** that reconciled DB (all four applied as no-ops) **and** a
completely fresh, empty database created from scratch (`sqlx migrate run`
end-to-end, zero manual intervention) — so any new environment, or anyone
re-bootstrapping from zero, should no longer hit any of these four walls.
---
## 1. Deploy backend changes