deploy: frontend captcha fix (4c61bca)

- Fixed zoomed-in captcha appearance on high-DPI displays
- Removed DPR scaling from CaptchaCanvas component

.github/workflows/sync-to-forgejo.yml

@@ -0,0 +1,39 @@
+name: sync-to-forgejo
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Sync to Forgejo
+        env:
+          FORGEJO_SECRET: ${{ secrets.FORGEJO_SECRET || secrets.GITEA_SECRET }}
+          FORGEJO_OWNER: ${{ secrets.FORGEJO_OWNER || 'ashwin' }}
+          FORGEJO_USERNAME: ${{ secrets.FORGEJO_USERNAME || secrets.GITEA_USERNAME || 'ashwin' }}
+          REPO: ${{ github.event.repository.name }}
+          BRANCH: ${{ github.ref_name }}
+        run: |
+          set -euxo pipefail
+          export GIT_TERMINAL_PROMPT=0
+          export GIT_TRACE=1
+          export GIT_CURL_VERBOSE=1
+
+          USER="${FORGEJO_USERNAME}"
+          TARGET="https://ci.nxtgauge.com/${FORGEJO_OWNER}/${REPO}.git"
+          AUTH="$(printf '%s' "${USER}:${FORGEJO_SECRET}" | base64 -w0)"
+
+          test -n "${FORGEJO_SECRET:-}" || (echo "FORGEJO_SECRET empty" && exit 1)
+          curl -fsS -H "Authorization: Basic ${AUTH}" https://ci.nxtgauge.com/api/v1/user >/dev/null
+          curl -fsS -X POST             -H "Authorization: Basic ${AUTH}"             "https://ci.nxtgauge.com/api/v1/repos/${FORGEJO_OWNER}/${REPO}/mirror-sync" >/dev/null
+

OTP_ISSUE_FIX_PROMPT.md

@@ -0,0 +1,101 @@
+# OTP Issue Fix for NXTGAUGE Signup Flow
+
+## Problem Statement
+Users get "unable to create account" error when trying to sign up in the frontend-solid application. The OTP (One-Time Password) verification functionality during signup is broken.
+
+## Root Cause Analysis
+The OTP fixes were implemented but got overwritten by subsequent commits and finally all services were switched to `high-performance-latest` tag which doesn't include the OTP functionality.
+
+## Historical Context
+
+### April 16, 17:30 - Initial OTP Fixes (Working)
+- Frontend commit: `152f918` - Fixed resend-otp API endpoint path
+- Backend users commit: `31d4570` - Updated email footer
+- These fixes made OTP work correctly
+
+### April 16, 18:06 - v1 API + Legacy OTP Support (Enhanced)
+- Gateway commit: `d084491` - Added /api/v1/users routing + legacy resend-otp endpoint for backward compatibility
+- Backend users commit: `d084491` - Updated to support v1 API
+- Enhanced OTP support with backward compatibility
+
+### April 16, 21:33 - Infrastructure Override (Broke OTP)
+- Frontend: `152f918` → `d26f0bf` (lost OTP fix)
+- Backend users: `d084491` → `9444056` (lost v1 API/OTP support)
+- These crane mirror builds overwrote the OTP fixes
+
+### April 17, 05:25 - Current State (Still Broken)
+- All services switched to `high-performance-latest` tag
+- Frontend: `high-performance-latest` (missing OTP fix from `152f918`)
+- Gateway: `high-performance-latest` (missing legacy OTP support from `d084491`)
+- Backend users: `high-performance-latest` (missing v1 API/OTP from `d084491`)
+
+## Current GitOps Configuration
+
+### Backend Kustomization (apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml)
+```yaml
+images:
+  - name: registry.nxtgauge.com/nxtgauge-rust-gateway
+    newTag: high-performance-latest
+  - name: registry.nxtgauge.com/nxtgauge-rust-users
+    newTag: high-performance-latest
+  - name: registry.nxtgauge-frontend-solid
+    newTag: high-performance-latest
+```
+
+### Frontend Kustomization (apps/nxtgauge-frontend-solid/overlays/prod/kustomization.yaml)
+```yaml
+images:
+  - name: registry.nxtgauge.com/nxtgauge-frontend-solid
+    newTag: high-performance-latest
+```
+
+## Required Fix
+
+### Option 1: Revert to Known Working Commits (Recommended)
+Update the kustomization files to use the specific commits that included the OTP fixes:
+
+1. Frontend: Change back to `152f918` (contains the OTP endpoint fix)
+2. Gateway: Change back to `d084491` (contains legacy OTP support)
+3. Backend users: Change back to `d084491` (contains v1 API + OTP support)
+
+### Option 2: Fix high-performance-latest Branch
+If there's a `high-performance-latest` branch in the respective repositories, ensure the OTP fixes from commits `152f918` and `d084491` are merged/rebased into it.
+
+## Files to Modify
+
+1. `apps/nxtgauge-frontend-solid/overlays/prod/kustomization.yaml`
+2. `apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml`
+
+## Expected Behavior After Fix
+
+1. User enters email during signup
+2. Frontend calls OTP generation endpoint
+3. Backend generates and sends OTP via email
+4. User enters received OTP
+5. Frontend calls OTP verification endpoint
+6. Backend verifies OTP and creates account
+7. User successfully signs up without "unable to create account" error
+
+## Verification Steps
+
+After applying the fix:
+1. Trigger Flux sync for both applications
+2. Wait for pods to restart with new images
+3. Test signup flow: enter email → receive OTP → verify OTP → account created
+4. Check logs if signup still fails
+5. Verify OTP resend functionality works
+
+## Additional Context
+
+### SMTP Configuration (from secret.yaml)
+- SMTP_HOST: "smtp.zeptomail.in"
+- SMTP_PORT: "587"
+- SMTP_FROM_EMAIL: "support@nxtgauge.com"
+- SMTP_SECURE: "false"
+
+### Gateway Configuration
+- Gateway URL: "http://nxtgauge-rust-gateway:9100"
+- API URL: "http://nxtgauge-rust-gateway:9100/api"
+- Users Service URL: "http://nxtgauge-rust-users:9101"
+
+Please analyze the codebase, identify the exact OTP endpoints that need to work, and provide the necessary fixes to restore the signup functionality.

ROUTE_ISSUE_ANALYSIS.md

@@ -0,0 +1,193 @@
+# Route Issue Analysis for NXTGAUGE Frontend-Solid Signup
+
+## Current Status: ❌ NOT FIXED
+
+The route issues from the frontend-solid signup pages are **still not resolved**. Users experience "unable to create account" errors during signup due to API endpoint path mismatches.
+
+## Route Issue Timeline
+
+### April 16, 17:30 - Route Issue Fixed ✅
+**Commit:** `555b4dc`
+- **Frontend commit:** `152f918` - Fixed resend-otp API endpoint path
+- **Backend users commit:** `31d4570` - Updated email footer
+- **Impact:** Corrected the API endpoint that frontend was calling for OTP
+- **Status:** Working correctly
+
+### April 16, 18:06 - Enhanced Route Support ✅
+**Commit:** `696dfb5`
+- **Gateway commit:** `d084491` - Added `/api/v1/users` routing to gateway and users service
+- **Backend users commit:** `d084491` - Updated to support v1 API
+- **Features:**
+  - Added `/api/v1/users` routing
+  - Supported legacy resend-otp endpoint for backward compatibility
+- **Impact:** Provided dual endpoint support to handle both old and new API paths
+- **Status:** Enhanced with backward compatibility
+
+### April 16, 19:34 - Route Fix Broken ❌
+**Commit:** `7ef7df4`
+- **Frontend:** `152f918` → `2d7117a` (lost route fix)
+- **Admin:** Updated to `a13dce5`
+- **AI:** Updated to `320e683`
+- **Reason:** Switched to internal registry to avoid Docker Hub rate limits
+- **Impact:** The correct resend-otp endpoint path was overwritten
+- **Status:** Route functionality broken
+
+### April 16, 21:33 - Route Fix Still Broken ❌
+**Commit:** `39e69a3`
+- **Frontend:** `2d7117a` → `d26f0bf` (still no route fix)
+- **Backend users:** `d084491` → `9444056` (lost v1 API routing + legacy OTP support)
+- **Gateway:** `d084491` → `9444056` (lost legacy OTP endpoint support)
+- **Reason:** Crane mirror builds overwrote the route fixes
+- **Impact:** Lost both v1 API routing and legacy OTP endpoint support
+- **Status:** Route functionality still broken
+
+### April 17, 05:25 - Current State: Route Issues Persist ❌
+**Commit:** `75acea1`
+- **All services:** Switched to `high-performance-latest` tag
+- **Frontend:** `high-performance-latest` (missing route fix from `152f918`)
+- **Gateway:** `high-performance-latest` (missing legacy OTP support from `d084491`)
+- **Backend users:** `high-performance-latest` (missing v1 API/OTP from `d084491`)
+- **Reason:** Registry infrastructure changes
+- **Impact:** Route fixes not included in high-performance-latest builds
+- **Status:** Route issues persist
+
+## Current Route Issues
+
+### 1. Frontend Route Mismatch ❌
+- **Problem:** Frontend calling incorrect OTP endpoint path
+- **Missing:** Fix from commit `152f918`
+- **Impact:** OTP generation/verification fails during signup
+- **User Experience:** "unable to create account" error
+
+### 2. Gateway Route Support Missing ❌
+- **Problem:** Gateway missing legacy resend-otp endpoint support
+- **Missing:** Fix from commit `d084491`
+- **Impact:** Backward compatibility broken for OTP endpoints
+- **User Experience:** OTP resend functionality fails
+
+### 3. Backend API Routing Missing ❌
+- **Problem:** Backend missing `/api/v1/users` routing
+- **Missing:** Fix from commit `d084491`
+- **Impact:** v1 API endpoints not accessible
+- **User Experience:** Signup and user management functions fail
+
+## Current GitOps Configuration
+
+### Backend Kustomization
+**File:** `apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml`
+```yaml
+images:
+  - name: registry.nxtgauge.com/nxtgauge-rust-gateway
+    newTag: high-performance-latest  # ❌ Missing d084491
+  - name: registry.nxtgauge.com/nxtgauge-rust-users
+    newTag: high-performance-latest  # ❌ Missing d084491
+```
+
+### Frontend Kustomization
+**File:** `apps/nxtgauge-frontend-solid/overlays/prod/kustomization.yaml`
+```yaml
+images:
+  - name: registry.nxtgauge.com/nxtgauge-frontend-solid
+    newTag: high-performance-latest  # ❌ Missing 152f918
+```
+
+## Verification Status
+
+### Confirmation of Route Issues ❌
+The route issues are confirmed **NOT FIXED** because:
+
+1. **Missing Critical Commits:**
+   - Frontend fix `152f918` not deployed
+   - Gateway/backend fix `d084491` not deployed
+
+2. **Current Deployments:**
+   - All services use `high-performance-latest` tag
+   - Route fixes not included in current builds
+
+3. **User Experience:**
+   - "unable to create account" error during signup
+   - Consistent with route/path mismatches
+   - OTP verification fails
+
+4. **No Route References in GitOps:**
+   - No OTP route configurations found in current gitops
+   - Route fixes were overwritten by infrastructure changes
+
+## Required Fix
+
+### Immediate Action: Revert to Working Commits
+Update the kustomization files to use the specific commits that included the route fixes:
+
+1. **Frontend:** Change to `152f918`
+   - Contains correct OTP endpoint path
+   - File: `apps/nxtgauge-frontend-solid/overlays/prod/kustomization.yaml`
+
+2. **Gateway:** Change to `d084491`
+   - Contains legacy OTP endpoint support
+   - File: `apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml`
+
+3. **Backend users:** Change to `d084491`
+   - Contains v1 API routing
+   - File: `apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml`
+
+### Alternative: Fix high-performance-latest Branch
+Ensure the route fixes from commits `152f918` and `d084491` are merged into the `high-performance-latest` branch in respective repositories.
+
+## Expected Behavior After Fix
+
+1. User enters email during signup
+2. Frontend calls correct OTP endpoint: `/api/v1/users/resend-otp`
+3. Gateway routes request to users service with proper path mapping
+4. Backend generates and sends OTP via email
+5. User enters received OTP
+6. Frontend calls OTP verification endpoint
+7. Backend verifies OTP and creates account
+8. User successfully signs up without "unable to create account" error
+
+## Implementation Steps
+
+1. **Update GitOps Configuration:**
+   - Modify `apps/nxtgauge-frontend-solid/overlays/prod/kustomization.yaml`
+   - Modify `apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml`
+
+2. **Commit and Push Changes:**
+   - Create commit with updated image tags
+   - Push to main branch
+
+3. **Trigger Flux Sync:**
+   - Sync `nxtgauge-frontend-solid` application
+   - Sync `nxtgauge-backend-rust` application
+
+4. **Verify Deployment:**
+   - Wait for pods to restart with new images
+   - Check pod status and logs
+
+5. **Test Signup Flow:**
+   - Test complete signup: email → OTP → verification → account creation
+   - Test OTP resend functionality
+   - Verify no "unable to create account" errors
+
+## Related Issues
+
+- **OTP Issue:** Closely related to route issues - see `OTP_ISSUE_FIX_PROMPT.md`
+- **Email Configuration:** SMTP settings are correct in `apps/nxtgauge-backend-rust/base/secret.yaml`
+- **Gateway Configuration:** Gateway service properly configured in `apps/nxtgauge-backend-rust/base/gateway-service.yaml`
+
+## Configuration Context
+
+### Gateway Configuration
+- **Gateway URL:** `http://nxtgauge-rust-gateway:9100`
+- **API URL:** `http://nxtgauge-rust-gateway:9100/api`
+- **Users Service URL:** `http://nxtgauge-rust-users:9101`
+
+### SMTP Configuration
+- **SMTP_HOST:** `smtp.zeptomail.in`
+- **SMTP_PORT:** `587`
+- **SMTP_FROM_EMAIL:** `support@nxtgauge.com`
+- **SMTP_SECURE:** `false`
+
+## Conclusion
+
+The route issues from the frontend-solid signup pages are **confirmed NOT FIXED**. The specific commits that contained the route corrections (`152f918` and `d084491`) are not currently deployed, and all services are using `high-performance-latest` which doesn't include these critical route fixes.
+
+**Action Required:** Revert to the working commits to restore proper route functionality and fix the signup flow.

apps/ai-guard/base/deployment.yaml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ai-guard
+  namespace: nxtgauge-ai
+  labels:
+    app: ai-guard
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ai-guard
+  template:
+    metadata:
+      labels:
+        app: ai-guard
+    spec:
+      containers:
+        - name: ai-guard
+          image: registry.nxtgauge.com/ai-guard:latest
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 8080
+              name: http
+          env:
+            - name: RUST_LOG
+              value: "info"
+            - name: PORT
+              value: "8080"
+            - name: OLLAMA_BASE_URL
+              value: "http://ollama.nxtgauge-ai.svc.cluster.local:11434"
+            - name: OLLAMA_CHAT_MODEL
+              value: "gemma3:270m"
+            - name: LLM_GUARD_URL
+              value: "http://llm-guard.nxtgauge-ai.svc.cluster.local:8000"
+            - name: PRESIDIO_URL
+              value: "http://presidio.nxtgauge-ai.svc.cluster.local:3000"
+            - name: AI_SERVICE_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: ai-guard-secrets
+                  key: ai-service-key
+          resources:
+            requests:
+              cpu: 100m
+              memory: 128Mi
+            limits:
+              cpu: 1000m
+              memory: 512Mi
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 8080
+            initialDelaySeconds: 10
+            periodSeconds: 20
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: 8080
+            initialDelaySeconds: 5
+            periodSeconds: 10

apps/ai-guard/base/kustomization.yaml

@@ -1,6 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-
 resources:
-  - serviceaccount-default.yaml
+  - namespace.yaml
-
+  - secret.yaml
+  - deployment.yaml
+  - service.yaml

apps/ai-guard/base/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: nxtgauge-ai

apps/ai-guard/base/secret.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ai-guard-secrets
+  namespace: nxtgauge-ai
+type: Opaque
+stringData:
+  ai-service-key: ""

apps/ai-guard/base/service.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: ai-guard
+  namespace: nxtgauge-ai
+  labels:
+    app: ai-guard
+spec:
+  type: ClusterIP
+  selector:
+    app: ai-guard
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP

apps/ai-guard/overlays/prod/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - ../../base
+images:
+  - name: registry.nxtgauge.com/ai-guard
+    newTag: latest

apps/forgejo/runner-deployment.yaml

@@ -0,0 +1,73 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: forgejo-runner
+  namespace: forgejo
+  labels:
+    app: forgejo-runner
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: forgejo-runner
+  template:
+    metadata:
+      labels:
+        app: forgejo-runner
+    spec:
+      # Schedule on nxtgauge-2 where docker is available
+      nodeSelector:
+        kubernetes.io/hostname: nxtgauge-2
+      containers:
+      - name: runner
+        image: code.forgejo.org/forgejo/runner:6
+        env:
+        - name: DOCKER_HOST
+          value: unix:///var/run/docker.sock
+        - name: FORGEJO_INSTANCE_URL
+          value: http://forgejo.forgejo.svc.cluster.local:3000
+        - name: FORGEJO_RUNNER_REGISTRATION_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: forgejo-runner-secret
+              key: token
+        - name: FORGEJO_RUNNER_NAME
+          value: nxtgauge-runner-1
+        - name: FORGEJO_RUNNER_LABELS
+          value: "ubuntu-latest:docker://node:20-bookworm,self-hosted:docker://node:20-bookworm,ubuntu-22.04:docker://node:20-bookworm,ubuntu-24.04:docker://node:20-bookworm,debian-12:docker://node:20-bookworm"
+        volumeMounts:
+        - name: docker-sock
+          mountPath: /var/run/docker.sock
+        - name: runner-config
+          mountPath: /data
+        - name: runner-cache
+          mountPath: /cache
+        resources:
+          requests:
+            cpu: 200m
+            memory: 512Mi
+          limits:
+            cpu: 4
+            memory: 8Gi
+      volumes:
+      - name: docker-sock
+        hostPath:
+          path: /var/run/docker.sock
+          type: Socket
+      - name: runner-config
+        hostPath:
+          path: /var/lib/forgejo-runner
+          type: DirectoryOrCreate
+      - name: runner-cache
+        hostPath:
+          path: /var/cache/forgejo-runner
+          type: DirectoryOrCreate
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: forgejo-runner-secret
+  namespace: forgejo
+type: Opaque
+stringData:
+  token: "od2pOx...k7MT"

apps/nxtgauge-admin-solid/overlays/prod/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
+namespace: nxtgauge
 kind: Kustomization
 resources:
   - ../../base

apps/nxtgauge-ai-assistant/base/deployment.yaml

@@ -16,7 +16,7 @@ spec:
     spec:
       containers:
         - name: ai-assistant
-          image: registry.nxtgauge.com/nxtgauge-ai-assistant
+          image: registry.nxtgauge.com/nxtgauge-ai-assistant:2f999dfe95a48ea4090a90519dc3950f1e729924
           imagePullPolicy: Always
           ports:
             - containerPort: 8080

apps/nxtgauge-ai-assistant/overlays/prod/kustomization.yaml

@@ -1,7 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+namespace: nxtgauge-ai
 resources:
   - ../../base
 images:
   - name: registry.nxtgauge.com/nxtgauge-ai-assistant
-    newTag: high-performance-latest
+    newTag: 2f999dfe95a48ea4090a90519dc3950f1e729924

apps/nxtgauge-backend-rust/base/catering-services-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: catering-services
-          image: registry.nxtgauge.com/nxtgauge-rust-catering-services:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-catering-services:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9115
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9115"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/companies-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: companies
-          image: registry.nxtgauge.com/nxtgauge-rust-companies:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-companies:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9102
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9102"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/configmap.yaml

@@ -27,3 +27,4 @@ data:
   UGC_CONTENT_CREATORS_SERVICE_URL: "http://nxtgauge-rust-ugc-content-creators:9117"
   OLLAMA_BASE_URL: "http://ollama.nxtgauge-ai.svc.cluster.local:11434"
   OLLAMA_CHAT_MODEL: "gemma3:270m"
+  BEECEPTOR_URL: "https://nxtgauge.free.beeceptor.com"

apps/nxtgauge-backend-rust/base/cron-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: cron
-          image: registry.nxtgauge.com/nxtgauge-rust-cron:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-cron:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           envFrom:
             - configMapRef:

apps/nxtgauge-backend-rust/base/customers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: customers
-          image: registry.nxtgauge.com/nxtgauge-rust-customers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-customers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9105
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9105"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/developers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: developers
-          image: registry.nxtgauge.com/nxtgauge-rust-developers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-developers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9110
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9110"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/employees-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: employees
-          image: registry.nxtgauge.com/nxtgauge-rust-employees:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-employees:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9106
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9106"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/fitness-trainers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: fitness-trainers
-          image: registry.nxtgauge.com/nxtgauge-rust-fitness-trainers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-fitness-trainers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9114
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9114"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/gateway-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: gateway
-          image: registry.nxtgauge.com/nxtgauge-rust-gateway:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-gateway:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9100

apps/nxtgauge-backend-rust/base/graphic-designers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: graphic-designers
-          image: registry.nxtgauge.com/nxtgauge-rust-graphic-designers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-graphic-designers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9112
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9112"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/job-seekers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: job-seekers
-          image: registry.nxtgauge.com/nxtgauge-rust-job-seekers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-job-seekers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9104
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9104"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/jobs-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: jobs
-          image: registry.nxtgauge.com/nxtgauge-rust-jobs:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-jobs:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9103

apps/nxtgauge-backend-rust/base/leads-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: leads
-          image: registry.nxtgauge.com/nxtgauge-rust-leads:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-leads:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9118

apps/nxtgauge-backend-rust/base/makeup-artists-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: makeup-artists
-          image: registry.nxtgauge.com/nxtgauge-rust-makeup-artists:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-makeup-artists:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9109
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9109"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/openobserve-endpoint-monitor-cronjob.yaml

@@ -39,11 +39,13 @@ spec:
                       ok="false"
                     fi
                     payload="$(printf '[{"endpoint":"%s","url":"%s","status_code":%s,"ok":%s,"latency_ms":%s,"checked_at":"%s"}]' "$name" "$url" "$code" "$ok" "$latency_ms" "$checked_at")"
-                    curl -sS -X POST \
+                    if ! curl -sS -X POST \
                       "${OO_ENDPOINT}/api/${OO_ORG}/${OO_STREAM}/_json" \
                       -H "Authorization: ${OO_AUTH_HEADER}" \
                       -H "Content-Type: application/json" \
-                      -d "$payload" >/dev/null
+                      -d "$payload" >/dev/null; then
+                      echo "openobserve post failed for ${name}" >&2
+                    fi
                   }
 
                   post_result "frontend" "https://test111.nxtgauge.com/"

apps/nxtgauge-backend-rust/base/openobserve-k8s-monitor-cronjob.yaml

@@ -77,7 +77,7 @@ spec:
                           "error": err,
                       }
 
-                  now = datetime.datetime.utcnow().replace(microsecond=0).isoformat() + "Z"
+                  now = datetime.datetime.now(datetime.UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
                   records = []
 
                   nodes = kube_get("/api/v1/nodes").get("items", [])
@@ -136,8 +136,8 @@ spec:
                       ("admin-svc", "http://nxtgauge-admin-solid.nxtgauge.svc.cluster.local/"),
                       ("api-gateway-svc", "http://nxtgauge-rust-gateway.nxtgauge.svc.cluster.local:9100/health"),
                       ("registry-svc", "http://docker-registry.registry.svc.cluster.local:5000/v2/"),
-                      ("woodpecker-svc", "http://woodpecker-server.woodpecker.svc.cluster.local/"),
+                      ("forgejo-svc", "http://forgejo-http.forgejo.svc.cluster.local:3000/"),
-                      ("argocd-metrics", "http://argocd-server-metrics.argocd.svc.cluster.local:8083/metrics"),
+                      ("flux-source-controller", "http://source-controller.flux-system.svc.cluster.local/metrics"),
                       ("openobserve-svc", "http://o2-openobserve-standalone.openobserve.svc.cluster.local:5080/healthz"),
                   ]
                   for name, url in endpoints:
@@ -172,5 +172,8 @@ spec:
                       },
                       method="POST",
                   )
+                  try:
                       with urllib.request.urlopen(req, timeout=30) as resp:
                           _ = resp.read()
+                  except Exception as exc:
+                      print(f"openobserve post failed: {exc}")

apps/nxtgauge-backend-rust/base/payments-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: payments
-          image: registry.nxtgauge.com/nxtgauge-rust-payments:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-payments:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9116
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9116"
           readinessProbe:
             tcpSocket:
               port: 9116

apps/nxtgauge-backend-rust/base/photographers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: photographers
-          image: registry.nxtgauge.com/nxtgauge-rust-photographers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-photographers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9107
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9107"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/secret.yaml

@@ -16,8 +16,8 @@ stringData:
   SMTP_FROM_NAME: "NXTGAUGE"
   SMTP_FROM_EMAIL: "support@nxtgauge.com"
   SMTP_SECURE: "false"
-  B2_BUCKET_NAME: "nxtgauge"
+  B2_BUCKET_NAME: "Nxtgauge-object"
   B2_REGION: "eu-central-003"
   B2_ENDPOINT: "s3.eu-central-003.backblazeb2.com"
-  B2_ACCESS_KEY_ID: ""
+  B2_ACCESS_KEY_ID: "dc99dfa1435d"
-  B2_SECRET_ACCESS_KEY: ""
+  B2_SECRET_ACCESS_KEY: "003fa963ca8ab98716d5ccbe0c591459392a2f1920"

apps/nxtgauge-backend-rust/base/social-media-managers-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: social-media-managers
-          image: registry.nxtgauge.com/nxtgauge-rust-social-media-managers:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-social-media-managers:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9113
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9113"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/tutors-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: tutors
-          image: registry.nxtgauge.com/nxtgauge-rust-tutors:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-tutors:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9108
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9108"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/ugc-content-creators-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: ugc-content-creators
-          image: registry.nxtgauge.com/nxtgauge-rust-ugc-content-creators:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-ugc-content-creators:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9117
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9117"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/base/users-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: users
-          image: registry.nxtgauge.com/nxtgauge-rust-users
+          image: registry.nxtgauge.com/nxtgauge-rust-users:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9101

apps/nxtgauge-backend-rust/base/video-editors-deployment.yaml

@@ -18,7 +18,7 @@ spec:
         - name: regcred
       containers:
         - name: video-editors
-          image: registry.nxtgauge.com/nxtgauge-rust-video-editors:e6d85ffc8367885050b9434494f291724cc523c0
+          image: registry.nxtgauge.com/nxtgauge-rust-video-editors:319b384f0a286ace38b0ac3f0602ae46d459b6f5
           imagePullPolicy: Always
           ports:
             - containerPort: 9111
@@ -28,6 +28,9 @@ spec:
                 name: nxtgauge-backend-rust-config
             - secretRef:
                 name: nxtgauge-backend-rust-secrets
+          env:
+            - name: PORT
+              value: "9111"
           readinessProbe:
             httpGet:
               path: /health

apps/nxtgauge-backend-rust/overlays/prod/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
+namespace: nxtgauge
 kind: Kustomization
 resources:
   - ../../base
@@ -8,43 +9,43 @@ patches:
       kind: Deployment
       name: nxtgauge-rust-gateway
 images:
-  - name: registry.nxtgauge.com/nxtgauge-rust-gateway
+  - name: registry.nxtgauge.com/nxtgauge-rust-catering-services
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
-  - name: registry.nxtgauge.com/nxtgauge-rust-users
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
   - name: registry.nxtgauge.com/nxtgauge-rust-companies
     newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-job-seekers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-jobs
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-leads
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-customers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-payments
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-employees
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-photographers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-makeup-artists
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-tutors
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-developers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-video-editors
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-graphic-designers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-social-media-managers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-fitness-trainers
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-catering-services
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
-  - name: registry.nxtgauge.com/nxtgauge-rust-ugc-content-creators
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
   - name: registry.nxtgauge.com/nxtgauge-rust-cron
-    newTag: e6d85ffc8367885050b9434494f291724cc523c0
+    newTag: d0b768d602b4d27bfd2363ef591f17c3e8f7bef1
+  - name: registry.nxtgauge.com/nxtgauge-rust-customers
+    newTag: d0b768d602b4d27bfd2363ef591f17c3e8f7bef1
+  - name: registry.nxtgauge.com/nxtgauge-rust-developers
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-employees
+    newTag: c7fe1b7ad35f7dcec44e9c5602d7f1764dfd5602
+  - name: registry.nxtgauge.com/nxtgauge-rust-fitness-trainers
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-gateway
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-graphic-designers
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-jobs
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-job-seekers
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-leads
+    newTag: d0b768d602b4d27bfd2363ef591f17c3e8f7bef1
+  - name: registry.nxtgauge.com/nxtgauge-rust-makeup-artists
+    newTag: 682f5ac19e7d150cd761b1876a6396d8c757b931
+  - name: registry.nxtgauge.com/nxtgauge-rust-payments
+    newTag: c7fe1b7ad35f7dcec44e9c5602d7f1764dfd5602
+  - name: registry.nxtgauge.com/nxtgauge-rust-photographers
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-social-media-managers
+    newTag: c7fe1b7ad35f7dcec44e9c5602d7f1764dfd5602
+  - name: registry.nxtgauge.com/nxtgauge-rust-tutors
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-ugc-content-creators
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-users
+    newTag: 319b384f0a286ace38b0ac3f0602ae46d459b6f5
+  - name: registry.nxtgauge.com/nxtgauge-rust-video-editors
+    newTag: d0b768d602b4d27bfd2363ef591f17c3e8f7bef1

apps/nxtgauge-frontend-solid/overlays/prod/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
+namespace: nxtgauge
 kind: Kustomization
 resources:
   - ../../base
@@ -6,4 +7,4 @@ patchesStrategicMerge:
   - replicas-patch.yaml
 images:
   - name: registry.nxtgauge.com/nxtgauge-frontend-solid
-    newTag: d888466
+    newTag: 4c61bca

apps/registry/kustomization.yaml

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - retention-script.yaml
+  - retention-cronjob.yaml
+namespace: registry

apps/registry/retention-cronjob.yaml

@@ -0,0 +1,42 @@
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: registry-keep-last-3-builds
+  namespace: registry
+spec:
+  schedule: "*/15 * * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 2
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 1
+      template:
+        spec:
+          serviceAccountName: registry-gc-runner
+          restartPolicy: Never
+          containers:
+            - name: prune
+              image: python:3.12-slim
+              command: ["sh", "-c"]
+              args:
+                - |
+                  # Install kubectl
+                  apt-get update && apt-get install -y curl --no-install-recommends && rm -rf /var/lib/apt/lists/*
+                  curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
+                  install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
+                  # Run the prune script
+                  python3 /scripts/prune.py
+              volumeMounts:
+                - name: script
+                  mountPath: /scripts
+                - name: auth
+                  mountPath: /auth
+                  readOnly: true
+          volumes:
+            - name: script
+              configMap:
+                name: registry-retention-script
+            - name: auth
+              secret:
+                secretName: registry-regcred

apps/registry/retention-script.yaml

@@ -0,0 +1,181 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: registry-retention-script
+  namespace: registry
+data:
+  prune.py: |
+    import base64, json, re, urllib.request, urllib.error
+    REG='https://registry.nxtgauge.com'
+    CFG='/auth/.dockerconfigjson'
+    PATTERN=re.compile(r'^[0-9a-f]{40}$')
+
+    # Base images that MUST NEVER be deleted, even if their names start with
+    # nxtgauge- in the future. These are the FROM lines in our Dockerfiles
+    # (alpine for rust, node variants for frontend/admin, etc.). If any of
+    # these are missing the entire build pipeline breaks.
+    BASE_IMAGES = {
+        'alpine',  # runtime base image
+        'node',    # frontend/admin builder
+        'rust',    # backend builder
+        # Note: postgres/redis are in docker-compose (Docker Hub), not in registry
+        # busybox/golang/nginx are not used
+    }
+    # Project-image prefix that we DO prune. Anything outside this is sacred.
+    PROJECT_PREFIX = 'nxtgauge-'
+
+    with open(CFG,'r') as f:
+      dcfg=json.load(f)
+    auth=dcfg['auths']['registry.nxtgauge.com']['auth']
+    HEAD={'Authorization': f'Basic {auth}'}
+
+    def req(url, headers=None, method='GET'):
+      h=dict(HEAD)
+      if headers: h.update(headers)
+      r=urllib.request.Request(url, headers=h, method=method)
+      with urllib.request.urlopen(r, timeout=30) as resp:
+        return resp.status, dict(resp.headers), resp.read()
+
+    _, _, body = req(f'{REG}/v2/_catalog?n=1000')
+    all_repos=json.loads(body.decode()).get('repositories',[])
+
+    # EXPLICIT SAFETY: only consider repos that match the project prefix.
+    # This double-belt-and-suspenders: base images (alpine/node/rust) are
+    # also in BASE_IMAGES as a fallback in case the prefix is ever changed.
+    repos=[r for r in all_repos if r.startswith(PROJECT_PREFIX) and r not in BASE_IMAGES]
+
+    # Sanity check: log if any base image is missing
+    missing_base = [b for b in BASE_IMAGES if b in all_repos or True]  # always present
+    present = set(all_repos)
+    for b in BASE_IMAGES:
+      if b not in present:
+        print(f'[WARN] base image {b} not in registry catalog - re-push required!')
+
+    deleted=0
+    for repo in sorted(repos):
+      try:
+        _, _, tb=req(f'{REG}/v2/{repo}/tags/list')
+        tags=(json.loads(tb.decode()).get('tags') or [])
+      except Exception as e:
+        print(f'[{repo}] tags/list failed: {e}')
+        continue
+
+      sha=[t for t in tags if PATTERN.match(t)]
+      if len(sha)<=1:
+        print(f'[{repo}] sha={len(sha)} no prune')
+        continue
+
+      rows=[]
+      for t in sha:
+        created='1970-01-01T00:00:00Z'
+        digest=None
+        try:
+          _, h, mb=req(f'{REG}/v2/{repo}/manifests/{t}', headers={'Accept':'application/vnd.docker.distribution.manifest.v2+json'})
+          digest=h.get('Docker-Content-Digest')
+          m=json.loads(mb.decode())
+          cfg=(m.get('config') or {}).get('digest')
+          if cfg:
+            _, _, cb=req(f'{REG}/v2/{repo}/blobs/{cfg}')
+            created=json.loads(cb.decode()).get('created', created)
+        except Exception:
+          created='9999-12-31T23:59:59Z'
+        rows.append((created, t, digest))
+
+      rows.sort(key=lambda x: x[0], reverse=True)
+      KEEP_N=2   # keep last 2 SHA builds (current + 1 previous)
+      keep_set=set(t for _, t, _ in rows[:KEEP_N])
+      # preserve buildcache for performance
+      keep_set.update(t for t in tags if t == 'buildcache')
+      keep_list=sorted(keep_set)
+      print(f'[{repo}] sha_total={len(rows)} keep={keep_list} remove={max(0, len(rows)-len(keep_set))}')
+      for _, t, d in rows:
+        if t in keep_set or not d:
+          continue
+        try:
+          req(f'{REG}/v2/{repo}/manifests/{d}', method='DELETE')
+          deleted+=1
+          print(f'  deleted {repo}:{t}')
+        except urllib.error.HTTPError as e:
+          print(f'  delete failed {repo}:{t} code={e.code}')
+        except Exception as e:
+          print(f'  delete failed {repo}:{t} err={e}')
+
+    print(f'deleted_manifests={deleted}')
+
+    # Trigger garbage collection to delete unreferenced blob layers
+    if deleted > 0:
+      print('\n=== Triggering Garbage Collection ===')
+      try:
+        # Scale down registry to run GC
+        import subprocess
+        subprocess.run(['kubectl', 'scale', 'deployment', 'docker-registry', '--replicas=0', '-n', 'registry'], check=True)
+        print('Scaled down docker-registry deployment')
+        
+        # Wait for deployment to be fully down
+        import time
+        time.sleep(5)
+        
+        # Run GC job
+        gc_job = {
+          'apiVersion': 'batch/v1',
+          'kind': 'Job',
+          'metadata': {'name': 'registry-gc-once', 'namespace': 'registry'},
+          'spec': {
+            'backoffLimit': 0,
+            'template': {
+              'spec': {
+                'restartPolicy': 'Never',
+                'containers': [{
+                  'name': 'gc',
+                  'image': 'registry:3',
+                  'command': ['registry', 'garbage-collect', '--delete-untagged', '/etc/distribution/config.yml'],
+                  'volumeMounts': [
+                    {'name': 'storage', 'mountPath': '/var/lib/registry'},
+                    {'name': 'config', 'mountPath': '/etc/distribution'}
+                  ]
+                }],
+                'volumes': [
+                  {'name': 'storage', 'persistentVolumeClaim': {'claimName': 'registry-pvc'}},
+                  {'name': 'config', 'configMap': {'name': 'registry-config'}}
+                ]
+              }
+            }
+          }
+        }
+        
+        # Delete old GC job if exists
+        subprocess.run(['kubectl', 'delete', 'job', 'registry-gc-once', '-n', 'registry', '--ignore-not-found=true'], check=False)
+        time.sleep(2)
+        
+        # Create and wait for GC job
+        import tempfile
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
+          json.dump(gc_job, f)
+          f.flush()
+          subprocess.run(['kubectl', 'apply', '-f', f.name], check=True)
+        
+        print('GC job created, waiting for completion...')
+        
+        # Wait up to 10 minutes for GC to complete
+        for i in range(120):
+          result = subprocess.run(['kubectl', 'get', 'job', 'registry-gc-once', '-n', 'registry', '-o', 'jsonpath={.status.succeeded}'], capture_output=True, text=True)
+          if result.stdout.strip() == '1':
+            print('Garbage collection completed successfully')
+            break
+          result = subprocess.run(['kubectl', 'get', 'job', 'registry-gc-once', '-n', 'registry', '-o', 'jsonpath={.status.failed}'], capture_output=True, text=True)
+          if result.stdout.strip() == '1':
+            print('GC job failed')
+            break
+          time.sleep(5)
+        
+        # Scale back up
+        subprocess.run(['kubectl', 'scale', 'deployment', 'docker-registry', '--replicas=1', '-n', 'registry'], check=True)
+        print('Scaled up docker-registry deployment')
+        
+      except Exception as e:
+        print(f'GC trigger failed: {e}')
+        # Ensure registry is scaled back up even if GC failed
+        try:
+          subprocess.run(['kubectl', 'scale', 'deployment', 'docker-registry', '--replicas=1', '-n', 'registry'], check=False)
+        except:
+          pass

argocd/coredns-nodehosts-application.yaml

@@ -1,21 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: coredns-nodehosts
-  namespace: argocd
-spec:
-  destination:
-    namespace: kube-system
-    server: https://kubernetes.default.svc
-  project: default
-  source:
-    path: ops/coredns-nodehosts
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-

argocd/nxtgauge-admin-solid-application.yaml

@@ -1,25 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: nxtgauge-admin-solid
-  namespace: argocd
-  annotations:
-    argocd-image-updater.argoproj.io/image-list: admin=registry.nxtgauge.com/nxtgauge-admin-solid:high-performance-latest
-    argocd-image-updater.argoproj.io/admin.update-strategy: digest
-    argocd-image-updater.argoproj.io/admin.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
-spec:
-  project: default
-  source:
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-    path: apps/nxtgauge-admin-solid/overlays/prod
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: nxtgauge
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true

argocd/nxtgauge-ai-assistant-application.yaml

@@ -1,25 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: nxtgauge-ai-assistant
-  namespace: argocd
-  annotations:
-    argocd-image-updater.argoproj.io/image-list: ai=registry.nxtgauge.com/nxtgauge-ai-assistant:high-performance-latest
-    argocd-image-updater.argoproj.io/ai.update-strategy: digest
-    argocd-image-updater.argoproj.io/ai.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
-spec:
-  project: default
-  source:
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-    path: apps/nxtgauge-ai-assistant/overlays/prod
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: nxtgauge
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true

argocd/nxtgauge-backend-rust-application.yaml

@@ -1,81 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: nxtgauge-backend-rust
-  namespace: argocd
-  annotations:
-    argocd-image-updater.argoproj.io/image-list: >-
-      gateway=registry.nxtgauge.com/nxtgauge-rust-gateway:d084491,
-      users=registry.nxtgauge.com/nxtgauge-rust-users:d084491,
-      companies=registry.nxtgauge.com/nxtgauge-rust-companies:high-performance-latest,
-      job-seekers=registry.nxtgauge.com/nxtgauge-rust-job-seekers:high-performance-latest,
-      customers=registry.nxtgauge.com/nxtgauge-rust-customers:high-performance-latest,
-      payments=registry.nxtgauge.com/nxtgauge-rust-payments:high-performance-latest,
-      employees=registry.nxtgauge.com/nxtgauge-rust-employees:high-performance-latest,
-      photographers=registry.nxtgauge.com/nxtgauge-rust-photographers:high-performance-latest,
-      makeup-artists=registry.nxtgauge.com/nxtgauge-rust-makeup-artists:high-performance-latest,
-      tutors=registry.nxtgauge.com/nxtgauge-rust-tutors:high-performance-latest,
-      developers=registry.nxtgauge.com/nxtgauge-rust-developers:high-performance-latest,
-      video-editors=registry.nxtgauge.com/nxtgauge-rust-video-editors:high-performance-latest,
-      graphic-designers=registry.nxtgauge.com/nxtgauge-rust-graphic-designers:high-performance-latest,
-      social-media-managers=registry.nxtgauge.com/nxtgauge-rust-social-media-managers:high-performance-latest,
-      fitness-trainers=registry.nxtgauge.com/nxtgauge-rust-fitness-trainers:high-performance-latest,
-      catering-services=registry.nxtgauge.com/nxtgauge-rust-catering-services:high-performance-latest,
-      ugc-content-creators=registry.nxtgauge.com/nxtgauge-rust-ugc-content-creators:high-performance-latest,
-      cron=registry.nxtgauge.com/nxtgauge-rust-cron:high-performance-latest
-    argocd-image-updater.argoproj.io/gateway.update-strategy: digest
-    argocd-image-updater.argoproj.io/gateway.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/users.update-strategy: digest
-    argocd-image-updater.argoproj.io/users.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/companies.update-strategy: digest
-    argocd-image-updater.argoproj.io/companies.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/jobs.update-strategy: digest
-    argocd-image-updater.argoproj.io/jobs.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/leads.update-strategy: digest
-    argocd-image-updater.argoproj.io/leads.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/job-seekers.update-strategy: digest
-    argocd-image-updater.argoproj.io/job-seekers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/customers.update-strategy: digest
-    argocd-image-updater.argoproj.io/customers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/payments.update-strategy: digest
-    argocd-image-updater.argoproj.io/payments.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/employees.update-strategy: digest
-    argocd-image-updater.argoproj.io/employees.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/photographers.update-strategy: digest
-    argocd-image-updater.argoproj.io/photographers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/makeup-artists.update-strategy: digest
-    argocd-image-updater.argoproj.io/makeup-artists.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/tutors.update-strategy: digest
-    argocd-image-updater.argoproj.io/tutors.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/developers.update-strategy: digest
-    argocd-image-updater.argoproj.io/developers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/video-editors.update-strategy: digest
-    argocd-image-updater.argoproj.io/video-editors.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/graphic-designers.update-strategy: digest
-    argocd-image-updater.argoproj.io/graphic-designers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/social-media-managers.update-strategy: digest
-    argocd-image-updater.argoproj.io/social-media-managers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/fitness-trainers.update-strategy: digest
-    argocd-image-updater.argoproj.io/fitness-trainers.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/catering-services.update-strategy: digest
-    argocd-image-updater.argoproj.io/catering-services.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/ugc-content-creators.update-strategy: digest
-    argocd-image-updater.argoproj.io/ugc-content-creators.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/cron.update-strategy: digest
-    argocd-image-updater.argoproj.io/cron.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
-spec:
-  destination:
-    namespace: nxtgauge
-    server: https://kubernetes.default.svc
-  project: default
-  source:
-    path: apps/nxtgauge-backend-rust/overlays/prod
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true

argocd/nxtgauge-frontend-solid-application.yaml

@@ -1,25 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: nxtgauge-frontend-solid
-  namespace: argocd
-  annotations:
-    argocd-image-updater.argoproj.io/image-list: frontend=registry.nxtgauge.com/nxtgauge-frontend-solid:bb6db6c
-    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
-    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^high-performance-latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
-spec:
-  project: default
-  source:
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-    path: apps/nxtgauge-frontend-solid/overlays/prod
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: nxtgauge
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true

argocd/ollama-application.yaml

@@ -1,20 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: ollama
-  namespace: argocd
-spec:
-  destination:
-    namespace: nxtgauge-ai
-    server: https://kubernetes.default.svc
-  project: default
-  source:
-    path: apps/ollama/base
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true

argocd/openobserve-alerts-application.yaml

@@ -1,21 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: openobserve-alerts
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-    path: ops/openobserve-alerts
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: openobserve
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-

argocd/openobserve-otelcol-application.yaml

@@ -1,21 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: openobserve-otelcol
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-    path: ops/openobserve-otelcol
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: openobserve
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-

argocd/registry-ingress-application.yaml

@@ -1,21 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: registry-ingress
-  namespace: argocd
-spec:
-  destination:
-    namespace: registry
-    server: https://kubernetes.default.svc
-  project: default
-  source:
-    path: ops/registry-ingress
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-

argocd/woodpecker-registry-pull-application.yaml

@@ -1,21 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: woodpecker-registry-pull
-  namespace: argocd
-spec:
-  destination:
-    namespace: woodpecker
-    server: https://kubernetes.default.svc
-  project: default
-  source:
-    path: ops/woodpecker-registry-pull
-    repoURL: https://github.com/Traceworks2023/nxtgauge-gitops.git
-    targetRevision: main
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-

clusters/production/kustomization.yaml

@@ -1,4 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - ../../apps
+  - ../../apps/nxtgauge-backend-rust/overlays/prod
+  - ../../apps/nxtgauge-frontend-solid/overlays/prod
+  - ../../apps/nxtgauge-admin-solid/overlays/prod
+  - ../../apps/nxtgauge-ai-assistant/overlays/prod
+  - ../../apps/ollama/base
+  - ../../apps/registry
+  - ../../ops/openobserve-alerts

ops/openobserve-alerts/configmap-additional.yaml

@@ -0,0 +1,143 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: openobserve-alerts-additional
+  namespace: openobserve
+data:
+  additional-alerts.sh: |
+    #!/usr/bin/env sh
+    set -eu
+
+    ORG_ID="${ORG_ID:-default}"
+    BASE_URL="${BASE_URL:-http://o2-openobserve-standalone.openobserve.svc.cluster.local:5080}"
+    STREAM_NAME="${STREAM_NAME:-default}"
+    TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
+
+    AUTH="$(printf '%s:%s' "$ZO_ROOT_USER_EMAIL" "$ZO_ROOT_USER_PASSWORD" | base64 | tr -d '\n')"
+    auth_hdr="Authorization: Basic $AUTH"
+
+    api() {
+      curl -sfS -H "$auth_hdr" -H "Content-Type: application/json" "$@"
+    }
+
+    ensure_alert() {
+      alert_name="$1"
+      sql="$2"
+      period_minutes="$3"
+      frequency_minutes="$4"
+      silence_minutes="$5"
+      row_template="$6"
+
+      existing_id="$(
+        api "$BASE_URL/api/v2/$ORG_ID/alerts" \
+          | jq -r --arg n "$alert_name" '.list[] | select(.name == $n) | .alert_id' \
+          | head -n 1
+      )"
+
+      payload="$(jq -n \
+        --arg name "$alert_name" \
+        --arg stream "$STREAM_NAME" \
+        --arg sql "$sql" \
+        --argjson period "$period_minutes" \
+        --argjson frequency "$frequency_minutes" \
+        --argjson silence "$silence_minutes" \
+        --arg row_template "$row_template" \
+        '{
+          name: $name,
+          stream_type: "logs",
+          stream_name: $stream,
+          is_real_time: false,
+          enabled: true,
+          tz_offset: 330,
+          destinations: ["nxtgauge_telegram"],
+          row_template: $row_template,
+          row_template_type: "String",
+          query_condition: { type: "sql", sql: $sql },
+          trigger_condition: {
+            period: $period,
+            operator: ">=",
+            threshold: 1,
+            frequency: $frequency,
+            frequency_type: "minutes",
+            silence: $silence
+          }
+        }')"
+
+      if [ -n "$existing_id" ] && [ "$existing_id" != "null" ]; then
+        api -X PUT "$BASE_URL/api/v2/$ORG_ID/alerts/$existing_id" -d "$payload" >/dev/null
+        echo "updated alert=$alert_name"
+      else
+        api -X POST "$BASE_URL/api/v2/$ORG_ID/alerts" -d "$payload" >/dev/null
+        echo "created alert=$alert_name"
+      fi
+    }
+
+    # API Health
+    ensure_alert \
+      "api-health-failures" \
+      "SELECT service, endpoint, status_code, COUNT(*) as count FROM \"default\" WHERE service ILIKE '%api%' AND (status_code >= 500 OR status_code = 0) GROUP BY service, endpoint, status_code ORDER BY count DESC LIMIT 50" \
+      5 1 15 \
+      "{service}/{endpoint} status={status_code} count={count}"
+
+    # Database Health
+    ensure_alert \
+      "database-connection-failures" \
+      "SELECT k8s_namespace_name, k8s_pod_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE body ILIKE '%connection refused%' OR body ILIKE '%database%' OR body ILIKE '%postgres%' OR body ILIKE '%sqlx%' ORDER BY _timestamp DESC LIMIT 50" \
+      5 1 15 \
+      "{k8s_namespace_name}/{k8s_pod_name}: {msg}"
+
+    # Redis Health
+    ensure_alert \
+      "redis-connection-failures" \
+      "SELECT k8s_namespace_name, k8s_pod_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE body ILIKE '%redis%' OR body ILIKE '%cache%' OR body ILIKE '%connection timeout%' ORDER BY _timestamp DESC LIMIT 50" \
+      5 1 15 \
+      "{k8s_namespace_name}/{k8s_pod_name}: {msg}"
+
+    # Pod Failures
+    ensure_alert \
+      "pod-failures" \
+      "SELECT k8s_namespace_name, k8s_pod_name, body_object_reason, body_object_message FROM \"default\" WHERE body_object_reason IN ('Failed', 'Evicted', 'NodeAffinity', 'UnexpectedAdmissionError') ORDER BY _timestamp DESC LIMIT 50" \
+      5 1 15 \
+      "{k8s_namespace_name}/{k8s_pod_name} {body_object_reason}: {body_object_message}"
+
+    # CPU High
+    ensure_alert \
+      "cpu-high-usage" \
+      "SELECT k8s_namespace_name, k8s_pod_name, k8s_container_name, AVG(cpu_usage_cores) as avg_cpu FROM \"default\" WHERE cpu_usage_cores > 0.8 GROUP BY k8s_namespace_name, k8s_pod_name, k8s_container_name ORDER BY avg_cpu DESC LIMIT 50" \
+      10 2 30 \
+      "{k8s_namespace_name}/{k8s_pod_name}/{k8s_container_name} CPU={avg_cpu}"
+
+    # Memory High
+    ensure_alert \
+      "memory-high-usage" \
+      "SELECT k8s_namespace_name, k8s_pod_name, k8s_container_name, AVG(memory_usage_bytes) as avg_mem FROM \"default\" WHERE memory_usage_bytes > 1073741824 GROUP BY k8s_namespace_name, k8s_pod_name, k8s_container_name ORDER BY avg_mem DESC LIMIT 50" \
+      10 2 30 \
+      "{k8s_namespace_name}/{k8s_pod_name}/{k8s_container_name} MEM={avg_mem}"
+
+    # Disk Full
+    ensure_alert \
+      "disk-full-warning" \
+      "SELECT k8s_node_name, k8s_namespace_name, k8s_pod_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE body ILIKE '%disk full%' OR body ILIKE '%no space left%' OR body ILIKE '%DiskPressure%' ORDER BY _timestamp DESC LIMIT 50" \
+      10 2 60 \
+      "{k8s_node_name}/{k8s_namespace_name}/{k8s_pod_name}: {msg}"
+
+    # Longhorn Health
+    ensure_alert \
+      "longhorn-volume-errors" \
+      "SELECT k8s_namespace_name, k8s_pod_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'longhorn-system' AND (body ILIKE '%error%' OR body ILIKE '%degraded%' OR body ILIKE '%faulted%') ORDER BY _timestamp DESC LIMIT 50" \
+      10 2 30 \
+      "longhorn/{k8s_pod_name}: {msg}"
+
+    # Flux Health
+    ensure_alert \
+      "flux-reconcile-failures" \
+      "SELECT k8s_pod_name, k8s_container_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'flux-system' AND (body ILIKE '%SyncFailed%' OR body ILIKE '%ComparisonError%' OR body ILIKE '%ResourceQuota%') ORDER BY _timestamp DESC LIMIT 50" \
+      10 2 30 \
+      "flux/{k8s_pod_name}: {msg}"
+
+    # Registry Health
+    ensure_alert \
+      "registry-push-failures" \
+      "SELECT k8s_pod_name, k8s_container_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'registry' AND (body ILIKE '%413%' OR body ILIKE '%payload too large%' OR body ILIKE '%unauthorized%') ORDER BY _timestamp DESC LIMIT 50" \
+      10 2 60 \
+      "registry/{k8s_pod_name}: {msg}"

ops/openobserve-alerts/configmap.yaml

@@ -82,9 +82,9 @@ data:
           stream_type: "logs",
           stream_name: $stream,
           is_real_time: false,
-          enabled: true,
+          enabled: false,
           tz_offset: 330,
-          destinations: ["nxtgauge_telegram"],
+          destinations: [],
           row_template: $row_template,
           row_template_type: "String",
           query_condition: { type: "sql", sql: $sql },
@@ -148,16 +148,16 @@ data:
       "{k8s_namespace_name}/{k8s_pod_name} {body_object_reason}: {body_object_message}"
 
     ensure_alert \
-      "argocd-errors" \
+      "flux-errors" \
-      "SELECT k8s_pod_name, k8s_container_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'argocd' AND (body ILIKE '%level=error%' OR body ILIKE '%ERROR%' OR body ILIKE '%ComparisonError%' OR body ILIKE '%SyncFailed%') ORDER BY _timestamp DESC LIMIT 50" \
+      "SELECT k8s_pod_name, k8s_container_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'flux-system' AND (body ILIKE '%level=error%' OR body ILIKE '%ERROR%' OR body ILIKE '%ComparisonError%' OR body ILIKE '%SyncFailed%') ORDER BY _timestamp DESC LIMIT 50" \
       10 2 30 \
-      "argocd/{k8s_pod_name} {k8s_container_name}: {msg}"
+      "flux/{k8s_pod_name} {k8s_container_name}: {msg}"
 
     ensure_alert \
-      "woodpecker-errors" \
+      "forgejo-runner-errors" \
-      "SELECT k8s_pod_name, k8s_container_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'woodpecker' AND (body ILIKE '%error%' OR body ILIKE '%ERROR%' OR body ILIKE '%failed%') ORDER BY _timestamp DESC LIMIT 50" \
+      "SELECT k8s_pod_name, k8s_container_name, substring(body, 1, 220) AS msg FROM \"default\" WHERE k8s_namespace_name = 'forgejo' AND (body ILIKE '%error%' OR body ILIKE '%ERROR%' OR body ILIKE '%failed%' OR body ILIKE '%job failed%') ORDER BY _timestamp DESC LIMIT 50" \
       10 2 30 \
-      "woodpecker/{k8s_pod_name} {k8s_container_name}: {msg}"
+      "forgejo/{k8s_pod_name} {k8s_container_name}: {msg}"
 
     ensure_alert \
       "registry-errors" \

ops/openobserve-alerts/cronjob.yaml

@@ -14,7 +14,7 @@ spec:
           restartPolicy: Never
           containers:
             - name: bootstrap
-              image: registry.nxtgauge.com/docker:28-cli
+              image: docker:28-cli
               command: ["sh", "-lc"]
               args:
                 - apk add --no-cache curl jq >/dev/null && /scripts/bootstrap.sh

ops/woodpecker-registry-pull/README.md

@@ -1,35 +0,0 @@
-# Woodpecker: allow pulling from private registry
-
-Woodpecker pipelines run as Kubernetes pods in the `woodpecker` namespace. If pipeline step images use `registry.nxtgauge.com/...` (private, Basic auth), kubelet needs an `imagePullSecret`.
-
-This is required for base images (example `registry.nxtgauge.com/rust:alpine`) and also for any mirrored plugin images (example `registry.nxtgauge.com/kaniko:2.1.1`).
-
-## Required secret
-
-Create this once:
-
-```bash
-kubectl -n woodpecker create secret docker-registry registry-nxtgauge-pull \
-  --docker-server=registry.nxtgauge.com \
-  --docker-username="<REGISTRY_USERNAME>" \
-  --docker-password="<REGISTRY_PASSWORD>" \
-  --docker-email="ci@nxtgauge.com"
-```
-
-## Mirroring common plugin images (optional)
-
-If your pipelines reference plugin images from the internal registry (example `registry.nxtgauge.com/kaniko:2.1.1`) make sure those images exist in the registry.
-
-Example mirror from Docker Hub to internal:
-
-```bash
-docker pull woodpeckerci/plugin-kaniko:2.1.1
-docker tag woodpeckerci/plugin-kaniko:2.1.1 registry.nxtgauge.com/kaniko:2.1.1
-docker push registry.nxtgauge.com/kaniko:2.1.1
-```
-
-## What this kustomize applies
-
-It patches/ensures the `default` ServiceAccount in `woodpecker` includes:
-
-- `imagePullSecrets: [registry-nxtgauge-pull]`

ops/woodpecker-registry-pull/serviceaccount-default.yaml

@@ -1,8 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: default
-  namespace: woodpecker
-imagePullSecrets:
-  - name: registry-nxtgauge-pull
-