All checks were successful
build-and-release / build (tutors) (push) Successful in 2m36s
build-and-release / build (users) (push) Successful in 4m52s
build-and-release / build (jobs) (push) Successful in 41s
backend-integration-tests / ai-credits (push) Successful in 45s
build-and-release / build (fitness-trainers) (push) Successful in 1m52s
build-and-release / build (catering-services) (push) Successful in 1m59s
build-and-release / build (social-media-managers) (push) Successful in 3m3s
build-and-release / build (employees) (push) Successful in 1m50s
build-and-release / build (graphic-designers) (push) Successful in 1m54s
build-and-release / build (ugc-content-creators) (push) Successful in 2m46s
build-and-release / build (companies) (push) Successful in 2m6s
build-and-release / build (payments) (push) Successful in 1m55s
build-and-release / build (makeup-artists) (push) Successful in 2m38s
build-and-release / build (cron) (push) Successful in 2m22s
build-and-release / build (job-seekers) (push) Successful in 3m9s
build-and-release / build (customers) (push) Successful in 2m38s
build-and-release / build (developers) (push) Successful in 2m23s
build-and-release / build (photographers) (push) Successful in 2m51s
build-and-release / build (video-editors) (push) Successful in 2m45s
build-and-release / build (gateway) (push) Successful in 44s
- Add llm_moderation_check() — async call to LiteLLM /moderations endpoint (routes to OpenAI Moderation API, free) for hate/sexual/violence detection. Fails open: if LiteLLM is unreachable, message is allowed through with a warning log so core chat is never broken by a network hiccup. - Add spawn_guard_log() — fire-and-forget tokio task that persists every guard rejection (keyword AND moderation) to ai_guard_violations table. - Wire both guards into ai_chat_message, ai_chat_ask, and ai_chat_stream. - Add migration: ai_guard_violations(id, user_id, message_excerpt, guard_type, reason, categories JSONB, created_at). - Add GET /api/admin/ai/guard-events with filter by guard_type, pagination. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
16 lines
992 B
SQL
16 lines
992 B
SQL
-- AI Guard Violations — records every message blocked by the content guard.
|
|
-- Migration: 20260815000001
|
|
|
|
CREATE TABLE IF NOT EXISTS ai_guard_violations (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
user_id UUID REFERENCES users(id) ON DELETE SET NULL,
|
|
message_excerpt TEXT NOT NULL, -- first 200 chars, never the full text
|
|
guard_type VARCHAR(40) NOT NULL, -- 'keyword' | 'length' | 'flood' | 'moderation_api'
|
|
reason TEXT NOT NULL, -- human-readable rejection reason
|
|
categories JSONB, -- OpenAI moderation categories (moderation_api only)
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_ai_guard_violations_user_id ON ai_guard_violations(user_id);
|
|
CREATE INDEX IF NOT EXISTS idx_ai_guard_violations_created_at ON ai_guard_violations(created_at DESC);
|
|
CREATE INDEX IF NOT EXISTS idx_ai_guard_violations_guard_type ON ai_guard_violations(guard_type);
|