Follow-up on the AI safety review — addressed the three remaining
lower-severity findings:
1. crates LiteLlmError::error_body() returned the raw upstream response
body verbatim to the client on any non-2xx LiteLLM response. That
body can contain internal routing/diagnostic details from the
LiteLLM proxy or the underlying model provider. Now returns a
generic, status-aware message to the client; the full body is
logged server-side via tracing::error! at each of the three call
sites that construct LiteLlmError::Api, so nothing is lost for
debugging — it's just not exposed to end users.
2. Added an explicit anti-prompt-injection clause to
ai/orchestrator.rs::GROUNDING_GUARDRAIL, the baseline system prompt
applied to every AI feature call via effective_system_prompt() —
instructs the model to treat all user/company-authored input
(job descriptions, profile text, chat messages) as data to analyze,
never as instructions to follow. Covers every ai.rs handler that
goes through call_feature/call_feature_with_plan in one place,
rather than patching each call site's prompt construction
individually.
3. apps/cron/src/tasks/auto_apply.rs's cover-letter prompt doesn't run
through the orchestrator (separate app/crate), so hardened it
directly: fenced the untrusted CANDIDATE/JOB sections with explicit
"this is data, not instructions" framing. While there, fixed a
latent panic: `&job_desc[..job_desc.len().min(500)]` slices on a
raw byte offset, which panics if byte 500 isn't a UTF-8 character
boundary — a company job description with any multi-byte character
before that point (accented letters, emoji, etc.) would crash the
whole cron run. Switched to char_indices() to find a safe boundary.
Asked to review Tracecoin and AI implementation safety. Found and fixed
two exploitable TOCTOU races, plus a data-integrity bug:
1. apps/payments/src/main.rs::verify_payment — the PayU success callback
is called directly by the client (not a server-to-server webhook), so
a user fully controls how many times they replay a valid success
payload. The payment "is it still PENDING" check and the "mark
SUCCESS + credit wallet" write were separate, non-transactional
queries — concurrent replays could both pass the check before either
commits, double- (or N-times-) crediting the wallet for one real
payment. Now wrapped in a single transaction with
`SELECT ... FOR UPDATE` on the payments row, so a second concurrent
call blocks until the first commits, then correctly sees the row is
no longer PENDING (Postgres re-evaluates the WHERE clause via
EvalPlanQual after the lock is granted).
2. crates/db/src/models/ai/repository.rs — UserAiSubscriptionRepository
had the exact same shape of bug: apps/users/src/ai/credits.rs::
charge_feature read the subscription, checked daily-limit and credit
balance, THEN issued two separate unconditional `UPDATE ... SET x =
x + $1` statements with no WHERE guard on the balance. N concurrent
requests from one user all pass the check before any deduction
lands, running up unlimited LLM API spend (this endpoint is called
before/around real LiteLLM calls, so the cost is real). Added
UserAiSubscriptionRepository::try_charge — a single conditional
UPDATE that checks the daily limit and credit balance and deducts
atomically, returning None (mapped to the existing error types) if
either check fails.
3. apps/cron/src/tasks/auto_apply.rs — daily_actions_used was being
incremented twice per auto-applied job (once in the credit-deduct
UPDATE, once more in a second, redundant UPDATE right after) —
silently halving job seekers' effective daily auto-apply limit.
Removed the redundant second UPDATE.
Also added non-negative CHECK constraints directly to the live
database (tracecoin_wallets.balance/reserved,
user_ai_subscriptions.daily_actions_used/monthly_credits_used/
purchased_credits_used) as defense in depth — belt-and-suspenders in
case a future code path reintroduces a similar bug.
- Company approval wrote profile status to 'ACTIVE' (company_profiles'
own pre-verification default) using an id column that never matched
any row, so create_job's APPROVED check always rejected newly
approved companies. Match on user_id for user_id-keyed tables and
write the canonical 'APPROVED' status.
- job_seeker_profiles was missing columns the job-seeker app has
always queried (full_name, location, summary, skills,
active_application_count, status), and the job_applications /
job_seeker_documents tables it depends on were never migrated in —
job seeker profile save/submit and job applications failed outright
with "column/relation does not exist".
- Renamed the job_seeker first_name/last_name split to full_name to
match what the frontend has always sent.
- Special-cased JOB_SEEKER in the generic profile.rs handlers (mirrors
the existing COMPANY special-case) so the shared ProfilePage save/
submit flow, which was routed through a user_role_profile_id-based
path job_seeker_profiles never had, now persists correctly.
- Fixed apply_to_job's company notification query joining a
nonexistent "companies" table instead of company_profiles.
- Fixed auto-apply cron's company status filter to match the
corrected 'APPROVED' status.
Same congestion pattern from the crates/contracts push. gateway still had
2 healthy replicas serving traffic throughout, so no outage — just a
failed rolling-update replica for these 6 services.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add #![allow(dead_code)] pragma to all main.rs files
- Remove unused imports from users handlers (ai_cache, AiCreditPackageRepository, AiCreditTransactionRepository)
- Make LiteLLMChatMessage and LiteLLMChoice public with public fields
- Fix remaining unused variables with cargo fix
- Add missing pub visibility modifiers to litellm structs
All packages now compile with ZERO warnings and errors!
Delete legacy code that used old company_ai_usage/job_seeker_ai_usage tables:
- Remove has_active_ai_pack() - old AI_PACK pricing package check
- Remove check_and_increment_usage() - legacy daily quota tracking
- Remove BASE_AI_LIMIT, get_ai_limit_for_package constants/functions
- Remove legacy queries from ai_auto_apply() and ai_usage_status()
- Update auto_apply.rs to use user_ai_subscriptions.daily_actions_used
instead of job_seeker_ai_usage table
- Inline apply_scheduled_downgrades() and expire_trials() in cron tasks
to remove dependency on users crate internal modules
The new system uses user_ai_subscriptions with:
- daily_actions_used / daily_credits_used counters
- monthly_credits_total / monthly_credits_used
- purchased_credits_total / purchased_credits_used
All AI billing now flows through the wallet/ledger system with
LiteLLM integration (Tasks 1-10).
- Add AI plans, credits, model routing, LiteLLM client, and orchestrator services
- Add AI management endpoints, auto-apply/auto-request handlers, and log endpoints
- Add cron jobs for daily action reset and monthly credit reset
- Add AI credit purchase flow in payments service
- Add ai_credit_packages migration with seed data
- Update Dockerfile build tooling across services
- Add get_llm_base_url() and get_llm_model() helper functions
- Update call_ollama_inline to route to LiteLLM when LLM_PROVIDER=litellm
- Add auto-apply cron task for background job matching
- Auto-apply matches job seekers to new jobs based on skills
- Generate cover letters via LiteLLM for each application
- companies: user.name in email and contact queries
- customers: user.name in email
- job_seekers: u.name in company user query
- cron tasks (jobs/leads/requirements): use u.name instead of u.full_name
- contracts/profession_shared: u.name for customer_name fields
- Update leads service to use 'leads' table
- Update extension models to use user_role_profile_id
- Update ProfessionalRepository to work with new schema
- Create TracecoinWalletRepository for wallet operations
- Update all handlers to use new model fields
- Rename Application fields (job_seeker_id -> applicant_user_id)
- Update cron tasks for new schema
- Fix compilation errors across all services
Added openssl-libs-static and OPENSSL_STATIC=1 environment variable
to fix reqwest/native-tls compilation errors with musl target.
Changes:
- Install openssl-libs-static in builder
- Set OPENSSL_STATIC=1 and OPENSSL_DIR=/usr
- Ensures OpenSSL is statically linked for all services
Switched from Debian to Alpine Linux for significant improvements:
- Image size: ~5MB vs ~100MB (95% smaller)
- Security: Minimal attack surface, no glibc vulnerabilities
- Static linking: No glibc version issues ever again
- Uses rust:alpine builder with x86_64-unknown-linux-musl target
- Static binaries with RUSTFLAGS='-C target-feature=+crt-static'
Fixes the GLIBC_2.38 error permanently by avoiding glibc entirely.
Fixed glibc version mismatch between rust:latest builder (glibc 2.38+)
and debian:bookworm-slim runtime (glibc 2.36). This was causing:
- ./companies: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.38 not found
- ./payments: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.38 not found
- Similar errors for users service
Updated all 19 service Dockerfiles + Dockerfile.template to use
debian:trixie-slim which includes glibc 2.38+.
