AI regression testing

--- name: ai-regression-testing description: Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code. origin: ECC --- # AI Regression Testing Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it : creating systematic blind spots that only automated tests can catch. ## When to Activate - AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic - A bug was found and fixed : need to prevent re-introduction - Project has a sandbox/mock mode that can be leveraged for DB-free testing - Running /bug-check or similar review commands after code changes - Multiple code paths exist (sandbox vs production, feature flags, etc.) ## The Core Problem When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern: `` AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists ` **Real-world example** (observed in production): ` Fix 1: Added notification_settings to API response → Forgot to add it to the SELECT query → AI reviewed and missed it (same blind spot) Fix 2: Added it to SELECT query → TypeScript build error (column not in generated types) → AI reviewed Fix 1 but didn't catch the SELECT issue Fix 3: Changed to SELECT * → Fixed production path, forgot sandbox path → AI reviewed and missed it AGAIN (4th occurrence) Fix 4: Test caught it instantly on first run PASS: ` The pattern: **sandbox/production path inconsistency** is the #1 AI-introduced regression. ## Sandbox-Mode API Testing Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing. ### Setup (Vitest + Next.js App Router) `typescript // vitest.config.ts import { defineConfig } from "vitest/config"; import path from "path"; export default defineConfig({ test: { environment: "node", globals: true, include: ["__tests__/**/*.test.ts"], setupFiles: ["__tests__/setup.ts"], }, resolve: { alias: { "@": path.resolve(__dirname, "."), }, }, }); ` `typescript // __tests__/setup.ts // Force sandbox mode : no database needed process.env.SANDBOX_MODE = "true"; process.env.NEXT_PUBLIC_SUPABASE_URL = ""; process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY = ""; ` ### Test Helper for Next.js API Routes `typescript // __tests__/helpers.ts import { NextRequest } from "next/server"; export function createTestRequest( url: string, options?: { method?: string; body?: Record<string, unknown>; headers?: Record<string, string>; sandboxUserId?: string; }, ): NextRequest { const { method = "GET", body, headers = {}, sandboxUserId } = options || {}; const fullUrl = url.startsWith("http") ? url : http://localhost:3000${url}; const reqHeaders: Record<string, string> = { ...headers }; if (sandboxUserId) { reqHeaders["x-sandbox-user-id"] = sandboxUserId; } const init: { method: string; headers: Record<string, string>; body?: string } = { method, headers: reqHeaders, }; if (body) { init.body = JSON.stringify(body); reqHeaders["content-type"] = "application/json"; } return new NextRequest(fullUrl, init); } export async function parseResponse(response: Response) { const json = await response.json(); return { status: response.status, json }; } ` ### Writing Regression Tests The key principle: **write tests for bugs that were found, not for code that works**. ``typescript // tests/api/user/profile.test.ts import { describe, it, expect } from "vitest"; import { createTestRequest, parseResponse } from "../../helpers"; import { GET, PATCH } from "@/app/api/user/profile/route"; // Define the contract : what fields MUST be in the response const REQUIREDFIELDS = [ "id", "email", "fullname", "phone", "role", "createdat", "avatarurl", "notificationsettings", // ← Added after bug found it missing ]; describe("GET /api/user/profile", () => { it("returns all required fields", async () => { const req = createTestRequest("/api/user/profile"); const res = await GET(req); const { status, json } = await parseResponse(res); expect(status).toBe(200); for (const field of REQUIREDFIELDS) { expect(json.data).toHaveProperty(field); } }); // Regression test : this exact bug was introduced by AI 4 times it("notificationsettings is not undefined (BUG-R1 regression)", async () => { const req = createTestRequest("/api/user/profile"); const res = await GET(req); const { json } = await parseResponse(res); expect("notificationsettings" in json.data).toBe(true); const ns = json.data.notification_settings; expect(ns === null || typeof ns === "object").toBe(true); }); }