Insurance Compliance Copilot

A self-built demo — not client work — built to show end-to-end AI product engineering: a citation-grounded regulatory-compliance copilot over public-domain US insurance law. Ask a claims-handling question, get an answer grounded in cited statute/regulation passages; for change-proposing actions, get a human-approved diff to a structured compliance artifact. It runs live on Cloudflare Workers.

Live: copilot.edshav.com · Code: github.com/edshav/insurance-compliance-copilot

The workflow

A compliance reviewer asks a question against the controlling regulations and gets a cited answer plus a reviewable, human-approved change:

1. Retrieve the controlling statute/regulation passages.
2. Answer, grounded in those passages with section-level citations — or refuse if retrieval doesn’t support an answer.
3. Locate the SOP/policy clause that conflicts.
4. Propose a minimal diff to bring that clause into conformance with the cited regulation.
5. Require human accept/reject before anything is written.
6. Record the decision and its citations as an append-only audit trail.

That loop — question → cited law → grounded answer → flagged clause → minimal diff → human approval → audit trail — is the centerpiece. Everything else exists to make it trustworthy.

Why the evals are the point

The credibility anchor is the eval harness, not the chat UI. A golden Q&A set is mined from California DOI market-conduct examination reports — regulator- authored ground truth that cites the exact statute sections an insurer allegedly violated. Each answer is scored for faithfulness and citation accuracy, with an adversarial no-answer set that scores the system on correctly refusing when no retrieved passage supports an answer.

It’s wired into CI via Promptfoo: a GitHub Action runs the golden set on every PR and blocks merges below a regression floor. The headline is a measured number on this corpus — the full pipeline lifts retrieval citation MRR +0.26 (0.57 → 0.83) over a true-naive baseline.

Architecture

Four layers on Cloudflare Workers:

• Retrieval — section-aware ingest + contextual chunking; hybrid retrieve (dense + sparse/BM25, RRF fusion, reranking) behind an agentic loop gated by query complexity; generation with section-level citations under a refuse-if-unsupported guard.
• Agent UI — generative-UI answer cards (cited passage + source links + an impact view), the HITL diff surface, and step-by-step SSE streaming of how each answer was assembled. The same read + propose capabilities are exposed as MCP tools — with no write tool, so the protocol can read and propose but the accept/reject stays a human action.
• Evals — the golden + adversarial no-answer sets and the Promptfoo CI gate above.
• Observability — hand-instrumented OpenTelemetry GenAI spans → Langfuse.

Stack: TanStack Start (React) + TypeScript, Workers AI (bge-m3 embed, bge-reranker-base rerank, llama-3.1-8b-instruct-fast generate), and Postgres + pgvector reached from the Worker via Hyperdrive.

Scope & honest caveats

A reference-grounding prototype, not filing-ready legal determinations. It surfaces and cites controlling regulatory text and proposes human-reviewed changes; it does not render authoritative compliance rulings. The MVP scope is deliberately narrow — one state’s claims-handling regulations, one structured artifact type, and the single review loop above — a single tool-calling agent, not a multi-agent system.