I verify AI-built software before it breaks in front of paying users. AI Assurance Engineer. M.S. Cybersecurity. I write about testing AI systems — and run fixed-fee verification audits for founders shipping with AI. Work with me →
Four Surfaces, One Agent: What the Testing Framework Missed Until I Tested the UI
Background The risk-based framework scores and gates test suites. The AI extension does the same for agents and LLM features: prompt injection, disclosure, least privilege, the whole adversarial list. This week I ran both against a real one: a client-facing AI agent sitting on top of a read-only workforce-data API, tested against a plan a colleague had already written. Ten endpoints, one chat window, several thousand rows of other people’s payroll data behind each API key. ...
Shift Left, For Real: Testing and Security Belong in the Spec
Background The risk-based framework I built scores and gates test suites. The AI extension does the same for agents and LLM features. Both answer how much testing, and where it blocks a merge. Neither answers when. The Problem Lately, when has been the problem. A feature ships, the PR lands, and that’s the first time QA sees it. Testing becomes archaeology: reconstruct what the feature was supposed to do from the diff, then work out what wasn’t tested. ...
Testing What You Can't Enumerate: A QA Framework for AI Agents and LLMs
Background Last month I wrote about the risk-based testing framework I built to keep QA on pace with agentic coding: twenty test categories five phases a scoring rubric that decides where the effort goes. Then the products started shipping AI features. They were the same products as before, handling PII and other regulated, person-level business data. The Problem The original framework breaks in four places. You can’t enumerate the input space. 80% line coverage means something. “80% of prompts” means nothing. There is no denominator. ...
Lean Software Quality Assurance in a Regulated Industry
Background Over the last week, I’ve been building a risk-based software testing and quality assurance framework for regulated products. The products I work on handle PII, worker and employer data, payrolling, staffing and placement, benefits, prescreening, compliance, Vendor Management Systems (VMSs), Managed Service Providers (MSPs), etc. The products also serve clients and workers in the US and in jurisdictions across the EU, the UK, and Canada. The Problem As agentic coding tools increasingly got better and faster at writing code, the developers on our team started writing code quicker than the QA team could keep up with. ...