AI Verification Architect

Chicago, IL

About the Company

Our client is a well-established, private equity-backed leader in a specialized B2B services and software space, supporting a large global customer base through a portfolio of complementary technology and data platforms. The organization is investing heavily in an “AI First” engineering strategy and is building out a new function to ensure quality and trust are engineered into every AI-driven system from the ground up.

Position Summary

The AI Verification Architect is the quality authority for the company’s AI First engineering organization, owning a four-layer Verification Gate (automated testing, static analysis, security scanning, and architecture conformance) that every AI-generated software increment must pass before production.

This is the next evolution beyond the SDET role. Rather than automating regression for deterministic code, you’ll build verification systems for probabilistic, agentic outputs, judging whether an AI agent did the right thing, not just whether it ran without errors.

Reporting to the Director of Engineering and partnering daily with the AI Solution Architect and AI Engineers, you will build and govern an adversarial Review Agent, automate business-side UAT to run at deployment speed, and operate the production observability pipeline that catches escaped defects and model drift, closing the Fail Fast / Fix Fast loop that keeps AI delivery rapid and reliable.

This is a hands-on, high-ownership role, measured by whether what ships delivers its intended outcomes, how quickly defects are caught and resolved, and the trust the business places in the company’s AI systems. Strong communication matters as much as technical depth.

Responsibilities

AI First Verification Framework & Delivery

• Own, govern, and continuously refine a four-layer Verification Gate as the organizational standard across every AI-generated software increment
• Build, govern, and orchestrate verification agents across the pipeline: an adversarial Review Agent that challenges every generated artifact for architecture, security, standards, and logic errors, and Testing Agents covering unit, integration, acceptance, and end-to-end validation
• Design and automate business-side UAT cycles so acceptance testing runs at pipeline speed rather than human review speed, while maintaining human-on-the-loop oversight for business intent validation and final production acceptance decisions
• Translate specs and acceptance criteria into executable verification contracts: automated test suites, evaluation scorers, and behavioral assertions
• Partner with AI Engineers throughout development to ensure features are designed for testability

Production Observability & Escaped Defect Detection

• Deploy and operate production observability agents that surface regressions, behavioral drift, and quality degradation before they compound
• Operate AI-powered incident triage agents that trace production failures to their source spec, commit, and producing engineer, and surface defect escape rate metrics for engineering leadership
• Champion the Fail Fast / Fix Fast feedback loop: triage production quality signals, route defects with full trace context and reproduction steps, and track resolution velocity as a shared engineering health metric
• Establish and monitor quality SLAs for AI outputs in production, covering accuracy, faithfulness, latency, security posture, and defect escape rate

Engineering Leadership & Standards

• Define and enforce quality standards, verification criteria, and release gates across the AI First engineering organization
• Partner with the AI Solution Architect to shape the verification layer of the company’s AI First architecture
• Champion shift-left verification practices, bringing quality constraints into spec authoring and acceptance criteria definition
• Capture patterns from test failures, evaluation anomalies, and quality signals, and feed them back into a shared context store to continuously raise the quality ceiling

Required Qualifications

Core Experience

• 3–6 years of software engineering experience with a strong emphasis on test engineering, quality automation, or SDET work, including building and deploying test frameworks for SaaS platforms
• Hands-on experience building automated test frameworks, quality gates, and CI/CD-integrated verification pipelines
• Hands-on proficiency across the test tooling spectrum, including automation frameworks such as Playwright or Selenium, xUnit or nUnit, and pytest, plus AI evaluation and observability tooling, backed by strong fundamentals in the Microsoft stack (C#/.NET, React/TypeScript, RESTful Web APIs, SQL Server / Azure SQL)
• Experience integrating multi-layer quality gates into CI/CD pipelines, including automated release blocking, static analysis, and security scanning (SAST/dependency audit)
• Exposure to applying AI and agentic tools to testing and quality work, or a clear aptitude and motivation to learn. Hands-on experience verifying AI-generated outputs is a plus, not a prerequisite
• Experience with, or strong interest in building, automated acceptance and UAT frameworks
• Excellent written and verbal communication skills
• A demonstrated capacity to learn quickly and a growth mindset

AI & LLM Ecosystem Proficiency (Preferred)

• Exposure to building or working with AI agents (adversarial review, testing, or anomaly-detection agents) using Claude Code or equivalent frameworks
• Experience evaluating LLM outputs and AI agent behavior, including evaluation frameworks and scoring methodologies
• Familiarity with observability for AI systems: tracing agent sessions, monitoring output quality, detecting hallucinations or behavioral drift
• Exposure to leading LLM APIs, including tool use, structured outputs, and streaming
• Applied software engineering discipline in AI systems: version control and CI/CD for verification infrastructure, prompt versioning, evaluation dataset management

Highly Desired Qualifications

• Experience with security scanning and static analysis tools such as Veracode, SonarQube, Snyk, or equivalent SAST platforms
• Hands-on experience with Playwright for end-to-end testing, including MCP-based test execution in agentic development workflows
• Experience with enterprise observability platforms including Azure Monitor, Application Insights, and Log Analytics
• Experience with Azure cloud infrastructure and enterprise system integrations such as D365 F&O and Salesforce or equivalent CRM
• Bachelor’s degree in Computer Science, Information Systems, or equivalent professional experience