Four case studies, end to end

Problem

Testing was fragmented across manual teams and multiple tools. Regression cycles ran long, locators drifted constantly, automation was flaky, and nobody had a single view of release risk. Agents running on third-party platforms could not be instrumented from the inside.

Approach

Treat quality itself as an agentic workflow. Six specialised agents each own one step - analysis, design, generation, execution, failure analysis and self-healing - with a human quality gate holding the release. Test from the outside, the way a customer experiences the system.

Architecture

JIRA / OpenAPI / live URL feeds the Analyst Agent, then Test Designer, Test Generator, Executor, Failure Analyst and Self-Heal Agent, ending at a Quality Gate wired into GitLab CI/CD, Slack and Grafana.

Testing

Six test types from one pipeline: functional, end-to-end, REST API, performance, security via OWASP ZAP, and cross-browser. Scenarios are generated from acceptance criteria rather than written by hand, including negative and edge cases.

Automation

An async job queue drives isolated runners executing Playwright and Selenium workflows in parallel. The Self-Heal Agent recovers from UI drift using deterministic locator strategies and DOM heuristics first, with LLM proposals only as a fallback.

Metrics

Problem

LLM output is non-deterministic, so a pass/fail assertion is the wrong instrument. The same prompt returns different words every run, hallucinations surface only on the paths nobody tested, and a model upgrade can silently regress tone, policy compliance or accuracy.

Approach

Split the judgement in two. An LLM judge scores the subjective dimensions against explicit rubrics; deterministic checks handle everything that can be proved - schema, PII, policy, required disclosures. Anything that can be checked deterministically never goes to a model.

Architecture

Golden datasets feed a scenario generator producing adversarial and functional cases. Each response passes through the judge layer and the deterministic layer in parallel, with results written to a versioned baseline for candidate-vs-baseline diffing.

Testing

Adversarial suites built for the failure modes that matter: hallucination traps, multi-turn context drift, prompt injection, policy violations and edge cases that hand-written suites routinely miss.

Automation

Evaluations run as a CI gate. Every candidate is scored against the current baseline, regressions are attributed to the dimension that moved, and fail-closed guardrails block a release rather than logging a warning.

Metrics

Problem

Enterprise suites were serial, slow and brittle. A single DOM change broke dozens of tests, flaky failures trained the team to ignore red builds, and runs were too slow to sit inside a deployment pipeline.

Approach

Make the suite fast enough to be a gate and stable enough to be trusted. Parallel execution across isolated runners, a layered locator strategy that survives DOM change, and honest quarantine for genuinely flaky tests instead of blanket retries.

Architecture

An async job queue distributes work to isolated Playwright and Selenium runners, sharded for parallel execution across browsers, with results aggregated into a single report and pushed to CI/CD, Slack and Grafana.

Testing

Functional, end-to-end and cross-browser coverage alongside REST API and performance suites, exercised against the same scenario set so a regression is caught wherever it surfaces.

Automation

Self-healing locators with deterministic strategies and DOM heuristics, drift alerts on retry, and CI/CD webhook integration so every merge triggers the relevant slice rather than the whole suite.

Metrics

Problem

Financial products spanning Lending, Payments, Insurance and Collections were validated largely by hand. Full cycle testing took 54 hours, releases waited on manual regression, and quality standards varied between country teams.

Approach

Transform the platform and the organisation together. Automate the regression backbone, virtualise unstable downstream services, put observability behind the pipeline, and establish shared engineering standards and delivery governance across all three sites.

Architecture

API-first automation with Selenium UI coverage on top, WireMock service virtualisation to decouple from unstable dependencies, and Grafana observability across CI/CD pipelines feeding automated quality gates.

Testing

Risk-based regression across all four financial product lines, plus performance and customer experience testing, prioritised by exposure rather than by module.

Automation

Parallel execution against optimised regression suites with automated quality gates, replacing the manual sign-off that had been sitting on the critical path of every release.

Metrics