Testing AI, Duplicate Test Detection, and the Death of Cucumber

[00:00:00]Is your test suite quietly billing you every single time it runs? Your agent wrote 30 new tests this week, but how many of them actually test anything new?

[00:00:08]And the guy who created Cucumber says he got it wrong. Here's what he built instead. Find out more in this episode of the Test Guild News Show for the week of July 20th. So grab your favorite cup of coffee and tea and let's do this.

[00:00:20]First up, if your team is testing anything with the model behind it, your test suite might be quietly running up a bill. And here's the exact line one team drew to stop it. Let's check it out.

[00:00:31]This by Debbie O'Brien who's published a piece on how her team test an AI product without paying for every run. And the problem she describes is specific.

[00:00:39]Testing a chat product means every single step is a real model call.

[00:00:43]Running the message, each specialized reply, and the evaluator that reads the whole conversation. Multiply that across five exercises and every suite run, and the cost climbs while the test gets lower and flakier. Her team's answer was to swap the model and keep everything else real. Playwright drives the actual desktop app, and a harness wraps the three functions that would otherwise hit a model, returning script replies instead. Messages still save through the real chat API and the real UI renders, so the test will still catch genuine regression issues. And Debbie is blunt about the risk here. A test that looks free, but quietly makes one real call, is worse than no test because you trust it and the bill creeps up anyway. She shipped a fix for exactly that leak. And with all the news about all the AI costs that's going on in the real world, I think Debbie really has a great solution for this that you definitely should check out using that link down below. So what agents cost you in tokens? Here's what they cost you in bloat. Because when an agent can write 30 tests in a session, nobody has to ask whether any of them are new.

[00:01:43]So Nuwan, founder and CEO of TestChip, has released a new free open-source CLI called Semantic Graph. And the problem it targets is one that agents have been creating.

[00:01:53]When a code agent can write dozens of Playwright tests in a single session, the bottleneck moves from authoring to governance. Things like, are the new tests distinct and useful, or are they duplicates of what you already have? He writes that an agent rewarded for adding a test has no innate sense that something already exists slightly reworded. So, your test suites will balloon with tests that assert the same behavior under different titles, or differ only in fixture data or selector phasing. And this obviously inflates CI time, confuses failure triage, and gives a false sense of depth while the behavioral breadth stalls. So, the CLI scans your spec and test files, embeds each test canonical text using an embedding model, clustered by similarities, and lays the results out in 2D, and names the clusters with a lightweight LLM pass, and serves a local interactive UI. Agents scoring at or above 0.92 is flagged as a strong dedupe candidate. And what's great about it is that it runs locally, no database and no test chip account required, and you don't need an embedded API key, also.

[00:02:55]All right, staying on tools you can install today, this next one is free, open source, and it's a single command to try against your own API. And I found this on Oscar's LinkedIn profile. He's the co-founder and CTO at Test Army.

[00:03:07]He's released Scout, which is an open source API testing CLI built for AI agents. And it works by parsing your open API spec, executing instrumental requests, and records structured findings and coverage reports. So, your agent can test your API before you even deploy it. And the part testers should care about is the safety design and requests are limited to the configured base URL host. So, post, put, patch, and delete are blocked by default and require an explicit opt-in. And as we talked about earlier with the token cost, there's a rate limited and request budget across the whole run, and credentials are pulled from environment variables at request time, and redacted from output. And it also ships with a bunch of bundled agent skills you install with a single command, plus a suggested prompt that tells agents exercise realistic user flows rather than reviewing endpoints in isolation and ask before touching anything that it writes. All right, for all you cucumber haters, this announcement caught me by surprise. It's by the actual creator of cucumber. Aslak has announced on LinkedIn a new open source tool called Vara and the framing is really unusual because he launched cucumber 18 years ago, co-wrote a book, co-founded a company, trained thousands of people, then sold up and walked away about 6 years ago. But this is his attempt to correct what he sees as its mistakes. He wrote how Gherkin felt too stilted and constraining. And he also admits that cucumber doesn't play well with other test runners outside the JVM. And the editor support was poor or is poor. So, what did he do? He created Vara. And what pushed him to do this was to build again with agents. After a couple of years coding with them, he wants to know whether the agent actually translated his prompts and specs into what he intended. So, all you need to do is write plain markdown, no Gherkin involved. The project pitch is executable documentation for humans and agents, turning your docs into tests.

[00:04:55]So, each paragraph runs as its own example with fresh state and when the markdown disagrees with the code, the test fails. There's also drift detection. And it supports a whole bunch of languages like TypeScript, Java, Python, Ruby, and Rust. And he's also asking for your feedback, good or bad, so you can find all those links in the comments down below. All right, we covered a lot of tooling, but this next one is all about the system around the tooling. And this is by David Burns who's a well-known Selenium committer and automation expert. And he published a piece last week arguing that testing feels when it feels like a chore bolted on to the end of a sprint. And I definitely agree when testing feels like extra work.

[00:05:31]He writes, "People skip it, rush it, or resent it, and that's exactly when critical issues reach production." His problem with shift left is how companies actually do it, forcing developers to run heavy test suites right before release just moves the bottleneck. David also wants testers acting as quality architects, building automated guardrails into the daily workflow. He says this is now survival because AI agents are writing committing code at speeds while staying blind to nuance, context, and more importantly, business risk. So, try to manually review all that at the end of a sprint and the volume will crush you and your pipeline.

[00:06:07]His answer is three layers running locally. Automated formatting and structure checks on every save.

[00:06:13]Real-time code analyzes inside the editor working like spell check for code and local commit gates in a block commit when a unit test fails or a critical vulnerability shows up so broken code never reaches the cloud. All right, next up is our webinar of the week. So, this week I'll be hosting a webinar along with Perfecto to go all over autonomous testing in action, functional performance, and mobile testing from a single prompt. The setup is one a lot of you will recognize. Fragmented tools, minimal script maintenance, and sprawling coverage has pushed QA past its limits and the test volume explodes.

[00:06:45]Traditional automation simply can't keep pace. So, we'll be joined by two awesome experts, Don Jackson and also Steven Feloney. I'm going to walk you through some of the known issues and also give you a walk-through of the Perfecto autonomous testing, which is the AI-driven platform that orchestrates functional performance and mobile testing from a single chat interface using natural language.

[00:07:06]You set the intent and the platform handles the rest.

[00:07:09]You get to see how to define performance and functional tests once in plain language and run them across web, mobile, and how AI-driven orchestration cuts maintenance to near zero and how unified execution and reporting gives you one view of quality. They'll also cover fully traceability from requirements to release and you see three-step workflows live. Describe your intent, let AI orchestrate, and analyze results in one place. You don't want to miss it. Check it out using the link down below and hope to see you there.

[00:07:37]All right, and as we close, a big question I get asked all the time, if AI writes most of the code, what exactly is your job? One of the people who's been living in this problem since the beginning, since the Bing search days, just wrote an awesome resource you need to know more about. So, Jason Arbon has a new book he just released called Testing AI, Engineering Confidence in Nondeterministic Systems. It runs about 21 chapters across five parts, and I'll be sitting down with Jason on an upcoming podcast, so stay tuned for that. The core argument in this book is one testers should really get to know.

[00:08:07]When systems are nondeterministic, quality is not proven by one good run.

[00:08:11]It's measured across samples, slices, traces, failures in time. And when something important fails, the next question isn't only what failed, it's why the evaluation system didn't catch it sooner. He also talks about a role shift where a developer who prompts an agent, waits, then clicks around the results to see if it works has become what he's calling a confidence engineer.

[00:08:33]So, the chapters cover a bunch of things like sampling and uncertainty, statistical tests for AI quality, judges and humans disagreement, building evals that matter, release readiness, and the anti-patterns that create false confidence, security and godrails governance. All right, before we go, a quick thank you because last week wasn't about me being behind the screen. Big shout-out to everyone who came to our Minneapolis and Chicago for the Test Guild IRL tour last week. Great turnout on both stops, and the conversations after the sessions were some of the best part of it as well. So, it was real testers, real problems. So, thank you everyone for showing up. Thank you for our awesome sponsor Test Mu. Thank you for your questions, and thank you everyone who stayed up late to keep talking. And if you were there, I would love you to drop a comment down below to tell everyone what you took home from it. All right, for links of everything we covered in this news episode, head over to those links down below. So, that's it for this episode of the Test Guild news show. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack pipeline automation awesomeness. And as always, test everything and keep it good.

[00:09:32]Cheers.