Learning Log 1: How to TDD in the age of AI

After a year on the road, it's time to get back to the keyboard. But what's the best way to do it these days? That's what I need to figure out before my next job interviews. I'm starting with a talk by Matteo Vaccari on TDD in the age of AI. My questions: Can I trust tests that an agent writes itself? What's the point of defining test cases as specifications, and when in the development process should I interact with the agent? I had a small example built for me, injected some bugs, and wrote down what I learned. Enjoy!

After two years away from Germany, I'm finally back home. If you've followed my sporadic updates, you know we first lived in Canada and then set off on a year-long trip around the world. It feels crazy to be back in Germany after all this time. It isn't even that cold yet, and still I'm freezing as if it were the middle of winter. After the endless summer we had on our trip, the trees are now finally turning yellow and orange. Our journey took us through Central and South America to Southeast and East Asia. We experienced so much that I still need time to process it all, but there's no chance to catch our breath just yet.

My savings have shrunk considerably over the past year, so I need a new source of income. That's why it came at just the right time when two recruiters reached out to me at the start of the week. I'm happy about it, of course, but now I have a lot of catching up to do so I don't embarrass myself in interviews. I can count on one hand how often I found time to write code last year. My former colleague Chris from my time at DNAstack told me he now lets agents do all his programming. How does that work, and can you get clean code out of it? Time to find out!

The Talk: Green or Not Green, Is the Feature Done or Is the Agent Fibbing?

I'm starting with a talk by Matteo Vaccari: A Harness for Behaviour: Ensuring Ai-generated Code Does What We Want - TDD in the Age of AI. In it, Matteo describes his approach to developing with agents. In short: when programming with agentic AI, we can't trust the agents. We also can't trust the tests an agent writes, keyword reward hacking. But there's also a weakness at the very beginning of development, in the first iteration of a test. Birgitta Böckeler has looked into TDD in the agent loop (article). She points out: if the agent writes both the test and the code, a red test only shows that the agent ran it and saw it fail. It doesn't tell you whether it failed for the right reason. So what's the solution?

The solution Matteo Vaccari proposes: the acceptance criteria are written as customer tests. The cases are stored as specifications in files, in his example as YAML tables, and a small test runner executes them. The system stays a black box. The agent is not allowed to rewrite these files to make the tests pass. It has to change the code. This is close to Ivett Ördög's "Approved Scenarios" approach, which Böckeler names as a good building block for trust in tests. She describes it as semi-manual testing with a dedicated runner per application: you review scenarios thoroughly and freeze them, and every later violation has to be approved again.

My aha moment: I had understood Matteo to mean that an agent uses the specifications to generate the tests and the code. Instead, the YAML files contain the test cases, and the test classes run them like a parameterized test. Smart, because that makes it obvious right away when an agent tries to cheat by changing a YAML file.

Learning by Doing

Okay, enough theory. What could this approach look like in a real project? To understand it, I first discussed what I'd learned from the video with Claude (Opus). That conversation produced a prompt, which I then used to have an agent build an example for me locally. It was meant to show me what the YAML cases, the reader, and the runner actually look like.

The result of the prompt was a solid Java repository. The agent stored the requested cases as YAML files in the project, and a helper class reads them into the tests. It reminds me a lot of behavior-driven development with Cucumber.

# Normal cases: price from the catalog (EUR) in the requested currency.
cases:
 - name: Pear in USD
   given:
     rates: { USD: "1.10" }
   when:
     price: { item: pear, currency: USD }
   then:
     price: "2.75"

 - name: Euro needs no rate, even when the rate service is down
   given:
     rate_service: unavailable
   when:
     price: { item: apple, currency: EUR }
   then:
     price: "1.00"

class ServiceCasesTest {

   @TestFactory
   Stream<DynamicTest> serviceCases() {
       return YamlCaseLoader.tests("service", c -> {
           PriceService service = new PriceService(givenRates(c), new PriceConverter(), message -> { });

           PriceService.Result result = service.price(c.text("/when/price/item"), c.text("/when/price/currency"));

           if (c.has("/then/price")) {
               assertNull(result.error(), "unexpected error");
               assertEquals(c.text("/then/price"), result.price().toPlainString());
           } else {
               assertEquals(c.text("/then/error"), String.valueOf(result.error()));
           }
       });
   }
   private static FakeExchangeRateClient givenRates(Case c) {
       FakeExchangeRateClient rates = new FakeExchangeRateClient();
       if (c.has("/given/rate_service") && c.text("/given/rate_service").equals("unavailable")) {
           rates.makeUnavailable();
       }
       c.data().at("/given/rates").properties()
               .forEach(e -> rates.setRate(e.getKey(), new BigDecimal(e.getValue().asText())));
       return rates;
   }
}

Then I tried it out and introduced two bugs into the code myself: a different rounding mode and a wrong return value in the PriceService. Both times a test failed. The refactoring also went through without problems, so the decoupling works. Of course, the code is very simple. Whether this holds up in day-to-day work remains to be seen. What are your experiences with it?

My Updated Workflow

I think this is how I'll implement features from now on:

1. Describe the feature in plan mode.
2. Design the test cases together with the AI and store them as YAML files.
3. Let the agent implement the feature without changing the approved cases.
4. Review the result.

The protection against agent trickery comes from the separation: the cases are data, the runner is code. A changed expected value stands out immediately in the diff. A loosened assertion in test code is much easier to miss. Matteo's main emphasis is "data instead of code".

A year ago, I already gained some experience with agentic development at DNAstack. Every test an agent wrote back then, and they loved writing lots of tests, I had to check. As my colleagues adopted agent-supported development more and more, their pull requests kept getting bigger. As a result, I spent more time reviewing PRs than working on my own tickets. To take pressure off reviewers and fight cognitive debt, meaning the growing gap between the existing code and how much of it the team actually understands, we started writing smaller pull requests. That helped us become productive again. So I was glad to see that Javier López Fernández, in his article "How to talk with AI?", also considers one big review at the end to be outdated. He wants small interactions with the AI: one single test assertion or a short interface definition per step, something you can check in seconds, combined with trunk-based development and continuous review. To me, that sounds less like handing programming off to an agent and more like working together, similar to pair programming. My workflow, on the other hand, lets the agent build an entire feature on its own. So far I prefer Javier's approach. From my experience, I don't trust agents to write high-quality code. We'll see whether the agentic loop or agentic pairing performs better. I have an open source project where I want to try this out soon. Stay tuned!

So What Have We Learned?

This deep dive into Matteo's talk and the articles by Birgitta Böckeler and Javier López Fernández made a few things clearer for me. I can't simply trust an agent or the tests it writes itself. I already knew that from my work at DNAstack. If the test cases are fixed as data up front, any change to them stands out in review, as long as I create the test files myself first. In my small example, the tests caught both injected bugs, and the refactoring didn't break anything either.

In my view, there's no best practice yet. My impression is that there are two camps: small steps with continuous review and trunk-based development (López Fernández) on one side, and specification up front with an agent loop (Vaccari, building on Ivett Ördög's Approved Scenarios) on the other.

Where I'm Still Stuck

The test runner is Java code, and in my example nothing protects it. Only the YAML files are locked by a rule in the CLAUDE.md. So an agent could still cheat in the test runner.

Here you can pick a camp. If you're in the optimistic camp, you'll use even more AI: a second agent checks whether the cases are really being read and executed. It would have a different goal, namely finding problems, while the first one is supposed to make the tests pass. Whether that works, I don't know. Both agents usually run on the same model, and a reviewer agent might take shortcuts too.

In the other camp, which is more critical of AI, I'd guess humans are needed more. Only a manual review can really make sure the tests test what they're supposed to. I suspect we won't get around that.

What's Next!

This is just the beginning of the series. Next, I want to understand what a harness actually is and what spec files are. After that, it's on to token economics: When do I use which model, how do I select it automatically for each task, how do I cut costs, and how do I track token usage over time? There's also the question of whether code should even be read anymore. And what is this new ATDD all about? As you can see, I still have a mountain of open questions ahead of me, but first I need to sort through the photos from our trip and swap my shorts for long pants.

While I was writing this post, I received the sad news that Patrick Stewart has passed away. To me, he will always be the best captain of the Enterprise. Rest in peace and Godspeed, Captain 🖖