Yeah, having your agent write 3x the code in exhaustive tests (I tried this recently and got 600 lines of tests for my 100 lines of code!) sure makes things look great, but when you actually look at the content of the tests they’re meaningless. Good tests validate the use of design patterns, ensure that dependencies hold, and are meaningful (e.g. shortcut debugging by setting up useful state) when they break.

This was really good, and second leaning on property testing. I’ve had really good outcomes from setting up Schemathesis and getting blanket coverage for stuff like “there should be no request you can generate as logged in user A that let’s you do things as or see things that belong to user B”, as well as “there should be no request you can find to any API endpoint that can trigger a 5xx response”

Test theatre is exactly the right framing. The tests are syntactically correct, they run, they pass but do they actually prove anything?

I've found the best way to achieve that is to force the agent to do TDD. Better to get it to do Outside-in TDD. Even better to get it to run Outside-in TDD, then use mutation testing to ensure it has fully covered the logic.

I've written about this and have a POC here for those interested: https://www.joegaebel.com/articles/principled-agentic-softwa...

You mean an LLM wrote an article for you. I got bored not even halfway through because it wrote a wall of text but it didn't actually say anything.

Actually, every single word was hand typed. You'll probably notice that in areas where I could improve my grammar. It's understandable that you hit the wall of text and felt a bit dismayed by the length- hence the TLDR at the top and the example repo :)

Test theatre isn’t new. Most people writing tests do the exact same thing, testing implementation.

This is why people don't see the problem with tests that agents generate after the implementation. They look exactly like what they write.