Years ago, I used mutation testing in a pretty big Java project. Let me tell you: You don't want so much coverage. It's usually a waste of time and makes changes to existing code very annoying. Keep mutation testing/high coverage for a small selection of _very important_ code, but don't overdo it
pfdietz 119 days ago [-]
But did it lead to the finding of additional bugs?
looperhacks 117 days ago [-]
It's very likely that over the years, it found something. But we didn't have anything to compare against, because it was always mutation testing from day one.
aDyslecticCrow 119 days ago [-]
I read about mutation testing in a course of test oriented programming. And i am unsure to this day of what domain it's meant for.
The most strict and safety critical code i have encountered, was specifying its system behaviour in a state machine diagram of a ISO specification.
Yet it was too state based for effective unit testing. Instead it opted to assert the living hell out of the codebase and be paranoid about even the stability of the ram it wrote to.
I have a hard time imagining a domain that is both;
1. Easily to coverage test to a high enough degree that mutating testing can be used.
2. Needs pedantic testing to the point of absurdity, perhaps already using boundary value diagrams for its test-cases.
And so mutating testing feels quite odd to me, and selling it as an AI tooling is even more insane. It feels like far more effort than to just not use AI.
mkw5053 119 days ago [-]
My arc of using AI code generation tools started by steadily adding more and more guardrails. I realized the returns were diminishing while cost, complexity, and latency climbed. Mutation testing made it obvious I had gone too far. While there is a time and place for it, I completely lost the flow and was fighting against my dev setup. I’ve since found a better balance: the most strict, opinionated dev environment, especially on types and linting, with the least code and configuration.
swader999 119 days ago [-]
Fascinating idea. The price stopped me right in my tracks.
The most strict and safety critical code i have encountered, was specifying its system behaviour in a state machine diagram of a ISO specification.
Yet it was too state based for effective unit testing. Instead it opted to assert the living hell out of the codebase and be paranoid about even the stability of the ram it wrote to.
I have a hard time imagining a domain that is both;
1. Easily to coverage test to a high enough degree that mutating testing can be used.
2. Needs pedantic testing to the point of absurdity, perhaps already using boundary value diagrams for its test-cases.
And so mutating testing feels quite odd to me, and selling it as an AI tooling is even more insane. It feels like far more effort than to just not use AI.
https://research.google/pubs/state-of-mutation-testing-at-go...