During the last couple of days, I’ve had some issues because some specific test fails in TeamCity.
What is hard to understand is why they fail sporadically, I’ve performed executions of the same test in my environment, but I have not been able to reproduce the behavior. But in the TeamCity server they present a really unpredictable behaviour. I was able to identify what is the test that fails every time during the execution in TeamCity.
Trying to find a solution to this problem (and also to be able to run all the test successfully), I found a very interesting article from Martin Fowler. The name of the article is Eradicating Non-Determinism in TestsEradicating Non-Determinism in Tests (a highly recommended reading by the way :)).
In this article Martin defines a non-deterministic test as tests that sometimes pass and sometimes fail….
If you continue reading the article, Martin explains why non-deterministic test are dangerous in some way for your project, the following paragraph has been taken from the reading:
Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite. As a result they need to be dealt with as soon as you can, before your entire deployment pipeline is compromised.1
One of the suggestions in the reading is to identify and isolate(quarantine) those test but keep an eye on that (don’t forget they need to be fixed anyway :)).
I hope you enjoy reading it 🙂