Why it is necesary to erradicate non-deterministic test?

During the last couple of days, I’ve had some issues because some specific test fails in TeamCity.

What is hard to understand is why they fail sporadically, I’ve performed executions of the same test in my environment, but I have not been able to reproduce the behavior. But in the TeamCity server they present a really unpredictable behaviour. I was able to identify what is the test that fails every time during the execution in TeamCity.

Trying to find a solution to this problem (and also to be able to run all the test successfully), I found a very interesting article from Martin Fowler. The name of the article is Eradicating Non-Determinism in TestsEradicating Non-Determinism in Tests (a highly recommended reading by the way :)).

In this article Martin defines a non-deterministic test as  tests that sometimes pass and sometimes fail….

If you continue reading the article, Martin explains why non-deterministic test are dangerous in some way for your project, the following paragraph has been taken from the reading:

Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite. As a result they need to be dealt with as soon as you can, before your entire deployment pipeline is compromised.1

One of the suggestions in the reading is to identify and isolate(quarantine) those test but keep an eye on that (don’t forget they need to be fixed anyway :)).

I hope you enjoy reading it 🙂


About Michael Hidalgo

Michael is a Software Developer Engineer based on San José, Costa Rica. He leads the OWASP Chapter from Costa Rica. You can take a look at my blog at http://blog.michaelhidalgo.info/
This entry was posted in Browser Automation, Selenium WebDriver and tagged . Bookmark the permalink.

One Response to Why it is necesary to erradicate non-deterministic test?

  1. Pingback: Implementing best practices on TeamMentor Automation project based on learned leasons | TeamMentor Development and Testing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s