In software testing, you’ll hear the terms “red” and “green”. These refer to the status of tests, which have a binary result: pass (green) or fail (red). This is pretty basic at the unit level (for one test), but it is equally important at the system level (for all tests). Test frameworks will present a “summary” color status of green (meaning ALL tests pass), or red (meaning ONE or more tests fail). That’s it: a single failed test causes the whole system to go to a red state (meaning failed). There is no yellow, or even lavender, lime, or tangerine. (And no prizes given for “good attendance”.) Having a binary result is important, because it confronts a major weakness in our human nature to ignore or de-emphasize problems, and paint overly optimistic descriptions. 99% does NOT cut it. This really means ALL green. If any break, then you stop everything and fix that broken test. Here’s why: If you accept that some of your tests are failing, accepting the notion that “well, most of them are good”, then you are left in a murky situation where you don’t really know if a change broke something. After all…was that 23 or 24 tests that were broken yesterday? Never rely on memory. It’s so easy for a bug to be introduced, and if you don’t have an “all green” policy, then you really don’t know if this code change broke anything, and you lose the value of having that “watchdog”, guarding your code. That really is a sea-change, when you have all your tests passing, since they now can take on a different role, a more pro-active role of guarding code quality. This allows you to add unit tests to your build process, as part of Continuous Integration (CI), so whenever anyone checks in any code, all the unit tests are automatically run, and if any were broken, the build should fail as well (since the “build” is now more than just compiling code, but also executing tests.)
What if want to add unit-test execution to your build but have a LOT of failing tests, without time to fix them? I’m going to make a controversial statement here, and it’s very intentional. If a set of failing tests are all that stop you from adding unit-test execution to your build process, then I would say to mark those failing tests as “[Ignore]” (or comment them out, or whatever you need to do). Yes, I can hear screams from purists right now, but if this sacrifice is what it takes to get a test-verified build process, then you seriously need to consider just that. (Yes, fix any tests you can, but regardless, don’t let failing tests stop you.) You’ve probably heard it before: “The PERFECT is the enemy of the GOOD.” Don’t allow a hopeless yearning for perfection stop you from doing what you know is good and right. So if disabling a few tests gets you to a test-verified build (which is a VERY GOOD thing), then do what you need to do. Cut some throats. Having said that, I will emphasize this should be a ONE-TIME concession, for the purpose of adding unit-test execution to your build process. Going forward, disabling a unit test is NOT the way to solve a failed test. At that time, you have the following options:
- Fix the application code, if this truly was a bug in the code
- Fix the test, if it no longer has correct assumptions about the code (which often is the case, as the application evolves)
- Remove the test, if it is obsolete. This is an extension of #2. Since unit tests are intended to be small, testing discrete points, and since the natural code process sometimes involves major refactoring of classes, it’s reasonable to say that SOMETIMES a given test may become obsolete, where it really no longer server a meaningful purpose. In that case, yes, it’s alright to remove the test. But this should be rare.
In the end, you want your unit tests to serve a very important role of guarding the integrity of your code, and the only way that can be accomplished is if you uphold a policy ensuring ALL GREEN. Uphold this, and you’ll be doing yourself a big favor in the long run.