The Joomla! Bug Squad has been going strong for about five months now, and things couldn't be better. For the most part, we've been using a manual testing technique, simply because it's fast, easy, and flexible. We've had a few people interested in unit testing, especially for maintenance procedures, but for the most part unit testing hasn't caught on.

Okay, I'll admit, even I was skeptical about the value of unit testing. I knew it had benefits, but I really wondered if all the effort required to build unit tests was really worth it. Then, I encountered an issue with JCache. The patch was an easy fix, but I had no idea how to test the patch thoroughly. Then, it occurred to me that perhaps this was a good candidate for the famed unit tests!

Realizing a perfect trial by fire opportunity for the hard work that Alan Langford and others have done on the Joomla! Unit Testing Framework, I contacted Alan to see what we could do to test these JCache issues. Within a few hours (and significant help from the expert), I'd written a series of tests for various aspects of JCache. As it turns out, testing not only confirmed the patch had indeed fixed those issues, but, as an added bonus, testing also revealed several other issues we were not aware of.

It was a powerful lesson to learn that by investing two hours in writing these unit tests, I was not only able to validate the patch, but also identify several problems which months of stable release usage had not yet revealed. (Yes, fixes for those issues will be included in Joomla! 1.5.4). For me, the question of the value of unit testing has been answered. Not only is unit testing worth it, but it's necessary! Testing manually is GREAT, but Unit Testing can identify things manual testing just can't possibly find.

So, what's the ideal testing scenario? I'd say it's a mixture of both manual and unit testing. Manual testing still has a place for verifying the types of issues that are simply unrealistic for automated testing. Those issues might include usability questions, design or styling validation. However, unit testing easily catches quite a few issues that months of manual testing cannot, no matter how much you look for it. The best testing approach is to combine the best of both worlds; unit testing for areas that can be isolated, and manual testing for the system as a whole/external interactions.

The Bug Squad has shown how a test based maintenance procedure can rapidly and dramatically increase the stability of the project. Coupling those approaches with solid Unit Testing will surly lead to an even better platform.

For more information about the Bug Squad, see the Wiki and the Google Group. Community members are encouraged to join in the fun and help out with the Bug Squad. To get started, contact Wilco Jansen or me. Your involvement is welcome!