Unit Testing for Diogenes =========================== Unit testing is the process of writing lots of little pieces of test code which each test some part of the functionality of a software system. By having a comprehensive set of test cases which are satisfied by the production code, we can have real confidence in our software, instead of just "hoping" it works. What is a unit test? ---------------------- The general structure of all unit tests is as follows: * Setup pre-conditions (the state of the system prior to the execution of the code under test) * Run the code to be tested, giving it all applicable arguments * Validate that the code ran correctly by examining the state of the system after the code under test has run, and ensuring that it has done what it should have done. The pre-conditions are normally either set at the beginning of the test method, or in a general method called setUp() (see below for the structure of test code). Running the code itself is normally making a call to the function to be tested, constructing an object, or, for Web applications, running the top-level web script (using require()). Validating the post-run state of the system is done by examining the database, system files, and script output, and using various assert methods to communicate the success or failure of various tests to the user. Testing Web applications such as Diogenes is relatively easy, especially the user interface side of things. Unlike traditional GUI applications, every state change of a web application is defined by the database and filesystem, the user's session, and the values passed from the web browser through $_REQUEST and $_SERVER. Since all of these aspects are relatively easy to control, setting the pre-run state of the application is quite simple. There are also relatively few discrete states that your web application can be in, because all of the system control has to be made through the constrained interface above. Testing post-conditions, as well, is simple. If you want to ensure that a certain thing is part of the post-run display, you can just interrogate the HTML output from the test run, which is all text and can be tested with assertRegExp() and assertSubStr(). Again, the only state available is in the user's session, database, and filesystem, all of which are easy to interrogate. Test cases and test methods ----------------------------- (This is where things get a bit hairy -- hold on, and perhaps read it twice) A test case is a collection of tests which are related in some way. Each test case is represented in our testing framework by a class subclassed from PHPUnit_TestCase. Each test is a method on the test case whose name starts with 'test'. You should name the tests appropriately from there. Test cases can have a couple of special methods defined on them, called setUp() and tearDown(). The first method is called before each of the test methods is called, and the second is called after each test method has run. The setUp() and tearDown() methods are the primary reason for grouping test methods together. Methods should, as much as possible, grouped into test cases with common setUp() and tearDown() requirements. So how do I write a Unit Test? ------------------------------- Put together a snippet of code which sets up the state of the application, then run the code to be tested, and check the result. This snippet of code should be placed in a test case in a method named test[Something](). Each test method takes no arguments and returns nothing. Have a look at the existing test code in the testing/ directory of the diogenes source distribution for examples of how to write test code. When you create a new test case, you need to tell the test code runner that it needs to run the new test code. There is an array in the alltests script which defines the files and test cases to be run. Each key in the array specifies a filename to be read (without the .php extension), while the value associated with the key is an array of class names in that file. What should I test? --------------------- The naive answer would be "everything". However, that is impractical. There are just too many things that could be tested for a policy of "test everything" to allow any actual code to be written. It helps to think of tests as pulls of the handle on a poker machine. Each pull costs you something (some time to write it). You "win" when a test that you expected to pass fails, or when a test you expected to fail passes. You lose when a test gives you no additional useful feedback. You want to maximise your winnings. So, write tests that demonstrate something new and different about the operation of the system. Before you write any production code, write a test which defines how you want the system to act in order to pass the test. Run the test suite, verify that the test fails. Now, modify the production code just enough so the test passes. If you want the system to do something that can't be expressed in one test, write multiple tests, each one interspersed with some production development to satisfy *just* *that* new test. This is, in my experience, the best way to ensure that you have good test coverage, whilst minimising the production of tests which add no value to the system. How do I retro-fit unit testing onto an existing codebase? ------------------------------------------------------------ Diogenes already has a significant amount of code written, which would take hundreds of tests to cover completely. There is little point in going back and writing tests for all of this functionality. It appears to work well enough, so we should just leave it as-is. However, from now on, every time you want to make some modification (whether it be a refactoring, a bug fix, or a feature addition), write one or more test cases which demonstrate your desired result: Refactoring: Write tests surrounding the functionality you intend to refactor. Show the test cases accurately represent the desired functionality of the system by ensuring they all run properly. Then perform the refactoring, ensuring you haven't broken anything by making sure the tests all still run properly. Bug fix: Write one or more tests which shows the bug in action -- in other words, it hits the bug, and produces erroneous results. Then modify the system so that the test passes. You can be confident that you've fixed the bug, because you have concrete feedback in the form of your test suite that the bug no longer exists. Feature addition: Put together some tests which invoke the feature you want to add, and test that the feature is working as it should. Naturally, these tests will fail at first, because the feature doesn't exist. But you then modify the production code to make the feature work, and you stop when your tests all pass. Over time, as old code gets modified, the percentage of code covered by the tests will increase, and eventually we will have a comprehensively tested piece of software. During modifications, if you manage to break something accidentally, write a test to show the breakage and fix it from there. If you broke it once, there's a good chance it'll break again when someone else modifies it, and there should be a test to immediately warn the programmer that they've broken something. How do I run the unit tests? ------------------------------ The primary script that executes all of the unit tests is the 'alltests' script in the testing/ directory of the distribution. However, the output of this script is one line for every test that passes or fails. To help highlight the test failures, there is a 'run' script, which filters out all of the passes, and only shows you the failures. Very useful. So, your regular test run will be done with ./run, but if you want to see a thousand passes, run ./alltests. Both of these should be run from the testing/ directory. Running it from elsewhere isn't likely to get good results.