Platemail News
Sunday, January 09, 2005
Extreme Testing:
I spent a little time last week re-aquainting myself with some of the philosophy behind Extreme Programming, and wondering how it might apply to a project like Platemail. A lot of the practices seem a little heavy-weight for this sort of project, and some just don't really apply, but there are a few ideas that really make a lot of sense.For example, there is no customer as such in our development, and it seems a little early to be worrying about release schedules and project velocity. The practices that really struck a chord with me were those based around structured testing, simplicity of design coupled with frequent refactoring, and collective code ownership. Here, I'd like to say a few things about testing.
Every developer agrees on the importance of thorough testing, but it seems to me that in practice, it doesn't really happen to the required extent. An ideal scenario would be to have a test for every piece of functionality in the system, testing each function, class and subsystem with all the corner cases. Such a suite could be run quickly and easily in a fully automated manner, so after every iteration of development, however small or large, you could be certain that the functionality hasn't regressed. Each new piece of code introduced to the source tree would be accompanied by a set of tests which would defend that functionality in future. Each bug that is found would be illustrated with a repeatable test case, which is also added to the test suite once the fix is implemented. I suspect that such a scenario is impractical for most projects, for several reasons.
One such reason is that although writing simple tests in conjunction with the development is probably not a burden, maintaining the tests may well become one. When a class is refactored to remove a function, or split a function into several smaller ones, a unit test for that function will fail. This test then needs to be updated, and care needs to be taken that no cases that the original test covered are now missed, thus opening the door for a bug to go unnoticed. Similarly, as the test suite becomes larger and more complex, it may outgrow the testing system which is in place. Soon, maintaining the testing infrastructure itself becomes a significant task.
A second issue which needs to be overcome is the mechanism of testing a given piece of functionality. The simple examples used in articles about testing are trivial to test - writing a test to show that the equals method of an object is performing correctly is not a problem. This is enough to get started; indeed, I've added a few simple tests for Platemail, which verify various functions of the logging class.
But testing the more complex aspects of Platemail is a challenge which will require much more thought. To test the parts of the code which talk to an IMAP server, for example, would require either an IMAP server, or something pretending to be one. To create a complex, repeatable test with either is a non-trivial task. To create a suite of tests that simulate different aspects of the IMAP protocol, and the variations used by different IMAP implementations, is a significant piece of work.
These issues are by no means insurmountable, but they do provide some food for thought as Platemail develops. As I add small unit tests for individual pieces of core functionality, I'm starting to look into practical ways of automating testing of Platemail as a whole. There are frameworks for automating GUI testing, and there are always Expect scripts for talking to a command line, or even talking a protocol like IMAP. I'm also considering creating a database of various formats and sizes of emails, to test that Platemail can correctly handle them all. Each time an email is found that Platemail doesn't like, it can be added to the database.
But even these will only test a small portion of Platemail will eventually contain. We'll ultimately have to test that user settings are saved and restored correctly between settings. If Platemail's data formats change between versions, we'll need to verify that the behaviour is correct when loading data that was created by a different version. As if all this wasn't enough, I'd like to be able to test different JVMs at different versions of Java on different platforms. So testing-wise, there's plenty of work ahead in the future, to say nothing of the project's implementation itself.
posted by Neil Campbell 10:41 PM
