This post is a reply to Eric Holscher's call for suggestions to extend and improve the documentation about testing in Django. By no means am I an expert in testing, but I thought I would share some thoughts I've had about this. It is more like a dump of ideas that come from my own experience, and hopefully that will contribute slightly to the debate.

The existing documentation is already pretty good but it is still a bit scarce. It is true that there is a gap in this area and that testing doesn't get the level of attention it deserves (I'm talking, broadly, in the Django community). Developers who regularly write tests already know how good testing is; but there should be more education provided to newcomers so that testing becomes a more common practice. And this would, in turn, benefit the community as a whole. By the way, Eric should be thanked for leading the way in this quest. If you haven't yet, you should definitely check out the great screencasts and tutorials he's posted on his blog about this topic.

Personally, I didn't know anything about testing until I discovered Django and started trying to write patches for it. I found that pretty much every patch had to include tests or it wouldn't stand a chance to be checked in by the core devs. I originally thought this was a bit of a stubborn ideology, but after diving into it I quickly realised how important and significant tests were. Put simply, Django would NOT be this good (in terms of features) and this reliable (in terms of robustness) if it didn't have that comprehensive test suite. I encourage anybody to have a look at it; this is one of the most pedagogic ways to understand Django's inner-workings and best practices.

Anyways, let's cut the crap. In the remaining of this post I'll talk about some misconceptions which I believe exist amongst the developers who are new to testing. Then I'll list some advantages I've noticed in writing tests for my Django apps. And finally I'll give some suggestions to improve the existing documentation, since that's what this is all about.

Misconceptions about testing

"Testing is hard, tedious and not much fun"

The word "test" sounds a bit like "spinach", doesn't it? When you were a kid, didn't your mom try to make you eat that disgusting green porridge saying that it was good for you? Well, when I first heard about testing that's exactly the feeling I had: everyone says it's good for you but no one really feels like giving the first bite. Maybe one of the reasons is that by writing tests you are very likely to find bugs in your code; and finding new bugs is never a good feeling. But, hey! Isn't it better to find bugs early on rather than when you least expect them?

So yes, testing requires discipline. But it is NOT hard! In fact, if you look at it closely, the code used in tests is usually damn straightforward. If you are scared by tests then you should realise that it is completely irrational. Simply jump into it! It won't take long before you recognise the benefits and it will give you the most rewarding feeling. There's even room for being creative when writing tests.

Believe it or not, but writing tests can get addictive. Oh, by the way, I now eat spinach and I like it :)

"Testing adds work load"

"What? I'm already swamped with development work and you want me write tests on top of that?!"

I think this would be a quite natural reaction from someone who's never tried testing. Well, let's put it this way: writing tests doesn't add work load, it just makes you reorganise the way you approach development.

You know, I'm going to tell you a secret: you already do "testing". Ok, this is not exactly a secret, but what I mean is that you must already be doing some kind of "human" testing: you open your browser, type in the URL, fill out some funky values in your forms (e.g. "test1", "dfdsfsdf", "blah"), click "Submit" and then check that it works of fails as expected. You're doing that, aren't you?

Writing "machine" tests will cut a lot of that work load. Even better, you can run those tests a zillion times each day for free, if you use a bot! Really, the machine should execute the tests, not you! I see the writing of tests more like a replacement (not complete, but at least partial) of the human tests that we already do on a daily basis.

"Testing takes time and slows development down"

I think there is a common belief that writing tests is time-consuming and that it slows you down in the development of your apps. In fact, if you look at the overall process, I contend that it actually makes you save time. And it does so at many different levels.

It's an investment: Save now for later

Here's a silly question: Do you want to find bugs and fix them now while you're building your app, or do you want to wait for your app to shamefully crash in front of the world? Obviously you don't want the world to know that your app contains bugs, and testing does help with that.

Like I said before, unless you're a total genius with a Python compiler implanted in your brain, it's very much likely that you will discover bugs in your code while writing tests. And that's in essence the whole beauty of testing. By killing bugs early on you can save a lot of the time which you would otherwise spend debugging when the bugs emerge days or weeks later. Also, for some reasons it seems to be much easier to think of all the edge cases when writing tests than when simply testing the app in the browser.

The other great thing about writing tests during development is that your mind is still very familiar with the code, so you can quickly identify the source of the bugs and fix them. Debugging weeks after means that you have to spend some extra time re-familiarising yourself with the code, before being able to do anything: W.A.S.T.E of time. Of course, writing tests as you go won't prevent some lurking bugs to appear once in a while, but if you are rigorous enough most bugs will be killed before they're even born.

Writing tests might seem a daunting task. But as development unfolds, it's just a matter of adding a few lines of code each time you add a feature or fix a bug. It is a capitalisation process and a solid investment of your time. It's like you keep putting money in the bank. During times of crisis like we're in, it's good to have at least something reliable! :)

Don't worry about the aesthetics

I don't know about you, but I often find myself spending 70% of the time fine-tuning small cosmetic details rather than writing actual code that does actual stuff. I can't help it, but I just hate developing something and testing it in the browser if it looks too wonky and is too naked visually. It just irritates me.

I'm pretty sure this happens to a lot of us. We can easily get distracted by the graphics and presentation of an app while we should be writing proper code instead. Testing really helps with that. When you're writing tests you naturally focus on getting the sh*t done. I believe this is because the purpose of testing is to make sure that your app WORKS well, not that it LOOKS good. You will do the graphic design work later, or even delegate it to someone else.

So, if you get in the habit of writing tests regularly, you will automatically cut a lot of time that you would otherwise spend in procrastination dealing with the aesthetical and superficial aspects of your app.

Don't worry about the browser

This follows directly on the previous point: if you don't get distracted by the aesthetics while writing tests it is simply because you don't have to use the browser. As I mentioned earlier, testing your app manually by clicking links, typing fake data, etc. can take a lot of time if you add all those tasks up.

Plus, as you gain experience and get better at writing tests you will also get faster at identifying and killing bugs. But could your browser be faster, or could you ever click and type faster? Probably not.

Obviously, you do need sometimes to check that your app works fine in the browser. But writing tests allows you to do that less often. As a challenge, I once tried to develop a simple, but big enough Django app (which processed some forms to add records in the database and also sent and processed confirmation emails), while writing tests in parallel and never using the browser at all. Then, once the core of the code was completed I tried it in the browser for the first time, and it just worked. All that was left to do was fine-tuning the templates and doing all the styling. If I can, I'll try to write another blog post explaining the process I've followed for that one.

Don't worry about database schema

Something that I think is just great is that the database is re-created each time you run the test suite. This is particularly useful when you start developing a new app, with new models. Indeed, during these early stages the models are likely to change a lot as you develop the app's specifications. If you do human/manual tests, it requires that you first create your database with syncdb, and then, if you want to make changes you have to do them manually in the database (unless you're using one of the emerging solutions for that problem, like django-evolution or South, but those are not quite fully functional yet).

If instead you write "machine" tests then you don't have to worry as much about experimenting with the database schema because the changes will be taken into account straight away the next time the suite is run. You don't need to do anything more than simply change your model declarations. Believe me, that alone saves a great deal of time and frustration.

Other advantages of testing

In the previous section I have already mentioned a few advantages of testing. That was to specifically respond to misconceptions some developers may have. Here I'll list a few other advantages that are important in my eyes.

First, as I already said before, testing allows you to anticipate bugs and kill them before they even exist. Another great benefit is that it prevents regression. Typically, regression means that something that used to work doesn't work anymore because of the insertion of some new code. In other words: you've broken your app while you were in fact trying to improve it. There is no worse feeling that breaking some good code on which you've spent time and sweat. If you have written tests that check your code is working properly, then those tests will fail and shout at you the next time you break your code, therefore preventing you from releasing a half-broken app. If you find a new bug, then fix it and write tests for it, so you're sure that you're done with it: that bloody bug won't come around again!

Second, writing tests forces you to have a critical look and to think effectively about the strengths and flaws of your app. After all, writing tests is about trying to break your app in any imaginable way. This forward-thinking process also greatly helps in the specification stage. If you're not enthusiast about drawing diagrams or drafting use cases, and you just want to dive into the code, then testing is for you. Testing makes you think about the big picture because it requires you to ask yourself questions like "What is this piece of code supposed to do and not do?" or "What would be appropriate and inappropriate input and output for this function?" Still, you're doing that thinking at the same time as you're programming, so you can satisfy you thirst for coding throughout the process :)

Finally, tests give you confidence. And confidence means both credibility towards your clients and peace of mind for yourself. For all the reasons enumerated above, you know that if you keep up with tests it is less and less likely your app will break. Priceless.

So, what should we do?

So, after all this wordiness, what is it we should do to entice more Django developers into writing tests and improving the (already awesome) Django's test framework? The existing documentation is already pretty good, but I believe there's a lack of practical examples with a good balance of best practices and purely technical considerations. Here are a few suggestions.

Strategies for testing

There should be some documentation explaining some good strategies in testing Django apps. It is out of question to give a full lecture on test-driven development, but at least some good hints and start points would be welcome. Here are some examples of strategies I think would be worth expanding on in the documentation:

• When you fix a bug, you want to fix the root cause, not the symptoms. Conversely, when you're testing you want to make sure the symptoms don't show up again. Functions, models, views, etc. should be treated as black boxes, and the tests should check that the inputs and outputs are correct, not how the job is actually done inside the black box.
• It would be good to give hints on the sequence in which things should be tested. Personally, I would advocate starting to test the small elements (e.g. custom form fields) before the bigger ones (e.g. the views). I would also recommend to tests all imaginable bad scenarios for a given element before moving to the next bigger element, so to ensure you build your app on solid grounds.
• There should be some pointers as to what is important to test in your app and what isn't. As I said, testing can be addictive and you can rapidly find yourself writing tons of tests that are useless. It's important to keep thinking: Is this test necessary? What should I really be testing? What are the priorities?
• There should be how-to guides explaining the best practices for testing a view, a middleware, a decorator, a template, a model (and a model field), a form (and a form field), a template tag, the sending of emails, etc. Obviously there is not a single way to test each of these, but providing a few alternatives would constitute a very good starting point.

Project-specific testing

The existing documentation explains quite well how to set up tests for individual apps, but there is very little information on how to set up project-specific tests. You might want, for example, to check that some specific data is systematically added to your database when the project is up and running. Again, I don't think there is one single way to do this, since the needs may vary from one project to another or from one environment to another. But it would be great if people could share their own tips about it, and maybe some best practices will emerge. I'm planning to write up another blog post soon to explain one particular setup I've made on a recent project. Stay tuned for more.

Tips and tricks

There should be a section in the documentation with some tips and tricks that make testing easier. Hey, I'll start with one! When I'm in development phase I need to run the test suite quite often, and that can take quite a while to process each time. So, what I usually do is use a SQLite database for testing, which is way faster than others I know. Then, once in a while I run the suite on the system that will be used in production (e.g. MySQL).

Conclusion

In conclusion I would say that, even if writing tests as you go might seem a bit time-consuming in the first place, it will save you a great deal of time and frustration in the long run. Django helps lever the foundations of your app quite quickly, and then testing should help consolidate them by closing the gaps and tightening the screws. You will then be waiting for the earthquake with a big smile on your face (sorry for the bad metaphor :) )

To finish, I recommend you to check out the slides from the Django master class that was given by Jeremy Dunck, Jacob Kaplan-Moss, and Simon Willison at OSCON 07. There are some excellent tips on unit testing and other areas of Django. Remember also that the best place to look for examples is Django's test suite itself. Dozens (hundreds!) of people have contributed to it over the years so you will find different styles of testing in it. You should also have a look at the tests included in the most popular third party apps, which should provide plenty of inspiration.

Phew... this post ended up being much longer than I anticipated. If you've read until this point, thank you :) I hope this contributes in some way to the debate, and I hope that at least this will help demystify testing for beginners. I'll post more on this topic, so stay tuned.