Julien Phalip

One great advantage of using Vagrant is that it allows to work with virtual machines (VMs) that provide cleanly separated, sandboxed environments. I find this so convenient that I systematically build individual VMs for every single Web project I work on. This setup definitely works great. However, since I frequently switch between several projects multiple times during the day, I often end up having multiple VMs running at the same time. This eventually causes a lot of RAM to be consumed on my laptop and it then becomes necessary to shut down some of the VMs that I don’t need to use any more.

Vagrant already offers a pretty simple command to turn off a VM:

$ vagrant halt

With this command, Vagrant attempts a graceful shutdown of the VM (e.g. by issuing a halt in Linux). However, it sometimes happens that, when something gets screwed up within the VM’s environment, this command simply freezes and never completes. Also, if you wish to turn off multiple VMs, you’ll have to execute this command multiple times: once for each VM from within its corresponding directory on your host machine.

So, to make life a little bit easier, I wrote a simple Python script (full code provided in the gist below) that directly issues a poweroff command to VirtualBox. This command has the same effect on a VM as pulling off the power cable on a real computer. While I would obviously recommend against doing this in production, it is generally perfectly safe to do in a development environment. Besides, it makes the VM shut down really fast. By passing the --all parameter you may also run this script just once to instantly shut down all running VMs.

To make this script executable from anywhere, I recommend placing it inside your PATH (e.g. ~/bin/) and giving it the execution flag (e.g. chmod u+x poweroff.py). Here’s an example running the script:

$ poweroff.py --all
2 VM(s) currently running...
Powering off VM: d8ec66a6-6455-416a-969b-be44fc094c91...
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Powering off VM: d3ce89aa-5700-4060-87cd-1e04b8c8cef2...
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Hopefully this will of some use to the Vagrant users out there. When Vagrant eventually (soon!) supports VMWare, this script could be easily updated to work with it as well.

https://gist.github.com/eaab558be836f423f927

Quick note: if you’re using Python < 2.7 you will first need to install the argparse module:

$ pip install argparse

There are situations where real time is important in the behaviour of some views. For example, one could write an anti-spam mechanism to prevent robots or ill-intended users to post a form too often and too rapidly. In this article I’ll present a strategy for testing such views.

Let’s take a very simple example, in which we’ll let users post messages. The time constraint we’ll set is that users have to wait at least 15 minutes between 2 consecutive postings, otherwise their message will be ignored. I’ll start by giving out the code. First, the model:

from django.db import models
from django.contrib.auth.models import User

class Message(models.Model):
    user = models.ForeignKey(User, related_name='messages')
    datetime = models.DateTimeField()
    message = models.CharField(max_length=100)
    
    def __unicode__(self):
        return self.message

Nothing complicated here. Just a simple model to record the user and the time at which the message is posted.

Then, we will use a setting to determine the interval of time a user should wait before posting another message (You will see later why using a setting is crucial here). In our example we pick 15 minutes. So, add the following to your settings.py file:

MESSAGE_INTERVAL = 60 * 15 # 15 minutes expressed in seconds

Now, on to the view:

from datetime import datetime, timedelta

from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist
from django.http import HttpResponse

from models import Message

def send_message(request):
    message_interval = timedelta(seconds=settings.MESSAGE_INTERVAL)
    now = datetime.now()
    try:
        Message.objects.get(user=request.user, datetime__gte=now - message_interval)
        # A message has been posted too recently, so ask the user to wait a bit
        return HttpResponse('Be patient, try again later.')
    except ObjectDoesNotExist:
        # The user waited long enough, so we can create the message
        Message.objects.create(user=request.user, message=request.POST['message'], datetime=now)
        return HttpResponse('Thanks for your message.')

It’s a pretty simple view, but let me just explain what’s going on in there. First, we retrieve the time interval from the settings (it should be 15 minutes, or whatever value you’ve set). Then we check if a message has been posted by the same user during the last 15 minutes. If there is one, return an error notice telling the user he needs to wait; otherwise save the message and return a success notice.

All good! Now, how do we go about testing this? I’ll start by giving out the code:

from time import sleep

from django.test import TestCase
from django.conf import settings
from django.contrib.auth.models import User

from models import Message

message_interval = 1 # seconds
not_long_enough = 0.7 # seconds
long_enough = 1.3 # seconds

class TimeBasedTesting(TestCase):
    
    def setUp(self):
        self.old_MESSAGE_INTERVAL = settings.MESSAGE_INTERVAL
        settings.MESSAGE_INTERVAL = message_interval
        User.objects.create_user('testuser', 'testuser@example.com', 'testpw')
   
    def tearDown(self):
        settings.MESSAGE_INTERVAL = self.old_MESSAGE_INTERVAL

    def test_message(self):
        self.client.login(username='testuser', password='testpw')
        
        # First message
        response = self.client.post('/send_message/', { 'message': 'First try!' })
        self.assertEquals(response.content, 'Thanks for your message.')
        
        # Wait enough
        sleep(long_enough)
        response = self.client.post('/send_message/', { 'message': 'Second try!' })
        self.assertEquals(response.content, 'Thanks for your message.')
        
        # Don't wait enough
        sleep(not_long_enough)
        response = self.client.post('/send_message/', { 'message': 'Third try!' })
        self.assertEquals(response.content, 'Be patient, try again later.')
        
        # Wait enough
        sleep(long_enough)
        response = self.client.post('/send_message/', { 'message': 'Fourth try!' })
        self.assertEquals(response.content, 'Thanks for your message.')
        
        # Check what messages have been recorded
        self.assertEquals(str(Message.objects.all()), '[<Message: First try!>, <Message: Second try!>, <Message: Fourth try!>]')

Something we cannot afford is wait for 15 minutes to run each test. This is exactly why we put the waiting time interval into a setting — because then we can override it in our tests. The setUp() method first makes a backup of the setting (stored in old_MESSAGE_INTERVAL) and then overrides it before the tests are run. The tearDown() method eventually restores the setting when the tests are complete so there’s no risk to interfere with other applications or with other tests.

In this example we have set the time interval message_interval to 1 second, and we use two other variables to simulate whether the user waits long enough or not long enough between each posting. Then, to simulate the user waiting we simply use the built-in python function sleep(). It’s as simple as that! This test should take approximately 5 seconds to run.

Finally, there’s no problem if for some reasons you do not want to have the time interval in your settings. In fact, you can put it anywhere you like. The key is to move it to a variable out of your view so it can easily be overridden by your tests.

Hope it helps!

This post is a reply to Eric Holscher’s call for suggestions to extend and improve the documentation about testing in Django. By no means am I an expert in testing, but I thought I would share some thoughts I’ve had about this. It is more like a dump of ideas that come from my own experience, and hopefully that will contribute slightly to the debate.

The existing documentation is already pretty good but it is still a bit scarce. It is true that there is a gap in this area and that testing doesn’t get the level of attention it deserves (I’m talking, broadly, in the Django community). Developers who regularly write tests already know how good testing is; but there should be more education provided to newcomers so that testing becomes a more common practice. And this would, in turn, benefit the community as a whole. By the way, Eric should be thanked for leading the way in this quest. If you haven’t yet, you should definitely check out the great screencasts and tutorials he’s posted on his blog about this topic.

Personally, I didn’t know anything about testing until I discovered Django and started trying to write patches for it. I found that pretty much every patch had to include tests or it wouldn’t stand a chance to be checked in by the core devs. I originally thought this was a bit of a stubborn ideology, but after diving into it I quickly realised how important and significant tests were. Put simply, Django would NOT be this good (in terms of features) and this reliable (in terms of robustness) if it didn’t have that comprehensive test suite. I encourage anybody to have a look at it; this is one of the most pedagogic ways to understand Django’s inner-workings and best practices.

Anyways, let’s cut the crap. In the remaining of this post I’ll talk about some misconceptions which I believe exist amongst the developers who are new to testing. Then I’ll list some advantages I’ve noticed in writing tests for my Django apps. And finally I’ll give some suggestions to improve the existing documentation, since that’s what this is all about.

The word “test” sounds a bit like “spinach”, doesn’t it? When you were a kid, didn’t your mom try to make you eat that disgusting green porridge saying that it was good for you? Well, when I first heard about testing that’s exactly the feeling I had: everyone says it’s good for you but no one really feels like giving the first bite. Maybe one of the reasons is that by writing tests you are very likely to find bugs in your code; and finding new bugs is never a good feeling. But, hey! Isn’t it better to find bugs early on rather than when you least expect them?

So yes, testing requires discipline. But it is NOT hard! In fact, if you look at it closely, the code used in tests is usually damn straightforward. If you are scared by tests then you should realise that it is completely irrational. Simply jump into it! It won’t take long before you recognise the benefits and it will give you the most rewarding feeling. There’s even room for being creative when writing tests.

Believe it or not, but writing tests can get addictive. Oh, by the way, I now eat spinach and I like it :)

“What? I’m already swamped with development work and you want me write tests on top of that?!”

I think this would be a quite natural reaction from someone who’s never tried testing. Well, let’s put it this way: writing tests doesn’t add work load, it just makes you reorganise the way you approach development.

You know, I’m going to tell you a secret: you already do “testing”. Ok, this is not exactly a secret, but what I mean is that you must already be doing some kind of “human” testing: you open your browser, type in the URL, fill out some funky values in your forms (e.g. “test1”, “dfdsfsdf”, “blah”), click “Submit” and then check that it works of fails as expected. You’re doing that, aren’t you?

Writing “machine” tests will cut a lot of that work load. Even better, you can run those tests a zillion times each day for free, if you use a bot! Really, the machine should execute the tests, not you! I see the writing of tests more like a replacement (not complete, but at least partial) of the human tests that we already do on a daily basis.

I think there is a common belief that writing tests is time-consuming and that it slows you down in the development of your apps. In fact, if you look at the overall process, I contend that it actually makes you save time. And it does so at many different levels.

Here’s a silly question: Do you want to find bugs and fix them now while you’re building your app, or do you want to wait for your app to shamefully crash in front of the world? Obviously you don’t want the world to know that your app contains bugs, and testing does help with that.

Like I said before, unless you’re a total genius with a Python compiler implanted in your brain, it’s very much likely that you will discover bugs in your code while writing tests. And that’s in essence the whole beauty of testing. By killing bugs early on you can save a lot of the time which you would otherwise spend debugging when the bugs emerge days or weeks later. Also, for some reasons it seems to be much easier to think of all the edge cases when writing tests than when simply testing the app in the browser.

The other great thing about writing tests during development is that your mind is still very familiar with the code, so you can quickly identify the source of the bugs and fix them. Debugging weeks after means that you have to spend some extra time re-familiarising yourself with the code, before being able to do anything: W.A.S.T.E of time. Of course, writing tests as you go won’t prevent some lurking bugs to appear once in a while, but if you are rigorous enough most bugs will be killed before they’re even born.

Writing tests might seem a daunting task. But as development unfolds, it’s just a matter of adding a few lines of code each time you add a feature or fix a bug. It is a capitalisation process and a solid investment of your time. It’s like you keep putting money in the bank. During times of crisis like we’re in, it’s good to have at least something reliable! :)

I don’t know about you, but I often find myself spending 70% of the time fine-tuning small cosmetic details rather than writing actual code that does actual stuff. I can’t help it, but I just hate developing something and testing it in the browser if it looks too wonky and is too naked visually. It just irritates me.

I’m pretty sure this happens to a lot of us. We can easily get distracted by the graphics and presentation of an app while we should be writing proper code instead. Testing really helps with that. When you’re writing tests you naturally focus on getting the sh*t done. I believe this is because the purpose of testing is to make sure that your app WORKS well, not that it LOOKS good. You will do the graphic design work later, or even delegate it to someone else.

So, if you get in the habit of writing tests regularly, you will automatically cut a lot of time that you would otherwise spend in procrastination dealing with the aesthetical and superficial aspects of your app.

This follows directly on the previous point: if you don’t get distracted by the aesthetics while writing tests it is simply because you don’t have to use the browser. As I mentioned earlier, testing your app manually by clicking links, typing fake data, etc. can take a lot of time if you add all those tasks up.

Plus, as you gain experience and get better at writing tests you will also get faster at identifying and killing bugs. But could your browser be faster, or could you ever click and type faster? Probably not.

Obviously, you do need sometimes to check that your app works fine in the browser. But writing tests allows you to do that less often. As a challenge, I once tried to develop a simple, but big enough Django app (which processed some forms to add records in the database and also sent and processed confirmation emails), while writing tests in parallel and never using the browser at all. Then, once the core of the code was completed I tried it in the browser for the first time, and it just worked. All that was left to do was fine-tuning the templates and doing all the styling. If I can, I’ll try to write another blog post explaining the process I’ve followed for that one.

Something that I think is just great is that the database is re-created each time you run the test suite. This is particularly useful when you start developing a new app, with new models. Indeed, during these early stages the models are likely to change a lot as you develop the app’s specifications. If you do human/manual tests, it requires that you first create your database with syncdb, and then, if you want to make changes you have to do them manually in the database (unless you’re using one of the emerging solutions for that problem, like django-evolution or South, but those are not quite fully functional yet).

If instead you write “machine” tests then you don’t have to worry as much about experimenting with the database schema because the changes will be taken into account straight away the next time the suite is run. You don’t need to do anything more than simply change your model declarations. Believe me, that alone saves a great deal of time and frustration.

In the previous section I have already mentioned a few advantages of testing. That was to specifically respond to misconceptions some developers may have. Here I’ll list a few other advantages that are important in my eyes.

First, as I already said before, testing allows you to anticipate bugs and kill them before they even exist. Another great benefit is that it prevents regression. Typically, regression means that something that used to work doesn’t work anymore because of the insertion of some new code. In other words: you’ve broken your app while you were in fact trying to improve it. There is no worse feeling that breaking some good code on which you’ve spent time and sweat. If you have written tests that check your code is working properly, then those tests will fail and shout at you the next time you break your code, therefore preventing you from releasing a half-broken app. If you find a new bug, then fix it and write tests for it, so you’re sure that you’re done with it: that bloody bug won’t come around again!

Second, writing tests forces you to have a critical look and to think effectively about the strengths and flaws of your app. After all, writing tests is about trying to break your app in any imaginable way. This forward-thinking process also greatly helps in the specification stage. If you’re not enthusiast about drawing diagrams or drafting use cases, and you just want to dive into the code, then testing is for you. Testing makes you think about the big picture because it requires you to ask yourself questions like “What is this piece of code supposed to do and not do?” or “What would be appropriate and inappropriate input and output for this function?” Still, you’re doing that thinking at the same time as you’re programming, so you can satisfy you thirst for coding throughout the process :)

Finally, tests give you confidence. And confidence means both credibility towards your clients and peace of mind for yourself. For all the reasons enumerated above, you know that if you keep up with tests it is less and less likely your app will break. Priceless.

So, after all this wordiness, what is it we should do to entice more Django developers into writing tests and improving the (already awesome) Django’s test framework? The existing documentation is already pretty good, but I believe there’s a lack of practical examples with a good balance of best practices and purely technical considerations. Here are a few suggestions.

There should be some documentation explaining some good strategies in testing Django apps. It is out of question to give a full lecture on test-driven development, but at least some good hints and start points would be welcome. Here are some examples of strategies I think would be worth expanding on in the documentation:

• When you fix a bug, you want to fix the root cause, not the symptoms. Conversely, when you’re testing you want to make sure the symptoms don’t show up again. Functions, models, views, etc. should be treated as black boxes, and the tests should check that the inputs and outputs are correct, not how the job is actually done inside the black box.
• It would be good to give hints on the sequence in which things should be tested. Personally, I would advocate starting to test the small elements (e.g. custom form fields) before the bigger ones (e.g. the views). I would also recommend to tests all imaginable bad scenarios for a given element before moving to the next bigger element, so to ensure you build your app on solid grounds.
• There should be some pointers as to what is important to test in your app and what isn’t. As I said, testing can be addictive and you can rapidly find yourself writing tons of tests that are useless. It’s important to keep thinking: Is this test necessary? What should I really be testing? What are the priorities?
• There should be how-to guides explaining the best practices for testing a view, a middleware, a decorator, a template, a model (and a model field), a form (and a form field), a template tag, the sending of emails, etc. Obviously there is not a single way to test each of these, but providing a few alternatives would constitute a very good starting point.

The existing documentation explains quite well how to set up tests for individual apps, but there is very little information on how to set up project-specific tests. You might want, for example, to check that some specific data is systematically added to your database when the project is up and running. Again, I don’t think there is one single way to do this, since the needs may vary from one project to another or from one environment to another. But it would be great if people could share their own tips about it, and maybe some best practices will emerge. I’m planning to write up another blog post soon to explain one particular setup I’ve made on a recent project. Stay tuned for more.

There should be a section in the documentation with some tips and tricks that make testing easier. Hey, I’ll start with one! When I’m in development phase I need to run the test suite quite often, and that can take quite a while to process each time. So, what I usually do is use a SQLite database for testing, which is way faster than others I know. Then, once in a while I run the suite on the system that will be used in production (e.g. MySQL).

In conclusion I would say that, even if writing tests as you go might seem a bit time-consuming in the first place, it will save you a great deal of time and frustration in the long run. Django helps lever the foundations of your app quite quickly, and then testing should help consolidate them by closing the gaps and tightening the screws. You will then be waiting for the earthquake with a big smile on your face (sorry for the bad metaphor :) )

To finish, I recommend you to check out the slides from the Django master class that was given by Jeremy Dunck, Jacob Kaplan-Moss, and Simon Willison at OSCON 07. There are some excellent tips on unit testing and other areas of Django. Remember also that the best place to look for examples is Django’s test suite itself. Dozens (hundreds!) of people have contributed to it over the years so you will find different styles of testing in it. You should also have a look at the tests included in the most popular third party apps, which should provide plenty of inspiration.

Phew… this post ended up being much longer than I anticipated. If you’ve read until this point, thank you :) I hope this contributes in some way to the debate, and I hope that at least this will help demystify testing for beginners. I’ll post more on this topic, so stay tuned.

A common pattern in websites is when a few pages are protected and require a login to be accessed. The @login_required decorator often comes in handy for these situations. But, another pattern which is quite common is when most of the site is protected, with just a few exceptions of pages that remain public (e.g. frontpage, registration page, etc.). In that case, it can be quite tedious to decorate all of the views with @login_required, and it can be easy to forget to decorate some of them.

So, I came up with a simple system which by default protects every view and then lets you explicitly tell which views should be public. This makes things both easier and less error-prone.

The core of that system is contained in the following middleware code:

import re
from django.conf import settings
from django.contrib.auth.decorators import login_required
from path.to.your.decorators import PublicView

class LoginRequiredMiddleware(object):    
    def __init__(self):
        self.public_patterns = []
        self.public_views = []
        if hasattr(settings, 'PUBLIC_VIEWS'):
            for view_path in settings.PUBLIC_VIEWS:
                view = self.get_view(view_path)
                self.public_views.append(view)            
        if hasattr(settings, 'PUBLIC_PATHS'):
            for public_path in settings.PUBLIC_PATHS:
                self.public_patterns.append(re.compile(public_path))

    def get_view(self, view_path):
        i = view_path.rfind('.')
        module_path, view_name = view_path[:i], view_path[i+1:]
        module = __import__(module_path, globals(), locals(), [view_name])
        return getattr(module, view_name)

    def matches_public_view(self, view):
        if self.public_views:
            for public_view in self.public_views:
                if view == public_view:
                    return True
        return False

    def matches_public_path(self, path):
        if self.public_patterns:
            for pattern in self.public_patterns:
                if pattern.match(path) is not None:
                    return True
        return False

    def process_view(self, request, view_func, view_args, view_kwargs):
        if request.user.is_authenticated() or isinstance(view_func, PublicView) or self.matches_public_path(request.path) or self.matches_public_view(view_func):
            return None
        else:
            return login_required(view_func)(request, *view_args, **view_kwargs)

To install this middleware, simply copy and paste the above code anywhere in your project, for example in a file called middleware.py. Then, update the MIDDLEWARE_CLASSES setting in your project’s settings file:

MIDDLEWARE_CLASSES = (
    ...
    'path.to.your.middleware.LoginRequiredMiddleware',
)

You’ll notice that, at the top of the middleware code above, there is an import of the PublicView class. You need to update that import path after having copied/pasted the following snippet anywhere in your project, for example in a decorators.py file:

try:
    from functools import update_wrapper
except ImportError:
    from django.utils.functional import update_wrapper  # Python 2.3, 2.4 fallback.

from django.contrib.auth.decorators import _CheckLogin

def login_not_required(view_func):
    """
    Decorator which marks the given view as public (no login required).
    """
    return PublicView(view_func)


class PublicView(object):
    """
    Forces a view to be public (no login required).
    """
    def __init__(self, view_func):
        if isinstance(view_func, _CheckLogin):
            self.view_func = view_func.view_func
        else:
            self.view_func = view_func
        update_wrapper(self, view_func)
        
    def __get__(self, obj, cls=None):
        view_func = self.view_func.__get__(obj, cls)
        return _PublicView(view_func)
    
    def __call__(self, request, *args, **kwargs):
        return self.view_func(request, *args, **kwargs)

The above code contains a new decorator (@login_not_required) which will be explained in detail in a moment.

At this point, all of your views will require you to log in, including the login page itself. So, we now need to specify the few views that should be public. There are three different ways at your disposal: using a special decorator, listing the public views, or listing the public URL paths.

Thanks to the new @login_not_required you can explicitly force a view to be public. Here’s an example:

from path.to.your.decorators import login_not_required

@login_not_required
def frontpage(request):
    ...

In this case, the frontpage view will be properly displayed even if you’re not logged in.

If you don’t have direct access to modify a view’s code (e.g., it’s in a third-party application), you still can force that view to be public by adding it to the new PUBLIC_VIEWS setting in your settings file. Here’s an example if you’re using the django.contrib.auth system and the django-registration application:

PUBLIC_VIEWS = [
    'django.contrib.auth.views.login',
    'django.contrib.auth.views.password_reset_done',
    'django.contrib.auth.views.password_reset',
    'django.contrib.auth.views.password_reset_confirm',
    'django.contrib.auth.views.password_reset_complete',
    'registration.views.register',
    'registration.views.activate',]

The third and last way is to directly specify the URL paths (as regular expressions) for the pages you want to be public. This can be useful, for example, if a page is rendered by a generic view. It is also useful if you are serving your media files statically via Django (only recommended in development mode). For that, you need to add the PUBLIC_PATHS setting in your settings file. Here’s an example:

PUBLIC_PATHS = [
    '^%s' % MEDIA_URL,
    '^/accounts/register/complete/$', # Uses the 'direct_to_template' generic view
    ]

That’s it! By using this technique your site will be protected effectively and it will be easy to maintain. I hope it helps! Any comment or remark is very welcome ;)

Search is a feature that is — or at least, should be — present on most sites containing dynamic or large content.

There are a few projects around to tackle that. Here’s a non-exhaustive list: djangosearch, django-search (with a dash), django-sphinx.

Those search engines are great, but they seem like overkill if you just need a simple search feature for your CMS or blog.

To deal with that, I’ve come up with a generic and simple trick. All you need is copy/paste the following snippet anywhere in your project:

import re

from django.db.models import Q

def normalize_query(query_string,
                    findterms=re.compile(r'"([^"]+)"|(\S+)').findall,
                    normspace=re.compile(r'\s{2,}').sub):
    ''' Splits the query string in invidual keywords, getting rid of unecessary spaces
        and grouping quoted words together.
        Example:
        
        >>> normalize_query('  some random  words "with   quotes  " and   spaces')
        ['some', 'random', 'words', 'with quotes', 'and', 'spaces']
    
    '''
    return [normspace(' ', (t[0] or t[1]).strip()) for t in findterms(query_string)] 

def get_query(query_string, search_fields):
    ''' Returns a query, that is a combination of Q objects. That combination
        aims to search keywords within a model by testing the given search fields.
    
    '''
    query = None # Query to search for every search term        
    terms = normalize_query(query_string)
    for term in terms:
        or_query = None # Query to search for a given term in each field
        for field_name in search_fields:
            q = Q(**{"%s__icontains" % field_name: term})
            if or_query is None:
                or_query = q
            else:
                or_query = or_query | q
        if query is None:
            query = or_query
        else:
            query = query & or_query
    return query

What the above does is generate a django.db.models.Q object (see doc) to search through your model, based on the query string and on the model’s fields that you want to search. Importantly, it also analyses the query string by splitting out the key words and allowing words to be grouped by quotes. For example, out of the following query string…

'  some random  words "with   quotes  " and   spaces'

…the words 'some', 'random', 'words', 'with quotes', 'and', 'spaces' would actually be searched. It performs an AND search with all the given words, but you could easily customise it to do different kinds of search.

Then, your search view would become as simple as:

def search(request):
    query_string = ''
    found_entries = None
    if ('q' in request.GET) and request.GET['q'].strip():
        query_string = request.GET['q']
        
        entry_query = get_query(query_string, ['title', 'body',])
        
        found_entries = Entry.objects.filter(entry_query).order_by('-pub_date')

    return render_to_response('search/search_results.html',
                          { 'query_string': query_string, 'found_entries': found_entries },
                          context_instance=RequestContext(request))

And that’s it! I use this on a site that has about 10,000 news items and it works pretty fast…

Now you have no excuse not to add a search box to your site! ;)

A few months ago I’ve conducted my first case study. For 2.5 months I’ve followed a film score project between a Melbourne-based filmmaker and a Sydney-based composer. It has been a fantastic experience, and although I improvised quite a bit, the results have been extremely rich and valuable.

Now I’m confronted to writing the report for that case study. That is not an easy task. The best place to start is probably with Robert Yin’s books — Case Study Research - Design and Methods and Applications of Case Study Research.

I think the most difficult part of the report is to build the case for you case study: justify that it was the right way to go and that you’ve set up the proper environment to collect meaningful data. Analysing and summarising your findings is the fun part, and I can’t wait to get to that point :)

In this post I share some thoughts on one way to customise the Django’s admin interface beyond what, I believe, it was originally designed for. Well, at least it’s an approach that I used to bring django-treemenus’ codebase up to the NewForms-Admin’s API, while preserving the app’s original behaviour.

First, you may want to check the latest release of django-treemenus (0.6). In that release I’ve completely refactored the code to use all the goodness of NFA. Backward incompatible changes are minimal if you weren’t using the extension system, and from the user’s point of view everything is pretty much the same as before. The result is quite satisfactory: the amount of code was reduced by more than half, every known issue was fixed, and it is now much easier to extend/hack this app for those who are interested.

Doing that refactoring made me realise even more how great NFA is. Still, I did not quite want to use it the “standard” way. Basically, I wanted to keep the URL scheme that was used in previous versions of treemenus. For example:

/admin/treemenus/menu/1/            -> The menu #1 edit page.
/admin/treemenus/menu/1/items/add/  -> Add an item to menu #1.
/admin/treemenus/menu/1/items/9/    -> The item #9 edit page, within menu #1.

Also, I did not want to allow the items to be edited directly without the context of the menu they belong to. Therefore, I wanted to both avoid having a MenuItemAdmin class freely accessible from the admin’s index page, and avoid enabling the following URLs:

/admin/treemenus/menuitem/
/admin/treemenus/menuitem/9/

To achieve that, I have first overriden the call method in the customised MenuAdmin class. I wish this could be done a bit more cleanly, so I’ll probably open a ticket one day, proposing to add a simple extra hook which would greatly simplify the customisation of URL routing in the admin.

Then, because every single request would systematically be routed to the MenuAdmin class, I’ve used a private instance — that is, not “officially” registered — of MenuItemAdmin as a proxy to manipulate the menu items. For, example, here’s how the MenuItemAdmin’s add_view is proxied:

def add_menu_item(self, request, menu_pk):
    ...
    menuitem_admin = MenuItemAdmin(MenuItem, self.admin_site, menu)
    return menuitem_admin.add_view(request, extra_context={ 'menu': menu })

To understand how it works, let’s follow the route that is taken when an item is added to a given menu. First, the URL to visit is /admin/treemenus/menu/1/items/add/. This will be routed to the MenuAdmin’s __call__ method, which in turn will pass on the request to the above-mentioned add_menu_item method. There, a private instance of MenuItemAdmin is created and the request is passed on to its own add_view method. After that, NFA takes over and does its wonders to process the form and create the new item in database. The same approach is applied for all the other views: change, delete and the custom move up/down.

All this may sound complicated, but it is in fact pretty simple. If you’re interested, it’s probably best to check out the source code as it should speak for itself. At least, it will probably speak better than I’ve tried to in this post :)

NFA is a fantastic improvement to the Django’s admin system, and browsing into its depths taught me some good lessons and good practices in Python and Django programming. Now, I also believe that there is still some room for a few simple backward compatible changes that would greatly improve its customisability. All the “hacks” I’ve done here would then become trivial, and that would open many opportunities for customising admin apps. Anyway, I’ll probably post more about that in a few weeks, when things “settle down” a bit after the awesome and most anticipated Django 1.0 is released.

I’d be glad to hear any idea/criticism about this approach, so feel free to drop a line or two in the comments ;)

It is important to be aware that dates are spelled differently in different countries (e.g. dd/mm/yyyy in Australia or mm/dd/yyyy in the US). This is why it is a good idea to let the user select their preferred date format and store it into their user profile. For example, you may store the values "%d/%m/%Y" or "%m/%d/%Y" in that user’s profile. That way, you may display dates in the format chosen by the user throughout the site.

Now, if the site contains many forms with date fields (say, for example, you’re building a calendar application), it can be a bit repetitive and annoying to check and assign the date format for every form in every view. To go around that, I came up with a simple trick. It all happens in the following class:

class FormWithFormattedDates(forms.ModelForm):
    def __init__(self, *args, **kwargs):
        date_format = None
        if 'date_format' in kwargs:
            date_format = kwargs['date_format']
            del kwargs['date_format'] # The ModelForm's __init__ method doesn't expect to receive that argument, so take it out.
        super(FormWithFormattedDates, self).__init__(*args, **kwargs)
        if date_format is not None:
            for (field_name, field) in self.fields.items():
                if isinstance(field, forms.fields.DateField):
                    field.input_format = [date_format]
                    field.widget = forms.widgets.DateTimeInput(format=date_format)

What this class does is explore what’s in the form, and make sure that all date fields will display and validate with the given date format.

Then, all you need to do is to make all your forms that contain date fields inherit from the above class. For example:

# Model
class Event(models.Model):
    title = models.CharField(max_length=100)
    start_date = models.DateField()
    end_date = models.DateField()

#Form
class EventForm(FormWithFormattedDates):
    class Meta:
        model = Event

The next step is then in the view. When you instantiate the form, just pass it the user’s preferred format as parameter:

def add_new_event(request):
    if request.method == 'POST':
        form = EventForm(data=request.POST, date_format=request.user.get_profile().date_format)
        if form.is_valid():
            ...

All the validation logic is taken care of by the FormWithFormattedDates class. This allows to both keep the views’ code very simple, and also to not have to worry about per-user validation as it all happens automatically.

Update: I could add that you could use the same approach to let the user select their preferred time format (e.g. “1pm” or “13:00”), or, in fact, with any kind of validation that needs to be operated on a large number of forms in your site and where that validation also depends on the user’s preferences.

I have just packaged a new release 0.5 for django-treemenus

That release should only concern people working on Django’s development version after the merge of the newforms-admin branch. I also hear that Django 1.0 alpha has just been released, so that’s good timing ;)

If you’re using Django’s trunk prior the NFA merge, then you can stick to 0.4.

I’ve also included the German translation kindly provided by Thomas Kerpe (thanks Thomas!). Available languages are now: English, French, Dutch, Russian and German. Please keep sending me your translations and they will be included in future releases.

Also, I’d be very interested to hear testimonials of people using this app. How do you use it? Do you use the menu extension mechanism? How would you like to see this app improved? Any feedback/criticism is very welcome ;)

I have just released the version 0.4 of django-treemenus

It does not contain code modifications so you don’t necessarily have to upgrade if you’re currently using it. In fact, this release integrates more languages, so you may be interested if you’re not happy with the standard English version.

Thank you to Maxim Oransky for marking a couple of missing strings and for providing the Russian translation. Thank you also to Ido Sebastiaan van Oostveen for providing the Dutch translation. I’ve also added French locale, so that’s now 4 languages including English. Please send me your translations in other languages and I’ll integrate them in future releases.

Django-treemenus has been quite stable it seems since the last release. I use it in several projects, and my clients like it simplicity of use (in particular the user-friendly representation of the tree structure in the admin).

I’d be happy to hear more feedback, so please let me know how you use this, if you’ve extended or improved it. Any suggestion or testimonial is very welcome!

Finally, a couple of things I’m planning to work on in the near future: handle caching and better integration with newforms-admin.

UPDATE: This blog has since been moved to Tumblr

I finally managed to take a bit of time to set up this blog. It was implemented in Django and based on the excellent Coltrane (along with template-utils and comment-utils) by James Bennett.

For more technical details, the typography is enhanced with typogrify, tagging is done with django-tagging, and the code highlighting is managed by pygments. I did the graphic design myself, quite simple but well enough for the purpose of this site.

This blog will mostly be a place for me to discuss the progress of my PhD research, web development subjects or my interests in music and photography.

Stay tuned for more, and grab the feeds of the categories/tags that are of interest for you. I’d be happy to discuss so feel free to leave your comments on the various entries of this blog.