Julien Phalip

Jul 11

A script to quickly access the source of installed python packages

It is sometimes useful to access the source of an external Python package in your current virtualenv, for example to inspect its code or to place breakpoints while debugging.

If you use virtualenv_wrapper then you may use the cdsitepackages command. However, this doesn’t help much for packages that have been installed in “editable” mode (that is, typically in the <virtualenvpath>/src/ folder).

So I wrote a little bash & zshell script to quickly access any package, regardless of where they are installed in your virtualenv. This script also allows tab completion to make things even faster:

(myenv)$ goto <TAB>
admin_tools      curses           encodings        feincmstools     lib2to3          paramiko         ratelimit
appconf          dateutil         example          gargoyle         logging          password_reset   requests
bsddb            debug_toolbar    example_module   gunicorn         markdown         PIL              setuptools
compiler         distutils        example_project  hotshot          modeldict        pip              sorl
compressor       django           fabfile          idlelib          mptt             psycopg2         south
ctypes           email            feincms          jsonfield        nexus            pyrepl

(myenv)$ goto d<TAB>
dateutil       debug_toolbar  distutils      django

(myenv)$ goto django
(myenv)$ pwd

Here’s the source:


To install this script in your user’s bin/ folder, simply run the following:

(myenv)$ mkdir -p ~/bin/
(myenv)$ wget https://gist.github.com/jphalip/5967635/raw/goto.sh -P ~/bin/
(myenv)$ source ~/bin/goto.sh

To make this script accessible in all future terminal sessions, simply add source ~/bin/goto.sh to your user’s bash or zshell profile.

PS: Thanks to Carlos Nepomuceno for providing some useful tips about pkgutil on the Python users mailing list.

Feb 22

Fast shutdown of development Vagrant/VirtualBox virtual machines

One great advantage of using Vagrant is that it allows to work with virtual machines (VMs) that provide cleanly separated, sandboxed environments. I find this so convenient that I systematically build individual VMs for every single Web project I work on. This setup definitely works great. However, since I frequently switch between several projects multiple times during the day, I often end up having multiple VMs running at the same time. This eventually causes a lot of RAM to be consumed on my laptop and it then becomes necessary to shut down some of the VMs that I don’t need to use any more.

Vagrant already offers a pretty simple command to turn off a VM:

$ vagrant halt

With this command, Vagrant attempts a graceful shutdown of the VM (e.g. by issuing a halt in Linux). However, it sometimes happens that, when something gets screwed up within the VM’s environment, this command simply freezes and never completes. Also, if you wish to turn off multiple VMs, you’ll have to execute this command multiple times: once for each VM from within its corresponding directory on your host machine.

So, to make life a little bit easier, I wrote a simple Python script (full code provided in the gist below) that directly issues a poweroff command to VirtualBox. This command has the same effect on a VM as pulling off the power cable on a real computer. While I would obviously recommend against doing this in production, it is generally perfectly safe to do in a development environment. Besides, it makes the VM shut down really fast. By passing the --all parameter you may also run this script just once to instantly shut down all running VMs.

To make this script executable from anywhere, I recommend placing it inside your PATH (e.g. ~/bin/) and giving it the execution flag (e.g. chmod u+x poweroff.py). Here’s an example running the script:

$ poweroff.py --all
2 VM(s) currently running...
Powering off VM: d8ec66a6-6455-416a-969b-be44fc094c91...
Powering off VM: d3ce89aa-5700-4060-87cd-1e04b8c8cef2...

Hopefully this will of some use to the Vagrant users out there. When Vagrant eventually (soon!) supports VMWare, this script could be easily updated to work with it as well.


Quick note: if you’re using Python < 2.7 you will first need to install the argparse module:

$ pip install argparse

Update July 29th, 2013: Updated the script to work with Vagrant 1.1+

Update September 23th, 2013: As pointed out by Alex Dergachev in the comments, vagrant halt --force can in fact be used for the same thing. However, this script also allows to shutdown all currently-running VMS with the --all option, which you may still find useful.

Jan 26

Testing time-based views in Django

There are situations where real time is important in the behaviour of some views. For example, one could write an anti-spam mechanism to prevent robots or ill-intended users to post a form too often and too rapidly. In this article I’ll present a strategy for testing such views.

Let’s take a very simple example, in which we’ll let users post messages. The time constraint we’ll set is that users have to wait at least 15 minutes between 2 consecutive postings, otherwise their message will be ignored. I’ll start by giving out the code. First, the model:

from django.db import models
from django.contrib.auth.models import User

class Message(models.Model):
    user = models.ForeignKey(User, related_name='messages')
    datetime = models.DateTimeField()
    message = models.CharField(max_length=100)
    def __unicode__(self):
        return self.message

Nothing complicated here. Just a simple model to record the user and the time at which the message is posted.

Then, we will use a setting to determine the interval of time a user should wait before posting another message (You will see later why using a setting is crucial here). In our example we pick 15 minutes. So, add the following to your settings.py file:

MESSAGE_INTERVAL = 60 * 15 # 15 minutes expressed in seconds

Now, on to the view:

from datetime import datetime, timedelta

from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist
from django.http import HttpResponse

from models import Message

def send_message(request):
    message_interval = timedelta(seconds=settings.MESSAGE_INTERVAL)
    now = datetime.now()
        Message.objects.get(user=request.user, datetime__gte=now - message_interval)
        # A message has been posted too recently, so ask the user to wait a bit
        return HttpResponse('Be patient, try again later.')
    except ObjectDoesNotExist:
        # The user waited long enough, so we can create the message
        Message.objects.create(user=request.user, message=request.POST['message'], datetime=now)
        return HttpResponse('Thanks for your message.')

It’s a pretty simple view, but let me just explain what’s going on in there. First, we retrieve the time interval from the settings (it should be 15 minutes, or whatever value you’ve set). Then we check if a message has been posted by the same user during the last 15 minutes. If there is one, return an error notice telling the user he needs to wait; otherwise save the message and return a success notice.

All good! Now, how do we go about testing this? I’ll start by giving out the code:

from time import sleep

from django.test import TestCase
from django.conf import settings
from django.contrib.auth.models import User

from models import Message

message_interval = 1 # seconds
not_long_enough = 0.7 # seconds
long_enough = 1.3 # seconds

class TimeBasedTesting(TestCase):
    def setUp(self):
        self.old_MESSAGE_INTERVAL = settings.MESSAGE_INTERVAL
        settings.MESSAGE_INTERVAL = message_interval
        User.objects.create_user('testuser', 'testuser@example.com', 'testpw')
    def tearDown(self):
        settings.MESSAGE_INTERVAL = self.old_MESSAGE_INTERVAL

    def test_message(self):
        self.client.login(username='testuser', password='testpw')
        # First message
        response = self.client.post('/send_message/', { 'message': 'First try!' })
        self.assertEquals(response.content, 'Thanks for your message.')
        # Wait enough
        response = self.client.post('/send_message/', { 'message': 'Second try!' })
        self.assertEquals(response.content, 'Thanks for your message.')
        # Don't wait enough
        response = self.client.post('/send_message/', { 'message': 'Third try!' })
        self.assertEquals(response.content, 'Be patient, try again later.')
        # Wait enough
        response = self.client.post('/send_message/', { 'message': 'Fourth try!' })
        self.assertEquals(response.content, 'Thanks for your message.')
        # Check what messages have been recorded
        self.assertEquals(str(Message.objects.all()), '[<Message: First try!>, <Message: Second try!>, <Message: Fourth try!>]')

Something we cannot afford is wait for 15 minutes to run each test. This is exactly why we put the waiting time interval into a setting — because then we can override it in our tests. The setUp() method first makes a backup of the setting (stored in old_MESSAGE_INTERVAL) and then overrides it before the tests are run. The tearDown() method eventually restores the setting when the tests are complete so there’s no risk to interfere with other applications or with other tests.

In this example we have set the time interval message_interval to 1 second, and we use two other variables to simulate whether the user waits long enough or not long enough between each posting. Then, to simulate the user waiting we simply use the built-in python function sleep(). It’s as simple as that! This test should take approximately 5 seconds to run.

Finally, there’s no problem if for some reasons you do not want to have the time interval in your settings. In fact, you can put it anywhere you like. The key is to move it to a variable out of your view so it can easily be overridden by your tests.

Hope it helps!

Nov 15

Misconceptions about testing (and what we should do about them)

This post is a reply to Eric Holscher’s call for suggestions to extend and improve the documentation about testing in Django. By no means am I an expert in testing, but I thought I would share some thoughts I’ve had about this. It is more like a dump of ideas that come from my own experience, and hopefully that will contribute slightly to the debate.

The existing documentation is already pretty good but it is still a bit scarce. It is true that there is a gap in this area and that testing doesn’t get the level of attention it deserves (I’m talking, broadly, in the Django community). Developers who regularly write tests already know how good testing is; but there should be more education provided to newcomers so that testing becomes a more common practice. And this would, in turn, benefit the community as a whole. By the way, Eric should be thanked for leading the way in this quest. If you haven’t yet, you should definitely check out the great screencasts and tutorials he’s posted on his blog about this topic.

Personally, I didn’t know anything about testing until I discovered Django and started trying to write patches for it. I found that pretty much every patch had to include tests or it wouldn’t stand a chance to be checked in by the core devs. I originally thought this was a bit of a stubborn ideology, but after diving into it I quickly realised how important and significant tests were. Put simply, Django would NOT be this good (in terms of features) and this reliable (in terms of robustness) if it didn’t have that comprehensive test suite. I encourage anybody to have a look at it; this is one of the most pedagogic ways to understand Django’s inner-workings and best practices.

Anyways, let’s cut the crap. In the remaining of this post I’ll talk about some misconceptions which I believe exist amongst the developers who are new to testing. Then I’ll list some advantages I’ve noticed in writing tests for my Django apps. And finally I’ll give some suggestions to improve the existing documentation, since that’s what this is all about.

Misconceptions about testing

“Testing is hard, tedious and not much fun”

The word “test” sounds a bit like “spinach”, doesn’t it? When you were a kid, didn’t your mom try to make you eat that disgusting green porridge saying that it was good for you? Well, when I first heard about testing that’s exactly the feeling I had: everyone says it’s good for you but no one really feels like giving the first bite. Maybe one of the reasons is that by writing tests you are very likely to find bugs in your code; and finding new bugs is never a good feeling. But, hey! Isn’t it better to find bugs early on rather than when you least expect them?

So yes, testing requires discipline. But it is NOT hard! In fact, if you look at it closely, the code used in tests is usually damn straightforward. If you are scared by tests then you should realise that it is completely irrational. Simply jump into it! It won’t take long before you recognise the benefits and it will give you the most rewarding feeling. There’s even room for being creative when writing tests.

Believe it or not, but writing tests can get addictive. Oh, by the way, I now eat spinach and I like it :)

"Testing adds work load"

"What? I’m already swamped with development work and you want me write tests on top of that?!"

I think this would be a quite natural reaction from someone who’s never tried testing. Well, let’s put it this way: writing tests doesn’t add work load, it just makes you reorganise the way you approach development.

You know, I’m going to tell you a secret: you already do “testing”. Ok, this is not exactly a secret, but what I mean is that you must already be doing some kind of “human” testing: you open your browser, type in the URL, fill out some funky values in your forms (e.g. “test1”, “dfdsfsdf”, “blah”), click “Submit” and then check that it works of fails as expected. You’re doing that, aren’t you?

Writing “machine” tests will cut a lot of that work load. Even better, you can run those tests a zillion times each day for free, if you use a bot! Really, the machine should execute the tests, not you! I see the writing of tests more like a replacement (not complete, but at least partial) of the human tests that we already do on a daily basis.

"Testing takes time and slows development down"

I think there is a common belief that writing tests is time-consuming and that it slows you down in the development of your apps. In fact, if you look at the overall process, I contend that it actually makes you save time. And it does so at many different levels.

It’s an investment: Save now for later

Here’s a silly question: Do you want to find bugs and fix them now while you’re building your app, or do you want to wait for your app to shamefully crash in front of the world? Obviously you don’t want the world to know that your app contains bugs, and testing does help with that.

Like I said before, unless you’re a total genius with a Python compiler implanted in your brain, it’s very much likely that you will discover bugs in your code while writing tests. And that’s in essence the whole beauty of testing. By killing bugs early on you can save a lot of the time which you would otherwise spend debugging when the bugs emerge days or weeks later. Also, for some reasons it seems to be much easier to think of all the edge cases when writing tests than when simply testing the app in the browser.

The other great thing about writing tests during development is that your mind is still very familiar with the code, so you can quickly identify the source of the bugs and fix them. Debugging weeks after means that you have to spend some extra time re-familiarising yourself with the code, before being able to do anything: W.A.S.T.E of time. Of course, writing tests as you go won’t prevent some lurking bugs to appear once in a while, but if you are rigorous enough most bugs will be killed before they’re even born.

Writing tests might seem a daunting task. But as development unfolds, it’s just a matter of adding a few lines of code each time you add a feature or fix a bug. It is a capitalisation process and a solid investment of your time. It’s like you keep putting money in the bank. During times of crisis like we’re in, it’s good to have at least something reliable! :)

Don’t worry about the aesthetics

I don’t know about you, but I often find myself spending 70% of the time fine-tuning small cosmetic details rather than writing actual code that does actual stuff. I can’t help it, but I just hate developing something and testing it in the browser if it looks too wonky and is too naked visually. It just irritates me.

I’m pretty sure this happens to a lot of us. We can easily get distracted by the graphics and presentation of an app while we should be writing proper code instead. Testing really helps with that. When you’re writing tests you naturally focus on getting the sh*t done. I believe this is because the purpose of testing is to make sure that your app WORKS well, not that it LOOKS good. You will do the graphic design work later, or even delegate it to someone else.

So, if you get in the habit of writing tests regularly, you will automatically cut a lot of time that you would otherwise spend in procrastination dealing with the aesthetical and superficial aspects of your app.

Don’t worry about the browser

This follows directly on the previous point: if you don’t get distracted by the aesthetics while writing tests it is simply because you don’t have to use the browser. As I mentioned earlier, testing your app manually by clicking links, typing fake data, etc. can take a lot of time if you add all those tasks up.

Plus, as you gain experience and get better at writing tests you will also get faster at identifying and killing bugs. But could your browser be faster, or could you ever click and type faster? Probably not.

Obviously, you do need sometimes to check that your app works fine in the browser. But writing tests allows you to do that less often. As a challenge, I once tried to develop a simple, but big enough Django app (which processed some forms to add records in the database and also sent and processed confirmation emails), while writing tests in parallel and never using the browser at all. Then, once the core of the code was completed I tried it in the browser for the first time, and it just worked. All that was left to do was fine-tuning the templates and doing all the styling. If I can, I’ll try to write another blog post explaining the process I’ve followed for that one.

Don’t worry about database schema

Something that I think is just great is that the database is re-created each time you run the test suite. This is particularly useful when you start developing a new app, with new models. Indeed, during these early stages the models are likely to change a lot as you develop the app’s specifications. If you do human/manual tests, it requires that you first create your database with syncdb, and then, if you want to make changes you have to do them manually in the database (unless you’re using one of the emerging solutions for that problem, like django-evolution or South, but those are not quite fully functional yet).

If instead you write “machine” tests then you don’t have to worry as much about experimenting with the database schema because the changes will be taken into account straight away the next time the suite is run. You don’t need to do anything more than simply change your model declarations. Believe me, that alone saves a great deal of time and frustration.

Other advantages of testing

In the previous section I have already mentioned a few advantages of testing. That was to specifically respond to misconceptions some developers may have. Here I’ll list a few other advantages that are important in my eyes.

First, as I already said before, testing allows you to anticipate bugs and kill them before they even exist. Another great benefit is that it prevents regression. Typically, regression means that something that used to work doesn’t work anymore because of the insertion of some new code. In other words: you’ve broken your app while you were in fact trying to improve it. There is no worse feeling that breaking some good code on which you’ve spent time and sweat. If you have written tests that check your code is working properly, then those tests will fail and shout at you the next time you break your code, therefore preventing you from releasing a half-broken app. If you find a new bug, then fix it and write tests for it, so you’re sure that you’re done with it: that bloody bug won’t come around again!

Second, writing tests forces you to have a critical look and to think effectively about the strengths and flaws of your app. After all, writing tests is about trying to break your app in any imaginable way. This forward-thinking process also greatly helps in the specification stage. If you’re not enthusiast about drawing diagrams or drafting use cases, and you just want to dive into the code, then testing is for you. Testing makes you think about the big picture because it requires you to ask yourself questions like “What is this piece of code supposed to do and not do?” or “What would be appropriate and inappropriate input and output for this function?” Still, you’re doing that thinking at the same time as you’re programming, so you can satisfy you thirst for coding throughout the process :)

Finally, tests give you confidence. And confidence means both credibility towards your clients and peace of mind for yourself. For all the reasons enumerated above, you know that if you keep up with tests it is less and less likely your app will break. Priceless.

So, what should we do?

So, after all this wordiness, what is it we should do to entice more Django developers into writing tests and improving the (already awesome) Django’s test framework? The existing documentation is already pretty good, but I believe there’s a lack of practical examples with a good balance of best practices and purely technical considerations. Here are a few suggestions.

Strategies for testing

There should be some documentation explaining some good strategies in testing Django apps. It is out of question to give a full lecture on test-driven development, but at least some good hints and start points would be welcome. Here are some examples of strategies I think would be worth expanding on in the documentation:

Project-specific testing

The existing documentation explains quite well how to set up tests for individual apps, but there is very little information on how to set up project-specific tests. You might want, for example, to check that some specific data is systematically added to your database when the project is up and running. Again, I don’t think there is one single way to do this, since the needs may vary from one project to another or from one environment to another. But it would be great if people could share their own tips about it, and maybe some best practices will emerge. I’m planning to write up another blog post soon to explain one particular setup I’ve made on a recent project. Stay tuned for more.

Tips and tricks

There should be a section in the documentation with some tips and tricks that make testing easier. Hey, I’ll start with one! When I’m in development phase I need to run the test suite quite often, and that can take quite a while to process each time. So, what I usually do is use a SQLite database for testing, which is way faster than others I know. Then, once in a while I run the suite on the system that will be used in production (e.g. MySQL).


In conclusion I would say that, even if writing tests as you go might seem a bit time-consuming in the first place, it will save you a great deal of time and frustration in the long run. Django helps lever the foundations of your app quite quickly, and then testing should help consolidate them by closing the gaps and tightening the screws. You will then be waiting for the earthquake with a big smile on your face (sorry for the bad metaphor :) )

To finish, I recommend you to check out the slides from the Django master class that was given by Jeremy Dunck, Jacob Kaplan-Moss, and Simon Willison at OSCON 07. There are some excellent tips on unit testing and other areas of Django. Remember also that the best place to look for examples is Django’s test suite itself. Dozens (hundreds!) of people have contributed to it over the years so you will find different styles of testing in it. You should also have a look at the tests included in the most popular third party apps, which should provide plenty of inspiration.

Phew… this post ended up being much longer than I anticipated. If you’ve read until this point, thank you :) I hope this contributes in some way to the debate, and I hope that at least this will help demystify testing for beginners. I’ll post more on this topic, so stay tuned.

Oct 23

Site-wide login protection (and public views)

A common pattern in websites is when a few pages are protected and require a login to be accessed. The @login_required decorator often comes in handy for these situations. But, another pattern which is quite common is when most of the site is protected, with just a few exceptions of pages that remain public (e.g. frontpage, registration page, etc.). In that case, it can be quite tedious to decorate all of the views with @login_required, and it can be easy to forget to decorate some of them.

So, I came up with a simple system which by default protects every view and then lets you explicitly tell which views should be public. This makes things both easier and less error-prone.


The core of that system is contained in the following middleware code:

import re
from django.conf import settings
from django.contrib.auth.decorators import login_required
from path.to.your.decorators import PublicView

class LoginRequiredMiddleware(object):    
    def __init__(self):
        self.public_patterns = []
        self.public_views = []
        if hasattr(settings, 'PUBLIC_VIEWS'):
            for view_path in settings.PUBLIC_VIEWS:
                view = self.get_view(view_path)
        if hasattr(settings, 'PUBLIC_PATHS'):
            for public_path in settings.PUBLIC_PATHS:

    def get_view(self, view_path):
        i = view_path.rfind('.')
        module_path, view_name = view_path[:i], view_path[i+1:]
        module = __import__(module_path, globals(), locals(), [view_name])
        return getattr(module, view_name)

    def matches_public_view(self, view):
        if self.public_views:
            for public_view in self.public_views:
                if view == public_view:
                    return True
        return False

    def matches_public_path(self, path):
        if self.public_patterns:
            for pattern in self.public_patterns:
                if pattern.match(path) is not None:
                    return True
        return False

    def process_view(self, request, view_func, view_args, view_kwargs):
        if request.user.is_authenticated() or isinstance(view_func, PublicView) or self.matches_public_path(request.path) or self.matches_public_view(view_func):
            return None
            return login_required(view_func)(request, *view_args, **view_kwargs)

To install this middleware, simply copy and paste the above code anywhere in your project, for example in a file called middleware.py. Then, update the MIDDLEWARE_CLASSES setting in your project’s settings file:


You’ll notice that, at the top of the middleware code above, there is an import of the PublicView class. You need to update that import path after having copied/pasted the following snippet anywhere in your project, for example in a decorators.py file:

    from functools import update_wrapper
except ImportError:
    from django.utils.functional import update_wrapper  # Python 2.3, 2.4 fallback.

from django.contrib.auth.decorators import _CheckLogin

def login_not_required(view_func):
    Decorator which marks the given view as public (no login required).
    return PublicView(view_func)

class PublicView(object):
    Forces a view to be public (no login required).
    def __init__(self, view_func):
        if isinstance(view_func, _CheckLogin):
            self.view_func = view_func.view_func
            self.view_func = view_func
        update_wrapper(self, view_func)
    def __get__(self, obj, cls=None):
        view_func = self.view_func.__get__(obj, cls)
        return _PublicView(view_func)
    def __call__(self, request, *args, **kwargs):
        return self.view_func(request, *args, **kwargs)

The above code contains a new decorator (@login_not_required) which will be explained in detail in a moment.

Declaring public views

At this point, all of your views will require you to log in, including the login page itself. So, we now need to specify the few views that should be public. There are three different ways at your disposal: using a special decorator, listing the public views, or listing the public URL paths.

Using a Decorator

Thanks to the new @login_not_required you can explicitly force a view to be public. Here’s an example:

from path.to.your.decorators import login_not_required

def frontpage(request):

In this case, the frontpage view will be properly displayed even if you’re not logged in.

Listing public views

If you don’t have direct access to modify a view’s code (e.g., it’s in a third-party application), you still can force that view to be public by adding it to the new PUBLIC_VIEWS setting in your settings file. Here’s an example if you’re using the django.contrib.auth system and the django-registration application:


Listing URL public paths

The third and last way is to directly specify the URL paths (as regular expressions) for the pages you want to be public. This can be useful, for example, if a page is rendered by a generic view. It is also useful if you are serving your media files statically via Django (only recommended in development mode). For that, you need to add the PUBLIC_PATHS setting in your settings file. Here’s an example:

    '^%s' % MEDIA_URL,
    '^/accounts/register/complete/$', # Uses the 'direct_to_template' generic view

That’s it! By using this technique your site will be protected effectively and it will be easy to maintain. I hope it helps! Any comment or remark is very welcome ;)