Wouldn’t it be nice to be able to deploy a Django webapp on your own server just based on a git repo URL? I set out to try this using the neat-looking family of “application” chef cookbooks. These can be used to setup nginx, gunicorn and a Django webapp. With a few more chef recipe extensions we can add the necessary MySQL (or other) database bits, and we’ll have a pretty comprehensive deployment.

The main aim of this post is to describe how go as quickly as possible from zero to a fully chef-deployed Django app. This is not a substitute for learning chef of course.

I explicitly don’t want to use chef enterprise (or even chef-server) for this exercise, to keep things simple and to focus on the necessities. Instead, we’ll be adding the -z, or –local-mode flag to almost every command, to run things in “local”, or “chef-zero” mode. The app used here will be based on the Django 1.7 tutorial, you can get the code for that from github.com/m01/django_tutorial. The chef cookbook I wrote is available at github.com/m01/chef-django_tutorial. There’s also a TL;DR version of this post.

Setting up a “chef-repo”

The chef-repo will contain all the cookbooks and other data chef needs for provisioning. This is the one-off task you’ll need to do, once you’ve got this you can spin up more nodes and deploy your webapp on them with little effort.

Prerequisites

You’ll need git setup, including setting up the user.name and user.email parameters. This seems to be required by the ‘knife cookbook site install’ utility. You’ll also need to ensure EDITOR is set to something you’re comfortable with. For the lazy with a fresh test VM:

sudo apt-get update && sudo apt-get install git
export EDITOR=sensible-editor
git config --global user.name "ubuntu"
git config --global user.email "ubuntu@example.com"

Next, install the chef-client using the instructions on the official website, e.g. for Ubuntu 14.04:

curl -L https://www.opscode.com/chef/install.sh | sudo bash

Building your chef-repo

First, we need a skeleton chef-repo, which will house our cookbooks:

cd ~
git clone git://github.com/opscode/chef-repo.git
cd chef-repo
# avoid accidents
git remote rm origin

Install cookbook dependencies (for later)

You could do this using Berkshelf, librarian or other tools, but I’m trying to keep the dependencies and setup cost to a minimum. At the time of writing, the latest version of the application cookbook is too new for the application_python cookbook, so we have to use the older version 3.0.0. This needs to be installed last to avoid the broken dependency resolution mechanism updating it while installing application_python or application_nginx.

cd cookbooks
knife cookbook site install -z application_python
knife cookbook site install -z application_nginx
knife cookbook site install -z application 3.0.0
knife cookbook site install -z database

Your Django application cookbook

Download mine (quick & easy)

If you just want to play, add mine to your cookbooks and continue to the next step. You can git clone it as shown below, or if you want to do it “properly”, look into berkshelf, librarian or knife-github-cookbooks,

git clone https://github.com/m01/chef-django_tutorial django_tutorial
# we're done with cookbooks for now
cd ..

Create your own

Otherwise, use knife to create a new cookbook:

knife cookbook create -z django_tutorial

Next, you’ll want to edit the django_tutorial/recipes/default.rb file. Whenever you add new dependencies, add them to django_tutorial/metadata.rb. The general idea of what you need to do is, on the chef recipe side:

1. Setup any directories, user accounts etc (as required)
2. Setup database stuff (unless you’re planning on sticking with sqlite)
3. Setup the general web application, including the git URL that your code should come from.
4. Add the django specific bits, see application_python. This will setup the django app, but not run it.
5. Setup gunicorn as explained in the application_python readme. This will result in your app running (but on port 8000 or whatever).
6. Setup nginx (using application_nginx) for proxying requests from port 80 to your gunicorn instance, and for serving your static files.

If you get stuck, feel free to look at my recipe and use that as an example; it’s not that long. I found I also needed to make some small changes to my django application:

• Ensure the Django application has a requirements.txt file
• Ensure gunicorn and the database library are featured in requirements.txt
• Ensure the STATIC_ROOT setting is os.path.join(BASE_DIR, “static”) to make your life easy
• Ensure you can use local_settings.py to override settings in settings.py

Accessing private repos

There are probably lots of ways to do this, but one quick & easy mechanism involves generating a personal access token on Github, and using the HTTPS URL for cloning the repo like this:

application 'foo_app' do
  ...
  repository 'https://<github username>:<personal access token>@github.com/your/repo.git'
  ...
end

Creating an application server “role”

The application_nginx cookbook comes with some clever logic in it to automatically populate the nginx loadbalancer configuration with the IP addresses and ports of the application servers running the Django webapp. This is probably great for a real-world deployment, but it does make our life a little bit more complicated at this stage. We need to define a “role” for the application server, otherwise the loadbalancer configuration will just contain an empty list of nodes to proxy requests to. The application server role name is by default <appname>_application_server, so let’s create that role:

knife role create -z django_tutorial_application_server

This will fire up an editor. You’ll want to add your recipe to the run_list, like this:

...
 "run_list": [
   "recipe[django_tutorial]"
 ]
...

You could at this stage create another role for the loadbalancer (see the application_nginx docs for details) and make sure that only VMs with that role install the nginx bits.

Run the chef-client!

Now we can run the chef-client in local mode to deploy the application.

sudo chef-client --local-mode -o "role[django_tutorial_application_server]"

If all goes well, that should just run without any errors. Now fire up your browser, and point it at your server’s IP address. You should see a familiar debug message telling you which URLs might be worthwhile visiting (/admin, /polls). The database will only contain whatever “python manage.py syncdb” would put in there. If you want to create a django superuser, you’ll need to source the virtualenv’s activate file, and then create the superuser manually (or write some chef code to do this?):

source /srv/webapps/django_tutorial/shared/env/bin/activate
cd /srv/webapps/django_tutorial/current
python manage.py createsuperuser

If that didn’t work…

Then here are a few hints:

• Add “-l debug” to the chef client command. This will generate very verbose output, but at least you can see the command output of e.g. python manage.py syncdb, which will help you debug issues (it might tell you of missing modules, or incorrect configs etc).

Deploying your Django application on the next node

This is the bit where having a chef-server would be handy (see below for details). The knife utility has a neat bootstrap function, which you should be able to just give the ssh login details, and with 1 command your new box should be provisioned.

Minimalist approach

This method involves sending a tarball of the whole chef-repo to the other node, installing chef on the other node, and then running the chef-client in local mode, just like we did above. This is a bit cumbersome, but it does not require any additional one-off setup on the box you’re working on.

Tar up the chef-repo with: cd ~; tar czvf chef-repo.tar.gz chef-repo

# use 'echo PASSWORD | sudo -S' to get around sudo password prompt.

# 1. install chef
ssh $YOURDESTINY -- "echo $PASSWORD | sudo -S echo 'sudo ok' &&  curl -L https://www.opscode.com/chef/install.sh | sudo bash"

# 2. scp over a tarball with everything chef cares about
scp chef-repo.tar.gz $YOURDESTINY:

# 3. Extract tarball, cd into chef-repo and run the chef-client
ssh $YOURDESTINY -- "echo $PASSWORD | sudo -S echo 'sudo ok' && tar xzf chef-repo.tar.gz && cd chef-repo && sudo chef-client --local-mode -o 'role[django_tutorial_application_server]'"

# done.

chef-zero

[Updated September 17, 2014]

With this method, you basically run a local, minimalist chef-server. You’ll still need to “upload” your chef repo to the local chef-zero server (like you would with a “real” chef server), but at least you don’t need to move tarballs around, and you keep your code in one place. This reduces the risk of having outdated code scattered all over the place. I see this as the “testing version” of using a proper chef server (see below for that).

I initially couldn’t get this method to work at first, but thanks to Christine Draper‘s comment I found a way:

One-time setup

1. Set up a fake pem key as explained at the beginning of this post (grab the wget command)
2. Set up .chef/knife.rb as explained in the “Multiple nodes with a single chef-zero” section of this post.
3. mkdir -p $HOME/.chef/local-mode-cache/clients

Bootstrap second node

# in one terminal run:
knife serve --chef-zero-host <IP address the other node can reach>

# in another terminal:
# "upload" your repo to the chef-zero server's local cache:
cd ~/chef-repo
knife upload --chef-repo-path `pwd` .

# and then bootstrap the new node

knife bootstrap $YOURDESTINY --ssh-user ubuntu \
    --sudo --ssh-password $PASSWORD \
    -N newnodename \
    --run-list 'role[django_tutorial_application_server]'

chef-server/chef-enterprise

If you’ve got a chef-server setup, or a chef enterprise account (at the time of writing the latter is free for up to 5 nodes), then this is probably the method of choice.

First, ensure the machine you’re working from is setup to use the chef server. Then, upload/update your recipes & role definitions:

cd ~/chef-repo; knife upload .

And bootstrap a node with something like:

knife bootstrap $DEST_IP \
    --ssh-user ubuntu --sudo --ssh-password ubuntu \
    -N mynewnode \
    --run-list 'role[django_tutorial_application_server]'

I believe you can even use chef to spawn the node on EC2 straight away for you, but I just wanted to show the basics here.