Posts de ‘Francisco Souza’

[Francisco Souza] Speaking at OSCON 2014

Saturday, April 12th, 2014

Wow, one year without any posts! But I’m trying to get back…

This is a very short post, just to tell everybody that this year, I will have the opportunity to speak at OSCON 2014. I’m speaking about tsuru, and check more details of the talk in the tsuru blog.

[Francisco Souza] Speaking at OSCON 2014

Saturday, April 12th, 2014

Wow, one year without any posts! But I’m trying to get back…

This is a very short post, just to tell everybody that this year, I will have the opportunity to speak at OSCON 2014. I’m speaking about tsuru, and check more details of the talk in the tsuru blog.

[Francisco Souza] Try out Tsuru: announcing limited preview

Thursday, April 11th, 2013

A few days ago, Tsuru got some attention in the news. After reading about Tsuru, and seeing some of its capabilities, people started asking for a way to try Tsuru. Well, your claims were attended! We’re preparing a public cloud that will be freely available for beta testers.

TL;DR: go to tsuru.io/try, signup for beta testing and get ready to start deploying Python, Ruby, Go and Java applications in the cloud.

What is Tsuru?

Tsuru is an open source platform as a service that allows developers to automatically deploy and manage web applications written in many different platforms (like Python, Ruby and Go). It aims to provide a solution for cloud computing platforms that is extensible, flexible and component based.

You can run your own public or private cloud using Tsuru. Or you can try it in the public cloud that Globo.com is building.

What is Tsuru public cloud? What does “beta availability” means?

Tsuru public cloud will be a public, freely available, installation of Tsuru, provided by Globo.com. “Beta availability” means that it will not be available for the general Internet public.

People will need to subscribe for the beta testing and wait for the confirmation, so they can start deploying web applications on Tsuru public cloud.

Which development platforms are going to be available?

Tsuru already supports Ruby, Python, Java and Go, so it is very likely that these platforms will be available for all beta users.

It’s important to notice that adding new platforms to Tsuru is a straightforward task: each development platform is based on Juju Charms, so one can adapt charms available at Charm Store and send a patch.

How limited is it going to be?

We don’t know what’s the proper answer for this question yet, but don’t worry about numbers now. There will be some kind of per-user quota, but it has not been defined yet.

People interested in running applications in the Tsuru public cloud that get to use the beta version will have access a functional environment where they will be able to deploy at least one web application.

When will it be available?

We’re working hard to make it available as soon as possible, and you can help us get it done! If you want to contribute, please take a look at Tsuru repository, chose an issue, discuss your solution and send your patches. We are going to be very happy helping you out.

What if I don’t want to wait?

If you want an unlimited, fully manageable and customized installation of Tsuru, you can have it today. Check out Tsuru’s documentation and, in case of doubts, don’t hesitate in contacting the newborn Tsuru community.

[Francisco Souza] Try out Tsuru: announcing limited preview

Thursday, April 11th, 2013

A few days ago, Tsuru got some attention in the news. After reading about Tsuru, and seeing some of its capabilities, people started asking for a way to try Tsuru. Well, your claims were attended! We’re preparing a public cloud that will be freely available for beta testers.

TL;DR: go to tsuru.io/try, signup for beta testing and get ready to start deploying Python, Ruby, Go and Java applications in the cloud.

What is Tsuru?

Tsuru is an open source platform as a service that allows developers to automatically deploy and manage web applications written in many different platforms (like Python, Ruby and Go). It aims to provide a solution for cloud computing platforms that is extensible, flexible and component based.

You can run your own public or private cloud using Tsuru. Or you can try it in the public cloud that Globo.com is building.

What is Tsuru public cloud? What does “beta availability” means?

Tsuru public cloud will be a public, freely available, installation of Tsuru, provided by Globo.com. “Beta availability” means that it will not be available for the general Internet public.

People will need to subscribe for the beta testing and wait for the confirmation, so they can start deploying web applications on Tsuru public cloud.

Which development platforms are going to be available?

Tsuru already supports Ruby, Python, Java and Go, so it is very likely that these platforms will be available for all beta users.

It’s important to notice that adding new platforms to Tsuru is a straightforward task: each development platform is based on Juju Charms, so one can adapt charms available at Charm Store and send a patch.

How limited is it going to be?

We don’t know what’s the proper answer for this question yet, but don’t worry about numbers now. There will be some kind of per-user quota, but it has not been defined yet.

People interested in running applications in the Tsuru public cloud that get to use the beta version will have access a functional environment where they will be able to deploy at least one web application.

When will it be available?

We’re working hard to make it available as soon as possible, and you can help us get it done! If you want to contribute, please take a look at Tsuru repository, chose an issue, discuss your solution and send your patches. We are going to be very happy helping you out.

What if I don’t want to wait?

If you want an unlimited, fully manageable and customized installation of Tsuru, you can have it today. Check out Tsuru’s documentation and, in case of doubts, don’t hesitate in contacting the newborn Tsuru community.

[Francisco Souza] Using Juju to orchestrate CentOS-based cloud services

Saturday, July 28th, 2012

Earlier this week I had the opportunity to meet Kyle MacDonald, head of Ubuntu Cloud, during FISL, and he was surprised when we told him we are using Juju with CentOS at Globo.com. Then I decided to write this post explaining how we came up with a patched version of Juju that allows us to have CentOS clouds managed by Juju.

For those who doesn’t know Juju, it’s a service orchestration tool, focused on devops “development method”. It allows you to deploy services on clouds, local machine and even bare metal machines (using Canonical’s MAAS).

It’s based on charms and very straightforward to use. Here is a very basic set of commands with which you can deploy a Wordpress related to a MySQL service:

% juju bootstrap% juju deploy mysql% juju deploy wordpress% juju add-relation wordpress mysql% juju expose wordpress

These commands will boostrap the environment, setting up a bootstrap machine which will manage your services; deploy mysql and wordpress instances; add a relation between them; and expose the wordpress port. The voilà, we have a wordpress deployed, and ready to serve our posts. Amazing, huh?

But there is an issue: although you can install the juju command line tool in almost any OS (including Mac OS), right now you are able do deploy only Ubuntu-based services (you must use an Ubuntu instance or container).

To change this behavior, and enable Juju to spawn CentOS instances (and containers, if you have a CentOS lxc template), we need to develop and apply some changes to Juju and cloud-init. Juju uses cloud-init to spawn machines with proper dependencies set up, and it’s based on modules. All we need to do, is add a module able to install rpm packages using yum.

cloud-init modules are Python modules that starts with cc_ and implement a `handle` function (for example, a module called “yum_packages” would be written to a file called cc_yum_packages.py). So, here is the code for the module yum_packages:

import subprocessimport traceback

from cloudinit import CloudConfig, util

frequency = CloudConfig.per_instance

def yum_install(packages):    cmd = ["yum", "--quiet", "--assumeyes", "install"]    cmd.extend(packages)    subprocess.check_call(cmd)

def handle(_name, cfg, _cloud, log, args):    pkglist = util.get_cfg_option_list_or_str(cfg, "packages", [])

    if pkglist:        try:            yum_install(pkglist)        except subprocess.CalledProcessError:            log.warn("Failed to install yum packages: %s" % pkglist)            log.debug(traceback.format_exc())            raise

    return True

The module installs all packages listed in cloud-init yaml file. If we want to install `emacs-nox` package, we would write this yaml file and use it as user data in the instance:

#cloud-configmodules: - yum_packagespackages: [emacs-nox]

cloud-init already works on Fedora, with Python 2.7, but to work on CentOS 6, with Python 2.6, it needs a patch:

--- cloudinit/util.py 2012-05-22 12:18:21.000000000 -0300+++ cloudinit/util.py 2012-05-31 12:44:24.000000000 -0300@@ -227,7 +227,7 @@         stderr=subprocess.PIPE, stdin=subprocess.PIPE)     out, err = sp.communicate(input_)     if sp.returncode is not 0:-        raise subprocess.CalledProcessError(sp.returncode, args, (out, err))+        raise subprocess.CalledProcessError(sp.returncode, args)     return(out, err)

I’ve packet up this module and this patch in a RPM package that must be pre-installed in the lxc template and AMI images. Now, we need to change Juju in order to make it use the yum_packages module, and include all RPM packages that we need to install when the machine borns.

Is Juju, there is a class that is responsible for building and rendering the YAML file used by cloud-init. We can extend it and change only two methods: _collect_packages, that returns the list of packages that will be installed in the machine after it is spawned; and render that returns the file itself. Here is our CentOSCloudInit class (within the patch):

diff -u juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py juju-0.5-bzr531/juju/providers/common/cloudinit.py--- juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py 2012-05-31 15:42:17.480769486 -0300+++ juju-0.5-bzr531/juju/providers/common/cloudinit.py 2012-05-31 15:55:13.342884919 -0300@@ -324,3 +324,32 @@             "machine-id": self._machine_id,             "juju-provider-type": self._provider_type,             "juju-zookeeper-hosts": self._join_zookeeper_hosts()}+++class CentOSCloudInit(CloudInit):++    def _collect_packages(self):+        packages = [+            "bzr", "byobu", "tmux", "python-setuptools", "python-twisted",+            "python-txaws", "python-zookeeper", "python-devel", "juju"]+        if self._zookeeper:+            packages.extend([+                "zookeeper", "libzookeeper", "libzookeeper-devel"])+        return packages++    def render(self):+        """Get content for a cloud-init file with appropriate specifications.++        :rtype: str++        :raises: :exc:`juju.errors.CloudInitError` if there isn't enough+            information to create a useful cloud-init.+        """+        self._validate()+        return format_cloud_init(+            self._ssh_keys,+            packages=self._collect_packages(),+            repositories=self._collect_repositories(),+            scripts=self._collect_scripts(),+            data=self._collect_machine_data(),+            modules=["ssh", "yum_packages", "runcmd"])

The other change we need is in the format_cloud_init function, in order to make it recognize the modules parameter that we used above, and tell cloud-init to not run apt-get (update nor upgrade). Here is the patch:

diff -ur juju-0.5-bzr531.orig/juju/providers/common/utils.py juju-0.5-bzr531/juju/providers/common/utils.py--- juju-0.5-bzr531.orig/juju/providers/common/utils.py 2012-05-31 15:42:17.480769486 -0300+++ juju-0.5-bzr531/juju/providers/common/utils.py 2012-05-31 15:44:06.605014021 -0300@@ -85,7 +85,7 @@

 def format_cloud_init(-    authorized_keys, packages=(), repositories=None, scripts=None, data=None):+    authorized_keys, packages=(), repositories=None, scripts=None, data=None, modules=None):     """Format a user-data cloud-init file.

     This will enable package installation, and ssh access, and script@@ -117,8 +117,8 @@         structure.     """     cloud_config = {-        "apt-update": True,-        "apt-upgrade": True,+        "apt-update": False,+        "apt-upgrade": False,         "ssh_authorized_keys": authorized_keys,         "packages": [],         "output": {"all": "| tee -a /var/log/cloud-init-output.log"}}@@ -136,6 +136,11 @@     if scripts:         cloud_config["runcmd"] = scripts

+    if modules:+        cloud_config["modules"] = modules+     output = safe_dump(cloud_config)     output = "#cloud-config\n%s" % (output)     return output

This patch is also packed up within juju-centos-6 repository, which provides sources for building RPM packages for juju, and also some pre-built RPM packages.

Now just build an AMI image with cloudinit pre-installed, configure your juju environments.yaml file to use this image in the environment and you are ready to deploy cloud services on CentOS machines using Juju!

Some caveats:

  • Juju needs a user called ubuntu to interact with its machines, so you will need to create this user in your CentOS AMI/template.
  • You need to host all RPM packages for juju, cloud-init and following dependencies in some yum repository (I haven’t submitted them to any public repository):
  • With this patched Juju, you will have a pure-centos cloud. It does not enable you to have multiple OSes in the same environment.

It’s important to notice that we are going to put some effort to make the Go version of juju born supporting multiple OSes, ideally through an interface that makes it extensible to any other OS, not Ubuntu and CentOS only.

[Francisco Souza] Using Juju to orchestrate CentOS-based cloud services

Saturday, July 28th, 2012

Earlier this week I had the opportunity to meet Kyle MacDonald, head of Ubuntu Cloud, during FISL, and he was surprised when we told him we are using Juju with CentOS at Globo.com. Then I decided to write this post explaining how we came up with a patched version of Juju that allows us to have CentOS clouds managed by Juju.

For those who doesn’t know Juju, it’s a service orchestration tool, focused on devops “development method”. It allows you to deploy services on clouds, local machine and even bare metal machines (using Canonical’s MAAS).

It’s based on charms and very straightforward to use. Here is a very basic set of commands with which you can deploy a Wordpress related to a MySQL service:

% juju bootstrap% juju deploy mysql% juju deploy wordpress% juju add-relation wordpress mysql% juju expose wordpress

These commands will boostrap the environment, setting up a bootstrap machine which will manage your services; deploy mysql and wordpress instances; add a relation between them; and expose the wordpress port. The voilà, we have a wordpress deployed, and ready to serve our posts. Amazing, huh?

But there is an issue: although you can install the juju command line tool in almost any OS (including Mac OS), right now you are able do deploy only Ubuntu-based services (you must use an Ubuntu instance or container).

To change this behavior, and enable Juju to spawn CentOS instances (and containers, if you have a CentOS lxc template), we need to develop and apply some changes to Juju and cloud-init. Juju uses cloud-init to spawn machines with proper dependencies set up, and it’s based on modules. All we need to do, is add a module able to install rpm packages using yum.

cloud-init modules are Python modules that starts with cc_ and implement a `handle` function (for example, a module called “yum_packages” would be written to a file called cc_yum_packages.py). So, here is the code for the module yum_packages:

import subprocessimport traceback

from cloudinit import CloudConfig, util

frequency = CloudConfig.per_instance

def yum_install(packages):    cmd = ["yum", "--quiet", "--assumeyes", "install"]    cmd.extend(packages)    subprocess.check_call(cmd)

def handle(_name, cfg, _cloud, log, args):    pkglist = util.get_cfg_option_list_or_str(cfg, "packages", [])

    if pkglist:        try:            yum_install(pkglist)        except subprocess.CalledProcessError:            log.warn("Failed to install yum packages: %s" % pkglist)            log.debug(traceback.format_exc())            raise

    return True

The module installs all packages listed in cloud-init yaml file. If we want to install `emacs-nox` package, we would write this yaml file and use it as user data in the instance:

#cloud-configmodules: - yum_packagespackages: [emacs-nox]

cloud-init already works on Fedora, with Python 2.7, but to work on CentOS 6, with Python 2.6, it needs a patch:

--- cloudinit/util.py 2012-05-22 12:18:21.000000000 -0300+++ cloudinit/util.py 2012-05-31 12:44:24.000000000 -0300@@ -227,7 +227,7 @@         stderr=subprocess.PIPE, stdin=subprocess.PIPE)     out, err = sp.communicate(input_)     if sp.returncode is not 0:-        raise subprocess.CalledProcessError(sp.returncode, args, (out, err))+        raise subprocess.CalledProcessError(sp.returncode, args)     return(out, err)

I’ve packet up this module and this patch in a RPM package that must be pre-installed in the lxc template and AMI images. Now, we need to change Juju in order to make it use the yum_packages module, and include all RPM packages that we need to install when the machine borns.

Is Juju, there is a class that is responsible for building and rendering the YAML file used by cloud-init. We can extend it and change only two methods: _collect_packages, that returns the list of packages that will be installed in the machine after it is spawned; and render that returns the file itself. Here is our CentOSCloudInit class (within the patch):

diff -u juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py juju-0.5-bzr531/juju/providers/common/cloudinit.py--- juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py 2012-05-31 15:42:17.480769486 -0300+++ juju-0.5-bzr531/juju/providers/common/cloudinit.py 2012-05-31 15:55:13.342884919 -0300@@ -324,3 +324,32 @@             "machine-id": self._machine_id,             "juju-provider-type": self._provider_type,             "juju-zookeeper-hosts": self._join_zookeeper_hosts()}+++class CentOSCloudInit(CloudInit):++    def _collect_packages(self):+        packages = [+            "bzr", "byobu", "tmux", "python-setuptools", "python-twisted",+            "python-txaws", "python-zookeeper", "python-devel", "juju"]+        if self._zookeeper:+            packages.extend([+                "zookeeper", "libzookeeper", "libzookeeper-devel"])+        return packages++    def render(self):+        """Get content for a cloud-init file with appropriate specifications.++        :rtype: str++        :raises: :exc:`juju.errors.CloudInitError` if there isn't enough+            information to create a useful cloud-init.+        """+        self._validate()+        return format_cloud_init(+            self._ssh_keys,+            packages=self._collect_packages(),+            repositories=self._collect_repositories(),+            scripts=self._collect_scripts(),+            data=self._collect_machine_data(),+            modules=["ssh", "yum_packages", "runcmd"])

The other change we need is in the format_cloud_init function, in order to make it recognize the modules parameter that we used above, and tell cloud-init to not run apt-get (update nor upgrade). Here is the patch:

diff -ur juju-0.5-bzr531.orig/juju/providers/common/utils.py juju-0.5-bzr531/juju/providers/common/utils.py--- juju-0.5-bzr531.orig/juju/providers/common/utils.py 2012-05-31 15:42:17.480769486 -0300+++ juju-0.5-bzr531/juju/providers/common/utils.py 2012-05-31 15:44:06.605014021 -0300@@ -85,7 +85,7 @@

 def format_cloud_init(-    authorized_keys, packages=(), repositories=None, scripts=None, data=None):+    authorized_keys, packages=(), repositories=None, scripts=None, data=None, modules=None):     """Format a user-data cloud-init file.

     This will enable package installation, and ssh access, and script@@ -117,8 +117,8 @@         structure.     """     cloud_config = {-        "apt-update": True,-        "apt-upgrade": True,+        "apt-update": False,+        "apt-upgrade": False,         "ssh_authorized_keys": authorized_keys,         "packages": [],         "output": {"all": "| tee -a /var/log/cloud-init-output.log"}}@@ -136,6 +136,11 @@     if scripts:         cloud_config["runcmd"] = scripts

+    if modules:+        cloud_config["modules"] = modules+     output = safe_dump(cloud_config)     output = "#cloud-config\n%s" % (output)     return output

This patch is also packed up within juju-centos-6 repository, which provides sources for building RPM packages for juju, and also some pre-built RPM packages.

Now just build an AMI image with cloudinit pre-installed, configure your juju environments.yaml file to use this image in the environment and you are ready to deploy cloud services on CentOS machines using Juju!

Some caveats:

  • Juju needs a user called ubuntu to interact with its machines, so you will need to create this user in your CentOS AMI/template.
  • You need to host all RPM packages for juju, cloud-init and following dependencies in some yum repository (I haven’t submitted them to any public repository):
  • With this patched Juju, you will have a pure-centos cloud. It does not enable you to have multiple OSes in the same environment.

It’s important to notice that we are going to put some effort to make the Go version of juju born supporting multiple OSes, ideally through an interface that makes it extensible to any other OS, not Ubuntu and CentOS only.

[Francisco Souza] Setting up a Django production environment: compiling and configuring nginx

Monday, November 7th, 2011

Here is another series of posts: now I’m going to write about setting up a Django production environment using nginx and Green Unicorn in a virtual environment. The subject in this first post is nginx, which is my favorite web server.

[Francisco Souza] Setting up a Django production environment: compiling and configuring nginx

Sunday, November 6th, 2011

Here is another series of posts: now I’m going to write about setting up a Django production environment using nginx and Green Unicorn in a virtual environment. The subject in this first post is nginx, which is my favorite web server.

This post explains how to install nginx from sources, compiling it (on Linux). You might want to use apt, zif, yum or ports, but I prefer building from sources. So, to build from sources, make sure you have all development dependencies (C headers, including the PCRE library headers, nginx rewrite module uses it). If you want to build nginx with SSL support, keep in mind that you will need the libssl headers too.

Build nginx from source is a straightforward process: all you need to do is download it from the official site and build with some simple options. In our setup, we’re going to install nginx under /opt/nginx, and use it with the nginx system user. So, let’s download and extract the latest stable version (1.0.9) from nginx website:

% curl -O http://nginx.org/download/nginx-1.0.9.tar.gz% tar -xzf nginx-1.0.9.tar.gz

Once you have extracted it, just configure, compile and install:

% ./configure --prefix=/opt/nginx --user=nginx --group=nginx% make% [sudo] make install

As you can see, we provided the /opt/nginx to configure, make sure the /opt directory exists. Also, make sure that there is a user and a group called nginx, if they don’t exist, add them:

% [sudo] adduser --system --no-create-home --disabled-login --disabled-password --group nginx

After that, you can start nginx using the command line below:

% [sudo] /opt/nginx/sbin/nginx

Linode provides an init script that uses start-stop-daemon, you might want to use it.

nginx configuration

nginx comes with a default nginx.conf file, let’s change it to reflect the following configuration requirements:

  • nginx should start workers with the nginx user
  • nginx should have two worker processes
  • the PID should be stored in the /opt/nginx/log/nginx.pid file
  • nginx must have an access log in /opt/nginx/logs/access.log
  • the configuration for the Django project we’re going to develop should be versioned with the entire code, so it must be included in the nginx.conf file (assume that the library project is in the directory /opt/projects).

So here is the nginx.conf for the requirements above:

user  nginx;worker_processes  2;

pid logs/nginx.pid;

events {    worker_connections  1024;}

http {    include       mime.types;    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                     '$status $body_bytes_sent "$http_referer" '                     '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile           on;    keepalive_timeout  65;

    include /opt/projects/showcase/nginx.conf;}

Now we just need to write the configuration for our Django project. I’m using an old sample project written while I was working at Giran: the name is lojas giranianas, a nonsense portuguese joke with a famous brazilian store. It’s an unfinished showcase of products, it’s like an e-commerce project, but it can’t sell, so it’s just a product catalog. The code is available at Github. The nginx.conf file for the repository is here:

server {    listen 80;    server_name localhost;

    charset utf-8;

    location / {        proxy_set_header    X-Real-IP   $remote_addr;        proxy_set_header    Host        $http_host;        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://localhost:8000;    }

    location /static {        root /opt/projects/showcase/;        expires 1d;    }}

The server listens on port 80, responds for the localhost hostname (read more about the Host header). The location /static directive says that nginx will serve the static files of the project. It also includes an expires directive for caching control. The location / directive makes a proxy_pass, forwarding all requisitions to an upstream server listening on port 8000, this server is the subject of the next post of the series: the Green Unicorn (gunicorn) server.

Not only the HTTP request itself is forwarded to the gunicorn server, but also some headers, that helps to properly deal with the request:

  • X-Real-IP: forwards the remote address to the upstream server, so it can know the real IP of the user. When nginx forwards the request to gunicorn, without this header, all gunicorn will know is that there is a request coming from localhost (or wherever the nginx server is), the remote address is always the IP address of the machine where nginx is running (who actually make the request to gunicorn)
  • Host: the Host header is forwarded so gunicorn can treat different requests for different hosts. Without this header, it will be impossible to Gunicorn to have these constraints
  • X-Forwarded-For: also known as XFF, this header provide more precise information about the real IP who makes the request. Imagine there are 10 proxies between the user machine and your webserver, the XFF header will all these proxies comma separated. In order to not turn a proxy into an anonymizer, it’s a good practice to always forward this header.

So that is it, in the next post we are going to install and run gunicorn. In other posts, we’ll see how to make automated deploys using Fabric, and some tricks on caching (using the proxy_cache directive and integrating Django, nginx and memcached).

See you in next posts.

[Francisco Souza] Setting up a Django production environment: compiling and configuring nginx

Sunday, November 6th, 2011

Here is another series of posts: now I’m going to write about setting up a Django production environment using nginx and Green Unicorn in a virtual environment. The subject in this first post is nginx, which is my favorite web server.

This post explains how to install nginx from sources, compiling it (on Linux). You might want to use apt, zif, yum or ports, but I prefer building from sources. So, to build from sources, make sure you have all development dependencies (C headers, including the PCRE library headers, nginx rewrite module uses it). If you want to build nginx with SSL support, keep in mind that you will need the libssl headers too.

Build nginx from source is a straightforward process: all you need to do is download it from the official site and build with some simple options. In our setup, we’re going to install nginx under /opt/nginx, and use it with the nginx system user. So, let’s download and extract the latest stable version (1.0.9) from nginx website:

% curl -O http://nginx.org/download/nginx-1.0.9.tar.gz% tar -xzf nginx-1.0.9.tar.gz

Once you have extracted it, just configure, compile and install:

% ./configure --prefix=/opt/nginx --user=nginx --group=nginx% make% [sudo] make install

As you can see, we provided the /opt/nginx to configure, make sure the /opt directory exists. Also, make sure that there is a user and a group called nginx, if they don’t exist, add them:

% [sudo] adduser --system --no-create-home --disabled-login --disabled-password --group nginx

After that, you can start nginx using the command line below:

% [sudo] /opt/nginx/sbin/nginx

Linode provides an init script that uses start-stop-daemon, you might want to use it.

nginx configuration

nginx comes with a default nginx.conf file, let’s change it to reflect the following configuration requirements:

  • nginx should start workers with the nginx user
  • nginx should have two worker processes
  • the PID should be stored in the /opt/nginx/log/nginx.pid file
  • nginx must have an access log in /opt/nginx/logs/access.log
  • the configuration for the Django project we’re going to develop should be versioned with the entire code, so it must be included in the nginx.conf file (assume that the library project is in the directory /opt/projects).

So here is the nginx.conf for the requirements above:

user  nginx;worker_processes  2;

pid logs/nginx.pid;

events {    worker_connections  1024;}

http {    include       mime.types;    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                     '$status $body_bytes_sent "$http_referer" '                     '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile           on;    keepalive_timeout  65;

    include /opt/projects/showcase/nginx.conf;}

Now we just need to write the configuration for our Django project. I’m using an old sample project written while I was working at Giran: the name is lojas giranianas, a nonsense portuguese joke with a famous brazilian store. It’s an unfinished showcase of products, it’s like an e-commerce project, but it can’t sell, so it’s just a product catalog. The code is available at Github. The nginx.conf file for the repository is here:

server {    listen 80;    server_name localhost;

    charset utf-8;

    location / {        proxy_set_header    X-Real-IP   $remote_addr;        proxy_set_header    Host        $http_host;        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://localhost:8000;    }

    location /static {        root /opt/projects/showcase/;        expires 1d;    }}

The server listens on port 80, responds for the localhost hostname (read more about the Host header). The location /static directive says that nginx will serve the static files of the project. It also includes an expires directive for caching control. The location / directive makes a proxy_pass, forwarding all requisitions to an upstream server listening on port 8000, this server is the subject of the next post of the series: the Green Unicorn (gunicorn) server.

Not only the HTTP request itself is forwarded to the gunicorn server, but also some headers, that helps to properly deal with the request:

  • X-Real-IP: forwards the remote address to the upstream server, so it can know the real IP of the user. When nginx forwards the request to gunicorn, without this header, all gunicorn will know is that there is a request coming from localhost (or wherever the nginx server is), the remote address is always the IP address of the machine where nginx is running (who actually make the request to gunicorn)
  • Host: the Host header is forwarded so gunicorn can treat different requests for different hosts. Without this header, it will be impossible to Gunicorn to have these constraints
  • X-Forwarded-For: also known as XFF, this header provide more precise information about the real IP who makes the request. Imagine there are 10 proxies between the user machine and your webserver, the XFF header will all these proxies comma separated. In order to not turn a proxy into an anonymizer, it’s a good practice to always forward this header.

So that is it, in the next post we are going to install and run gunicorn. In other posts, we’ll see how to make automated deploys using Fabric, and some tricks on caching (using the proxy_cache directive and integrating Django, nginx and memcached).

See you in next posts.

[Francisco Souza] Go solution of the Dining philosophers problem

Monday, October 31st, 2011

Solution for the dining philosophers problem using the Go programming language.