Posts de ‘Igor Sobreira’

[Igor Sobreira] Improving performance of Django test suite

Wednesday, September 19th, 2012

I’m working on a small django application these days which offers templates to override django-filebrowser’s templates to remove the Grappelli dependency.

It’s a small app, consisting 99% of templates. There are only 19 tests which take 35 seconds to execute! All of them performs at least one request using django test client, no selenium, no real HTTP requests, and still 35 seconds!

Tests speed very slow

After a run with cProfile I’ve noticed many calls to hash functions, specifically django.utils.crypto.pbkdf2() and it’s helper _fast_hmac. Then I came up with this very complex patch to my settings:

+    PASSWORD_HASHERS = (
+        'django.contrib.auth.hashers.UnsaltedMD5PasswordHasher',
+    ),

Now the tests execute on 6 seconds!

Tests speed improved

You probably want a safer hash function for your user passwords in production, but there is no problem to use a simpler one for tests. This extensible way to deal with password hashers was introduced in Django 1.4, you can read more about it on the docs.

[Igor Sobreira] Vagrant plugin to take screenshots

Wednesday, March 14th, 2012

Published on: 13/03/2012 22:19h

I’ve been using Vagrant a lot at work. These days we were facing some problems when our VMs were booting and a friend, jbochi, came up with an idea to take a screenshot from a running VM.

He found this command on VBoxManage:

$ VBoxManage controlvm [vm-uuid] screenshotpng [output-filename]

And it helped a lot! That’s when we had de idea to wrap this command in a script to keep using. Initially I’ve created a small python script, but then reading about vagrant plugins (hltbra’s suggestion) I’ve decided to create one.

The vagrant-screenshot plugin is very simple. You just need to install the gem and use:

$ gem install vagrant-screenshot

Here is an usage example:

$ vagrant screenshot -o[vagrant] Taking screenshot for default[vagrant] Screenshot saved on screenshot-default.png

By default it takes screenshots for all active VMs. The -o (--open) option opens the images when done (only works on OS X for now)

See the help for more details

$ vagrant screenshot --help

PS.: The problem I’ve mentioned we were facing ended up in a pull request to vagrant, but unfortunately it wasn’t accepted :o)

By Igor Sobreira

[Igor Sobreira] Vagrant plugin to take screenshots

Tuesday, March 13th, 2012

I’ve been using Vagrant a lot at work. These days we were facing some problems when our VMs were booting and a friend, jbochi, came up with an idea to take a screenshot from a running VM.

He found this command on VBoxManage:

$ VBoxManage controlvm [vm-uuid] screenshotpng [output-filename]

And it helped a lot! That’s when we had the idea to wrap this command in a script to keep using. Initially I’ve created a small python script, but then reading about vagrant plugins (hltbra’s suggestion) I’ve decided to create one.

The vagrant-screenshot plugin is very simple. You just need to install the gem and use:

$ gem install vagrant-screenshot

Here is an usage example:

$ vagrant screenshot -o
[vagrant] Taking screenshot for default
[vagrant] Screenshot saved on screenshot-default.png

By default it takes screenshots for all active VMs. The -o (--open) option opens the images when done (only works on OS X for now)

See the help for more details

$ vagrant screenshot --help

PS.: The problem I’ve mentioned we were facing ended up in a pull request to vagrant, but unfortunately it wasn’t accepted :o)

[Igor Sobreira] Book review: Being Geek

Thursday, December 29th, 2011

Published on: 28/12/2011 23:50h

I’ve finished reading this book, Being Geek, by Michael Lopp, an experienced engineering manager from Silicon Valley. And recommend it…

It’s a book about geeks professionalism, career. I liked the way the author exposes his ideas, he’s clear and pragmatic. Right in the second chapter there are these three questions I think it’s worth pointing out:

  1. Are you actively defining the technical direction for your product?
  2. Do you understand what you need to do in order to grow?
  3. Are you hitting your dates? Are you meeting your commitments? Are you doing what you say you’re going to do?

That’s it! That’s what you need to ask yourself every day. Those answers will make you realize if this job is being useful for you to archive your professional goal in your life.

Later in the book he talks about task management, to-do lists and so on. And a very interesting one he calls “The Trickle List”. This is not a simple to-do list for that day. It’s a list of tasks you do every day with a bigger goal in mind, in the future. If you want to be a better engineer, every day you may want to learn something, improve on something, read something. I suddenly realized I was missing this list…

Well, I liked the book. I was not very excited reading the first chapters. But as I continued reading it just got better and better. The author writes comfortably about our career, being an engineer or a manager. The book is full of good advices based on experience. Hiring process, remote working, giving presentations, etc.

By Igor Sobreira

[Igor Sobreira] Book review - Being Geek

Wednesday, December 28th, 2011

Book Being Geek

I’ve finished reading this book, Being Geek, by Michael Lopp, an experienced engineering manager from Silicon Valley. And recommend it…

It’s a book about geeks professionalism, career. I liked the way the author exposes his ideas, he’s clear and pragmatic. Right in the second chapter there are these three questions I think it’s worth pointing out:

  • Are you actively defining the technical direction for your product?
  • Do you understand what you need to do in order to grow?
  • Are you hitting your dates? Are you meeting your commitments? Are you doing what you say you’re going to do?

That’s it! That’s what you need to ask yourself every day. Those answers will make you realize if this job is being useful for you to archive your professional goal in your life.

Later in the book he talks about task management, to-do lists and so on. And a very interesting one he calls “The Trickle List”. This is not a simple to-do list for that day. It’s a list of tasks you do every day with a bigger goal in mind, in the future. If you want to be a better engineer, every day you may want to learn something, improve on something, read something. I suddenly realized I was missing this list…

Well, I liked the book. I was not very excited reading the first chapters. But as I continued reading it just got better and better. The author writes comfortably about our career, being an engineer or a manager. The book is full of good advices based on experience. Hiring process, remote working, giving presentations, etc.

[Igor Sobreira] Command line interface to FMS Admin

Monday, December 26th, 2011

Published on: 25/12/2011 13:29h

I’ve been working with Flash Media Server for the last 8 months now, and
one tool that really annoys me is the Administration Console. I miss a
command line interface to accomplish tasks like see all live streams, reload
an app, etc.

That’s why I’ve started the fms-admin-api. It’s a command line
client to Flash Media Server Administration API.
Since it’s written in ruby, it’s possible to use the ruby client class directly.
Use rubygems to install:


$ gem install fms-admin-api

The way it works is really simple. Here is an example: there is a reloadApp() method available
that accepts the arguments host, auser,
apswd and appInst. The command line interface just replaces
from camelCase syntax to underscore:


$ fmsapi reload_app –host=fms.example.com –auser=fms –apswd=secret –app_inst=live

There are improvements to be made
and help is welcome.

By Igor Sobreira

[Igor Sobreira] Command line interface to FMS Admin

Sunday, December 25th, 2011

I’ve been working with Flash Media Server for the last 8 months now, and one tool that really annoys me is the Administration Console. I miss a command line interface to accomplish tasks like see all live streams, reload an app, etc.

That’s why I’ve started the fms-admin-api. It’s a command line client to Flash Media Server Administration API. Since it’s written in ruby, it’s possible to use the ruby client class directly. Use rubygems to install:

$ gem install fms-admin-api

The way it works is really simple. Here is an example: there is a reloadApp() method available that accepts the arguments host, auser, apswd and appInst. The command line interface just replaces from camelCase syntax to underscore:

$ fmsapi reload_app --host=fms.example.com --auser=fms --apswd=secret --app_inst=live

There are improvements to be made and help is welcome.

[Igor Sobreira] Change object after saving all inlines in Django Admin

Wednesday, October 19th, 2011

Published on: 12/02/2011 11:45h

UPDATE Aug, 2011: This patch added a new hook to ModelAdmin called save_related(). Now you don’t the hack described bellow anymore.


Admin is an awesome Django builtin app to create nice CRUDs for your models, and offers a lot of customizations hooks. You can personalize the templates, perform custom filters, modify newly created objects and if that’s not enough, you can always create your own view to do something it doesn’t by default.

The admin docs are great, I’m not going to explain how it works. The intention here is to show a way to modify an object, after it’s saved, and other objects related to it using inlines are saved too.

Once you’ve configured your admin interface with inlines, you end up with something similar to:

from django.contrib import admin
from fooapp.models import Foo, Related

class RelatedInline(admin.TabularInline):
    model = Related

class FooAdmin(admin.ModelAdmin):
    inlines = [RelatedInline]

admin.site.register(Foo, FooAdmin)

this mean whens you’re adding a Foo, since Related has a foreign key to it, django will display a few forms to add Relateds in the same page.

Imagine you need to do something with the new foo instance that needs to now how many related objects it has. Admin already has methods you can override to do something after Foo is saved, like ModelAdmin.save_model(). See how it works:

class FooAdmin(admin.ModelAdmin):
    inlines = [RelatedInline]

    def save_model(self, request, obj, form, change):
        obj.save()
        # do something with obj.related_set.all()
        # OPS! it's empty!

admin.site.register(Foo, FooAdmin)

the problem here is that save_model() is called before the inlines are saved.

Let’s find out how it works. Open django source code, specifically in django.contrib.admin.options.py go to add_view(), it’s the view called when you are creating an object. As you can see, when the request method is “POST” it goes through all validation, for the main form and all related formsets (your inlines). And if all of then are valid it calls save_model(), then save_formset() for each formset. The interesting piece is shown below:

class ModelAdmin(BaseModelAdmin):

    # ...

    def add_view(self, request, form_url='', extra_context=None):

        # ... all validation here ...

        if all_valid(formsets) and form_validated:
            self.save_model(request, new_object, form, change=False)
            form.save_m2m()
            for formset in formsets:
                self.save_formset(request, form, formset, change=False)

            self.log_addition(request, new_object)
            return self.response_add(request, new_object)

Notice here that the last method it calls is response_add(), and it passes the created object, that’s all we need! If you see the change_view() method (witch it the view called when you’re editing an object) it calls a similar method: response_change().

Now we can solve our problem doing something like this:

class FooAdmin(admin.ModelAdmin):
    inlines = [RelatedInline]

    def response_add(self, request, new_object):
        obj = self.after_saving_model_and_related_inlines(new_object)
        return super(FooAdmin, self).response_add(request, obj)

    def response_change(self, request, obj):
        obj = self.after_saving_model_and_related_inlines(obj)
        return super(FooAdmin, self).response_change(request, obj)

    def after_saving_model_and_related_inlines(self, obj):
        print obj.related_set.all()
        # now we have what we need here... :)         return obj

There are two things to keep in mind when using these methods:

  • They are not documented. So probably it’s not part of the API admin expects you to override. Thankfully django publishes excellent release notes including backwards incompatible changes. And I hope you have good test coverage to know if it will break when you update django :)
  • Their intention is to handle the response of the view, as it docstring says: “Determines the HttpResponse for the add_view stage”. So if you do something there not related to it, it works, wont make much sense if you look at the overall architecture.

Maybe django will add a more specific hook to solve this issue later, but I hope it helps you for now.

By Igor Sobreira

[Igor Sobreira] Adding methods dynamically in Python

Wednesday, October 19th, 2011

Published on: 06/02/2011 20:50h

Given the dynamic nature of Python you can do many things in runtime, like add methods dynamically to an object or class. This is particularly useful when writing unit tests.

Here is the simplest way, adding a method to and object:

class Person(object):
    pass

def play():
    print "i'm playing!"

p = Person()
p.play = play
p.play()

note that play is just a function, it doesn’t receive self. There is no way to p knows that it’s a method. If you need self, you must create a method and then bind to the object:

from types import MethodType

class Person(object):
    def __init__(self, name):
        self.name = name

def play(self):
    print "%s is playing!" % self.name

p = Person("igor")
p.play = MethodType(play, p)
p.play()

In these examples, only the p instance will have play method, other instances of Person won’t. To accomplish this we need to add the method to the class:

class Person(object):
    def __init__(self, name):
        self.name = name

def play(self):
    print "%s is playing!" % self.name

Person.play = play

p1 = Person("igor")
p1.play()

p2 = Person("joh")
p2.play()

note that we don’t need to create a method with types.MethodType here, because all functions in the body of a class will become methods and receive self, unless you explicit say it’s a classmethod or staticmethod.

Adding methods to builtin types

Python doesn’t allow to add methods to builtin types, actually to any type defined in C. The unique way around this is to subclass the type, here is an example:

class UpperList(list):
    pass

def to_upper(self):
    for index, item in enumerate(self):
        self[index] = item.upper()

UpperList.to_upper = to_upper

l = UpperList(['i','g','o','r'])
l.to_upper()
print l

Here is a nice comparison on dynamically adding methods in Python and Ruby.

By Igor Sobreira

[Igor Sobreira] Mocks, Stubs, Fakes and Spies

Wednesday, October 19th, 2011

Published on: 15/12/2010 01:05h

Reading the Continous Delivery book I found a good definition of mocks, fakes, stubs and spies:

  • “Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production. A good example of this is the in-memory database.
  • Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.
  • Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
  • Mocks are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don’t expect and are checked during verification to ensure they got all the calls they were expecting.”

This terminology was taken from xUnit Test Patterns book.

I’ve already seen cases where people write lot’s of unit tests, abusing from mock objects, but without good integration and acceptance tests you can’t really guarantee that each scenario is really working as expected.

“It’s very easy to misuse mocks to write tests that are both pointless and fragile, using them simply to assert the specific details of the workings of some code, rather than an assertion of interactions with collaborators.”

A couple links if you want to read more about it: