Planeta Globo.com

One of the core premisses of RESTful web services is that HTTP should be seen as an application protocol rather than just a transport protocol. It comprises a whole bunch of semantics that allows us to build robust distributed systems. And for some cases, when multiple consumers manipulate the same resource, therefore changing its state, the solution should be robust enough to prevent the system to get into a race condition.

But how HTTP could prevent that?

HTTP provides a simple but powerful mechanism for aligning resource states by making use of entity tag or ETag and conditional request headers. An ETag is anything that uniquely identifies an entity, such as the ID associated with a persisted resource, a checksum of the entity headers and body, etc. If this resource changes—that is, when one or more of its headers, or its entity body, changes—then the entity tag changes accordingly, reflecting this new resource state.

When a response contains an ETag associated to a resource state and you want to continue working with this same resource, it’s recommended to use this tag in subsequent requests (called conditional requests), otherwise the resource state might eventually become out of sync with service one, returning something like a 409 Conflict.

Conditional requests happens when the current ETag is supplied to a conditional request header, such as If-Match or If-None-Match, when user is requesting to update a resource for example. The service will then check the precondition, by comparing the current resource ETag with the one provided in the request. If it’s satisfied than the server proceeds and process the request, otherwise it concludes that the resource has changed and responds with a 412 Precondition Failed.

Example

Given an online shop for home goods, where two people— Admin1 and Admin2 —are responsible for administrating its contents. In our scenario both administrators are trying to change the state of the same product (the Weber BBQ), around the same time. Admin1 wants to lower the product price down to $300.00 and Admin2 wants to change its state to “Not Available”. Firstly, both administrators GET the current product state independently of one another by doing the following request:

GET /product/1 HTTP/1.1
Host: myshop.com

Returning the following resource (product) as response. Note that the service’s response contains an ETag header.

HTTP/1.1 200 OK
Content-Length: 265
Content-Type: application/xml
ETag: "686897696a7c876b7e"

<product>
  <name>Weber Family BBQ</name>
  <description>Great for parties and cooks a neat roast too.</description>
  <price>$399.00</price>
  <status>In Stock</status>
</product>

When Admin1 does a conditional PUT, including an If-Match header with the ETag value from the previous GET.

PUT /product/1 HTTP/1.1
Host: myshop.com
If-Match: "686897696a7c876b7e"

<product>
  <name>Weber Family BBQ</name>
  <description>Great for parties and cooks a neat roast too.</description>
  <price>$300.00</price>
  <status>In Stock</status>
</product>

And as the product state hasn’t changed since the last request, then the request is thus successful! Notice that the response returns an updated ETag value, reflecting the new product state.

HTTP/1.1 204 No Content
ETag: "616898r96a8cy86b8eee11"

Little time after Admin1 has updated the product, Admin2 does another PUT to the same product, including the same If-Match header with the ETag value from the GET request.

PUT /product/1 HTTP/1.1
Host: myshop.com
If-Match: "686897696a7c876b7e"

<product>
  <name>Weber Family BBQ</name>
  <description>Great for parties and cooks a neat roast too.</description>
  <price>$399.00</price>
  <status>Not Available</status>
</product>

The service then determines that someone is trying to change the same product, using an out-of-date resource representation (ETags are different!), and responds with a 412 Precondition Failed code. No race conditions whatsoever!

HTTP/1.1 412 Precondition Failed

Conclusion

Although ETags and conditional request headers make up a powerful mechanism for dealing with concurrency, one thing to keep in mind is that, depending on the amount of computation performed by the server to generate an ETag, response times might increase considerably. So use it only if you need it!

In the beginning of the year I posted about the ways you can use Hamcrest out of test code, together with hamcrest-collections. This combination allows us to write different kinds of matchers to select and reject items from lists, as well as applying map and reduce to them. After a while making use of them on my current project, I wanted to share what I liked a lot, and what I didn’t like a lot. My friend Liz Douglass has also written a post sharing our experience, and I will just complement it a bit…

What I Liked A Lot

There’s no much to write here, as we all know that this combination is quite powerful when you’re looking for writing code that reads more like english language, making it much easier to express the intent of your code. Not to mention that we get rid of for loops everywhere in the codebase.

What I Didn’t Liked A Lot

One aspect I didn’t like since the beginning when implementing Reducers is that they are coupled to one specific type. It reduces a list of one type into a result of the same type. And from the Wikipedia definition of Map and Reduce…

“Map” step: The master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes.

“Reduce” step: The master node then takes the answers to all the sub-problems and combines them in a way to get the output - the answer to the problem it was originally trying to solve.

… we can see that it doesn’t mention that the result should be of the same type as the original one, after applying the reducer. And that’s exactly what I wanted to do instead. People said I was trying to combine both Map and Reduce into a single implementation. I kind of disagree with that, because the fact that I am reducing a list into a result of a different type, it doesn’t mean that I am transforming the original input (like multiplying each item in a list of integers by 2, before concatenating them). Confusing?

Let me try to explain it using an example. Given we have a list of integers…

List list = Lists.create(1, 2, 3);

and that I want to concatenate these numbers into a string. With the current hamcrest-collections implementation, that would be possible doing something like…

Iterable listOfStrings = FunctionMapper.map(list, new Function() {
    public String apply(Integer number) {
        return String.valueOf(number);
    }
});

String result = Reduction.reduce(listOfStrings, new Reducer() {
    public String apply(String first, String previous) {
        return first.concat("+").concat(previous);
    }
});

Assert.assertEquals("1+2+3", result);

It’s quite a lot of code just to concatenate a list of numbers! One day while pairing with Tom Czarniecki, we decided to reimplement the Reduction and Reducer classes, so that we could create more flexible and simple Reducer implementations, and of course writing almost half the lines of code.

public interface Reducer {
    U apply(T first, U previous);
}

public class Reduction {

    public static  U reduce(List list, U initialValue, Reducer reducer) {
        U currentValue = initialValue;
        for (T item : list) {
            currentValue = reducer.apply(item, currentValue);
        }
        return currentValue;
    }
}

To do the same number concatenation with this new implementation, is just a matter of defining a new Reducer, that concatenates them in a string.

Assert.assertEquals("1+2+3", Reduction.reduce(list, "", new Reducer() {
    public String apply(Integer first, String previous) {
        return previous.concat("+").concat(String.valueOf(first));
    }
}));

Much simpler!

Advantages Of This New Implementation

1. Do I have to mention again that it’s much cleaner?
2. We use java.util.List instead of Iterable.
3. No exception is thrown if the list to reduce is empty. It just uses the initial value provided on the reducer implementation.
4. Flexibility to reduce a list into a result of any type.

Hope you enjoy it!

One of the big challenges faced by distributed teams is how to get over the communication gap created by the physical distances that separates them. We all know that communication, either verbal or non-verbal, is fundamental for any project to be delivered successfully. When a team is good at communicating, they cultivate a more effective sense of collectivity and cooperation, having faster feedback, by sharing information (knowledge) and having valuable discussions.

But this is not quite the real world for distributed development teams. It’s much harder, not to say almost impossible, to know what exactly is happening on each other’s mind. What problems and technical challenges are they facing? What are they doing now? What points are they considering when designing a new feature? How important is for them to write tests? Are they following the project development standards?

Blame the “Bandwidth Limited” Communication Tools!

Software development teams, by the nature of their work, needs to discuss and assess different ideas to solve complex problems. And they are very difficult to communicate when using tools such as email or telephone, which on the book they call “bandwidth limited”. And those are exactly the ones available for most distributed teams. So face-to-face communication suits better for this kind of discussions, using the assistance of diagrams or sketches, not to mention the use of body language. This would give us immediate feedback, just by looking into the other person’s eyes, which communicate understanding.

* Extracted from The Organization and Architecture of Innovation: Managing the Flow of Technology (with some modifications).

And as you can’t always minimize distances to allow verbal communication, you have to look for other ways, and maximizing non-verbal communications is definitely a road to go down.

Some Bad Outcomes

Poor Code Quality

• Code Duplication (see)

It’s quite usual. For example, the guy wants to load a XML file as a String so that he can perform some assertions over the result. He will implements something like a FileLoader class. But what he doesn’t know is that another developer has already implemented a class with this behaviour.

• Reinventing the wheel (see)

This is partially caused by lack of communication and partially a result of the programmer’s discipline. When adding a new library to the project the team must have a discussion and look for the benefits earned by using it. Before adding a XML parsing library that you’re used to, have a quick chat with the team will let you know if is there any other parsing library being used. Maybe someone could make a walk-through with you on it. But it is your responsibility to know how to use it afterwards.

• Code For The Others (and for yourself)

When coding, you should always ask yourself if your peers would be able to understand what are you producing. Better still, you should ask if you would easily understand it again in a couple of weeks from now. It’s quite common when coding, you get contextualized with what you need to do to deliver that functionality. This context will always get lost after finishing, unless you share it with the others or document it. There are some good materials out there that shows you how to write clean and readable code.

• Broken Builds

In a distributed team, a broken build not only just affects the people in your room, it also affects people in rooms into other cities. So reverting a broken build should be taken into account, specially when you have a slow build, then definitely the commiter would get himself into a big problem! Imagine a long build that takes about 30 minutes for example, and someone commits something broken. If he fixes it really quickly, it still may take 1 hour for the other team to be able to commit its changes and consequently 1 hour and a half lost in productivity in the other cities. It’s all about communication - the quicker the build, the quicker the feedback. So a fast and successful build is mandatory!

Fear of Refactoring

Poor code quality results in fear of refactoring. Who hasn’t been in a situation, working on a tightly coupled system, where it was quite hard to do any refactoring? Any attempt would propagate the changes deep in the source code, ending up shaving the yak, not going anywhere.

Absence of Trust

I see this one as a result of the other two I mentioned above. When your team is biased to go off the tracks when trying to comply with code standards, some precautionary measures are generally created to avoid the worse.

I’ve seen a case where a pair, assigned to implement a story, and almost completing the development, ended up realising that another pair was also looking at it. Don’t ask me why!

I’ve also seen people creating triggers on the version control system, so that for each commit from one team, the other received an email with all the commit information. This is good in one side, because you can easily identify cowboy commiters that don’t write tests. But this is also used to check if the code is acceptable, reverting if not!

My Current Experience

The team I’m currently working with is facing some of these problems, and during all last week, when I was on the other side of the fence, visiting the other part of out team on Tasmania, this became even more highlighted. Although we were having daily stand-up meetings, I felt like I was missing something, specially because there was another team in Melbourne joining us and I still didn’t know how it was going to work out. Chatting with my friend Mark Needham about it, he recommended me a book called The Organization and Architecture of Innovation: Managing the Flow of Technology, where there’s a chapter dedicated exclusively to this point, and that I could probably get some ideas of how to overcome this problem.

Taking actions

It seems obvious that an organization that wants its technical staff members to communicate needs to ensure the distances among them are minimized. Unfortunately, the traditional and most common form of office configuration does just the opposite. Not to mention when they are in separated buildings.

The quote above also extracted from the book, doesn’t tell anything new, and that’s exactly one of the issues we wanted to fix. Now, with three teams we agreed that we would need to have them communicating face-to-face more often. So the rule is that every week we should have at least one person from each team visiting a different one. Apart from that, we are continuing with our daily stand-up meetings, each team separately, and later on another daily meeting, but between teams (in the Scrum world called Scrum of Scrums). This one involves, by default, only the iteration manager and the tech lead, but everyone else is also welcome to attend.

We also had to put more effort on improving the non-verbal communication, as they are more required on distributed teams. With this separation teams have to be even more strict with what they permit or not in the codebase. We introduced development tools such as Checkstyle and Compile With Walls to ensure this. Checkstyle acts as a hammer on misbehaved commiters and Compile With Walls ensures that project structure is being respected. Sometimes quite good threads (over IM or email) are created by people trying to understand why a Checkstyle rule has failed.

(Thanks to Tom Czarniecki for helping me with this one.)

ThoughtWorks Australia is looking for new talents!

This time we are hiring Senior QA Testing Consultants!
So if you want to work in this fast growing, unhierarchical consultancy, applying your knowledge of testing in a variety of client environments while constantly using the latest methodologies and technologies, you can continue reading this post, otherwise, just don’t bother

Working with us, you’ll get to work alongside truly talented teams and help them enhance their performance by bringing quality assurance to the forefront of clients’ minds. As well as ensuring the bug-free delivery of custom built software, you will also be working with clients to advise them on improving their test processes and teaching them about the very latest from the QA world.

Some Of The Duties

Our test processes are very different to many organisations. Testers are involved from the initial requirements gathering through implementation to deployment. They are always around to ask the awkward questions and try scenarios that analysts or developers are unlikely to dream up. They are involved when analysts are capturing requirements in the form of user stories. These stories are then converted into acceptance tests outlining specific scenarios. Testers play a big part in making sure those tests are well defined and complete so that developers know when they have finished implementing the functionality defined in a story. For more information, visit http://testing.thoughtworks.com.

Desired Experience

• Be a very hands-on tester who is comfortable across a whole range of functional testing including UAT, acceptance and system testing with tools like Fit, Fitnesse, Silk, Winrunner or any other automation tool
• Experience of participating in full life cycle development right from the requirements gathering and analysis phase
• Have worked on large, long term projects (more than 10 people, longer than 6 months)
• Enjoyment of working closely with developers, analysts and clients in a highly collaborative environment
• Exceptional communication skills
• An unrivalled passion for delivery

Also Highly Desirable

• Experience of creating test frameworks and strategy, choosing automated testing tools and creating testing standards
• Experience of, or interest in working with Open Source testing tools like Selenium and Sahi
• A knowledge of testing within an Agile development environment
• A background in OO development
• A track-record of innovation in testing
• Experience of working in an onsite, consultancy environment

So if you are interested, then click here to apply online. And just a quick reminder that ThoughtWorks offers Visa Sponsorship for candidates.

Last week I attended the Lean Thinking And Practices For IT Leaders workshop organised by ThoughtWorks. There we had the presence of Mary and Tom Poppendieck, my colleague Jason Yip and two consultants from KM&T. One of the things that I really liked about it was that it wasn’t only driven by presentations, but also by a lot of practical exercises, so we could get a better feeling of the benefits of applying these thinking and practices. One of the exercises we did was the Go-Kart game.

How it works?

Two teams are created (alpha and beta), and each one has to split up into five groups with the given responsibilities: disassembly, transportation, assembly, observation and time-keeping. They are given the task to completely disassemble, transport and re-assemble a Go-Kart as quick as possible, in a safe manner, while the observer write notes about problem points. The whole process is done twice, so that you can run it once, analyse the process used, based on feedback provided by the observer, and think of ways to improve it, before running the second time.

First Attempt

In our first attempt, all we knew was that we had to split the team into five groups. We had no idea of the necessity of a detailed process, but doing all the phases as fast as possible. Vikky, our team leader, proposed the creation of a manual with the detailed steps needed to assemble the kart, to be used by the assembly team. And that’s what we did!

Our marks

Planing time: 10 minutes
Disassembling time: 5 minutes
Assembling time: 12 minutes
Total time: 14 minutes 20 seconds
Quality of delivered product: OK

Problem Points (Gathered by observers)

• The team took seven minutes to get organised and start doing something.
• No leadership nomination. Vikky, one of the team members, had to auto-niminate herself as the team leader.
• Disassembly group didn’t notice differences on the washers and on the bolts, causing uncertainty and waste of time in the assembly group.
• Bottleneck on the transportation of the parts from one station to another. No one from disassembly group to pick up the parts, making the transporter keep holding them, stopping the process flow.
• The components needed to assemble specific parts of the car were not delivered together, making the assembly group wait for the remaining ones.
• Some members in the assembly group were in a rush to finish fast and ignored the manual, resulting in some mistakes.

Second Attempt

Before starting the second attempt we got together to discuss the problem points, coming up with some ideas of improvements. Here they are:

Improvements

• We nominated people on both disassembly and assembly groups to be in charge of handing and picking up parts from the transporter.
• We decided to hand the parts related to each other in chunks, so that they could be assembled straight away, eliminating the time wasted waiting for remaining parts.
• We nominated specialists for roles such as assembling the wheels, etc.
• We added one more member to the transportation group, to get rid of the bottleneck.

Instead of spending a long time planning, we did it the agile way, highlighting only things we knew at the time, very quickly, and running through, spiking and checking if we were actually carrying out with the improvements, before doing the official attempt. We found some problems, adjusted to them and immediately got organised for the second attempt.

Our marks

Planing time: 10 minutes
Disassembling time: 1 minute 50 seconds
Assembling time: 2 minutes 33 seconds
Total time: 3 minutes 45 seconds
Quality of delivered product: OK

Click here to see some photos of our team during the exercise.

Conclusion

Lean advocates that you should pursue perfection when improving your process - aiming to reduce effort, time, space, cost and mistakes - and I learnt that this applies to any organisation, of any size. Thus, from the process used on this game, collaboration, self-organisation, rapid feedback contributed a lot to our improvement, helping us to eliminate waste.

So, what could you do for your organisation?

Take a step back, take a look at the big picture of how things work in your company and ask yourself questions such as: How do we deliver? Does it takes longer to test and deploy our system than to develop it? Who do we depend on to put the system onto production? What is causing a bottleneck? What could I do to change this scenario? Answer these questions (or others you make up) and think of improvements.

It’s been a while since I read some interesting posts showing creative uses of Hamcrest library out of test code. Since then I’ve been proscrastinating to implement my own version, trying strongly typed java delegates.

Thankfully this week I came across a nice API called hamcrest-collections. It uses Hamcrest to implement features such as select, reject, map, reduce and zip familiar from languages like Ruby and Python.

Selectors

Selectors can be used to select or reject items that matches a given Matcher, from any iterable object. It reminds me the Specification Pattern from Domain-Driven Design, which is also used for querying objects that satisfies defined specifications.

public static final Person john = new Person("John", 28);
public static final Person nicole = new Person("Nicole", 12);
public static final Person ryan = new Person("Ryan", 23);
public static final Person nathan = new Person("Nathan", 18);

public static final List list() {
    return Arrays.asList(john, nicole, ryan, nathan);
}

The code below selects from the list of users defined above, the ones that are under twenty.

@Test
public void should_select_only_people_under_twenty_years_old() {
    List users = Person.list();
    Iterable underTwentyList = select(users, underAge(20));
    assertThat(underTwentyList, hasItems(nicole, nathan));
    assertThat(underTwentyList, not(hasItems(john, ryan)));
}

The code below rejects all the users that are under twenty.

@Test
public void should_reject_every_people_under_twenty_years_old() {
    List users = Person.list();
    Iterable aboveTwentyList = reject(users, underAge(20));
    assertThat(aboveTwentyList, hasItems(john, ryan));
    assertThat(aboveTwentyList, not(hasItems(nicole, nathan)));
}

Map and Reduce

Map is used to apply a function onto each item in any iterable object, whereas Reduce combines all these elements, applying a Reducer implementation. In our example, we map the timesTwo function, that doubles each element in the list, and then we reduce it by adding up all of them.

@Test
public void should_double_each_number_in_the_list_then_sum_all_of_them() {
    List numbers = Arrays.asList(1, 2, 3);
    MultiplyBy timesTwo = new MultiplyBy(2);

    Iterable result = map(numbers, timesTwo);
    assertThat(result, hasItems(2, 4, 6));

    Integer sum = reduce(result, new Sum());
    assertThat(sum, equalTo(12));
}

public class MultiplyBy implements Function {
    private Integer factor;

    public MultiplyBy(Integer factor) {
        this.factor = factor;
    }

    public Integer apply(Integer number) {
        return (int)number * factor;
    }
}

public class Sum implements Reducer {
    public Integer apply(Integer first, Integer second) {
        return first + second;
    }
}

Despite the bias created by some developers, that Hamcrest should not be used anywhere else but test code, specially after JUnit has defined it as its new matcher library, just ignore it and add these features to your runtime library, so that you can let your creativity drive you when developing. Get rid of “for” loops from your life!

After reading posts from some friends I decided to write my 2008 retrospective, so there it go!

Personal Life

• First year married
• Moved to the land down under.
• Tried to go to the gym, but it still seems like I am better as an investor

Professional Life

• Joined ThoughtWorks Australia.
• Tried to post more often on my blog.
• Became a Certified Scrum Master Of The Universe!
• Projects: 4
• Conferences attended: JAOO Sydney

Learning

• Languages learned: Erlang, Ruby, Clojure (started)
• Books bought: 15
• Books read:
1. Test Driven Development
2. My Job Went To India
3. Waltzing With Bears
4. Programming Erlang
5. Agile Software Development With Scrum
6. Agile Web Development With Rails
7. Agile Retrospectives
8. Programming Clojure (started)
9. How To Mind Map

2009 Resolutions

Still haven’t planned properly what to do for 2009, the only thing for now is continuing learning Clojure, and understand more about applying Lean principles into software development.