Effort Estimations Fallacy or Effort (only) Driven Project Planning

Countless books, blogs and studies have been written about the effort estimation process. Unless we talk about repetitive, so called “blue collar” jobs, the estimation of the time required to complete any complex activity has proven something hard to do right (these are my favorite studies: “Research suggests people underestimate numerical guesses when leaning left“, “Impact of irrelevant and misleading information on the effort estimate.”). My take on this is that the problem is not easy to solve as our work includes solving non-linear problems (or that the estimation of the effort to solve a problem is non-linear problem to solve). Before joining the big company, I had fun dealing with some of these problems in another field, and to my mind, from early days of working as a developer in a dev team, everything that had to do with the planning and the effort estimation was similar to the simulating annealing process, where the convergence to the global maximum requires sometimes taking random (not to say wrong) steps. In my previous blog, I talked about the “Cone of Uncertainty” that should shrink over time, which in a way is similar to the idea of the cooling in the simulating annealing algorithm.

I went to this length of explaining my “academical view” on this, as I would like you to remember two things: First, if you ever find yourself reading or trying to find a magic formula for effort estimation – you are wasting your time. Techniques, such as “functional points analysis”, have been invented by folks that never wrote single line of code. Secondly, the fact that the estimation is not a simple thing to do, doesn’t mean you should not do it, I am only saying, what ever you do, be careful about the data you get out of this process.

Many agile gurus, like Craig Larman, rather talk about a need for trust and transparency between stake holders and R&D, in order to make a project planning process more transparent and more engaging. More over, as the burden of success or failure of the project are often solely responsibility of the project manager after initial roadmap phase (which is wrong in many ways), discussions about the effort estimation becomes the central point of the project planning – generating mistrust between R&D and the rest of community at early stage of the project. WRONG!!

First of all, like mentioned  in my previous blog, project roadmap planning must be first about the product strategy and activities which are required to make the release success. Once that is clear, and first few iterations are completed, people will have better idea what is probably achievable in the current release. Before that phase is completed, any estimation or the feedback from R&D must be taken with a caution. At very beginning of the project, the focus of the  team, including R&D and PO, should not be on the effort estimation, but rather on understanding of the requirements, complexity of the problem and different possible paths that can lead to solution. Once this is clear, we are ready for the planning phase.

But before I explain how I do it (in the end, this is a blog about agileME), I still feel like introducing a little bit of methodology, sorry for that. When it comes to getting a number from the estimation process, they are today two main schools of thoughts: you either do the estimation in time, or in story points.

Some time back, Mike Cohn came up with a story point planning strategy: check out book or video link from google talk, or slides. In short, first you go with iterative estimation strategy, called a poker game. Here are the steps:

  • Each estimator is given a deck of cards, each card has a valid estimate written on it (Fibonacci series)
  • Customer/Product owner reads a story and it’s discussed briefly
  • Each estimator selects a card that’s his or her estimate
  • Cards are turned over so all can see them
  • Discuss differences (especially outliers)
  • Re-estimate until estimates converge

Finally fill the backlog with tasks, subtasks and story points. Once you have done so, as Mike says, you continue like this:

  • Determine how many iterations you have
  • Estimate velocity as a range (number of story points team can finish in one iteration)
  • Multiply low velocity × number of iterations  – These are “Will Have” items
  • Multiply high velocity × number of iterations and substruct from this the number of “Will Have” items in story points  – These are “Might Have items”

Picture taken from Mike Cohn’s presentation on the iteration planning

It sounds simple, but there are few “gotchas” that have hit every single scrum team I have seen so far:

  • it makes people focused too much on poker games too early in the project (you miss a big picture).
  • it feels like project management is all about story point estimation and simple counting (to be frank, few agile folks warn about  this trap, but still it happens way too often)
  • if you make your planning only based on the total number of story points, then you need to estimate every single little thing in story points. I hate this, and this is in contrary to Pareto rule from the previous blog. More over, you may wonder whether to estimate “framework things”. Then you have academic discussions on whether to estimate refactoring, etc. endless. Before you know it, people would spend weeks doing estimation without doing any real work, contrary to everything agile should bring. If you have inexperienced scrum master that needs to provide a roadmap planning, you will be doing poker games like crazy and end up with 800 stories in your backlog before you even start doing anything (flat backlog blackout syndrome)
  • people don’t learn from their own estimations. As mentioned earlier, estimations are always wrong, but if you do it in time, at least after a while you know better whether you are a pessimist or an optimist.
  • you can’t make that much out of the story points if the team is new in this
  • we are told that story points are about the complexity number of the feature, BUT everyone in his mind has the time dimension anyway (strange how few people are willing to admit this, don’t know why). For a fun, check out this TED video “Why we think it’s OK to cheat and steal (sometimes)” from Dan Ariely, and pay attention on the part of the experiment which includes tokens as a intermediate currency before being exchange for a real money.
  • if someone tells me that he will need 5 points to make something, I have no clue when this stuff will be done, more over, the guy doesn’t feel like making any commitment to the rest of the team. (You may wonder what this has to do with my previous statement of avoiding getting in situation “commitment from R&D only” to the stakeholders). This is different, it is about a guy having the best guess how much is going to take him to finish a task within one iteration. If there is no time information present, you never know whether the guy is in trouble during the development, whether he knows what he is talking about, and if you like, it is hard to make a peer pressure working (I am a hard core XP guy on this, to be frank).

 

Now, you may wonder why project managers like story points planning. I think for the same reason they liked waterfall, it gives them a fake feeling of having things under control once you have your numbers. Like by magic, story points remove all uncertainty of making the project a success.

If you by now think that I am not using story points at all, you would be wrong :). I do use them, but in a different way. First of all, story points discussion is very good way of discussing complexities – indeed. It is engaging. It also provides people with a “token platform” to discuss different aspects of the problem, and that level of indirection at that point of time is a good thing. The only real question is what you do with these discussions and the numbers that come out of them? Here is a little tip, while listening on discussions, I am interested in two things:

  1. How much opinions differ
  2. How complex the feature is

Complexity number gives you an idea about the nature of the probability distribution of the estimation for a given feature, while different opinions are giving you the range (or sigma) of the “Bell Curve” – if and only if, the complexity seems to be low (in that case we can talk about the Bell curve).

If you are faced with a project that is not evolving that much, and I guess this is the place where most of the “story points guys” are writing their studies and books, you are faced with rather predictable environment, and story point only project planning might work well. BUT, if you are faced with projects in which features exhibit high level of skewness, you better do things differently. One thing that strikes me the most about story point articles is how they tend to plot the distribution of story points per iteration (like in picture above) – and come with a bell curve always, while I would be more interested to plot estimated vs. actual points required to finish any given story. Well, the point is, you can’t, as this is never captured. Again, you may think, why the hell would someone need that, since you come up with the global distribution that is bell curved anyway? Again, that is only true, if all the features are of the same distribution, and that is not the case in more complex projects! Just ask yourself, how many times you have actually seen burn down  chart working? Really?

By now, you probably know why most of the project managers are either depressed, lunatics or alcoholics, as they need to deal with quite some uncertainty every single day, and provide always accurate planing. But there is a way out of this, more about it in my next blog post.

Agile Planning

In my previous post I introduced some basic principles of project management, so if you skipped that part, please read it first 🙂 ). Here are my rules of Agile planning:

  • Rule1: Ask PO to present the vision of the new release to the complete team. This is milestone M1. Everyone in the team deserves to know what is the ultimate goal of that release.
  • Rule2: Define “rules of the engagement” for the complete team, including PO. That includes iteration span (heartbeat of the project), kick off dates (day in a week and the exact hour), standup meetings, scrum of the scrum meetings, format of the kick offs and demos etc (link to my previous post on agile meetings). This needs to be presented by project manager at very first iteration to the complete team. There might be some suggestions and arguments – that is good as we need to work as a team, but once that is all cleared – no more discussion on that. Important part of a project manager role is to show consistency by example, and insist other people follow the same.  I am true believer in self organized teams as long as the rules of game are known to everyone.
  • Rule 3: Don’t wait longer that 20-30% of the full release cycle on scoping and effort estimation. More than this is a waste – less than this is a chaos.  The most important part of the work within first few weeks is to run workshops and understand underlying complexity. Use whiteboards and avoid any UML tools or ppt slides. This is also the time you can tests new frameworks. Do iterations and coding(spikes) from day1, more you wait, longer the uncertainty (“Cone of Uncertainty” is not shrinking).
  • Rule4: Once you reached high level of understanding of the project functionality, create the project plan (something like a skeleton planning discussed before). Skeleton plan should be a joined effort and known to everyone. Once that is ready, you have your next milestone – M2
  • Rule 5: After every iteration, re-adjust one of the parameters of “The Iron Triangle”: your release valves – functionality vs. effort.  This is done after every iteration (between PO, scrum master and some key developers). During the project lifecycle these two items will become more and more accurate.

Presentation of the project organization to the complete team – real example

Once all features are implemented – you have your “beta” release. How long it takes you from “beta” till GA date, depends on various things. Typically, agile folks hate to talk about the “hardening” of beta releases and preach GA quality at every iteration. In on of my future blog posts, I will spend more time on this topic.

Project milestones

What does this mean, and what not?

  • In Agile, you still need a good overall planning, and you need to have a product backlog
  • Not ALL functionality needs to be known upfront, but R&D has to have a good understanding of all use cases. Agile saves time by having detailed discussion as part of iteration planning, and you are making discussions more efficient, simply because you have accumulated knowledge (through the previous iterations) which you did not have at M1
  • PO can ask for a feature to be extended/changed during a project – but he always needs to take into account  “Iron Triangle Principle” (hence PO/R&D collaboration is important throughout the project lifecycle)
  • Later requirements can be added to the release, but be careful – closer to “beta”  date, bigger the risk of adding new things.
  • PO defines the ‘What’, The Scrum Team define the ‘How’ and they work together to agree on ‘When’.
  • Testers can’t know upfront all detail requirements, but should know all use cases. They need to be on board from day1.
Why and when is agile so successful? – my philosophical view
  • Any methodology that invites people to collaborate and motivates people to excel must work.
  • Frequent inspection and adaptation is the most effective concept in any complex and high risk activity
  • R&D and PO in agile context will eventually collaborate, because they are forced to play TIT for TAT with fast feedback cycle. Another phenomena – longer they work together, more they are willing to collaborate. This is well known fact from gaming theory.
  • R&D folks gets motivated by feeling truly in charge of things they know the best
  • “Control” is achieved by clear accountability for everyone’s work
  • Openness allows constructive peer pressure and unbiased norm for the complete team