Project Estimation Geometry
I recently read Joel Spolsky's On Software after reading Erick Sink's On The Business Of Software; both are books that I highly recommend. They both offer funny and informative insights from their respective blogs in book form on their various takes on the business of software development. Spolsky's discussion of the McDonalds theory of software development is spot on. Spolsky says that from McDonalds' University all the way to the exacting nature of the uniform delivery of what really has become a bland (albeit loved) product, McDonalds has really detailed how to create a uniform product across independently owned franchises. This would equate to an exactingly specified software development methodology that in theory could produce software in a predictable and uniform way. Spolsky says it can't and it won't. He's right.
The theory is if that senior developers with a track record of success write in exhausting detail how to build software, anyone can follow that methodology and get a uniform result every time. This theory is inherently wrong. Every software project is vastly different from every other and requires so many intangible decisions that ultimately lead to success and failure that it is impossible to create a methodology map that ensures a reliable, predictable, and manageable success every time. On every project, it comes down to how the practitioners on that project choose to spend their time on the tangible and intangible elements of the project. Hence, a single methodology dogmatically adhered to simply becomes an albatross around the neck of those forced to follow its tenants.
In this article, I will try to tell you a way you can tackle one aspect of a project—software estimation. I will tell you how I estimate projects and the geometry involved, and if unfettered by poorly judged external forces this way leads me to a successful delivery. But, the most important thing is that these are general guidelines and there is simply no substitute for individuals and teams that can succeed in their own way. Individuals and individual teams must be permitted to use their own judgment, for there is no more desirable outcome than a delivered and well received product. (Until software becomes an assembly-line process and by then, God willing, I'll be horizontal.)
Disclaimer: Any misunderstanding of Joel Spolsky's meaning or intent is my own.
Geometry applied in the context of software development project estimation refers to the size, shape, and relationships among all of the elements of a software project, including those practices whose direct result are tangibles and those time slots that are essentially intangible but necessary in some way. Many estimators get some of the tangible parts of estimation correct, but it is all of the intangible aspects that are often overlooked and result in delays, disappointments, and undelivered software.
Tangible elements (often called deliverables) can include project plans, requirements, detailed designs, the software, and time spent testing, debugging, documenting, and launching software. Many people know about these elements and some include these elements in estimations. Sadly, some do not include all of these elements. Intangibles such as water cooler time, illness, bad requirements, wrong requirements, changed requirements, unforeseen hard bugs in your code or the tool provider's code, learning, white board time, unit tests, arguments, meetings, distractions, failed or replaced hardware, absenteeism, car accidents, sustainable productivity levels—getting in and staying in "the zone"—reading FoxNews.com, checking out research.microsoft.com and more, are all too often overlooked, dismissed, or denied. But, these are the elements as likely to happen as coding that chew away at the schedule, willingly or not, and is what is most often overlooked in a project schedule. To account for the tangibles, methodologies like RUP were invented. To account for intangibles, methodologies like XP and Scrum were invented. But, again there is no substitute for having people who have demonstrated an ability to deliver.
Tip: Software projects that succeed often have a single-minded determination and focus on the number one deliverable, the software.
It is the tangible and intangible elements of software development that make up every single project's geometry. Thus, you can break all of the tangible items known down to minute tasks and try to add them, but this will not lead you to a singularly reliable estimate. For how are they added together, consecutively or concurrently? A Gantt chart may suggest the formation and consequently a starting point and ending point, but sadly Gantt charts can only show what is known at a point in time, and almost every Gantt chart (and project schedule I have seen) has left out the intangible elements of the estimate. And, if you include the intangibles in the estimate, it is likely to lead to the termination of the estimating-slacker. Failure to include intangibles in the project geometry is ignoring the humanity in the humans doing the work.
Unfortunately, believe it or not, it is all of the ways you spend your time that will dictate how long it takes to deliver the software, and few if any of these tangible elements should be negotiated out of the schedule and no amount of bullying, threatening, or carrot-and-stick approaches will mitigate the intangible elements of software-time in any sustainable or predictable way. This is why software projects are still very unpredictable.
As an alternative approach, one can try complex algorithms such as CoCoMo II invented and detailed by Dr. Barry Boehm—see CoCoMo on Wikipedia or read Dr. Boehm' s seminal book for more information—but I have found CoCoMo estimates to suggest that successful projects I have completed should have required as much as two times the amount of time and five times the number of resources (people) as the project actually took. Again, the practitioners will ultimately decide the outcome, but tools and point-in-time guesstimates [sic] can be useful and insightful starting points as long as these are made by the practitioners actually tasked to do the work.
The Process I Use
I can unabashedly tell you that, after twenty years, I have a way that I estimate projects based on dozens of projects that I participated in or observed closely. I can also tell you that some of these elements have met consistently with open or barely contained hostility. You have been cautioned.
Every manager needs to be able to predict outcomes and determine resources. Unfortunately, at the onset of a project this is almost impossible to do in any meaningful way. Too little is known about a project (of any size or complexity) at its onset to permit these estimates to be reliable. This is especially true if the estimates come from above. Manger deadlines are the worst kinds of guesses, uniformly, and almost always are taken to be schedule by mandate estimates (what programmers refer to as death marches [Yourdon, Death March]).
Note: If there is a live-or-die quality to a schedule—as in people will live or die or companies will live or die, and these should be very rare—a merciless willingness to trim unnecessary features and a do the hard stuff first approach can work. This should never, ever, be routine practice, and people know the difference. However, in such a circumstance you need a superhero and said superhero should be richly rewarded.