Lies, Statistics, and the PMO
Here at the words-and-facts factory, we spend our days juggling statistics: either those we've generated ourselves by developing research questionnaires, or those we've sought out from other purveyors of research questionnaires. After ten years or so of this kind of activity, one becomes an extremely skeptical consumer of statistics. Like Benjamin Disraeli, who famously railed against "lies, damned lies, and statistics," the more familiar you are with all the ways that statistics can mislead, the more critical eye you cast over any use of statistical information.
Project management is a field in which much research remains to be done. In the past five years, the number of studies related to PM has increased, and that's good. Here are just a few of the studies that have been done recently that address the value of the PMO:
CIO magazine with PMI: http://www2.darwinmag.com/learn/research/surveyreport.cfm?id=58
But not all research is created equal. I often see questionable assertions made about project management based on sketchy research data. So, be a careful consumer of PM research studies: If you are seeking to prove the value of a PMO, either by collecting external research to make your case, or by creating internal research to justify your existence, here are six important points to keep in mind.
1. What's in a name?
The term "Project Office" suffers from having a welter of definitions. To one company, it's a single person on a mission to promote a methodology. To another, it's a bank of project mentors that fans out across the enterprise to teach people how to fish. In some large capital project contractors, a Project Office might be a kind of war room for a single project. Some companies with mature project management practices have a true organization called a Project Office, with a Director and support staff and direct reports who oversee everything from project manager training to software selection and implementation, but this is still rare and usually confined to very large enterprises. (About 3% of companies have this level of Project Office organization, according to Center for Business Practices research.) So, even to discuss the topic, you first have to find out which Project Office paradigm the participants are coming from.
This all-over-the-map character isn't a bad thing: it's an essential part of the nature of a concept under development. While companies figure out how to make a concept into a set of workable practices, we can expect those practices to take many forms. However, in one way, the concept itself can be hurt when in-practice Project Offices are so variable. In the project management field, there isn't that much research being done, so whatever studies are done tend to be widely quoted (and sometimes misquoted: see Item 4 below.) One bad piece of information—an erroneous assumption drawn from confusing results to a badly-designed questionnaire—can have immense impact.
So, if you are creating a survey about PMOs, be sure to give the respondents a way to indicate what PMO means to them. A checklist of types of PMO organization with "check as many as apply" is one way to do this.
Another, related question might be length of time since implementation. A one-person, brand-new PMO cannot possibly have had time to have much of an effect on organizational factors like career planning, portfolio success, and the like. Always offer the "not applicable" option in multiple choice questions.
2. The question makes the answer.
In one recent PMO study I've seen, the results indicated that, for the majority of responding companies, those with a PMO had higher project failure rates.
There are a couple ways to explain this result. Maybe this entire group of companies surveyed were all really, really bad at PMO implementation. But that seems kind of unlikely. More likely is that the question didn't do much to elicit a description of reality. For example, in many companies no one is keeping track of project failure rates. Therefore, the known failure rate is ... zero. Along comes the PMO and begins collecting metrics. Aha! Project failure rate is, say, 15%. Does this mean failures went up? No, but that's what it will look like if all you ask is pre-PMO and post-PMO failure rates.
Survey design is both art and science and a poorly-designed questionnaire can obfuscate more than it illuminates. It would have been good to know things about these companies like:
- Did they take a baseline measurement of failures before they implemented the PMO?
- How long have they been tracking failures?
- What failure prevention measures have been undertaken by the best-performing companies in the study? By the worst-performing?
The problem of course is that, the more precise your questions are, the fewer of them you can ask. It's a law of questionnaire development that few people will respond to a survey questionnaire that comprises more questions than will fit on about one page. People have to be highly motivated to respond to longer surveys. That's why it's good to have some sort of pre-qualified survey group. Our organization does this by soliciting members for a research network. They are a self-selected group of people who are interested in both participating in research, and learning from it.