The problem of actual costs of space infrastructure overrunning estimated costs - often by large amounts - is one of the most chronic challenges in our space community. In the July 16 issue of Space News, for instance, articles addressed costs in excess of estimates for projects including NASA's Orbiting Carbon Observatory and the Kepler telescope, and the U.S. Air Force's Space Based Infrared System. These are but three examples of many others. In fact, overruns are almost the rule rather than the exception - when was there last an article about a project within its estimated budget? Or, for that matter, under budget? But before heading to the woodshed for 30 lashes, a word or two of perspective may be in order.

 

Ideal cost estimation, the perfect process that encounters no technical, time or other problems, has eluded all humankind. Stanley Engerman of the University of Rochester and Kenneth Sokoloff of the University of California, Los Angeles, have evaluated a dozen major U.S. public works projects, including construction of the Erie Canal, the interstate highway system, the original Superdome and Boston's Big Dig. The ratio of actual to projected cost in these examples ranges from about 1.1 to 20; for seven of the 12 the ratio is greater than 2.

 

In a November 2006 report, "Space Acquisitions: DoD Needs to Take More Action to Address Unrealistic Initial Cost Estimates of Space Systems," the U.S. Government Accountability Office (GAO) found current cost estimates had increased in the range of 44 percent to more than double original estimates for six ongoing space programs. Similarly, a 2004 Congressional Budget Office (CBO) report, "A Budgetary Analysis of NASA's New Vision for Space Exploration," found wide disparities between projected and actual costs in more than 70 space projects undertaken between 1977 and 2000.

 

Is the problem endemic to public works - the canals, roads and space projects undertaken and managed by government? Is there no problem in private projects? Hardly - just ask commercial real estate developers or the neighbors down the street renovating their home or building a new one.

 

Neither is the problem unique to the United States. Large cost overruns plagued construction of the Euro tunnel, the reconstruction of London's Wembley Stadium and the building of a large oil pipeline from the Caspian Sea to a Mediterranean port, a project undertaken jointly by Azerbaijan, Georgia and Turkey. In a 2003 study of 258 rail, road and airport construction projects in 20 nations, Danish researchers at the Delft University of Technology in the Netherlands found substantial cost escalation to be the rule, not the exception. And as readers of Space News well know, cost estimation challenges all international space organizations, not just our own.

 

The reasons for these results are several:

 

§   Engineering the uncertain: Uncertainty is inherent in new technology, making it difficult to estimate costs of unknowns. Professors James Quirk and Katsuaki Terasawa have dubbed the problem "pioneering bias in cost estimation" - the tendency for engineers and other experts to mis-estimate the cost of developing and testing out new technology. Of course, unexpected problems also arise even in dealing with proven technology such as plumbing repairs in old houses, or a highway corridor encountering a sacred ancient burial ground. Uncertainty by itself does not imply bias in one direction. But Quirk and Terasawa find that the tendency is to underestimate expected costs, which can lead to cost escalation, delays and outright cancellation of projects.

 

§   The difficulty of monitoring to avoid excessive profits, waste, fraud and inefficiency: These suspicions are at the very heart of contracts, whether for major weapons procurement, highway construction or that extra bathroom. The problem largely arises from asymmetric information and the inability to monitor a contract. We - or governments acting on our behalf - enter into a contract because the contractor has the information, specialized knowledge, equipment and other required inputs. In addition, it is difficult to audit the contractor's every action. For these reasons, what constitutes excessive profit, waste, fraud and inefficiency is often a matter of "they said, we said; we did, they didn't."

 

§   Pressures to underestimate costs: Such pressures can arise among competing programs within an agency or in the agency's relationship in advancing proposals before congressional funders. The GAO has found repeatedly in its interview-based and other studies that programs "produce optimistic estimates in order to gain approval for funding." The Danish researchers mentioned above studying transportation infrastructure found "grossly inaccurate" estimates over 70 years' of data, prompting them to write a strongly worded conclusion: The inaccuracy was by and large due to "strategic misrepresentation, that is, lying." A researcher at the Harvard Business School offers a more gentile description of this tendency - which is prevalent in the private sector as well - dubbing it "self-serving bias."

 

§   Reputations don't matter: If a contractor repeatedly lowballs estimates and incurs large overruns in actual costs, customers can spread the word. But if choice among contractors is limited, or government program managers repeatedly underestimate costs but are hard to fire, then the market discipline of a good reputation is limited. Consolidation has lessened competition within the traditional aerospace industry, blunting the effect of a bad reputation or, as important, rewarding a good reputation.

 

So is there anything to be done to improve cost estimation? Beware of any simple answer. Cost estimation and its related domain, contract theory, have some counterintuitive results. For instance, there are circumstances under which an incomplete contract (and provisional cost estimation) is a good idea. In this case, the parties to a contract prefer not to iron out all the details - that is, all the variables that may affect the conditions of the contractual relationship. In a study of the procurement of aircraft engines, researchers found that the cost of writing and enforcing a contract to cover all contingencies and eventualities in long-run projects was simply too expensive. The best contract leaves open some issues for future negotiation.

 

That said, some rules of the road include:

 

§   Demand information about a project's benefits, not just its costs. Imperfect cost estimates or excessive actual cost overruns still may result in programs that are effective in serving the public. In other words, cost growth per se is not an indicator of a socially undesirable program, nor would a project that is completed within its anticipated budget necessarily be an indicator of a socially desirable program. Decision-makers should require objective estimates of the putative future benefits of a project as well as its costs.

 

§   Discourage precision. Program managers, agency heads, budget managers and Congress should refuse point estimates and narrow ranges of estimates. They should tolerate and even demand uncertainty, requiring agencies to apply decision software and analytical approaches that transparently communicate confidence intervals and other related measures. What are the greatest risks in a project and how will they be managed? Much like a mutual fund manager, can government better act on behalf of the taxpayer to demand explanation of the upside and the downside of public investments?

 

§   Keep the fox out of the henhouse. Objectivity in cost estimation is hard to obtain. Engaging teams of private-sector analysts, provided they are individuals with their own professional reputations at stake, may be workable. Put their names - not their companies' - on the public report; much as under the new Public Company Accounting Reform and Investor Protection of Act of 2002, known to most of us as Sarbanes-Oxley, chief executives now personally sign and certify financial reports.

 

§   Maintain a public track record. Which government program managers continue to validate projects with large cost overruns, and which deliver better ratios of actual to projected costs? When better ratios are obtained, we need to make national heroes of our government program managers, who all too often receive too little credit for good work but much notoriety when something goes awry.

 

§   Reward good results. When projects come in below budget, savings are siphoned off for other things. For NASA centers, this perverse incentive could be eliminated by establishing a virtual bank in which projects that come in under projected cost keep the savings. These funds could be carried forward within the scientific domain - for example, savings on one telescope go into a pool to finance the next-generation telescope. Several years ago, NASA funded researchers at California Institute of Technology and Purdue University to experiment with how to design such a scheme. It's worth revisiting.

 

 

Molly Macauley is a senior fellow at Resources for the Future. Prabirendra Chatterjee is pursuing a doctorate from the Department of Economics at the University of Washington and was a summer intern at Resources for the Future. This article is based in part on a presentation by Macauley at the American Meteorological Society's 2007 Summer Public Policy Colloquium.