Fixing the Foam: Preventing Disaster, Getting Clear Picture

Armando Oliu, Final Inspection Team lead for the Shuttle program, speaks to reporters in 2004. CREDIT: NASA/KSC.

This story is Chapter 3 in an 11-part series by Florida Today.

NEW ORLEANS - The stuff that changed everything at NASA feels like the foam in life jackets on a fishing boat. It's super light, mostly air. Hold it in your hand and it defies common logic. How could a chunk of this foam bring down a mighty space shuttle?

Even the engineers who spent decades working on the foam, which prevents ice from growing on the outside of the fuel tank that holds a half-million gallons of two of the coldest substances on Earth, never feared the kind of catastrophe they saw in 2003.

They'd seen foam come off the tank for more than 20 years, usually popcorn-size pieces, and they had long ago decided they were only dealing with a maintenance problem. How could the equivalent of a foam beer cooler hurt an aircraft? But the shuttle turns reality inside out. The environment it flies through those first nine minutes from Earth to space is a different realm, and for a few fleeting seconds, even foam becomes a potential killer.

Steve Holmes is still coping with that new reality. Holmes, who lives with his wife, Pam, and two kids in Huntsville, Ala., is a shuttle fuel tank engineer who spent the last two years trying to figure out why chunks of foam insulation come off the external fuel tank and how to stop it.

NASA had tried to solve the problem about a dozen times with varying success, but never eliminated it. Now, with the foam implicated in the loss of a $2 billion spaceship and seven people's lives, trying wasn't good enough. Holmes and the tank team had only one option: success.

The prime fix was obvious -- get rid of the two big wedge-shaped ramps of foam that protect a V-shaped strut connecting the tank to the orbiter's nose.

One of the two triangular foam blocks ripped free of the tank, probably because of an air pocket or other flaw hidden inside the hand-molded ramps.

With the shuttle roaring upward, the lightweight foam slammed into the spaceship with the force of a ton of bricks -- literally.

But that wasn't the only fix. Post-accident testing of shuttle heat-shielding tiles and wing panels showed the materials were not as strong as NASA believed. Indeed, pieces of foam as small as a cupcake -- something a tenth the size and weight of what hit Columbia -- could be fatal.

So Holmes and hundreds of his co-workers had to stop anything bigger than three-hundredths of a pound -- a square about the size of a breakfast muffin -- from breaking off the tank during flight. It was a tall order beyond what NASA believed possible.

Since 2003, Holmes has shuttled from the tank's design center in Huntsville and the factory near New Orleans. He has missed soccer games and school functions, a lot of what's happening in the lives of his children, Madeleine, 14, and Thomas, 6.

"I've tried to make the birthdays and major holidays," said Holmes, a self-proclaimed space cadet who went to work for NASA in 1989 and joined the external tank project a decade later.

Members of the tank team went to work early every morning and came home late every night.

They constantly passed the posters plastered on hallways featuring shuttle commander Eileen Collins holding her grinning daughter, Bridget. The slogan: "Are you ready for us to go? Think safety."

No one publicly pointed fingers at the men and women of the tank program, but the team felt the pain anyway. And they were determined never to feel the agony of the Columbia seven's loss again.

"A lot of people did a lot of soul-searching about what could have been done different," Holmes said. "This is something that could have been taken care of a long time ago."

Among the long-accepted problems that had to be fixed: a foot-wide strip of extra-thick foam near where the round barrel of the tank meets the rounded pointy end. The ridge consistently shed Frisbee-sized pieces of foam.

Across 95 percent of the tank, robots spray a near-perfect layer of the white liquid foam, which hardens into the orangish color people are accustomed to seeing on the launch pad.

In hard-to-reach places with bumps, grooves or other odd surfaces, technicians spray or pour the foam into place. That's where most flaws hide.

The ridge -- called a flange -- is one of those handmade spots and one of the tank's weak points.

Engineers, tank builders and foam sprayers worked the problem side by side; everyone had ideas.

They sliced and diced another tank's foam looking for air pockets, cracks and defects.

They chilled the inside of metal panels coated with foam, trying to re-create the extreme temperatures the stuff endures at the launch pad and during flight.

Fixes designed by engineers were perfected in countless practice sessions by men who've been coating the tanks for 20 years.

The "sprayers" would work over and over on new techniques, experiencing failure upon failure, before finding something that worked.

One theory was that tiny cracks or holes could let gas get behind the foam, then rapidly expand as the temperature skyrockets during launch.

The resulting pressure pops foam off in chunks large and small. Experiment after experiment ended in frustration. Nothing worked.

Then, one day, epiphany. Spraying a test panel, and watching it react, the group figured it out.

Gas crept through the threads of bolts holding together two parts of tank. The bolts were hard to spray around, and gas could sneak into tiny paths around the threads.

"I think we've got it!" one of the guys shouted.

Repeated tests proved they were right. Engineers went off to figure out how to stop it.

The answer: Turn the bolts upside down so they were easier to reach with foam spray guns and fill the bolt threads with material to prevent gas from creeping past.

The sprayers practiced and practiced before they sprayed real foam on the real tank.

Finally, just before the New Year, the managers, engineers and factory workers celebrated as they loaded ET No. 120 onto a barge that crossed the Gulf of Mexico, sailed around the southern tip of Florida and ultimately drifted into a basin at Kennedy Space Center the first week of January.

Nobody relaxed, though. Discovery's tank was delivered, but every tank has to be perfect now.

"We shipped one tank, and we're getting ready to ship another one," Holmes said shortly after shipping the first tank. "It's going to be another three or four months before we're going to see any more breathing room."

KENNEDY SPACE CENTER
Engineer keeps cameras focused on shuttle

Armando Oliu, whose films first spotted the debris that doomed Columbia, is now in the middle of making sure the tank is fixed.

The trick for Oliu is balancing new camera shots against making sure he at least gets what he always had gotten before.

Oliu's team upgraded to high-definition television cameras. They repaired or replaced giant lenses -- really telescopes -- that are attached to some cameras along the coast north and south of the pad.

Cameras will be on boats, airplanes and even the shuttle itself. Never before have shuttle engineers had so many looks, so many angles, such detailed imagery of a launching spacecraft.

Once Discovery's in space, Oliu and

a team of engineers will spend more than two days studying thousands of frames.

That's just if everything looks good.

Oliu and colleague Bob Page will brief mission managers each of the first few days of the flight on what they see.

This time, they'll have the best of the best equipment to do the analysis. A powerful bank of computers hums in the film lab now. Giant screens and banks of monitors are everywhere.

Oliu painted the floors black on the advice of film experts because it would reduce the glare so people can see better -- the same reason movie theaters are black or very dark colors.

They bought a projector used at modern movie houses playing digital versions of Hollywood films.

"There are a few extremely rich individuals who would have this at home. Bill Gates, maybe," Oliu said.

Oliu never expected to become so engrossed in cameras. He came to NASA after the Challenger accident in 1986. A systems engineer, he spent time working on the external tank and the International Space Station before being asked to join the ice and debris team.

The engineers not only are experts on various shuttle systems; they're known for excruciating attention to detail in reviewing films for trouble.

They also do one of the most dangerous jobs in the program other than astronauts.

As the shuttle is fueled before launch, they walk up and down the evacuated 300-plus-foot tower looking for debris, cracks in the external tank's insulating foam, large chunks of ice, anything that could hit the delicate orbiter during liftoff.

Now, Oliu is in the middle of the most important inspection ever done on a shuttle.

The hours have been longer. The travel has been more extensive. But Oliu's team tries to balance work and family.

Oliu works hard. His wife, Jennifer, understands. But he's no workaholic.

"I have a level of sacrifice to make. But first is my family," he said. "I'll work to get the job done, but I'm not going to destroy my family for the space program. . . . Every individual's got to weigh that pull from work with what's right for the family."

He met Jennifer at a NASA picnic. She was a nurse who worked with the wife of one of Oliu's buddies. Armando and Jennifer hit it off and got married in 1997.

A few years later, Victoria was born. She's almost 6 now, a kindergartener in Cocoa Beach. She knows her dad works on the shuttle, but he says, "she's just getting old enough to understand. She still thinks I'm an astronaut, for God's sake."

Oliu's job, the investigation, the return-to-flight effort have put him in the spotlight. He has been on television several times. She notices a rocket lifting off or something space-related on the tube at home in Rockledge and she asks, "Daddy, are you going to be on TV again?"

"Hopefully not," he jokes, somewhat uncomfortable with being put forward to reporters to represent the work of dozens of others.

Off camera, he's still making the kind of frank, no-nonsense calls people have come to expect from him.

If he thinks your idea is bad, he'll say so. No hard feelings. Get the job done.

There's been plenty of that since Columbia.

As incredible as the shuttle tracking camera system always has been, it's not good enough anymore. The loss of seven astronauts exposed a host of seemingly innocuous problems that, left unfixed too long, turned deadly.

Broken tracking gear, out-of-focus cameras, glitchy remote control systems and human error all contributed to pictures that Oliu complained to bosses were "simply unacceptable" and "unusable" for engineers trying to make life-or-death decisions.

In the years since, Oliu and others have put every camera and related piece of equipment under the microscope.

Some wanted to replace almost everything with the latest technology.

For instance: There's a camera north of the pad near a place called Shiloh, the one and only camera that captured the leak of fuel from the Challenger booster rocket and helped solve the mystery of that accident. Some wanted to shut it down. A new one nearby gets the same look.

"Let's not take that one out just yet," Oliu said.

Why? Because shuttle engineers know this: You never know. Oliu wants to preserve at least the images that engineers have come to expect, at least for now. Once he sees the new cameras work, he'll feel safer.

"Let's not start deleting cameras," Oliu said. "We know what we can get out of this camera. We've gotten used to certain views from certain cameras. If you do everything brand new, it's going to be a mess."

Published under license from FLORIDA TODAY. Copyright ? 2005 FLORIDA TODAY. No portion of this material may be reproduced in any way without the written consent of FLORIDA TODAY.

         Fixing NASA: Complete Coverage of Space Shuttle Return to Flight