Retirees help maintain Hubble after 30+ years in orbit

Imagine trying to troubleshoot and repair an 8-track-tape player or Atari electronic arcade/video game from the 1970s. You might find yourself wishing for a visit from Christian Slater—as the phantom television character "Mr. Robot"—to even know where to begin.

Now, imagine the challenge of troubleshooting and making repairs to a vastly more complex technological wonder—NASA's Hubble Space Telescope (HST), which was first funded as an agency space-exploration mission in the 1970s. Having now exceeded even the wildest expectations for its lifetime and service to science—more than 30 years and counting—this remarkable spacecraft/scientific instrument does hit occasional snags in operation. When that happens, says NASA HST Deputy Project Manager, James Jeletic, the telescope will switch to a so-called safe mode, sometimes close the flap over its primary mirror and halt all but basic functions until NASA engineers can form a "tiger team" to find and fix the problem.

But for whenever electro-optomechanical hiccups strike HST (and other NASA space craft), the agency also tries to keep on call some of its retired personnel—experts who may have helped design original systems or their replacements now causing headaches as well as former staff and contractors with more system-wide knowledge.

They are people, Jeletic says, with institutional knowledge who can perhaps help tiger teams quickly locate key HST technical documentation, for example, or whose recollections of decision-making, perhaps not recorded, that can shed light on troubleshooting and problem solving.

"When we have a serious anomaly we ask, who were the experts in this area?" Jeletic says.

One such expert is NASA retired electrical engineer, Ron Barasch, who joined the HST project two years before the telescope was launched in April 1990. For HST and other NASA projects, he says he spent most of his career designing and building spacecraft command and data handling systems.

In mid-June, after being retired from NASA for nine years, Barasch says, "I got a phone call from the HST mission operations manager. The telescope had gone into a safe mode. It's a self-protection feature where the telescope is telling us, ‘Something went wrong, and I am just going to go into a safe state until you figure out what's wrong and get back to me when you're done, ok?'"

The tiger team starts with the known facts. In this case, for some reason, Barasch says, the main spacecraft computer and payload computer did not complete an electronic "handshake."

"And if one of them does not respond," Barasch says, "the system goes into a hold—the telescope shuts down the science instruments and places itself into a safe condition. The ground has to figure out what's going on." Safe mode, he adds, is but a symptom of a problem.

For the mid-June problem, the chief suspect was a high-energy particle flipping one bit of system memory. This cosmic radiation is a constant hazard in space, particularly for sensitive electronics. Barasch says the system has built-in capabilities to detect and correct one error, that is, a one-bit flip.

"If it has a two-bit error, the circuitry can detect it but not correct it and initially that's what the tiger team thought happened," Barasch says.

Because of the pandemic, Barasch and the rest of the tiger team worked together in cyberspace toward a solution that could be implemented in the HST control center at the Goddard Spaceflight Center in Greenbelt, MD, USA.

For big repairs not solvable from Earth, Hubble was designed to be serviced by astronauts in space. Five space shuttle missions had been tasked with repairing, upgrading, and/or replacing almost all systems on the telescope. On the last repair mission in 2009, the science instrument command and data handling system was replaced—with the system that caused the June troubles.

"Ron's an expert on that particular unit," Jeletic says."

"My name came up because of my history with the system," Barasch says.

Ron Barasch (right) receiving an award from then HST project manager, Frank Cepollina, shortly after a Hubble servicing mission by a NASA space shuttle in 1999. Credit: Ron Barasch

With Barasch onboard, the "tiger-team plus," as the HST project calls it, started poring over recent events with the HST science instrument command and data handling system. They used a so-called fishbone analysis to diagram system operations and, through a process of elimination, begin to home in on the problem. Eventually, he says their analyses led to two possible sources of error and, as luck would have it, the team could use the same fix for either problem.

Essentially, they took advantage of redundancies in the HST system to move the entire command and data handling system to one side only of the spacecraft. The problem was solved and the HST was once again operational.

"We obviously have schematics and other documents around to guide us," says Jeletic, "but you never really know what important questions can come up from 30 years ago." He recalls a situation involving voltage tolerance limits in HST electronics systems.

"Why did they pick those limits? That's where Ron comes in and can really help us out," Jeletic says.

One of Barasch's former NASA colleagues, retired mechanical engineer John Decker, says project teams at NASA for HST and other missions "keep contact lists with the names and numbers of people that they could call out in the event of an anomaly." It might be thought of as NASA's deep bench.

And Decker would know. He agreed to remain on the contact list even after he retired eight years ago, and he received one such call in March. The HST had switched over to safe mode—but failed to automatically close the giant flap of the barrel of the telescope.

Decker explains that whenever the flap on the barrel of HST is opened—an aperture door he helped design—the next step is to send a command for autonomous closure in the event of a safe-mode step down of operations.

But on 7 March, when HST did fall into safe mode, he says, mission control "noticed that the door didn't automatically close and they said, ‘uh oh, that is not good.'" Still, it was not as critical of a situation as it could have been, he says, because the HST's pointing system was still functioning properly, keeping it away from the Sun, which could severely damage the telescope if it ever shined down the barrel.

All of this happened at around 4 am on a Sunday morning, Decker says. "At 7 pm that night it occurred to them to call me and see if I had any ideas." Soon, he says he found himself in a "nice reunion" on Microsoft Teams with his former colleagues who are still actively supporting the Hubble program.

Both Barasch and Decker say they were surprised at how much came back to him. "I had been so immersed in that system 30 years ago that it's still pretty much in the memory banks," Barasch says. "I could remember the details of the circuitry, the operational signature, how long the motor would be at X speed—it was still very familiar to me."

In fact, Barasch says, his recall became so vivid he was able to recall relevant documents and how to locate them, as well as the names of the people involved during the development."

"One of my most vivid memories before the launch," Decker says, "was sitting in a room for months trying to think of everything that could possibly go wrong with every component of the Hubble mechanical system and have a plan to address it."

Barasch says it's important to remember why engineers like he and Decker devote such effort to making sure the Hubble Space Telescope can continue operating for as long as possible.

"We build tools for the scientists to do their research," he says. "And if those tools don't work, scientists who have spent years working on their proposals might have to start all over. And if the instrument fails altogether—all of those researchers' hope, the papers they were going to write, the discoveries...."

But of course, the Hubble Space Telescope did work and NASA hopes to keep it going for more years to come. Barasch says HST has processed 1.5 million images. "It makes me feel good that every single one of them came through my system," he says. "And every single one of them came through my aperture door," Decker adds.