NASA's Johnson Space Center told TechRepublic the whole story behind one of the most important tools it uses to train astronauts: virtual reality. Learn how the agency pulled off VR's greatest triumph.
RECOMMENDED FOR YOU
For the first time, astronaut Doug Wheelock hung on to a handrail outside the International Space Station (ISS), and looked down at the Earth.
"I know exactly where I am," he thought. "It looks just like the VR Lab."
That familiarity is a quick comfort not only for Wheelock, but for many astronauts past, present, and future, floating hundreds of miles above the Earth, trying to get a job done in one of the most hostile working environments imaginable.
In space, an astronaut's next minutes are never guaranteed. They have to adjust to the drastically modified rules of physics, and to a calmness and a slowness that masks danger.
Inside Wheelock's spacesuit, he was aware of the frightening silence of space, broken only by the voices coming in through his communications cap, the ventilation system of his suit, and his own heartbeat. Outside, temperatures were fluctuating between 300 degrees Fahrenheit and negative 300 degrees in 90-minute cycles.
Space is beautiful—but unforgiving.
Wheelock knew it would be like this.
His preparation from the Virtual Reality Lab at NASA's Johnson Space Center was both mental and physical.
As the spacewalk—or extra vehicular activity (EVA)—turns 50 in 2015, its history is inextricably linked to the development and use of a technology that's long been derided as a toy.
For as long as virtual reality has existed, even conceptually, its constant companions have been hype, gimmickry, and unmet expectations. While the commercial VR industry has waited for another shot at world domination, Johnson Space Center's VR Lab doggedly worked on ways to better train astronauts for one of the most dangerous excursions of their lives.
Virtual reality has matured and developed into a mission-critical training tool. NASA has pushed a number of different technologies to their limits to pull it off, without the glare of industry expectations or the pressure of packaging it into a consumer product. Their solution isn't flashy, but it's immensely powerful. And it works.
The main reason for the success of NASA's VR experiment is ridiculously obvious—there's no other way to replicate space here on Earth.
By the scruff of your neck
Johnson Space Center sprawls across nearly 1,700 acres of land near Nassau Bay, southeast of Houston, Texas. There are more than 200 buildings, all randomly numbered. The Virtual Reality Lab is in Building 9, which is mostly indistinguishable from the rest.
Through gray metal doors is a room that looks like a standard-issue office, with off-white walls and pressed wood desks. But, there are posters from different space missions everywhere. A section of the lab is set off by a narrow open doorway and two half walls made of plywood. The walls are draped with black sheets, and smattered with a few handfuls of glow-in-the-dark stars—the kind kids put on the ceiling above their beds. The stars were Evelyn Miralles' doing. She's the lead engineer, and a full-time staple in the lab for the past 16 years.
There's a tall cart with electronics, two office chairs EVA trainees can sit in, and on each side of the space are two metal cubes suspended from the floor and ceiling by wires. The wires look like a web, and that's why those cubes, which are really robots, are called Charlotte.
Then there are the head-mounted displays (HMDs). They're black with a band around the back, and are always a work in progress. The previous week, one of them got a revamp. The OLED displays now show 1920x1080p per eye at 20 to 60Hz, depending on the load, task, and complexity of the scene being rendered.
Put on one of the HMDs and you're floating above the truss of the ISS. The Earth spins down below, and the moon is small and barely visible over your shoulder.
Welcome to the closest experience there is to floating above the Earth.
VR face is a term for the dumbstruck look people get when they experience virtual reality. It's that slack-jawed mix of awe and obliviousness that other people are watching them while they're buried under a mound of plastic and sensors. The lab's rig also includes gloves with sensors, so when users hold up their hands and flip them around a few times, they're looking at the hands of an astronaut.
In the lab, VR face is really hard to avoid.
Miralles and team can fly whoever is sitting in the VR chair around the ISS, up close to the sides of the structure, even just outside the upside down dome observatory, the cupola. The lab's founder Dave Homan equated it to getting picked up by the scruff of your neck.
For a visitor passing through the lab, that quick trip around the ISS is a brain-melting party trick.
For the astronauts, it's critical preparation toward completing a complicated and dangerous assignment.
SEE: Video: 3 big takeaways for the future of virtual reality from NASA's VR Lab
A solution without a problem
For NASA, VR started out the same way it did for a lot of other folks in the 1990s: "Hey, here's this thing, is there anything we can actually do with it?"
In effect, Homan was the one asking that question.
He was working at NASA as an engineer, and had developed simulations and graphics to support analysis for tasks like maneuvering the space shuttle's robotic arm.
In the early 90s, NASA headquarters hosted a conference of sorts on virtual reality in Wyoming. Homan was the only person in his division who was interested in going.
So, he went to the conference and started thinking about how to use virtual reality in a "legitimate" way—perhaps for astronaut training. To help him figure that out, he recruited Brad Bell, who had worked with him since 1985.
RECOMMENDED FOR YOU
Initially, Homan and Bell thought they'd use VR to replace the training for the high dexterity tasks astronauts were practicing in the giant pool they called the Neutral Buoyancy Lab—skills like using tools, or loosening bolts. It didn't take long to realize that the pool was still better suited for that type of training.
However, that legitimate use they'd been looking for materialized in the form of a repair mission to the Hubble Space Telescope. Soon after the Hubble launched in 1990, excitement gave way to disappointment. The $2.5 billion telescope was blurry. The fix would be STS-61, a 1993 mission to correct a problem with the mirror and scrub the proverbial egg off the face of the telescope.
One of the first problems Homan wanted to address was how an EVA crew member and robotic arm operator had nowhere to practice their communication. If a spacewalker was upside down on the end of the robotic arm, it wasn't really clear what a command like "move me up" meant, and Homan said valuable time in orbit was spent sorting that out.
Virtual reality brought together both the simulations and graphics NASA had been working on, and the live EVA crew members and robotic arm operators.
"They could actually go through the entire EVA scenario, and work out what they meant by 'move me up' and 'move me down' and what coordinate system they were talking about," Homan said.
Using virtual reality also meant that training didn't have to conform to the physical restrictions of the pool.
In preparing for the repair mission, NASA had assembled a replica of part of the Hubble in the Neutral Buoyancy Lab, but had to split it in half as the whole telescope couldn't fit in the 40-foot-deep pool, and the astronauts had to practice working at its top.
Homan said that when they configured the telescope, arm, and orbiter in VR, they realized the arm couldn't actually reach the spot where the repairs needed to be done.
"You wouldn't want to find that out when you got out to orbit," Homan said.
They changed the configuration, modified it in the pool, and eventually, used that configuration to make the repair.
By 1994, the VR Lab introduced Simplified Aid for EVA Rescue (SAFER) training. Every astronaut is equipped with a jetpack, and though it hasn't happened before, they still have to anticipate the chance that they'd become detached and start free floating in space.
Yes, it's kind of like the movie Gravity, but Hollywood has a different version of physics. In real life, traveling in orbital space requires a different approach than flying in Earth's gravity.
The spacewalking astronauts who flew on STS-64 on the space shuttle Discovery—a mission that included the first untethered EVA in 10 years—trained with SAFER.
The next big addition to the VR Lab's training offerings was Charlotte. Originally, the robot, designed by aerospace manufacturing company McDonnell Douglas, was used as a way for support on the ground to tend to experiments on the shuttle while astronauts slept. Homan saw the opportunity to use it another way, and modified it to work Earth-side. Charlotte became a means to help astronauts maneuver objects no longer subject to the Earth's laws of physics, which means they're weightless, but still have mass and centers of gravity influenced by orbital mechanics. Wheelock put it like this:
"[If] you fly through [the ISS] like Superman, and there's a metal hatch waiting for you at the other end, it's not going to be a pretty picture. F still equals MA [force=mass times acceleration], so you're going to slam into that hatch unless you catch yourself on something."
In 2007, as part of STS-120, Wheelock had to disconnect a solar array from the top of ISS's truss and move it. It was 37,000 pounds. In space, you can hold it up with one finger, he said.
But, if that piece had started moving in a certain direction, and he were to try and hold it, it could have torn the glove off his hand.
Training with Charlotte means avoiding lost solar arrays, lost gloves, and worse.
"[The VR Lab] is the only place where we can get that hands on, tactile feel for mass distribution," Wheelock said.
And beyond that, Charlotte is adaptable. After all, when wearing the HMD, astronauts aren't looking at that metal cube anymore.
Following the 2003 space shuttle Columbia disaster, the shuttle program halted for two years. The next two shuttle missions were called Return to Flight missions, and incorporated a new tactic to ensure that what happened with Columbia wouldn't happen again.
Shuttles started undertaking the Rendezvous Pitch Maneuver, in which a shuttle would essentially show its underside to the space station, and astronauts aboard would photograph it to make sure there was no damage to the heating tiles on the shuttle, as there had been for Columbia.
That's not an easy task. Those on the station had to make sure they scanned the entire surface, and within a limited timeframe. The VR Lab modified Charlotte to use real camera lenses to project a simulation of the shuttle's belly onto the camera's sensor.
"To get the detail you need, you're zoomed in so tight, all you're seeing is just this wall of tiles, so it's like you're looking at a brick wall," said Jeff Hoblit, the VR Lab's contractor lead, and one of the first engineers in the lab.
Changing tech and an exploding HoloLens
Another way to trace the evolution of the VR Lab is through its hardware.
The current lab team—Miralles, Bell, Hoblit, plus lab manager Eddie Paddock, and engineer John Schweers—sit around and try and tally up just how many different HMDs they've had, and all the things they've done to them.
They've had six, with untold iterations in between. Here's the timeline:
1991 - Flight Helmet Virtual Research (LCD displays - 320x240)
1992 - N-Vision Datavisor (CRT displays - 1280x1024)
1994 - N-Vision Datavisor(CRT displays - 1280x1024)
2005 - N-Vision Datavisor (Micro-OLED displays - 800x600)
2012 - In-house built (LCD displays - 1280x720)
2015 - In-house built (OLED displays - 1920x1080)
Bell wasn't a huge fan of the N-Vision Datavisors—the distortion correction was done through pot adjusters on the side, and as it was analog, it would drift over time, which meant he'd get simulation sickness trying to readjust the helmet.
"He would actually wait until the end of the day [to adjust it] because he knew by the time he got done... he was going to be sick, so then he could just go home," Hoblit said.
Early headsets cost hundred of thousands of dollars. And, they weren't exactly plug and play.
Eventually, instead of buying new ones, the VR Lab started modifying the ones they had, and that hasn't changed.
"Brad uses his garage as basically a manufacturing plant," Paddock said.
Other tech in the lab has also been through multiple updates, for example, moving from powering the headsets with a 4-rack mainframe to using PCs. The cost of replacement was less than the cost of the annual maintenance of the mainframe. And then eventually, they just built their own computers.
It's been almost 20 years of scoping out the latest in displays, video cards, tracking systems, and whatever else emerges on the market and proves stable enough to integrate into the lab.
There are a lot moving parts to the lab's setup, and integrating new technologies takes time and care. In another example, the current tracking system is electromagnetic-based and uses a transmitter, and six-degree trackers. Electromagnetic means that occlusion isn't a problem, unlike in an optical solution where tracking can get lost if something is in the way of the sensor. When TechRepublic visited the lab in August 2015, the team was interested in VR and 3D graphics entertainment company Sixense's STEM System, which is also electromagnetic, but is wireless. A wireless system could interfere with the audio system they build into the headset. So, if they ever decide to integrate it, it won't be until they're dead sure it won't snag any other part of their setup.
The flock of consumer HMDs threatening to hit the market in early 2016 has piqued their interest. The team is looking at Oculus and all the other companies about to dive into the VR space.
The Space Station was supposed to get Microsoft HoloLens to try out, but the much-hyped augmented reality headset was among the cargo of the SpaceX rocket that exploded at the end of June 2015.
In any case, the new wave of VR and AR is very much welcome in Houston.
Though the lab has been staffed and supported by many people over the years, there are typically only between four and seven people working there full time. They don't have the time, manpower, or money to dig into researching issues like simulator sickness.
"We're very happy the money's being thrown into that [research] space because all of us who do VR are going to benefit from the dollars being spent that aren't ours," Bell said.
SEE: NASA Mission Control: Photos of the past and present at Johnson Space Center
Meet DOUG
It's easy to focus on the hardware. It's the thing you can hold in your hand or strap on your face. But in truth, it would be useless without the software behind it.
These days, the graphics engine that powers NASA's virtual reality training is the Dynamic Onboard Ubiquitous Graphics, or DOUG for short. And Miralles, they call her Dougalina.
She's supported the lab since just about the beginning, but started there full time around 2000 when she and Bell undertook the creation of a graphics engine that was, in a sense, portable. It would be the next evolution on the lab, making it possible to not just prepare in the lab in Houston, but up on the Space Station as well.
"It was pretty fun because we had to experiment a lot with different scripting languages and interfaces in the environment," Miralles said.
One of the most impressive aspects of DOUG is that it's an incredibly detailed 3D graphical replica of the ISS. Each piece is its own module—with more than 5 million polygons—and it changes every time the space station does.
Sitting in a Chinese restaurant in Houston, Miralles grabbed the salt and pepper shakers from the table to explain that as the Space Station was being constructed, adding on piece by piece, shaker to shaker, if you will, it was undergoing the same transformation in DOUG.
Earlier that day in the lab, Wheelock stood in front of a large monitor, describing a mission that involved coaxing a ripped solar array to retract properly. He couldn't initially remember which solar array it was, until Miralles zoomed in on the station using DOUG, and there it was—the tear he'd been describing.
Now DOUG is all over NASA, as well as the Space Station, and used in a variety of ways.
Before an EVA, the lab prepares the procedure (reconfigs) so that the spacewalkers can review the EVA using the 3D graphics, frame by frame.
Hoblit said that the Russians used to review their EVAs on paper, but once they saw what their US counterparts were doing, they asked if the lab could also use DOUG to prepare the Russian EVAs. Now the lab also prepares reconfigs for the Russians.
DOUG is also used for SAFER training aboard the ISS since astronauts are spending longer periods of time in space, and are experiencing long gaps of time since their last training. They couldn't send up additional hardware, but Bell came up with an idea for using an upside down laptop attached to a simple set of goggles as an HMD. By 2012, virtual reality training had arrived in space.
On a wall outside the lab, there's a print of the ISS pulled from DOUG. One of NASA's public affairs officers mentioned that it took him a surprisingly long time to realize that the image was from DOUG, and not a photo. It looks that real, even when you put your nose right up to it.
SEE: NASA's unsung heroes: The Apollo coders who put men on the moon (PDF)
Customers first
The VR Lab is the product of a lot of people, both engineers and astronauts.
That relationship is part of the reason the lab has evolved the way it has. As the lab prioritizes the needs of the astronauts, the astronauts advocate for the lab. Homan protected the lab from much of the bureaucratic oversight that would make it difficult to quickly adapt to the training needs of EVA crews.
From the beginning, Homan and Bell sought input from astronauts like Jeff Hoffman and John Young.
"We wanted to include the crew early on because we didn't want to create a solution that had no problem [to solve]," Bell said.
Playing to the astronauts' training needs helped the lab sidestep some of the signature problems that VR's run into over the years, like latency.
When Homan and the lab were starting out in the 90s, the graphics were classically mediocre and computers just didn't have the power to render the scenes fast enough. They found themselves having to choose between more detailed graphics or lower latency.
But in talking with the astronauts, it turned out they prefered the details to the lower latency, partly because the latency didn't matter that much.
In space, they move slowly and deliberately. Getting out a wrench is the most important task in the world, until it comes time for the next step. So, if it took a half second for the scene to catch up with orientation of the astronaut's head, that was actually okay because they were moving about that slow anyway.
Homan does remember one case, though, where latency proved a problem.
One afternoon, upper management came by the lab during a period where they were trying to secure funding. The Astronaut Office had been talking up the technology, but when the powers-that-be tried it, VR didn't exactly sell itself.
"On the ride back to their office, they were all complaining about how anybody could possibly want to do that. Actually, it had made most of them sick," Homan said.
Another familiar issue is the comfort of the HMDs. In the past, they've been heavy and awkward, sometimes enough to turn off casual observers.
But even if early on the helmets in the lab weren't all that comfortable to wear, dealing with them was by no means the most uncomfortable part of astronaut training.
One of the VR Lab's selling points is time.
"The fact that you can zip over to the VR Lab and do an hour of training in your shirt sleeves without having to go through the huge cost of doing an actual run in the pool gives virtual reality a huge advantage," said astronaut Clay Anderson.
For every astronaut in the pool, there are four other divers. Getting into the space suit is something of a production. If someone really needs to take one more run through of a task on a weekend, the folks in the lab will accommodate. And that includes times when something unexpected comes up on the ISS and an unplanned EVA has to happen, or one has to be modified. The VR Lab can do that rather quickly.
Miralles and the other members of the lab like to reiterate that every astronaut comes through the lab for training since it became a requirement in 1994. During the space shuttle program, the lab supported four to seven missions a year.
Astronauts could spend hours at a time under the helmet. The lab would be packed with trainers and crew members, and of course, the staff, ready to fix or modify if asked.
Wheelock remembers walking into the lab for the first time. He was nervous. It felt like something out of a James Bond film.
That didn't last long. Wheelock remembers training for a mission, one where he and his spacewalking partner Scott Parazynski would at one point pop up across from each other on opposite sides of a truss they were disconnecting. On one VR run through, Wheelock popped up over the side to see Hello Kitty peeking out at him from Parazynski's helmet.
"You can get yourself balled up in here, you're sweating, and you know everyone is watching, and you want to do well, and you want to survive," Wheelock said.
And while much of the training is skills-oriented, one of its best functions is to get astronauts in the right frame of mind for completing the task. There's a great advantage in just feeling as though the environment isn't entirely new.
"It means everything. It's not hard to realize that just beyond that thin little visor is instantaneous death. It's oblivion," Wheelock said.
It's space, so there's liable to be a surprise at any moment.
"That's why we show Hello Kitty," Miralles laughed, explaining that part of the training involves not only occasional comic relief, but some sensitivity to those going through this process. Sometimes an animated cat is the best tool for re-establishing focus.
That looseness is also part of the working culture of the lab. Homan said that while most labs at NASA forbid food and drink, for example, he always encouraged it.
"A lot of times, astronauts would just come over there to get away from everything, and sit down, and eat our candy," he said.
Miralles, Bell, Paddock, Schweers, and Hoblit have known each other a long time. It's an old cliche, but they really can finish each other's sentences, and do.
They crack jokes and erupt into short raucous bursts of laughter that sound like Statler and Waldorf from the Muppets.
All these years of working together leads to a certain comfort in expressing ideas, even if they seem a little wacky.
"For example, when Brad first presented the idea of the [upside down] laptop," Hoblit started.
"They laughed at me," Bell finished.
"We assumed he was joking," Hoblit said.
"I did a foam core mock-up and they were like 'oh, yeah, maybe that will work,'" Bell said.
"But when you first told us, we just shook our heads," Hoblit said. They break into laughter again.
Hoblit sees it like this: "When you have people you work with for that length of time, and everybody has a lot of respect for the other people, what their capabilities are, it actually makes for a very fun work environment because you're not afraid to tease each other, you don't feel threatened, nobody gets their feelings hurt."
Paddock thinks part of the reason everyone's gotten so comfortable is that NASA needs VR. Like the rest of NASA, the Johnson Space Center has seen plenty of layoffs due to budget cuts over the years, but VR training is not something they can just lob off in order to cut expenses.
SEE: Photos: Inside Johnson Space Center's Neutral Buoyancy Lab
Beyond 50 years of spacewalks
Without the space shuttles, training has slowed, but there's plenty else to do.
The lab also produces data products, and is involved with many other projects. For example, they've been working with an IMAX team that's making a movie about the ISS. When a certain event is going to occur, like SpaceX flying a mission to the station, they ask the VR lab which cameras placed in which spots on the Space Station will best capture the action. So, Miralles looks at the trajectory, checks the views from the windows, and picks the best cameras, and sends those images from DOUG to the IMAX team.
And as long as the Space Station is active, there will be projects to tackle. Miralles said they could see another spike with the rise of commercial space flight. Plus, the ISS will be in operation for at least another 10 years.
"That means we'll be here at least that long," Bell tells her.
"Oh, God!" she says and they dissolve into laughter again.
With almost two decades behind them, Miralles said they do think about what comes next for the VR Lab. For her, it's a concern that's specific to the lab, but also attached to a passion that's formed after a career of being a woman engineer.
She hasn't always been the only woman in the lab, but the ratio has shaped how she views women in STEM and the unique challenges they face learning how to communicate in an all-male environment—first hand, she felt the pressure to always be exactly sure of the next thing she was about to say, and had to learn to not worry about asking the questions she needed to ask.
It's made her more demonstrative in her outreach to young women who may turn away from STEM because they feel they don't fit the idea of what an engineer is.
Originally from Venezuela, Miralles grew up playing Lost in Space with her brothers, and eventually came to the US to get a bachelors in computer science and another in computer graphics. She also got an MBA in management of technologies.
Recently, she was serving as a judge for an American Institute of Aeronautics and Astronautics (AIAA) robotics competition for middle school and high school students. During one of the judging rounds, she got a string of phone calls. They were from her friend Samantha Cristoforetti, who was aboard the ISS. Cristoforetti left her a voicemail saying hello, and Miralles took it back to the group she was judging, a team of three girls, to show off not only this dispatch from space, but that is was from a woman astronaut.
"The sparkle in the eyes... it was so neat, and so sweet being able to share that moment with them," Miralles said.
Presence
On the morning of February 25, 2015, Miralles woke up and turned on her television.
Astronauts Terry Virts and Barry Wilmore were going to be embarking on EVA-31, and Miralles and Bell had been working on the reconfigs since December for that and EVA-30, which took place just a few days before.
On the ground in Houston, Miralles drank her coffee and watched Virts and Wilmore install a new communications system called the Common Communications for Visiting Vehicles (C2V2). It was going to require a lot of cabling, so Miralles had emailed Virts some advice—make sure you keep your tether attached.
Whenever there's an EVA, Miralles watches, at least the beginning, sometimes staying home an extra hour or two just to see that the task is underway and on schedule. Once she knows that's the case, she drives to work and immediately turns on the TV in the lab.
By the time astronauts exit the Space Station, they've rehearsed and reviewed the EVA so many times in the pool, in the lab, and on the station, Wheelock said they can practically do it with their eyes closed and feel their way along.
Miralles and the lab also know every single handrailing. Spending months with the crew members, training, prepping, advising, updating—it makes watching EVAs resonate on a personal level.
The ISS orbits roughly 250 miles above Earth, which is just about the distance from Atlanta, Georgia to Nashville, Tennessee. That means the work that the lab does comes to fruition a long way away.
The twist is that it doesn't quite feel that way, Miralles said. In fact, it's quite the opposite, and, perhaps, quite apropos to the world of virtual reality.
"We feel like we're there."
For more on this story...
Photos: Inside Johnson Space Center's Neutral Buoyancy Lab
NASA Mission Control: Photos of the past and present at Johnson Space Center
Video: 3 big takeaways for the future of virtual reality from NASA's VR Lab
Hide Comments