A Modern Moonshot
When NASA sent the first humans to the moon 50 years ago this year, the development of computers was in a nascent stage. The Apollo 11 spacecraft touted one of its own on board: just over 70 pounds and 24 by 12.5 by 6 inches.
Fifty years later, NASA is planning again to launch people to the moon, targeting a 2024 moon landing and later looking toward Mars through its Artemis program. Technology has advanced greatly in the last 50 years, which also transforms the risk.
For the moon mission, astronauts will travel in the Orion space vehicle, being built by Lockheed Martin. Like many government platforms today, the functions of the vehicle rely on software — and any bugs in the code could potentially catastrophic effects.
Ensuring the integrity of code falls in part on Justin Smith, the project manager for the Orion Multi-Purpose Crew Vehicle team at NASA’s Independent Verification & Validation (IV&V) Program. His team is checking that critical functions of the spacecraft work correctly.
“Does the software help the parachutes deploy on the line the right way right before it crashes into the water? Or do the jets fire at the right time or in the correct position?” Those are just a couple of the considerations that Smith noted Oct. 21 at the American Council for Technology-Industry Advisory Council Imagine Nation conference in Philadelphia.
To do the job effectively, Smith’s team adopted an “agile” software assurance process.
Moving beyond ‘paralysis by analysis’
“Agile is all about small, manageable pieces of work,” Smith said. Agile development — an alternative to a more traditional, linear waterfall approach, where each phase depends on results from the phase previous — can apply to software assurance, software development, workforce policies and the government’s ability to adopt to new technologies quickly.
“In the 21st century, agility powered by IT, powered by data and powered by people is how leading players execute,” Margaret Weichert, the Office of Management and Budget deputy director of management, said at the same conference.
The people piece was a critical part for Smith’s agile assurance of the Orion software, dividing into what he called “small self- organizing teams.” He started with two teams: one focusing on command and data handling, communications and tracking, and electrical power; and the other focused on entry, descent and landing of the spacecraft.
“So, whenever they would run into a problem or a different way to look at the software, they would help each other in the scrum meetings,” Smith said.
Scrum, of course, is the framework for agile, named after the rugby formation.
They later decided to take a different approach, moving away from grouping by technical background, he said.
“We’ve moved away from ‘this person has this specific knowledge of the vehicle, this person is really good at doing this kind of analysis and they work well with this person,’” Smith said. “We’ve done a lot of personality assessments.”
Smith said that he wanted to avoid “paralysis by analysis” to speed up his team’s decision-making. Smith’s team established retrospectives, which are opportunities for team members to openly talk about what worked and what didn’t work during the previous period — or “sprint,” though his team later stopped using sprints but continued retrospectives. Having these conversations, he said, were critical to getting members comfortable with faster decision-making.
“There’s always opportunities through our retrospectives to course correct,” Smith said.
But “that was very hard at the beginning. A lot of them were really scared to take chances because they felt they were just going to get hit over the head or lose their job if something went wrong.”
A model for the feds
In her speech in Philadelphia in late October, Weichert said that she wanted to knock down the “structural impediments” in government that are preventing the government from being agile. Across the civilian agencies, federal chief information officers are recognizing the impact that an agile approach can have on projects — and gradually attempting the transition.
But the agile development lifecycle from the developers that IV&V worked with created challenges: they couldn’t keep up with the process. The agile approach was releasing new code for the flight software every three months.
The process was “like nothing IV&V had seen before,” team members wrote in a white paper on the program. The document said team members became very frustrated trying to analyze the code because they couldn’t keep up. Ultimately, the inability to keep up introduced risks of “increasing cost to an already resource constrained program.”
“The developer was doing what they needed to do to build the flight software, and it was evident that these challenges would require IV&V to adapt much more than usual to perform effective analysis,” the team wrote.
Indeed, to keep up with the rapid development of Orion’s code, which totals 800,000 lines, IV&V had to change its approach to one that would allow it to quickly shift between areas of highest risk and provide assurance for those areas, and then rapidly shift focus again if priorities changed. NASA brought in an outside consultant from Carnegie Mellon’s Software Engineering Institute and then eventually set up an agile assurance framework.
With the proper approach in place, cost savings are a critical benefit of agile development for the taxpayer, said Department of Education CIO Jason Gray. He added that the flexibility also makes projects more efficient.
“It allows us to respond to the constantly changing environment in a much faster and more efficient way,” rather than feeling compelled to continue down a path to failure, Gray said.
Ultimately, the approach allows for a better product — and also allows the customer to have more input during the process. The result is ideally a “more user-centered design” that can help define requirements and ensure the team is “getting it right as we go,” said Small Business Administration CIO Maria Roat.
The Orion IV&V team’s agile approach changed how it releases its results, the team wrote. Instead of only reporting out what the software is doing that it shouldn’t, it also reports confirmation that the software is working the way it should. This approach provides a more “balanced assessment of the condition of the software.”
Ultimately, the approach provides insight that allows the decision-makers to “understand the level of risk perceived” by IV&V, they wrote.
Therein lies another benefit of agile: providing stakeholders with more transparency throughout the process. Agile development can often appease program area owners who have complained “for years” that they weren’t involved in the project processes, said Gary Washington, the Department of Agriculture CIO, adding that through agile development “we’re showing them where they fit in in a process.”
It’s important because “they actually see tangible progress in terms of development of systems and they are part of that whole activity,” he said.
Lessons learned from the moon mission
The agile framework increased the frequency of Smith’s team reports to Lockheed Martin from months to weeks. Some weeks, they can deliver several products.
The retrospective meetings that Smith’s team held were what he described as the most important piece of their successful adoption of agility. All the “great innovations” that came from his team, he said, started in those meetings that at the core reflect on successes and failures.
“They started with an idea — something that we weren’t doing well, and they were brave enough to talk about” Smith said. “They felt safe. And from there we talked as a group, we brainstormed ideas, and then, ultimately, we created a safe environment for that idea to go off.”
Retrospective meetings don’t have to take much time; as little as 10-15 minutes can make a difference, Smith said.
Meetings also serve as the project management component of the effort — identifying pockets of opportunity to drive efficiency.
Recognizing what Smith and his team are working toward, 50 years after their predecessors relied on clunky computers, is both surreal and inspiring. And, Smith adds, it is the best recruitment tool imaginable.
“It’s extremely cool,” Smith said. “As somebody who gets to follow the program, to watch what they’re doing and understand the mission — it’s awesome.
Andrew Eversden covers all things defense technology for C4ISRNET. He previously reported on federal IT and cybersecurity for Federal Times and Fifth Domain, and worked as a congressional reporting fellow for the Texas Tribune. He was also a Washington intern for the Durango Herald. Andrew is a graduate of American University.
