High above Wilson Boulevard in Eastern Foundry's new Rosslyn campus, Dave Zvenyach was discussing the central tenet of the new gospel of innovation in government: the benefits of failing.
"So my theory is government is not afraid of failure. My theory is government is afraid of failing unpredictably," said the executive director of 18F — the General Services Administration's digital services consultancy — at the Jan. 31 event.
The event was billed as the opening of government technology incubator Eastern Foundry's new Rosslyn campus, but it also served as an unofficial celebration of the Trump administration's tacit endorsement of 18F, which publicly emerged in a tweet from then-Chief Digital Officer Gerrit Lansing five days earlier.
"Assistant to POTUS for Intragovt and Tech Initiatives Reed Cordish at @18F/TTS Townhall just now: 'We have your back,' " Lansing wrote, effectively ending weeks of speculation over whether the new administration would scrap the Obama-era's "digital SWAT team."
The survival of 18F was anything but certain prior to inauguration, when prognosticators swirled rumors that the new administration would shutter the doors of the innovation office and others like the U.S. Digital Service — whose mission it was to salvage the disastrous rollout of Healthcare.gov — but after the White House's endorsement, Zvenyach was fielding questions about how to inject innovation into best-value procurements as a way to reshape the public sector.
"I think what we need to do is we need to give examples for where you can do great things," he said.
"We need to be able to build that sort of catalog of examples, of examples where we have successfully done the things that I'm talking about and say: 'Yeah, we tried it, it worked,' or 'We tried it and this component didn't work. Don't do that again, but the principle was the right thing.' And having that catalog of experience is what's going to transform government."
In a city so averse to the downside of risk that it pillories its most famous flops in marathon committee hearings, Zvenyach might seem like a blaspheming heretic run amok. But in certain corners of Washington looking to modernize the federal government, failure is the new success.
Failure as an option
While 18F and USDS often serve as the poster children for the injection of startup culture in the federal government, the concept of using Silicon Valley strategies to reimagine public service is currently permeating agencies like a viral meme.
"The ultimate accountability is for delivering improved government services to the public," said Matt Lira, senior adviser for House Majority Leader Kevin McCarthy, R-Calif., who has voiced support for 18F and USDS.
"As we can see from both what we hear every day from constituents, but also from elections, there is widespread frustration by the public with their interactions with government," Lira said. "The way things have been going isn't getting the job done, so our perspective is leveraging things like agile development to rapidly deliver improvements to public services, hopefully creating a more efficient, effective and accountable government."
Whether it's the incorporation of human-centered design in every mission from military health care to State Department diplomacy or federal agencies' attempts to rapidly shift data storage to cloud computing, the dogma that innovation disciples imbue centers on one basic principle: It's better to fail fast and correct course than fail big down the line.
But while Valley startups can use the financial cover of venture capitalists to fuel their experiments, agency leaders are ill-disposed to gamble taxpayers' money in such fashion.
Federal Communications Commission Chief Information Officer David Bray noted the disparity in a January 2016 Brookings column titled "Idea to retire: Leaders can't take risks or experiment," in which he called for a government-specific model for innovation.
"Whereas the motto in Silicon Valley and other innovation hubs around the world might be 'fail fast and fail often,' such a model is not going to work for public service, where certain endeavors absolutely must succeed and cannot waste taxpayer funds," he wrote.
"The only way public sector technologists will gain the expertise needed to respond to and take advantage of the digital disruptions occurring globally will be to do 'dangerous experiments' as positive change agents akin to what entrepreneurs in Silicon Valley also do."
But the speed of both new technology and the innovation it drives moves significantly faster than the checks and balances of government are designed to do. The incongruence of pairing lightning-fast solutions with the measured, layered response of government often puts agency leaders in a conundrum.
"We can't just incrementally make improvements as to what we are doing," Bray said in an interview with Federal Times. "So the conversation I was having at the [Feb. 8 Government of the Future] conference, I talked about how we went all-in on cloud computing.
"There was another agency that was talking about how we've got to think about it, we've got to categorize it, we've got to plan for it, and it didn't sound like the fastest approach. I think, however, I can sympathize with that agency. Because with all the pressure from the legislative side, as well as the executive side, as well as just public opinion, there's no upside to taking a risk and having it pay off in public service."
Bray said the equation for federal agencies to define acceptable risk is by identifying what objectives have to run on time and what needs to be experimented on and why.
"If something is not going right, identify it as rapidly as you can and redirect it," he said. "If you take a bet and you realize in 15 days or 30 days you need to change direction, that's OK. If you take a bet and you allow it to persist in a very bad course and a year and a half later you've got to say, 'Oh,' that's probably deadly."
Flak jacket in chief
Coming to the FCC in 2013, Bray said he was given some latitude to experiment on new solutions, but he had to strike the balance of delivering results with pacing change at a rate that accounted for the perspectives of agency stakeholders.
"Encourage a diversity of thought," he said.
The incorporation of different viewpoints is an integral concept in human-centered design, a Silicon Valley-
inspired philosophy often incorporated in innovation labs springing up throughout the executive branch.
The strategy suggests the best successes come from incorporating new perspectives and balancing them with established stakeholders to inject new ideas and develop them alongside experience to flush out blind spots.
The result is a meritocracy concept that breaks down hierarchy and rewards the best ideas. But for it to thrive, Bray said a key element is for C-suite managers to serve as a protective barrier, or flak jacket, to ensure that innovation can develop without the fear of failure, something he calls bottom-up development.
"In a rapidly changing environment, you have to be bottom-up because the edge has many more people aware of what is going on than the C-suite executive that is probably one, two or maybe even three steps away," he said.
"The advantage of the C-suite executive is you are seeing more perspectives, so you can serve as a better integrator, but don't think you are the person that is really rubbing up against reality."
The key ingredient for executives is to make sure employees have autonomy, Bray added, so that while new solutions develop, all of the stakeholders can voice their viewpoints and point out the blind spots of a plan.
"Acceptable risk is acceptable when you finally recognize it is a shared endeavor rather than an individual endeavor," he said. "C-suite executives should provide top cover for those members of the team that are experimenting after it's been a shared conversation and we've all collectively bought into this.
"If it goes well, all credit goes to the individual that ran with it. If it doesn't go well, then yes, it is your job to take the hits."
Planning for failure
Even when trying to game out a mission that allows for constant advancement and reassessment, setbacks can occur, which requires failing-fast strategies to not only be nimble but also forward-facing.
"Technology and innovation often require several iterations to be successful," GSA said in an email statement regarding 18F.
"In a sense, failing fast necessarily implies accountability. When you approach a program — or even an experiment within a program — you need to try and understand what success and failure look like before you start. Sometimes there's still uncertainty, but the goal is to ensure that you're focused on continuous learning and avoid repeating the same mistakes."
Bray also had to game potential failure scenarios when he designed the FCC's shift from dedicated servers to cloud computing — known as Operation Server Lift — in 2015.
The move required that the FCC physically ship its servers to a commercial site, where its 400 terabytes of data could be transferred to the commercial service provider that would oversee the cloud transfer.
The plan was a calculated risk because if anything happened to the servers, the agency's online operations could be compromised while it tried to recover the data.
"I wrote before we launched a blog, basically saying that the risk of doing nothing was worse that the risk of doing something in that case because we're already spending 85 percent of our budget" on legacy servers, he said. "That was intentional to lay in case something did go wrong. And I expected something to go wrong in the sense that there were so many moving parts.
"In a perfect world, we would have gotten additional funding, we would have replicated the servers, we wouldn't have physically had to move them ... but that would have cost additional money, which we, quite frankly, didn't have."
So Bray and his team went for the Hail Mary and shipped the entire server network, using two versions in two trucks leaving at two different times to account for any scenario where the servers could be damaged or delayed.
But when the servers arrived at the commercial provider, the cables to 200 servers weren't exactly mapped to reconnect them, despite the FCC having paid a vendor to verify that they were.
As detailed in The Enterprisers Project's two-part chronicle of Operation Server Lift, the cables issue required both FCC personnel and contractors to work 55 hours to reconnect the servers, expending time budgeted for unforeseen issues, before the successful cloud transfer could be made.
"That's where you have to have a good enough council that surrounds you so that when you are trying to make the call in the moment, 'Do I roll this back or do I keep on going,' you are getting multiple perspectives," Bray said.
The future of failure
While failing fast can provide agencies with opportunities to discover new breakthroughs, it requires the top cover of leadership, and the buck ultimately stops at the White House.
The Trump administration's backing of 18F and USDS signals its tacit appraisal of the value of fail-fast philosophies in government; but as with any venture, support is dependent on results.
"How do nimble organizations pivot quickly to deliver improved services?" Lira asked. "I think programs like this are able to learn very quickly what works and what doesn't and apply those to wider-scale platforms versus the status quo, waterfall method, which will fail over the span of years and lead to, obviously, resource questions, but also the ability to deliver meaningful outcomes to the public.
"It isn't about a lack of accountability. I think it's actually about having accountability, but having it occur much faster so the outcomes can be improved on a much more timely basis."
Bray said the Trump administration may see the advantages of acceptable risk, but what remains to be seen is in what ways they would like to use it to drive innovation.
"I think if they come from a venture capital background ... that will probably be more intuitive and innate to them," he said.
"I think risk taking is not something that is usually taught in schools of policy, but it is taught in business. The challenge is that every administration has the things that they absolutely want to do. If you bind that and say this is what we want to do, the question then is how to best do that?"
