The Department of Defense’s new electronic health record program once again faced sharp criticism, as the department’s testing office released a hugely negative report critiquing the system’s usability.
Based on initial operational testing and evaluation, MHS GENESIS was deemed neither operationally effective nor suitable, according to the memorandum dated April 30 and written by DoD Director for Operational Test and Evaluation Robert Behler.
“MHS GENESIS is not operationally effective because it does not demonstrate enough workable functionality to manage and document patient care. Users successfully performed only 56 percent of the 197 tasks used as Measures of Performance,” Behler wrote.
“MHS GENESIS is not operationally suitable because of poor system usability, insufficient training and inadequate help desk support. Survivability is undetermined because cybersecurity testing is ongoing.”
RELATED
According to the report, users submitted over 14,000 help desk ticket items, generated 22 high-severity incident reports for interoperability problems, experienced system outages and lag times, and ranked the usability of the system at 37 out of 100.
Behler recommended that the undersecretary of defense for acquisition and sustainment delay further deployment of the system, ultimately intended to service nearly 10 million beneficiaries, until all initial operational testing and evaluation can be completed and the Project Management Office corrects outstanding deficiencies.
The fact that users have been struggling to figure out the MHS GENESIS system at the four initial testing sites is not new. Reports surfaced in March 2018 that doctors were slowed down significantly in trying to use the new system, seeing far fewer patients and working overtime to do so.
However, DoD officials and spokespersons for Cerner and Leidos, the two companies handling the $4 billion contract for the EHR rollout, said that those initial difficulties were part of the plan for testing and improving the system before it went live to more facilities.
At the time, Stacy Cummings, program executive officer for the Program Executive Office, Defense Healthcare Management Systems, said that the program was still on track for its next phase of deployment along the Pacific Northwest in 2019 and for a full deployment in 2022. Cummings confirmed in a May 11, 2018, press call that that timeline is still in effect.
In fact, Cummings said that the findings in the DOT&E report didn’t come as a surprise to the MHS GENESIS team, as the findings matched user communications they had already been having with the four test centers prior to the report’s release.
“We think that the test community gave us exactly the recommendation that we would have given ourselves, which is that we needed to do some work, we needed to test it. And our further deployments were not scheduled to happen until 2019, so we remain in line with our acquisition timelines,” said Cummings.
The testing behind the DOT&E report occurred in 2017, according to Cummings, which meant that the MHS GENESIS team was already working to resolve many of the report’s problems by the time of its release.
“We completely agreed with their assessment, because that was exactly what we were doing,” Cummings said.
Representatives from some of the four initial operational capability sites in Washington State — Fairchild Air Force Base, Naval Hospital Oak Harbor, Naval Hospital Bremerton and Madigan Army Medical Center — attested to the fact that systems are already improving from the results initially found in the report.
“The whole idea for us was that we wanted to see consistent improvement in all of these things, and so the ones that really matter to us are the clinical delivery times and statistics,” Army Col. Michael L. Place, commander of the Madigan Army Medical Center, said.
Place said that from November 2017 to March 2018, clinic visits went up by 20 percent, admissions went up by 10 percent, prescriptions filled increased by 78 percent, barcode scanning was above the national average and increased by 22 percent, and the time to document went down.
Place added that the large number of trouble tickets filed by users is also ultimately a good thing for the system.
“As an IOC site, one of our roles, as we see it, is to find all of those things that need to be fixed. And so we take a certain, maybe you can call it perverse pleasure in finding all of the different things that we think could be better,” said Place.
Behler, in a May 11, 2018, statement to the Program Executive Office of the Defense Healthcare Management Systems, seemed to support the fact that the PMO is already actively working on recommendations found in the report.
“The MHS GENESIS PMO continues to address this complexity with a robust test program, and has rapidly incorporated lessons learned from testing,” wrote Behler.
“The PMO has developed a strong relationship with the users of MHS GENESIS by setting up a senior board, with representatives from all of the military services, to help identify ways to quickly improve the functionality and usability of MHS GENESIS.”
Jessie Bur covers federal IT and management.
