Finding a killer buried in a medical device's source code is not straightforward detective work. The directions for an implantable defibrillator might run over 100,000 lines - as long as War and Peace - and cover a multitude of possible actions that could take a decade for the device to run through. Fitzgerald's team of investigators doesn't have that kind of time, especially when patients are dying.
FDA officials declined to name the maker of the infusion pumps. (In 2006, Cardinal Health, of Dublin, Ohio, stopped production of its Alaris SE pumps because of a key-bounce error that reportedly killed two patients, including a 16-day-old baby that got 44.8 milliliters of intravenous nutrition, rather than 4.8 milliliters.)
During the investigation into the malfunctioning pumps, nurses complained about frequent keyboard errors, while the manufacturer blamed nurses for entering the wrong drug information and then failing to double-check, said Brian Fitzgerald, who heads the FDA's software specialists. The team thought the problem was a key-bounce error. Once they got a copy of the pump's source code, they quickly corroborated their suspicions.
"We could see if a key was pressed more than, like, 20 times in a second, the key would repeat," Fitzgerald said.
More often, though, clues are scarce and answers far from immediate. The team must pore over the entire code, looking for tiny flaws in the logic that, on the rare occasions it is summoned into action, could have disastrous consequences. No human has the brain power - or patience - to perform that work. Indeed, powerful computers must, in effect, crunch all the moves that a piece of software might take.
They run programs developed to find a bug that had caused Europe's Ariane 5 rocket to blow up in 1996. Since then, automakers, Microsoft and the federal government have started using the programs, called static analyzers.
"There really is sort of a revolution in the way these control systems are built now," said Rance Cleaveland, a computer science professor at the University of Maryland who has talked with the FDA about static analysis.
The FDA established its forensic software unit in 2004, after noticing that device makers were issuing more and more software-based recalls. By 2006, officials had learned from talking with North Carolina State University computer scientists that static analysis could also be used to investigate mishaps. "It was almost accidental," recalled Al Taylor, director of the FDA's electrical and software engineers.
The FDA's team employs about 10 mathematicians, computer scientists and a physicist who once designed military satellites. Their year-old laboratory on the agency's new campus - a former Navy warfare research site near Silver Spring - is cluttered with circuit boards, cables and desktop computers. Racks of servers blink and hum on the floor below, running probes of software code that controls wheelchairs, ventilators and proton beam therapy systems.
About two years ago, Fitzgerald recalled, the forensic software team was assigned to investigate a dialysis machine, in use for two decades, that suddenly began malfunctioning on patients with terminal illnesses. The team investigated but could not find a problem with the software. Six months passed before the manufacturer finally found a defect that only mattered when the machine worked nonstop, as it did on the terminally ill.
"We declared the software innocent," Fitzgerald said.
jonathan.rockoff@baltsun.com
