Archive for the 'FDA' Category

Technical Debt in Medical Software

Published on August 4, 2010 by Bob in FDA, Programming and Software Quality. 2 Comments Tags: , , .

Software development is software development. Most of the life cycle and quality issues faced in medical software are the same challenges for any software product. Technical Debt in Medical Software points out what technical debt is:

  • Complexity
  • Code Duplication
  • Documentation Debt
  • Testing Debt
  • Architectural Debt

A Martin Fowler article is referenced that nicely identifies the source of technical debt:

The benefits of paying down the debt are:

  • Increased R&D efficiency and improved time to market
  • Hitting commitment dates
  • Performance and technology upgrades

Of course if you don’t want to pay it off, there’s always the option to go bankrupt. This may have long-term advantages, but it will surely be a more expensive route. There is one statement in this regard that I think needs some qualification:

In this case the technical debt can be retired along with the legacy system, and like filing Chapter 11, you are no longer responsible to address all the sins of the past.

I know this refers to code sins, but just because you decide to do a re-write doesn’t mean you no longer have responsibility for the legacy product. You still have customers using the old software that you’re obligated to continue to support.  For FDA approved medical software, this is a legal requirement. Most of the time this means that the legacy code will need to be maintained and periodically updated in the field, sometimes even after the “new” product is released. This just makes the cost of bankruptcy even higher.

ISO 62304: The Harmonized Standard for Medical Device Software Development

Published on June 5, 2010 by Bob in FDA, Medical Devices and Software Quality. 3 Comments Tags: .

The FDA approved ISO 62304 as a recognized software development standard in 2009. Developing Medical Device Software to ISO 62304 gives a nice overview.

Besides providing a globally accepted development process one of the other practical components is the assignment of a safety class to individual software items and units:

  • Class A: No injury or damage to health is possible
  • Class B: Non-serious injury is possible
  • Class C: Death or serious injury is possible

Each classification changes the required documentation for the assigned software.

These standards will become more widely known as the FDA moves to regulate the proliferation of medical applications for personal and home use, most notably software that runs on mobile devices. I’ve discussed this before in When Cell Phones Become Medical Devices. As noted more recently in FDA oversight may extend throughout health IT:

… an FDA director stated flatly: “Under the Federal Food, Drug and Cosmetic Act, HIT software is a medical device.”

Broad FDA oversight at the QSR/62304 level will probably not happen, but change is certainly coming for many HIT companies.

The Elsmar Cove Forum IEC 62304 – Medical Device Software Life Cycle Processes has a lot of discussion on this topic. This is where I found a document checklist that is useful for understanding the process scope:

IEC62304_Checklist.xls (Excel spreadsheet)

The Software Quality Balancing Act

Published on May 15, 2010 by Bob in Agile, FDA, Medical Devices and Software Quality. 1 Comment Tags: , .

Andrew Dallas’s article Caution: V&V May Be Hazardous to Software Quality touches on a number of good points regarding software quality best practices.

Medical device software development V&V (also see here) and the documentation that goes with it have substantial costs. Any strategy that can reduce this overhead and still meet the necessary quality standards should be seriously considered.

The use of “incremental” software development approaches really refers to Agile methodologies.  I’ve talked about the use of Agile for medical device software development several times:

Most of the discussion revolves around the risks associated with this approach. The benefits of any process change have to be weighed against the possible risks that might be introduced.

Besides the importance of understanding what V&V documentation the FDA actually wants to see, Andrew makes a great point about producing quality software versus the V&V process (my highlight):

V&V is not software testing. Verification testing ensures specified requirements have been fulfilled. Validation testing ensures that particular requirements for a specific intended use can be consistently fulfilled.

Following the required FDA V&V processes alone is not sufficient to ensure software quality. You also have to adhere to software development best practices at all levels. For example, in addition to non-functional requirements there are many software quality factors that require careful design considerations and testing that you may decide are outside the scope of FDA reporting.  Deciding what to report and what to leave out is the balancing act.

To Validate and Verify: Software Issues Solved

Published on April 6, 2010 by Bob in FDA and Software Quality. 1 Comment Tags: .

Yours truly was interviewed for this article:

To Validate and Verify: Software Issues Solved

“V&V” is one of those topics that should be simple to understand, but for some reason is the source of a lot of confusion. This is evident in the comments on Software Verification vs. Validation.

It is also interesting to note that the differing interpretations of these definitions results in a wide variety of V&V strategies and plans. From a regulatory point of view there is no single right or wrong way to do it. It’s similar to the implementation of quality systems in general.  If you say you are going to do something you need to be able to prove that you’re actually doing it.

Why Healthcare IT is Not a Game Changer

Published on February 15, 2010 by Bob in EMR, FDA and Interoperability. 1 Comment Tags: , , , , , .

Last week I attended the WLSA/Continua Mobile Healthcare Symposium and the opening day of the Continua Health Alliance Winter Summit 2010.  Also, a couple of weeks ago I attended a few of the FDA Workshop on Medical Device Interoperability: Achieving Safety and Effectiveness sessions via a Webcast*.

Since I’m not going to HIMSS in Atlanta this year (starts Mar. 1) I thought now would be a good time to do some venting.

I’ve talked about HIT problems before, e.g. Healthcare Un-Interoperability and The EMR-Medical Devices Mess. With all of the ARRA/HITECH talk along with the National Healthcare debate raging it made me wonder how the issues facing device interoperability, wireless Healthcare, and HIT in general really fit in to the bigger picture.

After sitting though multiple sessions on a wide variety of topics presented by smart people the obvious hit me in the face:  The complexity of the issues are mind numbing. Everybody has good (and even great) ideas, but nobody has real solutions. Why is it that all this good HIT hasn’t translated into meaningful improvements in Healthcare?

For example. At first I thought the talk by Dr. Patrick Soon-Shiong might be heading somewhere interesting.  He presented a well structured view of the current Healthcare landscape that seemed to make a lot of sense. Then he plunged into the abyss with an in-depth discussion of transformational technologies (molecular data mining, Visual Evoked Potentials, etc.).  These developments could potentially lead to improvements in people’s health, but we never got to hear how any of the complex Healthcare delivery issues were going to be addressed.

Among his many endeavors Dr. Soon-Shiong is Chairman of  the National Coalition for Health Integration (NCHI). I think the “Zone of Complexity” point of view (see here — warning PDF) is a good starting point for understanding the position that Healthcare IT is in:

Also, following the diagram above is this statement:

However, currently, even when information is in digital formats, data are not accessible because they reside in different “silos” within and between organizations. In turn, the U.S. health system is hampered by inefficient virtual organizations that lack the mechanisms needed to engage in coordinated action.

The NCHI Integrated Health Platform (grid computing) is a good idea, but does it really even begin to provide the solution to these complex problems?

  1. They are taking a “bottom-up” approach to interoperability (system, data , and process) and trying to leverage existing technologies (like DICOM and HL7).  Makes sense. But other than academic or government institutions what’s the incentive for private  companies (like EMRs) to participate?
  2. How is an improved underlying infrastructure going to reduce the chaotic nature of the health delivery system (hospitals, insurance companies, Medicare, etc.)? It’s like putting the cart before the horse.

This is the dilemma. We can come up with clever and even ingenious technical solutions in our little IT world, but none of them are going to be game changers.   The availability of a great technologies are not enough to change the institutional processes that make an organization inefficient or communication ineffective.

The solution is in the people and the processes they follow. The best example I can think of is EMR adoption. Everybody knows why the rate of conversion from a paper to a paperless office is so low.  It’s mostly because of people’s resistance to change the way they’ve “always done it.”  Change is hard, and in this case HIT is the barrier to adoption, no mater how good the EMR solution is.

At the national level Healthcare IT only enables interoperability and improved data management.  The chaos can only be solved by first changing U.S. Healthcare delivery policies.  Whatever the changes are, they will then determine the incentives and processes that actually drive the system and put HIT to use.

For Healthcare IT, the NCHI is just one example. There are a whole bunch of other technology-driven initiatives that also have high hopes.  I’m not saying we should stop developing great technologies.  We just shouldn’t be surprised when they don’t change the world.

Happy Presidents Day!

UPDATE (8/4/10): Martin Fowler’s UtilityVsStrategicDichotomy post is another perspective on “IT Doesn’t Matter”.

*I thought the Webcast was very well done.  It had split screen (speaker and slides) along with multiple camera views that included the audience. The video quality wasn’t great (it really didn’t need to be) but the streaming was reliable.  Also, the web participants could chat among themselves and the on-site staff and ask the speaker questions.

The Challenges of Developing Software for Medical Devices

Published on February 8, 2010 by Bob in FDA, Medical Devices, Programming, Software Quality and Tools. 0 Comments Tags: , .

Developing Software for Medical Devices – Interview with SterlingTech gives a good overview of the challenges that especially face young medical device companies. In particular (my emphasis):

Make sure that your company has a good solid Quality System as it applies to software development. Do not put a Quality System into place that you can not follow. This is the cause of most audit problems.

I couldn’t have said it better myself, though I think that focusing on the FDA may distract you from why you’re creating software quality processes in the first place. The real purpose of having software design controls is to produce a high quality, user friendly, robust, and reliable system that meets the intended use of the device.  If your quality system does that, you won’t have to worry about FDA audits.

Since Klocwork is a static analysis tool company I also want to point out a recent related article that’s worth reading — and trying to fully understand:

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

Note the user comment by Bjarne Stroustrup.

UPDATE (2/9/10): Here’s another good code analysis article:

A Formal Methods-based verification approach to medical device software analysis

Medical Device Software on Shared Computers

Published on September 7, 2009 by Bob in FDA, Interoperability, Medical Devices and Networking. 6 Comments

ECG PCThe issues raised in Tim’s post Running Medical Device Software on Shared Computers literally opens Pandora’s box. Installation of medical device software on general purpose computers is an intractable problem.

It’s very similar to the complications associated with Networked Medical Devices, except worse.  An FDA approved device in a changing network environment is one thing.  Software that controls a medical device on a PC that is open for the user to install operating system upgrades, applications, and other device drivers is a recipe for disaster.

I don’t care how obsessed a vendor is, there is no way for a medical device manufacturer to verify proper operation for all possible hardware and software environments.

With today’s PC architectures, the highest risk area is at the device driver level. Running multiple devices that require even modest I/O bandwidth can cause interference that could result in lost or significantly delayed data. This is especially true with Windows XP or Vista that do not inherently provide any real-time data processing capabilities.

I think the best strategy is to provide stand-alone medical devices that have no dependencies on the PC hardware and software that may be available for down-stream data processing and display. This not only reduces compatibility risk, but it can also address mobility issues. With miniaturization and wireless capabilities, the medical device can now travel with the patient.

Also, with Pandora’s box safely closed, solving the networked medical device issues suddenly feels manageable.

UPDATE (9/15/09): Here’s an interesting take on this subject from the consumer perspective: Should Medical Devices Multitask?

When Cell Phones Become Medical Devices

Published on July 14, 2009 by Bob in FDA and Medical Devices. 3 Comments Tags: , , .

airstripMobile devices are quickly becoming the conduit of choice for collecting and disseminating clinical data.  The FDA will soon be forced to step in and take regulatory control.  It’s going to happen eventually.

Bradley Merrill Thompson does a good job of outlining the factors that lead to FDA oversight in the article FDA may regulate certain mobile phones, accessories. The Components vs. Accessories distinction is an important one for manufacturers — regulatory oversight is dependent on who buys it. The “intended use”, labeling, and marketing are also factors.

Because of its unique user interface, display, and broadband capabilities the Apple iPhone is a particularly attractive platform for medical applications. For example, the AirStrip OB application is available for download at the Apple App Store and is FDA cleared. Other modalities, like the Critical Care monitor application (shown) is still in testing.

The “intended use” issues are complex.  A cell phone that is used to communicate clinical information, e.g. to a PHR, essentially becomes part of a Networked Medical Device.

This mean that 510(k) premarket notification may also be necessary under the proposed Medical Device Data System (MDDS) rules.  If you read though what constitutes a MDDS, you can see how well the definitions fit mobile device functionality:

  • The electronic transfer or exchange of medical device data from a medical device.
  • The electronic storage and retrieval of medical device data.
  • The electronic display of medical device data.
  • The electronic conversion of medical device data from one format to another format.

Its not the end of the world to be classified as a medical device, but verification and validation of these functions are not a trivial endeavor (see here).

The FDA is almost certainly looking and will be taking action soon.

UPDATE (7/25/09): Here’s a mobile device that does not appear to have FDA approval: EKG On Your Mobile Wherever You Are

UPDATE (11/24/09): When will the FDA drop the gavel?

Software Verification vs. Validation

Published on March 26, 2009 by Bob in FDA and Software Quality. 20 Comments Tags: .

For some reason it just really bugs me that these two terms are incorrectly interchanged so frequently.

Part of the problem is that the document General Principles of Software Validation; Final Guidance for Industry and FDA Staff (2002) does not do a good job of differentiating actual verification and validation activities. They just call everything validation.

The recent MD&DI article Building Quality into Medical Device Software provides a pretty good overview of the these regulatory requirements, but is a another case in point.  The article talks about “software validation” at every step just like the FDA document.

Another similar article on this subject is Software Validation: Turning Concepts into Business Benefits.  It is also confused. e.g. (my highlight):

… software validation involves the execution of tests designed to cover each of the specific system requirements.

No, testing specific requirements is a verification activity! It’s no wonder most people are confused.

These definitions, Difference between Verification and Validation, are better as they highlight the sequencing of activities:

Verification takes place before validation, and not vice versa. Verification evaluates documents, plans, code, requirements, and specifications. Validation, on the other hand, evaluates the product itself.

From here (warning PDF):
verification vs. validation

Validation activities (usability testing, user feedback, etc.) are much harder to define, execute, and document properly than most verification testing.

Here are the golden rules:

Verification:  was the product built right?

Validation: was the right product built?

I guess I should get over it…

UPDATE (5/12/09):  Good definitions from here: Diagnosing Medical Device Software Defects Using Static Analysis:

Verification and validation are terms that are often used in software. However, it is important to understand the difference between these two distinct but complementary activities. Software verification provides objective evidence that the design outputs of a particular phase of the software development life cycle meet all of the specified requirements for that phase by checking for consistency, completeness, and correctness of the software and its supporting documentation. Validation, on the other hand, is the confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled.

UPDATE (8/6/09):  The importance of proper V&V can not be overstated. The FDA is watching: FDA still enforcing regulations for validation of enterprise software.

UPDATE (2/11/10): I just noticed that the guidance document link on the FDA site was changed and fixed it. When I reviewed the document I found that even though it was “issued” in Jan. 2002 it had been recently updated (11/6/09). The later sections (4, 5, and 6) still use the term validation generically, but the updated document does distinguish between verification and validation:

3.1.2 Verification and Validation

The Quality System regulation is harmonized with ISO 8402:1994, which treats “verification” and “validation” as separate and distinct terms. On the other hand, many software engineering journal articles and textbooks use the terms “verification” and “validation” interchangeably, or in some cases refer to software “verification, validation, and testing (VV&T)” as if it is a single concept, with no distinction among the three terms.

Software verification provides objective evidence that the design outputs of a particular phase of the software development life cycle meet all of the specified requirements for that phase. Software verification looks for consistency, completeness, and correctness of the software and its supporting documentation, as it is being developed, and provides support for a subsequent conclusion that software is validated. Software testing is one of many verification activities intended to confirm that software development output meets its input requirements. Other verification activities include various static and dynamic analyses, code and document inspections, walkthroughs, and other techniques.

Software validation is a part of the design validation for a finished device, but is not separately defined in the Quality System regulation. For purposes of this guidance, FDA considers software validation to be “confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled.” In practice, software validation activities may occur both during, as well as at the end of the software development life cycle to ensure that all requirements have been fulfilled.

More Software Forensics and Why Analogies Suck

Published on July 1, 2008 by Bob in FDA, Medical Devices and Software Quality. 2 Comments

There’s a recent article in the Baltimore Sun called Flaws in medical coding can kill which just rehashes static software  analysis (hat tip: FDA Trying to Crack Down on Software Errors).

I’ve discussed software forensics tools before. Yes, bad software has hurt and killed people, and there’s no excuse for it.  I still don’t think an expensive automated software tool is the silver bullet (which is implied by the article) for solving these problems.

But here’s what really bugs me:

“If architects worked this way, they’d only be able to find flaws by building a building and then watching it fall down”

This is a prime example of why analogies suck.  The quote is supposed to somehow bolster the FDA’s adoption of “new forensic technology”. If you stop and think about it, it does just the opposite.

I guess you first have to consider the source –  a VP of Engineering for a forensic software vendor. This is exactly what a you’d expect to hear in a sales pitch.

What’s truly ironic though is that a static analysis tool can only be used on source code! Think about it. Source code is the finished product of the software design and development process. Also, forensic science, by definition is the presentation of something that has already happened. It can only be done after the fact.

The logical conclusion you would draw from the analogy is that static analysis is probably useless because the building is already up!  If you step back and look at the full software quality process, this may well be true.

I’m not saying that static analysis tools don’t have value. Like all of the other software tools we use, they have their place.

Just beware when you try to use an analogy to make a point.

UPDATE (7/5/08):

Here’s another take on medical device bugs: When bugs really do matter: 22 years after the Therac 25.

UPDATE (7/16/08):
From Be Prepared: Software Forensics Gaining Steam at FDA, David Vogel of ­Intertech Engineering Associates says:

… that static tools are hyped to do more than they can actually deliver. “Static analysis looks for simple coding errors and does not apply heuristics to understand how it will perform dynamically because it is a static analysis tool”

I agree.

UPDATE (7/26/08):

Another reference : Are hospitals really safe?

UPDATE (9/16/08):

A couple more related articles:

Applying Static Analysis To Medical Device Software

Using static analysis to diagnose & prevent failures in safety-critical device designs

UPDATE (9/27/08):

Architecting Buildings and Software: Software architects are an important component in the creation of quality software and need to continue to refine and improve their role in the development process.  No matter how you try to bend and twist it though, the building analogy will always be problematic — so why bother? Maybe that “intuitive understanding” of the construction industry just distracts us from being innovative about what’s required to build software.

UPDATE (12/1/08): If Jeff wasn’t a programmer he’d be a farmer: Tending Your Software Garden

Subscribe

Categories

Twitter Updates