How do you know it works?

This is the tough one. The other questions are important, but relative to #5, answering them is pretty easy.  How to answer this question (i.e. accomplish this validation) is the source of a lot of confusion.

There are many business and technical considerations that go into the decision to use OTS or SOUP software as part of a medical device. Articles and books are available that include guidance and general OTS validation approaches. e.g. Off-the-Shelf Software: A Broader Picture (warning PDF) is very informative in this regard:

• Define business’ use of the system, ideally including use cases and explicit clarification of in-scope and out-of-scope functionality
• Determine validation deliverables set based on system type, system risk, project scope, and degree of system modification
• Review existing vendor system and validation documentation
• Devise strategy for validation that leverages vendor documentation/systems as applicable
• Create applicable system requirements specification and design documentation
• Generate requirements-traceable validation protocol and execute validation
• Put in place system use, administration, and maintenance procedures to ensure the system is used as intended and remains in a validated state

This is great stuff, but unfortunately it does not help you answer question #5 for a particular type of software. That’s what I want to try to do here.

OTS really implies Commercial off-the-shelf (COTS) software. The “commercial” component is important because it presumes that the software in question is a purchased product (typically in a “shrink-wrapped” package) that is designed, developed, and supported by a real company.  You can presumably find out what design controls and quality systems are in place for the production of their software and incorporate these findings into your own OTS validation.  If not, then the product is essentially SOUP (keep reading).

Contrast OTS with Software of Unknown Provenance (SOUP).  It is very unlikely that you can determine how this software was developed, so it’s up to you to validate that it does what it’s supposed to do.  In some instances this may be legacy custom software, but these days it probably means the integration of an open source program or library into your product.

This following list is by no means complete. It is only meant to provide some typical software categories and the strategies used for validating them.  Some notes:

• I’ve included a Hazard Analysis section in each category because the amount of validation necessary is dependent on the level of concern.
• The example requirements are not comprehensive. I just wanted to give you a flavor for what is expected.
• Always remember, requirements must be testable.  The test protocol has to include a pass/fail criteria for each requirement. This is QA 101, but is often forgotten.
• I have not included any example test protocol steps or reports.  If you’re going to continue reading, you probably don’t need help in that area.

Operating Systems

Examples:

• Windows XP SP3
• Windows 7 32-bit and 64-bit
• Red Hat Linux

Approach:

1. Hazard Analysis: Do a full  assessment of the risks associated with each OS.
• Pay particular attention to the hazards associated with device and device driver interactions.
• List all hazard mitigations.
• Provide a residual Level of Concern (LOC) assessment after mitigation — hopefully this will be negligible.
• If the residual LOC is major, then Special Documentation can still be provided to justify its use.
2. Use your full product verification as proof that the OS meets the OTS requirements. This has validity since your product will probably only be using a small subset of the full capabilities of the OS.  All of the other functionality that the OS provides would be out of scope for your product.
3. This means that a complete re-validation of your product is required for any OS updates.
4. There is no test protocol or report with this approach. The OS is considered validated when the product verification has been successfully completed.

Compilers

Examples:

• Visual Studio .NET 2010  (C# or C++)

Integrated Libraries

Examples:

• Log4Net (Log4J) — Log generation
• Hibernate/NHibernate  — Database Object/Relational Mapping

Version Control Systems

Examples:

• Subversion (+TortoiseSVN)
• Git (+TortoiseGit)

Issue Tracking Tools

Examples:

• Trac
• Jira
• FogBugz

Validation is a lot of work but is necessary to ensure that all of the tools and components used in the development of medical device software meet their intended functionality.

The complexity of the software employed in many medical devices has rendered inadequate traditional methods (testing) for demonstrating their safety.

The article then provides examples of the types of analyses that can be performed to better ensure safety.

Interesting read.

Here are some references:

BohrBug: Not necessarily easy to find, but once discovered is reproducible.

Heisenbug: The ever-annoying bug that can not be reliably reproduced.

Spin: An open-source software tool for formal verification of distributed software systems.

It seems to me that the editor and any other tool used to create the software is exactly that, a productivity tool. The end result (compiled binary installed on a validated PC configuration) is still going to go through verification and validation, therefore, it seems validating any of the items used to actually create the binary is unnecessary.

Any thoughts or guidance to help me understand this process?

This is a great question and the source of a lot of confusion.

The bottom line is that all third party tools (and libraries) used to construct or test FDA regulated software need to be validated.

You may think validating a compiler is unnecessary, but the FDA says otherwise — section 6.3 of the FDA Guidance on General Principles of Software Validation discussion includes “off-the-shelf software development tools, such as software compilers, linkers, editors, and operating systems.”

The form of the required documentation is detailed in the Off-The-Shelf Software Use in Medical Devices guidance document.  Section 2.1 has the questions that the OTS software BASIC DOCUMENTATION needs to answer:

1. What is it?
2. What are the Computer System Specifications for the OTS Software?
3. How will you assure appropriate actions are taken by the End User?
4. What does the OTS Software do?
5. How do you know it works?
6. How will you keep track of (control) the OTS Software?

There was one item I’d like to comment on:

The challenge is, of course, regarding regulatory management of open source frameworks. To a large degree open source software is anathema to the FDA regulatory process–and it relates to control and management of access.

OSS detested or loathed by the FDA? I don’t think so. The open source framework itself would not be subject to regulatory control.  The FDA does not really care where any specific software component comes from.  Also, security and  access management are certainly important, but I’m not sure of their applicability at the device connectivity level.

The FDA cares about intended use, efficacy, and safety of medical devices. All FDA regulated software is subject to the same design controls (§820.30) — design, risk analysis, verification, validation, etc. I.e. any company that included these open source software components would ultimately be responsible for proving these processes are followed.

Shahid N Shah’s OSCon 2011 Talk: The implications of open source technologies in safety critical medical device platforms does a good job of detailing these points (Will the FDA accept open source in safety-critical system? Yes!) as well as presenting an OSS connectivity software architecture.

In addition to the post-conference workshops, there is also a special preconference event: an open house at the Medical Device Plug and Play Interoperability program’s lab. September 7, from 4pm to 6pm attendees can tour the lab, interact with various demonstrations and chat with program staff.

Tim has put together another great conference.

• Just about all medical devices are computerized these days. Most will not harm or kill you if their software fails (Class I & II), but that’s no excuse for writing crappy code.
• As pointed out, the mission critical nature of mass transit systems (airplanes, subways, etc.) affords those industries a much higher level of scrutiny then cars or medical devices ever will. But that’s still no excuse for writing crappy code.

Even through drugs and airplanes need advance approval from authorities before being brought to the market, medical devices and software do not, at least in the United States.

• This statement is not correct. All medical devices, including diagnostic and therapeutic software-only products, require FDA clearance to be sold in the US market (see the Class I & II link above).  There are many exemptions, but a 510k pre-market approval is generally a minimum requirement. After you receive approval the FDA can pull your device off the market (the dreaded “recall”) at any time due to complaints or unsatisfactory audit results.
• The FDA QSR Subpart C (§ 820.30) looks a lot like DO-178B as quality system design controls go, but I’m sure the aviation standard enforcement is far more rigorous (well, at least I hope it is). It’s true, there are no coding standards for medical device software.  Good companies set their own development standards and practices — some even use static analysis! It’s all the other companies that don’t bother to do anything that you have to worry about.

I’m certain that static analysis technology has improved vastly in the four years since some of the articles below were written.  The challenge is that the complexity of medical device software and the systems they run on has also increased tremendously during that time. In particular, the explosion of high bandwidth wireless networks along with advances in handheld computing power and graphics capability (think iPhone/iPad, of course) is fundamentally changing the way medical devices will be developed and delivered to the market in the future.

Static analysis will remain a valuable tool for identifying critical software defects, but new methods will have to be developed for rooting out risks in the new network-connected, multi-touch world.

It’s sad to say, but you should probably be more than “a little uncomfortable.”

Other static analysis articles:

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World
More Software Forensics and Why Analogies Suck
Medical Device Software Forensics
Pascal’s 3 part static analysis series that starts here:
Guest Article: Static Analysis in Medical Device Software (Part 1) — The Traps of C

Hopefully they’re using an approved non-invasive blood pressure (NIBP) device like the SunTech module.

The built-in ability to e-mail results to family or a physician seems useful, but posting your blood pressure on Facebook or Twitter?  I don’t know…

Hat Tip: medGadget

… see what we can do to “agilify” our practices under these standards as we move forward.

It’s been three years since I wrote Agile development in a FDA regulated setting.  I’ll be interested to see if the application of “agile, high assurance activities” in this environment — and the associated issues — have changed since then.

UPDATE (10/23/10): Can and should agile be used for medical device development? Absolutely!

UPDATE (11/27/10): More discussion here: Can Agile Software Methods be used in medical device software development?

UPDATE (11/28/10): Agile Medical Device Software Development?

UPDATE (12/17/10): GE Healthcare Goes Agile

UPDATE (1/5/11): Missed this one: Four Reasons Medical Device Companies Need Agile Development

• Complexity
• Code Duplication
• Documentation Debt
• Testing Debt
• Architectural Debt

A Martin Fowler article is referenced that nicely identifies the source of technical debt:

The benefits of paying down the debt are:

• Increased R&D efficiency and improved time to market
• Hitting commitment dates
• Performance and technology upgrades

Of course if you don’t want to pay it off, there’s always the option to go bankrupt. This may have long-term advantages, but it will surely be a more expensive route. There is one statement in this regard that I think needs some qualification:

In this case the technical debt can be retired along with the legacy system, and like filing Chapter 11, you are no longer responsible to address all the sins of the past.

I know this refers to code sins, but just because you decide to do a re-write doesn’t mean you no longer have responsibility for the legacy product. You still have customers using the old software that you’re obligated to continue to support.  For FDA approved medical software, this is a legal requirement. Most of the time this means that the legacy code will need to be maintained and periodically updated in the field, sometimes even after the “new” product is released. This just makes the cost of bankruptcy even higher.

Besides providing a globally accepted development process one of the other practical components is the assignment of a safety class to individual software items and units:

• Class A: No injury or damage to health is possible
• Class B: Non-serious injury is possible
• Class C: Death or serious injury is possible

Each classification changes the required documentation for the assigned software.

These standards will become more widely known as the FDA moves to regulate the proliferation of medical applications for personal and home use, most notably software that runs on mobile devices. I’ve discussed this before in When Cell Phones Become Medical Devices. As noted more recently in FDA oversight may extend throughout health IT:

… an FDA director stated flatly: “Under the Federal Food, Drug and Cosmetic Act, HIT software is a medical device.”

Broad FDA oversight at the QSR/62304 level will probably not happen, but change is certainly coming for many HIT companies.

The Elsmar Cove Forum IEC 62304 – Medical Device Software Life Cycle Processes has a lot of discussion on this topic. This is where I found a document checklist that is useful for understanding the process scope:

IEC62304_Checklist.xls (Excel spreadsheet)

UPDATE (9/9/10): IEC 62304 – The Basics