Posts Tagged ‘mdds’

Full Cycle Teams in a FDA regulated setting

Monday, January 4th, 2021

Full Cycle DevelopersThe 200X hot topic was Agile development in a FDA regulated setting. Over a decade later this should (hopefully) be a settled issue. I can’t imagine anyone still doing water-fall these days. The new challenge for medical device companies is implementing Full Cycle Teams (FCTs), which is well described in Full Cycle Developers at Netflix — Operate What You Build.

This organizational structure increases the speed of feature delivery and allows for experimentation to further improve the customer experience. Tooling and automation ("paved roads") are key. The model that Netflix came up with:

"Full cycle developers" is a model where a development team, equipped with amazing developer productivity tools, is responsible for the full software life cycle: design, development, test, deploy, operate, and support.

If you work for a large enough enterprise, you likely have teams of people that provide the following functions:

  • Product development (creates and designs applications software and includes architects, product owners, and scrum masters)
  • Quality assurance (QA). They test the software. For a medical device company, we call this team Verification and Validation (V&V)
  • Site Reliability Engineering (SRE). Ensures scalability and reliability of the infrastructure and applications. They do performance testing and may implement some Chaos engineering techniques.
  • Development operations (DevOps). Manage the code repositories, shared development tools, CI/CD pipelines, middleware, databases, etc.
  • Infrastructure management (on-prem hardware and operating systems)
  • Cloud management (same as above,  but in the cloud)
  • Applications support (monitor and manage applications in production)

Do not confuse FCTs with "Full Stack Teams" (see Full Stack Pronounced Dead). This "stack" refers to technologies that are used to implement a typical web-based application (e.g. LAMP).

FCTs are about supporting functionality end-to-end (product idea to production), but both have the challenge of developer specialization in common. A FCT has to broaden their skill-set even further to include application/infrastructure deployment, monitoring, and support. This is the future!

Full Cycle Team Challenges for Medical Device Companies

The transformation from a legacy organization (as described above) to FCTs is made even more challenging for a medical device company creating software that has to maintain FDA regulatory controls (see Quality System Regulation Subpart C–Design Control § 820.30).

Below is a list of regulatory and transition considerations that impact the release process. Most are associated with keeping the Design History File (DHF) documentation up-to-date. The organizational challenge in a FCT world is figuring out who is responsible for these tasks.

Spoiler alert: The suggested answers should be obvious, but many times the best I can do is just ask the question. Every organization, and even different teams within a single organization, will have different solutions. These can be tough problems to solve. Don't shoot the messenger!

Medical Device Data System (MDDS)

Not all of your software may be under FDA Class II/III regulatory controls. Some could fall under MDDS, see Identifying an MDDS. There is still some risk associated with MDDS but special controls and premarket approval -- the 510(k) -- are not necessary (see MDDS Rule).

MDDS software requires the same QMS documentation (see MDDS Section VI-E. Current Good Manufacturing Practices (CGMP)/QS Regulation/MDR Compliance of the rule) so most of the items listed here still apply.

Also, see Comment 25 from the rule which addresses "modular software". I.e mixing MDDS components with medical device components. The response says "The MDDS regulation does not necessarily prevent modular implementation.", but the FDA can't make a "generalized determination" on the various ways these combinations may be made. This may be a situation you run into and the FDA suggests it is best to contact them if you have questions.

Validation and Verification

General Principles of Software Validation; Final Guidance for Industry and FDA Staff (PDF)

Based on the intended use and the safety risk associated with the software to be developed, the software developer should determine the specific approach, the combination of techniques to be used, and the level of effort to be applied. While this guidance does not recommend any specific life cycle model or any specific technique or method, it does recommend that software validation and verification activities be conducted throughout the entire software life cycle.

FDA guidance documents, and FDA regulations in general (e.g. IEC 62304), tell you what to do, but leave the how up to the organization.  

Let's highlight the SRS* to System test verification from the V-model. This is essentially end-to-end testing. In a microservice-based architecture, each FCT is likely responsible for different sets of services. These services may be dependent on the services provided by other teams.

Which team is responsible for ensuring that the entire system is functioning properly (i.e. end-to-end test protocols and results) after changes are made to one or more of these services?

In an ideal world, these end-to-end tests are completely automated, but even then someone still needs to maintain them.

Validation testing (was the right product built?) presents even more challenges as a single FCT is may only be responsible for a small portion of the entire product.

Risk analysis

From Medical Device Design Risk Management Basic Principles:

Risk analysis is typically done by a cross-functional team that may span multiple business units, but it is probably not unreasonable for the FCT Product Owner to drive this process and get the documentation updated as needed.

Traceability

From the FDA guidance:

A source code traceability analysis is an important tool to verify that all code is linked to established specifications and established test procedures.

Creating this documentation is well suited for automation. It still requires ensuring that all requirements and related test scenarios are properly tagged so they can be parsed to produce a release report.

Software Design Evidence

From the FDA guidance:

The Quality System regulation requires that at least one formal design review be conducted during the device design process. However, it is recommended that multiple design reviews be conducted (e.g., at the end of each software life cycle activity, in preparation for proceeding to the next activity).

This is a challenge for any Agile-based development process so is not specific to the FCT-based organization. Running formal design reviews as early in the development process as possible should be a team responsibility.

Manual Approval Gates

For many unregulated software products continuous integration (CI) and continuous delivery (CD) is a reality. I.e. Code can be pushed, run through the CI/CD pipelines, and delivered to customers without human intervention.

It is very unlikely (not impossible though I suppose, depending on the product) that this would occur for FDA-regulated software. Even with automated document generation, software deployment to production will still require human sign-off steps and audit trails.

Off-The-Shelf (OTS) Software

OTS/SOUP Software Validation documentation needs to be kept up-to-date. This is mostly a book-keeping exercise for OTS/SOUP that is part of the software product. For tools though, see OTS/SOUP Software Validation Strategies.

Another consideration to keep in mind for including 3rd party software into your product is the software license. The corporate (legal) policy should dictate license requirements, but teams would be aided by an automated tracking process.

Infrastructure

Installation, operational, and performance qualification -- IQ/OQ/PQ. FDA regulated software must have these processes in place to ensure that after any changes are made, the infrastructure continues to meet quality requirements.  With the microservice architecture becoming a best practice, the team would now be responsible for documenting the IQ/OQ/PQ for their particular microservice or container flavor(s).

Cloud Offerings

Serverless architectures (Note: I'm most familiar with AWS, so will use their cloud products as examples. Azure and GCP have similar offerings.) One of the key advantages of the LambdaFargate, RDS, and similar managed/SaaS products is their undifferentiated heavy lifting.  AWS is responsible for the care and maintenance of the underlying infrastructure and servers. For on-prem servers, this is something the organization spends significant time and money on, but these expenditures do not directly benefit the customer.  Serverless allows companies to focus their efforts on things that make a difference to their customers.

How do you ensure IQ/OQ/PQ quality when you don't have control over the servers that are running your application(s)?

Another consideration: Teams will need to take regulatory impact into consideration when selecting new cloud technologies.

Infrastructure as Code (IaC)

The use of IaC (e.g. CloudFormation or Terraform) may require new release cycle processes. I.e since this code is not part of the application, you may want to have a separate release cycle for when the infrastructure is updated. The same is true for container (Docker) code updates.

The FCT should be responsible for the IaC associated with their product as it directly impacts both functionality and performance.

Transformation

When thinking about transforming to a FCT-based organization, the 2019 AWS re:invent keynote by Andy Jassy comes to mind. His "transformation" is referring to migrating from on-premise to cloud infrastructure (AWS, of course), but I think the non-technical transformation recommendations he outlines (start: 5:04 end 11:48) are also applicable to the FCT organizational change:

I think aggressive goals (item #2) is particularly important. Legacy organizations have a lot of inertia that needs to be overcome in order to move things forward. Breaking those initial barriers is even more difficult when you're having to deal with regulatory requirements.

Bottom Line

FDA regulatory requirements add tasks and documentation to the software release process. This has always been the case for medical device companies, but how this additional work is managed when trying to implement full-cycle teams can be a complicated problem to solve.

Just like unregulated development, providing the tooling to automate these tasks is the key to allow teams to deliver quality software to customers more quickly.

---------------

*SRS, Software Requirements Specification. The old-school water-fall requirements document. I don't miss those days!

When Cell Phones Become Medical Devices

Tuesday, July 14th, 2009

airstripMobile devices are quickly becoming the conduit of choice for collecting and disseminating clinical data.  The FDA will soon be forced to step in and take regulatory control.  It’s going to happen eventually.

Bradley Merrill Thompson does a good job of outlining the factors that lead to FDA oversight in the article FDA may regulate certain mobile phones, accessories. The Components vs. Accessories distinction is an important one for manufacturers — regulatory oversight is dependent on who buys it. The “intended use”, labeling, and marketing are also factors.

Because of its unique user interface, display, and broadband capabilities the Apple iPhone is a particularly attractive platform for medical applications. For example, the AirStrip OB application is available for download at the Apple App Store and is FDA cleared. Other modalities, like the Critical Care monitor application (shown) is still in testing.

The “intended use” issues are complex.  A cell phone that is used to communicate clinical information, e.g. to a PHR, essentially becomes part of a Networked Medical Device.

This mean that 510(k) premarket notification may also be necessary under the proposed Medical Device Data System (MDDS) rules.  If you read though what constitutes a MDDS, you can see how well the definitions fit mobile device functionality:

  • The electronic transfer or exchange of medical device data from a medical device.
  • The electronic storage and retrieval of medical device data.
  • The electronic display of medical device data.
  • The electronic conversion of medical device data from one format to another format.

Its not the end of the world to be classified as a medical device, but verification and validation of these functions are not a trivial endeavor (see here).

The FDA is almost certainly looking and will be taking action soon.

UPDATE (7/25/09): Here's a mobile device that does not appear to have FDA approval: EKG On Your Mobile Wherever You Are

UPDATE (11/24/09): When will the FDA drop the gavel?

UPDATE (4/19/11): Navigating Regulatory Uncertainty for Smartphones

 

Connecting Computers to FDA Regulated Medical Devices

Wednesday, June 18th, 2008

Pete Gordon asked a couple of questions regarding FDA regulations for Internet-based reporting software that interface with medical devices. The questions are essentially:

  1. How much documentation (SRS, SDS, Test Plan) is required and at what stage can you provide the documentation?
  2. How does the FDA view SaaS architectures?

The type of software you're talking about has no real FDA regulatory over site. The FDA has recently proposed new rules for connectivity software. I've commented on the MDDS rules, but Tim has a complete overview here: FDA Issues New MDDS Rule. As Tim notes, if the FDA puts the MDDS rules into place and becomes more aggressive about regulation, many software vendors that provide medical devices interfaces will be required to submit 510(k) premarket approvals.

Dealing with the safety and effectiveness of medical devices in complex networked environments is on the horizon. IEC 80001 (and here) is a proposed process for applying risk management to enterprise networks incorporating medical devices.  My mantra: High quality software and well tested systems will always be the best way to mitigate risk.

Until something changes, the answer to question #1 is that if your software is not a medical device, you don't need to even deal with the FDA. The answer to question #2 is the same. The FDA doesn't know anything about SaaS architectures unless it's submitted as part of a medical device 510(k).

I thought I'd take a more detailed look at the architecture we're talking about so we can explore some of the issues that need to be addressed when implementing this type of functionality.

mdds2.jpg

This is a simplified view of the way medical devices typically interface to the outside world. The Communications Server transmits and receives data from one or more medical devices via a proprietary protocol over whatever media the device supports (e.g. TCP/IP, USB, RS-232, etc.).

In addition to having local storage for test data, the server could pass data directly to an EMR system via HL7 or provide reporting services via HTTP to a Web client.

There are many other useful functions that external software systems can provide. By definition though, a MDDS does not do any real-time patient monitoring or alarm generation.

Now let's look at what needs to be controlled and verified under these circumstances.

  1. Communications interaction with proper medical device operation.
  2. Device communications protocol and security.
  3. Server database storage and retrieval.
  4. Server security and user authentication.
  5. Client/server protocol and security.
  6. Client data transformation and presentation to the user (including printed reports).
  7. Data export to others formats (XML, CSV, etc.).
  8. Client HIPAA requirements.

Not only is the list long, but these systems involve the combination of custom written software (in multiple languages), multiple operating systems, configurable off-the-shelf software applications, and integrated commercial and open source libraries and frameworks. Also, all testing tools (hardware and software) must be fully validated.

One of the more daunting verification tasks is identifying all of the possible paths that data can take as it flows from one system to the next. Once identified, each path must be tested for data accuracy and integrity as it's reformatted for different purposes, communications reliability, and security. Even a modest one-way store-and-forward system can end up with a hundred or more unique data paths.

A full set of requirements, specifications, and verification and validation test plans and procedures would need to be in place and fully executed for all of this functionality in order to satisfy the FDA Class II GMP requirements. This means that all of the software and systems must be complete and under revision control. There is no "implementation independent" scenario that will meet the GMP requirements.

It's no wonder that most MDDS vendors (like EMR companies) don't want to have to deal with this. Even for companies that already have good software quality practices in place, raising the bar up to meet FDA quality compliance standards would still be a significant organizational commitment and investment.