Archive for the 'Agile' Category

Agile Software Development in Regulated Environments

As part of a series on High Assurance Agile Development in Regulated Environments is the article
Agile Software Development in Regulated Environments Example: Medical Devices. The purpose of this article and future posts is to introduce the FDA regulatory landscape and then

… see what we can do to “agilify” our practices under these standards as we move forward.

It’s been three years since I wrote Agile development in a FDA regulated setting.  I’ll be interested to see if the application of “agile, high assurance activities” in this environment — and the associated issues — have changed since then.

UPDATE (10/23/10): Can and should agile be used for medical device development? Absolutely!

UPDATE (11/27/10): More discussion here: Can Agile Software Methods be used in medical device software development?

UPDATE (11/28/10): Agile Medical Device Software Development?

UPDATE (12/17/10): GE Healthcare Goes Agile

UPDATE (1/5/11): Missed this one: Four Reasons Medical Device Companies Need Agile Development

The Software Quality Balancing Act

Andrew Dallas’s article Caution: V&V May Be Hazardous to Software Quality touches on a number of good points regarding software quality best practices.

Medical device software development V&V (also see here) and the documentation that goes with it have substantial costs. Any strategy that can reduce this overhead and still meet the necessary quality standards should be seriously considered.

The use of “incremental” software development approaches really refers to Agile methodologies.  I’ve talked about the use of Agile for medical device software development several times:

Most of the discussion revolves around the risks associated with this approach. The benefits of any process change have to be weighed against the possible risks that might be introduced.

Besides the importance of understanding what V&V documentation the FDA actually wants to see, Andrew makes a great point about producing quality software versus the V&V process (my highlight):

V&V is not software testing. Verification testing ensures specified requirements have been fulfilled. Validation testing ensures that particular requirements for a specific intended use can be consistently fulfilled.

Following the required FDA V&V processes alone is not sufficient to ensure software quality. You also have to adhere to software development best practices at all levels. For example, in addition to non-functional requirements there are many software quality factors that require careful design considerations and testing that you may decide are outside the scope of FDA reporting.  Deciding what to report and what to leave out is the balancing act.

The Desperate Need for Simplicity

Ted Neward’s article “Agile is treating the symptoms, not the disease” touches on several important points about the software industry.

  • Modern software development tools and technologies require a significant learning curve.
  • Development methodologies (like Agile) exist for managing complexity, but do not reduce the load of these technologies.
  • In the last decade there has been no “Next Big Thing”, like Access was in the 90s.

What’s most interesting to me is:

We are in desperate need of simplicity in this industry. Whoever gets that, and gets it right, defines the “Next Big Thing”.

What’s true in the broader software world is also generally true in Healthcare IT.  In HIT there has never been an Access equivalent, just a lot of pieces and parts trying unsuccessfully to work together.

The need was touched on in Liberate the Data!.  Simplicity is desperately needed in order to create the “First Big Thing” for HIT interoperability.

UPDATE (10/14/09):  More commentary:

Agile Stumbling?

James Shore has an interesting post, The Decline and Fall of Agile, that has generated a lot of good discussion over the last couple of weeks.

I’ve seen a shift in my business over the last few years.

I can’t disagree with the assessment of James’ own business trends (or his colleagues) . But I have to wonder how well his anecdotal sample extrapolates to the rest of the Agile industry?  Without some real numbers to back it up, how can he claim that Agile is either declining or will fail?

But those hundreds of successes will be drowned out by the thousands of failures.

That statement isn’t evidence of a trend; it’s a prediction of the future based to how he happens to be feeling right now. It would be nice if James (or anyone else) could provide some real evidence that the Agile movement is actually in decline — or at least it’s success rate relative to the alternatives, e.g. Waterfall.

Some more views on this are here:
James Shore: The Decline and Fall of Agile
Thoughts on the Decline and Fall of Agile
The Decline and Fall of Agile and How Scrum Makes it Hurt More

These are valuable discussions for developers and managers that don’t have direct personal experience with Agile methodologies but might be considering using them in the future.  It’s clear (to me anyway) that when it comes to software development management, the process is a difficult one no matter what path you take.

Software Development: Driven by what?

First a definition:

driv·en –adjective

  1. : having a compulsive or urgent quality
  2. : propelled or motivated by something — used in combination <results-driven>

Driven software development methodologies abound:

Many of these are encompassed by the iterative Agile software development methodologies. Collectively they are sometimes referred to as the XDD acronyms. As you might expect, these along with all of the other competing, contrasting, and overlapping development philosophies can cause a software developer much consternation. Confessions of a Software Developer* is a good example of the overload that can occur.

My reason for bringing up driven methodologies is not to complain about being overwhelmed by them (which, like most others, I am). It’s simply to point out the contradiction of X-Driven with the Merriam-Webster definition. I think this will help us better understand what should really be driving us.

Look closely at definition #2. Propelled or motivated by somethingresults-driven. What is that something? Ah ha!

The fundamental motivation for all of these development approaches is to:

Improve productivity and quality.

This is the result, the goal. Behavior, Model, Test, etc. are all just the means by which we are trying to achieve this desired result. It’s the result that we’re driven towards, not the methods and techniques we use to get there.

So, in order to make this distinction clear and to eliminate confusion in the future, I propose that all these methodologies be renamed from Driven to Guided. Think of them like you would a GPS system in your car, except these will allow you to find software Nirvana. TDD is now TGD, and the whole lot is known as XGD.

The point here is that you should not let any particular development philosophy blind you to what the real purpose of using it is in the first place. Being guided by a methodology helps me remember that better than when I’m driven by it. Also, the whole concept of being driven seems exclusionary to me. You shouldn’t hesitate use the pieces and parts of any combination of these techniques that best suites your needs.

Dilbert on Agile

In this mornings paper:

Dilbert on Agile

😆

Understanding Software Defects

We tend to focus a lot of attention on tools and methodologies for improving software quality. I thought it would be worth while taking a step back to try to understand what the root causes of software defects are. Fortunately there has been decades of research that have analyzed the most common sources of software defects.

After also looking at some related development sins, I’ll summarize what this new understanding means to me as a software developer.

An often sited article in IEEE Computer is Software Defect Reduction Top-10 List (Vol. 34, Issue 1, January 2001, 135-137) . Here’s a summary (from Software Engineering: Barry W. Boehm’s Lifetime Contributions to Software Development, Management, and Research):

  1. Developers take 100 times less effort to find and fix a problem than one reported by a customer.
  2. Half of software project work is wasted on unnecessary rework.
  3. Twenty percent of the defects account for 80% of the rework.
  4. Twenty percent of modules account for 80% of the defects and half the modules have no defects.
  5. Ninety percent of the downtime comes from 10% of the defects.
  6. Peer reviews catch 60% of the defects.
  7. Directed reviews are 35% more effective than nondirected ones.
  8. Discipline can reduce defects by 75%.
  9. High-dependability modules cost twice as much to produce as low-dependability ones.
  10. Half of all user programs contain nontrivial defects.

This list is based on empirical research and is a good starting point for understanding how to avoid predictable pitfalls in the software development process.

A broader perspective is provided by Pursue Better Software, Not Absolution for Defective ProductsAvoiding the “Four Deadly Sins of Software Development” Is a Good Start. Here are the four deadly sins:

The First Deadly Sin: Sloth — Who Needs Discipline?

The Second Deadly Sin: ComplacencyThe World Will Cooperate with My Expectations.

The Third Deadly Sin: MeagernessWho Needs an Architecture?

The Fourth Deadly Sin: IgnoranceWhat I Don’t Know Doesn’t Matter.

The SEI article concludes:

We believe that the practice of software engineering is sufficiently mature to enable the routine production of near-zero-defect software.

🙂 How can you not smile (or even LOL) at that statement? Despite that, I like the reduction of the problem into its most basic elements: human shortcomings. That’s why the conclusion is so preposterous — software development is a human activity, and a complex one at that. You’re trying to produce a high quality software solution that meets customer expectations, which is a difficult thing to do.

Another list of software development sins can be found in The 7 Deadly Sins of Software Development.
#1 – Overengineering (in complexity and/or performance)
#2 – Not considering the code’s readership
#3 – Assuming your code works
#4 – Using the wrong tool for the job
#5 – Excessive code pride
#6 – Failing to acknowledge weaknesses
#7 – Speaking with an accent (naming conventions)

There are some tool/language specific items here, but this list generally follows the same trend of discovering typical developer shortcomings that can be avoided.

Another source of software defects is poor project planning. More sins (deadly again) can be found in the Steve McConnell article: Nine Deadly Sins of Project Planning.

It’s pretty easy to see from these categorizations where a lot of the software development and management techniques, tools, and practices came from. As you might have expected, many are focused on human behavior and communication as a key component for improving software quality. For example, take the Agile Manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

This vision is very telling about what the manifesto writers considered to be a primary cause of software defects.

Another perspective is Fred Brooks’ famous 1986 ‘No Silver Bullet‘ paper (also see here) that distinguishes “accidental” repetitive tasks from “essential” tasks. From the article:

There is no single development, in either technology or in management technique, that by itself promises even one order-of-magnitude improvement in productivity, in reliability, in simplicity.

Even after twenty years of significant progress in software engineering, I believe that this is still a true statement.

Conclusion:

There are many complex factors that contribute to software defects. There is clearly no one-size-fits-all solution. As a developer, this means that I have to:

  1. Be aware of my own shortcomings and biases.
  2. Continually try to improve my development, communication, and management skills.
  3. Understand how each new tool or methodology that claims to improve software quality fits into the bigger picture of what I (both personally and as an organization) am trying to accomplish.

Medical Device Software Development – Going Agile

I’ve been involved in some informal discussions regarding the use of Agile methodologies for medical device software. The Medical Device and Diagnostic Industry (MD&DI) October 2007 article by Tim Bosch entitled Medical Device Software Development—Going Agile provides a good overview of the challenges that face medical device design and development organizations that want to embrace Agile. Here’s Figure 2 from the article:

A typical agile development process

I liked the organization ‘rejection, force fitting, or abandonment’ analysis. Changing organizational behavior is a difficult thing to do. Add in the documentation requirements and you can see why adopting Agile is an uphill battle. This is especially true for an organization that already has a history of doing software development the old fashioned way.

On the regulatory side, Tim references General Principles of Software Validation; Final Guidance for Industry and FDA Staff and claims that:

An agile development approach aligns well with this guidance.

I’m not so sure about that. As I’ve pointed out before, because of the validation requirements those guidelines are much better suited for the Waterfall development approach. That’s why most people do it that way. Agile can be applied, but it comes with increased cost and potential regulatory risk.

I think the advantages of Agile methodologies are real and application of them does have the potential to improve the functionality, cost effectiveness, and quality of medical device software. It’s good to see articles that detail the issues and provide a realistic strategy for achieving those goals.

Update: Agile development in a FDA regulated setting

I contacted Frank Jacquette regarding my previous port on this subject (Agile development in a FDA regulated setting). His experience using Agile methodologies for pure software medical device projects does not correspond with my conclusion regarding cost effectiveness and regulatory risks. Frank said:

It has been a long while since I wrote that article, but we’ve applied the same approach to some fairly significant systems and they’ve all come in on time and within budget.

He does agree that the regulatory risk is a legitimate concern, but their experience with clients and regulators has always been positive.

I want to thank Frank for so graciously responding to my inquiry.

He also pointed me to a presentation called Integrating lightweight software processes into a regulated environment (warning: PDF) by Adrian Barnes that I had not seen before. This is a far more detailed look at possible solutions for bridging the gap between “Agile Processes” and “Formal Processes”. The subject progression and graphics are very well done. It’s worth a careful look-through. I’ll let you be the judge, but I think Adrian’s conclusions have the same level of skepticism as mine. I broadly addressed cost effectiveness whereas he specifically deals with risk factors for his bridge solution. He has even less faith on the regulatory risk side: “A brave man would try to convince the FDA that Agile is OK”.

It’s always good to have multiple points-of-view on a subject.

Agile development in a FDA regulated setting

I ran across an interesting Agile v. FDA discussion the other day. For those that are not familiar with what a FDA regulated product means, I’ll give a brief overview.

In order to market and sell a medical device in this country — OUS (outside US) medical device regulations are different — you must have FDA approval. Most of the time, this involves a FDA Premarket Notification 510(k) submission. Your company must be registered with the FDA and is subject to periodic on-site visits by FDA inspectors to audit your quality system records. There are two important points here:

  1. Getting approval: In order to sell your device you not only have to prove safety and effectiveness, but you also have to demonstrate that the design and development of the device — including the software — follow Quality System Regulations. The Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices details these requirements.
  2. Keeping approval: After you receive 510(k) approval the FDA can pull your device off the market (the dreaded “recall”) at any time due to complaints or unsatisfactory audit results.

What this means is that your on-going software development process must adhere to a well defined quality system process. As noted in the discussion, the FDA guidance does not dictate that a particular process must be used. The quality system process itself is designed by you — actually, your entire company — and simply needs to ensure that you are designing and building a quality product. The difficult part is that you have to be able to prove to the outside world that you have actually followed that process.

I’ve spent just about my entire career developing software for devices that were under FDA regulatory control. In the old days (pre ~1996) the FDA did not have a good concept of the software development process as part of it’s quality system regulations and inspectors did not usually scrutinize software design and development. Nowadays, the FDA has a much clearer understanding of the software development process. It’s that understanding that is the one of the central issues with respect to adopting an Agile development process for software in medical devices.

Let’s first look at the FDA Good Manufacturing Practice (GMP – Quality System Regulation) requirements. It’s Subpart C–Design Control (§ 820.30) that’s of primary interest. Here’s the outline:

  1. General
  2. Design and development planning
  3. Design input (specifications)
  4. Design output (coding)
  5. Design review
  6. Design verification (Was the product built right?)
  7. Design validation (Was the right product built?)
  8. Design transfer
  9. Design changes
  10. Design history file

The critical issue for the software development process is that each of the items 3-7 require a formal review and approval step. This is the reason why most companies that develop medical device software have chosen to use a quality system process that follows the Waterfall model for their development.

Waterfall

This sequential approach is a natural fit for allowing you to review and document each step in the process.

Now let’s look at the Agile process. One of the better descriptions of the Agile development process is ‘The New Methodology’ by Martin Fowler. There a number of flavors of Agile (SCRUM, Extreme Programming (XP), etc.) that all try to encompass the Agile Manifesto. One advantage that the Agile process has over the Waterfall approach is it’s ability to adapt to the unpredictability of requirements and changing customer needs. This is handled through the use of Iterations. From the Martin Fowler paper:

The key to iterative development is to frequently produce working versions of the final system that have a subset of the required features. These working systems are short on functionality, but should otherwise be faithful to the demands of the final system. They should be fully integrated and as carefully tested as a final delivery.

The purpose of the iterative development is to facilitate requirements changes between each iteration. Agile Methodologies in a Validated Setting by Frank Jacquette proposes some steps to accomplish the use of iterative development in a FDA regulated environment.

Let’s assume that an Agile development process would be able to produce higher quality medical device software and that because of the customer focus of this process the resultant product would better meet market needs. Even with these assumptions, I think there are two major issues that need to be addressed:

  1. I just don’t see how a cost-effective quality system process implementation can be accomplished. Even if (and it’s a big if) the actual overall software development time was shorter, the extra costs incurred by the additional process controls, documentation and testing required for each iteration would far exceed those savings.
  2. The last point Mr. Jacquette makes is the other issue:

    Take the time to explain agile methodologies to your regulatory specialists and work hard to gain their understanding and agreement, because the burden of proof will be on them and you if an FDA auditor decides to take a peek under the hood.

    This seems like a huge risk to me.

The first question becomes: Is the possibility of an improved product (features and quality) worth the additional cost? I suppose if you had a development team with extensive Agile experience you could make the argument that it would be worth it. If not, I think the ROI (return on investment) analysis would be a difficult one to make.

The second question is a big unknown, which is why I think the risk is high. My experience with FDA auditors is generally good. They are professionals that are focused on getting a very specific job done. Since their interest is the entire quality system, the typical audit is a whirlwind affair as it is. The amount of time spent on design control (auditing the Design History File) is usually minimal. Even if you had received 510(k) approval with an Agile design control process, having to take the time to explain to an on-site FDA auditor (who in all likelihood has never seen your 510(k)) a methodology they probably have never even heard of is reason to worry.

Conclusion:

It seems to me that Agile methodologies have a long way to go before we see them commonly used in medical device software development. I’ve searched around and have found nothing to make me think that there is even a trend in this direction. Maybe it’s that Agile processes are just too new. They seem popular as a presentation topic (I’ve been to several), but I wonder how prevalent Agile is even in mainstream software development?

If you are (or have ever been) part of an Agile development team for a FDA regulated product I’d love to here about your experiences and how you were able to resolve the types of issues presented here.

Thanks!

Subscribe

Categories

Twitter Updates