Bob on Medical Device Software

We ran into an interesting problem recently that I have not been able to find documented anywhere.

We’re doing real-time USB data acquisition with .NET 2.0. The data bandwidth and processing isn’t overwhelming. Specifically, we expect data packets at 50 Hz — every 20 ms. Yet we were having horrible delay problems. In the end we found that Console.Writeline was the culprit!

To verify this a test program was written to measure the throughput of the following loop:

while (m_working) {
   Console.WriteLine(string.Format("Message {0} [{1}]",cnt++,mstr));
}

The length of mstr is varied and this loop is run for about 30 seconds. The results show the ms per message for increasing message lengths:

Console.Writeline is surprisingly slow!

We use log4net for our logging. With the timestamp and class information, a log message is typically greater than 100 characters. A single log message introduces at least a 20 ms delay in that case, with additional messages adding that much more. Even though debug logging would not be included in the released version, these significant delays make development difficult.

Not only do you need to make sure that there are no Console.Writeline's in your real-time threads you also need to remove the console appender (<appender-ref ref="ConsoleAppender"/>) from the log4net configuration. The log4net file appenders do not cause noticeable delays.

If you follow Microsoft comings and goings, one of the more interesting developments (at least to me) over the last 8 months has been the formation of a community that calls themselves ALT.NET.

As explained in What is ALT .NET?, the term was coined by David Laribee last year and describes a group of

like-minded individuals within the larger world of the Microsoft® .NET Framework who felt a growing frustration that Microsoft tooling, guidance, and .NET culture at large did not reflect or support an important set of core values.

The name is misleading because even though most members are from the .NET community, the group’s purpose is to promote a set of core values that are platform/language independent. To summarize from Jeremy’s article:

1. Keeping an eye out for a better way.
2. Adopt the best of any community.
3. Not content with the status quo — experimenting with techniques.
4. It’s the principles and knowledge that really matter.

The members of the ALT.NET group are distinguished technologist and many are productive bloggers, e.g. codebetter.com and Ayende@Rahien. Also, the discussion group altdotnet is very active (over 6200 posts since the beginning of the year) and lively. There are also periodic group meetings (see the ALT.NET site for links) that use Open Space Technology (OST) to organize conference agendas. Check out the interesting videos (by David Laribee) from the recent conference in Seattle.

So why are ALT.NETters not like the rest of us? We’re experienced developers that use modern tools and techniques, but we:

• Have never used enterprise-class frameworks and tools (e.g. Biztalk, P&P Application Blocks, ESB, TFS, etc.).
• Have never worked with a “Software Architect”. We have always had to design and develop our own systems.
• Have experimented with Agile development methodologies but have never been part of a “real” Agile team.
• Think Pair programming is an April Fool’s joke.
• As with Agile, we know about all the different “driven” software development approaches, but have never had the opportunity to fully embrace any of them.
• Have heard about Boo, Spec#, and F#, but have never used them.

This list could go on and on. Many have never used an ORM or the MVC design pattern either. The point isn’t what we know versus what they know. I’ve talked about Stereotyping Programmers before and how it’s just plain bad. I think the ALT.NET community has made a conscious effort to improve their inclusiveness.

The ALT.NET group is certainly on the cutting edge of useful and innovative software technologies and techniques. We may not understand everything they’re talking about, but the conversation is well worth listening to. Someday you may be faced with a challenge that will need just the type of solutions they’ve been discussing.

I’m not a computer scientist. I’m also not one of the many über programmers that create and analyze software frameworks and techniques. I simply design and develop software that attempts to meet my customer’s needs. To that end I’m always looking for the best tools available to get the job done.

Jeremy Miller states the importance of design patterns well:

I know many people blow off design patterns as ivory tower twaddle and silly jargon, but I think they’re very important in regards to designing user interface code. Design patterns give us a common vocabulary that we can use in design discussions. A study of patterns opens up the accumulated wisdom of developers who have come before us.

My opinion: You don’t need to be a rocket scientist to understand design patterns. Most are just common sense. Complex patterns are designed to solve complex problems. Design patterns should be thought of as a tool that you use just like any other. Don’t let the ‘ivory tower twaddle’ scare you away.

I think most people would agree that one of the key components to creating a successful software product is quality. I’ve developed .NET applications in the past and have experienced the difficulty of testing and maintaining the functionality of Winform forms and components when they are created with the default Visual Studio tools. If you’re not careful, here’s what you end up with:

Photo hat tip: Josh Smith from Using MVC to Unit Test WPF Applications — a very good and relevant article.

I should note here that the development of software for medical devices already has rigorous verification and validation processes to ensure quality. See FDA Good Manufacturing Practice (GMP - Quality System Regulation) subpart C–Design Control (§ 820.30 sections f & g). However, these requirements do not preclude the need for development techniques that make the software more robust and maintainable. On the contrary, the higher the software quality, the easier it is to meet these standards.

I’ve recently spent some time trying to select a GUI architecture that will allow us to create a more robust unit testing environment. This is why I started looking at Model-View-Controller (MVC) and Model-View-Presenter (MVP) design patterns. The need for these types of design patterns is twofold:

1. Separation of Concerns (SoC), which allows for
2. Improved Testability.

There are many articles and blog posts that describe MVC, MVP, and their numerous variations. These techniques have been around for many years but the current corner-stone comes from these Martin Fowler articles:

• GUI Architectures
• Passive View
• Supervising Controller

Once you understand these concepts you can start to grasp the trade-offs of all of the available MVC/MVP flavors. If you’re like me, one of the problems you’ll run into is that there are so many different approaches (and opinions) that you’ll be left wondering which is best to implement. The only advice you’ll get is that it’s a matter of choice. Great, thanks! From the article above, Josh puts it best:

If you put ten software architects into a room and have them discuss what the Model-View-Controller pattern is, you will end up with twelve different opinions.

This is when you turn from the theory and start looking for concrete implementations that might be suitable for your situation. Microsoft has released an ASP.NET MVC Framework as part of VS2008, but all of the Winform code samples I found were part of either blog posts or articles.

As you look at the different implementations (and relevant snippets), you quickly realize that following these design patterns requires significantly more work than just adding your application’s logic directly to the IDE generated delegates. The additional work is expected and is the trade-off for improved testability.

That’s fine, and worth it, but it’s still time and money. We do not have the resources, or experience, to undertake a full Test-Driven Development (TDD) effort. We will implement MVC/MVP on only the displays that we feel are the most vulnerable.

I’m not going to list all of the candidate examples I looked at. I will mention that Jeremy’s series of articles (here) dig deep into these issues and have lots of good code examples. Each approach has their pros and cons, just like the one I’ll present here. We’ll try to use it, but may end up with something else in the end. As we become more experienced, I suspect we’ll evolve into a customized solution that best meets our needs.

A Promising Candidate:

Implementing the Passive View — a Derivative of the Model-View-Control by Matthew Cochran.

This hybrid approach appealed to me for a couple of reasons. The first is that I spent several years doing Swing development, which uses a MVC that also allows multiple simultaneous views of a single model. I also like the event driven approach, which is not only heavily used in Java, but is also well supported in .NET. In any case, the View is passive and all of the important functional logic is centralized in the Controller class which can be easily tested with real or mock Model and View objects.

Matthew has done a good job of providing support generic classes that make implementation somewhat less cumbersome. The MvcControlBase class provides generic Control-View wiring while ChangeRequestEvents manages all events in a single class.

The project download provided by the article is a VS2008 solution. We’re still using VS2005, but I was able to back-port the project to VS2005 with only minor modifications that had no effect on functionality. The VS2005 project is available for download here:

MVC-PV-2005.zip (23K)

Final Thoughts:

I see adoption of MVC/MCP methodology for GUI development as a critical component for improvement in software reliability, quality, and long-term maintainability. Also, structuring the application side with MVC/MVP is only half the battle. Developing an effective testing strategy must go along with it in order to achieve these objectives. Until Microsoft provides an integrated Winforms MVC solution like they did for ASP.NET, we’ll just have to continue to roll our own.

I’d like to hear about your experiences, suggestions, and recommendations on any of these topics.

Thanks!

UPDATE (6-May-08): Here’s a good MVC article by Jeff Atwood: Understanding Model-View-Controller

UPDATE (16-Jun-08): Another reference: Everything You Wanted To Know About MVC and MVP But Were Afraid To Ask

Last night I went to the launch of the San Diego Connected Systems SIG (and here). Brian (along with Chris Romp) gave a great overview of BizTalk Server 2006 R2.

I have never used BizTalk and had little knowledge of its capabilities going in. BizTalk reference material and articles can be found in numerous places on the web — a good summary is Introducing BizTalk Server 2006 R2 (pdf).

My major take-aways from the presentation were:

• BizTalk is an enterprise class product — i.e. a heavy weight solution designed to scale for very large business needs (global reach, high throughput, tight control and policies, highly reliable).
• As such, the learning curve is steep.
• BizTalk uses a message oriented architecture designed to connect disparate systems of all types.
• Some of the key BizTalk tools include:
  • Sending and receiving messages with Adapters
  • Orchestrations
  • Business Rule Engine
  • Message processing with Pipelines
  • Message translation with Data Mapping
  • Business Activity Monitoring (BAM)
• BizTalk uses a publish/subscribe model that allows for asynchronous message handling.
• Most development tools are integrated into Visual Studio. Some of the visual message mapping needs present real GUI challenges.
• BizTalk Server 2006 R2 will include Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF) integration. WF will not replace Orchestrations.
• Microsoft SQL Server is used as the back-end database and is very tightly bound to BizTalk functionality and performance. The message persistence capability of the Message Box is a powerful built-in tool.
• The Microsoft Enterprise Service Bus (ESB) Guidance further uses BizTalk to support a loosely coupled messaging architecture.
• BizTalk will also be a key component of Microsoft’s new service-oriented architecture (SOA) framework called Oslo.

Because of its message handling architecture it’s easy to see how HL7 translation and routing could be accomplished. Microsoft provides accelerators (pre-defined schema, orchestration samples, etc.) for HL7 and HIPAA for this purpose.

It’s not hard to understand the importance of BizTalk in the larger Enterprise space. It appears to be benefiting from its years of prior experience and continued integration with other evolving Microsoft technologies. Overall, I was very impressed with BizTalk.

A Note on Special Interest Groups

I’m not only lucky to have a SIG like this in the area, but it’s also great to have people as knowledgeable (and friendly) as Brian and Chris running it. Great job guys!

I would encourage everyone to seek out and attend their local user/developer group meetings. Don’t just go for the free pizza (which usually isn’t that good anyway) — it’s a great way to improve yourself both technically and professionally. You’ll also get to meet new people that have the same interests as you.

I think that getting exposure to technologies that you don’t use in your day-to-day work can be just as rewarding as becoming an expert in your own domain. Learning about cutting-edge software (or hardware) is exciting no matter what it is. That new knowledge and perspective also has the potential to lead you down roads that you might not have considered otherwise.

Last night I attended a San Diego .NET Users Group meeting where the topic was LINQ. The presentation was done by Julie Lerman (http://www.thedatafarm.com/blog and http://blogs.devsource.com/devlife).

Since I have an interest in ORM (see here) I’ve done some reading on LINQ in the past. It’s always amazing to me how much more you seem to learn from a presentation. Especially when the speaker is well organized and knowledgeable and provides an engaging delivery. Great job Julie!

LINQ is very impressive. The new .NET Framework Orcas (VS 2008) language features include:

• Automatic Properties
• Extension Methods
• Lambda Expressions
• Query Syntax
• Anonymous Types

These not only provide a foundation for ORM work, but are also powerful .NET language and tool additions for just about any programming task.

This is a follow-up to the Developing a real-time data flow and control model with WCF post. My original plan was to write a full-fledged article on this. I’ve gotten some requests for the code, but it does not appear that I’m going to have time to complete the article in the near future. So I thought I’d just give a brief description here of what I’ve done so far and provide the code as is.

Please Note: The description provided is very brief and only meant as an overview. None of the implementation details are included here. It’s not a very complicated project. If you have some VS2005 development experience and are willing to dig into the code, you shouldn’t have a problem figuring it all out. Working through this code would provide a good first tutorial on developing with WCF. Some set-up is required (service installation) and is described below.

Motivation:

I originally conceived of this project because of some questions I’d heard from real-time C/C++ developers. They wanted to know about migrating from MFC/VC++ to .NET managed code. The primary concern was about the use of legacy device level code and how to manage future mixed development projects.

So my first thought was to demonstrate how easy straight forward it is to incorporate COM components into managed code with .NET wrappers. There are already many good articles on integrating old Win32 projects into .NET. e.g. Do you COM? Dealing with Legacy Projects. This project is a concrete example of how that can be done.

It also illustrates a model of the type of real-time data streaming and control typically required by a physiological monitor.

To extend that model, I wanted to show how WCF could be used as a network transport for that same data stream. Hence the previous post. The addition of a WCF client application that provided a real-time display of the data stream was only logical.

There are a number of directions that I had planned on taking this project, but that will have to wait for another day. I’m sure that you’ll come up with your own ideas along with numerous improvements.

The Code:

The download (below) is a Visual Studio 2005 solution, called RealTimeTemplate, with 6 projects (one is C++, all the rest are C#). Here is a diagram of the projects and their relationship. The horizontal arrows show the data flow as described above.

The projects are:

• SineWaveGenerator: This is a C++ COM component that generates buffers of multi-channel sine waves.
• SineWaveGenerator.Test: The NUnit test functions for SineWaveGenerator
• SineWaveWCFLib: This is the WCF server component.
• SineWaveWCFServer: This the Windows service that hosts the WCF service (SineWaveWCFLib).
• SineWaveWCFService.Test: The NUnit tests functions for SineWaveWCFService.
• RealTimeDisplayWinForm: This is the Windows Form class, and WCF client, that controls and displays the sine wave data provided by the service. The graphical display is done using the ZedGraph library.

Here is what the Windows Form looks like when the application is running.

Service Installation:

In order to run the sine wave display application, you’ll first have to install and start the SineWaveWCFService.

1. Build the solution in Debug configuration.
2. Execute the InstallSineWaveWCFService.bat in SineWaveWCFService\bin\Debug. The service can be un-installed with the UninstallSineWaveWCFService.bat script.
3. Run the Windows Services Manager (services.msc) and Start SineWaveWCFService.
4. Run the RealTimeDisplayWinForm project. Use the Add button to add sine wave displays.

The source code can be downloaded here:

RealTimeTemplate_20070923.zip (210K)

Enjoy!

This is follow-up to the Kernel Object Namespace and Vista post. Those previous findings were made using an administrative user in Vista. When I tried creating a ‘Session\AppName’ Mutex as a non-administrative user though, the application hung!

Just to be clear, here is how (simplified) I’m creating the Mutex:

string MutexName = @"Session\AppName";
bool mutexWasCreated;
MutexSecurity mSec = new MutexSecurity();
MutexAccessRule rule = new MutexAccessRule(
      new SecurityIdentifier(WellKnownSidType.WorldSid, null),
      MutexRights.FullControl,AccessControlType.Allow);
mSec.AddAccessRule(rule);
Mutex m = new Mutex(false, mutexName, out mutexWasCreated, mSec);

The hang occurs when the Mutex is created. By hang I mean that the process just spins its wheels sucking 50-60% of the CPU and will continue until it’s killed. Based on WinDbg analysis it’s either stuck in the underlaying Win32 CreateMutex() call or CreateMutex() is being called repeatedly. It’s probably the later.

When ‘Local\’ or ‘Global\’ are used, the Mutex is created fine! As noted before, ‘Local\’ doesn’t work for other reasons so I’m stuck using the ‘Global\’ namespace. Go figure?

Just a quick development note:

According to Kernel Object Namespaces objects can have one of three predefined prefixes — ‘Local\’, ‘Global\’, or ‘Session\’. For Win2K/XP I’ve always used the ‘Local\’ prefix, which works fine. My primary use is with a Mutex to determine that a single instance of an application is running (like here). I also use the Mutex from a system service to discover if a GUI application is available for messaging. When trying to run the some code on Vista I found that the ‘Local\’ namespace does not work when Mutex.OpenExisting() is called from a the system service which is owned by a different user (from the same user, it works fine). So it appears that the ‘Local\’ prefix in Vista has a different behavior for the client session namespace than it does in Win2K/XP.

I searched around for a solution, but was unable to find a definitive answer. I did find a post about the Private Object Namespace which alludes to Vista kernel changes, but that’s all. Here’s what I determined empirically:

The NO entries in the table mean that the namespace did not work. So, it appears that in order to support all three Windows versions I’d have to use the ‘Global\’ namespace. This is not a good solution. Unless I find another way, I’ll have to determine the OS version and select the appropriate namespace at runtime (’Session\’ for Vista, ‘Local\’ for Win2K/XP).

A Windows Communication Foundation (WCF) service is defined through its operations and data contracts. One of the major benefits of WCF is the ease with which a client can create and use these services through the automatically generated proxy classes. The service side is only half of the communications link though. Discovering the correct WCF configuration options that allow a solution to operate properly was not as easy as I thought it would be.

This post describes a specific WCF-based data control and streaming architecture. The primary goal of this service is to provide a continuous data stream (as buffers of short values) from a real-time data acquisition source. The client would then be able to display the data as it became available or store the data when directed by the user. In addition, the service allows the client to both get status information (Getters) and control certain attributes (Setters) of the underlying data source. This is illustrated here:

The DataBufferEvent is defined as a one-way callback and continuously delivers data to the client. The IsOneWay property is valid for any operation that does not have a return value and improves network performance by not requiring a return message. The Getters and Setters [for you Java folks, this has nothing to do with JavaBeans] can be called at any time. Changing a data source attribute with a Setter will probably affect the data stream, but it is the responsibility of the data source to ensure data integrity. The underlying transport binding must support duplex operation (e.g. wsDualHttpBinding or netTcpBinding) in order for this scenario to work.

Here is what an example (a sine wave generator) service interface looks like:

namespace SineWaveWCFLib
{
    [ServiceContract(SessionMode = SessionMode.Required,
        CallbackContract = typeof(ISineWaveWCFLibCallback))]
    public interface ISineWaveWCFLib
    {
        // Setters are one-way operations
        [OperationContract(IsOneWay = true)]
        void SetWaveformFrequency(int wfm, double freq);
        [OperationContract(IsOneWay = true)]
        void SetSamplingPeriod(int sper);
        [OperationContract(IsOneWay = true)]
        void SetBufferSize(int count);
        // Getters are two-way operations (default)
        [OperationContract]
        double GetWaveformFrequency(int wfm);
        [OperationContract]
        int GetWaveformCount();
        [OperationContract]
        int GetSamplingPeriod();
    }
    // Define the callback contract interface
    public interface ISineWaveWCFLibCallback
    {
        [OperationContract(IsOneWay=true)]
        void DataBufferEvent(short [] data);
    }
}

The service class is implemented as follows:

[ServiceBehavior(InstanceContextMode =InstanceContextMode.PerSession,
        ConcurrencyMode=ConcurrencyMode.Multiple)]
public class SineWaveWCFLib : ISineWaveWCFLib
{
    private ISineWaveWCFLibCallback callback = null;
    // Constructor
    SineWaveWCFLib()
    {
        // Initialize COM object and setup serv_DataBufferEvent callback.
        callback = OperationContext.Current.GetCallbackChannel< ISineWaveWCFLibCallback>();
        // Other initialization...
    }
    // When a buffer is ready serv_DataBufferEvent is called by the underlaying COM object
    // that is producing the data. GetDataBuffer turns pOutBuff into a short [] array.
    // None of that code is shown here.
    private void serv_DataBufferEvent(Array pOutBuff)
    {
        callback.DataBufferEvent(GetDataBuffer(pOutBuff));
    }
    // ISineWaveWCFService Members not shown.
}

The InstanceContextMode.PerSession mode is appropriate for this type of interface. Even though there is probably only a single data source, you still want to allow multiple service session instances to provide data simultaneously to different clients. The data source would be responsible for managing the multiple data requesters.

With the service side complete, all the client needs is to do is create the proxy classes (with either Visual Studio or Svcutil), setup the DataBufferEvent callback and call the appropriate control functions. My first client was a Winform application to display the data stream. The problem I ran into is that even though the data callbacks worked properly, I would often see the control functions hang the application when they were invoked.

It took quite a bit of searching around before I found the solution, which is here. You can read the details about the SynchronizationContext issues, but this caused me to spin my wheels for several days. The upside is that in trying to diagnose the problem I learned how to use the Service Trace Viewer Tool (SvcTraceViewer.exe) and the Configuration Editor Tool (SvcConfigEditor.exe, which is in the VS2005 Tools menu).

So after adding the appropriate CallbackBehavior attributes, here are the important parts of the client that allow this WCF model to operate reliably:

public partial class Form1 : Form
{
        private SineWaveWCFLibClient m_client;
        [CallbackBehavior(ConcurrencyMode=ConcurrencyMode.Reentrant,
                           UseSynchronizationContext=false)]
        internal class DataBufferEventHandler : SineWaveWCFLibCallback
        {
            private Form1 m_form;
            public DataBufferEventHandler(Form1 form)
            {
                m_form = form;
            }
            public void DataBufferEvent(BindingList< short> data)
            {
                m_form.UpdateWaveforms(data); // not shown: Uses Invoke to update display
            }
        }
 
        private void Form1_Load(object sender, EventArgs e)
        {
            InstanceContext site = new InstanceContext(new DataBufferEventHandler(this));
            m_client = new SineWaveWCFLibClient(site, "NetTcpBinding_SineWaveWCFLib");
            try
            {
                m_client.SetBufferSize(50);
            }
            catch (EndpointNotFoundException)
            {
                MessageBox.Show("Unable to connect to " + m_client.Endpoint.Address.ToString() +
                                        "!nQuiting application.");
                m_client = null;
                Application.Exit();
            }
        }
}

The first take-away here is that WCF is a complex beast. The second is that even though it’s easy to create proxy classes from a WCF service, you have to understand and take into account both sides of the communications link. It seems so obvious now!

That’s it. This WCF component is just part of a larger project that I’m planning on submitting as an article (with source code) on CodeProject, someday. If you’d like the get a copy of the source before that, just let me know and I’ll send you what I currently have.

Update: Proof of my first take-away: Callbacks, ConcurrencyMode and Windows Clients. Thanks Michele!

I’ve updated my SolutionZipper source and installer to version 1.3 on CodeProject. Here are the changes:

• Fixed a bug that was causing SZ to fail during the “Handle external projects” phase.
• Ignore VC++ Intellisense Database files, i.e. *.ncb.
• Ignore hidden files and folders.

I originally wrote this last year as simply a convenience function. Even though I use a source code control system (Subversion) at work, I still need a quick way to snapshot and backup my personal projects at home.

I recently started a solution that included a C++ project and noticed some problems. First was that there was no need to backup the VC++ Intellisense database file. The second problem might be related to one of these:

• Microsoft Visual Studio 2005 Professional Edition - ENU Service Pack 1 (KB926601)
• Visual Studio 2005 extensions for .NET Framework 3.0 (WCF & WPF), November 2006 CTP
• Microsoft ASP.NET 2.0 AJAX Extensions 1.0

I don’t know which one caused the problem, but after one of these was installed VS2005 had project list items that were not file system based — a project called <MiscFiles>? Anyway, this caused the search for external projects to fail.

There was a request to ignore Subversion (.svn) directories. This was a good idea so I just ignore all hidden directories and files. This also means that VS Solution User Option files (.suo) are not included in the zip file.