Since 2015, every medical device company operating in the US has been required to report to the Food and Drug Administration (FDA) on any adverse events related to what they produce. This includes fatalities, serious injuries and other material issues related to public safety.
Medical Device Reporting (MDR) regulation applies to both local manufacturers and foreign importers, even if their devices are only likely to cause an incident.
Manually handling this sort of requirement has proven burdensome, so more and more companies are turning to the latest advances in AI and machine learning to assist with meeting their obligations. Just as quality control in medical devices directly impacts the health and wellness of patients, quality control in the regulatory compliance of machine learning tools has a direct impact on the financial health of the parent company.
Clearly, there are heightened challenges for managers in optimizing machine learning solutions within a regulated environment. Specifically, there are five areas of concern that deserve extra attention by business leaders from the very beginning of any machine learning project.
1. Deploy regression analysis for clarity
How will you make sense of machine learning logic for the average user? Even in extremely complex fields like healthcare, users don’t feel comfortable without a simple explanation for why the model made predictions in a certain way. For most machine learning or deep learning techniques, explanations are hard to come by as they tend to operate as black boxes.
The programmers may understand the logic, but the end users are expected to simply accept the results. Fortunately, there are ways to derive a schema for the interpretations using a simpler model, such as linear regression or logistic regression. Although these merely approximate the logic, they are ideal for the purposes of user interpretability of the final conclusions.
2. Simplify the user interface (UI) for reporting
How are end users consuming the results of the machine learning model? Before you invest in a sophisticated but complex UI, consider implementing something much simpler, with end users in mind. Begin with monitoring and tracking how users view and interact with simple reports. As they build familiarity and mastery of the tools, you will be able to build out a more robust workflow that streamlines operations while retaining drill down complexity.
3. Build in iterative improvement cycles
How will human oversight feedback be provided back to the model? It’s okay if this involves some manual steps. Accuracy takes precedence over efficiency. Generally, consider it more important to ensure that when a user notices a false positive or a false negative, they are immediately able to flag the data in the system so that the machine learning algorithm can learn from these inputs and build more accurate models for the next set of reports.
4. Consolidate and streamline processing
How will you make sure the machine learning processes are optimally analyzing the data? A good example is in how system resources are allocated. Once a specific medical device is production or distribution, the machine learning processes should be redirected to generating models that only make predictions on records where circumstances have changed. This is especially true when the results are meant to impel users to take action. It makes little sense to keep re-predicting futures for those records that have been previously predicted and handled.
5. Continually search for improvements to the model
Finally, does the predictive capacity of the machine learning algorithm ever really reach perfection? At some point, these processes will reach a state of maximum reliability, but until then, managers should keep searching for ways to improve results. Every machine learning project should include a formalized process to tune the model production process and review metrics to ensure that existing models are on track. New sources of data and new ways of validating that data become available every year. Be absolutely confident that model accuracy metrics are improving over time.
Machine learning for human protection
The field of governance, risk and compliance (GRC) is one of the fastest growing sectors for machine learning applications, particularly when laws and oversight guidance documents are frequently changing in the healthcare sector. Healthcare firms tend to exhibit a sort of unique dichotomy in their approach to technology – patient-facing technology is often cutting edge while back-office systems remain traditional and very slow to adapt. The most successful firms are bringing the two sides into alignment, applying the processing power of advanced AI to solve the toughest problems both medically and organizationally.
Medical device manufacturers and importers have a highly useful tool at their disposal in the form of the FDA’s Manufacturer and User Facility Device Experience Database – (MAUDE). This contains a wealth of data from as far back as 1993 when companies reported this sort of data voluntarily. Although MAUDE data is not intended for use in evaluating rates of adverse events nor for comparing adverse events across devices, but machine learning specialists will find it invaluable in predictive capability testing. In the end, if even one patient can be saved from serious injury, these programs will have more than paid for themselves.