The meeting was ending at Mythical Medical.

“We believe this product is safe and performs as expected. We have received very few complaints, and the benefits outweigh risks when compared to
state-of-the-art products.” 

Few questioned that authoritatively delivered closing message as several stood up applauding, whistling, and shouting. Mythical Medical employee Kai-Lin, on the other hand, scratched her head whispering under her breath, “I don’t get it. What data do we have to support this standing ovation?” Her comment almost went unheard.

In this fictional story, I’ll guide Kai-Lin through an effective process that can acquire this data quickly. I will be addressing relevant topics and providing simple, practical solutions to Kai-Lin that can be implemented globally.

Show Me The Data

In spite of the standing ovation, Kai-Lin knew that optimism is not data. Confidence is good, but the facts on your side are better. Many manufacturers, looking across their total product portfolio, recognize they don’t have sufficient current, clinical data to re-license them under newer regulations, e.g., MDR/IVDR. Time is running out.

Although firms meet milestones and most of their key performance indicators, a familiar “Show me the data,” voice is being heard more and more worldwide in hallways, conference rooms, and over lunch. Product certificates will be expiring in less than a year, and regulators, along with Notified Bodies, have made it clear that, although opinions and beliefs are lovely, decisions for certificate renewals must be based on facts.

After Kai-Lin contacted me, I learned that she was right. There were several products that had insufficient clinical data to obtain approval. For the first project, I recommended using surveys to quickly obtain safety and performance data that, in turn, could be used to also create benefit-to-risk analyses within the current state-of-the-art of their product.

Surveys

Surveys represent one option or solution to obtain clinical data, but they need to be developed correctly and compliantly. They can be used to either be the primary source of safety and performance data, e.g., for low-risk devices, or they can be used to fill in the gaps for higher risk devices where conducting a clinical investigation or registry is not practical. For all risk classes, surveys might be targeted to address specific questions from patient or user groups, healthcare professionals, regulatory bodies, or an independent, accredited body.

Surveys can be developed as either open-responses or closed-responses. I have dealt with both, and they each have their purposes and challenges. However, for this article, I’ll be recommending closed-response questionnaires. Then, you can choose a Likert scale format (pronounced “Lick-ert”), the more dominant method used. Developed in 1932 by Rensis Likert to measure attitudes, the typical Likert scale is a 5- or 7-point ordinal measure used by respondents to rate the degree to which they agree or disagree with a statement. I initially recommend using a 5-scale approach.

One advantage of closed-response questions is that it focuses on the respondent’s attention to address the critical issue at hand. Second, uniformed data can easily be entered into a database for analyses. Finally, they appeal to those hesitant to respond to items that might require a short narrative (or novel) to explain themselves.

Surveys need to be scientifically and clinically developed with a sound statistical sample size estimation and subsequent analysis plan. Especially in global studies, be sure to utilize local resources to ensure the questionnaire is culturally appropriate. Finally, the time to complete the survey is essential. Initially aim for 10 minutes.

Step 1 Create Research Questions For The Data

Start with an initial target goal of about 8-12 questions based, in part, on risk, quality, and clinical gap assessments. Conduct an internal technical documentation assessment addressing what evidence currently exists and how old it is; data which is already known about the product. Assess its quality and relevance. Perform a systematic literature evaluation to either add to what is known or confirm what is unknown. Then, meet with whom I refer to as “manufacturer-friendly” healthcare professionals that desire to be a part of a solution. Include a few questions to assess internal consistency.

This is going to take time, and you want to work with someone willing to help as they apply their unique, expert insights. They do not necessarily need to be using your product. Minimally, they just need to know and understand devices like yours. Finally, identify specific safety and performance issues.

Create statements that might be addressed with a targeted survey. Use a Likert 5-scale approach. Initially, select one question that could be used for sample size calculations. Conduct a pilot survey in the field to test, analyze, and make final changes. Assess internal consistency with your questions and validate the survey to ensure it is collecting the information you intend to collect.

Next, develop an extensive interviewer training program to minimize interviewer bias, knowing it may need to be tweaked once you launch this activity in the field. It is also good to remind personnel to report product performance issues that they learn about. Present this strategic plan to the regulatory agency or Notified Body as part of an overall surveillance discussion. Notified Bodies are not allowed to consult, so manage internal expectations. Begin discussions and gain alignment on which question will be selected to ensure an adequate sample size is selected.

In Mythical Medical’s case, a guidewire was identified as their initial target. The following issues were identified as gaps in their technical documentation: (1) delivery to the lesion, (2) ability to cross the lesion, (3) procedure success without complications at hospital discharge, (4) procedure success identified by another device being delivered, e.g., imaging, balloon, stent, and finally, and (5) comparative information on another guidewire. Number 2 above was selected as the question for the sample size determination using a 5-scale Likert frequency survey form, e.g., never, rarely, sometimes, often, always. I have a list of 37 Likert-type scale responses and the top 10 Likert-type 5-scale responses that I would be pleased to share. Ten questions were developed for the final survey.

Step 2 Calculate A Sample Size For The Data

If done well, survey results for the sample population should reflect the results you would have gotten from the entire group. Use a validated sample size software program that would be accepted by competent authorities, e.g., Australia, Argentina, Brazil, Canada, China, European Union, India, Israel, Japan, Mexico, Portugal, Kingdom of Saudi Arabia, South Korea, Taiwan, Thailand, United Kingdom, United States of America, etc. An initial suggestion could be nQuery (www.statsols.com) or Power and Precision (https://www.power-analysis.com/index.php). I use both, but there are many others. The study will compare two groups using an identical scale to get information on two commonly used guidewires. The null hypothesis is that the effect size or response from both groups will be equal. We intend to disprove the null hypothesis.

This is the technical piece, but I will keep it high level for this simulation. In this case, the Wilcoxon/Mann-Whitney rank-sum test (ordered categories) for equal “n’s” was used. The alpha level was set at 0.05 (criteria used for statistical significance), a two-sided test was chosen, the number of categories was 5 (see above). Estimates were obtained from each of the 5 Likert responses for the question being used for sample size calculations based on a similar survey Mythical Medical found in the literature.

The initial results suggest sample sizes per group of n=75, n=100, n=125, and n=150 and estimated powers of 68%, 80%, 88%, and 93%, respectively. This is where the fun starts. The manufacturer needs to align internally in terms of risks they are willing to take. On the one hand, you want enough observations to obtain reasonably precise estimates of the parameter of interest, but also want to do this within a reasonable resource budget. The important thing is to quantify risks associated with the final sample size that is chosen. There is also the consideration of missing data, which in this simulation was estimated to be 3%.

In this case, Mythical Medical selected the third option using n=125 in each group with an estimated power of 88% to allow us to conclude that the response differs between the two groups. They did not want to consider the 3% assumption for missing data. There are other issues to discuss, but let’s keep it simple. From a budget perspective, we will have ten questions being answered by 250 participants that will give us 2,500 completed questions. Thus, assume you need $20,000 (direct survey costs), convert to your local currency. Next, adjust budgets considering your unique business model. Factors that might affect budgets could include (1) whether it will be in-sourced or outsourced, (2) translations, (3) travel expenses, (4) healthcare practitioner fees, and (5) fees associated with compliance with good clinical practice, privacy laws, safety reporting, etc. The total budget for this project if implementing all 5 factors above, could creep up to $100,000, though (direct and indirect costs).

Step 3 Collect The Data

Initially, Mythical Medical wanted to set this up online using online survey programs and then send out an email to those they wanted to complete the survey. That is undoubtedly a standard method used to conduct surveys. The challenge here, as discussed with Kai-Lin, is the company had limited experience conducting surveys for regulatory submission purposes. Those in the survey area are all too familiar with low return rates, potentially non-representative samples, inability to clarify the question, and too much time spent on demographic issues.

In this case, I recommend selling the value of conducting face-to-face interviews, especially since the company has little experience conducting hypothesis testing surveys. One benefit is it is a tremendous customer-facing activity. There are other secondary benefits when meeting one-on-one with your customers. Mythical Medical will also need to have the right personnel in the field meeting with customers. These are typically not from sales or marketing, per se, but from those having clinical expertise such as seen within clinical research, medical affairs, quality, regulatory, engineering, biostatistics, or safety. As more experience is obtained, recommendations could vary.

Step 4 Analyze The Data

The Mann-Whitney U test (a non-parametric test) was applied to evaluate data from each question. Although it is acknowledged in the literature, there has been a longstanding, fascinating debate regarding analyzing Likert-scale data; it is beyond the scope of this article. The discussion between the ordinalists (saying that you should use non-parametric tests) and the intervalists (arguing for parametric tests) is ongoing. Without going too far into the simulated results, within eight months, this approach yielded robust data from 10 questions per survey applied to two treatment groups. These results now are beneficial because they can be used, in part, to develop future surveys and serve as the basis to test new hypotheses in the future.

There are several software programs to assist you in running the analysis, e.g., Prism (which I use), R statistic, SAS, and SPSS. You can also purchase programs like Q1Macros (which I have also used) that embed statistical packages into EXCEL, which I have found really convenient. This is especially helpful to quickly run Mann-Whitney U tests once you have entered the data in an EXCEL spreadsheet.

Step 5 Data Becomes Inputs Into Key Processes

These results should become inputs back into your quality system and technical documentation processes. The results have direct relevance to design input. Any product performance issues that were uncovered should be processed in a timely manner too.

Likert 5-scale response sets are robust and can be applied to appraise frequency, quality, intensity, agreement, approval, awareness, importance, familiarity, satisfaction, and performance. The Likert scale is Rensis Likert’s best-contribution. He created the scale in 1932 as a part of his Ph.D. thesis and its usage has universal application.

Depending on the expertise involved, open-ended questions to assess product safety such as, “Were there any adverse events not identified in the Instructions for Use?” could be added in future surveys.

Implications To EU MDR/IVDR

Within the EU MDR/IVDR regulations, the following sixteen documents should be updated as appropriate, e.g., Strategy for Regulatory Compliance (Article 10 & Annex IX), Risk Management Plan and Report (Article 10 & Annex I), Clinical Development Strategy (Article 61), Clinical Evaluation Plan and Report (Article 61 & Annex XIV), Clinical Development Plan (Annex XIV), Clinical Investigation Plan and Report (Article 62, Article 63, & Annex XV), Post-Market Surveillance Plan (Article 84 & Annex III), Post-Market Surveillance Report for Class I devices (Article 85 & Annex III), Periodic Safety Update Report or PSUR (Article 86 & Annex III), Post-Market Clinical Follow-up Plan and Report (Article 61 & Annex XIV), Summary of Safety and Clinical Performance or SSCP (Article 32), and Claims (Article 7).

Closing Thoughts

When initially developing effective surveys, there are two steps that take the longest; Step 1 (creating the questions) and Step 3 (obtaining the data). Kai-Lin asked the right question early enough and chose a method to obtain the data to support that closing remark which led to a standing ovation. She actively participated in finding a solution and learned valuable skills along the way. The product was approved.

A colleague overheard that “under the breath comment.” Knowing the value survey results brought to the company and witnessing both how well she led the project and worked well with others, her colleague ensured she was recognized.

Of course, this was with Mythical Medical, but these principles also apply at your “Real Medical” too and you can implement them when you go back to work tomorrow. You know where your company has been and where it is going. Be that person who inspires greatness when clinical data is needed. Kai-Lin trusted her instinct and was part of both developing and implementing a simple survey strategy. You can, too!

David R Rutledge, Pharm.D., FCCP, FAHA, President & CEO, Global Strategic Solutions, LLC, Silicon Valley in California. david.rutledge@globalstrategicsolutions.com. +1 (630) 846-0350 cell.

Previously I read his first book, Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. (2009), Introduction to Meta-Analysis. West Sussex, United Kingdom, John Wiley & Sons, ISBN: 978-0-470-05724-7. Yet, I absolutely fell in love with his newest book, Borenstein M. (2019), Common Mistakes in Meta-Analysis and How to Avoid Them. Englewood, New Jersey, USA, Biostat Inc, ISBN: 978-1-7334367-1-7 (softcover). Hence, I want to review it for you.

Dr. Borenstein is acknowledged as a distinguished expert and is exceptionally positioned to author this landmark textbook. I have read the book and found it to be a critical addition to my library. It is helpful in expanding and developing my knowledge of meta-analysis, and especially in avoiding common mistakes when conducting systematic reviews of interventions. This is in its first edition (ISBN: 978-1-7334367-1-7 the softcover version) and available on Amazon.

The author has chosen to identify and explain 42 mistakes made when conducting these important analyses. He uses excellent, easily understood examples for you to apply and learn the principles of meta-analysis. The author explains why each is a mistake, the implications of the mistake, and how you can avoid the mistake.

For Those New To Meta-Analyses: You will never forget the history behind the term “forest plots” and will learn why you should refer to them as “forest plots” and not “Forest plots” even though you will see that term referenced in the literature.

If you are new to this area of statistical analysis, there is no better book for you to read at this early stage in your career. Don’t be afraid to be a beginner. It provides a historical perspective to help you understand where the industry was and where it is headed in terms of the evolution of meta-analysis when applied to social, behavioral, educational, medical, clinical, regulatory, or legal applications.

If you need to know the basics and wonder “Why do meta-analyses and what are the different models that can be used?” this book, starting with the preface and continuing through chapters 6 and 7, is for you. Before you get to page 42 you will learn what many don’t seem to know or remember; how to decide on the model to use upfront, prior to your analysis. Dr. Borenstein will give you a solid explanation to support that fundamental principle. Refer to chapter 4 to obtain websites for this book, the software program Comprehensive Meta-Analysis, upcoming workshops, and then register your email to receive updates to remain current and competent. Finally, review the glossary in chapter 15. Now you will be ready to actively participate in your first meta-analysis team meeting using your solid, fundamentally sound foundation.

For Those Seasoned Professionals: If you are a seasoned professional, grounded in the basics already, you will enjoy the following chapters with discussions regarding (1) issues and myths about statistical models, (2) heterogeneity, (3) mistakes related to significance testing, (4) publication bias, (5) mistakes in subgroup analyses, (6) features of the Comprehensive Meta-Analysis software program, and (7) how to correctly report the results of an analysis. Relevant examples are presented. Study them. For a more in-depth review, the author includes ten appendices that contain supplementary material which may be helpful in providing a more comprehensive understanding of your research question or problem.

Reflecting On My Experience Reading This Book: I personally found value in (1) knowing what the 42 mistakes were, (2) the easily understood Summary Sections the author provides, and (3) the “Putting It All Together” sections. He clarifies the issues that were previously ambiguous. The section on heterogeneity and prediction interval, an index of dispersion, was of enormous value and very practical to me. I learned what the “I squared” (I2) statistic correctly is and is not; it really doesn’t tell you how effects vary, despite the widespread misconception in the literature. It tells us what proportion of the variance in observed effects reflects variation in true effects, rather than sampling error. It does not tell us how much variation there is. Rely on the prediction interval instead. Finally, templates were provided to assist with reporting results that communicate clearly to my intended audience.

When the treatment effect (or effect size) is consistent from one study or observation to the next, meta-analysis can be used to identify this common effect. When the effect varies from one study or observation to the next, a meta-analysis may be used to identify the reason for the variation. For example, when you run the analysis, why does one population seem to not benefit and actually appear to be harmed, while another population has a small treatment effect, a third tends to have a modest effect, and finally some appear to have the largest treatment effect? Now it’s our opportunity to put those detective shoes on and set up strategies to identify, address or evaluate those differences in effects. Regulatory bodies, clinicians, and Notified Bodies will want to confirm that your product, for example, is working as intended in the approved population. Performing a meta-analysis is one way to assess product safety and performance.

In this life, few are given the responsibility or opportunity to change the world (or at least a piece of it where you live) by analyzing significant data correctly. Don’t model your comprehensive reviews and analyses on what others have published in the literature. Base it on the principles learned in this book. It is a 5-star, brilliantly written, expertly communicated reference that has received my highest recommendation. Avoid making the 43rd mistake by not having this as part of your library.

If you believe that decisions affecting people’s lives in areas of clinical, public health or policy settings should be informed by relevant research evidence using meta-analyses, then this review is dedicated to you. With the proper education, training and other easy to use tools that exist today, you too can be on the verge of changing the world where you live. How about your experiences? I’d love to hear more heartwarming stories from the analyses you have performed.

Key Websites For Your Continued Learning

https://meta-analysis-books.com/, www.cochrane.org/, https://www.meta-analysis-workshops.com/, and training.cochrane.org/.

David R Rutledge, Pharm.D. FCCP, FAHA
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350 cell
Based in Silicon Valley, California, USA

In this episode, David Rutledge from Global Strategic Solution will share with us 6 ways to collect clinical data for your Medical Devices. He will also help you understand which one is more suitable for your situation.

David also shared with us a presentation that you can download from the links listed below. Don’t hesitate to reach our to him if you have more questions.

• David Rutledge Linkedin: https://www.linkedin.com/in/david-rutledge-9291609/
• Download Presentation: https://mailchi.mp/easymedicaldevice/clinical_data_source

(Or watch on the official website: https://podcast.easymedicaldevice.com/55/ )

David R Rutledge, Pharm.D, FCCP, FAHA
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350 cell
Based in Silicon Valley area, California, USA

In this version of the script, the character Jon “David” Snow becomes the 998th Lord Commander of Evaluations of the Night’s Watch in Game of Thrones Season 05, Episode 02, which was supposed to premiere May 26, 2017.

One photo was found of Jon David Snow from this episode. Facial recognition software suggests it bears a striking resemblance to this article’s author when he attended a recent Halloween party.

After narrowly winning the vote, he slowly moves to the front of the darkened room. Less than half of those in the chamber did not vote for him and many just hate the new Lord Commander. There is silence.

He has an introspective, reflective and cold look on his face as he remembers that the European Union (EU) Medical Device Regulation (MDR) comes into force today. It gives manufacturers three years to meet the new requirements.

The enforcement deadline for compliance with the EU MDR is May 26, 2020. That deadline is looming large for many in the room. Some think it is a long time away. Others, like Jon David Snow, have more immediate concerns. He turns around and raises his finger for dramatic purposes.

Jon “David” Snow Addresses the Room

“My name is Jon David Snow, Lord Commander of Evaluations. We are not friends. We have never been friends. I suspect we won’t become friends today. This is not about friendship. This is about survival. Survival from the MDR White Walkers.

I give you this bag. Please pass it around. This is a gift. In it, you will find daggers made of Valyrian Steel or as you know them, sufficient clinical evidence daggers. One of my servants used this to kill an MDR White Walker. 

Shortly afterward, a small group of Notified Bodies known as Wildlings started chasing me. One of them, a very desirable, known and appealing one, started shooting arrows at me, telling me that, ‘You know nothing about the MDR, Jon David Snow.’

I know in my heart that those Notified Body Wildlings are not our enemies. They have their unique concerns with the MDR White Walkers themselves. We need to work together.

It is not the MDD/AIMD Kings and Queens that we should fear now. It is those White Walkers! The only known weakness that the MDR White Walkers have shown thus far is that they are killed instantly when struck with ‘sufficient clinical evidence’ weapons made of Valyrian steel. 

Don’t forget your past submissions. They were hard and it took time; time away from the ones we love. I won’t forget mine. What I am asking you today is to think about your future submissions. Winter is coming. MDR is just around the corner. Certificate renewals are coming. New product submissions are coming. It will not be like it used to be.

Your senior leaders can’t stop it. Our associations can’t stop it. Lords, Presidents, and CEOs can’t stop it and none of their knights can either. Notified Body Wildlings cannot stop it (He pauses to look at several Notified Body Wildlings in the room).

Only together, all of us, can we overcome this and even then, it may not be enough. But at least we will give those MDR White Walkers a fight. Who is willing to stand with me?”

Those in the room start chanting, thumping and pounding the tables, “Save our device. Save our device. Save our device!”

A raven suddenly flies into the room delivering Jon David Snow a message. It simply states, “EUDAMED will be delayed two years, but you won’t hear about it until Q4 of 2019.” The scene ends.

Lesson 1 – MDR date of application is coming

Characters in almost every other episode allegedly quote that the MDR “winter is coming,” referring to the date of application being May 26 2020. While you shouldn’t spend all of your time worrying about the future, you need to be mindful that hard, stressful, difficult times can happen, usually at the most inconvenient moments. It is the only way you can be prepared for them. If you need statistically and scientifically sound, compliant clinical data you have a few choices. Take a risk-based approach and use surveys (e.g., Likert), big data from large hospital networks (e.g. 1MData), investigator-sponsored studies, national registries, or single-arm or randomized clinical investigations. Some devices may require several approaches. Be sure to get the question correct in terms of what you are asking and ensure it aligns with either filling in a gap or helping to better understand the current information that exists on the device.

Lesson 2 – The mind needs the MDR like the sword needs a sharpening stone

Take the time to read the EU MDR and more than once. The preamble tells you why we need this legislation. The articles inform what needs to be done. The annexes suggest how to do it. Knowledge is power. The medical device directives have been out since 1990 and 1993 and we spent years discussing, challenging, and questioning that legislation. We will do the same for the EU MDR too, but a fundamental principle will be that you need solid, first-hand knowledge of what is written. Don’t rely on what others say it says.

Lesson 3 – A device’s death comes for everyone

Don’t get too emotionally attached to your medical device as it exists today. Iterations, advances in technologies, and changes in the state-of-the-art will occur. Quality management systems are set up to be living systems with one goal of incorporating feedback into medical device design. Challenge the device. If you don’t ask the right questions, you won’t get the right answers. Be analytical and critical of the inputs and outputs into the quality system and provide both alternatives and innovative solutions.

Lesson 4 – Sometimes you have to answer the call when it comes

Like Jon David Snow, usually, time picks you. You don’t pick the time for greatness or opportunities within an organization. The industry needs your personal leadership today. Optimism is not data. Confidence is good. Facts on your side are better! We are long past the days of relying on, “We believe the device is safe.” The EU MDR Chapter VI, Article 61, Annex I, and Annex XIV clearly speak to the need to have evidence or clinical data to base decisions on.

Manufacturers shall specify the level of clinical evidence. It needs to be aligned with the characteristics of the device and intended purpose in order to to demonstrate conformity to General Safety and Performance Requirements (Annex I). Discussing these requirements with senior leaders and those controlling budgets requires your leadership. Preparing for the MDR is like laughing with your family and friends. You never have enough time. Plans need to be in place, and execution on those plans is urgent.

Lesson 5 – Listen to your Notified Body

If the Notified Body wants chocolate, quit trying to sell them vanilla. One of the best ways to persuade a medical or scientific reviewer is with your ears—by listening to them. Scrutiny has never been higher, and decisions made by the Notified Body on why devices were approved and what levels evidence they required will become transparent to all when reported publicly through the new MDR-required Summary of Safety and Clinical Performance document for implantables and Class III devices (Article 32).

It’s Okay If You “Know Nothing”

It’s not an insult. The medical device world is vast and full of different cultures, histories, and viewpoints. We have the EU MDR, and although it is filled with useful and great information, there is still much to learn. You should always be open to learning and listening to other perspectives. We will figure this out together as Jon David Snow implies.

A closing thought. Finding clinical data or evidence in your clinical evaluation or technical documentation should not be a scavenger hunt for the reviewer. Remember, the value of your company is determined, in part, by the quality of your data. Survival from the MDR White Walkers is possible, but not simply by saying, “We believe our device is safe and performs as intended.”

Working together with your Notified Body and listening to their current interpretation of the EU MDR is an approach that could increase the likelihood of successful applications if they contain the evidence necessary to make those approval decisions. Draw your Valyrian steel “daggers of sufficient clinical evidence” now, all of you who are willing to fight! Survive the MDR White Walkers.

Will artificial intelligence (AI) and machine learning (ML) be the next big thing in medical devices? Will it be that new adventure for you?

These technologies have the potential to transform health care and medical devices and are at the center of a new revolution in this field. The ability of AI and ML software to learn from real-world feedback (or training) and improve its performance makes these technologies remarkably positioned to further the development of software as a medical device.

AI is a general name for ways that computers and computer programs build models of the world from data and examples. These models can then be used to answer questions, find patterns, predict futures, etc. It is broadly defined as the science and engineering of making intelligent machines, especially intelligent computer programs.

ML is an AI method that can be used to design and train software algorithms to learn from and act on data. Software data analytic engineers can use ML to create an algorithm that is “locked” so that its function does not change, or “adaptive” so its behavior can adjust over time based on new data inputs. It is this unlocked, active, real-world learning that is so very exciting.

AI and ML technologies differ from other software in that they have the potential to adapt and optimize device performance in real-time to continuously improve health care for patients. Many clinical, regulatory and quality personnel in the medical device industry are familiar with “inputs and outputs” within a quality management system. Conceptually, there are similarities here. However, the regulatory framework to support this will have to mature.

Demystifying Artificial Intelligence (AI) and Machine Learning (ML)

The figure below is commonly used in the field to introduce AI and ML. In simplistic terms, it is like a set of Russian dolls nested within each other. AI is an umbrella term for any computer program that does something smart. AI is not ML, per se. If there is something you can do in a few minutes or hours, AI, with the right rules, or “if-then” statements, in place should then be able to do that in a second or less. AI is the overarching discipline that covers anything related to making machines or devices smart. Once AI begins learning and starts making suggestions, then ML has kicked in. AI, and specifically ML, are techniques used to design and train software algorithms to learn from and act on data.

If you use the typical cruise control within your vehicle, then you are using the results of AI. Cruise control is not making decisions for you but using inputs and outputs to set the velocity you have chosen. Commonly used cruise control is not changing your speed based on driving conditions, so it’s not technically ML. If you have email spam filters or priority mail filters on your computer, then you are using AI. If you have a smartphone that uses location technology, maps can be used to show you where you are. Vast amounts of additional data can be fed into software programs to assist with finding fuel, food, hotels or resorts. All ML is AI, but not all AI is ML. As helpful as these AI software products are, they typically lack the ability to learn independently. They cannot think outside their code. Machine learning thus is a branch of AI that aims to give machines the ability to learn a task without pre-existing code.

However, if your smart phone begins to incorporate user-reported traffic incidents like construction and accidents, and learns from that, then it can suggest the fastest routes. If that program is learning from your previous choices, then it can suggest the types of fuel, food, or resorts that are best for you. These would be examples of ML. Other examples would be voice to text and typing auto correction.

AI and ML are good for large data sets where patterns are difficult to find using human intelligence alone. AI is good at understanding interactions. The challenge is to ask a specific, relevant question.

Nisha Talagala is the CEO of Pyxeda AI here in Silicon Valley and she provides her perspective on AI’s future potential.

“AI has tremendous potential for medical devices, with applications ranging from diagnostic assistance to personalized health, preventative care, and holistic wellness. The AI industry is maturing and moving from a research-oriented phase to broader commercial impact, driven by innovations in cloud tools and automation.”

-AI Club

Programs like http://aiclub.world, enabling individuals of varied domains and expertise levels to learn and apply AI to their fields, accelerate the broad production value of this transformational technology.” 

Demystifying Software as a Medical Device Under the EU MDR

With much of the recent emphasis of device manufacturers centering on meeting the requirements of the upcoming European Union (EU) Medical Device Regulation (MDR) and In Vitro Diagnostic Regulation (IVDR), pausing to briefly reflect on how this field might affect your work now, and in the future, is timely. It’s already affecting you at home anyway.

The EU MDR states that it is necessary to clarify that software in its own right, when specifically intended by the manufacturer to be used for one or more of the medical purposes set out in the definition of a medical device, qualifies as a medical device. See EU MDR Whereas statement (19) on page 4. However, software for general purposes, even when used in a healthcare setting, or software intended for lifestyle and well-being purposes is not a medical device. The qualification of software, either as a device or an accessory, is independent of the software’s location or the type of interconnection between the software and a device.

The EU MDR (Chapter 1, Article 2, page 15) provides the following definition of a “medical device” to better understand how software can be a medical device too. A medical device means any instrument, apparatus, appliance, software, implant, reagent, material or other article intended by the manufacturer to be used, alone or in combination, for human beings for one or more of the following specific medical purposes:

— diagnosis, prevention, monitoring, prediction, prognosis, treatment or alleviation of disease,

 — diagnosis, monitoring, treatment, alleviation of, or compensation for, an injury or disability,

 — investigation, replacement or modification of the anatomy or of a physiological or pathological process or state,

— providing information by means of in vitro examination of specimens derived from the human body, including organ, blood and tissue donations, and which does not achieve its principal intended action by pharmacological, immunological or metabolic means, in or on the human body, but which may be assisted in its function by such means.

On October 2019, the Medical Device Coordination Group (MDCG) released a helpful guidance document 2019-11, “Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR.” It gives clear direction that medical device software that is intended to be used, alone or in combination, for a purpose as specified in the definition of a “medical device” (above) in the EU MDR/IVDR will be strictly regulated.

The use of software as a medical device can have its own intended purpose and also drive or influence the use of a device or hardware for a medical purpose. Software then is classified on its own, based on its intended medical purpose. As you work your way through the risk classification algorithm, it cannot be classified in a risk class lower than the risk class of the medical device or hardware it is influencing. Not all software used in healthcare settings qualifies as a medical device; software used for staff planning, invoicing and simple searches would not qualify as a medical device.

Demystifying Software as a Medical Device within the Global Regulatory Environment

Let’s put this discussion into a broader, global regulatory context. The International Medical Device Regulators Forum (IMDRF) defines software as a medical device as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device. The USA Food and Drug Administration (FDA) under the Federal Food, Drug, and Cosmetic Act (FD&C Act) considers medical purpose as those purposes that are intended to treat, diagnose, cure, mitigate, or prevent disease or other conditions.

The FDA recently released a discussion paper on AI and ML software and its potential implications to the medical device industry (April 2019 https://www.fda.gov/media/122535/download). 

FDA has defined AI as, “A device or a product that can imitate intelligent behavior or mimics human learning and reasoning. Artificial Intelligence includes machine learning, neural networks, and natural language processing. Some terms used to describe artificial intelligence include computer-aided detection/diagnosis, statistical learning, deep learning, or smart algorithms.”

Demystifying How AI and ML can Transform Software as a Medical Device as We Know it Today

Intriguing applications could include more accurate diagnosis, earlier disease detection, identification of patterns or clinical data relationships previously undetected, and development of personalized diagnostics and/or therapeutics. One of the greatest benefits of AI and ML in software resides in its ability to learn from real-world use and experience, and its capability to improve its performance.

In 2018, the FDA authorized a type of clinical decision support software designed to analyze computed tomography (CT) results that may notify providers of a potential stroke in their patients. The agency also authorized an artificial intelligence-based screening device for detecting diabetic retinopathy, an eye disease that can cause vision loss in some diabetic patients.

The medical device industry is encouraged to commit to broader educational programs on the application of AI and ML. We need to transform from relatively unfamiliar engineers, clinical/regulatory/quality scientists, and healthcare practitioners to more informed, critical and key developers and users of this technology.

The intended use of AI and ML-based software as a medical device brings new possibilities but also requires a solid understanding of current regulations guiding software in this industry today and then having open discussions on how regulations and education might need to evolve.

Now to be a bit provocative. What’s good enough in AI and ML when applied to medical devices? Should we be solving 80%, 90% or 100% of the problem? There could be value in solving 80% of the problem, but do we know in what areas? How do we build and apply AI and ML that patients, healthcare practitioners, and regulators can trust? Finally, should we think more critically about testing these applications that learn, modify and change within a randomized clinical investigation setting? There are other provocative areas, such as:

1. What is the potential for bias as the software is learning?

2. Should lower risk devices be allowed to use “unlocked” software first?

3. Should manufacturers be allowed to let each device learn and evolve separately versus all together (or both)?

Using software as a medical device and then expanding and combining this to develop and apply additional algorithms create new, amazing, and wonderful possibilities. The hope is to deliver safe and effective device functionality that improves the quality of care that patients receive. There is a need to befriend AI and ML to both continue medical device innovation and remain competitive globally. In fact, there is a lot of room for innovation.

I started out by asking, “Will artificial intelligence (AI) and machine learning (ML) be the next big thing in medical devices?” What do you think? Will a new friendship with AI help you to choose new ideas, new experiences, and new adventures in the medical device field? I hope so because this is going to fun. Let’s go!

David R Rutledge, Pharm.D., FCCP, FAHA 
President & CEO
Global Strategic Solutions, LLC
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350 cell
www.globalstrategicsolutions.com

There exists vast amount of creative solutions to leverage real-world, state-of-the-art clinical data. It actually can be fun but has potential to create new workaholics if you have this passion. Patients, healthcare practitioners, patient advocacy groups, Notified Bodies (NBs), competent authorities, and the European Commission do not want your device to miserably be left behind. Real challenges exist for the medical device industry to transform from those old directives to the new European Union (EU) Medical Device Regulation (MDR).

Here is the problem. Manufacturers (and other stakeholders) recognize they don’t have enough current clinical data to either re-license their devices under the existing regulations or to license them under the upcoming EU MDR. They are running out of time. The industry is far behind in recognizing what is possible. Few are looking forward to 2020, but you can.

Until there is further guidance, there exist six proven methodologies or ways to actively and systematically gather, in a scientifically and valid manner, post-marketing evidence that can be used to update technical documentation files. This method can be applied to all classes of devices, but a risk-based, decision tree approach is suggested. In fact, using multiple approaches could be relevant for many devices. It depends, in part, on the scientific question being asked and how much existing, relevant clinical data the manufacture possesses. These assume relevant local and national regulations are followed, e.g., patient privacy laws, good clinical practice, safety reporting, etc.

These are ranked from least costly and lowest level of evidence to the more expensive, but highest level of evidence. A draft financial starting point is provided for each column in US dollars with one “M” representing $000. For example, $200 M would represent $200, 000. Please adapt these values to represent your business model and currency.

1.    Surveys 

Although this may be the lowest level and least costly option, real value can be obtained if scientifically and statistically created correctly and you get information from the right group. This could be a business opportunity to visit and engage customers, users, etc. directly (versus a mass mailing) using the appropriate manufacturer personnel, of course. Feedback is usually rapid and depending on the method used, can be either dialog-based or they just check the box.

Challenges exist. Potential downsides would be limited experience with surveys, low survey return rates, and potentially non-representative sample. Finally, “This could be a trap!” Many financial officers or budget owners will want all their devices to use surveys. Although it really depends on the question, generally it would be difficult to convince NBs and their medical reviewers that this is the right risk-based model for class III, implantables, or active devices. I have had those conversations and would love to hear your experiences in an “off the clock” setting, of course.

2.    Hospital Databases

If you need clinical data quickly, you will love this. As the healthcare industry is booming electronically, this represents a forgotten, clever, and shrewd decision tree method. If done correctly, many regulatory agencies or bodies have found value in reviewing this type of real-world device experience. Especially where several large hospitals are contributing. If you desire a new, exciting professional challenge, this opportunity is for you. I can personally attest that this was a viable pathway for re-licensure with the United States Food and Drug Administration, several NBs, and brilliant, thoughtful competent authorities, e.g., Australia, Argentina, Brazil, Canada, China, India, Israel, Japan, Mexico, Portugal, Saudi Arabia, Taiwan, Thailand, etc. Although the barrier was lower for acceptance from acute use products, it was also successfully applied with implantable devices.

With volumes of clinical information available within large, geographically based hospital networks, it is very difficult for regulators to dismiss what this valuable evidence represents.

Challenges exist. First, you need to identify both a local physician/user advocate because they know how to interpret what their data means (and what it does not mean) and their database manager who knows not only how to extract data elements, but also the validity and limitations of the system. Second, there are challenges with maintaining hospital contracts to obtain data long-term. Of course, in the real-world, users practice medicine. Thus, the information may not be representative of on-label or as intended in the approved population. This is actually good news. Manufacturers are expected to have processes in place to assist with identifying possible systematic misuse or off-label use of the device, with a view to verifying that the intended purpose is correct.

3.    Investigator-Sponsored Studies (ISS) aka Investigator-Initiated Trials/Studies/Research

These clinical investigations are submitted for funding to a manufacturer. The lead or principal investigator (PI) assumes regulatory responsibilities. These are usually orders of magnitude cheaper than industry-sponsored studies. They can provide important clinical evidence on device safety and performance. Contracting is usually with the institution, rather than the PI and remains in effect if the PI leaves; thus, allowing the investigation to continue by promoting a qualified, sub-investigator to that vacated role. This approach can help manufacturers with public procurement or tender processes. If aligned with clinical development plans, this approach could generate important, initial clinical data in other populations to assist with future product development.

Challenges exist. PIs may not be interested in core products (just not exciting to them) and prefer studying what’s new. Second, once the primary endpoint is reached, many are not excited to follow-up on research subjects year-over-year, for example. Research grants always have funding at significantly lower levels for follow-up visits. It’s a business or revenue stream, so it’s common for PIs to move to the next big project. Data quality may not be as high, usually because of lower levels of monitoring and limited audits. Next, some manufacturers have restrictions on “directing” the investigation. They are required to be more hands off. For those experienced with clinical investigations, it takes longer lead times to initiate a site(s) compared to the previous two methods mentioned above. Data may not be aligned with the product label. Seasoned PIs are typically more interested in pushing the product into risky populations versus overseeing a surveillance investigation. However, this would be an opportunity for newer PIs that are interested in working with the industry. The perception is it’s not real science and the results may be difficult to publish in major medical/scientific, peer-reviewed journals.

4.    National Registries

Where these exist, they are known within the industry because, in part, they may be required by law or supported by medical associations. Data is published on a regular basis in the medical literature or as publicly available annual reports since many countries are financially supporting these. It is predicted that the industry will be seeing more of these in the future. Manufacturers are encouraged to find ways to empower these national registries.

Challenges exist. First, there is limited ability to alter or change the database quickly. Not all device components are included. There is less flexibility to create downloads from the database “off schedule.” Additionally, data lag exists in terms of when data is uploaded from sites. This could be 6 to 9 months which is painful if a safety signal needs to be evaluated quickly. Finally, there is usually limited ability to go back and investigate a specific data point.

5.    Small Product Registries

Many manufacturers have experience running product registries. It’s becoming a core competency. The industry finds manufacture-friendly (and competent) PIs and sites, many of which desire to better understand technologies when used in their hands. Typically practice standards are different in various geographies and many have a desire to understand safety and performance in those environments. These registries can also assist with tender processes and complement product development activities.

Challenges exist. Few device registries exist on lower profit margin products. Similar to ISSs mentioned above, there is a long lead time to set up registries, recruit subjects, and then follow them until the primary endpoint is reached. The PI’s enthusiasm decreases if they are required to follow subjects year-over-year. Many PIs, in general, believe it’s not science after the primary endpoint is reached.

6.    Large Manufacturer Registries

The thought here is to set up an infrastructure and follow all products in one large registry. This is more common in smaller companies. Many times, there are limited exclusion criteria (all comers or more comers design). The manufacturer has more control over the database, monitors more frequently and performs scheduled audits to ensure higher integrity of the clinical data.

Challenges exist. This approach impacts R&D pipelines because of limited financial resources. The profitability of products may lead to a reassessment of lower-margin products. As mentioned for ISSs and small product registries, there exist lead times to start the registry, get to the endpoint and PIs find year-over-year follow-up boring.

Don’t let your device be left behind. These creative, strategic solutions can assist manufacturers to plan their evidence creation strategy, in order to improve the approval process. Get the right people involved to help with planning to identify or confirm safety, performance, or effectiveness, e.g., physicians, statisticians, users, NBs, database managers, regulators, advocacy groups, etc. Others could be involved too, e.g., consultants, advisors, clinical research organizations, and other relevant partners. These six strategies are all great sources of clinical data that can be included in clinical evaluations and post-market surveillance plans. There is no need to create new workaholics.

The thirst for data is hard to satisfy. It’s been my experience that data leads to data. Questions lead to answers, then more questions and the creation of more data. This aligns with EU MDR commitments to life-cycle or lifetime management of medical devices.

Predictability in product approvals is a competitive advantage. Implementing proven approaches like these increases this predictability. Pay attention to upcoming Common Specifications. Factor this in as you go through your decision tree. Once an infrastructure like this is set up, performing large-scale meta-analyses, or an analysis of an analysis, is the next logical, innovative step to better understand device safety, performance, and effectiveness.

Strategies like these can provide manufacturers with a platform for those discussions, which usually occur following gap analyses. It also will empower you to have fun now as you look forward to 2020 to celebrate the beginning of a new, EU MDR-focused year.

If you have success stories implementing strategies like these, I’d love to hear from you.

David R Rutledge, Pharm.D., FCCP, FAHA 
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350 cell

Although the bar was set high in an environment where scrutiny had never been more intense, Notified Bodies stepped up and succeeded with their new designation. The European Commission (EC) on Thursday added one additional Notified Body designated under the new Medical Devices Regulation (MDR). This brings the new total to five. Congratulations!

BSI-UK (0086) www.bsigroup.com 19JAN2019

TÜV SÜD (0123) www.tuev-sued.de/ps 27MAY2019

DEKRA (0124) www.dekra-certification.de 14AUG2019

IMQ ITALY (0051) www.imq.it 20AUG2019

TÜV Rheinland (0197) www.tuv.com/safety 26SEPT2019

For those manufacturers that have received or are about to receive notices that their Notified Body is withdrawing from MDD (or IVDD) and not pursuing MDR certification, huge challenges exist. Your hopes of relying on them to support the soft transition (2020-2024 under MDD or 2022-2024 under IVDD) have disappeared. Manufacturers are given 90 days to find a new Notified Body. How is that going?

The days of “like-to-like” transfers are gone; now it could take up to a year. A potential gap is likely to occur and there are consequences to your device certificates.

The MDR has prescriptive procedures for “Changes to designations and notifications” (see Article 46). Until then, manufacturers today should contact and petition the authority responsible for the Notified Body to identify the legal pathway forward. Procedures will vary by authority and should be considered emergency, temporary measures which will have some restrictions. In some cases, you will be requested to contact the authority where the authorized representative is located or where the manufacturer is located.

What’s been your biggest challenge? Although there are only 243 days left until 26MAY2020, there are still strategies that can be implemented to save your device.

David R Rutledge, Pharm.D., FCCP, FAHA, President & CEO, Global Strategic Solutions, LLC, david.rutledge@globalstrategicsolutions.com

However, some are concerned about the increased workload to initially create, update regularly, and then ensure consistencies within the quality management system. These reports must also be delivered in a cost-effective manner to the NB to review, validate and upload into Eudamed. Below are four initial tricks or techniques you may not be aware of to help you develop a SSCP. Please add your favorites too, if you are involved with implantable devices and class III devices that are not custom-made or investigational.

First, readability. For those SSCPs that also are intended for patients, Microsoft Word might help. Go to File, then Options, then Proofing, and check the box to show readability statistics. Now, after spell checking the document, readability statistics will be reported. Flesch Reading Ease is provided. This statistically calculates the readability of your content on a scale that ranges between 0-100. A lower score indicates that the content is more difficult to apprehend. If the score for your content is 100%, then your content is 100% readable, and it can be easily understood by everyone. Your goal here could be ~85%. Next, you will be provided the Flesch–Kincaid Grade Level. This helps you to know how many years of education one needs to understand your SSCP or, in other words, what reading level is needed. Your initial goal here could be grade level 6 or 7. Finally, there is a passive sentence percentage. Your initial goal here could be 20% to 25%.

You will need to strike a balance with these three readability statistics to create the perfect patient section of your SSCP. What about those technical, medical words that patients need to understand? Search for layman language suggestions for medical terminology. Manufactures could also review previous informed consent language, patient guides, or patient training materials to initially find the right words to use.

Second, the right people. Ensure the correct people are involved to get it right the first time. NBs will be looking to confirm consistencies among the submitted draft SSCP (and Instructions for Use) and the specific post-market surveillance (PMS) plan, post-market clinical follow-up (PMCF) plan, clinical evaluation plan, in addition to four reports, e.g., PMCF, clinical evaluation, risk management, and the periodic safety update report. If alignment exists, this could be evidence that the quality system might be functioning as intended. No one has time for re-working the SSCP because of communication challenges among important internal stakeholders.

Third, benefit-to-risk. Manufacturers have historically spent decades fine tuning risk assessments. The EU MDR now brings focus to benefit determinations, which should be based on evidence. Gone are the days of, “We believe.” This represents a huge challenge today. When listing potential benefits and risks, one of the first (of many) things that become evident is units are not the same. Thus, it is not a math problem; it is not possible to use simple math to objectively balance the potential benefits against potential risks. The timing, durations, and combinations of benefits and risks could also be different. Is this a challenge for you? Prior to receiving a guidance from MDCG or other groups, the US Food and Drug Administration recently published tools that I have used to facilitate initial discussions:

(i) Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian Device Exemptions (August 2019);

(ii) Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications (August 2019). There are two worksheets in this document that are useful to engage discussions about your device.

Fourth, changing regulatory environment. The SSCP is a required element within EU law. Specifically, listing relevant harmonized standards and common specifications. However, manufacturers are also developing global regulatory submission strategies. How are you doing keeping up to date with the changing regulatory environment not only in the EU, but worldwide? There are several ways to implement solutions, but there is one tool that can simplify this for you today. With changes occurring worldwide to medical device laws, standards, guidance documents, white papers, and EU MDR’s common specifications, delegating acts, and implementing acts, BSI’s Compliance Navigator is one tool designed to assist teams to manage regulatory information. This will not only ensure accuracy and provide a single source where information can be found within your organization, but it also provides “heads up” notices on what’s changing or coming up, which should help your planning. I found this tool to help teams work smarter, faster and maximize time to get products approved quicker and remain compliant. A nice additional feature with this tool is the assistance with interpreting standards with expert commentary, which could immediately help newly formed teams involved with creation of MDR-required documents like SSCPs.

Improving SSCP patient readability using Microsoft Word, getting the right people involved initially, understanding how to quantify benefit-to-risk, and keeping up to date with the changing regulatory environment in the EU (and elsewhere) are four solutions that can help both manufacturers and NBs. The results of which will provide health professionals and patients access to important information for device users and health professionals. A win-win-win.

By the way, for this document, Flesch Reading Ease was 37.9%, Flesch–Kincaid Grade Level was 11.9, and passive sentence percentage was 16.6%. I wonder if I should be using this for all those new procedures the MDR is requiring to either be created or updated? It would be a fun exercise knowing who needs to train to these.

Now, it is your turn. Please share your favorite techniques or tricks, if you are involved with SSCPs.

David R Rutledge, Pharm.D, FCCP, FAHA
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350
Based in the Silicon Valley area of California, USA