Notified Bodies, manufacturers, and regulators have leaders. Communication among these entities can be tough and complicated. Great leaders within these organizations sometimes only desire three things: (i) predictability, (ii) approvals on time, and (iii) quality submissions.

These represent shared common aspirations that provide a platform for mutual collaboration. Yet, there must be a way to structure a program to facilitate communications using metrics. At its heart will be the need for agreed-upon data. Product submission metrics did not exist at Mythical Medical. People should be brought together to take a deep dive into these processes and then agree on the future, immediate next steps.

Ideas Are Needed

Ideas are the natural-born enemy of the way things are. Kai-Lin, an employee at Mythical Medical, is unaware she is on a journey to greatness (see previous articles). What she is keenly aware of is that serious communication challenges exist between Mythical Medical and regulators as well as their Notified Body. She and her team are in unfamiliar territory.

Improvement in working relationships with their Notified Body will take priority now due, in part, to the increased number of planned product submissions.

In this fictional story, I’ll guide Kai-Lin through a process to generate ideas that can be used to create and implement a communication strategy. I will be addressing relevant topics and providing simple, practical solutions. These principles can be implemented globally with regulatory agencies, too.

 A Benchmark Is Needed

Previously we learned Kai-Lin was all about making decisions based on evidence. She demonstrated that by using surveys to acquire specific sufficient clinical evidence (safety and performance data, in this case, to fill in a clinical evidence gap) and later overseeing the creation of a framework to institute a benefit-to-risk process. Her reputation continues to grow with regulators and Notified Bodies all over the world. She and her team are respected. However now, she is lacking data again.

During our first meeting, I asked Kai-Lin, “How are discussions going between Mythical Medical and your Notified Body?” She replied, “Some describe it as challenging.” We discussed why that was, and it became clear that pieces of information, data that could be available to assist with communication were not known. Without data, there was no benchmark to measure it against. Without a benchmark, key performance indicators (KPIs) could not be developed. Thus, the first initiative or proposal would be to gather data using metrics, data that had to be agreed upon by her team, and their Notified Body. Kai-Lin, like many others, needed to discover those little-known knowns.

Agreeing on the Data to Collect

Most of the world make decisions by either guessing or using their gut. Having data to establish an initial benchmark is not only a useful tool for communication, but it is also a competitive advantage. The data collected initially will change or shift over time. Some call this data swirl.

Carve out time to bring important stakeholders together for a robust discussion. Agree on what the current pain-points are. Use this information, in part, to agree on the final data to collect. A key here is to filter out ideas of what could be measured to what should be measured. Although this will take some time and a little more effort, the desire to gain efficient product approvals will numb the pain it takes to get you there.

1. Little-Known Knowns: Predictability

Mythical Medical should work with their Notified Body on agreed-upon metrics related to predictability. Here are four metrics to better understand factors that affect predictability: (i) Number of pre-notifications provided versus the total numbers of submissions (percentage), (ii) Percentage of pre-notifications provided shorter than eight weeks before submission, (iii) Number of project cancellations versus total projects (percentage), and (iv) Number of postponed projects versus total projects (percentage). The first two metrics imply that early communication is essential. It can make or break a medical device’s successful journey to the market.

Depending on the software or analytical capabilities of the Notified Body, these four metrics could be modified, or others could be added. The point is having this type of information can be useful both to the Notified Body and the manufacturer. It makes for easy graphing, and it can be tracked over time.

2. Little-Known Knowns: Approval Timelines

In business, approvals coming ahead of or later than schedule cause both upstream and downstream manufacturing challenges for people, processes, production, and eventual implementation of launch strategies. Thus, the best-case scenario is to plan thoroughly and have your product approvals hit the targeted date.

These three metrics should be tracked: (i) Number of delayed projects (not postponed projects), (ii) Within project milestone dates, e.g., submission date, 1st Notified Body review round, 1st responses by the manufacturer, 2nd Notified Body review round, 2nd responses by the manufacturer, number of additional review rounds, certificate date related to review closing data, and (iii) Overall project timelines or planned versus actual achieved date.

Data within these little-known knowns can be used to understand better and then characterize specific issues related to submission processes at both organizations. It is okay that you might find embarrassing or frustrating elements. Those are better known than unknown.

Project managers will find information-rich data here. Evidence from these metrics should be segmented to analyze details within individual projects. Find trends in what works and what doesn’t work. Action items can be developed that will have positive effects on all future submissions.

3. Little-Known Knowns: Quality

If you are struggling with quality metrics during the submission and approval process with your Notified Body, here are five simple metrics to start with: (i) Number of questions per pre-notification, (ii) Number of screening deficiencies raised during administrative quality checks, (iii) Number (average/median) of review rounds per project, (iv) Number (average/median) of deficiencies per project or review module, (v) Number of post-approval activities, follow-up items, and certificate limitations.

You get what you measure. What each organization finds will eventually change who they will become in the future.

These metrics will create an initial understanding of the quality of your submissions. Otherwise, how would you know where challenges exist and who should be accountable?

A nice outcome here is that you can update your submission checklist and train employees to minimize these delays. You can look at these metrics per project team or product class.

Closing Thoughts

When communicating with Notified Bodies, manufacturers, and regulators, the single biggest conundrum is the illusion that communication has taken place. If you want to examine this conundrum, you must have metrics like these. Without them, your competitors will eat your mistakes. Therefore, it’s time to discover those little-known knowns.

Become a great leader within your organization and implement metrics that address: (i) predictability, (ii) approvals on time, and (iii) quality submissions. If you and your Notified Body are struggling to create these metrics, work together to agree to begin somewhere. Metrics will evolve or even swirl over time. That is the natural evolution as data matures or is analyzed. With this evidence, benchmarks can be created. Then, KPIs can be developed to report on progress against the desired result.

This approach seems simple on the surface. However, to manage Kai-Lin’s expectations, I jokingly informed her, “Half the people will love the data, half will hate the data, and half won’t understand it.” However, in time, jewels of truth and pearls of understanding will emerge. Someone has said, “Listen to the experienced people in your life–not because they’re always right, but because they have a better understanding of being wrong.”

To pursue excellence, it takes relentless effort. If you want to go beyond what’s expected, you must evolve and grow too. Little did she know (and those like her), that implementing these metrics would take her down a path to become a fantastic leader. She is on a journey to greatness.

Metrics like these are relatively new for most organizations, but with changes in product approval and post-approval regulations around the world, deadlines are rapidly approaching. It’s time to collect this data and have discussions about future submissions. Some will say it’s too early until, unfortunately, it’s suddenly too late.

The author would like to thank Bassil Akra of TÜV SÜD.

David R Rutledge, Pharm.D., FCCP, FAHA, President & CEO, Global Strategic Solutions, LLC, Silicon Valley in California. david.rutledge@globalstrategicsolutions.com +1 (630) 846-0350 cell. The homepage is www.globalstrategicsolutions.com. 

Previously I read his first book, Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. (2009), Introduction to Meta-Analysis. West Sussex, United Kingdom, John Wiley & Sons, ISBN: 978-0-470-05724-7. Yet, I absolutely fell in love with his newest book, Borenstein M. (2019), Common Mistakes in Meta-Analysis and How to Avoid Them. Englewood, New Jersey, USA, Biostat Inc, ISBN: 978-1-7334367-1-7 (softcover). Hence, I want to review it for you.

Dr. Borenstein is acknowledged as a distinguished expert and is exceptionally positioned to author this landmark textbook. I have read the book and found it to be a critical addition to my library. It is helpful in expanding and developing my knowledge of meta-analysis, and especially in avoiding common mistakes when conducting systematic reviews of interventions. This is in its first edition (ISBN: 978-1-7334367-1-7 the softcover version) and available on Amazon.

The author has chosen to identify and explain 42 mistakes made when conducting these important analyses. He uses excellent, easily understood examples for you to apply and learn the principles of meta-analysis. The author explains why each is a mistake, the implications of the mistake, and how you can avoid the mistake.

For Those New To Meta-Analyses: You will never forget the history behind the term “forest plots” and will learn why you should refer to them as “forest plots” and not “Forest plots” even though you will see that term referenced in the literature.

If you are new to this area of statistical analysis, there is no better book for you to read at this early stage in your career. Don’t be afraid to be a beginner. It provides a historical perspective to help you understand where the industry was and where it is headed in terms of the evolution of meta-analysis when applied to social, behavioral, educational, medical, clinical, regulatory, or legal applications.

If you need to know the basics and wonder “Why do meta-analyses and what are the different models that can be used?” this book, starting with the preface and continuing through chapters 6 and 7, is for you. Before you get to page 42 you will learn what many don’t seem to know or remember; how to decide on the model to use upfront, prior to your analysis. Dr. Borenstein will give you a solid explanation to support that fundamental principle. Refer to chapter 4 to obtain websites for this book, the software program Comprehensive Meta-Analysis, upcoming workshops, and then register your email to receive updates to remain current and competent. Finally, review the glossary in chapter 15. Now you will be ready to actively participate in your first meta-analysis team meeting using your solid, fundamentally sound foundation.

For Those Seasoned Professionals: If you are a seasoned professional, grounded in the basics already, you will enjoy the following chapters with discussions regarding (1) issues and myths about statistical models, (2) heterogeneity, (3) mistakes related to significance testing, (4) publication bias, (5) mistakes in subgroup analyses, (6) features of the Comprehensive Meta-Analysis software program, and (7) how to correctly report the results of an analysis. Relevant examples are presented. Study them. For a more in-depth review, the author includes ten appendices that contain supplementary material which may be helpful in providing a more comprehensive understanding of your research question or problem.

Reflecting On My Experience Reading This Book: I personally found value in (1) knowing what the 42 mistakes were, (2) the easily understood Summary Sections the author provides, and (3) the “Putting It All Together” sections. He clarifies the issues that were previously ambiguous. The section on heterogeneity and prediction interval, an index of dispersion, was of enormous value and very practical to me. I learned what the “I squared” (I2) statistic correctly is and is not; it really doesn’t tell you how effects vary, despite the widespread misconception in the literature. It tells us what proportion of the variance in observed effects reflects variation in true effects, rather than sampling error. It does not tell us how much variation there is. Rely on the prediction interval instead. Finally, templates were provided to assist with reporting results that communicate clearly to my intended audience.

When the treatment effect (or effect size) is consistent from one study or observation to the next, meta-analysis can be used to identify this common effect. When the effect varies from one study or observation to the next, a meta-analysis may be used to identify the reason for the variation. For example, when you run the analysis, why does one population seem to not benefit and actually appear to be harmed, while another population has a small treatment effect, a third tends to have a modest effect, and finally some appear to have the largest treatment effect? Now it’s our opportunity to put those detective shoes on and set up strategies to identify, address or evaluate those differences in effects. Regulatory bodies, clinicians, and Notified Bodies will want to confirm that your product, for example, is working as intended in the approved population. Performing a meta-analysis is one way to assess product safety and performance.

In this life, few are given the responsibility or opportunity to change the world (or at least a piece of it where you live) by analyzing significant data correctly. Don’t model your comprehensive reviews and analyses on what others have published in the literature. Base it on the principles learned in this book. It is a 5-star, brilliantly written, expertly communicated reference that has received my highest recommendation. Avoid making the 43rd mistake by not having this as part of your library.

If you believe that decisions affecting people’s lives in areas of clinical, public health or policy settings should be informed by relevant research evidence using meta-analyses, then this review is dedicated to you. With the proper education, training and other easy to use tools that exist today, you too can be on the verge of changing the world where you live. How about your experiences? I’d love to hear more heartwarming stories from the analyses you have performed.

Key Websites For Your Continued Learning

https://meta-analysis-books.com/, www.cochrane.org/, https://www.meta-analysis-workshops.com/, and training.cochrane.org/.

David R Rutledge, Pharm.D. FCCP, FAHA
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350 cell
Based in Silicon Valley, California, USA

However, some are concerned about the increased workload to initially create, update regularly, and then ensure consistencies within the quality management system. These reports must also be delivered in a cost-effective manner to the NB to review, validate and upload into Eudamed. Below are four initial tricks or techniques you may not be aware of to help you develop a SSCP. Please add your favorites too, if you are involved with implantable devices and class III devices that are not custom-made or investigational.

First, readability. For those SSCPs that also are intended for patients, Microsoft Word might help. Go to File, then Options, then Proofing, and check the box to show readability statistics. Now, after spell checking the document, readability statistics will be reported. Flesch Reading Ease is provided. This statistically calculates the readability of your content on a scale that ranges between 0-100. A lower score indicates that the content is more difficult to apprehend. If the score for your content is 100%, then your content is 100% readable, and it can be easily understood by everyone. Your goal here could be ~85%. Next, you will be provided the Flesch–Kincaid Grade Level. This helps you to know how many years of education one needs to understand your SSCP or, in other words, what reading level is needed. Your initial goal here could be grade level 6 or 7. Finally, there is a passive sentence percentage. Your initial goal here could be 20% to 25%.

You will need to strike a balance with these three readability statistics to create the perfect patient section of your SSCP. What about those technical, medical words that patients need to understand? Search for layman language suggestions for medical terminology. Manufactures could also review previous informed consent language, patient guides, or patient training materials to initially find the right words to use.

Second, the right people. Ensure the correct people are involved to get it right the first time. NBs will be looking to confirm consistencies among the submitted draft SSCP (and Instructions for Use) and the specific post-market surveillance (PMS) plan, post-market clinical follow-up (PMCF) plan, clinical evaluation plan, in addition to four reports, e.g., PMCF, clinical evaluation, risk management, and the periodic safety update report. If alignment exists, this could be evidence that the quality system might be functioning as intended. No one has time for re-working the SSCP because of communication challenges among important internal stakeholders.

Third, benefit-to-risk. Manufacturers have historically spent decades fine tuning risk assessments. The EU MDR now brings focus to benefit determinations, which should be based on evidence. Gone are the days of, “We believe.” This represents a huge challenge today. When listing potential benefits and risks, one of the first (of many) things that become evident is units are not the same. Thus, it is not a math problem; it is not possible to use simple math to objectively balance the potential benefits against potential risks. The timing, durations, and combinations of benefits and risks could also be different. Is this a challenge for you? Prior to receiving a guidance from MDCG or other groups, the US Food and Drug Administration recently published tools that I have used to facilitate initial discussions:

(i) Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian Device Exemptions (August 2019);

(ii) Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications (August 2019). There are two worksheets in this document that are useful to engage discussions about your device.

Fourth, changing regulatory environment. The SSCP is a required element within EU law. Specifically, listing relevant harmonized standards and common specifications. However, manufacturers are also developing global regulatory submission strategies. How are you doing keeping up to date with the changing regulatory environment not only in the EU, but worldwide? There are several ways to implement solutions, but there is one tool that can simplify this for you today. With changes occurring worldwide to medical device laws, standards, guidance documents, white papers, and EU MDR’s common specifications, delegating acts, and implementing acts, BSI’s Compliance Navigator is one tool designed to assist teams to manage regulatory information. This will not only ensure accuracy and provide a single source where information can be found within your organization, but it also provides “heads up” notices on what’s changing or coming up, which should help your planning. I found this tool to help teams work smarter, faster and maximize time to get products approved quicker and remain compliant. A nice additional feature with this tool is the assistance with interpreting standards with expert commentary, which could immediately help newly formed teams involved with creation of MDR-required documents like SSCPs.

Improving SSCP patient readability using Microsoft Word, getting the right people involved initially, understanding how to quantify benefit-to-risk, and keeping up to date with the changing regulatory environment in the EU (and elsewhere) are four solutions that can help both manufacturers and NBs. The results of which will provide health professionals and patients access to important information for device users and health professionals. A win-win-win.

By the way, for this document, Flesch Reading Ease was 37.9%, Flesch–Kincaid Grade Level was 11.9, and passive sentence percentage was 16.6%. I wonder if I should be using this for all those new procedures the MDR is requiring to either be created or updated? It would be a fun exercise knowing who needs to train to these.

Now, it is your turn. Please share your favorite techniques or tricks, if you are involved with SSCPs.

David R Rutledge, Pharm.D, FCCP, FAHA
david.rutledge@globalstrategicsolutions.com
+1 (630) 846-0350
Based in the Silicon Valley area of California, USA