The meeting was ending at Mythical Medical.
“We believe this product is safe and performs as expected. We have received very few complaints, and the benefits outweigh risks when compared to
state-of-the-art products.”
Few questioned that authoritatively delivered closing message as several stood up applauding, whistling, and shouting. Mythical Medical employee Kai-Lin, on the other hand, scratched her head whispering under her breath, “I don’t get it. What data do we have to support this standing ovation?” Her comment almost went unheard.
In this fictional story, I’ll guide Kai-Lin through an effective process that can acquire this data quickly. I will be addressing relevant topics and providing simple, practical solutions to Kai-Lin that can be implemented globally.
Show Me The Data
In spite of the standing ovation, Kai-Lin knew that optimism is not data. Confidence is good, but the facts on your side are better. Many manufacturers, looking across their total product portfolio, recognize they don’t have sufficient current, clinical data to re-license them under newer regulations, e.g., MDR/IVDR. Time is running out.
Although firms meet milestones and most of their key performance indicators, a familiar “Show me the data,” voice is being heard more and more worldwide in hallways, conference rooms, and over lunch. Product certificates will be expiring in less than a year, and regulators, along with Notified Bodies, have made it clear that, although opinions and beliefs are lovely, decisions for certificate renewals must be based on facts.
After Kai-Lin contacted me, I learned that she was right. There were several products that had insufficient clinical data to obtain approval. For the first project, I recommended using surveys to quickly obtain safety and performance data that, in turn, could be used to also create benefit-to-risk analyses within the current state-of-the-art of their product.
Surveys
Surveys represent one option or solution to obtain clinical data, but they need to be developed correctly and compliantly. They can be used to either be the primary source of safety and performance data, e.g., for low-risk devices, or they can be used to fill in the gaps for higher risk devices where conducting a clinical investigation or registry is not practical. For all risk classes, surveys might be targeted to address specific questions from patient or user groups, healthcare professionals, regulatory bodies, or an independent, accredited body.
Surveys can be developed as either open-responses or closed-responses. I have dealt with both, and they each have their purposes and challenges. However, for this article, I’ll be recommending closed-response questionnaires. Then, you can choose a Likert scale format (pronounced “Lick-ert”), the more dominant method used. Developed in 1932 by Rensis Likert to measure attitudes, the typical Likert scale is a 5- or 7-point ordinal measure used by respondents to rate the degree to which they agree or disagree with a statement. I initially recommend using a 5-scale approach.
One advantage of closed-response questions is that it focuses on the respondent’s attention to address the critical issue at hand. Second, uniformed data can easily be entered into a database for analyses. Finally, they appeal to those hesitant to respond to items that might require a short narrative (or novel) to explain themselves.
Surveys need to be scientifically and clinically developed with a sound statistical sample size estimation and subsequent analysis plan. Especially in global studies, be sure to utilize local resources to ensure the questionnaire is culturally appropriate. Finally, the time to complete the survey is essential. Initially aim for 10 minutes.
Step 1 Create Research Questions For The Data
Start with an initial target goal of about 8-12 questions based, in part, on risk, quality, and clinical gap assessments. Conduct an internal technical documentation assessment addressing what evidence currently exists and how old it is; data which is already known about the product. Assess its quality and relevance. Perform a systematic literature evaluation to either add to what is known or confirm what is unknown. Then, meet with whom I refer to as “manufacturer-friendly” healthcare professionals that desire to be a part of a solution. Include a few questions to assess internal consistency.
This is going to take time, and you want to work with someone willing to help as they apply their unique, expert insights. They do not necessarily need to be using your product. Minimally, they just need to know and understand devices like yours. Finally, identify specific safety and performance issues.
Create statements that might be addressed with a targeted survey. Use a Likert 5-scale approach. Initially, select one question that could be used for sample size calculations. Conduct a pilot survey in the field to test, analyze, and make final changes. Assess internal consistency with your questions and validate the survey to ensure it is collecting the information you intend to collect.
Next, develop an extensive interviewer training program to minimize interviewer bias, knowing it may need to be tweaked once you launch this activity in the field. It is also good to remind personnel to report product performance issues that they learn about. Present this strategic plan to the regulatory agency or Notified Body as part of an overall surveillance discussion. Notified Bodies are not allowed to consult, so manage internal expectations. Begin discussions and gain alignment on which question will be selected to ensure an adequate sample size is selected.
In Mythical Medical’s case, a guidewire was identified as their initial target. The following issues were identified as gaps in their technical documentation: (1) delivery to the lesion, (2) ability to cross the lesion, (3) procedure success without complications at hospital discharge, (4) procedure success identified by another device being delivered, e.g., imaging, balloon, stent, and finally, and (5) comparative information on another guidewire. Number 2 above was selected as the question for the sample size determination using a 5-scale Likert frequency survey form, e.g., never, rarely, sometimes, often, always. I have a list of 37 Likert-type scale responses and the top 10 Likert-type 5-scale responses that I would be pleased to share. Ten questions were developed for the final survey.
Step 2 Calculate A Sample Size For The Data
If done well, survey results for the sample population should reflect the results you would have gotten from the entire group. Use a validated sample size software program that would be accepted by competent authorities, e.g., Australia, Argentina, Brazil, Canada, China, European Union, India, Israel, Japan, Mexico, Portugal, Kingdom of Saudi Arabia, South Korea, Taiwan, Thailand, United Kingdom, United States of America, etc. An initial suggestion could be nQuery (www.statsols.com) or Power and Precision (https://www.power-analysis.com/index.php). I use both, but there are many others. The study will compare two groups using an identical scale to get information on two commonly used guidewires. The null hypothesis is that the effect size or response from both groups will be equal. We intend to disprove the null hypothesis.
This is the technical piece, but I will keep it high level for this simulation. In this case, the Wilcoxon/Mann-Whitney rank-sum test (ordered categories) for equal “n’s” was used. The alpha level was set at 0.05 (criteria used for statistical significance), a two-sided test was chosen, the number of categories was 5 (see above). Estimates were obtained from each of the 5 Likert responses for the question being used for sample size calculations based on a similar survey Mythical Medical found in the literature.
The initial results suggest sample sizes per group of n=75, n=100, n=125, and n=150 and estimated powers of 68%, 80%, 88%, and 93%, respectively. This is where the fun starts. The manufacturer needs to align internally in terms of risks they are willing to take. On the one hand, you want enough observations to obtain reasonably precise estimates of the parameter of interest, but also want to do this within a reasonable resource budget. The important thing is to quantify risks associated with the final sample size that is chosen. There is also the consideration of missing data, which in this simulation was estimated to be 3%.
In this case, Mythical Medical selected the third option using n=125 in each group with an estimated power of 88% to allow us to conclude that the response differs between the two groups. They did not want to consider the 3% assumption for missing data. There are other issues to discuss, but let’s keep it simple. From a budget perspective, we will have ten questions being answered by 250 participants that will give us 2,500 completed questions. Thus, assume you need $20,000 (direct survey costs), convert to your local currency. Next, adjust budgets considering your unique business model. Factors that might affect budgets could include (1) whether it will be in-sourced or outsourced, (2) translations, (3) travel expenses, (4) healthcare practitioner fees, and (5) fees associated with compliance with good clinical practice, privacy laws, safety reporting, etc. The total budget for this project if implementing all 5 factors above, could creep up to $100,000, though (direct and indirect costs).
Step 3 Collect The Data
Initially, Mythical Medical wanted to set this up online using online survey programs and then send out an email to those they wanted to complete the survey. That is undoubtedly a standard method used to conduct surveys. The challenge here, as discussed with Kai-Lin, is the company had limited experience conducting surveys for regulatory submission purposes. Those in the survey area are all too familiar with low return rates, potentially non-representative samples, inability to clarify the question, and too much time spent on demographic issues.
In this case, I recommend selling the value of conducting face-to-face interviews, especially since the company has little experience conducting hypothesis testing surveys. One benefit is it is a tremendous customer-facing activity. There are other secondary benefits when meeting one-on-one with your customers. Mythical Medical will also need to have the right personnel in the field meeting with customers. These are typically not from sales or marketing, per se, but from those having clinical expertise such as seen within clinical research, medical affairs, quality, regulatory, engineering, biostatistics, or safety. As more experience is obtained, recommendations could vary.
Step 4 Analyze The Data
The Mann-Whitney U test (a non-parametric test) was applied to evaluate data from each question. Although it is acknowledged in the literature, there has been a longstanding, fascinating debate regarding analyzing Likert-scale data; it is beyond the scope of this article. The discussion between the ordinalists (saying that you should use non-parametric tests) and the intervalists (arguing for parametric tests) is ongoing. Without going too far into the simulated results, within eight months, this approach yielded robust data from 10 questions per survey applied to two treatment groups. These results now are beneficial because they can be used, in part, to develop future surveys and serve as the basis to test new hypotheses in the future.
There are several software programs to assist you in running the analysis, e.g., Prism (which I use), R statistic, SAS, and SPSS. You can also purchase programs like Q1Macros (which I have also used) that embed statistical packages into EXCEL, which I have found really convenient. This is especially helpful to quickly run Mann-Whitney U tests once you have entered the data in an EXCEL spreadsheet.
Step 5 Data Becomes Inputs Into Key Processes
These results should become inputs back into your quality system and technical documentation processes. The results have direct relevance to design input. Any product performance issues that were uncovered should be processed in a timely manner too.
Likert 5-scale response sets are robust and can be applied to appraise frequency, quality, intensity, agreement, approval, awareness, importance, familiarity, satisfaction, and performance. The Likert scale is Rensis Likert’s best-contribution. He created the scale in 1932 as a part of his Ph.D. thesis and its usage has universal application.
Depending on the expertise involved, open-ended questions to assess product safety such as, “Were there any adverse events not identified in the Instructions for Use?” could be added in future surveys.
Implications To EU MDR/IVDR
Within the EU MDR/IVDR regulations, the following sixteen documents should be updated as appropriate, e.g., Strategy for Regulatory Compliance (Article 10 & Annex IX), Risk Management Plan and Report (Article 10 & Annex I), Clinical Development Strategy (Article 61), Clinical Evaluation Plan and Report (Article 61 & Annex XIV), Clinical Development Plan (Annex XIV), Clinical Investigation Plan and Report (Article 62, Article 63, & Annex XV), Post-Market Surveillance Plan (Article 84 & Annex III), Post-Market Surveillance Report for Class I devices (Article 85 & Annex III), Periodic Safety Update Report or PSUR (Article 86 & Annex III), Post-Market Clinical Follow-up Plan and Report (Article 61 & Annex XIV), Summary of Safety and Clinical Performance or SSCP (Article 32), and Claims (Article 7).
Closing Thoughts
When initially developing effective surveys, there are two steps that take the longest; Step 1 (creating the questions) and Step 3 (obtaining the data). Kai-Lin asked the right question early enough and chose a method to obtain the data to support that closing remark which led to a standing ovation. She actively participated in finding a solution and learned valuable skills along the way. The product was approved.
A colleague overheard that “under the breath comment.” Knowing the value survey results brought to the company and witnessing both how well she led the project and worked well with others, her colleague ensured she was recognized.
Of course, this was with Mythical Medical, but these principles also apply at your “Real Medical” too and you can implement them when you go back to work tomorrow. You know where your company has been and where it is going. Be that person who inspires greatness when clinical data is needed. Kai-Lin trusted her instinct and was part of both developing and implementing a simple survey strategy. You can, too!
David R Rutledge, Pharm.D., FCCP, FAHA, President & CEO, Global Strategic Solutions, LLC, Silicon Valley in California. david.rutledge@globalstrategicsolutions.com. +1 (630) 846-0350 cell.