Benefit-to-Risk Analyses
Some things make sense. They are easy, balanced, symmetrical, and predictable. That’s why Jae, an employee at Mythical Medical, liked mathematics, engineering, chemical reactions, and physics. Rules or laws are in place, and, for the most part, you follow them. Equations made sense, and he loved them.
He also joined a debate club because structuring logical arguments seemed to come naturally, too. That skill would come in handy at work. Regulations and laws, he would discover, were his friends too. They were so “black and white,” he thought.
Regulatory Affairs seemed to be the perfect fit to begin his professional career in the industry. Little did he know that a simple request would soon introduce him to a new world of “grey” within Regulatory Affairs.
In this fictional story, I’ll guide Jae through an effective process that provides a framework to organize thinking to craft a benefit-to-risk assessment. I will be addressing relevant topics and providing simple, practical solutions. These principles can be implemented globally.
The Simple Request
Kai-Lin, also an employee at Mythical Medical, asked Jae for help in creating a benefit-to-risk analysis (evaluation or assessment) for the company’s next product submission.
Previously (see previous articles) we learned Kai-Lin was all about making decisions based on evidence. Her reputation precedes her now because of her early work using surveys to create safety and performance data. Regulators and Notified Bodies all over the world respect her.
Her simple request today would soon change Jae’s Regulatory Affairs’ world forever. Things are not so simple as thinking in extremes like “black and white.” This perspective is dichotomous or polarized thinking.
Potential Benefit Versus Potential Risk
Jae initially set up two columns to guide his work on creating a benefit-to-risk analysis. This approach or method had worked so many times in the past. After a few hours, working alone, he had a wonderful, but small, limited number of items in the benefit column, but a massive, extensive list under the risk column. The two columns were not going to be easy to use this time.
After Jae contacted me, I learned that he was right. The exercise was not going to be like solving a math equation. I asked him to record the units of measurement out for every item listed under both columns. That’s when Jae realized that the units of measurement made the benefit-to-risk equation unsolvable. This really was like comparing apples to oranges. It is not possible today to use math to balance the benefits against risks objectively. There were other reasons for his frustrations, too, such as the frequency of events, severity, probability of having the event, the potential of having multiple events, and their duration.
Bring A Team Together
Jae did a fine job with his initial list. However, I advised him that to perform this analysis, others need to be involved. He should include those with clinical expertise such as seen within clinical research (clinical science), clinical evaluation, medical affairs, quality, risk management including product/clinical/software/cybersecurity representation, regulatory affairs, engineering, biostatistics, complaint handling, and safety.
This team should use all available clinical data, including results from clinical investigations or registries, hospital databases, usability, servicing records, CAPAs, customer/patient feedback, literature, etc.
Discuss Potential Benefits: Address these 12 issues
ISO14971:2019 defines benefit as having a positive impact or desired outcome of the use of a medical device on the health of an individual, or a positive impact on patient management or public health. Benefits can include a positive impact on clinical outcomes, the patient’s quality of life, outcomes related to diagnosis, positive impact from diagnostic devices on clinical outcomes, or positive impact on public health.
Identify Potential Medical Device Benefits
To begin the discussion of the potential benefits with Jae’s team, I suggested using a framework that includes addressing these 12 issues if they apply: (1) patient perspective, such as quality of life from a validated tool, (2) healthcare professionals and caregivers perspectives, (3) medical necessity, e.g., What are the other choices to patients/health care practitioners that would make this product a benefit?, (4) types of benefits (remember that claims should automatically be considered types of benefits), (5) magnitude of benefits in terms of time or scale so a metric can be used here, (6) likelihood or probability of experiencing one or more benefits, (7) reduction in the probability of death, (8) aiding improvement in patient function such as an organ system, (9) reducing the probability of loss of a function, (10) relief of symptoms, (11) duration of benefits in terms of curative, short-term, or long-term, and (12) benefits when compared to alternative therapies or state-of-the-art. This framework to organize thinking to craft a benefit-to-risk assessment allows Jae’s team to assemble evidence from one side of the equation, potential benefits that need to be considered in this analysis.
Now, with this wealth of information, Jae’s team should use these seven, broad benefit assessment criteria and then create narratives for each: (1) Types of benefits, (2) Magnitude of benefits, (3) Likelihood of experiencing one or more benefits, (4) Duration of effects, (5) Patients’ perspective on benefit, (6) Benefit factors for health care professionals or caregivers, and (7) Medical necessity compared to what is currently available, for example. If assumptions are being made, make them clear to the team.
Thus, benefits can range from how the patient feels or functions to how they survive. In some cases, there could be an acceptable surrogate outcome parameter that improves. Additionally, benefits could be related to the convenience of managing or diagnosing a disease or condition. Although manufacturers, in general, do not have a lot of experience documenting benefits, using this framework, becomes a fun activity.
Finally, uncertainties in benefits need to be addressed and might be related to sensitivity, specificity, accuracy, precision, reproducibility, wide confidence intervals, underpowered studies, missing data, or the impact of confounding interventions or physiological or pathophysiological factors.
Discuss Potential Risks: Address these 12 issues
According to ISO14971:2019, it is accepted that the concept of risk has two key components: (1) the probability of occurrence of harm and (2) the consequences of that harm, such as how severe it might be.
To begin the discussion of the potential risks with Jae’s team, I suggested using a framework that includes addressing these 12 issues, if they apply: (1) It is the patient who takes the risk for the promise of the potential benefit. How well does the patient understand the risk from the product?, (2) patient tolerance (or intolerance) to the product over time, (3) risks to healthcare professionals and caregivers, (4) severity of risks, (5) type of risks, such as product, clinical, software, and cybersecurity, (6) likelihood of experiencing one or more risks, (7) duration of exposure, (8) mitigation potential such as quantitative and qualitative risk control methods, (9) procedure-related versus device-related risks, (10) disease characteristics that could affect risks, (11) quantitative or qualitative residual risk estimates, and (12) risks from false-positive or false-negative results. The discussion should also include how risks might be mitigated or what measures can be in place to lower risks. The key here is to have the team agree on identifying residual risks. This framework to organize thinking to craft a benefit-to-risk assessment allows Jae’s team to assemble evidence from the other side of the equation, potential risks that will be considered in this analysis.
Now, with this information, use these six, broad risk assessment criteria and, once again, create narratives for each: (1) Types of harms or risks, (2) Magnitude or severity of harms or risks, (3) Likelihood of experiencing one or more harms or risks, (4) Duration of exposure to the population, (5) Patients’ perspective or tolerance (or intolerance) to harms or risks, and (6) False-positive or false-negative results. Again, if assumptions are being made, make them clear to the team.
Finally, uncertainties in risks need to be addressed and might be related to the product, procedure, subsequent treatments or tests, insufficient number of patients, differences in definitions, percentage of subjects that were lost-to-follow-up during a clinical investigation, protocol deviations, and user experience such as being inconsistent or not representative of likely real-world users.
Tell the Story
Is the overall residual risk acceptable in relation to potential benefits? Use the information gathered to identify potential benefits and potential risks and tell the story. When you put your mind to it, there are a lot of tools or potential sources of evidence in the benefit-to-risk toolbox that should be used depending on the risk class of the product. The crucial next step is to pull this all together to create a factual and evidence-based story that reveals how the potential benefits of the product outweigh potential risks. Knowing the uncertainties mentioned above could provide the manufacturer with information to develop and conduct prospective, post-market clinical (or performance) follow-up studies.
Regulatory bodies, as well as Notified Bodies, expect all relevant models, sizes, and device variants, to be covered in the benefit-to-risk analysis.
If the evidence does not support the conclusion that potential benefits outweigh residual risks, then the manufacturer should consider modifying the medical device, its intended use, or the target population. Perhaps other controls need to be in place.
Benefit-to-risk assessments are living analyses. They begin with justifying first-in-human studies. Next, they are revised to obtain approval to conduct pivotal investigations. Finally, assessments are continuously updated post-approval throughout the life cycle or lifetime of the device.
Medical therapies are not static and state-of-the-art changes. Benefit and risk perceptions change over time. Thus, benefit-to-risk analyses should be viewed as living systems that are to be revised regularly when new data or devices become available.
There are recent guidance documents and an ISO standard that you will find helpful as you apply this framework to organize thinking to craft benefit-to-risk assessments for your product:
(1) ISO14971:2019 Medical devices – Application of risk management to medical devices. https://www.iso.org/obp/ui#iso:std:iso:14971:ed-3:v1:en https://www.iso.org/standard/72704.html
(2) FDA Guidance “Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications” issued August 30, 2019. It contains worksheets and examples. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/factors-consider-when-making-benefit-risk-determinations-medical-device-premarket-approval-and-de
(3) FDA Guidance “Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian Device Exceptions” issued August 30, 2019. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/consideration-uncertainty-making-benefit-risk-determinations-medical-device-premarket-approvals-de
A textbook I would recommend is Safety Risk Management for Medical Devices by Bijan Elahi ISBN 978-0-12-813098-8 by Elsevier Publishing under the label of Academic Press 2018. https://www.elsevier.com/books/safety-risk-management-for-medical-devices/elahi/978-0-12-813098-8
Podcast on ways to collect Sufficient Clinical Evidence: https://www.youtube.com/watch?v=anQRz2weaqo
Podcast on the Benefit to Risk Analysis: https://www.youtube.com/watch?v=lwkTgNVnt9A
Closing Thoughts
Jae did create a benefit-to-risk assessment for Mythical Medical’s next product submission. He pulled a broad team together and led meetings using a logical, transparent framework. The results were presented for internal sign-off and were based on evidence that Kai-Lin required and not only on the belief that the product’s benefits outweighed its risks. Jae learned that benefit-to-risk-thinking is not always easy, balanced, symmetrical, or predictable.
Until manufacturers can mathematize benefit-to-risk analyses, the framework presented here can organize your thinking and allows immediate actions that your team can take and use now. It does require the use of “grey matter” (or your brain), but the outcomes are rewarding. It is flexible to use for all product risk classes.
David R Rutledge, Pharm.D., FCCP, FAHA, President & CEO, Global Strategic Solutions, LLC, Silicon Valley in California. +1 (630) 846-0350 cell.
Disclaimer
This is just a bit of fun. The purpose of articles like these is for the free exchange of questions, ideas, or comments regarding issues of global interest involving medical regulations and is not to be considered legal advice. These are just opinions and not advice; opinions may change over time, too. Use it at your own risk. This disclaimer will occasionally be posted or updated throughout the year.