PMA Advisory Panels: Do Their Votes Matter?
by Jeffrey N. Gibbs, J.D. and David A. Gibbs
Under the Federal Food, Drug, and Cosmetic Act, devices fall into three classes. The premarket approval (PMA) is the route to market for Class III devices, the highest risk medical devices. A Class III device is a device that cannot be classified as a Class I or II device because “insufficient information exists to determine that the application of” general and/or special controls would “provide a reasonable assurance of safety or effectiveness” and “is purported or represented to be for a use in supporting or sustaining human life or for a use which is of substantial importance in preventing impairment of human health” or “presents a potential unreasonable risk of illness or injury.”
PMAs represent a relatively small proportion of devices reviewed by the Food and Drug Administration (FDA). In 2017, for example, FDA approved only 46 PMAs while clearing 3,217 510(k) premarket notifications and authorizing 31 de novo applications.
Yet, even though PMAs constitute a minority of the new devices reviewed by FDA, they include some of the most commercially important and successful devices. Recently approved PMAs include a novel system for preserving lungs for transplantation, a screening test for colorectal cancer, an artificial iris, a test for pregnant women designed to assess the risk of spontaneous preterm delivery, and an implantable cardiac monitor to alert patients of a potential heart attack.
Because of their greater riskiness, PMA devices undergo a more rigorous review process. The PMA standard is the most demanding of the three primary device regulatory pathways: “reasonable assurance of safety and effectiveness.” To clear this hurdle, the applicant must meet the applicable regulatory standards, which include clinical data, bench testing, adequate labeling, and demonstrated adherence to the Quality System Regulation. Satisfying all of these elements is not a trivial task.
Yet for some PMA applicants, there is one more obstacle: the advisory panel. During this meeting, FDA solicits feedback from independent advisors. The process is public, and carefully scrutinized by competitors, investors, health care providers, and other stakeholders.
Although panel meetings are high visibility events, they are relatively rare. Initially, under the Medical Device Amendments of 1976, panel meetings were required for new Class III devices. Then, in 1990, Congress amended the law to provide that the referral of an application to a panel would only occur “on the Secretary of the Department of Health and Human Service’s own initiative” or “upon the request of an applicant unless the Secretary finds that the information in the application . . . substantially duplicates information which has previously been reviewed by a panel.” FDA recently issued a guidance document reaffirming that panels were not needed if the device presented issues that had already been addressed by a panel. Thus, many companies now include as part of their PMA a rationale why no panel is needed.
This is not the only major change in the panel process since it began. Initially, the panel meeting resulted in a single vote: a recommendation for or against approval. In 2010, FDA revised the process so that panel members were asked to vote on three separate topics: the safety, effectiveness, and benefit-to-risk ratio of the device. Simultaneously, FDA changed the voting process. Instead of going around the table and asking each panel member in turn to cast their vote publicly, FDA adopted simultaneous blinded voting. We will refer to this as the “new” system, as compared to sequential public voting, which we will refer to as the “old” system.
This change was described as a simple, procedural modification. The impact, though, appears to have resulted in—or is at least associated with—changes in panel behavior. For example, since blinded voting was put in place, there is—perhaps unexpectedly—greater consensus among panelists in voting than under the prior, sequential voting system.
Nor has this policy on voting been the only shift affecting the processing of PMAs. For example, as the result of User Fee negotiations, FDA made certain commitments regarding the PMA process, such as timeliness of reviews. FDA has also made other changes to the administrative processes for PMAs. However, none of these changes have altered the statutory standard for approving PMAs.
Given the perceived importance of panel meetings for the fate of PMAs, sponsors and FDA pour resources into their preparation. Unlike other device reviews where problems, blemishes, warts, and FDA concerns are hidden from view, this process is open for all to see. The process is very transparent, with the public not only able to watch the meeting itself but also to read the sponsor’s and FDA’s panel packages in advance. Applicants will spend months honing their panel presentation, practicing what they will say multiple times, and subjecting presenters to mock questions. Some companies specialize in helping device companies get ready for these meetings. Other companies hold seminars on the panel process and panel preparation.
This investment of time, effort, money, and anxiety stems in large part from the perception that a panel vote is a high stakes drama, a make-or-break event for the application. As a matter of law, an advisory panel is just that: advisory. FDA is free to reject or follow a panel’s recommendation. Nevertheless, as innumerable articles have stated, “FDA is not required to follow the recommendations of its advisory committees, but it usually does.”
While it is widely accepted that the panel’s recommendations are pivotal, there have been no studies to evaluate whether that is in fact the case. In fact, it is not known how important panels are in the PMA process, both in final outcome or in the timing of PMA approval. We therefore examined all non-humanitarian exemption PMA panel meetings that occurred between 2005 and 2016 to try to evaluate the relationship between panel votes and PMA outcome. We tested two hypotheses: 1) PMAs that received more “yes” votes from their panel were more likely to be approved, and 2) the more “yes” votes a PMA received, the more quickly it would be approved by the FDA.
We evaluated 37 committee hearings under the old system, with committee meetings between 2005 and mid-2010. Of these 37 hearings, 26 of the PMAs were approved and 11 were not approved. We evaluated 52 committee hearings under the new system, with committee meetings between mid-2010 and the end of 2016. Of these, 48 were approved and 4 have not been approved. PMAs were significantly more likely to be approved under the new panel system than under the old one (χ2=6.00, df=1, p<0.05). In essence, PMAs that have gone to panel under the new system have been very likely to be approved. This suggests that FDA is generally not sending PMAs to an advisory panel unless they meet a fairly robust threshold. In fact, of the 156 separate votes (52 panels, with three votes apiece) cast by the panels under the new system, a majority of the panel voted “no” only 20% of the time. And even a negative vote did not doom the application, because most of these applications with a negative vote on at least one ballot were ultimately approved.
We also analyzed whether the wait times from PMA submission to panel meeting and from panel meeting to decision differed under the old and new panel systems. (We recognize that these were not the only changes over this period potentially affecting timing.) Among the 89 PMAs we evaluated, we found information on PMA submission date, committee date, and FDA decision date for 77 of them (all were approved; 49 in the new system, 28 in the old system). There was not a significant difference in the wait times under the old and new systems due to the very large variation in wait times (two tailed t-tests) (Figure 1). However, the wait from committee meeting to decision date was 111 days longer under the old system than under the new one (354 days (standard deviation=389 days) vs. 243 days (standard deviation=275 days), respectively). This presumably is attributable to policies relating to PMA processing, e.g., user fee commitments, and not to the changes in the panel process. Removing the three of the 77 PMAs that were not approved also resulted in non-statistically significant differences in wait times under the two committee systems.
Regarding our first hypothesis (a more supportive panel results in a higher likelihood of approval), we found that panel support does matter for approval. There was a significant relationship between the fraction of “yes” votes (“yes” votes divided by “yes,” “no,” and “abstain” votes) and the likelihood of approval in the full set of 89 PMAs, although this was affected by which committee system was used (logistic regression—z-value for “yes” vote fraction=4.04, p-value for “yes” vote fraction<0.0001; z-value for voting system=-2.06, p-value for voting system<0.05). We found that the fraction of “yes” votes under the new system for safety and benefit-to-risk ratio were significantly correlated with the likelihood of approval (logistic regression for safety: z=2.36, p<0.05; for benefit-to-risk: z=2.05, p<0.05).
The fraction of “yes” votes for effectiveness was not significantly correlated with the likelihood of approval (z=0.127, p<0.9). One reason the fraction of “yes” votes for effectiveness was not significantly correlated with the likelihood of approval may be because four PMAs under the new system were not approved, and, of those, three had high fractions of “yes” votes for effectiveness. (Three of the four, though, lost on benefit-risk.)
Regarding our second hypothesis (a more supportive panel results in a faster decision for approval), we found that among the 74 applications under the old and new systems that were approved and had complete date information there was a significant negative relationship between the fraction of “yes” votes and the time between the committee hearing and the decision (linear regression, r2=0.076, p<0.01) . In other words, PMAs with panels that voted more in favor of the application had shorter wait times until approval by FDA. However, there was not a significant relationship between the fraction of “yes” votes and time between submission and committee hearing, nor was there a correlation between wait times from submission to committee hearing and committee hearing and decision.
Because more “yes” votes for a device were associated with shorter waits until approval, we further analyzed whether this was true for the three separate votes in the new voting system (Figure 2). In fact, there were significant negative correlations between the effectiveness vote and time until approval (linear regression, r2=0.165, p<0.005, and between the benefit-to-risk ratio vote and time until approval (linear regression, r2=0.114, p<0.05, (i.e. the more “yes” votes, the shorter the time until approval))). Although the correlations are significant, these are still not strong relationships; the effectiveness and benefit-to-risk ratio “yes” vote fractions did not explain much of the variation in time until approval. Curiously, the correlation between the safety vote and time until approval was not significant (linear regression, r2=0.044, p<0.1). In other words, the panel votes on effectiveness and benefit-to-risk provide some, but not a lot of insight into how long it will take FDA to reach a decision.
It is interesting that the “yes” vote fractions for effectiveness and benefit-to-risk are significantly correlated with the wait from committee to decision but that the “yes” vote fractions for safety and benefit-to-risk ratio are significantly correlated with likelihood of approval. Thus, the benefit-to-risk ratio vote appears to confer the most information about the fate of the PMA, both in terms of wait until decision and what the decision will be. This is perhaps to be expected given that the benefit-to-risk ratio incorporates feedback from the other two votes.
Finally, although the public focus is on the vote, FDA has repeatedly stated that it is interested in more than the final tally. The agency stresses that it also considers what panel members say during the meeting, not just how they vote. None of the findings here should be construed as meaning that the panel process is unimportant or should be taken lightly. Indeed, the favorable voting patterns may reflect the intense preparation companies go through.
The PMA panel meeting presents a rare piece of public drama in the device regulatory process. The sponsor’s device, after years of research, development, testing, and submissions becomes open to scrutiny. Examining the data on advisory committees, though, shows this event has become less common. Moreover, while getting to a panel does not guarantee approval, the odds are fairly high that an approval will eventually follow. How long that will take is correlated with what transpires at the panel: stronger panel support is generally predictive of a faster approval. And, if you could learn only one thing about the panel vote, knowing the outcome of the benefit-to-risk vote is probably the most useful.
McKenzie Cato, an associate at Hyman, Phelps & McNamara, P.C., Washington, D.C. assisted with the article.
- 21 U.S.C. § 360c(a)(1)(C).
- 21 C.F.R. Part 820.
- Medical Device Amendments of 1976, Pub. L. No. 94-295, § 2, 90 Stat. 539, 552 (codified at 21 U.S.C. § 360e(e)(2)) (“Upon receipt of an application . . . , the Secretary shall refer such application to the appropriate panel . . . for study and for submission . . . of a report and recommendation respecting approval of the application, together with all underlying data and the reasons or basis for the recommendation.”).
- Safe Medical Devices Act of 1990, Pub. L. No. 101-629, § 18(c), 104 Stat. 4511, 4528 (codified at 21 U.S.C. § 360e(c)(3)).
- FDA, Guidance for Industry and FDA Staff: Procedures for Meetings of the Medical Devices Advisory Committee, at 2 (Sept. 1, 2017).
- FDA, Summary of Changes to CDRH’s Advisory Committee Process, https://www.fda.gov/AdvisoryCommittees/CommitteesMeetingMaterials/MedicalDevices/ucm208485.htm (last visited Jan. 13, 2019).
- Jeffrey N. Gibbs & David A. Gibbs, PMA Advisory Panels: The Impact of FDA’s Change in Policy on Voting Pattern, Regulatory Focus, Regulatory Affairs Professionals Society (May 2018).
- FDA, MDUFA Performance Goals and Procedures, Fiscal Years 2018 through 2022 (Dec. 2, 2016), https://www.fda.gov/downloads/forindustry/userfees/medicaldeviceuserfee/ucm535548.pdf.
- FDA, Guidance for Industry: Advisory Committee Meetings – Preparation and Public Availability of Information Given to Advisory Committee Members, at 6-7 (Aug. 2008).
- Melissa Foster, FDA Panel Favors Approval of New AAA Stent Graft System, Healio (June 12, 2018), https://www.healio.com/cardiac-vascular-intervention/aneurysm-repair/news/online/%7B2ae8e5cf-8574-4b29-8d6c-060f09850bb6%7D/fda-panel-favors-approval-of-new-aaa-stent-graft-system.
- FDA publicly announces approvals but not non-approvals. In the case of the four PMAs with no announcement of an approval, two of the panel meetings were in February 2016, one was in October 2010, and one was in July 2010.
- A p-value is a measure of statistical strength; a p-value of <.05 is typically considered statistically significant. A z-value comes from a logistic regression and is used to assess the statistical significance of a result relative to some cutoff z-value.
- One of the companies received unanimous positive votes on all three ballots, but then went out of business.
- r2 is a measure of the proportion of the variance in the result variable that is explained by the explanatory variable(s).
- equation: waiting days=572.9–fraction “yes” vote*444.7
- equation: waiting days=539.3–fraction “yes” vote*411.6