Introduction to Regulatory Statistical Principles

1. Introduction to Regulatory Statistical Principles Peter A. Lachenbruch Oregon State University College of Public Health and Human Sciences (Ret.)

2. Conclusion “There can be few areas where the discipline of statistics is conducted with greater discipline. Pharmaceutical statisticians may be engaged in work that may sometimes involve routine calculations, but the regular application of statistical principles to produce high quality experiments, data and analysis promotes a professionalism that can itself be a source of satisfaction.” Stephen Senn (2000 – The Statistician)

3. Research vs. Development • Planned development vs. Exploratory analyses • Implications regarding significance tests • Journal Publication or Licensing • Understanding a biological process or demonstrating a therapeutic benefit • Assess efficacy and Safety • Submit data for review by regulators

4. The Cast of Characters • Sponsor – the entity that will market the product, be responsible for writing the marketing application (NDA, BLA etc.) • Usually have many skills - Biology, Biochemistry, Medicine, Regulatory Affairs, Biostatistics, etc. • These people have roles that change over time: biologists in pre-clinical times, biostatistics in clinical phases and pre-clinical, etc. • May be drug company, university, government agency, etc.

5. Characters (2) • Investigators – the scientists who conduct the trials. Must demonstrate their qualifications. May be physicians, biochemists, etc. Employees of sponsor or contracted to sponsor (e.g., a University scientist), CRO (contract research organization) • May also be independent scientists – e.g., many statisticians serve as consultants to drug companies

6. Characters (3) • Regulatory agency – in US the FDA plays this role. Has 6 centers: CBER, CDER, CDRH, CVM, CFSAN, NCTR. Most of what we will talk of here is related to the first three (Biologics, Drugs and Devices) • Requirements are governed by the authorizing laws and regulations that have evolved over many years. See the Code of Federal Regulations (CFR)

7. Two trials needed • For many years, the FDA required two “adequate and well-controlled” trials that showed clinical benefit to license a drug or biologic. The sole exception was vaccines which usually were studied in large (n>5000) trials. • Recently, the agency has adopted a more flexible approach, especially in rare diseases and conditions in which obtaining enough patients for two trials would be difficult. • If you wish to license a product with one trial, you should contact the agency and discuss the case .

8. ICH • The International Conference on Harmonization (ICH) is an effort to require the same standard of evidence for licensure in various regions of the world. • Joint by US, Europe, and Japan regulatory agencies and by the pharmaceutical industry organizations in those regions. • You can find links to these at www.ich.org

9. Phases of Drug Development • Pre-clinical: • Prior to the first human study, the sponsor must show evidence that the product does not have apparent unacceptable risks for humans • Done in animals: rats, mice • Carcinogenicity, teratogenicity • Stability, shelf-life, potency, contaminant detection • Studies of Maximum Tolerated Dose, route of administration, frequency of administration (also done in phase 1 studies • ICH S series (safety) and Q series (quality)

10. Phase 1 Studies • Must have a valid Investigational New Drug authorization – allows sponsor to conduct studies in human populations • Find MTD • Show a therapeutic response in a wide enough dose range that the product can be used safely • Primarily safety studies. May be in non-diseased population or a diseased population. Depends on the drug. • Often not randomized

11. Phase 2 Studies • Learn more about proper dose, schedule of administration, route of administration (oral, subcutaneous, intravenous) • Vaccines are frequently single dose, sometimes a booster dose is given • Therapeutic drugs may be given multiple times • Some products are given by multiple routes (at physician discretion) • Pain medications usually given until relief is obtained, so concern about cumulative dose • Chronic medication – e.g., cholesterol lowering medication, cardiovascular meds, diabetes medication may be given for long periods – lifetime. Must monitor for problems.

12. Phase 2, continued • Must pre-specify response/outcome • Can’t collect many outcomes and select one that seems to work well. This is exploratory analysis and is never accepted by regulators. • Analysis plan must be specified • Report to regulatory agency of results, including safety, efficacy, other data. • It is wise to study several doses, routes of administration, and schedules of administration • Sometimes there is information from products of a similar class that will assist in these studies

13. Phase 3 Studies • Must show superiority of the product • Two trials usually needed • Prefer there to be two different populations; e.g. tertiary care and primary care centers • Need reports for each trial; often detailed information on patients is given. • The statistical analysis plan (SAP) must be specified and shouldn’t be changed unless there is new evidence of changes in the endpoints. Changes after the data have been locked are highly suspicious.

14. Working with Regulators (with reference to FDA) Goal of both sponsor and FDA is to bring safe, effective products to market Sponsors want to present their product in the most favorable light (sometimes downplaying the bad things) Regulators (FDA especially) are cautious and don’t want to approve a product that is ineffective or has a poor balance of benefit and risk.

15. Regulatory work (2) • Note that regulatory agencies are not monoliths: • Advice may differ based on which center, division, or branch you deal with; sometimes reviewers will affect the advice • Advice you get may depend on what the favored research paradigm is, approach, custom of division/branch – e.g., how missing values are treated • Note that you can appeal the decision if you differ from the reviewer

16. Regulatory work (3) • Evidence standard: • Two “convincing” trials – p-value below 0.05 • What if the two convincing trials happen after 10 attempts? • Can use a negative binomial to suggest this isn’t persuasive. I find the probability of 2 successes by the 10th trial is 0.0186 under the null hypothesis • If p(success ) is 0.8, the probability of 2 successful trials by 4 trials is 0.97 • Usually need to show superiority, rather than non-inferiority although safer products can be persuasive

17. Regulatory work (4) • Single trial can be accepted if • A very large trial such as a vaccine trial (tens of thousands of observations) • A very serious and rare disease is being studied • Speak to the regulators! Many sponsors are reluctant to discuss their concerns with the FDA since they seem to fear that if they tell some problem, the FDA will focus on it. • This is a recipe for disasters. It’s better to avoid problems in interpretation early than have a fight later on.

18. Some suggestions • Know your reviewers • Perspective • Listen to any advice they offer and make counter-proposals if appropriate – reviewers will listen. They may not agree • Reviewers may change during your study – it’s important to have a record of any agreements; don’t rely on memory. • Don’t rely on biological arguments – clinical trials may contradict the biological argument

19. Types of Paperwork • IND or IDE • The application to conduct a clinical trial. No trial can proceed without one. Can be amended • Outline studies to be conducted, modifications to ongoing studies, where the study will be conducted, when it will be, etc. • What drugs, dosage, schedules, route of administration will be examined, etc. • Specify statistical analysis plan at an early amendment • Non-standard analyses can be proposed – should be justified • Any IND can be amended – studies are not cast in stone.

20. Paperwork (2) • One-arm studies (no control) are usually unacceptable • FDA has seen too many trials that have the single arm later shown to be inferior to a control arm • Non-randomized studies are notorious for having bias – the investigator knows what drug is given, rates the patients more highly than in a controlled trial.

21. Paperwork (3) • A famous study (Moertel) extracted results from many oncology studies and found the non-randomized ones always showed high response rates, while the randomized ones were much poorer. Some studies of this type have been accepted when the population is small and the natural history of the disease is well known.

22. Paperwork (4) • Make use of ICH guidelines and FDA guidelines. • These are not mandatory, but represent ‘best current thinking’ in their areas. Alternatives can be proposed. • Most important for clinical trials are E3, E9, E10 • Generalizing to a population • Clinical trials are conducted on a small subset of a population. In order for a product to be approved for marketing, FDA and other agencies want to be convinced that the results apply to the broader population. For this reason, sponsors either use broad entry criteria, or have multiple studies on different populations • This also includes patients or subjects of different ages, different physical conditions, or different hospitals.

23. Paperwork (5): Marketing applications • NDA and BLA are applications to market a new product. NDAs are used by CDER (center for drugs) and BLAs are used by CBER (center for biologics) • The BLA requires close scrutiny of the manufacturing process and facilities, because of the greater variability of biological products. • The application will include complete reports on all studies including patient listings, analysis according to the original statistical analysis protocol (SAP) as well as any exploratory analyses.

24. Paperwork (6):Marketing Applications • Full safety analyses: • Listing of all adverse events • By organ system • By frequency • Since neither FDA nor sponsors can define what events will occur a priori, such analyses tend to be exploratory. Some direction may be found in other drugs of the same class (e.g., a new statin or bet blocker, a new vaccine, etc.)

25. Regulator’s needs/wants Well-written – good grammar, concise, well-organized Correct – relevant tables, proper statistics (e.g. no women in prostate cancer denominators, etc.); no adults in tables referring to pediatric applications Proper number of digits (no spurious precision) Data set noted as source for each table/graph or analysis

26. Regulator’s needs (2) • Justify any statements – data should demonstrate this. • Be open and honest about the data • If there are problems, discuss them – show the warts as well as the beauty. • Do not lie about the data – if an analysis is determined after the data has been locked, it can be treated as exploratory. FDA will usually not accept ‘discovered hypotheses.’

27. Research paper vs. Marketing Application In research, the goal is to have a good paper published in a good journal. The author(s) may contact the editor to ascertain interest in the topic. The paper itself will rarely contain the data and the analysis will not be reproduced by referees. For a marketing application, the sponsor will have been working with the regulatory agency for years and will submit all data. The analysis will be reproduced by the reviewers.

28. Research vs. Marketing (2) For a marketing application, FDA and sponsor may have agreed on some points. These generally aren’t binding on either party. It is similar to the journal – in either case, the journal may decide it isn’t appropriate for the journal; the regulatory agency may decide it isn’t adequate for licensure if the data don’t show convincing evidence of efficacy The ultimate goal in regulation is to write a clear, good label for the product that the prescriber and public can understand

29. Reasons a marketing application may fail to be approved The application is not reviewable – data aren’t adequate, application is incomplete (sections missing) Data do not show evidence of efficacy There are safety concerns even if efficacy is demonstrated – in one case I know of, there were serious safety concerns but a blinded review of the data indicated that these were not the problem the investigators thought they were.

30. Options if Marketing Application is not Approved Respond to criticisms Appeal to Office Director or Center Director Appeal to FDA Ombudsman Have advocacy group place pressure on FDA Have congressman put pressure The last two are rarely effective since science usually has left the debate Conduct a new study

31. Understanding FDA language “Must” is directive – no options to do otherwise “No” means no “Recommend” is not directive – you should discuss your plans in detail “Should” is strong, but less directive than must. Generally, FDA will not comment on something without reviewing the data Show clinical benefit usually means they are not interested in means.

32. FDA language(2) • Time issues: • What happens at k weeks is different from what happens by k weeks or within k weeks • If confused, ask! • In reporting, define what you mean. • “clinically meaningful improvement” must be carefully defined and not rely on a physician’s impression • Be careful about interpreting FDA words: wanting a sensitivity analysis means FDA wants to be sure small changes in assumptions don’t affect the conclusions.

33. FDA language (3) • FDA will often look at subgroups to ensure the findings are robust. This is not doing lots of subgroup analyses to find something – it is more looking at various groups to ensure a finding is real. • In one study, a sponsor had found an effect, but FDA did some subgroup analysis and found 3 of 4 sites had no effect and one had a major effect. Subsequent investigation found that the study coordinator for the site in question had unblinded the randomization and gave the treatment to patients she thought would have a better chance of benefit.

34. Interacting with Regulators • Don’t ignore advice – the regulators have likely seen similar products and know pitfalls • Make your submissions clear – ensure your analysis answers the questions fully. If they differ from advice, explain why • Ensure your data are complete and consistent • Have a plan for dealing with missing values • Always present the pre-specified analyses and any others you have done. • Analyses not pre-specified will be considered exploratory

35. Interacting with Regulators (2) • In one example, I had been working on a 2 part model theory and a sponsor had a study that I thought was appropriate for the application. • I suggested it and the regulatory affairs person said “well do it!” • I responded that the sponsor should check it out to see if it was appropriate. • A week or two later, the statisticians had run a small simulation study and decided that a Mantel-Haenszel test would work better.

36. Interactions with Regulators (3) • Communicate with regulators • Some sponsors ban such communications unless a regulatory affairs representative is present or on the phone – can result in delays • Often the regulatory affairs person does not understand the statistical details being discussed • This is a message for the regulatory affairs people – don’t delay work to maintain your control – trust your statisticians and other staff.

37. Interactions with Regulators (4) • Working within a system • Concurrent documentation of procedures and studies • Include data cleaning and editing procedures • Data analysis methods and formulas • SOPs for lab tests and all measurements of patients • Normal ranges for key variables • How various labs were standardized

38. Interactions with Regulators (5) • Definitions: • Treatments – including dose, schedule, route • Population being studied • Null and alternative hypotheses • Tests to be conducted • Size of tests • Power of tests at alternatives • Guidances and Regulations, ICH

39. Contents of IND • For a phase 3 trial • Design of study – often involves two group comparisons; superiority vs. non-inferiority • Inclusion and exclusion criteria • Outcomes (pre-specified!), how measured, how standardized across investigators • Covariables (not too many – remember label must be fairly general) – phase 3 is NOT the time for variable selection • The covariates should be predictive of outcome, not of imbalance.

40. IND (2) • Sample size computation – often it’s good to give several alternatives for different size, power and effect sizes. Give details • Distinguish between non-inferiority and superiority trials/analysis

41. IND (3) • Assumptions involved in analysis • How will missing values be handled (maybe have several methods) • I prefer not using LOCF • Multiple imputation • Model based methods – likelihood, mixed models, GEE • Details of randomization: within site, balancing (minimization) • Proposed analysis – equations, references, justification of effect size based on pilot studies, the literature, etc.

42. IND (5) • Modifications of a study • It’s almost always done as details change – e.g., recruiting is slow, so inclusion criteria may be broadened (age, disease inclusion, extending duration of recruitment) • SAP can be changed as long as data are still blinded to sponsor. • Changing endpoints is possible early on, but after data are complete, changing becomes suspect even if sponsor swears they haven’t peeked.

43. IND (5) • Documenting work- YES • Data cleaning and editing steps – if programs are used, give them – regulator may not examine, but can be useful • Acceptable ranges for covariables and endpoints – • Handling if out of range (go to site and check?) • Handling inconsistencies in data • Software used – some differences due to different algorithms – e.g. SAS offers several definitions of percentiles, Stata does not • Distinguish between exploratory and confirmatory – generally not helpful to have p-values with exploratory analyses

44. Meeting the Regulators • Prepare a briefing book describing issues to be discussed • Don’t bring up new results at the meeting as the FDA will say to submit the information and they will respond. Extreme example of a consultant presenting new, un-reviewed data at an advisory committee meeting. • FDA will answer questions, often close to the meeting date making it difficult to reply to their concerns

45. Meetings (2) • Useless question: • Does the FDA agree that these results support licensure? • Better to ask “what additional analyses are needed to support licensure?” • More helpful discussions related to schedule of submissions, etc.

46. Meetings (3) • Type A meetings – when drug development is stalled and sponsor can’t proceed without FDA input – e.g., clinical hold (need to know what is needed to remove the hold), or to resolve a dispute between sponsor and FDA, or for a Special Protocol Assessment. • FDA will schedule a type A meeting within 30 days of receipt of a written request

47. Meetings (4) • Type B meetings occur at natural junctures during the course of drug development – e.g. pre-IND, end of phase 1, end of phase 2-pre phase 3, pre-NDA/BLA • Pose clear questions based on interaction with FDA, problems observed during study • Type C meetings are all other meetings regarding the development and review of a product.

48. Meetings (5) • Regulators want to approve products but only on the basis of convincing data. • Sponsors will discuss issues before the meeting (dress rehearsals) • The FDA review team will also discus issues in each of the areas: safety, efficacy, quality • Note that FDA reviewers are privy to information from other studies in the same class and comments may be related to problems observed in other products. The FDA can’t disclose specifics, but sponsors should be aware that issues mentioned may be relevant to them.

49. A Few Statistical Issues • Recent developments and methodological research of importance to FDA • Incorporating Bayesian methods into drug approval – informativeness of priors. CDRH has accepted these for years. • Genomics – large number of variables, relatively few patients. What standards should be used for approval? License a process or a product? Seems to suggest using theory rather than clinical data

50. A Few Statistical Issues (2) • Selecting the margin for non-inferiority studies (I’m not sure where this is at present) • Robust methods for complex statistical models (e.g., hierarchical models for non-normal data) • Follow on Biologics • Biologics equivalent to generics – since Biologics are much more variable than drugs, many tough issues • Adaptive Methods