1 / 40

Parallelism in practice USP Bioassay Workshop August 2010

Parallelism in practice USP Bioassay Workshop August 2010. Ann Yellowlees Kelly Fleetwood Quantics Consulting Limited. Contents. What is parallelism? Approaches to assessing parallelism Significance Equivalence Experience Discussion. Setting the scene: Relative Potency. RP :

john
Download Presentation

Parallelism in practice USP Bioassay Workshop August 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Parallelism in practiceUSP Bioassay Workshop August 2010 Ann Yellowlees Kelly Fleetwood Quantics Consulting Limited

  2. Contents • What is parallelism? • Approaches to assessing parallelism • Significance • Equivalence • Experience • Discussion

  3. Setting the scene: Relative Potency RP: ratio of concentrations of reference and sample materials required to achieve the same effect RP = Cref / Csamp

  4. Parallelism • One curve is a horizontal shift of the other • These are ‘parallel’ or ‘similar’ curves • Finney: • A prerequisite of all dilution assays

  5. Real data: continuous response

  6. Linear model(4 concentrations) Linear: Y = a+ β log (C) Parallel when the slopes β equal

  7. Four parameter logisticmodel 4PL: Y = γ+ (δ -γ) / [1 + exp (β log (C) – α)] Parallel when asymptotes γ ,δ slope β equal

  8. Five parameter logisticmodel 5PL: Y = γ+ (δ -γ) / [1 + exp (β log (C) – α) ]φ Parallel when asymptotes γ ,δ slopeβ asymmetry φ equal

  9. Is there evidence that the reference and test curves ARE NOT parallel? Compare unrestricted vs restricted models Test loss of fit when model restricted to parallel ‘p value’ approaches Traditional F test approach as preferred by European Pharmacopoeia Chi-squared test approach as recommended by Gottschalk & Dunn (2005) Tests for parallelismApproach 1

  10. Is there evidence that the reference and test curves ARE parallel? Equivalence test approach as recommended in the draft USP guidance (Hauck et al 2005) Fit model allowing non-parallel curves Confidence intervals on differences between parameters Pharmacopoeial disharmony exists!! (existed?) Approach 2

  11. In practice... Four example data sets Data set 1: 60 RP assays (96 well plates, OD: continuous) Data set 2: 15 RP assays (96 well plates, OD : continuous) Data set 3: 12 RP assays (96 well plates, OD : continuous) Data set 4: 60 RP assays (in vivo, survival at day x: binary*) * treated as such for this purpose; wasteful of data 11

  12. We have applied the proposed methods in the context of individual assay ‘pass/fail’ (suitability): Data set 1 Compare 2 ‘significance’ approaches Compare ‘equivalence’ with ‘significance’ Data sets 2, 3 Compare 2 ‘significance’ approaches Data set 4 Compare ‘F test’ (EP) with ‘equivalence’ (USP) In practice...

  13. Data set 1 60 RP assays 8 dilutions 2 independent wells per dilution 4PL a good fit (vs 5PL) NB precision 13

  14. Data set 1: F test and chi-squared test • F test: straightforward • Chi-squared test: • need to establish mean-variance relationship

  15. Data set 1: F test and chi-squared test • F test: • 12/60 = 20% of assays have p < 0.05 • Evidence of dissimilarity? • – OR – • Precise assay? • Chi-squared test: • 58/60 = 97% of assays have p < 0.05! • Intra-assay variability is low  differences between parallel and non-parallel model are exaggerated

  16. Data set 1: Comparison of approaches to parallelism

  17. Data set 1: Comparison of approaches to parallelism • Some evidence of ‘hook’ in model • Residual SS inflated

  18. Data set 1: Comparison of approaches to parallelism

  19. Data set 1: F test and chi-squared test RSSparallel = 159 RSSnon-parallel = 112 RSSp – RSSnp = 47 Pr(23>47) < 0.01 F test: P = 0.03

  20. Data set 1: F test and chi-squared test RSSparallel = 100.2 RSSnon-parallel = 99.0 RSSp – RSSnp = 1.2 Pr(23>1.2) = 0.75

  21. Data set 1: USP methodologyProve parallel • Lower asymptote:

  22. Data set 1: USP methodology • Upper asymptote:

  23. Data set 1: USP methodology • Scale:

  24. Data set 1: USP methodology • Criteria for 90% CI on difference between parameter values: • Lower asymptotes: (-0.235, 0.235) • Upper asymptotes: (-0.213, 0.213) • Scales: (-0.187, 0.187) • Applying the criteria: • 3/60 = 5% of assays fail the parallelism criteria • No assay fails more than one criterion

  25. Data set 1: Comparison of approaches to parallelism

  26. Data set 1: Comparison of approaches to parallelism • This plate ‘fails’ all 3 tests • USP: Lower asymptote

  27. Data set 1: Comparison of approaches to parallelism • Equivalence test: scales not equivalent • F test p-value = 0.60 • Chi-squared test p-value < 0.001

  28. Data set 2: Comparison of approaches to parallelism 28

  29. Data set 3: Comparison of approaches to parallelism 29

  30. Data set 4: Compare ‘F test’ with ‘equivalence’ Methodology for Chi-squared test not developed for binary data In practice...

  31. Data set 4 60 RP assays 4 dilutions 15 animals per dilution 31

  32. Data set 4: Comparison of approaches to parallelism

  33. Data set 4: Comparison of approaches to parallelism

  34. Broadly... F test Fail (?wrongly?) when very precise assay Pass (?wrongly?) when noisy Linear case: p value can be adjusted to match equivalence Chi-squared Fail when very precise assay (even if difference is small) If model fits badly – weighting inflates RSS (e.g. hook) 2 further data sets supported this USP Limits are set such that the extreme 5% will fail They do! Regardless of precision, model fit etc 34

  35. Stepping back… What are we trying to do? Produce a biologic to a controlled standard that can be used in clinical practice For a batch we need to know its potency With appropriate precision In order to calculate clinical dose 35

  36. Some thoughts Establish a valid assay Use all development assay results unless a physical reason exists to exclude them Statistical methodology can be used to flag possible outliers for investigation USP <111> applies this to individual data points Parallelism / similarity Are the parameter differences fundamentally zero? Or is there a consistent slope difference (e.g)? Equivalence approach + judgment for acceptable margin 36

  37. Some thoughts 2. Set number of replicates to provide required precision Combine RP values plus confidence intervals for reportable value Per assay, use all results unless physical reason not to (They are part of the continuum of assays) Flag for investigation using statistical techniques Reference behaviour Parallelism 4. Monitor performance over time (SPC) Reference stability Parallelism 37

  38. Which parallelism test? Our view: Chi squared test requires too many complex decisions and is very sensitive to the model F test not generally applicable to the assay validation stage Does not allow examination of the individual parameters Does not lend itself to judgment about ‘How parallel is parallel?’ The equivalence test approach fits in all three contexts With adjustment of the tolerance limits as appropriate 38

  39. Thank you • USP the invitation • Clients use of data • BioOutsource: www.biooutsource.com • Other clients who prefer to remain anonymous • Quantics staff analysis and graphics • Kelly Fleetwood (R), Catriona Keerie (SAS)

More Related