Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self

1. Statistical positivity criteria and multiplicity adjustment methods for the analysis of gated ICS assay data Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self Statistical Center for HIV/AIDS Research and Prevention (SCHARP) HIV Vaccine Trials Network (HVTN) Fred Hutchinson Cancer Research Center

2. Outline • Positivity criteria • Multiplicity adjustment methods • Simulations • Assay validation results • Summary

3. 5-way contingency table representation of the gated ICS data structure of one sample . . .

4. Positivity criteria • For each combination of peptide-pool stimulation, T-cell subset and cytokine subset, an assay result is categorized as positive or negative depending on whether or not the percentage of positive staining cells in the experimental wells is significant greater than that in the negative control wells • Response rates (number of vaccine recipients who have positive response) are the primary immunogenicity endpoint in most Phase I HIV-1 vaccine trials and one of the primary measurements in assessing surrogate endpoints in efficacy trials • Allow comparisons across labs, studies and population

5. Positivity criteria • Empirical positivity criteria: Background subtracted responses > = 0.05% and at least 3 times the background responses • Reasonable but no information on the control of false positive rate • Ignorance of multiple comparisons involved in assessment of an overall response • Statistical positivity criteria: multiplicity adjusted p-value < alpha • Optimizing sensitivity • False positive control of the assay • Accommodations of multiple comparisons

6. Notation For each sample, let and denote the cell counts of the kth, k =1, …, K, cytokine subset from the experimental well and negative control well, respectively. Then the joint distribution of the X’s: where n: total number of cells from the CD4 or CD8 T-cell subset D: dependence structure

7. Hypotheses • In addition to (or instead of) the omnibus test, • To maximize the advantage of multi-parameter flow data, of interest are: … … … Often times, K= 2^(# of cytokines) -1. However, more variables such as presence of at least one (or more) cytokines are also of interest and can to be tested.

8. Tests • Upper-tail two sample comparisons are constructed for each variable – the alternative hypotheses are that the stimulated responses are higher than the corresponding negative control responses. • Since some counts of events could be as small as 5, we use the Fisher’s exact test to account for the discreteness and sparseness of the data

9. Need for multiplicity adjustment • For each sample, there are at least 2^(#cytokines) tests are of interest for each T-cell subset and peptide pool combination • Without adjustment, it is too easy to reject null hypotheses, thereby reaching potentially erroneous conclusions • In the context of vaccine development and evaluation, it is more dangerous to claim a result to be true when it is not true than to fail to claim a result that is true • Need to use appropriate multiplicity adjustment methods to control family-wise error

10. Multiplicity adjustment methods The Bonferroni method can be conservative due to • Ignorance of the correlations structures among the multiple raw p values (permutation of the whole vector) • Equally weighting of clear false nulls and true nulls (stepwise procedure) • Ignorance of the discreteness of the sampling distributions (taking into account the precise discrete characteristics)

11. Multiple tests with discrete distributions(Westfall & Wolfinger, 1997) • Using the distribution of the minimum of the p values • Assuming the p-values are independent • Using the Bonferroni in equality – discrete Bonferroni adjusted p-values: , where Pi refer to the random null p values , where otherwise. And, pit are the observable values of the random p value for the ith variable

12. Multiple tests with discrete distributions • Using , the random null p values are generated by re-sampling (permuting) the nE + nN binary data vectors, taking the first nE for the re-sampled “experimental well” data and the last nN for the re-sampled “negative control well” data. • By re-sampling vectors, the correlation among binary outcomes is incorporated. • In addition, any redundancy (e.g., induced by including the composite category (any)) is explicitly accounted for in the adjusted p values • The step-down fashion can be easily added to further enhance the power

13. Multiple tests with discrete distributions • Using the independence-assuming adjustment • Using discrete Bonferroni adjustment • pit are obtained by exactly enumerating the discrete vectors • While the re-sampling multiplicity method does incorporate correlations, the fantastic gains over the Bonferroni method come from incorporating discreteness.

14. Other multiplicity adjustment methods • Tarone (1990) recognized the problem of the discretness of the sampling distribution, and proposed a modified Bonferroni method that incorporates only those variables that can possibly contribute to the multiplicity adjustment • Hommel and Krummenaur (1998)

15. Definition of power in multiple testing Assuming there are k null hypotheses Hi. Suppose some are true nulls, and all remaining hypotheses are false. Various definitions of power include: • The prob. of correctly rejecting at least one false null (minimal power) • The prob. of rejecting all false null hypotheses (complete power) • The prob. of rejecting exactly one false null, and • The prob. of rejecting a specific false null. • Minimal power is assessed in our simulations because the primary endpoint is defined as having a positive response in at least one comparison

16. Simulation results Sample size N = 50,000 Number of simulated data: 100 Number of permutations: 1000000 Alpha: 0.00001

17. Qualitative validation of 8-color ICS assays • Examine 20 HIV+ and 50 HIV- PBMC samples • Responses to any of Gag1, Gag2 or Nef peptide pool indicate a positive response • Statistical criteria for determining positivity: resampling-based DB multiplicity adjusted p-value <= 0.00001 HIV Status # positive/total % Negative Positive 3/50 20/20 6% 100% Note: False positives were most likely due to SEB contaminations

18. PTE peptide validation using 8-color ICS assays • Examine 10 HIV+ and 50 HIV- PBMC samples • Responses to any of Gag1, Gag2 , Nef, Pol1, pol2 or pol3 peptide pool indicate a positive response • Statistical criteria for determining positivity: resampling-based DB multiplicity adjusted p-value <= 0.00001 HIV Status # positive/total % Negative Positive 0/49 9/10 0% 90%

19. Summary • A multivariate binomial framework is used in determination of variable-specific positivity of an ICS assay result • Re-sampling based multiplicity adjustment, in tandem with the discrete Bonferroni method is used to handle multiple comparisons to incorporate both correlations and discreteness of the data • This method allows incorporation of additional variables or collapsing of multiple variables for comparisons • Simulation studies have shown higher power of the this method as compared to others • High sensitivity and specificity of the 8-color ICS assay were observed using the current statistical positivity criteria

20. Reference • Hommel, G. and Krummenauer, F. (1998). Improvements and modifications of Tarone’s multiple test procedure for discrete data. Biometric 54, 673 -681. • Leon, A. C. and Heo M. (2005). A comparison of multiplicity adjustment strategies for correlated binary endpoints. JBS 15: 839-55 • Tarone, R. E. (1990). A modified Bonferroni method for discrete data. Biometrics 46, 515-22. • Westfall, P. H., Wolfinger, R. D. (1997). Multiple tests with discrete distributions. Amer. Stat. 51:3-8 • Westfall, P. H., Young, S. S. (1989). P value adjustments for multiple tests in multivariate binomial models. JASA 84:780 -6 • Westfall, P. H. and Young, S. S. (1993). Resampling-based multiple testing: Examples and Methods for P-value Adjustment, New York: John Wiley & Sons. • Westfall, P. H., Tobias R. D., Rom D., Wolfinger R. D. and Hochberg Y. (1999). Multiple Comparisons and multiple tests using SAS. Cary, NC: SAS Institute Inc.