Systems Approaches to Disease Stratification

1. Systems Approaches to Disease Stratification Nathan Price Introduction to Systems Biology Short Course August 20, 2012

2. Goals and Motivation • Currently most diagnoses based on symptoms and visual features (pathology, histology) • However, many diseases appear deceptively similar, but are, in fact, distinct entities from the molecular perspective • Drive towards personalized medicine

3. Outline • Molecular signature classifiers: main issues • Signal to noise • Small sample size issues • Error estimation techniques • Phenotypes and sample heterogeneity • Example study • Advanced topics • Network-based classification • Importance of broad disease context

4. Molecular signature classifiers Overall strategy

5. Molecular signatures for diagnosis • The goals of molecular classification of tumors: • Identify subpopulations of cancer • Inform choice of therapy • Generally, a set of microarray experiments is used with • ~100 patient samples • ~ 104 transcripts (genes) • This very small number of samples relative to the number of transcripts is a key issue • Feature selection & model selection • Small sample size issues dominate • Error estimation techniques • Also, the microarray platform used can have a significant effect on results

6. Randomness • Expression values have randomness arising from both biological and experimental variability. • Design, performance evaluation, and application of classifiers must take this randomness into account.

7. Three critical issues arise… • Given a set of variables, how does one design a classifier from the sample data that provides good classification over the general population? • How does one estimate the error of a designed classifier when data is limited? • Given a large set of potential variables, such as the large number of expression levels provided by each microarray, how does one select a set of variables as the input vector to the classifier?

8. Small sample issues • Our task is to predict future events • Thus, we must avoid overfitting • It is easy (if the model is complicated enough) to fit data we have • Simplicity of model vital when data is sparse and possible relationships are large • This is exactly the case in virtually all microarray studies, including ours • In the clinic • At the end, want a test that can easily be implemented and actually benefit patients

9. Error estimation and variable selection • An error estimator may be unbiased but have a large variance, and therefore often be low. • This can produce a large number of gene sets and classifiers with low error estimates. • For a small sample, one can end up with thousands of gene sets for which the error estimate from the sample data is near zero!

10. Overfitting • Complex decision boundary may be unsupported by the data relative to the feature-label distribution. • Relative to the sample data, a classifier may have small error; but relative to the feature-label distribution, the error may be severe! • Classification rule should not cut up the space in a manner too complex for the amount of sample data available.

11. Overfitting: example of KNN rule N = 30 test sample; k = 3 N = 90

12. Example: How to identify appropriate models(regression… but the issues are the same) noise learn f from data

13. Linear…

14. Quadratic…

15. Piecewise linear interpolation…

16. Which one is best?

17. Cross-validation

18. Cross-validation

19. Cross-validation • Simple: just choose the classifier with the best cross-validation error • But… (there is always a but) • we are training on even less data, so the classifier design is worse • if sample size is small, test set is small and error estimator has high variance • so we may be fooling ourselves into thinking we have a good classifier…

20. LOOCV (leave-one-out cross validation)

21. mean square error: 0.96 mean square error: 2.12 best mean square error: 3.33

22. Estimating Error on Future Cases Data Set Resampling: Shuffled repeatedly into training and test sets. Average performance on test set provides estimate for behavior on future cases Can be MUCH different than behavior on training set Training Set Test Set NO information passage • Methodology • Best case: have an independent test set • Resampling techniques • Use cross validation to estimate accuracy on future cases • Feature selection and model selection must be within loop to avoid overly optimistic estimates

23. Classification methods • k-nearest neighbor • Support vector machine (SVM) • Linear, quadratic • Perceptrons, neural networks • Decision trees • k-Top Scoring Pairs • Many others

24. Molecular signature classifiers Example Study

25. Diagnosing similar cancers with different treatments ? GIST Patient LMS Patient • Challenge in medicine: diagnosis, treatment, prevention of disease suffer from lack of knowledge • Gastrointestinal Stromal Tumor (GIST) and Leiomyosarcoma (LMS) • morphologically similar, hard to distinguish using current methods • different treatments, correct diagnosis is critical • studying genome-wide patterns of expression aids clinical diagnosis • Goal: Identify molecular signature that will accurately differentiate these two cancers

26. Geman, D., et al. Stat. Appl. Geneti. Mol. Biol., 3, Article 19, 2004 Tan et al., Bioinformatics, 21:3896-904, 2005 Relative Expression Reversal Classifiers • Find a classification rule as follows: • IFgene A > gene BTHENclass1, ELSEclass2 • Classifier is chosen finding the most accurate and robust rule of this type from all possible pairs in the dataset • If needed, a set of classifiers of the above form can be used, with final classification resulting from a majority vote (k-TSP)

27. Rationale for k-TSP • Based on concept of relative expression reversals • Advantages • Does not require data normalization • Does not require population-wide cutoffs or weighting functions • Has reported accuracies in literature comparable to SVMs, PAM, other state-of-the art classification methods • Results in classifiers that are easy to implement • Designed to avoid overfitting • n = number of genes, m = number of samples • For the example I will show, this equation yields: • 10^9 << 10^20

28. Price, N.D. et al, PNAS 104:3414-9 (2007) 5 10 Classified as GIST 4 10 OBSCN expression 3 10 2 10 Clinicopathological Diagnosis X – GIST Classified as LMS O - LMS 1 10 1 2 3 4 5 10 10 10 10 10 C9orf65 expression Accuracy on data = 99% Predicted accuracy on future data (LOOCV) = 98% Diagnostic Marker Pair

29. Price, N.D. et al, PNAS 104:3414-9 (2007) RT-PCR Classification Results LMS GIST • 100% Accuracy • 19 independent samples • 20 samples from microarray study • including previously indeterminate case Price, N.D. et al, PNAS 104:3414-9 (2007)

30. Price, N.D. et al, PNAS 104:3414-9 (2007) Comparative biomarker accuracies C-kit gene expression GIST – X LMS – O 2-gene relative expression classifier Price, N.D. et al, PNAS 104:3414-9 (2007)

31. Price, N.D. et al, PNAS 104:3414-9 (2007) Kit Protein Staining of GIST-LMS Blue arrows - GIST Red arrows - LMS Accuracy as a classifier ~ 87%. • Top Row – GIST Positive Staining • Bottom Row – GIST negative staining

32. A few general lessons • Choosing markers based on relative expression reversals of gene pairs has proven to be very robust with high predictive accuracy in sets we have tested so far • Simple and independent of normalization • Easy to implement clinical test ultimately • All that’s needed is RT-PCR on two genes • Advantages of this approach may be even more applicable to proteins in the blood • Each decision rule requiring the measurement of the relative concentration of 2 proteins

33. Network-based classification

34. Chuang, Lee, Liu, Lee, Ideker, Molecular Systems Biology 3:40 Network-based classification • Can modify feature selection methods based on networks • Can improve performance (not always) • Generally improves biological insight by integrating heterogeneous data • Shown to improve prediction of breast cancer metastasis (complex phenotype)

35. Rationale: Differential Rank Analysis (DIRAC) Price, N.D. et al, PNAS, 2007 • Networks or pathways inform best targets for therapies • Cancer is a multi-genic disease • Analyze high-throughput data to identify aspects of the genome-scale network that are most affected • Initial version uses a priori defined gene sets • BioCarta, KEGG, GO, etc. • Differential rank conservation (DIRAC) for studying • Expression rank conservation for pathways within a phenotype • Pathways that discriminate well between phenotypes Eddy, J.A. et al, PLoS Computational Biology (2010)

36. Differential Rank Conservation …across pathways in a phenotype tightly regulated pathway Highest conservation g3 g3 g3 g3 …across phenotypes for a pathway g1 g2 g2 g2 g1 g1 g1 g2 g4 g4 g4 g4 shuffled pathway ranking between phenotypes weakly regulated pathway GIST LMS g6 g5 g7 g7 g3 g4 g8 g8 g7 g6 g2 g1 Lowest conservation g7 g8 g6 g6 g1 g3 g5 g5 g5 g8 g4 g2

37. Visualizing global network rank conservation … … …

38. Visualizing global network rank conservation Average rank conservation across all 248 networks: 0.903 … … …

39. Global regulation of networks across phenotypes Highest rank conservation Lowest rank conservation Eddy et al, PLoS Computational Biology, (2010)

40. Global regulation of networks across phenotypes Highest rank conservation Lowest rank conservation Tighter network regulation:normal prostate Looser network regulation:primary prostate cancer Loosest network regulation:metastatic prostate cancer Eddy et al, PLoS Computational Biology, (2010)

41. Differential Rank Conservation …across pathways in a phenotype tightly regulated pathway Highest conservation g3 g3 g3 g3 …across phenotypes for a pathway g1 g2 g2 g2 g1 g1 g1 g2 g4 g4 g4 g4 shuffled pathway ranking between phenotypes weakly regulated pathway GIST LMS g6 g5 g7 g7 g3 g4 g8 g8 g7 g6 g2 g1 Lowest conservation g7 g8 g6 g6 g1 g3 g5 g5 g5 g8 g4 g2

42. Differential rank conservation of the MAPK network

43. DIRAC classification is comparable to other methods Cross validation accuracies in prostate cancer

44. Eddy et al, PLoS Computational Biology, (2010) Differential Rank Conservation (DIRAC): Key Features • Independent of data normalization • Independent of genes/proteins outside of network • Can show massive/complete perturbations • Unlike Fischer’s exact test (e.g. GO enrichment) • Measures the “shuffling” of the network in terms of the hierarchy of expression of he components • Distinct from enrichment or GSEA • Provides a distinct mathematically classifier to yield measurement of predictive accuracy on test data • Stronger than p-value for determining signal • Code for the method can be found at our website: http://price.systemsbiology.net

45. Global Analysis of Human Disease Importance of broad context to disease diagnosis

46. The envisioned future of blood diagnostics

47. Next generation molecular disease-screening

48. Why global disease analyses are essential • Organ-specificity: separating signal from noise • Hierarchy of classification • Context-independent classifiers • Based on organ-specific markers • Context-dependent classifiers • Based on excellent markers once organ-specificity defined • Provide context for how disease classifiers should be defined • Provide broad perspective into how separable diseases are and if disease diagnosis categories seem appropriate

49. GLOBAL ANALYSIS OF DISEASE-PERTURBED TRANSCRIPTOMES IN THE HUMAN BRAIN Example case study

50. Multidimensional scaling plot of brain disease data