1 / 19

Distributions and Concepts in Probability Theory

This session covers essential distributions and concepts in probability theory, their application in machine learning, and key techniques such as biased and unbiased estimators. Learn about the Exponential Family, Conjugate Priors, and how to calculate Expectation and Variance in various distributions.

pwilcox
Download Presentation

Distributions and Concepts in Probability Theory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Distributions and Concepts in Probability Theory 10701 Recitation PengtaoXie

  2. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

  3. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

  4. Distributions • Usage in machine learning • Expectation and variance • Bernoulli, Beta, multinomial, Dirichlet, Gaussian

  5. Usage of distributions in ML • Gaussian: Least Square Regression, Mixture of Gaussians, Kalman Filtering, Gaussian Markov Random Field, Gaussian Process • Multinomial: Hidden Markov Model, Mixture of Gaussians, Latent Dirichlet Allocation, Naive Bayes classifier • Dirichlet: Latent Dirichlet Allocation, Dirichlet Process • Bernoulli: Logistic Regression, switching variables in Graphical Models • Beta: Beta Process

  6. Expectation and variance • Expectation: the average value of a random variable under its probability distribution • Variance: a measure of how much variability there is in x around its mean value

  7. Distributions Random Variable Discrete Random Variable Continuous Random Variable Two Outcomes Multiple Outcomes Bernoulli Distribution Multinomial Distribution Gaussian Distribution Conjugate Conjugate Beta Distribution Dirichlet Distribution Conjugate

  8. Bernoulli • Model binary variable {0,1} • Probability mass function • Expectation

  9. Multinomial distribution • Model variables taking K possible states • 1-of-K coding • Probability mass function • Expectation

  10. Beta • Prior of the parameter in Bernoulli distribution • Probability density function • and are pesudo counts Refer to note1.pdf

  11. Dirichlet distribution • Prior of parameters in multinomial distribution • Probability density function • are pesudo counts

  12. Univariate Gaussian distribution • Model continuous variables • Probability density function • Expectation and variance

  13. Multivariate Gaussian distribution • Defined on a continuous random vector • Probability density function • Expectation and covariance

  14. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

  15. Exponential Family Distribution • A class of distributions sharing a certain form • Natural parameters and sufficient statistics • Special cases: Bernoulli, Beta, multinomial, Dirichlet, Gaussian • Moment generating property Refer to note2.pdf

  16. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

  17. Conjugate Prior both Beta both Dirichlet both Gaussian From the same distribution Refer to note3.pdf

  18. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

  19. Estimator Bias • Bias of an estimator • Unbiased estimator and biased estimator • Example: MLE for Gaussian mean and variance Refer to note4.pdf

More Related