1 / 16

Content Analysis: Reliability

Content Analysis: Reliability. Kimberly A. Neuendorf, Ph.D. Cleveland State University Fall 2011. Reliability. Generally—the extent to which a measuring procedure yields the same results on repeated trials (Carmines & Zeller, 1979) Types: Test-retest: Same people, different times.

mika
Download Presentation

Content Analysis: Reliability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Content Analysis:Reliability Kimberly A. Neuendorf, Ph.D. Cleveland State University Fall 2011

  2. Reliability Generally—the extent to which a measuring procedure yields the same results on repeated trials (Carmines & Zeller, 1979) Types: Test-retest: Same people, different times. Intracoder reliability. . . Alternative-forms: Different people, same time, different measures. Internal consistency: Multiple measures, same construct. Inter-rater/Intercoder: Different people, same measures.

  3. Index/Scale Construction Similar to survey or experimental work e.g., Bond analysis—Harm to female, sexual activity Need to check internal consistency reliability (e.g., Cronbach’s alpha)

  4. Intercoder Reliability Defined: The level of agreement or correspondence on a measured variable among two or more coders What contributes to good reliability? careful unitizing, codebook construction, coder training (training, training!)

  5. Reliability Subsamples Pilot and Final reliability subsamples Because of drift, fatigue, experience Selection of subsamples Random, representative subsample “Rich Range” subsample Useful for “rare event” measures Reliability/variance relationship

  6. Intercoder Reliability Statistics - 1 Types Agreement Percent agreement Holsti’s Agreement beyond chance Scott’s pi Cohen’s kappa Fleiss’ multi-coder extension of kappa Krippendorff’s alpha(s) Covariation Spearman rho Pearson r Lin’s concordance correlation coefficient (rc)

  7. Reliability Statistics – 2 See handouts on (a) Bivariate Correlation and (b) Pearson’s and Lin’s Compared

  8. Reliability Statistics - 3 Core assumptions of coefficients “More scholarship is needed”—these coefficients have not been assessed!

  9. Reliability Statistics - 4 My recommendations Do NOT use percent agreement ALONE Nominal/Ordinal: Kappa (Cohen’s, Fleiss’) Interval/Ratio: Lin’s concordance Calculate via PRAM Reliability analyses as diagnostics, e.g., Problematic variables, coders (“rogues”?), variable/coder interactions Confusion matrixes (categories that tend to be confused)

  10. Reliability Statistics - 5 “Standards” for Minimums for Rel. Stats. Percent Agreement: 90%?? Kappa (Cohen’s, Fleiss’): .40 minimally, .60 OK, .80 good Pearson correlation; Lin’s concordance: .70 (~50% shared variance) --???

  11. Reliability Statistics - 6 • The problem of the “extreme” or “skewed” distribution • Can have a % agreement of .95 and a Cohen’s kappa of -.10!!! • Why? • What to do?

  12. PRAM: Program for Reliability Analysis with Multiple Coders Written by rocket scientists! Trial version available from Dr. N!

  13. pause

More Related