Reliability and Validity

1. Reliability and Validity Quality of Data

2. Are we testing what we think we’re testing?

3. Quantitative Data • Reliability • Validity • Face • External • Internal

4. Reliability • Implies that the same data would have been collected each time over repeated tests/ observations. • Would a particular technique (or survey question) yield the same result each time? • “Did you go to church last week?” vs. “How many times have you been to church in your life?” • Reliability does not ensure accuracy. • Taken from Babbie, E.

5. Reliability • Problem if interpret questions differently • Poorly worded questions • Inconsistent coding: coding errors as with open-ended questions • Lack of definition of key terms

6. Reliability • Poorly worded: Does the library have adequate facilities and equipment for physically disabled students • Better: Can patrons in wheelchairs retrieve books from the browsing collection?

7. Reliability (indicators) • Pretest • Repeat question(s) • Test/retest • Split half and Parallel • Interscore or scorer

8. Validity • A term to describe a measure that accurately reflects the concept it is intended to measure. • Which is a more “valid” indicator of intelligence- an IQ score, or number of hours spent studying? • Ultimate validity cannot be proven, but can be supported by face, internal, and external measures. • Babbie, E.

9. Types of Validity • Face validity: The quality of an indicator/ question/ test that makes it a reasonable measure of a variable. • Church attendance is an indication of religiosity. • Number of grievances filed is an indicator of worker morale

10. Internal Validity • Approximate truth about inferences regarding causal relationships • Typically applied to studies using inferential statistics (i.e. quantitative measures) than descriptive or observation studies. • Especially useful for studies assessing affects of programs • Only applicable to the study in question- not generalizable. Why not? • Key question: Whether observed changes can be attributed to your program (the cause) and NOT other possible impacts/ causes.

11. Internal Validity Trochim, W.

12. Internal Validity • History or specific events: raises the issue that some variable other than the independent variable accounted for the change in the dependent variable. E.G.: the length of time between conducting the pretest and posttest may have a detrimental effect. • Maturation: the change results from biological or psychological processes, which occurred over time, and not from the treatment itself. Maturation becomes more a concern the longer the period between the pretest and posttest • Pretesting: may affect the dependent variable. Pretesting may alert participants or educate them about the topic under investigation. Therefore if subjects are administered a posttest, their performance may reflect a marked improvement • Measuring instruments or observational techniques: These—not the treatment—may account for the change in the dependent variable. Further, the validity of study findings may have been influenced by the fact that the evaluators as observers, raters, graders, interviewers, and coders gained experience, became tired, obtained a more complete understanding of the project, or eased their expectations of test subjects

13. Internal Validity (continued) • A nonrandom assignment of subjects to groups may signify that the groups were dissimilar from the beginning. Therefore any change might be attributed to the differential selection of subjects, rather than the actual treatment. • Statistical regression refers to the tendency for extreme scores to regress or move toward the common mean of subsequent measures. The assignment of subjects to a particular test group on the basis of extreme views may affect study findings.

14. Internal Validity (continued) • Mortality refers to the possibility that some subjects may have dropped out of the study after completion of the pretest but before the administration of the posttest. In such instances, every effort should be made to identify any common patterns or characteristics to ensure that any difference between a group’s pretest and posttest scores cannot be attributed to the loss of subjects. • Interaction refers to the fact that more than one of the previous threats might be in play. This is especially likely in those cases where subjects were not randomly assigned to groups and the evaluation was based on existing, intact groups.

15. External Validity • The approximate truth of generalizations drawn from a study. • The degree to which conclusions drawn from your study sample would hold true to other persons in other places at other times • Trochim, W.

16. External Validity Trochim, W.

17. External Validity • Example: institutions of higher education in Massachusetts: control, highest degree offered, and some characteristics of library (staff number, budget, and volume number) • Return rate? • Do respondents differ from non-respondents as a group?

18. Validity • Content validity (for achievement test): How well does the test sample what the students learned? How well does a standardized test cover what was taught in the information literacy program?

19. Validity (continued) • Criterion-related (predictive) (attitude test to predict performance in a library skills program): Who well does the test predict achievement for college freshmen? • Criterion-related (diagnostic): How well does the test diagnose current problems with library use?

20. Validity (continued) • Construct validity: How well does the test measure comprehension of library use? Does a test on the use of an OPAC really measure effective and efficient use rather than one’s ability to read test items?

21. Qualitative Study Equivalent • Credibility • Dependability • Confirmability • Transferrability

22. Qualitative Reliability • Researcher is the “instrument”- how to test for reliability? • Provide details of method, and abundance of evidence • Provide evidence of qualifications as observer • Make assumptions (and possible biases) clear • State research questions clearly • Use early stages of study to generate focus • Observe for an adequate period of time, across a full range of activities • Collect data from multiple sources • Save data for reanalysis

23. Qualitative Validity • Depends upon reliablity. Like reliability, asserted by documenting steps • Triangulation- data from different sources/ methods • Full documentation of data- “chain of evidence” • Logical connections between data and conclusions • Conscious and deliberate inclusion of data that might not support thesis • Preparedness to entertain alternatives • Self-reflection, acknowledgement of own biases • Review of preliminary reports by objective observers • Awareness of limitations • Gorman and Clayton

24. Qualitative Study: Increasing Reliability and Validty -Gorman and Clayton

25. Qualitative Study: Increasing Reliability and Validity cont’d -Gorman and Clayton

26. For a sweeping study When conduct it? For how long? How deal with reliability and validity? Course evaluation Example

27. References • Babbie, E. (2005). The basics of social research. Belmont, CA: Wadsworth Publishing. • Gorman, G.E. & Clayton, P. (2005). Qualitative research for the information professional: A practical handbook. London: Facet Publishing. • Trochim, W. M. K. (2006). Research methods knowledge base. Retrieved July 8, 2008 from http://www.socialresearchmethods.net/kb/intval.php • http://www.socialresearchmethods.net/kb/external.php