Reliability and Validity in Research

1. Reliability andValidity in Research Spring 2013University of Missouri -St. Louis

2. Believing what you read? • There is a need for reliable and valid data on student learning outcomes. Reliability: the extent to which an assessment tool is consistent or free from random error in measurement • Validity concerns the degree to which inferences about students based on their test scores are warranted. Validity: the extent to which an assessment tool measures what it is intended to measure

3. Validity • Validity has been defined as referring to the appropriateness, correctness, meaningfulness, and usefulness of the specific inferences researchers make based on the data they collect. • It is the most important idea to consider when preparing or selecting an instrument. • Validation is the process of collecting and analyzing evidence to support such inferences.

4. Evidence of Validity • There are 3 types of evidence a researcher might collect: • Content-related evidence of validity • Content and format of the instrument • Criterion-related evidence of validity • Relationship between scores obtained using the instrument and scores obtained • Construct-related evidence of validity • Psychological construct being measured by the instrument

5. Content-related Evidence • A key element is the adequacy of the sampling of the domain it is supposed to represent. • The other aspect of content validation is the format of the instrument. • Attempts to obtain evidence that the items measure what they are supposed to measure typify the process of content-related evidence.

6. Criterion-related Evidence • A criterion is a second test presumed to measure the same variable. • There are two forms of criterion-related validity: • Predictive validity: time interval elapses between administering the instrument and obtaining criterion scores • Concurrent validity: instrument data and criterion data are gathered and compared at the same time • A Correlation Coefficient (r) indicates the degree of relationship that exists between the scores of individuals obtained by two instruments.

7. Construct-related Evidence • Considered the broadest of the three categories. • There is no single piece of evidence that satisfies construct-related validity. • Researchers attempt to collect a variety of types of evidence, including both content-related and criterion-related evidence. • The more evidence researchers have from different sources, the more confident they become about the interpretation of the instrument.

8. How can validity be established? • Quantitative studies: • measurements, scores, instruments used, research design • Qualitative studies: • ways that researchers have devised to establish credibility: member checking, triangulation, thick description, peer reviews, external audits

9. Reliability • Refers to the consistency of scores or answers provided by an instrument. • Scores obtained can be considered reliable but not valid. • An instrument should be reliable and valid, depending on the context in which an instrument is used.

10. Reliability, continued • In statistics or measurement theory, a measurement or test is considered reliable if it produces consistent results over repeated tests. • Refers to how well we are measuring whatever it is that is being measured (regardless of whether or not it is the right quantity to measure).

11. Reliability, continued • Unlike the common understanding, in these contexts “reliability” does not imply a value judgment • Your car always starts/doesn’t start • Your friend is always/ never late

12. Reliability of Measurement

13. Errors of Measurement • Because errors of measurement are always present to some degree, variation in test scores are common. • This is due to: • Differences in motivation • Energy • Anxiety • Different testing situation

14. Observational Studies • Some characteristics cannot be measured through a test • Unobtrusiveness • Multiple sources of error • Reliability depends on the extent to which observers agree

15. How can reliability be established? • Quantitative studies? • Assumption of repeatability • Qualitative studies? • Reframe as dependability and confirmability

16. Reliability and Validity

17. Reliability and Validity • Why do we bother? • Terms used in conjunction with one another • Quantitative Research: R & V are treated as separate terms • Qualitative Research: R & V are often all under another, all encompassing term • Semi-reciprocal relationship