Educational Research

Educational Research Chapter 5 Selecting Measuring Instruments Gay and Airasian

Collecting Data • The collection of data is an extremely important part of all research endeavors, for the conclusions for a study are based on what the data reveal. • As a result, the kind (s) of data to be collected, the method (s) of collection to used, and the scoring of the data need to be considered with care.

In this chapter: Define data Present several types of instruments that can be used to collect data in a research study Different properties that scores are assumed to possess “Data”

1) Explain what is meant by the term “data” 2) Explain what is meant by the term ‘instrumentation” 3( Name three ways in which data can be collected by researchers Explain what is meant by the term “data-collection instrument” Describe five types of researcher-completed instruments used in educational research Describe five types of subject-completed instruments used in educational research Objectives: By the end of this chapter you should be able to:

Explain what is meant by the term ‘unobtrusive measures” and give two examples of such measures Name four types of measurement scales and give an example of each Name three different types of scores used In educational research and give an example of each Objectives

Objectives: • Describe briefly the difference between norm-referenced and criterion-referenced instrument • Describe how to score, tabulate, and code data for analysis

Identify the variable Operationally define the variable Locate data (measures, observations, documents with questions and scales) Collect data on instruments yielding numeric scores Self-efficacy for learning from others Level of confidence that an individual can learn something by being taught by others 13 items on a self-efficacy attitudinal scale from Bergin (1989) Scores of each item ranged from 0-10 with 10 being “completely confident.” Flow of Activities in Collecting Data Flow of Activities Example

Data Collection • Scientific and disciplined inquiry requires the collection, analysis, and interpretation of data • Data – the pieces of information that are collected to examine the research topic • Issues related to the collection of this information are the focus of this chapter

Data Collection • Terminology related to data • Constructs – abstractions that cannot be observed directly but are helpful when trying to explain behavior • Intelligence • Teacher effectiveness • Self esteem

Identify Data Options: Specify Variables • Independent Variables • Dependent Variables • Intervening Variables • Control • Moderating • Confounding

Identify Data Options: Operationalize Variables • Operational Definition: The specification of how the variable will be defined and measured • typically based on the literature • often found in reports under “definition of terms” • Sometimes the researcher must construct it

Some Times When Operational Definitions Would Be Helpful Figure 2.2 8

Which of the Following Definitions Are Operational? Page 34 1. As shown by enthusiasm in class 2. As judged by the student’s math teacher using a rating scale she developed 3. As measured by the “Math Interest” questionnaire 4. As shown by attention to math tasks in class 5. As reflected by achievement in mathematics 6. As indicated by records showing enrollment in mathematics electives 7. As shown by effort expended in class 8. As demonstrated by number of optional assignments completed 9. As demonstrated by reading math books outside class 10. As observed by teacher aides using the “Mathematics Interest” observation record 7

Data Collection • Data terminology (continued) • Operational definition – the ways by which constructs are observed and measured • Weschler IQ test • Virgilio Teacher Effectiveness Inventory • Tennessee Self-Concept Scale • Variable – a construct that has been operationalized and has two or more values

WHAT ARE DATA? • The term "data" refers to the kinds of information researchers obtain on the subjects of their research. • The term "instrumentation" refers to the entire process of collecting data in a research investigation.

KEY QUESTIONS • An important consideration in the choice of an instrument to be used in a research investigation is validity: • the extent to which results from it permit researchers to draw warranted conclusions about the characteristics of the individuals studied.

It involves not only the selection or design of the instruments but also the conditions under which the instruments will be administered. 1. Where? -- location 2. When? - - Time 3. How often?- - Frequency 4. Who? --administration of the instruments How you answer these questions may affect the data obtained! CONDITIONS

The data provided by any instrument may be affected by nay or all of the preceding considerations If administered incorrectly, disliked Noisy or inhospitable conditions Subjects are exhausted Every instrument if it is of any value must allow researchers to draw accurate conclusions about the capabilities or other characteristics of the people being studied Good Instruments?

VALIDITY, RELIABILITY, AND OBJECTIVITY • 1) Validity • An important consideration in the choice of an instrument to be used in a research investigation is validity: • the extent to which results from it permit researchers to draw warranted conclusions about the characteristics of the individuals studied.

Reliability and Objectivity • 2) A reliable instrument is one that gives consistent results. • 3) Whenever possible, researchers try to eliminate subjectivity from the judgments they make about the achievement, performance, or characteristics of subjects. That is, the researchers try to be objective.

USABILITY • Is it easy to use? • How long will it take to administer? • Are directions clear? • Is it appropriate for the ethnic group or other groups to whom it will be administered? • How easy is it to score? • To interpret the scores?

Practical Questions • How much does it cost? • Do equivalent forms exist? • Have any problems been reported? • Does Evidence of its reliability and validity exist? • Save time, energy and headaches!!!

Who Provides the Information • Research instruments can be classified in many ways. • Some of the more common are in terms of who provides the data, the method of data collection, who collects the data, and what kind of response they require from the subjects.

Data Obtained • Research data are data obtained by directly or indirectly assessing the subjects of the study. • Self-report data are data provided by the subjects of the study themselves. • Informant data are data provided by other people about the subjects of a study.

Researcher Instruments • Many types of researcher-completed instruments exist. • Some of the more commonly used are rating scales, interview schedules, tally sheets, flowcharts, performance checklists, anecdotal records, and time-and-motion logs.

Subject Instruments • The types of items or questions used in subject-completed instruments can take many forms, • but they all can be classified as either selection or supply items.

Subject Instruments • The types of items or questions used in subject-completed instruments can take many forms, but they all can be classified as either selection or supply items.

Subject Instruments • There are also many types of instruments that are completed by the subjects of a study rather than the researcher. • Some of the more commonly used of this type are questionnaires; self-checklists; attitude scales; personality inventories; achievement, aptitude, and performance tests; projective devices; and socimetric devices.

Subject Instruments (con.t) • Examples of selection items include true-false items, multiple-choice items, matching items, and interpretive exercises. Examples of supply items include short-answer items and essay • questions.

Where Did the Instruments come From? • 1) Find and administer a previously existing instrument of some sort, or • 2) administer an instrument the researcher personally developed or had developed by someone else • An excellent source for locating already available tests is the ERIC Clearinghouse on Assessment and Evaluation.

Data Collection • Measurement scales • Nominal – categories • Gender, ethnicity, etc. • Ordinal – ordered categories • Rank in class, order of finish, etc. • Interval – equal intervals • Test scores, attitude scores, etc. • Ratio – absolute zero • Time, height, weight, etc.

Four Types of Measurement Scales Figure 7.25 SCALE EXAMPLE Nominal Gender Ordinal Position in race Interval Temperature (in Fahrenheit) Ratio Money 50

An Ordinal Scale: The Winner of a Horse Race Figure 7.27 51

Data Collection • Types of variables • Categorical or continuous • Categorical variables reflect nominal scales • Continuous variables reflect ordinal, interval or ratio scales • Independent or dependent • Independent variables are the purported causes • Dependent variables are the purported effects

Measurement Instruments • Types of instruments (continued) • Affective (continued) • Scales used for responding to items on affective tests • Likert • Semantic differential • Thurstone • Guttman • Rating scales

Examples of Items from a Likert Scale Measuring Attitude toward Teacher Empowerment Figure 7.14 Instructions: Circle the choice after each statement that indicates your opinion. 1. All professors of education should be required to spend at least six months teaching at the elementary or secondary level every five years. Strongly Strongly agree Agree Undecided Disagree disagree (5) (4) (3) (2) (1) 2. Teachers’ unions should be abolished. Strongly Strongly agree Agree Undecided Disagree disagree (1) (2) (3) (4) (5) 3. All school administrators should be required by law to teach at least one class in a public school classroom every year. Strongly Strongly agree Agree Undecided Disagree disagree (5) (4) (3) (2) (1) 44

Example of the Semantic Differential Figure 7.15 45 Instructions: Listed below are several pairs of adjectives. Place a checkmark (a) on the line between each pair to indicate how you feel. Example Hockey: exciting :_____:_____:_____:_____:_____:_____:_____:_____: dull If you feel that hockey is very exciting, you would place a check in the first space next to the word “exciting.” If you feel that hockey is very dull, you would place a checkmark in the space nearest the word “dull.” If you are sort of undecided, you would place a checkmark in the middle space between the two words. Now rate each of the activities that follow [only one is listed]: Working with other students in small groups friendly :_____:_____:_____:_____:_____:_____:_____:_____: unfriendly happy :_____:_____:_____:_____:_____:_____:_____:_____: sad easy :_____:_____:_____:_____:_____:_____:_____:_____: hard fun :_____:_____:_____:_____:_____:_____:_____:_____: work hot :_____:_____:_____:_____:_____:_____:_____:_____: cold good :_____:_____:_____:_____:_____:_____:_____:_____: bad laugh :_____:_____:_____:_____:_____:_____:_____:_____: cry beautiful :_____:_____:_____:_____:_____:_____:_____:_____: ugly

Measurement Instruments • Issues for cognitive, aptitude, or affective tests • Bias – distortions of a respondent’s performance or responses based on ethnicity, race, gender, language, etc. • Responses to affective test items • Socially acceptable responses • Accuracy of responses • Response sets • Problems inherent in the use of self-report measures and the use of projective tests

Criterion-Referenced vs. Norm-ReferencedEvaluation Instruments Page 158 48 • Criterion-referenced: A student . . . • spelled every word in the weekly spelling list correctly. • solved at least 75 percent of the assigned problems. • achieved a score of at least 80 out of 100 on the final exam. • did at least 25 push-ups within a five-minute period. • read a minimum of one nonfiction book a week. • Norm-referenced: A student . . . • scored at the 50th percentile in his group. • scored above 90 percent of all the students in the class. • received a higher grade point average in English literature than any other student in the school. • ran faster than all but one other student on the team. • and one other in the class were the only ones to receive A’s on the midterm.

Selection of a Test • Designing you own tests • Get help from others with experience developing tests • Item writing guidelines • Avoid ambiguous and confusing wording and sentence structure • Use appropriate vocabulary • Write items that have only one correct answer • Give information about the nature of the desired answer • Do not provide clues to the correct answer • See Writing Multiple Choice Items

Selection of a Test • Test administration guidelines • Plan ahead • Be certain that there is consistency across testing sessions • Be familiar with any and all procedures necessary to administer a test

Identify Data Options: Select Scales of Measurement • Nominal (Categorical): categories that describe traits or characteristics participants can check • Ordinal: participants rank order a characteristic, trait or attribute

Identify Data Options: Select Scales of Measurement • Interval: provides “continuous” response possibilities to questions with assumed equal distance • Ratio: a scale with a true zero and equal distances among units

Record and Administer Data Collection: Locate or Develop an Instrument • Develop your own instrument • Locate an existing instrument • Modify an existing instrument

Record and Administer Data Collection: Obtain Reliable and Valid Data • Validity: the ability to draw meaningful and justifiable inferences from the scores about a sample or a population • Types of validity • Content (representative of all possible questions that could be asked) • Criterion-referenced (scores are a predictor of an outcome or criterion they are expected to predict • Construct (determination of the significance, meaning, purpose and use of the scores)

Record and Administer Data Collection: Develop Administrate Procedures for Data Collection • Develop standard written procedures for administering an instrument • Train researchers to collect observational data • Obtain permission to collect and use public documents • Respect individuals and sites during data gathering

Illustration of Types of Evidence of Validity Figure 8.1 52

Reliability and Validity Figure 8.2 53

Methods of Checking Validity and Reliability Table 8.2, page 180 54

More About Research: Threats to Internal Validity in Everyday Life Box 9A, page 199 55 • Consider the following commonly held beliefs: • Because “failure” often precedes “suicide,” it is therefore the cause of “suicide.” (probable history and mortality threat) • Boys are genetically more talented in mathematics than are girls. (probable subject attitude and location threats) • Girls are genetically more talented in language than are boys. (probable location and subject attitude threats) • Minority students are less academically able than students from the dominant culture. (probable subject characteristics, subject attitude, location, and instrumentation threats) • People on welfare are lazy. (probable subject characteristics, location, and history threats) • Schooling makes students rebellious. (probable maturation and history threats) • A policy of temporarily expelling students who don’t “behave” improves a school’s test scores. (probable mortality threat) • Indoctrination changes attitude. (probable testing threat) • So-called miracle drugs cure intellectual retardation. (probable regression threat) • Smoking marijuana leads eventually to using cocaine and heroin. (probable mortality threat)

Educational Research