Group-Level Measurement

1. 1 Group-Level Measurement Katherine Klein University of Pennsylvania kleink@wharton.upenn.edu CARMA Presentation February 2007

2. 2 Why Group-Level Measurement? Burgeoning of multilevel theory and research in last 25 years Great progress in conceptualizing and measuring group-level constructs Especially shared constructs Continuing challenges and opportunities Especially regarding configural constructs

3. 3 A Few Terms and Assumptions I�ll refer to groups but much or all of what I say will apply as well to organizations, departments, stores, etc. I�ll focus on the creation and use of original survey measures to assess group constructs. I�ll address statistical issues in passing only. But see past CARMA presenters including James LeBreton, Gilad Chen, Paul Bliese, Dan Brass, Steve Borgatti, and others

4. 4 Roadmap Fundamentals: Theory First Construct Types: Global, Shared, and Configural Constructs Practicalities and Technicalities Survey Wording Sampling Qualitative Groundwork Single-source Bias Justifying Aggregation Opportunities and Challenges The Configuration of Diversity Social Network Analysis

5. 5 Fundamentals: Theory First Constructs are our building blocks in developing and in testing theory. High quality measures are construct valid. The development of construct valid measures thus begins with careful construct definition. Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000): Global, shared, or configural.

6. 6 Global Constructs Relatively objective, easily observable, descriptive group characteristics. Originate and are manifest at the group level. Examples: Group function, size, or location. No meaningful within-group variability. Measurement is generally straightforward.

7. 7 Shared Constructs Group characteristics that are common to group members Originate in group members� attitudes, perceptions, cognitions, or behaviors Which converge as a function of attraction, selection, socialization, leadership, shared experience, and interaction. Within-group variability predicted to be low. Examples: Group climate, norms, leader style. Measurement challenges are well understood.

8. 8 Configural Group-Level Constructs Group characteristics that describe the array, pattern, dispersion, or variability within a group. Originate in group member characteristics (e.g., demographics, behaviors, personality, attitudes) But no assumption or prediction of convergence. Examples: Rates, diversity, fault-lines, social networks, team mental models, team star or weakest member. Measurement challenges are less well understood.

9. 9 A Related Framework: Chan�s (1988) Composition Typology Shared Constructs Direct consensus models (e.g., group norms) Referent shift models (e.g., team efficacy) Configural Constructs Dispersion model (e.g., climate strength) Additive models (e.g., mean group member IQ) Multilevel, Homologous Models Process model (e.g., efficacy-performance relationship)

10. 10 Construct Definition Complexities:An Example: Shared Leadership Shared leadership �A dynamic, interactive influence process among individuals in work groups in which the objective is to lead one another to the achievement of group goals� [It] involves peer, or lateral, influence and at other times involves upward or downward hierarchical influence� Conger & Pearce, 2003, p. 286 Is this a shared construct, or a configural construct, or � ?

11. 11 Construct Definition Complexities:An Example: Shared Leadership Well, how would you measure it? Shared team leadership as a shared construct �Team members share in the leadership of this team.� �Many team members provide guidance and direction for other team members.� Shared team leadership as a configural construct (network density): �To what extent do you consider _____ an informal leader of the team?�

12. 12 Construct Definition Complexities:An Example: Shared Leadership Calling it a �referent shift� construct is not the answer. Referent shift is a measurement strategy, not a construct type Shifting the referent in an unthinking manner can be quite problematic: The members of my team� �Express confidence that we will achieve our goals� �Will recommend that I am compensated more if I perform well� �Are friendly and approachable� �Rule with an iron hand�

13. 13 A Quick Recap Theory first: Define and explain the nature of your group-level constructs. Is it a clearly objective description of the group? If yes, a global construct. Do you expect within-group agreement? If yes, a shared construct. Does it describe the group in terms of the pattern or array of group members on a common attribute? If yes, a configural construct.

14. 14 Now What? Having defined your constructs, the goal is to create measures that: Are construct valid Show homogeneity within (shared constructs) Show variability between (all group-level constructs) Practicalities and technicalities Survey wording Sampling Qualitative groundwork Minimizing single-source bias Testing for aggregation

15. 15 Survey Wording:Global Constructs Draw attention to objective descriptions of each group. Gather data from experts and observers (SMEs) who can provide valid information about the groups in question. No need to gather data from individual respondents within groups Use language that fits your sample.

16. 16 Survey Wording: Shared Constructs Draw attention to shared group characteristics Use a group referent rather than individual referent to enhance: Within group agreement Between group variability Predictive validity Gather data from individual respondents so within-group agreement can be assessed. Actual consensus methods (discussion prior to group survey completion) work well but are labor-intensive.

17. 17 Survey Wording:Configural Constructs Draw attention to individual group member characteristics by using an individual referent. Gather data from experts and observers (SMEs) who can provide valid information regarding individual group members, or gather data from individual respondents within groups. The challenge is perhaps less in the survey wording than in operationalizing the array or pattern of interest.

18. 18 Sampling Substantial between-group variability is essential. Seek samples in which groups vary considerably on the constructs of interest Whether they are global, shared, or configural. Statistical power reflects both: Group sample size (n of groups) Within-group sample size When group size is large (number of respondents per group), measures of shared constructs are more reliable. More research needed on power in multilevel analyses.

19. 19 Qualitative Groundwork The survey wording and sampling guidelines seem fairly obvious and easy, but � Check your assumptions in the field prior to survey data collection. Are you measuring the right �groups�? Example: Grocery stores or departments? Is there meaningful between-group variability? Example: Fast food chain Are you measuring the right variables, and not too many of them? Beware the blob.

20. 20 Single-Source Bias Group-level correlations between measures of shared group constructs may be disturbingly high. Examples: Transformational and transactional leadership Task, emotional, and procedural conflict Aggregation does not �average away� response biases. Rather, group members may share response biases Halo, logical consistency, social desirability Response bias may be particularly influential when respondents must make subtle distinctions among constructs.

21. 21 Single-Source Bias:Beating the Blob Survey measures Choose and measure truly distinct constructs Use different survey response formats Survey design Keep survey items measuring distinct constructs separate. Help respondents recognize the distinction between leadership types, or conflict types, for example.

22. 22 Single-Source Bias:Beating the Blob Survey analysis Randomly split the within-group sample of respondents during data analysis. All receive the same survey, but half provide IV and the other half provide the DV for analyses Survey administration Randomly split the within-group sample of respondents during data administration. Respondents receive distinctive surveys. Half receive the IV survey and the other half receive the DV survey.

23. 23 A Quick Recap Having Defined our constructs Written our survey items Conducted qualitative groundwork Sampled appropriately Taken steps to reduce single source bias We�re almost ready for hypothesis testing But first: We need to justify aggregation

24. 24 Justifying Aggregation Why is this essential? In the case of shared constructs, our very construct definitions rest on assumptions regarding within- and between-group variability. If our assumptions are wrong, our construct �theories,� our measures, and/or our sample are flawed and so are our conclusions. So, test both: Within group agreement The construct is supposed to be shared, but is it really? Between group variability (reliability) Groups are expected to differ significantly, but do they really?

25. 25 Justifying Aggregation: rwg(j) Developed by James. Demaree, & Wolf (1984) Assesses agreement in one group at a time. Compares actual to expected variance. Answers the question: How much do members of each group agree in their responses to this item (or this scale)? Highly negatively correlated with the within group standard deviation Valid values range from 0 to 1 Rule of thumb: rwg(j) of .70 or higher is acceptable

26. 26 Justifying Aggregation: rwg Common to report average or median rwg(j) for each group for each variable: If rwg(j) is below .70 for one or more groups, check: Does the group have low rwg(j) values on several variables? Do many groups have low rwg(j) values on this variable? Remember: rwg(j) indicates within-group agreement, not between-group variability. Beware: When variance in a group exceeds expected variance, out of range rwg(j) result. Random-group re-sampling provides a useful alternative. James, Demaree, & Wolf (1984) Schmidt & Hunter (1989) Kozlowski & Hattrup (1992) James, Demaree, & Wolfe (1993) Lindell & Brandt (1997) Random-group re-sampling provides a useful alternative. James, Demaree, & Wolf (1984) Schmidt & Hunter (1989) Kozlowski & Hattrup (1992) James, Demaree, & Wolfe (1993) Lindell & Brandt (1997)

27. 27 Justifying Aggregation: h2 Assesses between-group variance relative to total variance, across the entire sample. Based on a one-way ANOVA Answers the question: To what extent is variability in the measure predictable from group membership? The F-test provides a test of significance The larger the sample of individuals, the more likely eta2 is to be significant. Beware: h2 may be inflated when group sizes are small (under 25 individuals per group) But, this is an easy way to begin tests of aggregation

28. 28 Justifying Aggregation: ICC(1) Assesses between-group variance relative to total variance Based on a one-way ANOVA Answers the question: To what extent is variability in the measure predictable from group membership? The F-test provides a test of significance Based on h2 but controls for the number of predictors relative to the total sample size, so ICC(1) is not biased by group size.

29. 29 Justifying Aggregation: ICC(2) Assesses the reliability of the group means (i.e., between-group variance) in a sample, based on ICC (1) and group size. Answers the question: How reliable are between-group differences on the measure? Reflects ICC(1) and within-group sample size Example: If ICC(1) = .20 and: Mean group size is 5, expected ICC(2) = .56 Mean group size is 20, expected ICC(2) = .71

30. 30 Justifying Aggregation: An Example

31. 31 A Quick Recap The hope is that we have successfully: Defined our constructs. Written our survey items. Conducted qualitative groundwork. Collected data from a large sample of groups. Taken steps to reduce single source bias. Justified aggregation. And moved on to test our hypotheses. So, what remains?

32. 32 Opportunities and Challenges:The Configuration of Diversity Configural constructs describe the array, pattern, dispersion, or variability within a group. The easy example is diversity Demographic diversity Climate strength But even the easy example isn�t so easy: What is the definition of diversity? And how should it be measured?

33. 33 The Configuration of Diversity A starting definition of diversity: The distribution of differences among the members of a group with respect to an attribute, X, such as age, ethnicity, conscientiousness, positive affect or pay. Okay, but what�s maximum diversity? Which team has maximum age diversity? 20, 20, 20, 70, 70, 70 20, 30, 40, 50, 60, 70 20, 20, 20, 20, 20, 70 20, 70, 70, 70, 70, 70

34. 34 The Configuration of Diversity Diversity isn�t one thing. It�s three things: Separation, Variety, or Disparity The three types differ in: Meaning or substance Pattern or shape Likely consequences Appropriate operationalization Blurring across these distinctions leads to fuzzy theory, misguided operationalizations, and potentially invalid research conclusions

35. 35 The Configuration of DiversityExample: Three Research Teams Team S Members differ in their view of qualitative research. Half of the team members respect it, half don�t. Team V Members differ in their discipline. 1 psychologist, 1 sociologist, 1 anthropologist, etc. Team D Members differ in their rank 1 senior professor, others are incoming graduate students.

36. 36 Diversity as Separation Differences in group members� position, attitude, or opinion along a continuum Min: Every member has the same opinion Max: Two polarized extreme factions Theory: Similarity-attraction Operationalization: Standard deviation

37. 37 Diversity as Variety Differences in kind or category Min: Every member is the same type Max: Each group member is a different type Theory: Requisite variety, cognitive resource heterogeneity Operationalization: Blau�s index of categorical differences

38. 38 Diversity as Disparity Differences in concentration or proportion of valued assets or resources Min: Every member has an equal portion of the resource Max: One member is �rich� and all others are �impoverished� Note: Disparity is asymmetric Theory: Inequality, relative deprivation, tournament compensation Operationalization: Coefficient of variation (SD/Mean)

39. 39 The Configuration of Diversity:A Recap Theory first Separation is about position, attitude, or opinion At maximum: Polarized factions Variety is about knowledge or information. At maximum: One of a kind Disparity is about resources or power. At maximum: One towers over others Operationalize accordingly The coefficient of variation is not a default or catch-all

40. 40 Opportunities and Challenges: Social Network Analysis Multilevel analysis and social network analysis have developed along separate paths. Rich opportunities for cross-fertilization. Social network analysis provides a means to conceptualize and operationalize configural constructs. Illuminating the pattern or array of interpersonal ties within a group

41. 41 Opportunities and Challenges: Social Network Analysis Many of our shared constructs appear to rest on tacit, often fuzzy, assumptions about interpersonal ties with groups. Examples: Cohesion, communication, coordination, knowledge sharing, shared leadership, conflict But we know little about the configuration of interpersonal ties � the structures � that underlie our shared constructs and measures.

42. 42 An Example: Social Network Analysis and Shared Team Conflict When teams report high task or emotional conflict, what is the structure of interpersonal ties within the team? As a starting point: How dense are positive (advice) ties? How dense are negative (difficulty) ties?

43. 43 An Example: Social Network Analysis and Shared Team Conflict Task and emotional conflict: The blob r = .83 Advice density and negative tie density: More weakly correlated r = -.36 Task conflict (mean task and emotional conflict), advice density, and negative tie density Team Conflict and Advice Density: r = -.47 Team Conflict and Difficulty Density r = .40

44. 44 Negative Tiesin a Low Conflict Team