Measuring IT Success

1. Measuring IT Success Reza Torkzadeh Department of MIS University of Nevada Las Vegas www.unlv.edu/faculty/reza/ Organizational Systems Research Association 21st Annual Conference Las Vegas, Nevada

2. Measuring IT Success • Success paradigms • Success measures • Usefulness • Satisfaction • Impact • Usage • E-commerce success measures • Theory and practice issues

3. Design vs outcome perspective There are at least two perspectives on evaluating information systems: • Design perspective - has a strong tradition in the MIS field, calls for evaluating information systems relative to design specifications or user needs. • Outcome perspective - calls for performance-related evaluations that focus on outcomes.

4. Design vs outcome perspective • The measurement of IT success has progressed from ‘design’ paradigm to an ‘outcome’ paradigm. • As we move from user satisfaction to downstream success measures, evaluation must look at outcomes rather than design intention. • User acceptance, perceived usefulness, and ease of use are appropriate criteria for evaluating developmental progress.

5. System to Value Chain

6. Alternative theory base • The innovation process model provides a framework for describing stage dependent information technology success criteria and suggests alternative theory bases for developing success measures. Cooper and Zmud Mgmt. Sc. 36(2) 1990

7. Changing Standards for Evaluating Info. Systems Perceived usefulness User satisfaction Usage time Usage pattern/impact Process Model Initiation Adoption Adaptation Acceptance Routine Use Infusion Innovation Process Model

8. Perceived usefulness • Usefulness • e.g., work more quickly, job performance, increase productivity, effectiveness • Ease of use • e.g., easy to learn, controllable, clear & understandable, flexible Davis, MIS Quarterly 13(3)

9. End user computing satisfaction • Content • Accuracy • Ease of use • Format • Timeliness Doll & Torkzadeh, MIS Quarterly 12(2)

10. IT impact on work • Task productivity • Management control • Customer satisfaction • Task innovation Torkzadeh & Doll, Omega 27, 1999 • Task learning?

11. IT impact • When we evaluate the impact as people use technology, we are not directly assessing the technology or the people. Rather, we are evaluating a complex socio-technical phenomenon defined by the interaction of people and technology in a social and organizational context. This unpredictable nature of the impact construct enhances its theoretical significance for both domains of MIS and social sciences.

12. IT impact • A conception of technology’s impact that is limited to productivity and/or management control is rooted in an industrial paradigm that ignores organizationally relevant impacts essential to the success and survival of modern organizations.

13. IT impact • Modern management perspective on IT must go beyond ‘routine use’ and explore ways that have substantial implications for the nature of work, productivity, learning, and diffusion of management functions throughout work force.

14. E-commerce success • A five-factor 21-item instrument that measures means objectives in terms of: • Internet product choice • Online payment • Internet vendor trust • Shopping travel • Internet shipping errors

15. E-commerce success • A four-factor 16-item instrument that measures fundamental objectives in terms of: • Internet shopping convenience Internet ecology • Internet customer relation • Internet product value Torkzadeh & Dhillon ISR, 13(2)

16. User evaluation • “What is needed for user evaluations to be an effective measure of information technology success is the identification of some specific user evaluation construct, defined within a theoretical perspective that can usefully link underlying systems to their relevant impacts.” Goodhue, Mgmt. Sc. 41, 1995

17. System-use • In Labor and Monopoly Capital, Braverman argues that the widespread introduction of information technology in manufacturing, services, and white collar work has improved productivity and management control, but has not changed the labor process itself.

18. System-use • Braverman sees information technology being used to facilitate the vertical specialization of work with management/staff • planning the employee’s work, • monitoring performance, and • taking corrective action.

19. System-use • Using a post-industrial perspective, Hirschhorn argues that information technology is used in new ways that have decisive implications on the general texture of life. Worker skill is not the skill of executing, but the skill of problem solving and the ability to learn and innovate. Work becomes heavily dependent on information processing, increasingly abstract, and mediated by sensing mechanisms.

20. System-use • In studying the impact of IT, we cannot just study what workers do, but rather how prepared they are to do what they might have to do. As Hirschhorn argues, the overall impact of technology is to diffuse the management functions. Empowered workers can use information technology to solve problems in innovative ways, plan their own work, and monitor/control their own performance. We must re-think the old organizational designs.

21. System-use • Zuboff distinguishes between the deskilling impact of ‘automating’ and enhanced cognitive skills (sense making, inferential reasoning, systemic thinking) needed in an ‘informated’ environment.

22. System-use • Weick argues that information technology utilization requires ongoing structuring and sense making. Work is largely cognitive and time alone is not a good measure of either the amount of work done or the extent of computer usage.

23. System-use constructs • Decision support • Problem solving • Decision rationalization • Work integration • Horizontal integration • Vertical integration • Customer service Doll & Torkzadeh , I&M, 33, 1998

24. Theory and Practice • The productivity of substantive research activities depends upon efforts to improve theory and measurement development • No instrument should be used without questioning the procedures used to develop it and the appropriateness of the measure for the research being examined.

25. Theory and Practice • As an applied discipline, the development of information system measurement is impaired by confusion concerning the opportunities and challenges of: • relying upon other disciplines for theory development, • modifying such theories to fit the IT research domain, and • developing original IT theory.

26. Theory and Practice • Measurement development can occur in one of three contexts: • strong theory, • no theory, and • weak theory.

27. Theory and Practice • It is important to identify underlying assumptions concerning the opportunities and challenges posed by borrowing theory from other disciplines or conceptualizing original theory. • These underlying assumptions frame individual thinking on whether and how to develop IT theories.

28. Theory and Practice • Important theoretical questions that guide instrument development: • What is the domain of MIS research? • What is the purpose of an instrument? Is it to evaluate the effectiveness or value of an application or to predict attitudes/behaviors vis-a-vis an application? • To what extent is the attitude-behavior research tradition of social and cognitive psychology applicable to MIS research and issues important to managers?

29. Theory and Practice • What Theory • The attitude-behavior literature evolved out of research domains in social and cognitive psychology that may be quite different than those experienced by the MIS research community. • The research in social and cognitive psychology contains a rich variety of behavioral phenomena and often focuses on emotionally charged issues (e.g., capital punishment, birth control prejudice, etc.)

30. Theory and Practice • Thus, attitude research in this domain emphasizes the affective rather than the cognitive (e.g., belief) dimension of attitudes. • The bulk of attitude research focuses primarily on affect rather than cognitive and behavioral dimensions. • This emphasis on emotional issues of attitudes suggests the need for caution when borrowing concepts and techniques for use in MIS.

31. Theory and Practice • How much do we want evaluations of information systems to be based on affective (emotional) vs. cognitive responses from users? • Moreover, research results have been disappointing. • Simple models of the attitude-behavior relationship are yielding to more complex models of the conditions under which attitudes can predict behavior.