What can we expect of Evaluating ICTs?

1. What can we expect of Evaluating ICTs? Larry Stillman, Centre for Community Networking Research Monash University, Australia ccnr.net < larrys@vicnet.net.au >

2. Just what do we mean by ICT networks anyway? • Geographically based? • Socially based (Wellman) • Virtual communities of interest • Defining the above helps to clarify/confuse our research questions and policy decisions • See Loader’s paper

3. Parsimony • We never have enough $, £, €! • We never have enough time! • We don’t have the skills etc • What is practical and based on good theory?

4. What is Evaluation? • A means of determining the value, merit, and worth of a program/project • A form of disciplined inquiry akin to research and policy analysis • A rigorous technique for knowledge discovery that joins together the qualitative and quantitative, pragmatically • Typically, a balance between the interests of the funder and the participants/subjects

5. Evaluation / R & D for ICT – Identified Needs • Practical community-level research methods • Building community and agency awareness and training regarding ICTs ie Capacity • Business planning skills • Collaboration among agencies, including neighbouring ICT and telecentre projects. • Roman, R. and R. D. Colle (2002). Themes and Issues in Telecentre Sustainability. (Accessed 22 July 2002.http://idpm.man.ac.uk/idpm/diwpf10.htm)

6. Action Research • A cycle of theory, discovery & research, analysis, reflection and new activity/questions • Ideally suited to new areas of activity • Uses the participants as partners • Can use qualitative and quantitative methods

7. Current Project • Working with Making the Network on their planning and management tools which have a strong participative action research component to add an evaluative component • Working with a large non-profit in Australia to trial it, and elsewhere • PROCESS with community organisations is a fundamental for capturing ‘fuzzy changes’ • The accountability aspect comes through the ‘Criteria of Worth’ tool • Outcome: an international tool kit for action-ICT research

8. Network Planning Source: Makingthenetwork.org

9. Dimensions of possible evaluative interest to ICT projects Line of Evidence/data source #1 (Quantitative)* Line of evidence/data source #2 (Quantitative)* Conclusion (value, merit and worth ) Planning /Resources + - ? Community-User Involvement + + + Community/Social Capital - + ? Management + + - Evaluation: Data and Analysis Matrix for Criteria of Worth 1

10. Evaluation: Data and Analysis Matrix for Criteria of Worth

11. Qualitative Methods • [Case studies of individuals or organisations] • Participant observation • Focus groups and open-ended discourse • Stories • Multimedia • Document review

12. Possibilities for a quantitative standard • Learning Communities: UNESCO’s Four Pillars of Learning • Learning to Know • Learning to Do • Learning to Be • Learning to Live Together • CTC Net survey tools • Social capital surveys: ICT as a particular form of social network • Social Capital consists of networksof social relations characterised by norms of trust and reciprocity” • The ‘bridging and bonding’ electronic network effects on individuals and virtual/ earth-based communities • See Clement and others for a large-scale proposal

13. Issues to be resolved in an action-research perspective • Resources and skills are major constraints in a community setting; this isn’t the same as funded academic social science • How to measure the virtual, especially the geographically dispersed • Use of cluster evaluation methods (Kellogg Foundation • Use the technology effectively, with caveats (‘the trap’ of over-reliance on complex sytems)

14. Sources • Making the Net Work materials (www.makingthenetwork.org) • Clement, A., L. Shade, et al. (2002). Toward an Evaluation Framework for Community Learning Networks.(Accessed: 11 November 2002.http://www.fis.utoronto.ca/research/iprp/) • Hurworth, R. (1996). "Qualitative Methodology. Some questions and answers about analysis of qualitative data in evaluation." Evaluation News and Comment5(2): 63-64. • Loader, B (2002) Whither Electronic Communities? Community informatics in principle and practice( http://www.ccnr.net/searchconf/loader.htm) • Miles, M. B. and A. M. Huberman (1988). Drawing Valid Meaning from Qualitative Data: Toward a Shared Craft. Qualitative Approaches to Evaluation in Education. D. M. Fetterman. NY, Praeger. • McConney, A., A. Rudd, et al. (2002). "Getting to the Bottom Line: A Method for Synthesizing Findings Within Mixed Method Program Evaluations." American Journal of Evaluation23(2): 121-140. • Stone, W. (2001). Measuring Social Capital. Towards a theoretically informed measurement framework for researching social capital in family and community life. Melbourne, Institute of Family Studies. • Wellman, B. (2001). Computer Networks as Social Networks.(Accessed: 14 September 293.http://www.chass.utoronto.ca/~wellman/publications/index.html) • Wellman, B., A. Quan Haase, et al. (2001). Does the Internet Increase, Decrease, or Supplement Social Capital? Social Networks, Participation, and Community Commitment.(Accessed: June 22 2001.http://www.chass.utoronto.ca/~wellman/publications/index.html)