A Service of the Children’s Bureau, a Member of the T/TA Network

1. A Service of the Children’s Bureau, a Member of the T/TA Network NHSTES Symposium Testing a model for Transfer of Learning June 10, 2011

2. University at Albany University of Maryland University of Denver University of Southern Maine National Indian Child Welfare Association Fordham University Michigan State University University of Iowa University of Michigan Portland State University Children’s Bureau/ ACF/DHHS NCWWI Partners www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

3. NCWWI Purpose • Build the capacity of the nation’s child welfare workforce and improve outcomes for children, youth, and families through activities that support the development of child welfare leaders www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

4. Leadership Academy for Middle Managers (LAMM) • Facilitate the ability of middle managers to apply leadership skills for sustainable systems change to improve outcomes for children, youth, and families • Provide a comprehensive learning opportunity comprising of pre-training preparation, 5 days of leadership development in a residential setting, coaching before, during and after training, and post-training peer networking opportunities www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

5. Leadership Academy for Supervisors (LAS) Online, self-directed, interactive learning experience, using real-life examples • 5 asynchronous sessions/synchronous learning networks • Core curriculum of 30 contact hours • Specialized supplementary training modules www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

6. Leadership Academy Evaluation Design • Pre-post, follow-up design • Self-ratings of competencies • Understanding Performance • Learning Transfer Systems Inventory • Adapted from Holton, 1996 • Ratings of training, self-efficacy • Transfer of learning • Mixed-methods approach • Web-based survey, focus groups, 6-mo interviews www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

7. Hypothesized Transfer of Learning Model Individual Readiness Training Transferability • Content Validity • Instructional Design • Trainer Effectiveness • Self-Efficacy • Learner Readiness Learning Expected Return on Investment Post-training Leadership Skills • Transfer effort – Performance Expectations • Expected Positive Outcomes – Personal • Expected Positive Outcomes – Agency Motivation to Transfer • Peer Support • Supervisor Support • AgencyClimate for Change • Opportunity to Use Work Environment moderation Transfer Support • Coaching • Training/Assistance • Peer Networks External Events Transfer of Learning Implementation of Change Initiative 3, 6, 12-month Competencies

8. Motivation to Transfer Model Individual Readiness Training Transferability • Self-Efficacy • Learner Readiness • Content Validity • Instructional Design • Trainer Effectiveness Learning Expected Return on Investment Post-training Leadership Skills • Transfer effort – Performance Expectations • Expected Positive Outcomes – Personal • Expected Positive Outcomes – Agency Motivation to Transfer

9. Method for Analysis • Assigned survey items to constructs based on a priori theory and exploratory factor analysis • Constructed scores for each construct by averaging item scores • Analysis sample = 114 LAMM participants from 3 training cohorts • Used averaged item scores in path analysis • Tested hypothesized model • Revised model using modification indexes www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

10. Original Path Model Individual Readiness: Self-Efficacy Individual Readiness: Learner Readiness Training: Content Validity Training: Trainer Effectiveness Poor fit χ2(6)=38.01, p < .001 CFI=.805 RMSEA=.228 SRMR=.048 Expected ROI: Personal Positive Outcomes Leadership Skills (Post-Training) Leadership Skills (Pre-Training) Expected ROI Transfer Effort– Performance Expectations Motivation to Transfer

11. Revised Path Model Individual Readiness: Self-Efficacy Training: Content Validity Leadership Skills (Pre-Training) .20* Training: Trainer Effectiveness .17* .44*** .17* Leadership Skills (Post-Training) .29** .38** Excellent fit χ2(4)=1.527, p = .82 CFI=1.000 RMSEA<.001 SRMR=.012 Expected ROI Transfer Effort – Performance Expectations .30*** Motivation to Transfer

12. Modified Motivation to Transfer Model Individual Readiness Training Transferability • Self-Efficacy • Learner Readiness • Content Validity • Instructional Design • Trainer Effectiveness Learning Expected Return on Investment Post-training Leadership Skills • Transfer effort – Performance Expectations • Expected Positive Outcomes – Personal • Expected Positive Outcomes – Agency Motivation to Transfer

13. Challenges and next steps • Developing an effective measurement model to test the theoretical model • CFA as well as EFA • Gathering enough data to test the model using consistent measurement tools while remaining flexible in evaluation approach to capture changes in approach • Developing good measures of TOL • competencies at 3 months and 6 months • TOL items about implementing CI www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

14. Discussion Questions • How does this model align or diverge from current models of Transfer of Learning? • What utility does the model offer in terms of designing and evaluating child welfare training? • What are some innovative strategies that researchers are using to measure TOL that move beyond self-report? www.ncwwi.org A Service of the Children’s Bureau, a Member of the T/TA Network

15. Thank You www.ncwwi.orgA Service of the Children’s Bureau, a Member of the T/TA Network