1 / 19

Some perspectives on the importance of policy evaluation

Some perspectives on the importance of policy evaluation. Joost Bollens HIVA- K.U.Leuven. Evaluation of ALMP’s. Why evaluate ALMP’s ? How to measure effectiveness ? Some practical issues Unanswered questions. ALMP’s. Active Labour Market Policies Training for the unemployed ;

etta
Download Presentation

Some perspectives on the importance of policy evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Some perspectives on the importance of policy evaluation Joost BollensHIVA- K.U.Leuven Joost Bollens

  2. Evaluation of ALMP’s • WhyevaluateALMP’s? • Howtomeasureeffectiveness? • Some practical issues • Unansweredquestions Joost Bollens

  3. ALMP’s • Active LabourMarketPolicies • Training for the unemployed; • Private sector incentiveschemes (wage subsidies, start-upgrants,…); • Direct employment programmes; • Counseling, monitoring, job search assistance, sanctions; Joost Bollens

  4. Public expenditureonALMP’s Joost Bollens

  5. WhyevaluateALMP’s? • Active policies : beneficialeffects • Strong beliefs • Is thisreally the case?  Impact evaluation • Different programmes in one country • Allequallyeffective? Joost Bollens

  6. WhyevaluateALMP’s? • Evidencebasedpolicy : givenevaluationresults, decideto: • Continue the programme • Expand the programme • Restructureorredesign the programme • Abolish the programme • In the end : a matter of accountability Joost Bollens

  7. Evaluation • Processevaluation • How is the programmeimplemented?, Management quality?, Proper design?, Selectionprocesses?,… • Impact evaluation : effectiveness • Efficiency : costeffectiveness • Twoequallyeffective programmes may have a quite different cost per participant Joost Bollens

  8. Impact evaluation • Effectiveness : a lot of possibleoutcomes • % of participantsthatfind a job, % thatleaveunemployment, % thatfind a stable job orstableemployment,…, • % thatfind a decent job, effectsonhealth, psychologicaleffects, effectsonwell-being Joost Bollens

  9. Gross versus net effectiveness • Observedoutcome : effect of programmeparticipation + effect of factors outside the programme • Therefore, if we observethat 6 monthsafter finishing the programme e.g. 60% of the participants do have a job, thiscannotentirelybeattributedtoprogrammeparticipation : even without participating in the programme, someunemployedwould have found a job within 6 months Joost Bollens

  10. Net effectiveness • In order tofind the proper impact of the programme (the “valueadded”, or the “net effectiveness”, or the “impact”) , we have to correct the observedgross effect : Net effect = [Gross effect] - [the % of participantsthatwould have found a job even without participating] • Sinceparticipantscannot at the same time benon-participants, the red quantitycannotbeobserved (“counterfactual”) and must beestimated Joost Bollens

  11. Estimatingcounterfactuals • Non-experimentalapproaches(includingquasi-experiments) • Several, more orless sophisticated approaches • Basically : compose a comparisongroup of personswho are comparabletoparticipants, BUT whodidnotparticipate • Potentialweakness : comparabilitynot complete, e.g. dueto (self-) selectioneffects. Example : motivation Joost Bollens

  12. Estimatingcounterfactuals • Experimentalapproaches • Basically : take the group of personswho are willingtoparticipate in a programme, and randomlyassign half of themto a experimentalgroup, and half of themto a controlgroup • Experimentalgroup is allowedtoparticipate, controlgroupnot • Results of controlgroup serve as counterfactual • Advantage : betterguaranteeforcomparability, factors like e.g. motivationwillon average be the same in bothgroups • Strong resistance in a lot of countriestothisapproach : “unequaltreatment”. However, given the cost of ALMPs and the intrinsicuncertainty as totheireffects, thisshouldbereconsidered Joost Bollens

  13. Some practical issues • Planning helps • Plan before the introduction of a newprogramme • However, avoid the evaluation of a brand newprogramme • Radicallychanging (orabolishing) a programmebefore the end of the evaluationmakes the resultssomewhat irrelevant Joost Bollens

  14. Somepotentialconflicts • Time is onourside? • Policy makers, evaluation sponsors, programme administrators want immediatelyevaluationresults↔evaluatorwillinsistthat a thoroughevaluationtakes time • Impact evaluationresultsnecessarilywillonlybeavailablesome time afterparticipation • The resulting “this is old stuff”-argument is not per sevalid Joost Bollens

  15. Someotherpotentialconflicts • Different expectations: “usableinformation” (e.g. whatcanbeusedto fine tune the programme) ↔whereasevaluatorsoften are (somewhatmyopically?) in the first place interested in the validity of their impact estimates • Makeevaluation more usefulbyuncoveringrelationshipbetweeneffectiveness and design aspects Joost Bollens

  16. Someotherpotentialconflicts • Moreover, policy makers etc. onlyseemtobeinterested in impact estimateswhen these are positive, whilenegativeresultsoften are downplayedoroutrightneglected • (apparently?) contradictoryconclusions • Meta-analysiscan help Joost Bollens

  17. Remainingquestions 1 Is net effectivenessrelatedto … • …specificgroups? What does (doesn’t) workforwhom and why (not)? • …combination of severalpolicies? Order? • …timing of intervention ? • …labourmarketinstitutions? • … intensityor “dose” orduration? Joost Bollens

  18. Remainingquestions 2 Is net-effectiveness different between … • …public versus private provider? • …local versus nationalprogramme? • …favourable and unfavourable business cycleconditions? • …short run and long run ? (locking in?) • …sample in evaluationstudy, and futureparticipants? (externalvalidity) Joost Bollens

  19. Macro-effects of ALMP’s • Thusfar: effect onparticipants • Butalso : • Effect onnon-participants? Substitution, displacement, deadweight loss,….; • General equilibriumeffects • Effectsonemployment, unemployment, productivity, matchingeffectiveness, …. • Very important, yet a lot of uncertainty Joost Bollens

More Related