Funding Audit Masterclass

1. Funding Audit Masterclass 7th December 2010

2. Funding Audit 2009/10Update Karl Bentley RSM Tenon

3. What we’ll be covering • Background • What we covered • What we found • Things to consider • Rumour control • The Four C’s

4. Background to the audit • Some 85 providers have been audited • Selection for the majority was on a sample basis, others based on risk or results of previous audits • We did 19 College audits and mentored PFA for a further six • Two qualified audits, one as the data did not reflect the funding, other was a technical qualification – no impact on funding but data had lots of weaknesses • Overall there were signs of improvement, especially those we have audited for the last two years

5. What we covered • The usual suspects • Eligibility • Attendance/Existence • GLH for unlisted aims • Fee Waiver • Disadvantage Uplift • NVQ • Distance Learning • Unitisation • Partnership • Other funding streams • DSAT • EMA and ALG

6. So what did we find this time? • Can anyone guess what the issues were this time round?

7. The Results • Some marked improvements in some areas • Eligibility was less of an issue this year • Some improvements on GLH • A51a was actually being completed this time

8. The Results • GLH is still proving to be a problem • Skills for life and Supported Learning • 19+ NVQ provision • Functional Skills has been a real problem this year GLH on ILR and register did not reconcile in a lot of cases Under and over claims View is that some Colleges treated them as key skills, forgetting that they were unlisted aims

9. The Results continued • Start, withdrawal and planned end dates • A51a still proving to be a problem – BTEC National Awards, Certificates and Diplomas were better, but still not right • Colleges had not seen the guidance issued for Construction Diplomas though • DSATs – Plenty of issues again, Early Completers report revealed some interesting results

10. A simple request • When the sample is sent we include a document call the ‘Prepared by Provider Report’ • Please read it! • It tells you exactly what you need to provide to us at the start of the audit – like on day one, not ready half way through the week • Please make sure your other funding streams reports are ready

11. Rumour Control • Not processing your 16-18 withdrawals before LR01 will not help to improve your learner numbers for future years • It will catch up with you and you’ll be having to explain why the numbers changed later in year • Will probably make you far more likely to be on the risk radar for SFA and YPLA • Audit simplification will only come with a simplification to the funding – we come at the end, not the start!

12. The Four C’s • Did anyone challenge the auditors this year? • Communicate, Check, Challenge and Complain • It works • We’re aware of one case where the auditors were instructed to remove a management letter point

13. DSAT: Things to consider Timetable your DSATs Don’t run every report every month Some will only need running towards the end of the year Prioritise your reports – think about the funding first! Personal pet hates… Learners way past their expected end date but still active Internally certificated qualifications should not be left as completed, awaiting result for a long time!

14. Beyond DSAT … drakelane.co.uk Presented by Ed Drake

15. Aims • To highlight how ‘cleaning’ ILR data via LIS and DSAT is only • part of the validation process • To stress the importance of setting your data standards higher • in order to approach audits with greater confidence

16. Starting point … • ILR data is “clean” according to LIS validation • … and … • ILR data is “clean” according to DSAT validation Inaccurate data Accurate data

17. How clean is “clean”? • “Clean” LIS and DSAT validated data may still reflect the • following: • Clearly incorrect learning aim references • Inconsistencies between 1st and 2nd year data • Significant funding under-claims • Significant funding over-claims • - Missing activity

18. Suggested remedial / further action 1. Write / run additional data validation reports • Sample check to ensure that the ILR data is representative • of actual activity

19. Suggested additional validation reports • Comparison of actual (delivered) GLH against ‘listed’ (funded) GLH

20. Suggested additional validation reports • Comparison of current year outturn against previous year outturn (including a range of like-for-like analyses)

21. Suggested additional validation reports • Comparison of current year outturn against previous year outturn (including a range of like-for-like analyses)

22. Suggested additional validation reports • Comparison of current year outturn against curriculum plan • Consistency of ILR data between years for programmes which run over more than one funding year – including: • Existence • Learning aim reference • Funding stream • Start and planned end dates • Guided learning hours • Remission status

23. Suggested additional validation reports • Additional funding under/over-claim reports – including in relation to: • 16-18 full time entitlement (‘enhanced’ funding) • Tuition fee remission • Disadvantage • Funding adjustments (A51a) • Improving / ‘fixing’ DSAT reports • Reconciling any 2 ILRs to understand / explain differences

24. Audit cycle When to undertake these data checks? … • Yesterday!! • Now!! • Ongoing Note: The days when ILR data could be ‘tidied-up’ by the end of the year have long gone

25. SCORE • Update to SCORE 2010 released yesterday • Additional 12 reports (eg. potential A51a triggers) • FREE SCORE 2009 software available for evaluation purposes ADaM • In-year monitoring of changes commences (post-LR01) • Update due by end of January (additional nesting reports)

26. Employer Responsive Timeliness Reports Ellie Fraizer, Data Quality & Assurance Manager 7th December 2010 Funding Audit Masterclass, Lakeside Centre - Birmingham

27. Why they are being introduced How they are being calculated A look at the reports When and where they will be available ER Timeliness Reports

28. Skills Funding Agency Guidance Note 3 Need to set timeliness standards for efficient and transparent financial reconciliation for Employer Responsive funding All starts must be reported within 2 reporting months All achievements and leavers/withdrawals must be reported within 3 reporting months ER Timeliness Reports – Why

29. EXAMPLE For a learner starting an aim in month X, a record for that start must be included in the return at the end of month X+1 (at the latest) Learner starts on 1st September 2010. This must be reported for the first time in the return for September (ER02) or October 2010 (ER03). That is, at the latest, in the return that closes on 4th November 2010 (ER03) ER Timeliness Reports – How

30. Reflections on methodology Matching: No unique reference for matching aims (A05 cannot be used). Aims treated as the same if they have a common: Provider Number (L01) Learner Reference Number (L03) Learning Aim Reference (A09) Learning Aim Start Date (A27) Consistent with methodology used for matching aims in TTG success rates ER Timeliness Reports – How

31. Reflections on methodology Instances where a single learner has more than one aim with same Learning Aim Reference (A09) and Learning Start Date (A27) are treated as a single aim Changes in A09, A27 and A40 will not be factored into the calculation Late accredited aims may be flagged as a late start Methodology and coding included in zip file ER Timeliness Reports – How

32. Released 1st December! Provider Gateway Document Exchange Provider Data Quality Reports, DQ-ER folder Report enquiries (methodology etc) and Gateway/Doc X access issues, please contact the Service Desk: servicedesk@thedataservice.org.uk ER Timeliness Reports – When & Where

33. Summary and conclusions Nick Linford

34. Avoiding the pitfalls of audit Rule 1: Understand the types of data quality Rule 2: Do not only prepare for audit Rule 3: Maintain data quality all the time Rule 4: Know what is important for you Rule 5: Ensure SMT and Governors understand how important you are Rule 6: Rule 5 keeps us all in a job and the college out of trouble 

35. Data quality Important to understand the types of data quality Validity (LIS error reports) Credibility (LIS warning reports, DSATs, Ofsted, ILR FAQs) Completeness (Data Dashboards and Annex P thresholds) Timeliness (ILR Hard Closes and new ER reports) Also understand that ILR data should not always be accurate…….. e.g. The learning aim is what was planned, not what was actually achieved.

36. It’s not only the funding auditor Skills Funding Agency Account Manager Ofsted (Jo Parkman) Local Authority & YPLA (soon to be EFA) The learners (see Personal Learning Record) And of course your college (LR01 withdrawals…) P.S. Framework for Excellence results published publically next week

37. Maintain data quality all the time It is no longer advisable to wait until the end of the year before cleaning the data prior to audit ~ Monthly ER timeliness report ~ Ofsted only need to give 2 weeks notice ~ Monitoring performance in-year (all the time) is a great way to drive up efficiencies Do you need to pay for internal auditors, and where you do are you being honest (and are they any good)?

38. What’s most important to you? Are all 66 DSAT reports critical to you? Biggest potential pitfalls ~ Unlisted functional skills (incl. embedding) ~ Use of the A51a discount field (esp. construction) ~ Eligibility for FL2 and FL3 entitlement ~ Transfers and learning aim changes ~ Guided Learning Hours (incl. listed aims) Note: If there has been an error it is ok to change it at any time

39. SFA Policy Statement highlights how important you are Higher Risk Providers & Persons and Agency Contracting (03/11/10) “a small number of providers have tarnished this otherwise positive relationship, and have achieved financial gain at the expense of both learners and the public purse, whether through ignorance or design.” Additional criteria for rejection of providers “previous activities that have resulted in significant repayment of funding (£100k or more)” “two or more instances where the Agency’s staff have audited the provision of the bidder and identified significant concerns resulting in a qualified opinion” ACTOR helps implement this

40. Nationally produced ILR audit reports ~ LIS errors~ LIS warnings~ OLDC validation (ULN and Employer Identifier)~ DSAT reports~ Data Credibility and Completeness (Dashboards, Annex P thresholds)~ ER Timeliness

41. Other ILR audit tools ~ SCORE and ADaM ~ Your own system validations and reports School 6th forms don’t know how lucky they are! School 6th forms don’t know how lucky they are!

42. School Success Rates YPLA have recently published school success rates following an FoI request from a journalist 23% unknown outcomes for Academies 16% unknown outcomes for School 6th Forms In total 160,000 exams taken where the outcomes were unknown Always nice to end a seminar on a high (AKA feeling smug!)

43. Any questions? Sign up for free updates at www.lsect.com