Auditing 101: A primer in foundations of conducting for-cause and not -for-cause audits

1. Auditing 101: A primer in foundations of conducting for-cause and not-for-cause audits Terry VandenBosch, PhD, RN, CIP, CCRP Managing Director Office of Human Research Compliance Review University of Michigan

2. Outline Concepts from Internal Auditing Profession Systematic Audit Process Reporting, Documentation and Record Retention Evaluation & Metrics Knowledge, skills and credentials

3. What is the State of Our Field? Emerging field Newer enterprise for Institutions and Academia Models from FDA biomonitoring & industry sponsor monitoring The nature, timing and extent of procedures are not evidence-based and not standardized We don’t know each other, no national group

4. Activities IRB Review-Say what you will do or have done Audit-Show what you will do or have done Routine, not-for-cause Directed, for-cause

5. Unlike IRB -- No Regulations on Audit PROCESS! NOT an FDA Biomedical Inspection!

6. A Definition of Human Subjects Protections Internal QA/QI Audit Internal HRPP review is an independent, objective, assurance and consulting activity designed to add value and improve an organization’shuman subjects research compliance program. It helps investigators and the organization to accomplish their objectives of human subjects participant protections by bringing a systematic, disciplined approach to evaluate, to verify and to improve those protections as appropriate. Adapted from IIA, 1999

7. Important Concepts Independent of research operations Objective-impartial, unbiased judgments Assurance uses systematic, disciplined approach to evaluate, to verity and to improve Consulting and Education Overall adds value & improves Compliance

8. The Three Foundations of Effective Internal Audit Services D U E C A R E Effective Internal Audit Services O B J E C T I V I T Y C O M P E T E N C E

9. Systematic, Disciplined Approach • Planning and preparation • Performing the onsite engagement • Sufficient, reliable, relevant & useful information • Appropriate analysis and evaluation • Condition-”what is” • Criteria-”what should be” • Consequences-”so what”, analyze risks • Causes-“root cause why”, Determines preventive actions • Record relevant information to support conclusions (workpapers) • Communicating • Monitoring Progress • Evaluating • Record Keeping & Retention

10. Symptom Versus Root Cause Remediation PROCESS IMPROVEMENT FUNDAMENTAL SOLUTION SYMPTON #1 REMEDIATION SYMPTON #2 REMEDIATION ROOT CAUSE ANALYSIS SYMPTON #1 SYMPTON #2 • Source: “Using System Archetypes to Identify Failure Patterns in Acquisition” by Diane Gibson, Linda Levine, and William Novak (Carnegie Mellon University Software Engineering Institute). Cited in Chong, E. E. (2010). Overcoming Checkbox Compliance. Internal Auditor, 64(6), 57.

11. Proportions of Research in the Portfolio Behavioral Science Portfolio Health Science Portfolio Biomedical Science Portfolio

12. Research Risks • Biomedical & Health • Social Sciences • Confidentiality • Discrimination/economic • Investigator initiated • Federal funding • Industry sponsor (monitored studies)?

13. Types of Routine, Not-for-Cause • Decide on: • “Unit” of analysis • Focused vs Comprehensive • Timing, e.g. enrollment • Audit/adherence to GCP • Selection: Random/judgment • Special focus • e.g., two parent signatures for studies approved under 45 CFR 46.406 • E.g., NSR devices • IRB request • Study start up

14. Directed, For-Cause Is there serious or continuing noncompliance? Providing data for internal QA/QI Providing a report for federal agencies

15. Faculty Acceptance • Faculty Acceptance • Service Philosophy • Enthusiasm (love or passion) for research • Transparency • Communication • Tailored Education • Objective/Independent • Authority clear • Reporting relationship • Institutional Official • IRB

16. SOPs Not-for-Cause • Annual risk assessment and compliance review plan • How many? • Resources & risk • Notification and Scheduling • Review Methods • Review Observations that may be serious or continuing noncompliance made during a RRERR (escalation plan) • Post-review activities for RRERRS

17. Methods • Not-for-Cause-”stop & go” • For-Cause-”needle in a haystack” • Observation • Interview/discussion • Records audit • Paper • Electronic (Records, charts, IRB) • Phone/email/Internet

18. Preparation • Notify Investigator • What should you tell them? • Who else to notify • Ask number of subjects enrolled • Space to work • Schedule time • Interview • onsite record review • How much investigator time? • Prepare tools • Running identifier for each page • Page 1 of __ • Auditor signature & date

19. Preparation • Study protocol • IRB concerns or issues during approval • Informed consent • Other attachments, e.g. surveys • Risk level • Confidentiality protections • Amendments • Adverse events • FDA regulated or federal funding • For-Cause • Presenting issue • Organizational policies and procedures • Due process • Arrange for Witness

20. Engagement-Opening Phase • Opening discussion • Small talk (Phatic talk) • Introduction to auditor & audit process steps • Relationship to IRB • Authority of office • Purpose of the audit • Describe type of review • For-Cause • Not-for-cause

21. Engagement-Working Phase • Discussion • PI discuss challenges in the study • PI oversight (Other oversight in for-cause) • Questions about adherence to IRB approved protocol • Confidentiality protections • Tailored Education • Record Review • Space to work • Orientation to organization of records • Access to subject source documents • Access to research records

22. Engagement-Working Phase • Selection of study records • At least 10% (more in for-cause) • Random, systematic, judgment sample • Organized auditing tools • Access to medical records as appropriate • Study team/PI availability for questions • Keep within a time frame...unless...

23. Engagement-Closing Phase • Positive aspects of review • Summary of any noncompliance • Explore causes of noncompliance • “Why” or “cause” of noncompliance determines preventive actions • Propose corrective action/preventive action to be confirmed by IRB • May need to communicate before final report, i.e., additional items, IRB perspective • Discuss next steps in review

24. Report and Follow up • Work collaboratively with the IRB • Work with compliance/IO in for-cause • Boundaries of authority for follow up • Describe corrective actions • Describe preventive actions (Root Cause Analysis) • study team not-for-cause • Other HRPP units/depts in for-cause • Due professional care

25. Report of Observations • “Observation” vs “Finding” • Language examples • Report • Succinct, clear, constructive, fact-based, timely, accurate • Report templates • Not-for-cause • For-cause • Background • Timeline • Scope • Observations including root cause analysis • Conclusions • Recommendations (corrective actions/preventive actions) • Responses

26. Report of Observations (cont’d) • Study summary/abstract • Scope of review • What was reviewed? Who attended discussions? • Could your findings be replicated? Do the data support the conclusions in for-cause? • Current study status • Narrative of observations • Exemplary practice • What done correctly • Any “observations” • May send PI draft of report for feedback on “facts”

27. Language Findings vsobservations Violations vsdeviations Assist you with implementing human subjects protections Noncompliance recommendations vs corrective actions “Human subjects protections” from whom is the subject being protected? Routine review vscompliance assessment Audit vs review Deficiencies vs corrective actions

28. Follow Up/Monitoring Progress • PI/study team follow up or auditor? • Onsite follow up by auditor or other? • Close out memo/letter when progress completed • For-cause • May have ongoing CAPA plan for HRPP units

29. Record Retention & Storage SOP-define number of years retention Define what is stored

30. Start Up Education & Assurance • AFTER IRB Approval • BEFORE subjects enrolled • Education • Issues • Is noncompliance possible? • Day-to-day study management • CRF/Data collection instruments • Study tools • IRB collaboration • But the IRB just approved my study • IRB issues

31. Start Up Education & Assurance • Report • Start up review completed • Any corrective actions? • Best wishes in their research program • Follow Up • Problems-later follow up visits • Suspensions possible? • Follow up until corrective actions completed?

32. Concepts of PI Audit Evaluation • The review process • Time reasonable • Clear communications • Convenient appointments • OHRCR staff • Respectful & professional • Timely response • Understand study unique aspects & challenges

33. PI Evaluation of Audit • Your audit experience • Clearly describe practices, policies, guidance & regulations • Additional knowledge • Satisfaction with audit process • Usefulness of audit to other faculty

34. Tracking Metrics • How many audits? • A function of resources & methods • Department level data • Risk level • Number of enrolled subjects • Department • Overall accomplishments • Qualitative • Metrics of observations