Application Auditing Scope, Approach, & Execution

1. Risk-based Application AuditingScope, Approach, & Execution January 2009 Michael Kirk, CIA, CISA

2. Introduction Mike Kirk, CIA, CISA • Application experience includes: Oracle, PeopleSoft, JD Edwards, and industry-specific and customized applications for healthcare, insurance, energy/utility, manufacturing, construction, and environmental industries. • Board member of the Central Ohio Chapter of ISACA

3. Introduction JD Edwards

4. Overview Risk Considerations for Application Auditing • Defining Your Scope • Developing Your Approach • Execution • Q & A

5. Where do you start? Scope Audit Methodology • Understand the environment • Understand business & technical processes • Assess risks & controls • Develop improvements

6. Defining Your Scope • Understand the business environment the application supports Develop a complete understanding of the business process flow (inputs, processing, & outputs), and, the data flow, including interfaces

7. Defining Your Scope • Understand the application’s technical environment • IT control environment • “Off-the-shelf” or customized • Legacy or web-based • Housed internally or external provider • Developed in-house or 3PP • User population • Dispersion of users

8. Defining Your Scope 1 Item out-of-stock? Confirm customer order Notify shipping Send info to Acct A Start Critical Process Maps Data-Flow/Interface Diagrams 3 2 Confirm receipt Close books Log revenue Create invoice A End Information System Critical Risk / Control Points # • Understand the business process and technical environment • Assess business information risks

9. Defining Your Scope Business & Information Risks: • Regulatory compliance • F/S integrity • PCI-related • Integration of an acquisition • Data privacy • Integrity of the application • Process control validation

10. Lessons Learned on Scope Critical dependencies: personnel, technical, external providers [BCP] “Canned” still seems to get modified Don’t forget spreadsheets & report-writers! Leverage existing flow diagrams Invest the time to understand the process flow... Adds value to your internal client You may find that you are now the expert!

11. Developing Your Approach Top-down Approach: Auditing around the application… Information Technology General Controls (ITGCs) Bottom-up Approach: Auditing the insides… Application Controls Testing

12. Top-down Approach Relationship between ITGCs and application controls • Development • Change Management • Security Administration • Operations

13. Top-down Approach Boundaries of Business, General, and Application Controls Source: COBIT 4.1 Framework

14. Top-down Approach Development AI2 Acquire and Maintain Application Software AI2.7 Development of Application Software • Ensure that automated functionality is developed in accordance with design specifications, development and documentation standards, QA requirements, and approval standards. Source: COBIT 4.1 Framework

15. Top-down Approach Change Management AI6 Manage Changes AI6.1 Change Standards and Procedures • Set up formal change management procedures to handle all requests (including maintenance and patches) for changes to applications, procedures, processes, system and service parameters, and the underlying platforms in a standardised manner. Source: COBIT 4.1 Framework

16. Top-down Approach Security Administration AI2 Acquire and Maintain Application Software AI2.4 Application Security and Availability • Address application security and availability requirements in response to identified risks and in line with the organisation’s data classification, information architecture, information security architecture and risk tolerance. Source: COBIT 4.1 Framework

17. Top-down Approach Change Mgt Application Security Database Security Development Network Security Process Controls Application Controls

18. Lessons Learned on Top-down • Security roles and functionality • Software Development Life Cycle • Test plans & supporting documentation • Testing to break vs. testing to pass

19. Lessons Learned on Top-down • Dispersion and diversity of the business, processes, and technology compounds the effort • Don’t underestimate the value of a thorough general controls review!

20. Bottom-up Approach • Auditing the functionality and control effectiveness of the application can only be determined based on the maturity level and effectiveness of the ITGCs Auditing the insides… Application Controls Testing

21. Bottom-up Approach Business Access and Application Management Data Source Data Processing Reporting Controls Origination Setup Controls Controls Input Processing Output

22. Bottom-up Approach 1 Item out-of-stock? Confirm customer order Notify shipping Send info to Acct A Start Critical Process Maps Data-Flow/Interface Diagrams 3 2 Confirm receipt Close books Log revenue Create invoice A End Information System Critical Risk / Control Points # • Understand the business process and technical environment • Assess business information risks

23. Bottom-up Approach • Understand the transaction process related to the application… identify key transactions • Assess Application Controls

24. Bottom-up Approach Screens to Test Transaction and Data Flow Detail

25. Bottom-up Approach Screens to Test Application Walk Through: Key Screens

26. Bottom-up Approach Boundaries of Business, General, and Application Controls Source: COBIT 4.1 Framework

27. Bottom-up Approach Application Controls • AC1 Source Data Preparation and Authorisation • Ensure that source documents are prepared by authorised and qualified personnel following established procedures… • AC2 Source Data Collection and Entry • Establish that data input is performed timely • AC3 Accuracy, Completeness and Authenticity Checks • Ensure that transactions are accurate, complete and valid. Validate data that were input, and edit or send back for correction as close to the point of origination as possible. Source: COBIT 4.1 Framework

28. Bottom-up Approach Application Controls • AC4 Processing Integrity and Validity • Maintain the integrity and validity of data throughout the processing cycle. • AC5 Output Review, Reconciliation and Error Handling • Establish procedures and associated responsibilities to ensure that… verification, detection and correction of the accuracy of output occurs. • AC6 Transaction Authentication and Integrity • When sharing data between internal applications and business/operational functions… maintain integrity. Source: COBIT 4.1 Framework

29. Bottom-up Approach • Testing Documentation • Application Function (screen mapping) • Function Description • Testing Procedures • Control Observations (testing results) & Recommendations • Key Risks & Impact • Lots of screen shots!

30. Bottom-up Approach sample

31. Bottom-up Approach Testing Procedures • Good code vs. not-so-good code • Sequence checks • Limit checks • Range checks • Validity checks • Reasonableness checks • Table lookups • Existence checks • Completeness checks • Duplicate checks • Logical relationships

32. System-Based Data Entry Integrity Controls Edits Description Sequence Checks The control number follows sequentially and any control numbers out of sequence or duplicated are rejected or noted on an exception report for follow-up purposes. For example, invoice numbers are systematically and generated sequentially generated and cannot be overridden. Limit Checks Data should not exceed a predetermined amount. For example, it may be determined that quantities ordered should not exceed a maximum of 500 each, or that a sales commission cannot exceed 12% without an override. If a quantity exceeds the limit, the data would be rejected for further verification or authorization. Range Checks Data should be within a predetermined range of values. For example, date ranges. Once books are closed for the current month, entries are not permitted for the previous month’s date range. Validity Checks Programmed checking of the data validity in accordance with predetermined criteria. For example, department & expenses relationships, account codes, company numbers, vendor numbers, customer numbers, invoice types, etc. Reasonableness Checks Input data are matched to predetermined reasonable limits or occurrence rates. For example, the tolerances exist within purchasing application to allow for quantity variations or unit price variations based on preset tolerable limits. Table Lookups Input data are agreed to predetermined criteria maintained in a data table of possible values. Essentially the same type of controls as validity – any pull-down menu, pre-defined field… an employee must exist within the EMPMSTR table for payroll processing. Existence Checks Data are entered correctly and agree to valid predetermined criteria. For example, a picking pack slip can only be generated for an existing order number, or in order to receive against a P.O., it must exist. Completeness Checks Data fields should always contain data and not zeros or blanks. A company number, accounting unit, and account code for a general ledger entry are required fields prior to the record being added to the system. A “hard stop” is received by the user if the required fields are not completed. Completeness check controls can be utilized by individual field or by data entry screen. Duplicate Checks New transactions are matched to those previously input to ensure that they have not already been entered. For example, accounts payable invoices cannot be entered twice for the same vendor invoice. Logical Relationship Checks If a particular condition is true, then one or more additional conditions or data input relationships may be required to be true and consider the input valid. Company, accounting unit, & account relationships exist. For example the marketing department cannot charge an expense to lease expenses, only to accounts logically tied to that department.

33. Bottom-up Approach Testing Procedurescontd. • Field formats are defined & locked • “Grayed” fields… better yet, linked to user and displayed only if applicable to functionality • On screen, visual feedback • Not-so-good… better have monitoring reports as mitigating controls!

34. Lessons Learned on Bottom-up • Applications pre-dating the current climate of control • Baselining • Value of an implementation project audit • Application functionality… best if conducted throughout testing stage • End user training & desktop procedures

35. Lessons Learned on Bottom-up • Monitoring controls… issues typically arise from the one $1,000,000 transaction, not from the million $1 transactions • Please remember to conduct audit work in a Test environment • Don’t underestimate the time commitment! • Emerging technology trends… web-based, PDAs, now smartphones

36. Summary • Risk considerations for application auditing • Risk-based approach drives increased efficiency + decreased costs = organizational value

37. Questions & Discussion Mike Kirk t: 614.403.7700 e: m_kirk01@yahoo.com