Consistency and Repeatability in Analytics (Using M2) wfunk@kennellinc

1. Consistency and Repeatability in Analytics (Using M2)wfunk@kennellinc.com

2. Consistency and Repeatability in Analytics • 1) Context: The practice of analytics requires discipline to ensure results are repeatable, consistent and reliable. • 2) Purpose: This presentation will describe best practices for reporting • 3) Outcome: After attending this session, participants will meet the objectives described on the next slide. FOR OFFICIAL USE ONLY

3. Objectives • Attendees can: • List best practices for preparing reports in M2 • Utilize the M2 data dictionary. • Retrieve a copy of the M2 Data Dictionary • Describe an analysts reponsibilities with respect to data protection. • Describe a good M2 deliverable product. • Track work on repeated projects for anomalies • Keep proper documentation associated with the use of M2.

4. Motivation • Reliable analysts usually become very valuable to an organization. • Especially true when analysts use ad-hoc systems to prepare studies. • Ad-hoc offers maximum flexibility for question answering • But there is always the risk that the user will not properly retrieve/handle data. • This presentation offers tips on the optimal use of M2, to achieve reliability, consistency, and repeatability. FOR OFFICIAL USE ONLY

5. Preparing M2 Reports FOR OFFICIAL USE ONLY

6. Restricted vs. Unrestricted M2 • There are two options for M2 access: • M2 Restricted: • Contains direct identifiers for beneficiaries (i.e. name, SSN) • Contains rank of sponsor • Most users do not need the restricted M2. • Appropriate for care managers or other people who need to know who the person is on a record in M2 • Can only retrieve <10K rows of data. • M2: • Encrypted person identifiers (mostly), grouped rank only • Considered safer, but still contains protected health information. • Can only retrieve up to 500K rows of data at a time. Use the Unrestricted M2 when you can!!!!!

7. Report Writing • Check to see if there is a corporate report that meets your needs. • Handbook can be obtained from M2 Infoview FOR OFFICIAL USE ONLY

8. Report Writing • If you find a candidate report, read the documentation carefully before using the report. • [Corporate Reports session on Tuesday at the Symposium] • If there is no available corporate report • Use available resources • M2 Data Dictionary • WISDOM notes • DCO Refresher Sessions • Navy Newsletters • Navy DCOs • Reports previously done FOR OFFICIAL USE ONLY

9. M2 Data Dictionary FOR OFFICIAL USE ONLY

10. M2 Data Dictionary • M2 Data Dictionary is published once per month • http://www.tricare.mil/ocfo/bea/functional_specs.cfm • Or even easier, just Google “M2 Data Dictionary”! • Excel workbook • Tabs for each type of data • Additional Reference tabs • Hyperlinked for easy navigation • May have to click “enable content” for navigation to work. Use the M2 Data Dictionary when you write reports!!!!! FOR OFFICIAL USE ONLY

11. M2 Data Dictionary M2 Alerts are not all that useful.

12. It says population, so this must be something about counting people?? • What does 0005 mean? What about age group ‘A” or Beneficiary Category ‘DA’? FOR OFFICIAL USE ONLY

13. M2 Data Dictionary • Click on M2 Data Dictionary and open it. • Click on the Table of Contents tab. • Each file is listed with a hyperlink to the page that describes the file. • Click on “Population Summary”. • Data elements are listed in the same order as in M2. • Review the header row. • Review the rows for “Age Group Code” and “Beneficiary Category”. • Review the row for “Catchment Area ID”. Click on the DMISID link. FOR OFFICIAL USE ONLY

14. To use the DD: Click on the title that represents the data file you are looking at • In this case, it’s population summary. FOR OFFICIAL USE ONLY

15. Header Information Above. • Detailed data descriptions in each row below FOR OFFICIAL USE ONLY

16. Using the DD • Description of valid values for age group code FOR OFFICIAL USE ONLY

17. Using the DD • Description of valid values for Beneficiary Category FOR OFFICIAL USE ONLY

18. Using the DD • No values are listed for Catchment Area ID • But if you click on the link….. FOR OFFICIAL USE ONLY

19. Using the DD • DMISIDs that represent the area being reported on, and associated names FOR OFFICIAL USE ONLY

20. Notice how the values you see in this report line up with what you just saw in the dictionary? • 2,759 Active Duty Family members ages 0-4 live within 40 miles of Bassett ACH in Alaska FOR OFFICIAL USE ONLY

21. M2 Data Dictionary • As with Catchment Area DMISID, when a variable in M2 has too many valid values to list, either: • Use the hyperlinks to click to an appendix with information, OR • Use reference tables within M2 • There are a few variables that do not have valid value lists in the DD or in an M2 reference table. • Might have to resort to web searches or other means in these cases • Sometimes the data dictionary tells you not to use a variable. • Back to table of contents and click on “Purchased Care Non-Institutional Detail” • Look at the comment for “Number of Visits, Raw” • DD contains all variables, restricted or not, you just won’t be able to see restricted variables if you only have regular access. FOR OFFICIAL USE ONLY

22. Using the DD • After reading this note, users should not retrieve this element and called what they report a “visit”. • Very important to always check the DD. FOR OFFICIAL USE ONLY

23. M2 Data Dictionary • The M2 Data Dictionary is an excellent resource to use, but it is not always correct. • This MTF Service Area definition is incorrect. • MTF Service Areas apply to inpatient and ambulatory clinics. • Has been reported and will be corrected. • Be careful! Really evaluate your data! FOR OFFICIAL USE ONLY

24. Other Resources • WISDOM CD: • Given to WISDOM Attendees • Detailed descriptions of data in M2 and how to use it. • Reports and more reports • Step by step instructions for Business Objects. • WISDOM instructors contact information. • WISDOM DCOs: • Conducted monthly, for all M2 users. • Materials are posted on Infoview (like the corporate reports) • Navy Analytics Newsletters (also in Infoview) • Navy DCOs (coming soon) FOR OFFICIAL USE ONLY

25. Previous Reports • Many questions are similar • Similar cohorts • Active Duty? Prime? Diabetics? • Etc… • Most users save their reports so they can be referred to again. • Can be cumbersome to find things though • It’s also a very good idea to keep a running list of criteria that you use to address questions for easy reference. FOR OFFICIAL USE ONLY

26. Example of a Running List of Query Criteria FOR OFFICIAL USE ONLY

27. Data Status Table • Run the Data Status table to check for freshness of data and any comments that may be applicable. FOR OFFICIAL USE ONLY

28. Report Checking • Once your report is written, make liberal use of slice and dice to check your work. • Check each variable to ensure that the content is what you expect. • Check counts of data for reasonability. • Look at monthly data to be sure you understand trends and such. • Double check calculations by doing crosstabs and by doing hand calculations where appropriate. FOR OFFICIAL USE ONLY

29. Report Checking • If you are running the same report regularly, keep track of how the data changes over time • Use that knowledge to know what to expect. • Example of same report being run once per month, with counts by month. Keeping adding data w/ each new run. FOR OFFICIAL USE ONLY

30. Report Checking • Be careful of row limits! • Check to make sure you don’t get “Partial Results” • Your data will be wrong if you use them! • Tiny little error message at the bottom of the screen! FOR OFFICIAL USE ONLY

31. Report Checking • Row limits can cause analytic problems: • If you need data that exceeds the row limit: • If you have both restricted and unrestricted access, see if your query can be written if the unrestricted universe. This will give you 490K extra rows to work with. • If you still need more than your row limit allows there are not many good choices. • Some users run their queries in pieces (i.e. month at a time) and then piece results back together. • If you do this, be sure you are authorized. (i.e. both universes are PHI, be careful what you do!). • Also, do very carefully, because you can introduce more human error this way. • Make a checklist of all of the pieces you need, check off that you got them, and run summary queries to confirm you put things together properly. FOR OFFICIAL USE ONLY

32. Data Protection FOR OFFICIAL USE ONLY

33. Row Limits • M2 obviously contains protected health information (PHI) and personally identifiable information (PII) • PII: Generally defined by the Privacy Act • PHI: Defined by HIPAA. • Currently both the M2 and M2 Restricted Universes contain PII and PHI • There are many rules and regulations regarding the use of PII and PHI FOR OFFICIAL USE ONLY

34. Data Protection When using M2 data, what do you need to know? • Which data fields are considered PII or PHI? • Which data fields are sensitive? • When you use these types of data, how do you protect yourself and the beneficiaries in the data? FOR OFFICIAL USE ONLY

35. PHI or PII? • PII • Defined by the Privacy Act • Such as names, Social Security Numbers, etc.. • ‘Small cell’ (<=30) data fields (i.e. age 90+, or Alaskan Indian Race). • DEERS Person ID (EDIPN): • Has recently been classified as PII by the TMA Privacy Office. • DEERS Person ID is available in both the M2 and M2 Restricted Universes. • Must be handled as you would a social security number. • DEERS Person ID will be encrypted in regular M2 in a future update. • The DEERS Person ID of the user is visible on the screen in M2. • Take particular caution if you are a beneficiary and an M2 User! FOR OFFICIAL USE ONLY

36. Sensitive Data • Some data are not covered under laws but are deemed sensitive • Some examples include: • Procurement sensitive data: Such as Ingredient cost and dispensing fee in purchased care pharmacy data • Proprietary Data: Such as CPT Codes and Descriptions. • Restricted use: Such as cause of death FOR OFFICIAL USE ONLY

37. PII or PHI? • PHI • Defined by HIPAA. • HIPAA is much stricter than the Privacy Act. • More than just PII. HIPAA seeks to further restrict deduction of identities, in addition to just restricting access to direct identifiers. • Stiffer fines for breaches (25K per HIPAA, 1K per Privacy Act). • HIPAA Limited Data Sets are a type of PHI but less strict than “Full PHI”. • Most research is done with HIPAA Limited Data Sets (LDS). • Lower hurdle to gain access (Privacy Office). • Still subject to all the rules and regulations of PHI, though. FOR OFFICIAL USE ONLY

38. Specifically Covered Elements • Names • Any geographic designation lower than state level (zip codes, catchment area, PRISM area, etc.). Any geography < 20K people. • Any date related to a person below the level of year (FM, Service Date, Date of Birth, Age (if >89) • Telephone Numbers, Fax Numbers, Email Addresses • Social Security Numbers • Medical Record Number (FMP/SSN) • Health Plan Beneficiary Number (EDIPN, DEERS Beneficiary ID) • Any other unique identifying number • (Other HIPAA elements are not generally available in MHS systems) The PHI bar is set very high by HIPAA FOR OFFICIAL USE ONLY

39. HIPAA • HIPAA Limited Data Sets (LDS) are PHI, but dates (except DOB) and geographic designations are allowed. • Absolutely essential elements in health research and analytics • Especially dates, where the temporal relationship between events must be understood. • M2 is a PHI system. Even the non-restricted version of M2 does not qualify as a limited data set. FOR OFFICIAL USE ONLY

40. HIPAA • Under HIPAA, a citizen has the right to information about who has accessed their PHI. • Implemented as the “Chain of Custody” requirement. • Must document use/release of PHI. Done with a HIPAA Log. • After writing a report, also consider whether the data use needs to be tracked. • Must have data use agreements in place to release external to the organization. • HIPAA breaches are very serious matters. Reporting of breaches are expected immediately. • The MHS has the biggest HIPAA breach in history, with CHCS data in San Antonio. Has resulted in a billion dollar lawsuit. FOR OFFICIAL USE ONLY

41. OUCH! FOR OFFICIAL USE ONLY

42. Sample HIPAA Log • Date of File • Disposition Date • Fed Ex Number • Destruction Witnessed By • Delivery Mechanism • Date Data Retrieved • User • Company/Organization • Task Order • Project Name • External Client/POC • Received From External Source File? • Sent To External Source? • Date Delivered • External Agency DUA # • File Names and Locations • Protected Info • Tracking is required by HIPAA for all PHI handling, not just M2 • Well-suited for a spreadsheet FOR OFFICIAL USE ONLY

43. Best Practices • Have all users in an organization use the same template for a HIPAA log. • Assign someone to be the HIPAA Log Coordinator at your location. • Have all users submit the log to the Coordinator on a routine basis for consolidation. • Review with DUA custodian or supervisor routinely to ensure data are being properly used and documented. FOR OFFICIAL USE ONLY

44. Best Practices • Physically secure your data • Locked cabinets, locked doors, locked file cabinets • Implement technological safeguards • Encryption • Mark copies FOUO, etc • Protect passwords/log-on information • Protect terminal while logged in (lock your workstation) • Do not leave CAC card unattended FOR OFFICIAL USE ONLY

45. Best Practices • Don’t release (or re-release) PHI data w/o authorization • Check DUA if a contractor or consider need to know if internal government. • Don’t publish PHI or PII. • Limit reports to retrieve the ‘minimum necessary’ information. • Destroy data when you are done with it (cross-cut shred, overwrite and delete, or deGauss) • Deliver data securely • Secure FTP, Encrypted Drives FOR OFFICIAL USE ONLY

46. M2 Deliverable Products FOR OFFICIAL USE ONLY

47. M2 Deliverable Products • Since M2 allows ad-hoc querying, there is always a risk that reports will not yield the results you want! • Ad-hoc can be your best friend or your worst enemy! • A good M2 deliverable product will contain detailed documentation so that recipients of data understand what they got. • Most users document in spreadsheets, with multiple tabs if needed. • Screen shot of query panel(s). • Screen shots of additional math/manipulation or language to describe it. • List of filters used with an indication of why. • Snapshot of the data status table and other metadata. FOR OFFICIAL USE ONLY

48. Screen Shot Data Elements Retrieved Using the DMIS ID Table Filters Applied FOR OFFICIAL USE ONLY

49. Screen Shot of Additional Calculations Created a variable called “% Diff” FOR OFFICIAL USE ONLY

50. Picture of a slice and dice panel. • Similar to Excel Pivot Tables • Enables users to reshape/filter data within the M2 application. FOR OFFICIAL USE ONLY