The Importance of Data in Functional Testing

1. The Importance of Data in Functional Testing James Lyndsay Workroom Productions

2. Data is important to functional testing. • A system is programmed by its data. • Functional testing suffers if data is poor. • Good data is vital to reliable test results. • Good data can help keep testing on schedule.

3. Problems caused by poor data • The following problems can be caused by poor data; • Unreliable test results • Degradation of test data over time • Increased test maintenance budget • Reduced flexibility in test execution • Obscure results and bug reports • Larger proportion of problems can be traced to poor data • Less time spent hunting bugs • Confusion between developers, testers and business • Requirements problems can be hidden in inadequate data • Simpler to make test mistakes • Unwieldy volumes of data • Business data not representatively tested • Inability to spot data corruption caused by bugs • Poor database integrity

4. Topics • 1) Recognising types of data • 2) Organising the data • 3) Involving the Business • 4) Data Load and Maintenance • 5) Testing the data

5. Types of Data Environmental data tells the system about its technical environment. Setup data tells the system about the business rules. Input data information input by day-to-day system functions. • Fixed input data is available at the start of the test. • Consumable input data forms the test input.

6. Organising the Data • Good data increases data reliability, reduces data maintenance time and can help improve the test process. • Good data assists testing, rather than hinders it. • Good data is based on; • 1) Permutations • 2) Partitioning • 3) Clarity

7. Data Permutations 1 • Permutations are familiar from test planning • Utility Customer Care / Billing Example: • A customer can have one of three products (1,2,3). They may be billed Monthly or Quarterly, be High or Low value, and their last bill was either Paid or Unpaid. There are 3x2x2x2 = 24 combinations - and so the number of possible permutations climbs rapidly as system complexity increases. • By requiring that the list holds not all possible combinations, but all possible pairs, the list can be reduced. All possible pairs; M1, M2, M3. Q1, Q2, Q3. H1, H2, H3. L1, L2, L3. P1, P2, P3. U1, U2, U3. MH, ML, QH, QL, MP, MU, QP, QU. HP, LP, HU, LU. • The following six permutations contain all the pairs; • Customer Account Product Care Bill • 1 M 1 H P • 2 M 2 L P • 3 M 3 H U • 4 Q 1 L U • 5 Q 2 H U • 6 Q 3 L P

8. Data Permutations 2 • Permutation is appropriate when: • Fixed input data consists of many rows • Fields are independent • You want to do many tests without loading / you do not load fixed input data for each test. • Permutation helps because: • Achieves good test coverage without having to construct massive datasets • Can perform investigative testing without having to set up more data • Can be used to test other data - particularly setup data • Permutation is familiar from test planning. • Reduces the impact of functional/database changes

9. Partitioned data • Data load/reload can be inconvenient. Data can be partitioned into: • 1) Safe area • Used for enquiry tests, usability tests etc. No test changes the data, the area can be trusted. Many testers can use simultaneously • 2) Change area • Used for tests which update/change data. Data must be reset or reloaded after testing. Used by one tester at a time. • 3) Scratch area • Used for investigative update tests and those which have unusual requirements. Existing data cannot be trusted. Used at own risk! • Data can be partitioned by machine / database / instance. Can also be partitioned by disciplined use of text / value fields.

10. Clarity • Developers and the Business don’t need to understand test data / data requirements - so some of them won’t. We can make our data clearer by using available free text fields; • Customer Name Account Product Care Bill • HP1 Monthly M 1 H P • LP2 Monthly M 2 L P • HU3 Monthly M 3 H U • LU1 Quarterly Q 1 L U • HU2 Quarterly Q 2 H U • LP3 Quarterly Q 3 L P • Clarity helps because: • Improves communication within and outside the team • Reduces test errors caused by using the wrong data • Helps when checking data after input • Helps in selecting data for investigative tests

11. Data and the Business • ‘The Business’ is good at looking at data; • Easier to understand than tests • Can be compared with existing systems • Advantages • Helps focus when requirements are vague • Helps UAT • Increases trust and understanding • Helps early user identification of problems • Disadvantages • Data creep • Vague requirements can lead to vague data • Incomplete data can lead to incomplete testing

12. Ways of loading data • Using the system you're trying to test • Manually entered • Automated tool, making keystrokes into application • Using a data load tool • All new data, created for testing • Old data, selected for testing / filtered and migrated • Complete set, migrated and loaded, identical but for personal details • Not loaded • Already set up for testing / Left in the system • You are working on the live system

13. Frequency of data load • At the start of testing • With each release • First thing Monday • Whenever I want • Before every test • NB: Straw poll of ‘When does the data usually get loaded’ also came back with the following answers; • Before I ever got involved • The developers left it there • The last testers / tests left it • Whenever we get enough time • After we've found out what shape the database is • When we know what it means

14. Data Maintenance • When and why? • Replacing consumed data • Repairing broken data • Responding to change - database schema, code, requirements • New test requirements • Problems: • Sizeable task - can be a substantial fraction of overall test maintenance • Prone to error • Performed by more than one group

15. Solutions to Loading and Maintenance headaches • 1) Automate data load and maintenance where possible • 2) Control / measure data change • 3) Recognise and prepare for problems • 4) Use good data

16. Testing the data • Systems are programmed by their data . . . • If the functionality of your system is at all affected by the setup data; • TEST THE DATA • 1) Does the current setup data affect the application appropriately? • 2) Will changes made to the setup data have the desired effect? • To incorporate into functional testing, you need; • organised data • good load/maintenance • business involvement

17. Testing the data 2 • Use a known dataset to: • Show functionality correctly applied • Assess effect of changes in setup data • Aims: • Improve overall testing • Help get the data right before live operation • Increased confidence in re-configuration • Help pinpoint bugs in live operation

18. Further data issues • Operational Profiles • Non-functional testing • Data verification

19. Topics: conclusion • 1) Recognising types of data • Environmental Data • Setup Data • Input Data - Fixed and Consumable • 2) Avoiding common problems • Recognise the problems • Use good data • Involve the Business • Automate loading and maintenance • Test your data

20. Summary • Plan the data for maintenance and flexibility • Know your data, and make its structure and content transparent • Use the data to improve understanding throughout testing and the business • Test setup data as you would test functionality James Lyndsay Workroom Productions www.workroom-productions.com jdl@workroom-productions.com