Performance and Reliability 101

1. Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com

2. A little about me • Like • Long walks on the beach • Red wine • Dislike • Mean people • Early mornings • Encourage questions throughout presentation • Although I may hold off if I am going to address it with material in a future slide. • You may have to ask again

3. OK Seriously… • Spent bulk of my career (14 years) at SAP • By way of Business Objects acquisition • By way of Crystal Decisions acquisition • By way of Seagate Software IMG • 5 years experience in customer support • Discovered impatience, dislike for people • 9 years of performance and reliability (P&R) testing for Crystal Reports in Crystal/Business Objects Enterprise product • Currently in my second year with Ping Identity

4. Why are we here? • Types of testing that make up P&R • Design • What is the goal of each test type? • What does it prove/disprove? • Execution • How is the test run? • Results Analysis • How to figure out if the test passed or failed? • Best Practices (Tips/Tricks/Suggestions/Filler) • Suggestions for root cause analysis

5. So… What are these test types that I speak of? • Types of P&R tests • Load • Scalability • Endurance • Stress • Reliability

6. Load

7. Load Testing • Performance equivalent of functional “smoke” test • Functional test/workflow executed under “load” • Typically “load” is in the form of concurrent users • Executed with a Load Generator tool • Load Runner, JMeter, QALoad, Grinder, etc… • Does the component stand up? • Does the test pass functionally? For all users? • Does it crash? Does the system grind to a halt? • Metrics to consider • Response time (average, 90th percentile, min, max) • Throughput • CPU and memory utilization on the target system

8. Scalability

9. Scalability Testing • Executed as a series of Load Tests • Workload Scalability • Vary the user load from test to test • Resource Scalability • Vary the resources from test to test • Functional success • Error rate “too high”, scalability results are meaningless • How does performance change from test to test? • Response time (average, 90th percentile, min, max) • Throughput • CPU and memory utilization on the target system • Do not discount single user performance • A system can exhibit linear scalability, but still perform poorly

10. Endurance

11. Endurance Testing • Also know is “Soak” testing • Load test executed over an extended duration • Typically overnight or over the weekend • Proves “reliability” of the system • Consistency of functional results • Very first result same as very last and all those in between? • Depending on requirements, error rate > 0 can be acceptable • Consistency of performance • Does response time or throughput degrade over time? • Consistency of resource utilization • Are we leaking memory? • How does CPU usage look over the duration?

12. Stress

13. Stress Testing • Often mistakenly referred to as “Load” testing • Best thought of as “extreme” load testing • Resiliency of the system when pushed beyond limits • 150% to 200% of the “nominal” load for the system • Half the system resources suggested for a given load • CPUs, memory, network bandwidth, etc… • Looking for “graceful failure” • Best: System returns “Too Busy” • Acceptable: System slows down, maybe some requests time out • Better: effective error messaging so that uses know system is maxed out • Bad: Crash • Worst: Unpredictable results, misleading error messages

14. Reliability

15. Reliability Testing • Negative condition Load Testing • Test resiliency under error conditions • Error condition code paths typically don’t get the same coverage as the “happy path” • Is the system consistent under constant error conditions? • Are results consistent and predictable over time? • Consistency of resource utilization • Error conditions are notorious for resource leaks • Security tests • i.e.: Denial of Service

16. Random Suggestions (Time Filler) • Choose workflows that fit the “80/20 rule” • Some workflows need P&R testing, others don’t. Choose wisely. • Use sufficient hardware for your Load Generator application • Size your client hardware like you would your target system • Don’t use “intrusive” validation in your test cases • Heavy test validation will slow down your test and affect concurrency • Avoid use of “intrusive” monitoring when possible • Beware of logging • Logging is useful, but can kill performance • Visualize your results • A picture is worth a thousand words. Who doesn’t like charts? • Include context (resource utilization of the systems under test)

17. So what do I do if I think there is a problem? • Too slow? • Is your system tuned? • Ensure you have not configured a bottleneck in your deployment • Try a profiling tool • Can show which areas of the code are taking the most time • Add some lightweight logging to code • Add “timing code” to log out elapsed time in functions/paths • Use a stack dumping utility • Repeated stack dumps can show where you are “stuck” • Using too much memory or leaking? • Try a profiling tool • Can show • Add “size” logging for container classes • Can show you if your containers are growing unbounded

18. Questions?