1 / 19

Memory issues in production systems

Memory issues in production systems. Production system. Restricted access Application, DB, Application server, log files Debugging, monitoring Investigation should not affect users Reproducing memory issue is not the best idea. Memory leaks. What is memory leak

tal
Download Presentation

Memory issues in production systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Memory issues in production systems

  2. Production system • Restricted access • Application, DB, Application server, log files • Debugging, monitoring • Investigation should not affect users • Reproducing memory issue is not the best idea

  3. Memory leaks • What is memory leak • java.lang.OutOfMemory: heap size • other symptoms - extensive garbage collection • Hard to discover • Users don't recognize it • Hard to investigate • Few information • Sometimes hard to fix • Design issues

  4. Investigation of memory issues • Log files • application logs, access logs, gc logs • Monitoring behavior of running application • JConsole • Heap dump Analysis • JHAT, MAT, Netbeans

  5. Log files • Motivation • find issues • find patterns leading to issues • Problems • combining different logs (access log, application log) • bad visibility of root cause – not visible or hidden in big amount of minor issues • memory problems trigger only few types of messages

  6. Investigating log files • Application logs • Log free memory once per minute • Access logs • Analysis of traffic and business use cases • GC logs • Show problems with GC (stop the world)

  7. Monitoring via JMX • Tools • JConsole • Motivation • More information than in logs • Monitoring tests • Problems • Not always applicable on production system (security reasons) • Test might not cover all scenarios

  8. Heap dump analysis • Tools • JMAP, JHAT, MAT, Netbeans • Motivation • Offline investigation of behavior during specific time frame • Comparison of different heap dumps • Problems • Access to heap dumps (security)

  9. Creating Heap dump • jmap -dump:live,format=b,file=heap.dump <pid> • Part of JDK • Used to create heap dump on demand • JVM parameters • -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/directory/subdirectory • Produces file java_pid<pid>.hprof • Used for production system, creates heap dump when OutOfMemory is thrown

  10. Tools for heap dump analysis • JHAT • Part of JDK 6 • Running as web application • IDE • Netbeans, MAT (Eclipse plugin) • Commercial tools

  11. Heap dump analysis - JHAT • Heap Histogram • List of classes with number of instances and used memory • Referrers vs Referees • Navigating in object graph • Example: Form → String → [c

  12. Heap dump analysis – Netbeans, MAT • Dominator tree • Object x dominates object y if every path in object graph from root to y goes through x • Shallow vs Retained heap • Shallow heap is memory used by object • Retained set of X is set of objects which would be removed by GC if X is removed by GC • Retained heap is sum of shallow heap in retained set • Other queries • Leak suspects • Top consumers

  13. Heap dump Analysis - OQL • Object query Language • Different dialect for each tool • JHAT • Simple SQL combined with javascript • callback functions • MAT • More SQL like (Subquery, union, distinct) • Predefined functions (dominator)

  14. OQL Examples • JHAT • select heap.findClass('org.apache.catalina.util.ServerInfo').statics.serverNumber.toString() • MAT • SELECT serverNumber.toString() FROM OBJECTS org.apache.catalina.util.ServerInfo

  15. Experiences • Bugs • Incorrect usage of external libraries or frameworks • Design issues

  16. Bugs

  17. Usage of frameworks • Hibernate • Short time usage of big amount of memory • Too big first level cache • Good understanding of framework is a must

  18. Design Issues • Overused session scope • Big pages • Big buffer in page context • Page context referenced by tag handlers • Heavy load causes big memory usage • Fix might require changing use cases • Reusing functionality

  19. Interesting links • http://blogs.sun.com/sundararajan/entry/querying_java_heap_with_oql • OQL help for JHAT • http://www.eclipse.org/mat/downloads.php • Standalone MAT • www.google.com

More Related