How to analyze thread dumps using a Java Thread Dump Analyzer

Subject

Thread dumps allow a user to see a snapshot of what Artifactory is doing at the time the thread dump is taken so it is a good resource to analyze when it comes to performance troubleshooting.  A Java thread dump analyzer tool makes it easy to compare, read, and analyze multiple thread dumps.

Affected Versions
Any Artifactory version

Details
There is more than one way to generate thread dumps:  

Artifactory is able to generate multiple thread dumps from it’s Information Bundle:
https://www.jfrog.com/confluence/display/RTF/Getting+Support#GettingSupport-CollectinganInformationBundle

There are also other systematic ways to generate a thread dump:
https://jfrog.com/knowledge-base/how-do-i-get-a-thread-dump/

Any of these methods can be used to generate thread dumps that can be read and analyzed by a thread analyzer tool.

Resolution
It is important to make sure that a thread dump is generated during the time of while the problem is happening.  If thread dumps are taken too close to one another, there may be too much noise and it would be more difficult to draw out useful information. Taking the thread dumps too far apart could potentially cause a miss in the data that could be vital.  A general rule of thumb is to take 5 thread dumps, one at every 10 seconds. 

After the thread dumps are taken, upload the files into a thread analyzer tool such as fastThread Java Thread Dump Analyzer:
http://fastthread.io/

It may time a little bit of time for the analyzer tool to parse analyze the thread dumps but after it completes, a summary will be provided to show an analytical breakdown of the the threads and it’s states.

Thread Information:

  • Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number).
  • Priority: Represents the priority of the threads.
  • Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.)
  • Thread status: Represents the status of the threads.
  • Thread callstack: Represents the call stack information of the threads. 

Thread States:

  • NEW: The thread is created but has not been processed yet.
  • RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITINGstatus due to the OS's resource distribution.)
  • BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock.
  • WAITING: The thread is waiting by using a wait, join or park method.
  • TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.) 

*Note: If a thread dump is too large for the thread analyzer tool, the following Linux command is useful for splitting the file by size:
split -b <file size> <thread dump file> <segment file>

For example: If we have a thread dump, named myfile, that is 140k and we executed the following command:
split -b 40k myfile segment

This will output four 40KB files: segmentaa, segmentab, segmentac, and segmentad

*Thread information and thread states were referenced from: https://dzone.com/articles/how-analyze-java-thread-dumps