Threaddump analysis with Dumpling CLI

Dumpling CLI is fully scriptable, console based tool for threaddump analysis. It is GitHub hosted project available under the terms of MIT license. It values maximal flexibility, reliability and convenience. How can it help you?

Installation

Dumpling CLI is distributed in a form of a self contained jar file hosted on OSS Sonatype maven repository. Preferred way to install it is using dumpling.sh script:

$ wget https://raw.githubusercontent.com/olivergondza/dumpling/master/dumpling.sh
$ chmod +x dumpling.sh

When used for the first time, the script will pick-up latest dumpling.jar and place it into current directory. Run ./dumpling.sh help to see what features offers.

Predefined queries

Dumpling implements several subcommands invoked the same way we just run help. Some of these are queries, predefined reports operating on a process runtimes. Process runtime is a Dumpling abstraction used to represent the state of threads and locks in the JVM at a given time. Every query is (or can be) provided with a process runtime using --in option. Such runtime can be obtained from threaddump log, existing process identified by its PID or over JMX connection.

Deadlock detection

Unsurprisingly, deadlocks query detects deadlocks in provided runtime. All deadlocks found are reported in a form of a short report. Overall number of deadlocks is returned as command exit code.

./dumpling.sh deadlocks --in threaddump:deadlock-jstack.log

Produces report of all deadlocked threads found in deadlock-jstack.log file. Such report can look like this:

Deadlock #1:
"Thread A" daemon prio=10 tid=1481750528 nid=27336
    Waiting to <0x40dce6960> (a class.of.monitorB)
    Acquired   <0x49c5f7990> (a class.of.monitorC)
    Acquired * <0x404325338> (a class.of.monitorA)
"Thread B" daemon prio=10 tid=47091108077568 nid=17982
    Waiting to <0x404325338> (a class.of.monitorA)
    Acquired * <0x40dce6960> (a class.of.monitorB)


Deadlocks: 1

Full header of all involved threads are reported with related monitors/synchronizers. Note that threads contributing to the deadlock are highlighted using *.

Thread blockage

Visualizing what tasks are blocking other tasks is priceless in understanding how lock contention affects application execution. That is why dumpling introduces blocked-blocking relationship: thread A blocks thread B (or thread B is blocked by thread A) when A can not proceed until B releases either a monitor or a lock. Therefore, a thread can block any number of other threads while itself it can be blocked by at most one thread.

The blocking-tree query arranges threads into a tree-like hierarchy based on blocked-blocking relationship. Threads that are not blocked itself but blocking other threads are reported as roots. Blocked threads are attached under the threads they are blocked by forming the rest of the hierarchy. Like so:

$ ./dumpling.sh blocking-tree --in threaddump:threaddump.log
"a" prio=10 tid=139918763327488 nid=31954
	"aa" prio=10 tid=139918763409408 nid=31955
		"aaa" prio=10 tid=139918763419648 nid=31957
	"ab" prio=10 tid=139918763413504 nid=31956

"b" prio=10 tid=139918763425792 nid=31958
	"ba" prio=10 tid=139918763431936 nid=31959



All threads: 6; Roots: 2

Report shows 2 separated blockage trees containing 6 threads in total. It clearly illustrates that thread aaa is blocked by aa that is itself blocked by a.

Both queries effectively reduces possibly large listings into concise reports. If more details are required, --show-stack-traces prints stacktraces of all involved threads.

Writing custom queries

However convenient can predefined queries be, creating custom ones offers whole new world of flexibility. For that purpose, groovy query exposes whole runtime in a form of Dumpling DSL to user provided groovy script. All you need to call to discover number of threads in running java process is:

$ ./dumpling.sh groovy --in process:4242 <<< "D.runtime.threads.size()"
7

In practice, most of the groovy queries perform thread filtering based on custom criteria. To avoid writing D.runtime.threads.grep { ... } over and over again, there is a grep query to do that. Filtering threads based on the status can be as simple as:

$ ./dumpling.sh grep "thread.status.waiting" --in threaddump:my_threaddump.log

For more information on how to use the domain specific language exposed by Dumpling, consult the reference documentation and javadoc.

Conclusion

Those are some of the most prominent features Dumpling has to offer. Consult the project homepage for more detailed documentation.

Tags: dumpling

PoorlyWritten

by olivergondza