XRAY: How to Collect Xray Thread Dumps

Ariel Kabov
2023-01-22 11:09

thread dump is a snapshot of the state of all threads that are part of the process.
In Xray we have 4 main processes: Server, Indexer, Analysis, Persist, which require collecting information from each separately.

To collect Xray thread dumps, run the following cURL commands:
Note: The profiling endpoint is only accessible via localhost.

curl "localhost:8000/debug/pprof/goroutine?debug=2" > "server.dump.$(date +%Y%m%d%H%M%S).td"
curl "localhost:7000/debug/pprof/goroutine?debug=2" > "analysis.dump.$(date +%Y%m%d%H%M%S).td"
curl "localhost:7002/debug/pprof/goroutine?debug=2" > "indexer.dump.$(date +%Y%m%d%H%M%S).td"
curl "localhost:7003/debug/pprof/goroutine?debug=2" > "persist.dump.$(date +%Y%m%d%H%M%S).td"

Another useful debugging tool is CPU profiling, which can be collected by:

curl "localhost:8000/debug/pprof/profile?seconds=60" > "server.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out"
curl "localhost:7000/debug/pprof/profile?seconds=60" > "analysis.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out"
curl "localhost:7002/debug/pprof/profile?seconds=60" > "indexer.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out"
curl "localhost:7003/debug/pprof/profile?seconds=60" > "persist.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out"

To collect a Heap Memory profile:

curl "localhost:8000/debug/pprof/heap" > "server.heap.$(date +%Y%m%d%H%M%S).out"
curl "localhost:7000/debug/pprof/heap" > "analysis.heap.$(date +%Y%m%d%H%M%S).out"
curl "localhost:7002/debug/pprof/heap" > "indexer.heap.$(date +%Y%m%d%H%M%S).out"
curl "localhost:7003/debug/pprof/heap" > "persist.heap.$(date +%Y%m%d%H%M%S).out"