A thread dump is a snapshot of the state of all threads that are part of the process.
In Xray we have 4 main processes: Server, Indexer, Analysis, Persist, which require collecting information from each separately.
To collect Xray thread dumps, run the following cURL commands:
Note: The profiling endpoint is only accessible via localhost.
curl "localhost:8000/debug/pprof/goroutine?debug=2" > "server.dump.$(date +%Y%m%d%H%M%S).td" curl "localhost:7000/debug/pprof/goroutine?debug=2" > "analysis.dump.$(date +%Y%m%d%H%M%S).td" curl "localhost:7002/debug/pprof/goroutine?debug=2" > "indexer.dump.$(date +%Y%m%d%H%M%S).td" curl "localhost:7003/debug/pprof/goroutine?debug=2" > "persist.dump.$(date +%Y%m%d%H%M%S).td"
Another useful debugging tool is CPU profiling, which can be collected by:
curl "localhost:8000/debug/pprof/profile?seconds=60" > "server.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out" curl "localhost:7000/debug/pprof/profile?seconds=60" > "analysis.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out" curl "localhost:7002/debug/pprof/profile?seconds=60" > "indexer.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out" curl "localhost:7003/debug/pprof/profile?seconds=60" > "persist.cpu-profile-60s.$(date +%Y%m%d%H%M%S).out"
To collect a Heap Memory profile:
curl "localhost:8000/debug/pprof/heap" > "server.heap.$(date +%Y%m%d%H%M%S).out" curl "localhost:7000/debug/pprof/heap" > "analysis.heap.$(date +%Y%m%d%H%M%S).out" curl "localhost:7002/debug/pprof/heap" > "indexer.heap.$(date +%Y%m%d%H%M%S).out" curl "localhost:7003/debug/pprof/heap" > "persist.heap.$(date +%Y%m%d%H%M%S).out"