Step 1: Identify the unresponsive window in Router logs
When the platform becomes briefly unresponsive, start with the Router service logs (router-service.log). Look for topology/health changes where one or more services on a node flip to UNHEALTHY / UNHEALTHY_PEER and later return to HEALTHY.
Example (abbreviated):
[jfrou] Topology changes details: External Topology changed: art1 state changed
(jfrt@... UNHEALTHY, jfmc@... UNHEALTHY, jfac@... UNHEALTHY_PEER, ...)
...
[jfrou] Topology changes details: External Topology changed: art1 state changed
(jfrt@... HEALTHY, jfmc@... HEALTHY, jfac@... HEALTHY, ...)
Note:
-
The node / hostname
-
The start and end timestamps of the unhealthy window
-
Which services were marked unhealthy
-
If several Java microservices on the same node go unhealthy together (for example Artifactory jfrt, Mission Control jfmc, Access jfac), treat that as a strong hint of a node-wide JVM pause (such as GC stop-the-world), not an isolated application bug in one service.
Step 2: Correlate readiness probes on the affected Java service
Next, open the service log for an affected Java microservice around the same timestamps. For example in $JFROG_HOME/var/log/, check
Access-service.log, artifactory-service.log, or mc-service.log
Router readiness probes call these endpoints periodically. During a GC stop-the-world (STW) pause, the JVM does not run application threads, so probe requests sit until the pause ends.
What to look for:
-
Around the unhealthy window, many readiness requests that appear to complete at nearly the same time (or show a multi-second gap with no completions, then a burst). That pattern usually means probes were blocked together while the JVM was paused, then flushed when STW ended.
2026-08-10T07:38:00.857Z GET /api/v1/system/readiness 200
2026-08-10T07:38:05.861Z GET /api/v1/system/readiness 200
2026-08-10T07:38:10.858Z GET /api/v1/system/readiness 200
2026-08-10T07:38:27.064Z GET /api/v1/system/readiness 200 ← three at the same ms
2026-08-10T07:38:27.064Z GET /api/v1/system/readiness 200
2026-08-10T07:38:27.064Z GET /api/v1/system/readiness 200
2026-08-10T07:38:32.770Z GET /api/v1/system/readiness 200
-
Also check application logs for HikariCP starvation warnings. Around the same unhealthy window, search the affected Java service log (for example artifactory-service.log, mc-service.log) for “Thread starvation or clock leap detected (housekeeper delta=...)”:
[jfmc] [WARN] [c.z.h.p.HikariPool] - HikariPool-1 - Thread starvation or clock leap detected
(housekeeper delta=1m6s...)
Step 3: Confirm GC stop-the-world in JVM GC / safepoint logs
Artifactory and other Java services write JVM GC logs (for example artifactory-jvm-gc-*.log). Open/enable the GC log for the same node and time window. Enable / enhance GC and safepoint logging (Java 9+) via system.yaml extraJavaOpts for the affected service, then restart that service:
## Example for Artifactory — adjust path for your install layout
shared:
extraJavaOpts:
“-Xlog:gc*=info,safepoint*=info:file=/opt/jfrog/artifactory/var/log/artifactory-jvm-gc.log:time,uptime,level,tags:filecount=10,filesize=50M”
For Mission Control or Access, use the corresponding service section (mc: / access:) and an appropriate log path under that service’s var/log.
Important:
Append to existing extraJavaOpts; do not remove heap/-Xmx settings already in place.
How to read a long STW pause in the GC log: G1 pause summary (duration is the last value in ms; also check Real):
[795832.484s][info][gc ] GC(14570) Pause Young (Normal) (G1 Evacuation Pause)
1989M->2032M(8192M) 48364.124ms
[795832.484s][info][gc,cpu] GC(14570) User=0.69s Sys=1.27s Real=48.36s
Interpretation:
Pattern
|
Meaning
|
Real multi-second, and User + Sys ≪ Real
|
Wall-clock STW was long, but the JVM did little CPU work. Most of the pause was waiting — commonly swap page-ins, disk I/O, or OS scheduling. Heap may barely move (e.g. 1989M->2032M).
|
Real multi-second, and User + Sys ≈ Real
|
The pause was mostly CPU-bound GC work (marking/copying/evacuation on CPU). Still an application STW (readiness/probes stall), but the next step is heap sizing / allocation rate / GC tuning — not primarily swap.
|
High Sys relative to User (with long Real)
|
Notable time in the kernel — often page faults, I/O, or other syscalls (swap and storage pressure show up here).
|
High User relative to Sys (with long Real)
|
GC threads were busy in user-space GC work (CPU-heavy collection).
|
Step 4: Remediation — reduce or eliminate long STW pauses
Long STW means application threads (including readiness) were stopped. Fix depends on why wall time was long:
A. Wait-dominated pause (Real ≫ User + Sys) - Likely: swap, storage latency, heavy page faults, CPU steal/throttling, noisy neighbor.
Check: free -h, swapon, vmstat (si/so), iostat, %steal / mpstat.
Remediate:
-
Reduce/eliminate swap (vm.swappiness=1 or disable swap on dedicated nodes)
-
Fit combined JVM RSS in RAM; add RAM or lower -Xmx
-
Fix slow disks / I/O saturation
-
Address CPU steal / oversubscribed VMs if %steal is high
B. CPU-dominated GC pause (User + Sys ≈ Real, often high User) - Likely: large live heap, allocation spikes, too-small heap causing frequent/expensive collections, humongous objects, lingering Full/long Remark.
Check: GC log — pause frequency, heap before/after, humongous regions, allocation rate.
Remediate:
-
Right-size heap (enough headroom; not so large it promotes huge STW on a small host)
-
Reduce allocation churn / large object spikes with Java JFR if identifiable
-
Ensure single node isn’t oversubscribed on CPU during GC
-
Optional after sizing: -XX:+AlwaysPreTouch
Step 5: If unhealthy windows continue after remediation, open a ticket with JFrog Support and include:
-
Router logs covering the unhealthy ↔ healthy transitions
-
Service request/readiness logs showing the probe gap/burst
-
GC logs with unified logging as above (gc* + safepoint*)
-
A Java Flight Recorder (JFR) capture spanning an incident (or a representative load period).
-
Enable in system.yaml:
shared:
extraJavaOpts:
"-XX:StartFlightRecording=name=jfrog-stw,settings=default,disk=true,maxage=2h,filename=/opt/jfrog/artifactory/var/log/jfrog-stw.jfr"
-
Run /opt/jfrog/artifactory/app/third-party/java/bin/jcmd when unresponsiveness manifests:
/opt/jfrog/artifactory/app/third-party/java/bin/jcmd -l
/opt/jfrog/artifactory/app/third-party/java/bin/jcmd <PID> JFR.check
/opt/jfrog/artifactory/app/third-party/java/bin/jcmd <PID> JFR.dump name=jfrog-stw
filename=/opt/jfrog/artifactory/var/log/jfrog-stw-dump.jfr