ARTIFACTORY: How to fix Node Outage Due to Front-End Service Out of Memory

Introduction

Issue: Can not access the UI

The issue was traced back to the front-end service running out of memory, which led to the service being terminated unexpectedly. This resulted in the front-end of the affected node becoming unresponsive.

JavaScript heap out of memory message in console.log

<--- Last few GCs --->

[122913:0x572a230] 5444285197 ms: Mark-sweep (reduce) 2036.5 (2084.0) -> 2035.1 (2084.0) MB, 947.6 / 0.1 ms  (average mu = 0.162, current mu = 0.132) allocation failure scavenge might not succeed
[122913:0x572a230] 5444286228 ms: Mark-sweep (reduce) 2036.4 (2084.0) -> 2035.2 (2084.0) MB, 1020.5 / 0.2 ms  (average mu = 0.090, current mu = 0.010) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb0a860 node::Abort() [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 2: 0xa1c193 node::FatalError(char const*, char const*) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 3: 0xcf9a6e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 4: 0xcf9de7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 5: 0xeb1685  [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 6: 0xeb2166  [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 7: 0xec068e  [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 8: 0xec10d0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
 9: 0xec404e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
10: 0xe8558a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
11: 0x11fe2d6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/jfrog/artifactory/app/third-party/node/bin/node]
12: 0x15f2d39  [/opt/jfrog/artifactory/app/third-party/node/bin/node]
2023-11-14T11:57:08.824Z [36m[jfrou][0m [31;1m[ERROR][0m [3475135c66c6f3bc] [healthcheck.go:66             ] [main                ] [] - Checking health of service 'jffe_000-art2' using URL 'http://localhost:8070/readiness' returned an error: Get "http://localhost:8070/readiness": dial tcp [::1]:8070: connect: connection refused

The root cause of the outage was identified as an Out of Memory (OOM) condition within the front-end service. The OOM situation led to the failure of the JavaScript heap, which triggered a series of errors and ultimately caused the service to terminate.


Resolution

To prevent similar incidents in the future, the following solutions are recommended:

1. Upgrade Artifactory:
Upgrade Artifactory to the latest version, such as version 7.90.X. The latest versions include optimizations and fixes that can help mitigate memory leaks and improve overall stability.
2. Scheduled Restarts:
Implement regular restarts of the Artifactory services to prevent memory leaks from accumulating over time. This can help maintain a stable environment and reduce the likelihood of similar OOM issues.


Conclusion

By either upgrading to the latest version of Artifactory or implementing regular restarts, the risk of front-end service failures due to out-of-memory conditions can be significantly reduced, ensuring better reliability and uptime for the system.