The Issue
Artifactory 7.x supports ipv6 and in an environment with ipv6 enabled, it will automatically bind to both ipv4 and ipv6 addresses.
In such case, tomcat will use ipv6 dns lookup method Java_java_net_Inet6AddressImpl_lookupAllHostAddr instead of Java_java_net_Inet4AddressImpl_lookupAllHostAddr
We have discovered that in some kubernete environments, such method could failed DNS look up and crash JVM and this will create random restart on your artifactory pod.
Identify the issue
To identify the issue, we first need to look at the core dump that’s automatically generated during and JVM crash.
Stack: [0x00007f81746e1111,0x00007f81747e2111], sp=0x00007c82747dc075, free space=2000k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0xe5281] gaih_inet.constprop.7+0x311
C [libc.so.6+0xe6d1b] getaddrinfo+0x12b
C [libnet.so+0x5e56] Java_java_net_Inet6AddressImpl_lookupAllHostAddr+0x96
j java.net.Inet6AddressImpl.lookupAllHostAddr(Ljava/lang/String;)[Ljava/net/InetAddress;+0 java.base@11.0.13
Next, we can run the following command to further identify if the kubernete cluster is indeed with ipv6 enabled and if artifactory is on ipv6:
bash-4.4$ cat /etc/hosts
# Kubernetes-managed hosts file (host network).
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Netstat command shows artifactory java is on tcp6
bash-4.4$ netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.1:8046 0.0.0.0:* LISTEN 4130/jf-router
tcp 0 0 127.0.0.1:8047 0.0.0.0:* LISTEN 4130/jf-router
tcp 0 0 127.0.0.1:8049 0.0.0.0:* LISTEN 4130/jf-router
tcp 0 0 127.0.0.1:8086 0.0.0.0:* LISTEN 4350/jf-metadata
tcp 0 0 127.0.0.1:8061 0.0.0.0:* LISTEN 4521/jf-event
tcp 0 0 127.0.0.1:8062 0.0.0.0:* LISTEN 4521/jf-event
tcp 0 0 127.0.0.1:8036 0.0.0.0:* LISTEN 4864/jf-observabili
tcp 0 0 127.0.0.1:8070 0.0.0.0:* LISTEN 4883/node
tcp 0 0 127.0.0.1:8071 0.0.0.0:* LISTEN 5039/jf-integration
tcp 0 0 127.0.0.1:8072 0.0.0.0:* LISTEN 5039/jf-integration
tcp6 0 0 :::8045 :::* LISTEN 5286/java
tcp6 0 0 127.0.0.1:8015 :::* LISTEN 5286/java
tcp6 0 0 :::8081 :::* LISTEN 5286/java
tcp6 0 0 :::8082 :::* LISTEN 4130/jf-router
tcp6 0 0 127.0.0.1:8091 :::* LISTEN 5286/java
tcp6 0 0 127.0.0.1:8040 :::* LISTEN 5286/java
Workaround
The work around to this issue is to force tomcat to use ipv4 over ipv6 with javaoption
-Djava.net.preferIPv4Stack=true
We can pass this to artifactory system.yaml under
shared:
extraJavaOpts: "-Djava.net.preferIPv4Stack=true"
You can change system.yaml setting directly in the secret as a temporary workaround:
1. Get systemYaml secret:
kubectl get secret d-artifactory-systemyaml -o jsonpath="{.data}"
2. Base64 decode the value to get the systemYaml content. Edit the extraJavaOpts key to add the following value ( If extraJavaOpts does not exist, you can add one):
shared:
extraJavaOpts: "-Djava.net.preferIPv4Stack=true"
3. Base64 encode the modified systemYaml content again and run
kubectl edit secret d-artifactory-systemyaml
Replace the encoded value under system.yaml
If this helps address the issue, you might want to update “javaOpts” inside your value.yaml and update your release.
Run netstat to confirm if the above settings are working. You should see:
tcp 0 0 127.0.0.1:8091 0.0.0.0:* LISTEN 5082/java
tcp 0 0 127.0.0.1:8061 0.0.0.0:* LISTEN 4225/jf-event
tcp 0 0 127.0.0.1:8030 0.0.0.0:* LISTEN 4886/jf-connect
tcp 0 0 127.0.0.1:8062 0.0.0.0:* LISTEN 4225/jf-event
tcp 0 0 127.0.0.1:8035 0.0.0.0:* LISTEN 4886/jf-connect
tcp 0 0 127.0.0.1:8036 0.0.0.0:* LISTEN 4547/jf-observabili
tcp 0 0 127.0.0.1:8070 0.0.0.0:* LISTEN 4381/node
tcp 0 0 127.0.0.1:8071 0.0.0.0:* LISTEN 4711/jf-integration
tcp 0 0 127.0.0.1:8072 0.0.0.0:* LISTEN 4711/jf-integration
tcp 0 0 127.0.0.1:8040 0.0.0.0:* LISTEN 5082/java
tcp 0 0 0.0.0.0:8045 0.0.0.0:* LISTEN 5082/java
tcp 0 0 127.0.0.1:8046 0.0.0.0:* LISTEN 3864/jf-router
tcp 0 0 127.0.0.1:8047 0.0.0.0:* LISTEN 3864/jf-router
tcp 0 0 127.0.0.1:8015 0.0.0.0:* LISTEN 5082/java
tcp 0 0 127.0.0.1:8049 0.0.0.0:* LISTEN 3864/jf-router
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 5082/java
tcp 0 0 127.0.0.1:8086 0.0.0.0:* LISTEN 4069/jf-metadata
tcp6 0 0 :::8082 :::* LISTEN 3864/jf-router