Why Won’t a Kubernetes Xray Work with a Non-Kubernetes Artifactory?

Patrick Russell
2021-04-06 17:57

Relevant Versions: Artifactory 7.X and Xray 3.X

It's sometimes the case that Artifactory will be installed on a Virtual Machine, while the user wants Xray to be installed on Kubernetes. For example, Artifactory could have been installed long ago on a VM, and was upgraded to 7.X, while the Xray 3.X will be a fresh installation. 

Since Kubernetes can handle Xray's microservices easily, why not go with a K8 Xray install?

Resolution

The answer lies in an often overlooked System Requirement of the JFrog Platform: Artifactory and Xray need to be on the same internal network

In other words, Artifactory's host needs a direct connection to Xray's host, and vice-versa (Xray needs to reach Artifactory). Direct internal communication needs to be possible. 

In most other installation cases, this requirement presents no problem. For example, take an Artifactory installed with Docker-Compose and an Xray installed via the Debian / RPM installer

In these kinds of setups, the internal IP addresses work as expected:

Note that trying to use the Docker Container's IP will not work (This would be set in the Artifactory system.yaml's shared.node.ip argument):


 

Xray cannot connect to Artifactory because the Artifactory IP address is not available to Xray's Router microservice. The Artifactory IP address is sent when Xray connects over the JFrog URL, and is used for subsequent health checks.

In past versions such as Artifactory 6.X and Xray 2.X, communications were done over the JFrog URL only. In these earlier versions, there was also an Xray Base URL, which was how Artifactory communicated over to Xray. This allowed for things such as Load Balancers and Reverse Proxies to be between the two services.

In the current Platform versions of Artifactory and Xray, communications are done using the service's IP Address. In a lot of cases this is an Internal IP Address, while the JFrog URL is an External IP address. There is no straightforward approach to changing this behavior.

This presents a problem when trying to run just one service in Kubernetes, because Kubernetes-hosted pods have both an Internal IP and an External IP:

In this illustration, the Xray-Server microservice has the internal K8 IP of 172.0.0.9, with an external K8 IP of 10.0.0.9. After an initial cluster registration completes (Using the JFrog URL), Artifactory will try to run health checks to the Xray Internal IP.

This results in timeouts and connection errors recorded in the Artifactory Router logs.

In other circumstances, it should be possible to update the Xray system.yaml's shared.node.ip argument to use the External K8 IP. However, because of Kubernetes there are problems with this approach. Mainly that the External IP address is by default dynamically allocated. Also, a single LoadBalancer setting can proxy multiple internal pods.

Because of these limitations, you should consider a Kubernetes host to be its own internal network. It's not compatible with a different network, even in the same Cloud environment.

You should not try installing one of these services in Kubernetes if other services are not in this same K8 network. It has to be either all-kubernetes, or not use kubernetes at all.