Relevant versions: Xray 2.X and Xray 3.X - Instructions and paths are for 3.X but the overall process is similar in 2.X
Xray HA requires a RabbitMQ broker per node (i.e., if you have two Xray nodes, you’ll need two RabbitMQ brokers). During setup, if the ports are not open or your machines are not communicating properly, RabbitMQ may fail to cluster correctly. This will result in messages getting lost and an unusable Xray HA.
Diagnosing the Issue
Let's take a look at a broken RabbitMQ cluster. In the example below, we'll use the command, rabbitmqctl cluster_status on two nodes each. Both nodes should print the same thing, but in this problem example, they won't.
First, you generally need to be the Xray Linux user to run these rabbitmqctl commands. The binary to use is in the Xray_Home "app" folder, usually, it can be found here:
# As the root user, log in as the Xray account
sudo su - xray
# Navigate to the Xray App folder
# The RabbitMQ binaries are within the third-party/rabbitmq/sbin folder
cd /opt/jfrog/xray/app/third-party/rabbitmq/sbin
#Call the cluster_status command using the ctl binary in the sbin folder
./rabbitmqctl cluster_status
In this broken setup, the Cluster Name and other fields for each node are different, each Running Nodes section shows only a single node as the sole node in operation.
The following are some of the Xray errors you might encounter when trying to edit and save your indexed repositories:
[2018/05/22 23:34:39 UTC] [EROR] (jfrog.com/xray/service/permission_service.(*PermissionService).updateOtherNodesAndUiClients:719)
Failed to reload permissions cache on other nodes: timeout waiting for reply to sync action 'ReloadPermissionCache' from nodes [eplus-xray]
[2018/05/22 23:34:39 UTC] [EROR] (jfrog.com/xray/service/permission_service.(*PermissionService).RemoveResources:397)
Failed to update other nodes / ui clients: timeout waiting for reply to sync action 'ReloadPermissionCache' from nodes [eplus-xray]
[2018/05/22 23:34:39 UTC] [EROR] (jfrog.com/xray/handlers/binary_managers.BinManagerHandler.SendRepos:490)
Failed to remove specific repositories from permissions: timeout waiting for a reply to sync action 'ReloadPermissionCache' from nodes [eplus-xray]
Reclustering
If you see these symptoms or are experiencing any of these problems, here’s our recommended solution.
1] First, check the network.
Please ensure the following ports are open for RabbitMQ communications:
-
4369 (epmd, a peer discovery service used by RabbitMQ nodes and CLI tools)
-
5672 (RabbitMQ’s listening port)
-
25672 (RabbitMQ’s inter-node HA port)
-
15672 (RabbitMQ’s dashboard UI and HTTP API port, which is optional, but helpful)
Ensure that each of your machines can reach all of those ports via telnet, cURL may not work since RabbitMQ uses AMP rather than HTTP.
Also ensure that the RabbitMQ Hostname is correct. RabbitMQ uses the "hostname -s" command to determine what its hostname should be. In some environments this hostname won't work; an IP address or other URL needs to be used instead. The hostname is used in the join_cluster command outlined below.
2] Choose one of the installed Xray nodes to be the Primary. Don't do anything else on your "primary" Xray node. If you're unsure which is supposed to be the primary, pick one arbitrarily. You'll have the other nodes join this Primary's cluster.
3] On the remaining Xray nodes, execute the following Rabbitmqctl commands:
#Halt the local RabbitMQ "app" - The Rabbitmq server itself should be running still
./rabbitmqctl stop_app
#Join the Primary's cluster - I.E. run this command from xray-2
./rabbitmqctl join_cluster rabbit@xray-1
#Start the local RabbitMQ "app"
./rabbitmqctl start_app
4] Repeat step [3] on all Xray nodes until all of them have joined the Primary cluster
5] Verify the cluster state by re-running cluster_status. There should now be all the nodes in the printout in both places:
./rabbitmqctl cluster_status
Basics
Cluster name: rabbit@xray-1.us-central1-c.internal
Disk Nodes
rabbit@xray-1
rabbit@xray-2
Running Nodes
rabbit@xray-1
rabbit@xray-2
[...]
If you get an Erlang distribution error, it means you have an Erlang cookie mismatch in your /var/lib/rabbitmq/.erlang.cookie file. As RabbitMQ requires the cookies to be identical, copy the contents of the cookie from your Primary node to all of your other nodes, then restart the service:
[$XRAY_HOME/app/third-party/rabbitmq/sbin/]
./rabbitmqctl stop
rabbitmq-server -detached
Then, similar to the steps above:
# Stop the RabbitMQ app:
rabbitmqctl stop_app
# Reset the app:
rabbitmqctl reset
# Cluster the app:
rabbitmqctl join_cluster rabbit@<Hostname>
# Restart the app:
rabbitmqctl start_app
# Remirror you queues:
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'Don't forget to check the rabbitmqctl cluster_status afterward.
Looking at the Queues section under the RabbitMQ dashboard, we can see that each node will display 'ha-all' in the features column, indicating that the policy for syncing queues is in place.
In the Queues section of your RabbitMQ dashboard, confirm that each of your nodes is displaying ha-all in the Features column. This will indicate that your syncing queues policy is in place and functioning properly:
You can also execute a DB sync, and follow its progress by directly accessing both servers in the UI. The progress bar should update for both (nearly) simultaneously.
More information, from RabbitMQ’s Clustering Guide, is available here.