Diagnosing the Issue

How to Recluster RabbitMQ HA for Xray HA

AuthorFullName__c
Loren Yeung
articleNumber
000004139
ft:sourceType
Salesforce
FirstPublishedDate
2019-07-08T22:43:34Z
lastModifiedDate
2024-03-10T07:47:42Z
VersionNumber
10

Let's take a look at a broken RabbitMQ cluster. In the example below, we'll use the command, rabbitmqctl cluster_status on two nodes each. Both nodes should print the same thing, but in this problem example, they won't.

First, you generally need to be the Xray Linux user to run these rabbitmqctl commands. The binary to use is in the Xray_Home "app" folder, usually, it can be found here:
 

# As the root user, log in as the Xray account
sudo su - xray

# Navigate to the Xray App folder
# The RabbitMQ binaries are within the third-party/rabbitmq/sbin folder
cd /opt/jfrog/xray/app/third-party/rabbitmq/sbin

#Call the cluster_status command using the ctl binary in the sbin folder
./rabbitmqctl cluster_status


In this broken setup, the Cluster Name and other fields for each node are different, each Running Nodes section shows only a single node as the sole node in operation.
 

image


The following are some of the Xray errors you might encounter when trying to edit and save your indexed repositories:

[2018/05/22 23:34:39 UTC] [EROR] (jfrog.com/xray/service/permission_service.(*PermissionService).updateOtherNodesAndUiClients:719)
Failed to reload permissions cache on other nodes: timeout waiting for reply to sync action 'ReloadPermissionCache' from nodes [eplus-xray]

[2018/05/22 23:34:39 UTC] [EROR] (jfrog.com/xray/service/permission_service.(*PermissionService).RemoveResources:397)
Failed to update other nodes / ui clients: timeout waiting for reply to sync action 'ReloadPermissionCache' from nodes [eplus-xray]

[2018/05/22 23:34:39 UTC] [EROR] (jfrog.com/xray/handlers/binary_managers.BinManagerHandler.SendRepos:490)
Failed to remove specific repositories from permissions: timeout waiting for a reply to sync action 'ReloadPermissionCache' from nodes [eplus-xray]