XRAY: Why is one of the Rabbitmq nodes in Xray not joining the existing cluster?

If one of the rabbitmq node in Xray HA keeps crashing and it can not join the existing cluster, follow the explanation/guide below:

You may see the following message in rabbitmq's crash.log

2022-08-XX 21:28:31.245 [error] <0.1158.0> Mnesia(rabbit@host_03): ** ERROR ** Mnesia on rabbit@host_03 could not connect to node(s) [rabbit@host_02]

Solution

Make sure that the rabbitmq section in system.yaml is correct.
It should be the following for a secondary node (the primary node is different). 
e.g.

shared:
  rabbitMq:
       autoStop: true
       active:
            node:
                name: <hostname of the active node>

Please see here - https://www.jfrog.com/confluence/display/JFROG/Installing+Xray#InstallingXray-HAInstallation

Further, to troubleshoot this issue, you can use the telnet command for a connection test and see if there is a firewall or other error (e.g. Unknown host error)
e.g.

telnet host_02 25672
Note that 25672 port is the rabbmitmq's clustering port and inter-node communication purpose.

If you see the telnet command is hanging, you should deal with the firewall settings and check that you can reach the proper host and port.
If you see the Unknown host error, it is DNS (Domain Name Server) issue. you should check the Network team Or you need to check /etc/hosts file and see if there is IP and hostname of the target node exist in /etc/hosts file
e.g.

10.xx.xxx.xx host_02