RabbitMQ (Split) Upgrade Guide

Challenges of Default Xray Installer Deployment in Large Clusters

Xray installers, by default, install the Application services and Rabbit MQ on the same node, where the application connects to the local RMQ instance for message publishing and consumption. We use classic queues and mirroring to all nodes in the cluster. This deployment works well with smaller clusters. However, this could cause issues for large clusters due to synchronised writes, synchronisation latency, resource overhead, etc. Also, with high replication, the chances of network partitions increase. In a partitioned scenario, the cluster might split, leading to inconsistent states across different nodes, further complicating message delivery and queue management.

The below diagram will consider the example setup, a configuration that contains 3 nodes of Xray and MQ on every node, with an external Postgres DB. We will explore how to tune this setup for better stability and resource utilisation.

The recommendation is to re-design it as follows: move to a 3-node cluster for X-ray and a three-node cluster for Rabbit MQ. This re-design offers several benefits, including improved stability, better resource utilisation, and enhanced scalability.

Note

Important - This model of deployment, with separate application and rabbit-mq cluster

Installation Procedure

Upgrade can be done in two ways. Transform the existing cluster by dedicating some nodes to RabbitMQ and converting the remaining nodes into an Xay only setup.

Upgrade an existing node to a RabbitMQ only node
1. Stop Xray and RabbitMQ.
2. Run the installer to deploy only RabbitMQ.
3. Verify that the RabbitMQ is online and is part of the cluster.
Upgrade an existing node to an Xray only node
1. Stop Xray and RabbitMQ.
2. Disconnect the RabbitMQ from the cluster.
3. Update the system.yaml (Provide the RabbitMQ urls )
4. Run the installer to deploy only Xray Instances.

Note

Note: Please refer the Challenges of Default Xray Installer Deployment in Large Clusters for more details

RPM/ DEB Upgrade

1) Upgrade an existing node to a RabbitMQ only node (RPM/DEB)

This is a scenario where you need to update one of your nodes to a RabbitMQ only node

Follow the below steps to upgrade.

Download the newer Xray RPM/DEB based installer(tar.gz) version from the JFrog Xray Downloads page.
Extract the tar.gz file:
```
tar -xvf jfrog-xray-<version>-rpm/deb.tar.gz
```
Note
Note: From 3.107 onwards, JFrog Xray installers have organized files into designated subfolders. After extracting the tar.gz file, ensure you navigate to the xray directory.
```
cd jfrog-xray-<version>-rpm/deb
```
Stop Xray and RabbitMQ.
Add autoStop: true flag and erlang cookie under RabbitMQ block in Xray system.yaml . The default cookie can be found under `xray/app/third-party/rabbitmq/.erlang.cookie`
```
shared:
    rabbitMq:
        autoStop: true
        erlangCookie:
            value:
```
Then, run the below command.
```
systemctl stop xray.service
```
Before proceeding, ensure the Xray services are shut down gracefully (it may take a while when the system is running)
Run the installer script.
```
./install.sh -t rabbitmq
```
Start RabbitMQ service.
```
systemctl start xray-rabbitmq.service
```
RabbitMQ service should start; you can verify it by checking the status.
```
systemctl status xray-rabbitmq.service
```

2) Upgrade an existing node to an Xray only node (RPM/DEB)

This is a scenario where you must update one node to an Xray only node.

Follow the below steps to upgrade.

Download the newer Xray RPM/DEB-based installer(tar.gz) version from the JFrog Xray Downloads page.
Extract the tar.gz file:
```
tar -xvf jfrog-xray-<version>-rpm/deb.tar.gz
```
Note
Note: From 3.107 onwards, JFrog Xray installers have organized files into designated subfolders. After extracting the tar.gz file, ensure you navigate to the xray directory.
```
cd jfrog-xray-<version>-rpm/deb
```
Perform this step if you are upgrading from a node that has Xray + RabbitMQ Disconnect RabbitMQ from Cluster.
1. Switch to Xray user (default: xray) and navigate to rabbitmq/sbin directory.
```
su xray
cd /opt/jfrog/xray/app/third-party/rabbitmq/sbin/
```
2. We will stop and remove the RabbitMQ node from the cluster in this step. Replace the <current-node> with the node name you are working on. You can use the cluster_status command to get the correct node name. And the <active-node> can be any other node in the cluster still online.
```
./rabbitmqctl cluster_status
./rabbitmqctl stop_app -n <current-node>
./rabbitmqctl forget_cluster_node <current-node> --node  <active-node>
```
3. Verify the node is not part of the cluster and no RabbitMQ services are running.
```
./rabbitmqctl cluster_status --node <active-node>
ps -aux | grep erl
ps -aux | grep epmd
```
Stop Xray service
```
systemctl stop xray.service
```
Before proceeding, ensure the Xray services are shut down gracefully (it may take a while when the system is running)
Run the installer script.
```
./install.sh -t xray
```
Before starting the Xray service, update the system.yaml file with the RabbitMQ cluster URLs and credentials for all nodes in the cluster. Ensure that all RabbitMQ node URLs are correctly listed and include the appropriate username and password for authentication. This configuration ensures Xray can communicate effectively with the RabbitMQ cluster and save the changes.
Change the order of the URLs to optimize load balancing and prevent all Xray nodes from connecting to the same RabbitMQ node. For example, if the RabbitMQ node URL order in the first Xray node is 1,2,3, then in the second node, it should be 2,3,1, and in the third node, it could be 3,1,2. This ensures that each Xray node connects to different RabbitMQ nodes, improving fault tolerance and distributing the load efficiently.
```
vi /var/opt/jfrog/xray/etc/system.yaml
```
Note
Note: If using a custom RabbitMQ password, ensure the same password is configured in Xray’s system.yaml file under the RabbitMQ section to maintain consistency.
Start Xray service:
```
systemctl start xray.service
```

Linux Archive Upgrade

1) Upgrade an existing node to a RabbitMQ only node (Linux Archive)

This is a scenario where you need to update one of your nodes to a RabbitMQ only node

Follow the below steps to upgrade.

Stop Xray and RabbitMQ.Add autoStop: true flag and erlang cookie under RabbitMQ block in Xray system.yaml . The default cookie can be found under `xray/app/third-party/rabbitmq/.erlang.cookie`
```
shared:
    rabbitMq:
        autoStop: true
        erlangCookie:
            value:
```
Then, run the below command.
```
cd $JFROG_HOME/xray/app/bin
./xray.sh stop ## as xray user
```
Or via service
```
systemctl stop xray.service
```
Wait for the services started by the Xray user to stop. Otherwise, manually kill all processes started by the Xray user.

Extract the contents of the compressed archive and go to the extracted folder.

mv jfrog-xray-<version>-linux.tar.gz /opt/jfrog/
cd /opt/jfrog
tar -xf jfrog-xray-<version>-linux.tar.gz

Replace the existing $JFROG_HOME/xray/app with the new app folder.For eg:

# Export variables to simplify commands
export JFROG_HOME=/opt/jfrog   ## this is your old xray dir
export JF_NEW_VERSION=/opt/jfrog/jfrog-xray-<version>-linux

# Remove app
rm -rf $JFROG_HOME/xray/app

# Copy new app
cp -fr $JF_NEW_VERSION/app $JFROG_HOME/xray/

# Make sure to chown again to set the right permissions for xray user
eg: chown -R xray:xray /opt/jfrog/xray

# Remove extracted new version
rm -rf $JF_NEW_VERSION

Start Xray service

xray/app/bin/xray.sh start|stop  ## As xray user

Or via service

systemctl <start|stop|status> xray.service

2) Upgrade an existing node to an Xray only node (Linux Archive)

This is a scenario where you must update one node to an Xray only node.

Follow the below steps to upgrade.

Perform this step if you are upgrading from a node that has Xray + RabbitMQ Disconnect RabbitMQ from Cluster.
1. Switch to Xray user (default: xray) and navigate to rabbitmq/sbin directory.
```
su xray
cd xray/app/third-party/rabbitmq/sbin/
```
2. We will stop and remove the RabbitMQ node from the cluster in this step. Replace the <current-node> with the node name you are working on. You can use the cluster_status command to get the correct node name. And the <active-node> can be any other node in the cluster still online.
```
./rabbitmqctl cluster_status
./rabbitmqctl stop_app -n <current-node>
./rabbitmqctl forget_cluster_node <current-node> --node  <active-node>
```
3. Verify the node is not part of the cluster and no RabbitMQ services are running.
```
./rabbitmqctl cluster_status --node <active-node>
ps -aux | grep erl
ps -aux | grep epmd
```
Stop the current server.
```
cd $JFROG_HOME/xray/app/bin
./xray.sh stop ## as xray user
```
Or via service
```
systemctl stop xray.service
```
Wait for the services started by the Xray user to stop. Otherwise manually kill all processes started by the Xray user.

Extract the contents of the compressed archive and go to the extracted folder.

mv jfrog-xray-<version>-linux.tar.gz /opt/jfrog/
cd /opt/jfrog
tar -xf jfrog-xray-<version>-linux.tar.gz

Replace the existing $JFROG_HOME/xray/app with the new app folder.

For eg:

# Export variables to simplify commands
export JFROG_HOME=/opt/jfrog   ## this is your old xray dir
export JF_NEW_VERSION=/opt/jfrog/jfrog-xray-<version>-linux

# Remove app
rm -rf $JFROG_HOME/xray/app

# Copy new app
cp -fr $JF_NEW_VERSION/app $JFROG_HOME/xray/

# Make sure to chown again to set the right permissions for xray user
eg: chown -R xray:xray /opt/jfrog/xray

# Remove extracted new version
rm -rf $JF_NEW_VERSION

Before starting the Xray service, update the system.yaml file with the RabbitMQ cluster URLs and credentials for all nodes in the cluster. Ensure that all RabbitMQ node URLs are correctly listed and include the appropriate username and password for authentication. This configuration ensures Xray can communicate effectively with the RabbitMQ cluster and save the changes.
Change the order of the URLs to optimize load balancing and prevent all Xray nodes from connecting to the same RabbitMQ node. For example, if the RabbitMQ node URL order in the first Xray node is 1,2,3, then in the second node, it should be 2,3,1, and in the third node, it could be 3,1,2. This ensures that each Xray node connects to different RabbitMQ nodes, improving fault tolerance and distributing the load efficiently.
```
vi $JFROG_HOME/xray/var/etc/system.yaml
```
Note
Note: If using a custom RabbitMQ password, ensure the same password is configured in Xray’s system.yaml file under the RabbitMQ section to maintain consistency.
Edit the file $JFROG_HOME/xray/var/etc/installerState.yaml and add the below line at the root level if not there or set it to a different value. Please note: This step is essential only to start xray services
```
installation_method: xray
```

Start Xray service:

xray/app/bin/xray.sh start|stop  ## As xray user

Or via service

systemctl <start|stop|status> xray.service

Docker Compose Upgrade

1) Upgrade an existing node to a RabbitMQ only node (Docker compose)

This is a scenario where you need to update one of your nodes to a RabbitMQ only node

Follow the below steps to upgrade.

Stop the service.

For Xray services

cd jfrog-xray-<version>-compose
docker-compose -p xray down

For RabbitMQ services

cd jfrog-xray-<version>-compose
docker-compose -p xray-rabbitmq down

Download the newer Xray docker compose based installer(tar.gz) from the JFrog Xray Downloads page.
Extract the newer tar.gz file:
```
tar -xvf jfrog-xray-<version>-compose.tar.gz
```
Note
Note: From 3.107 onwards, JFrog Xray installers have organized files into designated subfolders. After extracting the tar.gz file, ensure you navigate to the xray directory.
```
cd jfrog-xray-<version>-compose
```
Note
Note: For Docker Compose upgrades, merge any customizations if you have done any in your current docker-compose.yaml file to the new extracted version of the docker-compose.yaml file.
Copy the contents of the .env file in the previous installation to the newly created .env file in this archive without copying the versions, as this will affect the upgrade.
Run the installer script.
```
./config.sh -t rabbitmq
```
Note
Note: If you use a custom password for RabbitMQ, ensure the rabbitmq conf file has the correct password set before starting the service. Please refer here for more details.

Start RabbitMQ service.

docker-compose -p xray-rabbitmq -f docker-compose-rabbitmq.yaml up -d

RabbitMQ service should start; you can verify it by checking the logs
```
docker ps
docker logs -f xray_rabbitmq
```
(Optional) Access the UI through a browser to verify if the RabbitMQ nodes are up and running
Note
Note: The credentials provided below are the default ones. If you have customized the RabbitMQ password in’/opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf’, ensure you use the correct password when accessing the management UI.
```
http://<your-node-ip>:15672

Username:  guest
Password:  JFXR_RABBITMQ_COOKIE
```

2) Upgrade an existing node to an Xray only node (Docker compose)

This is a scenario where you must update one node to an Xray only node.

Follow the below steps to upgrade.

Stop the service.

For Xray services

cd jfrog-xray-<version>-compose
docker-compose -p xray down

For RabbitMQ services

cd jfrog-xray-<version>-compose
docker-compose -p xray-rabbitmq down

Download the newer xray docker compose based installer(tar.gz) from the JFrog Xray Downloads page.
Extract the newer tar.gz file:
```
tar -xvf jfrog-xray-<version>-compose.tar.gz
```
Note
Note: From 3.107 onwards, JFrog Xray installers have organized files into designated subfolders. After extracting the tar.gz file, ensure you navigate to the xray directory.
```
cd jfrog-xray-<version>-compose
```
Note
Note: For Docker Compose upgrades, make sure to merge any customizations if you have done any in your current docker-compose.yaml file to the new extracted version of the docker-compose.yaml file.
Copy the contents of the .env file in the previous installation to the newly created .env file in this archive without copying the versions, as this will affect the upgrade.
Run the installer script.
```
./config.sh -t xray
```
Upon running the command, you will be prompted with a question.
1. Have you disconnected Artifactory Xray pairings, except one prior to performing this upgrade (Refer http://service.jfrog.org/wiki/Xray+and+Artifactory+One+to+One+Pairing for more details) ? - You can choose y. This step was added for migrating from older 2.x to 3.x versions. If you are already on Xray 3.x, type y.
Before starting the Xray service, update the system.yaml file with the RabbitMQ cluster URLs and credentials for all nodes in the cluster. Ensure that all RabbitMQ node URLs are correctly listed and include the appropriate username and password for authentication. This configuration ensures Xray can communicate effectively with the RabbitMQ cluster and save the changes.
Change the order of the URLs to optimize load balancing and prevent all Xray nodes from connecting to the same RabbitMQ node. For example, if the RabbitMQ node URL order in the first Xray node is 1,2,3, then in the second node, it should be 2,3,1, and in the third node, it could be 3,1,2. This ensures that each Xray node connects to different RabbitMQ nodes, improving fault tolerance and distributing the load efficiently.
```
vi /root/.jfrog/xray/var/etc/system.yaml
```
Note
Note: If using a custom RabbitMQ password, ensure the same password is configured in Xray’s system.yaml file under the RabbitMQ section.
Start Xray service.
```
docker-compose -p xray up -d
```

Xray services should start; you can verify it by checking the logs.

docker ps
docker logs -f xray_router ## or any other xray services.