Xray Helm Upgrade

Note

When upgrading from Xray chart version 3.x to 3.x, due to breaking changes, use kubectl delete statefulsets <old_statefulset_xray_name> and run the helm upgrade.

Also, While upgrading from Xray chart version 3.x to 4.x, due to breaking RabbitMQ (whenrabbitmq.enabled=true) subchart changes, run the following commands.

$ kubectl delete statefulsets <old_statefulset_xray_name>
$ kubectl delete statefulsets <old_statefulset_rabbitmq_name>
$ kubectl delete pvc <old_PVC_rabbitmq_name>
$ helm upgrade --install xray --namespace xray jfrog/xray

Note

Downtime is required to perform an upgrade.

Offline Update

If you wish to do an offline update of the Xray database while running in pods, copy the downloaded data into the correct Xray container (scale down to 1 replica if needed before) and then using kubectl commands copy the file to the specific location.

For example:

kubectl cp ./comp_-6795364578871.zip <namespace>/xray-0:/opt/jfrog/xray/var/work/server/updates/component/comp_-6795364578871.zip -c xray-server
kubectl cp ./vuln_-6795364578871.zip <namespace>/xray-0:/opt/jfrog/xray/var/work/server/updates/vulnerability/vuln_-6795364578871.zip -c xray-server

Upgrading to Xray 3.70.0 or later from an Xray version prior to 3.70.0

If you are upgrading to Xray 3.70.0 or later from an Xray version prior to 3.70.0, your installation contains RabbitMQ 3.8.x. Feature flags in RabbitMQ have to be enabled before you can upgrade, since Xray 3.70.0 and later contains RabbitMQ 3.11.x and later.

Before you proceed with the upgrade, go to RabbitMQ Upgrade to version 3.11.x and set feature flags in RabbitMQ.

To upgrade Xray.

Update the existing deployed version to the updated version.

helm repo update

helm upgrade --install <myrelease> namespace xray jfrog/xray

If Xray was installed without providing a value to postgresql.postgresqlPassword (the password was autogenerated), follow these instructions.
1. Get the current password by running the following.
```
POSTGRES_PASSWORD=$(kubectl get secret -n <namespace> <myrelease>-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
```
2. Upgrade the release by passing the previously auto-generated secret.
```
helm upgrade <myrelease> jfrog/xray --set postgresql.postgresqlPassword=${POSTGRES_PASSWORD}
```
If Xray was installed without providing a value to rabbitmq.auth.password (the password was autogenerated), follow these instructions.
1. Get the current password by running the following.
```
RABBITMQ_PASSWORD=$(kubectl get secret -n <namespace> <myrelease>-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)
```
2. Upgrade the release by passing the previously auto-generated secret.
```
helm upgrade <myrelease> jfrog/xray --set rabbitmq.auth.password=${RABBITMQ_PASSWORD}
```
If Xray was installed with all of the default values (e.g., with no user-provided values for Rabbit/Postgres), follow these steps.
1. Retrieve all current passwords (RabbitMQ/PostgreSQL) as explained in the previous section.
2. Upgrade the release by passing the previously auto-generated secrets.
```
helm upgrade --install xray --namespace xray jfrog/xray --set rabbitmq.auth.password=<rabbit-password> --set postgresql.post
```
Access Xray from your browser at: http://<jfrogUrl>/ui/, then go to the Xray tab in the Application module in the UI.
Check the status of your deployed helm releases.
```
helm status xray
```

Upgrading the Chart Versions from 1.x/2.x to 3.x and Above

JFrog Xray 3.x is only compatible with JFrog Artifactory 7.x; to upgrade, you must first install JFrog Artifactory 7.x.
To upgrade from a version prior to 1.x, you will first need to upgrade to the latest version of 1.x
To upgrade from a version of 2.x, you first need to upgrade to the latest version of 2.x.

PostgreSQL Migration Process

The recommended migration process has two main steps:

Migrate the existing MongoDB data to an existing PostgreSQL.
Perform a full database export and import of PostgreSQL.

Prerequisite Step: Get the Details of the Existing Chart

Block user access to Xray (but do not shut down).

Get the service names OLD_PG_SERVICE_NAME, OLD_MONGO_SERVICE_NAME using the command below. For example: OLD_PG_SERVICE_NAME and OLD_MONGO_SERVICE_NAME values as <OLD_RELEASE_NAME>-postgresql and <OLD_RELEASE_NAME>-mongodb respectively.

$ kubectl get svc
NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
<OLD_RELEASE_NAME>-mongodb                 ClusterIP      10.101.56.69     <none>        27017/TCP                     114m
<OLD_RELEASE_NAME>-postgresql              ClusterIP      10.101.250.74    <none>        5432/TCP                      114m
<OLD_RELEASE_NAME>-rabbitmq-ha             ClusterIP      None             <none>        15672/TCP,5672/TCP,4369/TCP   114m
<OLD_RELEASE_NAME>-rabbitmq-ha-discovery   ClusterIP      None             <none>        15672/TCP,5672/TCP,4369/TCP   114m
<OLD_RELEASE_NAME>-xray-analysis           ClusterIP      10.104.138.63    <none>        7000/TCP                      114m
<OLD_RELEASE_NAME>-xray-indexer            ClusterIP      10.106.72.163    <none>        7002/TCP                      114m
<OLD_RELEASE_NAME>-xray-persist            ClusterIP      10.103.20.33     <none>        7003/TCP                      114m
<OLD_RELEASE_NAME>-xray-server             LoadBalancer   10.105.121.175   <pending>     80:32326/TCP                  114m

Save the previous passwords OLD_PG_SERVICE_NAME, OLD_MONGO_SERVICE_NAME or extract them from the secret of the existing PostgreSQL and MongoDB pods.

#Example:
OLD_PG_PASSWORD=$(kubectl get secret -n <namespace> <OLD_RELEASE_NAME>-postgresql -o jsonpath="{.data.postgres-password}" | base64 --decode)
OLD_MONGO_PASSWORD=$(kubectl get secret -n <namespace> <OLD_RELEASE_NAME>-mongodb -o jsonpath="{.data.mongodb-password}" | base64 --decode)

Stop the old Xray pods (scale down replicas to 0). The PostgreSQL and MongoDB pods will still remain active.

$ kubectl scale statefulsets <REPLACE_OLD_RELEASE_NAME>-rabbitmq-ha <REPLACE_OLD_RELEASE_NAME>-xray-analysis <REPLACE_OLD_RELEASE_NAME>-xray-indexer <REPLACE_OLD_RELEASE_NAME>-xray-persist <REPLACE_OLD_RELEASE_NAME>-xray-server --replicas=0

Migrate the Existing MongoDB Data to an Existing PostgreSQL

Run the helm install (not upgrade) with the new version say xray-new in the following way.

Verify that all probes are disabled.

--set router.livenessProbe.enabled=false --set router.readinessProbe.enabled=false --set indexer.livenessProbe.enabled=false --set analysis.livenessProbe.enabled=false --set server.livenessProbe.enabled=false --set persist.livenessProbe.enabled=false --set indexer.readinessProbe.enabled=false --set analysis.readinessProbe.enabled=false --set server.readinessProbe.enabled=false --set persist.readinessProbe.enabled=false

Point to the previous PostgreSQL pod (user, password,DATABASE).

--set postgresql.enabled=false --set database.user=<OLD_PG_USERNAME> --set database.password=<OLD_PG_PASSWORD> --set database.url="postgres://<SERVICE_NAME_POSTGRES>:5432/xraydb?sslmode=disable"

Point to the previous MongoDB (user, password,DATABASE) pod.

--set xray.mongoUsername=<OLD_MONGO_USERNAME> --set xray.mongoPassword=<OLD_MONGO_PASSWORD> --set xray.mongoUrl="mongodb://<SERVICE_NAME_MONGODB>:27017/?authSource=xray&authMechanism=SCRAM-SHA-1"

This will trigger the migration process like in the following example.

# Create a customvalues.yaml file
 
router:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
indexer:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
analysis:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
server:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false 
persist:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
postgresql:
  enabled: false
database:
  user: <OLD_PG_USERNAME>
  password: <OLD_PG_PASSWORD>
  url: "postgres://<SERVICE_NAME_POSTGRES>:5432/xraydb?sslmode=disable"
xray:
  mongoUsername: <OLD_MONGO_USERNAME>
  mongoPassword: <OLD_MONGO_PASSWORD>
  mongoUrl: "mongodb://<SERVICE_NAME_MONGODB>:27017/?authSource=xray&authMechanism=SCRAM-SHA-1"
  masterKey: <PREVIOUS_MASTER_KEY>
  jfrogUrl: <NEW_ARTIFACTORY_URL>
  joinKey: <JOIN_KEY>
rabbitmq:
  enabled: true
  auth:
    password: <PASSWORD>
rabbitmq-ha:
  enabled: false

Apply the values.yaml file during installation.

helm upgrade --install xray-new jfrog/xray -f customvalues.yaml

Stop the new Xray pods (scale down replicas to 0). Both PostgreSQL pods still remain active.

Create a customvalues.yaml file.

replicaCount: 0
postgresql:
  postgresqlPassword: <NEW_PG_PASSWORD>
rabbitmq:
  enabled: true
  auth:
    password: <PASSWORD>
rabbitmq-ha:
  enabled: false
xray:
  masterKey: <PREVIOUS_MASTER_KEY>
  jfrogUrl: <NEW_ARTIFACTORY_URL>
  joinKey: <JOIN_KEY>
unifiedUpgradeAllowed: true
databaseUpgradeReady: true

Apply the values.yaml file during installation.

helm upgrade --install xray-new jfrog/xray -f customvalues.yaml

Perform a Full Database Export and Import of PostgreSQL

To migrate PostgreSQL data between old and new pods.

Connect to the new PostgreSQL pod (you can obtain the name by running kubectl get pods).
```
$ kubectl exec -it <NAME> bash
```
Once logged in, create a dump file from the previous database usingpg_dump, and connect to the previous PostgreSQL chart.
```
$ pg_dump -h <OLD_PG_SERVICE_NAME> -U xray DATABASE_NAME > /tmp/backup.sql
```
After running this command, you will be prompted for a password; this password is the previous chart passwordOLD_PG_PASSWORD. This operation could take some time depending on the database size.
Once you have the backup file, you can restore it with this command.
```
$ psql -U xray DATABASE_NAME < /tmp/backup.sql
```
After running the command above, you will be prompted for a password; this is the current chart password. This operation could take some time depending on the database size.

Create a customvalues.yaml file.

# Create a customvalues.yaml file
xray:
  masterKey: <PREVIOUS_MASTER_KEY>
  jfrogUrl: <NEW_ARTIFACTORY_URL>
  joinKey: <JOIN_KEY>
rabbitmq:
  enabled: true
  auth:
    password: <PASSWORD>
rabbitmq-ha:
  enabled: false
postgresql:
  postgresqlPassword: <NEW_PG_PASSWORD>
unifiedUpgradeAllowed: true
databaseUpgradeReady: true

Apply the values file during the installation.

helm upgrade --install xray-new jfrog/xray -f customvalues.yaml

Restore access to the new Xray by running the command below to remove the old Xray deployment and Helm release.
```
run helm delete <OLD_RELEASE_NAME>
```

Xray should now be ready for use.

RabbitMQ Migration Process

Note

The RabbitMQ-HA chart has been removed from the 7.x chart versions; therefore, before upgrading to the 7.x chart versions, you will need to perform the RabbitMQ migration (assuming you are not using the Bitnami RabbitMQ). This section describes the steps for migrating from the RabbitMQ-HA chart to the Bitnami RabbitMQ chart beforeupgrading to chart version 7.x and above.

For this procedure, you will need to choose whether to migrate without existing queues (assuming that all queues are empty) or to migrate while Xray is down.

Migrating without Existing Queues

Important

This procedure assumes that all queues are empty.

While running the helm upgrade, make sure that there are no indexing or Watches running.

Upgrade Xray with Bitnami RabbitMQ (disabling RabbitMQ-HA).

yaml
     rabbitmq-ha:
       enabled: false
     rabbitmq:
       enabled: true
       auth:
         username: guest
         password: password

Migrating with Existing Queues

This migration option, which requires downtime, is intended for situations where there are unfinished tasks running in Xray, but the migration to Bitnami RabbitMQ is necessary.

Detach (disable) Xray in Artifactory.
Upgrade Xray with both RabbitMQs (RabbitMQ-HA and Bitnami RabbitMQ) and scale down the Xray services to 0 (replicaCount: 0).
Note
Both RabbitMQs should be scaled down to one replica. Both RabbitMQs should have the ha-mode: all set (by default they should already be set).
```
yaml
     xray:
     replicaCount: 0
     rabbitmq-ha:
       enabled: true
       replicaCount: 1
     rabbitmq:
       enabled: true
       replicaCount: 1
       auth:
         username: guest
         password: guest
```

Go into the Bitnami RabbitMQ pod and run the following command.

bash
    export OLD_RMQ=rabbit@`<RELEASE_NAME>`-rabbitmq-ha-0.`<RELEASE_NAME>`-rabbitmq-ha-discovery.`<NAMESPACE_NAME>`.svc.cluster.local && \
    rabbitmqctl stop_app && \
    rabbitmqctl join_cluster $OLD_RMQ && \
    rabbitmqctl start_app

The process of data synchronization betweenRabbitMQ-HA and the new BitnamiRabbitMQnode begins.

Verify the synchronization with the queue size and count using the command rabbitmqctl list_queues.
You can also view the synchronization status from the RabbitMQ dashboard of the old RabbitMQ (RabbitMQ-HA).

When all the data has been synchronized between the cluster nodes, run a helm upgrade to disable the RabbitMQ-HA. This removes the old RabbitMQ-HA and brings up the Xray services.

yaml
     xray:
     replicaCount: 1
     rabbitmq-ha:
       enabled: false
     rabbitmq:
       enabled: true
       replicaCount: 1
       auth:
         username: guest
         password: guest

Finally, remove the old node from the Bitnami RabbitMQ.

bash
     rabbitmqctl forget_cluster_node rabbit@`<RELEASE_NAME>`-rabbitmq-ha-0.`<RELEASE_NAME>`-rabbitmq-ha-discovery.`<NAMESPACE_NAME>`.svc.cluster.local

Enable Xray in Artifactory.

RabbitMQ Upgrade to version 3.11.x

Xray version 3.70.0 and later includes RabbitMQ version 3.11.x later. An upgrade to Rabbit MQ 3.11.x requires feature flags to be enabled. For more information, refer to Required Feature Flags in RabbitMQ 3.11.0.

Enable Feature Flags in Helm Installation

For Helm, we have added a pre-upgrade hook that enables feature flags in RabbitMQ before the upgrade to version 3.11.x and later. The hook runs automatically when you run the upgrade command. You can disable the hook for the subsequent upgrades.

Manual Migration Operation

You can also do the migration operation manually before you start the upgrade.

Run the following command inside the RabbitMQ container.

rabbitmqctl enable_feature_flag all

Ensure that you disable the pre-upgrade hook after you perform manual migration.

Disable Pre-upgrade Hook

Update the following values in the values.yaml file to disable the pre-upgrade hook.

rabbitmq.migration.enabled=false
rabbitmq.rabbitmqUpgradeReady=true

You can also pass these values along with the Helm upgrade command.

For example:

$ helm upgrade --install xray --namespace xray jfrog/xray --set rabbitmq.migration.enabled=false --set rabbitmq.rabbitmqUpgradeReady=true

Enable Feature Flags in Linux Installations

You need to manually run the migration operation for RabbitMQ before you start Xray upgrade for your Linux installation (RPM, Debian, and Linux Archive).

Run the following command in the nodes that run RabbitMQ.

sudo -H -u xray bash -c 'export RABBITMQ_FEATURE_FLAGS="drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,maintenance_mode_status,quorum_queue,stream_queue,user_limits,virtual_host_metadata"'
sudo -H -u xray bash -c '/opt/jfrog/xray/app/third-party/rabbitmq/sbin/rabbitmqctl enable_feature_flag all'

Enable Feature Flags in Docker Compose

Run the following command inside the RabbitMQ container.

rabbitmqctl enable_feature_flag all