Xray Helm Upgrade

JFrog Installation & Setup Documentation

Content Type
Installation & Setup
ft:sourceType
Paligo

Note

When upgrading from Xray chart version 3.x to 3.x, due to breaking changes, use kubectl delete statefulsets <old_statefulset_xray_name> and run the helm upgrade.

Also, While upgrading from Xray chart version 3.x to 4.x, due to breaking RabbitMQ (whenrabbitmq.enabled=true) subchart changes, run the following commands.

$ kubectl delete statefulsets <old_statefulset_xray_name>
$ kubectl delete statefulsets <old_statefulset_rabbitmq_name>
$ kubectl delete pvc <old_PVC_rabbitmq_name>
$ helm upgrade --install xray --namespace xray jfrog/xray

Note

Downtime is required to perform an upgrade.

Offline Update

If you wish to do an offline update of the Xray database while running in pods, copy the downloaded data into the correct Xray container (scale down to 1 replica if needed before) and then using kubectl commands copy the file to the specific location.

For example:

kubectl cp ./comp_-6795364578871.zip <namespace>/xray-0:/opt/jfrog/xray/var/work/server/updates/component/comp_-6795364578871.zip -c xray-server
kubectl cp ./vuln_-6795364578871.zip <namespace>/xray-0:/opt/jfrog/xray/var/work/server/updates/vulnerability/vuln_-6795364578871.zip -c xray-server

Upgrading to Xray 3.70.0 or later from an Xray version prior to 3.70.0

If you are upgrading to Xray 3.70.0 or later from an Xray version prior to 3.70.0, your installation contains RabbitMQ 3.8.x. Feature flags in RabbitMQ have to be enabled before you can upgrade, since Xray 3.70.0 and later contains RabbitMQ 3.11.x and later.

Before you proceed with the upgrade, go to RabbitMQ Upgrade to version 3.11.x and set feature flags in RabbitMQ.

To upgrade Xray.

  1. Update the existing deployed version to the updated version.

    helm repo update
    
    helm upgrade --install <myrelease> namespace xray jfrog/xray
  2. If Xray was installed without providing a value to postgresql.postgresqlPassword (the password was autogenerated), follow these instructions.

    1. Get the current password by running the following.

      POSTGRES_PASSWORD=$(kubectl get secret -n <namespace> <myrelease>-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
    2. Upgrade the release by passing the previously auto-generated secret.

      helm upgrade <myrelease> jfrog/xray --set postgresql.postgresqlPassword=${POSTGRES_PASSWORD}
  3. If Xray was installed without providing a value to rabbitmq.auth.password (the password was autogenerated), follow these instructions.

    1. Get the current password by running the following.

      RABBITMQ_PASSWORD=$(kubectl get secret -n <namespace> <myrelease>-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)
    2. Upgrade the release by passing the previously auto-generated secret.

      helm upgrade <myrelease> jfrog/xray --set rabbitmq.auth.password=${RABBITMQ_PASSWORD}
  4. If Xray was installed with all of the default values (e.g., with no user-provided values for Rabbit/Postgres), follow these steps.

    1. Retrieve all current passwords (RabbitMQ/PostgreSQL) as explained in the previous section.

    2. Upgrade the release by passing the previously auto-generated secrets.

      helm upgrade --install xray --namespace xray jfrog/xray --set rabbitmq.auth.password=<rabbit-password> --set postgresql.post
  5. Access Xray from your browser at: http://<jfrogUrl>/ui/, then go to the Xray tab in the Application module in the UI.

  6. Check the status of your deployed helm releases.

    helm status xray
Upgrading the Chart Versions from 1.x/2.x to 3.x and Above
  • JFrog Xray 3.x is only compatible with JFrog Artifactory 7.x; to upgrade, you must first install JFrog Artifactory 7.x.

  • To upgrade from a version prior to 1.x, you will first need to upgrade to the latest version of 1.x

  • To upgrade from a version of 2.x, you first need to upgrade to the latest version of 2.x.

PostgreSQL Migration Process

The recommended migration process has two main steps:

  1. Migrate the existing MongoDB data to an existing PostgreSQL.

  2. Perform a full database export and import of PostgreSQL.

Prerequisite Step: Get the Details of the Existing Chart
  1. Block user access to Xray (but do not shut down).

  2. Get the service names OLD_PG_SERVICE_NAME, OLD_MONGO_SERVICE_NAME using the command below. For example: OLD_PG_SERVICE_NAME and OLD_MONGO_SERVICE_NAME values as <OLD_RELEASE_NAME>-postgresql and <OLD_RELEASE_NAME>-mongodb respectively.

    $ kubectl get svc
    NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
    <OLD_RELEASE_NAME>-mongodb                 ClusterIP      10.101.56.69     <none>        27017/TCP                     114m
    <OLD_RELEASE_NAME>-postgresql              ClusterIP      10.101.250.74    <none>        5432/TCP                      114m
    <OLD_RELEASE_NAME>-rabbitmq-ha             ClusterIP      None             <none>        15672/TCP,5672/TCP,4369/TCP   114m
    <OLD_RELEASE_NAME>-rabbitmq-ha-discovery   ClusterIP      None             <none>        15672/TCP,5672/TCP,4369/TCP   114m
    <OLD_RELEASE_NAME>-xray-analysis           ClusterIP      10.104.138.63    <none>        7000/TCP                      114m
    <OLD_RELEASE_NAME>-xray-indexer            ClusterIP      10.106.72.163    <none>        7002/TCP                      114m
    <OLD_RELEASE_NAME>-xray-persist            ClusterIP      10.103.20.33     <none>        7003/TCP                      114m
    <OLD_RELEASE_NAME>-xray-server             LoadBalancer   10.105.121.175   <pending>     80:32326/TCP                  114m
  3. Save the previous passwords OLD_PG_SERVICE_NAME, OLD_MONGO_SERVICE_NAME or extract them from the secret of the existing PostgreSQL and MongoDB pods.

    #Example:
    OLD_PG_PASSWORD=$(kubectl get secret -n <namespace> <OLD_RELEASE_NAME>-postgresql -o jsonpath="{.data.postgres-password}" | base64 --decode)
    OLD_MONGO_PASSWORD=$(kubectl get secret -n <namespace> <OLD_RELEASE_NAME>-mongodb -o jsonpath="{.data.mongodb-password}" | base64 --decode)
  4. Stop the old Xray pods (scale down replicas to 0). The PostgreSQL and MongoDB pods will still remain active.

    $ kubectl scale statefulsets <REPLACE_OLD_RELEASE_NAME>-rabbitmq-ha <REPLACE_OLD_RELEASE_NAME>-xray-analysis <REPLACE_OLD_RELEASE_NAME>-xray-indexer <REPLACE_OLD_RELEASE_NAME>-xray-persist <REPLACE_OLD_RELEASE_NAME>-xray-server --replicas=0
Migrate the Existing MongoDB Data to an Existing PostgreSQL

Run the helm install (not upgrade) with the new version say xray-new in the following way.

  1. Verify that all probes are disabled.

    --set router.livenessProbe.enabled=false --set router.readinessProbe.enabled=false --set indexer.livenessProbe.enabled=false --set analysis.livenessProbe.enabled=false --set server.livenessProbe.enabled=false --set persist.livenessProbe.enabled=false --set indexer.readinessProbe.enabled=false --set analysis.readinessProbe.enabled=false --set server.readinessProbe.enabled=false --set persist.readinessProbe.enabled=false
  2. Point to the previous PostgreSQL pod (user, password,DATABASE).

    --set postgresql.enabled=false --set database.user=<OLD_PG_USERNAME> --set database.password=<OLD_PG_PASSWORD> --set database.url="postgres://<SERVICE_NAME_POSTGRES>:5432/xraydb?sslmode=disable"
  3. Point to the previous MongoDB (user, password,DATABASE) pod.

    --set xray.mongoUsername=<OLD_MONGO_USERNAME> --set xray.mongoPassword=<OLD_MONGO_PASSWORD> --set xray.mongoUrl="mongodb://<SERVICE_NAME_MONGODB>:27017/?authSource=xray&authMechanism=SCRAM-SHA-1"

    This will trigger the migration process like in the following example.

    # Create a customvalues.yaml file
     
    router:
      livenessProbe:
        enabled: false
      readinessProbe:
        enabled: false
    indexer:
      livenessProbe:
        enabled: false
      readinessProbe:
        enabled: false
    analysis:
      livenessProbe:
        enabled: false
      readinessProbe:
        enabled: false
    server:
      livenessProbe:
        enabled: false
      readinessProbe:
        enabled: false 
    persist:
      livenessProbe:
        enabled: false
      readinessProbe:
        enabled: false
    postgresql:
      enabled: false
    database:
      user: <OLD_PG_USERNAME>
      password: <OLD_PG_PASSWORD>
      url: "postgres://<SERVICE_NAME_POSTGRES>:5432/xraydb?sslmode=disable"
    xray:
      mongoUsername: <OLD_MONGO_USERNAME>
      mongoPassword: <OLD_MONGO_PASSWORD>
      mongoUrl: "mongodb://<SERVICE_NAME_MONGODB>:27017/?authSource=xray&authMechanism=SCRAM-SHA-1"
      masterKey: <PREVIOUS_MASTER_KEY>
      jfrogUrl: <NEW_ARTIFACTORY_URL>
      joinKey: <JOIN_KEY>
    rabbitmq:
      enabled: true
      auth:
        password: <PASSWORD>
    rabbitmq-ha:
      enabled: false
  4. Apply the values.yaml file during installation.

    helm upgrade --install xray-new jfrog/xray -f customvalues.yaml
  5. Stop the new Xray pods (scale down replicas to 0). Both PostgreSQL pods still remain active.

  6. Create a customvalues.yaml file.

    replicaCount: 0
    postgresql:
      postgresqlPassword: <NEW_PG_PASSWORD>
    rabbitmq:
      enabled: true
      auth:
        password: <PASSWORD>
    rabbitmq-ha:
      enabled: false
    xray:
      masterKey: <PREVIOUS_MASTER_KEY>
      jfrogUrl: <NEW_ARTIFACTORY_URL>
      joinKey: <JOIN_KEY>
    unifiedUpgradeAllowed: true
    databaseUpgradeReady: true
  7. Apply the values.yaml file during installation.

    helm upgrade --install xray-new jfrog/xray -f customvalues.yaml
Perform a Full Database Export and Import of PostgreSQL

To migrate PostgreSQL data between old and new pods.

  1. Connect to the new PostgreSQL pod (you can obtain the name by running kubectl get pods).

    $ kubectl exec -it <NAME> bash
  2. Once logged in, create a dump file from the previous database usingpg_dump, and connect to the previous PostgreSQL chart.

    $ pg_dump -h <OLD_PG_SERVICE_NAME> -U xray DATABASE_NAME > /tmp/backup.sql

    After running this command, you will be prompted for a password; this password is the previous chart passwordOLD_PG_PASSWORD. This operation could take some time depending on the database size.

  3. Once you have the backup file, you can restore it with this command.

    $ psql -U xray DATABASE_NAME < /tmp/backup.sql

    After running the command above, you will be prompted for a password; this is the current chart password. This operation could take some time depending on the database size.

  4. Create a customvalues.yaml file.

    # Create a customvalues.yaml file
    xray:
      masterKey: <PREVIOUS_MASTER_KEY>
      jfrogUrl: <NEW_ARTIFACTORY_URL>
      joinKey: <JOIN_KEY>
    rabbitmq:
      enabled: true
      auth:
        password: <PASSWORD>
    rabbitmq-ha:
      enabled: false
    postgresql:
      postgresqlPassword: <NEW_PG_PASSWORD>
    unifiedUpgradeAllowed: true
    databaseUpgradeReady: true
  5. Apply the values file during the installation.

    helm upgrade --install xray-new jfrog/xray -f customvalues.yaml
  6. Restore access to the new Xray by running the command below to remove the old Xray deployment and Helm release.

    run helm delete <OLD_RELEASE_NAME>

Xray should now be ready for use.

RabbitMQ Migration Process

Note

The RabbitMQ-HA chart has been removed from the 7.x chart versions; therefore, before upgrading to the 7.x chart versions, you will need to perform the RabbitMQ migration (assuming you are not using the Bitnami RabbitMQ). This section describes the steps for migrating from the RabbitMQ-HA chart to the Bitnami RabbitMQ chart beforeupgrading to chart version 7.x and above.

For this procedure, you will need to choose whether to migrate without existing queues (assuming that all queues are empty) or to migrate while Xray is down.

Migrating without Existing Queues

Important

This procedure assumes that all queues are empty.

While running the helm upgrade, make sure that there are no indexing or Watches running.

Upgrade Xray with Bitnami RabbitMQ (disabling RabbitMQ-HA).

yaml
     rabbitmq-ha:
       enabled: false
     rabbitmq:
       enabled: true
       auth:
         username: guest
         password: password
Migrating with Existing Queues

This migration option, which requires downtime, is intended for situations where there are unfinished tasks running in Xray, but the migration to Bitnami RabbitMQ is necessary.

  1. Detach (disable) Xray in Artifactory.

  2. Upgrade Xray with both RabbitMQs (RabbitMQ-HA and Bitnami RabbitMQ) and scale down the Xray services to 0 (replicaCount: 0).

    Note

    Both RabbitMQs should be scaled down to one replica. Both RabbitMQs should have the ha-mode: all set (by default they should already be set).

    yaml
         xray:
         replicaCount: 0
         rabbitmq-ha:
           enabled: true
           replicaCount: 1
         rabbitmq:
           enabled: true
           replicaCount: 1
           auth:
             username: guest
             password: guest
  3. Go into the Bitnami RabbitMQ pod and run the following command.

    bash
        export OLD_RMQ=rabbit@`<RELEASE_NAME>`-rabbitmq-ha-0.`<RELEASE_NAME>`-rabbitmq-ha-discovery.`<NAMESPACE_NAME>`.svc.cluster.local && \
        rabbitmqctl stop_app && \
        rabbitmqctl join_cluster $OLD_RMQ && \
        rabbitmqctl start_app

    The process of data synchronization betweenRabbitMQ-HA and the new BitnamiRabbitMQnode begins.

  4. Verify the synchronization with the queue size and count using the command rabbitmqctl list_queues.

    You can also view the synchronization status from the RabbitMQ dashboard of the old RabbitMQ (RabbitMQ-HA).

  5. When all the data has been synchronized between the cluster nodes, run a helm upgrade to disable the RabbitMQ-HA. This removes the old RabbitMQ-HA and brings up the Xray services.

    yaml
         xray:
         replicaCount: 1
         rabbitmq-ha:
           enabled: false
         rabbitmq:
           enabled: true
           replicaCount: 1
           auth:
             username: guest
             password: guest
  6. Finally, remove the old node from the Bitnami RabbitMQ.

    bash
         rabbitmqctl forget_cluster_node rabbit@`<RELEASE_NAME>`-rabbitmq-ha-0.`<RELEASE_NAME>`-rabbitmq-ha-discovery.`<NAMESPACE_NAME>`.svc.cluster.local
  7. Enable Xray in Artifactory.

RabbitMQ Upgrade to version 3.11.x

Xray version 3.70.0 and later includes RabbitMQ version 3.11.x later. An upgrade to Rabbit MQ 3.11.x requires feature flags to be enabled. For more information, refer to Required Feature Flags in RabbitMQ 3.11.0.

Enable Feature Flags in Helm Installation

For Helm, we have added a pre-upgrade hook that enables feature flags in RabbitMQ before the upgrade to version 3.11.x and later. The hook runs automatically when you run the upgrade command. You can disable the hook for the subsequent upgrades.

Manual Migration Operation

You can also do the migration operation manually before you start the upgrade.

Run the following command inside the RabbitMQ container.

rabbitmqctl enable_feature_flag all

Ensure that you disable the pre-upgrade hook after you perform manual migration.

Disable Pre-upgrade Hook

Update the following values in the values.yaml file to disable the pre-upgrade hook.

rabbitmq.migration.enabled=false
rabbitmq.rabbitmqUpgradeReady=true

You can also pass these values along with the Helm upgrade command.

For example:

$ helm upgrade --install xray --namespace xray jfrog/xray --set rabbitmq.migration.enabled=false --set rabbitmq.rabbitmqUpgradeReady=true