How To Troubleshoot Artifactory 7.x Upgrade Issues

David Pinhas
2020-06-09 06:39

When upgrading Artifactory 6.x to 7.x versions, you may encounter some issues.
This KB Article will share the most encountered issues and how to resolve them.
 

Issue – No valid installed license found

Error: 2020-05-20T12:09:02.734Z  [1;32m[jfrt ] [0;39m  [31m[WARN ] [0;39m [6cafdf57809c7477] [o.a.a.ConverterBlockerImpl:68 ] [ocalhost-startStop-1] – No valid installed license found. Blocking conversion
2020-05-20T12:09:02.735Z  [1;32m[jfrt ] [0;39m  [1;31m[ERROR] [0;39m [6cafdf57809c7477] [.a.c.ConvertersManagerImpl:215] [ocalhost-startStop-1] – Conversion failed. You should analyze the error and retry launching Artifactory. 
Error is: Converter can't run since no matching license found, please add new license

Answer: This error usually means that the Artifactory license was missing during the Artifactory boot process. Therefore, you will need to verify the artifactory.lic (artifactory.cluster.license if it is an HA setup) file exists under the $JFROG_HOME/artifactory/var/etc/artifactory/ directory. In case the file does not exist, please, create it manually and paste the license key inside the license (artifactory.lic) file.

Issue – Failed resolving the join key

Error: 2020-05-21T05:51:49.467Z [34;1m[jfmd ] [0m [31;1m[ERROR] [0m [2199530ec5479f2a] [keys.go:22] [main] – Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/artifactory/artifactory_home/var/etc/security/join.key
[init]
panic: Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/artifactory/artifactory_home/var/etc/security/join.key

Answer: This error occurs due to the join.key file missing while the Artifactory service is starting up. You may refer to our Wiki page on Managing Keys and check the path ~/opt/artifactory/artifactory_home/var/etc/security/ to verify whether the join.key file exists or not.

Issue – Master key mismatch

Error: java.lang.IllegalStateException: Master key mismatch. The provided master.key file does't match the DB fingerprint. Make sure your db.properties configurations are valid and the master key matches the DB you are trying to connect to.

Answer: This behavior occurs when the provided master.key mismatch the existing master.key in the database. In case the original master.key is present in the backup folder, you can copy it to $ARTIFACTORY_HOME/var/etc/security and restart the Artifactory service. 

In case the master.key is absent from the backup, to overcome this, we will need to regenerate the master.key by deleting the old entries from the database, as well as from the file system.

Before you delete the master.key entry from the database using the below queries, we highly recommend taking a backup prior to performing the following deletion queries:

Step1: Backup of existing entries, using below queries
SELECT * FROM access_configs where DATA like 'JE%';
SELECT * FROM access_users_custom_data where PROP_VALUE like 'JE%';
SELECT * FROM ACCESS_USERS_CUSTOM_DATA where PROP_KEY LIKE '%_shash'
SELECT * FROM access_master_key_status;
SELECT * FROM configs where DATA like 'JE%';
SELECT * FROM master_key_status;

Step 2: Delete the entries from the corresponding access tables
delete from access_configs where data LIKE 'JE%';
delete from access_users_custom_data where PROP_VALUE LIKE ‘JE%’;
delete from access_master_key_status where status = 'on';
delete from master_key_status where status = 'on';
delete from CONFIGS where data LIKE 'JE%';

Once the deletion of the master.key entry was successful, you will need to remove the master.key file under the $JFROG_HOME/var/etc/security directory.
Afterward, regenerate the master key using the command:

$openssl rand -hex 16

And place it in the $JFROG_HOME/var/etc/security directory by renaming it as master.key.
In addition, the database password is encrypted by the master.key file, therefore, it will need to be manually reverted to plain text. Change the password in the $ARTIFACTORY_HOME/var/etc/system.yaml file from encrypted to plain text.
After performing the above steps, please restart the Artifactory.

Issue – Local server is running as PRO/OSS

Error: [ERROR][0;39m [df66677ec8ef6f86] [ctoryContextConfigListener:115] [art-init ] – Application could not be initialized: Current Artifactory node last heartbeat is: 1588266832819. Stopping Artifactory since the local server is running as PRO/OSS but found other servers in registry.

Answer: This behavior usually occurs when upgrading an HA node as a Standalone Artifactory instance. We can distinguish the Artifactory type (HA or Standalone) by going through the Artifactory startup output and see ArtifactoryPro or ArtifactoryHA.
To resolve this, we will need to upgrade the Artifactory node according to the steps mentioned in HA Installation.

Issue – Relation "node_event_cursor" does not exist

Error: [jfrt ]ESC[0;39m ESC[1;31m[ERROR]ESC[0;39m [bfa28082ab7e28cd] [ctoryContextConfigListener:115] [art-init            ] – Application could not be initialized: ERROR: relation "node_event_cursor" does not exist

Answer: The following entry will appear if the database is missing the “node_event_cursor” table. To resolve this, we will need to recreate the table, along with the “node_event_priorities” table with the below queries:

CREATE TABLE node_event_cursor (
  operator_id       VARCHAR(255)  NOT NULL,
  event_marker      BIGINT        NOT NULL,
  CONSTRAINT operator_id_pk PRIMARY KEY (operator_id)
);

CREATE TABLE node_event_priorities (
  priority_id      BIGINT        NOT NULL,
  path             VARCHAR(1024) NOT NULL,
  type             VARCHAR(1024) NOT NULL,
  operator_id      VARCHAR(255)  NOT NULL,
  priority         SMALLINT      NOT NULL,
  timestamp        BIGINT        NOT NULL,
  retry_count      SMALLINT      NOT NULL,
  CONSTRAINT priority_id_pk PRIMARY KEY (priority_id)
);

In case you continue to encounter more errors regarding missing tables, please contact JFrog Support.

Issue – not permitted for a read-only connection, user or database

Error: 2020-04-21T20:03:36.117Z [jfrt ] [ERROR] [928f01c86de083be] [.s.d.v.c.DbSqlConverterUtil:87] [art-init            ] – Could not convert DB using v217_create_jobs_table converter due to DDL is not permitted for a read-only connection, user or database.
java.sql.SQLException: DDL is not permitted for a read-only connection, user or database.

Answer: To overcome errors similar to this one, we will need to remove the db.lock file from the path $JFROG_HOME/var/data/artifactory/derby. Afterward, please restart the Artifactory service.
In case you continue to encounter these errors, please make sure the ownership/permissions are valid for the folder.

Issue – Oracle database support libraries are missing

Error: 2020-03-05T15:36:10.378Z [34;1m[jfmd ] [0m [31;1m[ERROR] [0m [52f2c944d022d76b] [database_bearer.go:82         ] [main                ] – Oracle database support libraries are missing. Check https://www.jfrog.com/confluence/pages/viewpage.action?spaceKey=JFROG&title=Oracle for more details. [database]

Answer: This behavior usually occurs when the libraries path 'LD_LIBRARY_PATH' not configured with Oracle Instant Client library as recommended in Oracle-ConfiguringArtifactorytouseOracle wiki guide.

Issue – JDBC driver is missing from the required location

Error: java.net.URLClassLoader@7d907bac
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:292)
    ————-
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: Unable to load class: org.postgresql.Driver from ClassLoader:ParallelWebappClassLoader
  context: access

Answer: This behavior occurs due to the JDBC driver missing from the required location.
Artifactory 7.x is compatible with Java 11 and JDK is already bundled in the Application. In Artifactory 7.x while Artifactory is started, the JDBC driver will be copied from $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib location to $JFROG_HOME/artifactory/app/artifactory/tomcat/lib location.

Whenever we upgrade the Artifactory to 7.x, the below error can be encountered and the Artifactory service will not start.

Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter

The above errors means that the JDBC driver for the external Database which is being used is incompatible with Java 11. To overcome this error, change the JDBC driver to the compatible one under $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib and restart the Artifactory.

If the same error is encountered after changing the compatible JDBC driver, kindly navigate to $JFROG_HOME/artifactory/app/artifactory/tomcat/lib, remove all the JDBC drivers (old ones and the new ones that are copied) and restart the Artifactory.

In case you encounter any issues after performing the above steps, please contact JFrog Support for further assistance.