How to Troubleshoot Artifactory 7.x Upgrade Issues

How to Troubleshoot Artifactory 7.x Upgrade Issues

AuthorFullName__c
David Pinhas
articleNumber
000004823
ft:sourceType
Salesforce
FirstPublishedDate
2020-06-08T11:40:16Z
lastModifiedDate
2020-06-08
VersionNumber
8

When upgrading Artifactory 6.x to 7.x versions, you may encounter some problems. The following are some of the most encountered issues and how to resolve them.

Issue #1: No valid installed license found
Error:

2020-05-20T12:09:02.734Z  [1;32m[jfrt ] [0;39m  [31m[WARN ] [0;39m [6cafdf57809c7477] [o.a.a.ConverterBlockerImpl:68 ] [ocalhost-startStop-1] – No valid installed license found. Blocking conversion
2020-05-20T12:09:02.735Z  [1;32m[jfrt ] [0;39m  [1;31m[ERROR] [0;39m [6cafdf57809c7477] [.a.c.ConvertersManagerImpl:215] [ocalhost-startStop-1] – Conversion failed. You should analyze the error and retry launching Artifactory. 
Error is: Converter can't run since no matching license found, please add new license
Resolution: This error usually means that the Artifactory license was missing during the Artifactory boot process. Therefore, you’ll need to verify that the artifactory.lic file (or, if yours is an HA setup, the artifactory.cluster.license file) exists under the $JFROG_HOME/artifactory/var/etc/artifactory/ directory. If it does not exist, you’ll need to create it manually and paste the license key inside the artifactory.lic or artifactory.cluster.license file.

Issue #2: Failure to resolve the join key
Error:
2020-05-21T05:51:49.467Z [34;1m[jfmd ] [0m [31;1m[ERROR] [0m [2199530ec5479f2a] [keys.go:22] [main] – Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/artifactory/artifactory_home/var/etc/security/join.key
[init]
panic: Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/artifactory/artifactory_home/var/etc/security/join.key
Resolution: This error occurs due to the join.key file being missing when the Artifactory service is starting up. Take a look at our wiki page on Managing Keys HERE. Also, check the ~/opt/artifactory/artifactory_home/var/etc/security/ path to verify whether the join.key file exists or not.

Issue #3: Master key mismatch
Error:
java.lang.IllegalStateException: Master key mismatch. The provided master.key file does't match the DB fingerprint. Make sure your db.properties configurations are valid and the master key matches the DB you are trying to connect to.
Resolution: This behavior occurs when the provided master.key is a mismatch with the existing master.key in the database. If the original master.key is present in the backup folder, you can copy it to $ARTIFACTORY_HOME/var/etc/security and restart the Artifactory service. If the master.key is not in the backup folder, you’ll need to regenerate the master.key by deleting old entries from the database, as well as from the file system. To accomplish this, use the following deletion queries (but, before you do, we highly recommend making a backup of the database):

Step 1: Backup existing entries using the following queries:
  • SELECT * FROM access_configs where DATA like 'JE%'
  • SELECT * FROM access_users_custom_data where PROP_VALUE like 'JE%'
  • SELECT * FROM ACCESS_USERS_CUSTOM_DATA where PROP_KEY LIKE '%_shash'
  • SELECT * FROM access_master_key_status
  • SELECT * FROM configs where DATA like 'JE%'
  • SELECT * FROM master_key_status
Step 2: Delete the entries from the corresponding access tables, as follows:
  • delete from access_configs where data LIKE 'JE%'
  • delete from access_users_custom_data where PROP_VALUE LIKE ‘JE%’
  • delete from access_master_key_status where status = 'on'
  • delete from master_key_status where status = 'on'
  • delete from CONFIGS where data LIKE 'JE%'
Once you have successfully deleted the master.key entry, you’ll need to remove the master.key file from the $JFROG_HOME/var/etc/security directory. Thereafter, regenerate the master key using the following command:
$openssl rand -hex 16
Place this in the $JFROG_HOME/var/etc/security directory, renaming it as master.key. Additionally, as the database password is encrypted by the master.key file, you’ll need to go to the $ARTIFACTORY_HOME/var/etc/system.yaml file and manually change the password from encrypted to plain text. Afterwards, restart Artifactory for these changes to take effect.

Issue #4: Local server is running as PRO/OSS
Error:
[ERROR][0;39m [df66677ec8ef6f86] [ctoryContextConfigListener:115] [art-init ] – Application could not be initialized: Current Artifactory node last heartbeat is: 1588266832819. Stopping Artifactory since the local server is running as PRO/OSS but found other servers in registry.
Resolution: This behavior usually occurs when upgrading an HA node as a standalone Artifactory instance. You can identify your Artifactory type (HA or standalone) by going through your Artifactory startup output and see whether it’s listed as ArtifactoryPro or ArtifactoryHA. To resolve this issue, you’ll need to upgrade your Artifactory node by following the steps that are detailed HERE.

Issue #5: Relation "node_event_cursor" does not exist
Error:
[jfrt ]ESC[0;39m ESC[1;31m[ERROR]ESC[0;39m [bfa28082ab7e28cd] [ctoryContextConfigListener:115] [art-init            ] – Application could not be initialized: ERROR: relation "node_event_cursor" does not exist
Resolution: The following entry will appear if the database is missing the node_event_cursor table. To resolve this, you’ll need to recreate the table, along with the node_event_priorities table by using the following queries:
CREATE TABLE node_event_cursor (
  operator_id       VARCHAR(255)  NOT NULL,
  event_marker      BIGINT        NOT NULL,
  CONSTRAINT operator_id_pk PRIMARY KEY (operator_id)
);
CREATE TABLE node_event_priorities (
  priority_id      BIGINT        NOT NULL,
  path             VARCHAR(1024) NOT NULL,
  type             VARCHAR(1024) NOT NULL,
  operator_id      VARCHAR(255)  NOT NULL,
  priority         SMALLINT      NOT NULL,
  timestamp        BIGINT        NOT NULL,
  retry_count      SMALLINT      NOT NULL,
  CONSTRAINT priority_id_pk PRIMARY KEY (priority_id)
);
Should you continue to encounter errors regarding missing tables, please contact JFrog Support.

Issue #6: “not permitted for a read-only connection, user or database” error
Error:
2020-04-21T20:03:36.117Z [jfrt ] [ERROR] [928f01c86de083be] [.s.d.v.c.DbSqlConverterUtil:87] [art-init            ] – Could not convert DB using v217_create_jobs_table converter due to DDL is not permitted for a read-only connection, user or database.
java.sql.SQLException: DDL is not permitted for a read-only connection, user or database.
Resolution: To overcome errors such as this one, you’ll need to remove the db.lock file from the $JFROG_HOME/var/data/artifactory/derby path. Afterwards, restart the Artifactory service for your changes to take effect. Should you continue to encounter errors, please make sure your ownership/permissions are valid for this folder.

Issue #7: Oracle database support libraries are missing
Error:
2020-03-05T15:36:10.378Z [34;1m[jfmd ] [0m [31;1m[ERROR] [0m [52f2c944d022d76b] [database_bearer.go:82         ] [main                ] – Oracle database support libraries are missing. Check https://www.jfrog.com/confluence/pages/viewpage.action?spaceKey=JFROG&title=Oracle for more details. [database]
Resolution: This behavior usually occurs when the LD_LIBRARY_PATH libraries path hasn’t been configured, as recommended HERE, with the Oracle Instant Client library.

Issue #8: JDBC driver is missing from the required location
Error:
java.net.URLClassLoader@7d907bac
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:292)
    ————-
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: Unable to load class: org.postgresql.Driver from ClassLoader:ParallelWebappClassLoader
  context: access

Resolution: This behavior occurs due to the JDBC driver being missing from the required location. Artifactory 7.x is compatible with Java 11 and JDK comes bundled into the application. Accordingly, upon startup, the JDBC driver will be copied from the $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib folder to the $JFROG_HOME/artifactory/app/artifactory/tomcat/lib folder.

Whenever you execute an upgrade of Artifactory to 7.x, the following error may be encountered and the Artifactory service will not start:

Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter

This error means that the JDBC driver for the external database being used is incompatible with Java 11. To overcome this problem, change the JDBC driver to the compatible one, which can be found in the $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib folder. Thereafter, restart Artifactory for your changes to take effect.

Should you encounter the same error after making the change to the compatible JDBC driver, navigate to $JFROG_HOME/artifactory/app/artifactory/tomcat/lib, remove all JDBC drivers (both old and new), and restart Artifactory.

Issue #9: Ports are blocked by firewall
Error:

[ERROR] Router external port (8082) is blocked by firewall
[ERROR] Router internal port 8046 is blocked by firewall
[ERROR] Router traefik port (8049) is blocked by firewall
[ERROR] Router grpc port (8047) is blocked by firewall
[ERROR] Artifactory port (8081) is blocked by firewall
[ERROR] Access port (8040) is blocked by firewall
[ERROR] Access grpc port (8045) is blocked by firewall
[ERROR] Metadata Port (8086) is blocked by firewall
[ERROR] Frontend Port (8070) is blocked by firewall
[ERROR] Replicator Port (8048) is blocked by firewall

Resolution: This error indicates that Artifactory ports are all blocked by firewall rules. Resolving this issue can be done by allowing these ports in your firewall. A full list of ports Artifactory uses can be found on our System Resources wiki page.

Should you encounter any issues after performing all of these steps, please contact JFrog Support for further assistance.

Issue #10: When upgrading from Artifactory v6 to Artifactory v7 on a Windows environment, we usually encounter the following error:
 

[org.apache.tomcat.util.digester.Digester] [org.apache.tomcat.util.digester.Digester fatalError] - Parse Fatal Error at line 20 column 75: Element type "Connector" must be followed by either attribute specifications, ">" or "/>".
org.xml.sax.SAXParseException; systemId: file:/F:/Jfrog/artifactory/app/artifactory/tomcat/conf/server.xml; lineNumber: 20; columnNumber: 75; Element type "Connector" must be followed by either attribute specifications, ">" or "/>".


The reason why the issue occurred:

The following lines were added to system.yaml after the upgrade (those settings were migrated from version 6's server.xml):
 

 relaxedPathChars: '"[]"'
 relaxedQueryChars: '"[]"'


This is the result of parsing the above in server.xml:

<Connector port="8081" sendReasonPhrase="true" relaxedPathChars=''[]'' relaxedQueryChars=''[]'' maxThreads="200" />

Solution:
 
You can resolve the issue by removing the following lines from system.yaml (located at $JFROG_HOME/artifactory/var/etc/system.yaml).

 relaxedPathChars: '"[]"'
 relaxedQueryChars: '"[]"'

Thus, it forces Artifactory to use default values for relaxedPathChars and relaxedQueryChars parameters. You may inspect the full list of parameters and their default values on our System YAML wiki page.