What to Do When Your Artifactory Upgrade Fails

Patrick Russell
2021-09-14 08:55

The Artifactory Upgrade Failed – What Next?

How to troubleshoot Artifactory startup errors

Relevant versions: This information pertains to Artifactory versions 5.X and 6.X

Upgrading Artifactory is normally a straightforward process. Sometimes, however, something can go wrong and the process will fail. In such an instance, Artifactory will not start and an error will appear in one of its log files.

There can be many reasons for a failure, as both Artifactory upgrades and the overall startup process are complex operations. The following will provide you with some insights regarding problems you may be encountering, as well as ideas on how to troubleshoot them.

The Expected Startup Sequence

When troubleshooting an upgrade or a startup problem, you first need to identify where and when the failure is occurring. Artifactory is hosted on an Apache Tomcat server, which starts and stops using Unix system commands, with upgrades requiring a few extra process steps as compared to a normal startup.

Here's an outline of the startup process. You'll want to use these steps in the next section to troubleshoot the problem further based on where your startup got stuck. For example, if you can't start Artifactory by the systemctl method, try the manual start. This will also help JFrog Support know where to begin troubleshooting.

Starting Artifactory – Expected steps

1. A BASH signal is sent to start the Artifactory service

Service installation
systemctl start artifactory
## Or
system artifactory start
 
Manual start
$ARTIFACTORY_HOME/bin/artifactoryctl.sh start
## Or
$ARTIFACTORY_HOME/bin/artifactory.sh start
User-added image

2. Artifactory's Apache Tomcat begins the startup process 

At this stage, output from the startup attempt is recorded in the $ARTIFACTORY_HOME/logs/catalina folder only. You should find updated log entries in these log files:

catalina.out – or – catalina.<TIMESTAMP>.log:
Sep 10, 2019 9:46:34 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8081"]
[…]
[/home/jfrog/programs/artifactory-pro-5.10.3/tomcat/conf/Catalina/localhost/artifactory.xml]
2019-09-10 21:46:34 [UNDEFINED] [INFO ] Fetched Artifactory [artifactory.home=null] from servlet context

localhost.<timestamp>.log:
10-Sep-2019 17:20:19.392 INFO [localhost-startStop-4] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
[…]
10-Sep-2019 21:46:40.161 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring embedded WebApplicationContext

3. The Artifactory application starts and converts its linked systems

During an upgrade, your Artifactory system should detect that an upgrade is required and begin making system changes during this stage

Currently, the Artifactory application has two distinct subprocesses. There's the principal Artifactory process for the main binary repository system and an Access process that manages security systems, such as authentication. The two processes have separate log folders.

You should see updated log entries in both the:

$ARTIFACTORY_HOME/logs/artifactory.log
$ARTIFACTORY_HOME/access/logs/access.log

The Artifactory application will perform many actions, which can be divided into four (4) substages:
 

Stage 3A

Artifactory attempts to connect to its bundled Access server, which should have started at around the same time. This is a local web application hosted by default on the URL https://127.0.0.1:8040/access.

There's a separate startup sequence for the Access server, which can be found in the $ARTIFACTORY_HOME/access/logs/access.log folder.

Stage 3B – Upgrades only 

The first major change that's executed in this substage is a database migration. Artifactory converts its database schema from its current version to an upgraded schema:
 

[Artifactory.log – Derby database conversion]
2019-09-10 20:57:15,376 [art-init] [INFO ] (o.a.s.d.v.ArtifactoryDBVersion:121) – Starting database conversion from 6.5.18(60518900) to 6.12.1(61201900)
2019-09-10 20:57:15,380 [art-init] [INFO ] (o.a.s.d.v.c.DbSqlConverterUtil:93) – Starting schema conversion: /conversion/derby/derby_v214_replication_errors.sql
[…]
2019-09-10 20:57:15,579 [art-init] [INFO ] (o.a.s.d.v.c.DbSqlConverterUtil:95) – Finished schema conversion: /conversion/derby/derby_v218_add_bundle_files_idx.sql
2019-09-10 20:57:15,580 [art-init] [INFO ] (o.a.s.d.v.ArtifactoryDBVersion:125) – Finished database conversion from 6.5.18(60518900) to 6.12.1(61201900)

This SQL conversion is automated. The Artifactory code was written with the expectation that the exact database schema of the preupgrade Artifactory version is in place. This automation performs the upgrade in separate steps, the exact SQL commands will vary depending on the detected starting version. Each version of Artifactory builds on these steps, an upgrade from 5.4 to 6.4 will start with the same steps as upgrading from 5.4 to 5.5 and add additional SQL. 

Of all the stages, this is the one that's most likely to be the cause of an upgrade failure due to the hard-coded nature of these changes mentioned earlier. Troubleshooting this step is described below in Troubleshooting section 3B.

Stage 3C – Upgrades only

Artifactory converts its Global Configuration Descriptor to the newer version (usually a minor syntax change):

2019-09-10 20:57:15,808 [art-init] [INFO ] (o.a.d.r.CentralConfigReader:71) – Converting artifactory.config.xml version from 'v217' to 'v223'

Stage 3D

The remaining steps in a standard Artifactory startup should occur, ending with an Artifactory successfully started message: 

[The much-sought-after log lines]
###########################################################
### Artifactory successfully started (43.224 seconds)   ###
###########################################################

If you see the ASCII text above, your application should have started properly.

Troubleshooting Upgrade Problems

If Artifactory will not start, don't panic. If this is an HA cluster, your remaining secondary nodes should still be running with no primary on the old version. If this is a standalone Artifactory Pro instance, there should be a maintenance window in place and users should expect an outage during this period of time.

Troubleshooting Stage 1 – "service artifactory start" did not work

If you see a Systemd error when trying to start Artifactory, first make sure this is a real error. Sometimes an upgrade will cause your Linux Systemd to report an error when the Tomcat server started anyway:

sudo systemctl start artifactory
Job for artifactory.service failed because the control process exited with error code.
See "systemctl status artifactory.service" and "journalctl -xe" for details.

Check for the exact reason via the recommended journalctl -xe command. Running both of the commands above and reviewing the output(s) produced may indicate a clear error, such as a Linux permission problem.

You can also try checking for the presence of your Java/Artifactory PID:

ps aux | grep artifactory

[Expected output on a running Artifactory]

User-added image

If the PID is not present, then there's likely a problem with your service installation (Artifactory Systemd scripts). If yours is a Zip installation, try rerunning the $ART_HOME/bin/installService.sh script.

If none of the steps above help (or yours is an RPM or Debian installation), the problem has been identified as a systemD error. You can can bypass the systemD to continue the upgrade. To proceed, you'll want to manually start Artifactory by going to the bin folder (/opt/jfrog/artifactory/bin in RPM and Debian installations) and triggering the artifactoryManage.sh script:

./bin/artifactoryManage.sh start
+++ dirname ./bin/artifactoryManage.sh
++ cd ./bin
++ pwd
[… A lot of debug output is printed to standard out …]
+ '[' 1 -eq 0 ']'
+ echo 'Artifactory Tomcat started in normal mode'
Artifactory Tomcat started in normal mode
+ exit 0

Troubleshooting Stage 2 – Tomcat won't start Artifactory

If you see your Artifactory PID, but no Artifactory webpage, the most likely cause is a lower-level Apache Tomcat error

Tomcat can typically start Artifactory without any problems. However, the Tomcat server relies on a few system files to run Artifactory. If things get stuck at this stage, it's probable there will be no output in the $ART_HOME/logs/artifactory.log file. If there is Artifactory log output, skip to Stage 3.

Also check the log files in your $ARTIFACTORY_HOME/logs/catalina folder. Chances are good an error was recorded in either the $ART_HOME/logs/catalina/catalina.out file or $ART_HOME/logs/catalina/localhost.<Timestamp>.log file.

Usually the problem will be identified as either a system issue or license error. For example, Tomcat won't start Artifactory if your Artifactory license has expired and you tried to upgrade. Accordingly, before commencing an upgrade, make sure you have a valid license installed. Apache Tomcat won't even start the Artifactory webapp if it detects an expired license.

Troubleshooting Stage 3 – Artifactory failed to upgrade something

Stage 3.A – Access won't start

As from Artifactory version 5.4, Access is a separate service that comes as part of the Artifactory bundle. If Access doesn't start, look at your $ARTIFACTORY_HOME/access/logs/access.log file for details on why the startup failed. One possibility might be related to Artifactory's database connection. The failure reason is recorded in the $ART_HOME/logs/catalina/catalina.out file, too.

Solutions might include Bootstrapping the Access-Admin . If you're unsure of how best to proceed, forward the entire logs directory to JFrog Support.

Stage 3.B – The Database Migration failed

Artifactory must convert its existing database schema to the next version. This process is somewhat delicate because an Artifactory upgrade cannot proceed unless your database schema starts as an exact match of the hard-coded set of tables and indexes that the application has been set to change. To continue the upgrade, each SQL statement must succeed. 

Artifactory expects an exact set of SQL Tables, Indexes, and Table Columns to match the old database schema when converting. Usually there is a JDBC SQL error in the logs that indicates the problem area. 

Some common errors may result from manually adding an index file or table column to the database long before the upgrade was started. Depending on the error, undoing such changes may allow your upgrade to continue. One way to do this is by reloading a backup of the old version of Artifactory and attempt the upgrade again.

Please consult with your DBA and JFrog Support before manually modifying your Artifactory database or rolling back an upgrade.

Stage 3.C – Artifactory fails to upgrade its Config Descriptor

If this occurs, the problem should have been recorded in your artifactory.log file. To resolve this issue, you'll need to manually correct and bootstrap a config descriptor from your file system. Locate the $ART_HOME/etc/artifactory.config.latest.xml file, it will likely be the old XML version, which is ok:

[Expected log entries during a config bootstrap]
2019-09-10 23:04:44,973 [art-init] [INFO ] (o.a.d.r.CentralConfigReader:92) – The current Artifactory config schema namespace is 'http://artifactory.jfrog.org/xsd/2.2.3' The provided config does not seem to be compliant with it.
2019-09-10 23:04:45,026 [art-init] [INFO ] (o.a.d.r.CentralConfigReader:71) – Converting artifactory.config.xml version from 'v222' to 'v223'

Thereafter, fix the XML syntax, attempt to bootstrap the file by renaming it to $ART_HOME/etc/artifactory.config.import.xml, and restart Artifactory

Stage 3.D – Artifactory won't start after succeeding in the first 3 steps

Once you arrive at this stage, almost everything should be ready to go. If Artifactory refuses to start because of a remaining bad configuration or similar problem, you'll probably need to contact JFrog Support To obtain additional assistance.