The Artifactory Upgrade Failed – What Next?

Patrick Russell
2019-10-08 21:16

The Artifactory Upgrade Failed – What Next?

How to troubleshoot Artifactory startup errors

Relevant versions: Artifactory 5.X – 6.X

Upgrading Artifactory is normally a straightforward process. Sometimes, something goes wrong and the process fails. In this state, Artifactory will not start and will show an error in one of its log files.

There are many possible reasons for the failure, as both Artifactory upgrades and the overall startup process are complex operations. This article is intended to be a first-stop guide to troubleshoot upgrade or startup-related problems. 

The Expected Startup Sequence

When troubleshooting an upgrade or startup problem, we first need to identify where the failure is happening. Artifactory is hosted on an Apache Tomcat server, which starts and stops using Unix system commands. The upgrade does a few extra steps when compared to a "normal" startup.

This is an outline of the startup process. Please use these steps in the next section to troubleshoot the problem further based on where your startup got stuck.

Starting Artifactory – Expected steps

1. A BASH signal is sent to start the Artifactory Service

Service installation
systemctl start artifactory
## Or
system artifactory start
 
Manual start
$ARTIFACTORY_HOME/bin/artifactoryctl.sh start
## Or
$ARTIFACTORY_HOME/bin/artifactory.sh start

2. Artifactory's Apache Tomcat begins the startup process 

At this stage, output is recorded only in the $ARTIFACTORY_HOME/logs/catalina folder. You should find updated log entries in these log files:

catalina.out – or – catalina.<TIMESTAMP>.log:
Sep 10, 2019 9:46:34 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8081"]
[…]
[/home/jfrog/programs/artifactory-pro-5.10.3/tomcat/conf/Catalina/localhost/artifactory.xml]
2019-09-10 21:46:34 [UNDEFINED] [INFO ] Fetched Artifactory [artifactory.home=null] from servlet context

localhost.<timestamp>.log:
10-Sep-2019 17:20:19.392 INFO [localhost-startStop-4] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
[…]
10-Sep-2019 21:46:40.161 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring embedded WebApplicationContext

3. The Artifactory application starts and converts its linked systems

Once Tomcat has started running the "artifactory.war" and "access.war" web applications, the Artifactory system should detect that an upgrade is required and begin making system changes. 

Currently the Artifactory application has two separate subprocesses. There is a main Artifactory process for the main Binary Repository system, and an Access process that manages security systems such as authentication. The two processes have separate log folders.

You should see updated log entries in both the $ARTIFACTORY_HOME/logs/artifactory.log and $ARTIFACTORY_HOME/access/logs/access.log files throughout this final stage. 

The Artifactory application will perform many actions, which can be divided into 4 sub-stages:

Stage 3A

 Artifactory attempts to connect with its bundled Access server, which should have started at around the same time. This is a local web application hosted by default on the URL http://127.0.0.1:8040/access. 

There is a separate startup sequence for the Access server, found in the $ARTIFACTORY_HOME/access/logs/access.log location.

Stage 3B – Upgrades only 

The first major change done is a database migration. Artifactory converts its database schema from the current version to an upgraded schema:

[Artifactory.log – Derby database conversion]
2019-09-10 20:57:15,376 [art-init] [INFO ] (o.a.s.d.v.ArtifactoryDBVersion:121) – Starting database conversion from 6.5.18(60518900) to 6.12.1(61201900)
2019-09-10 20:57:15,380 [art-init] [INFO ] (o.a.s.d.v.c.DbSqlConverterUtil:93) – Starting schema conversion: /conversion/derby/derby_v214_replication_errors.sql
[…]
2019-09-10 20:57:15,579 [art-init] [INFO ] (o.a.s.d.v.c.DbSqlConverterUtil:95) – Finished schema conversion: /conversion/derby/derby_v218_add_bundle_files_idx.sql
2019-09-10 20:57:15,580 [art-init] [INFO ] (o.a.s.d.v.ArtifactoryDBVersion:125) – Finished database conversion from 6.5.18(60518900) to 6.12.1(61201900)

This SQL conversion is automated, and expects the exact database schema of the pre-upgrade Artifactory version. It performs the upgrade in separate steps, depending on the detected starting version. Each version of Artifactory builds on these steps.

Of all the stages, this one is the most likely cause of an upgrade failure. Troubleshooting this step is described below.

Stage 3C – Upgrades only

Artifactory converts its Global Configuration Descriptor to the newer version (Usually a minor syntax change):

2019-09-10 20:57:15,808 [art-init] [INFO ] (o.a.d.r.CentralConfigReader:71) – Converting artifactory.config.xml version from 'v217' to 'v223'

Stage 3D

The rest of the steps in a standard Artifactory startup should occur, ending with an "Artifactory successfully started" message: 

[The much-sought-after log lines]
###########################################################
### Artifactory successfully started (43.224 seconds)   ###
###########################################################

If you see the above ASCII text, the application should have started and all is well.

Troubleshooting Upgrade Problems

If Artifactory is not starting, the first step is to not panic. If this is an HA cluster, the remaining secondary nodes should still be running on the old version with no primary. If this is a standalone Artifactory Pro instance, there should be a maintenance window in place and users should expect an outage during this period of time.

Troubleshooting Stage 1 – "service artifactory start" did not work

If you see a Systemd error when trying to start Artifactory, the first step is to make sure this is a real error. Sometimes an upgrade will cause the Linux Systemd to report an error when the Tomcat server started anyway:

sudo systemctl start artifactory
Job for artifactory.service failed because the control process exited with error code.
See "systemctl status artifactory.service" and "journalctl -xe" for details.

Check for the exact reason via the suggested "journalctl -xe" command. Running the above commands and reviewing the output may indicate a clear error, like a Linux permission problem.

Try checking for the Java / Artifactory PID:

ps aux | grep artifactory

[Expected output on a running Artifactory]

If the PID is not present, there is likely a problem with the Service Installation (The Artifactory SystemD scripts). If this is a Zip installation, try re-running the $ART_HOME/bin/installService.sh script.

If none of the above steps help (Or this is an RPM or Debian installation), the problem has been identified and can be bypassed to continue the upgrade. You can manually start Artifactory by going to the "bin" folder (/opt/jfrog/artifactory/bin in RPM and Debian installations) and triggering the "artifactoryManage.sh" script:

./bin/artifactoryManage.sh start
+++ dirname ./bin/artifactoryManage.sh
++ cd ./bin
++ pwd
[… A lot of debug output is printed to standard out …]
+ '[' 1 -eq 0 ']'
+ echo 'Artifactory Tomcat started in normal mode'
Artifactory Tomcat started in normal mode
+ exit 0

Troubleshooting Stage 2 – Tomcat won't start Artifactory

If you see an Artifactory PID and you see no Artifactory webpage, the most likely cause is a lower-level Tomcat error. Apache Tomcat normally can start Artifactory without any problems. The Tomcat server relies on a few system files to run Artifactory. 

If things are stuck at this stage, there will most likely be no output in the $ART_HOME/logs/artifactory.log file (If there is Artifactory log output, skip to Stage 3).

Check the log files in the $ARTIFACTORY_HOME/logs/catalina folder. Chances are good an error was recorded in either the $ART_HOME/logs/catalina/catalina.out or $ART_HOME/logs/catalina/localhost.<Timestamp>.log file.

Usually there is either a system issue or license error causing the problem. For example, Tomcat won't start Artifactory if the Artifactory License has expired during an upgrade.

Make sure there is a valid license installed in Artifactory during an upgrade. Apache Tomcat won't start the Artifactory webapp if it detects an expired license during an upgrade.

Troubleshooting Stage 3 – Artifactory failed to upgrade something

Stage 3.A – Access won't start

The first thing to check is why the Access server isn't starting. This is a separate service bundled in Artifactory 5.4 and up. Look in the $ARTIFACTORY_HOME/access/logs/access.log file for details on why the startup failed, it might be related to Artifactory's database connection. The failure reason is recorded in the $ART_HOME/logs/catalina/catalina.out file too.

Solutions might include Bootstrapping the Access-Admin. If you are unsure of the next steps, forward the entire log file containing the error to JFrog Support.

Stage 3.B – The Database Migration failed

Artifactory has to convert its existing database schema to the next version. This process is somewhat delicate. The database schema is expected to exactly match a hard-coded set of tables and indexes. In order to continue the upgrade, each SQL statement must succeed. 

Artifactory expects an exact set of SQL Tables, Indexes, and Table Columns to match the old database schema when converting. Usually there is a JDBC SQL error in the logs that indicates the problem area. 

Some common errors may result from manually adding an index file or table column to the database long before the upgrade was started. Undoing such changes may allow the upgrade to continue, depending on the error.

One thing to try is to reload a backup of the old version of Artifactory and try the upgrade again.

Please consult with your DBA and JFrog Support before manually modifying your Artifactory database or rolling back an upgrade.

Stage 3.C – Artifactory fails to upgrade its Config Descriptor

At this stage, Artifactory is trying to convert its XML descriptor but ran into an issue. This problem should have been recorded to the artifactory.log file with details on the XML issue.

To solve this issue, you will need to manually correct and bootstrap a config descriptor from the file system. Locate the $ART_HOME/etc/artifactory.config.latest.xml file, it will likely be the old XML version, which is ok:

[Expected log entries during a config bootstrap]
2019-09-10 23:04:44,973 [art-init] [INFO ] (o.a.d.r.CentralConfigReader:92) – The current Artifactory config schema namespace is 'http://artifactory.jfrog.org/xsd/2.2.3' The provided config does not seem to be compliant with it.
2019-09-10 23:04:45,026 [art-init] [INFO ] (o.a.d.r.CentralConfigReader:71) – Converting artifactory.config.xml version from 'v222' to 'v223'

Fix the XML syntax, then attempt to bootstrap the file. This is done by renaming the file to $ART_HOME/etc/artifactory.config.import.xml and restarting Artifactory.

Stage 3.D – Artifactory won't start after succeeding in the first 3 steps

Once this stage is hit, almost everything is ready to go. If Artifactory refuses to start because of a remaining bad configuration or similar error, troubleshooting this problem will most likely involve working with JFrog Support.