ARTIFACTORY: Why does my Linux Archive service installation fail/degrade after an upgrade?

Nir Ovadia
2022-07-03 09:17

Subject 

Why does my Linux Archive service installation fail/degrade after an upgrade?
 

Affected Versions

All 7.x linux archive installs that are installed as a service. This means you use systemd (systemctl/service) to control the application, and had run the $ARTIFACTORY_HOME/app/bin/installService.sh beforehand.
 

Description

Please note, all scripts mentioned (other than the setenv.sh script) are in $ARTIFACTORY_HOME/app/bin/
 
This issue can cause a variety of symptoms, from encoded characters (such as ‘/’ encoded as %2f in npm packages) not being parsed correctly, to tomcat logs not showing up, to the instance not taking in custom java options (such as increased memory). The reasoning behind this is because most of the JAVA_OPTS applied to the instance are missing. The explanation is a bit long-winded, but if you’re looking for a solution you can find a simple and short one at the bottom of the page.
 
So why is this happening? To understand, we need to go to the $ARTIFACTORY_HOME/app/bin directory, where most of the main elements in play reside. In here there is an artifactory. default file, which sets some defaults for Artifactory to use. It adds the JFrog and Tomcat home variables
 export JF_PRODUCT_HOME=/opt/jfrog/artifactory 
export TOMCAT_HOME=$JF_PRODUCT_HOME/app/artifactory/tomcat

 
and sets a lot of the default JAVA_OPTS, including the “-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH” that is required for some npm requests. It’s a small and simple file to set up defaults for Artifactory. In 7.x the systemYaml has an extraJavaOpts: line to set your own Java Options (such as increasing the -Xmx value), and on startup the application takes in the JAVA_OPTS from this file and appends the extraJavaOpts to apply them all.
 
If you start artifactory through a script, you would use “artifactory.sh start”. In the script, before the init() function that starts up Artifactory, it runs a few setup steps and sets some environment variables. As you can see in the snippet below, the artifactory.default file is called to set all of the variables in that file
 artDefaultFile="${ARTIFACTORY_BIN_FOLDER}/artifactory.default"
 
. ${artDefaultFile} || errorArtHome "ERROR: $artDefaultFile does not exist or not executable"
PRODUCT_ETC=${JF_PRODUCT_HOME}/var/etc
ART_ETC=$PRODUCT_ETC/$ARTIFACTORY_NAME
ACCESS_ETC=$PRODUCT_ETC/$ACCESS_NAME
REPLICATOR_ETC=$PRODUCT_ETC/$REPLICATOR_NAME
 
init

 
and then later the script calls TOMCAT_HOME (set in artifactory.default) to start up Tomcat and Catalina.
 
So it’s pretty simple so far, Artifactory script -> calls file to setup environment -> set up Tomcat -> start Artifactory. Now the thing is when you call the installService.sh script to set up the service, it doesn’t use artifactory.sh. We have a different file, artifactoryManage.sh, that is compatible with the systemctl/systemd procedure. So if I run the installService.sh file it creates a /lib/systemd/system/artifactory.service file (might be different depending on OS, you can run a systemctl status artifactory to see the file location). Looking into the file, we see the “start” and “stop” commands are using “artifactoryManage.sh start/stop”
 ExecStart=/opt/jfrog/artifactory/app/bin/artifactoryManage.sh start
ExecStop=/opt/jfrog/artifactory/app/bin/artifactoryManage.sh stop

 
The main difference between artifactory.sh and artifactoryManage.sh is that artifactory.sh calls Tomcat and the artifactory.default file directly, but artifactoryManage.sh uses a specific file to call them. This file is the setenv.sh file (more info from tomcat here https://tomcat.apache.org/tomcat-7.0-doc/RUNNING.txt). The installService.sh file performs a copy of the file from the default location to the intended TOMCAT_HOME location like so
 cp ${serviceFiles}/setenv.sh ${TOMCAT_HOME}/bin/setenv.sh && \
 

which is by default from $ARTIFACTORY_HOME/app/misc/service/setenv.sh to  $ARTIFACTORY_HOME/app/artifactory/tomcat/bin/setenv.sh. If you check out the setenv.sh file, it looks like this
 #!/bin/bash
. ${JF_PRODUCT_HOME}/app/bin/artifactory.default
echo "Max number of open files: $(ulimit -n)"
echo "Using JF_PRODUCT_HOME: ${JF_PRODUCT_HOME}"
echo "Using JF_ARTIFACTORY_PID: $JF_ARTIFACTORY_PID"
 
export CATALINA_TMPDIR=${JF_PRODUCT_HOME}/var/work/artifactory/tomcat/temp
export CATALINA_OPTS="$CATALINA_OPTS $JAVA_OPTIONS -Djruby.bytecode.version=1.8"
export CATALINA_HOME="$TOMCAT_HOME"

As you can see, it calls the artifactory.default file, sets the CATALINA_OPTS (tomcat webapp that artifactory runs on) to use the JAVA_OPTS set, and sets the CATALINA_HOME for the webapp. The next part gets a bit complicated if we look at the scripts, but essentially the artifactoryManage.sh calls the tomcat startup script, which calls the catalina.sh script, which looks for setenv.sh with CATALINA_HOME and runs the file. 
 
So to summarize, with the installService.sh command the setenv.sh gets moved to the Tomcat home directory, and uses artifactoryManage.sh. This script calls Tomcat’s startup script, which calls catalina.sh, which uses some of the environment variables to call setenv.sh. Unlike the artifactory.sh script, this one relies a lot more on calling other scripts.
 
Here is where the issue arises. When you run an upgrade for a Linux Archive instance, you replace the full /app/ directory (to update the binaries in that directory). A side effect of that is that the setenv.sh script is no longer in the Tomcat home, it was only there because we ran the installService.sh. This means when we run systemctl start artifactory, it fails to set a lot of the environment variables that set up the environment around it so it starts up with a limited amount of hard coded defaults (no artifactory.default variables, no catalina_home, etc). In some situations due to the low default resources this can even fail the startup, while in others you see degradation and failures due to other symptoms.
 

Resolution

The explanation of the issue was long, but the resolution is fairly simple. All we need to do is restore the setenv.sh to the tomcat home location, so we can just run installService.sh again. Once doing that, the instance can be started up and runs with the proper resources and variables. As a note for future upgrades, any time you upgrade the linux archive install you should run installService.sh as well before starting up.