Introduction
Artifactory relies on a database to store all configurations and metadata necessary for its operation. For Artifactory to function properly, this database must be continuously accessible and connected.
To facilitate this connection, Artifactory uses a JDBC driver and a master key . The JDBC driver enables the application to communicate with the database, while the master key is essential for securing sensitive information, such as database credentials, by encrypting and decrypting data.
JFrog highly recommends using an external PostgreSQL database, and we supply the JDBC driver. However, we support other database types for which the user is required to manually install the JDBC driver. Please see supported database types and versions for Artifactory here.
This article outlines steps for troubleshooting common issues related to database connectivity and performance, helping us ensure that our Artifactory remains operational as expected.
Artifactory Failed Connecting to the Database
Common indicators in the log files for database connection issue:
- ‘master.key’ is missing or wrong:
[security_keys.go:172 ] [main ] [] - Master key is missing. Pending for 10 seconds with 5m0s timeout
- Wrong database credentials:
[jfmd ] [WARN ] [ ] [jobs.go:98 ] [main ] - [RETRY] Initiating database connection: failed to connect to `host=<HOSTNAME> user=artifactory database=artifactory`: server error (FATAL: password authentication failed for user "artifactory" (SQLSTATE 28P01)). Retrying in 1s ... (7/120) [database]
- Missing or incorrect JDBC driver:
[jfrt ] [ERROR] [8fcfc331962784f0] [tifactoryHomeConfigListener:55] [Catalina-utility-1 ] - Failed initializing Home. Caught exception: java.lang.RuntimeException: Failed to load driver class org.postgresql.Driver in either of HikariConfig class loader or Thread context classloader
- Connecting to the wrong database - If an external database is used, but upon startup we see that Artifactory started up connecting to the bundled Derby database:
[.i.DbInitializationManager:177] [Catalina-utility-1 ] - Database: Apache Derby 10.14.2.0 - (1828579). Driver: Apache Derby Embedded JDBC Driver 10.14.2.0 - (1828579) Pool: derby
These indicators would likely be followed by general errors - “router readiness test” failures:
Readiness test failed with the following error: "required node services are missing or unhealthy"
These errors would be printed in a loop until the services reached the timeouts.
Resolution
To ensure that Artifactory can establish a successful connection to the database, we need to make sure that the database is configured to listen to the Artifactory server’s IP address, modifying the database configuration files (pg_hba.conf + postgresql.conf for PostgreSQL, my.cnf for MySQL etc.)
It's important to verify that the master.key is correct and in place within $JFROG_HOME/artifactory/var/etc/security (upon fresh installation, the master.key may be created automatically if not specified otherwise).
Additionally, ensure that the JDBC driver .jar is in the correct location on the Artifactory server and is loaded properly during startup. The correct driver and how to set up your database can be seen in our documentation.
The driver should be placed in the directory:
$JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib
** Note
Upon startup post upgrade or installation, the driver is copied from this location to the JVMs directories.
By keeping the driver under this location, it will not be deleted during upgrades, as the $JFROG_HOME/artifactory/app directory is overwritten with new binaries.
**
Next, we need to ensure that the JDBC driver .jar file has the correct ownership and permissions.
Ownership: The .jar file should be owned by the user that runs Artifactory. By default, both the user and the group are named ‘artifactory’.
Permissions: The .jar file must have the appropriate permissions to be executed.
We can set the correct ownership and permissions using the following commands:
chown -R artifactory:artifactory $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib/* chmod +x $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib/*.jar
It should look like this:
Additionally, ensure that the file has the correct ownership and permissions in the JVM directories where it will be used, post start up of the service, after they are copied to these locations.
Artifactory JVM:
Access JVM:
Next, verify the connection configuration for the database in the system.yaml file:
Connection String: Ensure that the database connection string in the system.yaml file is accurate. This string typically includes the database host, port, and database name, and may follow different params (e.g, SSL).
Credentials: Check that the database username and password are correct. Note that once Artifactory connects to the database for the first time, the password will be encoded in the system.yaml file.
After confirming these details, we can validate the entire system.yaml file to prevent any syntax or structural issues. You can use an online YAML validator, such as https://www.yamllint.com/ to check for the following:
- Proper indentation
- Absence of duplicate properties
- Overall file validity
Database Performance Issues - Slowness, Latency or Crashes
If we experience performance issues with Artifactory, such as slowness, latency, or crashes, it’s important to conduct a thorough analysis of the database and system resources, including:
- Monitor Metrics: Review monitoring tools and database metrics during the timeframe of the incident. Look for spikes in CPU usage, RAM consumption, or network bandwidth. Excessive usage in these areas may indicate that the database is overloaded and could benefit from resource tuning.
- Resource Tuning: If we identify resource overload, consider adjusting settings for resource allocation. This may include resizing your database instance and/or increasing the maximum number of Open Connections.
*Recommendation*
See the following Architecture Manual for sizing reference. Refer to page 37-38 for more information.
**
- Identify Long-Running Queries: Query the database for any long-running AQLs (Artifactory Query Language queries). These queries can often cause performance bottlenecks.
For any unusually long-running AQLs, consider rewriting them to be more efficient, with a focus on accuracy and precision. This can help reduce execution time and minimize resource consumption.
By following these steps, we can better understand and address the underlying causes of database performance issues, ensuring that Artifactory runs smoothly.
Conclusion
Keeping Artifactory’s connection to the database stable and efficient is vital for ensuring smooth operations. This article outlined key steps for troubleshooting common connectivity and performance issues.