ARTIFACTORY: How to Resolve Derby Migration Failure ('service.properties' Missing) Error During the Upgrade

Introduction 

This article addresses a startup failure that can occur during Artifactory upgrades from version 7.90 onwards when the instance is using the embedded Derby database. Starting with Artifactory 7.90, the Access service was separated to run on its own dedicated JVM.

For installations using Derby, this separation requires the creation of a new, dedicated Access Derby database at $JFROG_HOME/artifactory/var/data/access/derby. The startup process then copies the Derby data from the main Artifactory database to this new Access location. If there is insufficient disk space to complete this copy, the migration fails, resulting in the Artifactory service failing to start.


Problem

When Artifactory upgrades to a version that separates the Access service (e.g. 7.90+), the startup process executes a mandatory data migration:

1. A new Derby database directory is created for the Access service: $JFROG_HOME/artifactory/var/data/access/derby.

2. The startup process copies the data from the main Artifactory Derby database ($JFROG_HOME/artifactory/var/data/artifactory/derby) to the new Access location.

3. This copy requires temporary free disk space that is approximately equal to the current size of the main Artifactory Derby folder. For example, if the main Artifactory Derby folder is 100GB, an additional 1000GB of free space is needed.

4. If the VM runs out of disk space during this large data copy/migration, the process fails mid-write.

5. The system fails to finalize the new database and generate the critical service.properties file.

The resulting error observed in the Artifactory Access logs is:

SQL State : XBM0A Error Code : 45000 Message : The database directory '/var/opt/jfrog/artifactory/data/access/derby' exists. However, it does not contain the expected 'service.properties' file. Perhaps Derby was brought down in the middle of creating this database. You may want to delete this directory and try creating the database again.

Attempting to manually copy the service.properties file from the main Artifactory Derby folder will typically lead to deeper corruption errors, such as a Container Missing error (SQL State : 40XD2).


Resolution

The ultimate cause is lack of disk space for the transactional copy. The resolution is to provide the required free space and then force a clean retry of the migration.

Step-by-Step Instructions

1. Stop Artifactory Service

   1. Ensure the Artifactory service is fully shut down.

2. Calculate and Allocate Disk Space

   1. Determine the size of your main Artifactory Derby folder ($JFROG_HOME/artifactory/var/data/artifactory/derby).

   2. Allocate sufficient disk space to the VM to have at least this size free, plus a buffer. (e.g. if the folder is 100GB, ensure you have well over 100GB of free space).

3. Delete Corrupted Access Derby Folder

   1. Remove the entire contents of the corrupted Access Derby directory to allow for a clean migration attempt on restart.

rm -rf /var/opt/jfrog/artifactory/data/access/derby

4. Start Artifactory Service

   1. Start the Artifactory service.

   2. With sufficient disk space, the system will now complete the migration process by copying the data (e.g. the full 100GB), generate the service.properties file automatically upon completion, and Artifactory will start successfully.

   3. Verification: All Artifactory data, including repositories and Access information (users, groups, permissions), will remain intact.

Recommendations for Future Upgrades

• Migrate to an External Database: Due to restricted functionalities, the embedded Derby database can impact the performance and resilience of your Artifactory instance. Hence, JFrog highly recommends avoiding the usage of the Derby database for your Artifactory production environments. JFrog recommends migrating to an external database like PostgreSQL to ensure system stability.

• Backup and Restore: Always ensure the Artifactory VM and database are fully backed up and restorable before initiating any major upgrade.

• Verify Disk Space: Prior to upgrading, verify available disk space is sufficient to accommodate temporary data expansion, which, as demonstrated here, can be substantial when migrating Derby databases.