How to check integrity of binaries in Artifactory database against filestore?

Joshua Han
2019-07-08 22:30

ISSUE: Artifactory errors out with [ERROR] (o.a.r.c.e.GlobalExceptionMapper:48) – Binary provider has no content for … in artifactory.log

POSSIBLE REASONS:
1. Artifactory's data folder is not configured correctly, pointing to a wrong directory
1. binary checksum files in Artifactory's data folder is moved/removed externally, without being known to Artifactory

TROUBLESHOOTING:
You may run one of following methods to quickly check integrity of binaries in the database against filestore

Method 1 (Single NFS only)

1. Get a list of binaries from Artifactory's database by querying sha1 values from "binaries" table

e.g. For MySQL:
a. SELECT sha1 FROM binaries Order By sha1 ASC;
b. save the output as a file

2. Get a list of binaries from Artifactory's filestore.

a. Go to Artifactory's filestore folder (e.g. ../ha-data/filestore/)
b. Run following command
find . -mindepth 2 -type f -printf '%fn' | sort >> binaries-in-filestore.out

3. Find the differences between the results above using your favorite diff tool
e.g. 
User-added image

Method 2 (Single filestore only)

While the method above gives the most efficient way to compare DB and FIlestore, there is a user plugin that does this easier. Please note that this plugin only works with Artifactory instances that use a simple filesystem-based filestore (usually a local filesystem store, or an NFS mount).
https://github.com/jfrog/artifactory-user-plugins/tree/master/filestore/filestoreIntegrity

Method 3 (S3, Sharded store and others)
If you wish to find broken binaries but are using a more complex filestore, such as S3, HDFS, or a sharded store, you can use this script instead. Note that said script is not particularly fast or efficient, and may not be suitable for use on large Artifactory instances.
https://github.com/jfrog/artifactory-scripts/tree/master/filestoreIntegrity

 

RECOVERY
Recover the missing binary files from your backup locations