Checksum-Based Storage Implementation

JFrog Installation & Setup Documentation

Content Type
Installation & Setup
ft:sourceType
Paligo

The following sections provide more information on how checksum-based storage features are implemented in Artifactory.

Deduplication

Artifactory stores any binary file only once, and this is what we call once and once only storage. First time a file is uploaded, Artifactory runs the required checksum calculations when storing the file. However, if the file is uploaded again (to a different location, for example), the upload is implemented as a simple database transaction that creates another record mapping the file's checksum to its new location. There is no need to actually store the file again in storage. No matter how many times a file is uploaded, the filestore only hosts a single copy of the file.

Copying and Moving Files

Copying and moving a file are implemented by simply adding and removing database references, and the performance of these actions corresponds to that of a database transaction.

Deleting Files

Deleting a file is also a simple database transaction in which the corresponding database record is deleted. The file itself is not directly deleted, even if the last database entry pointing to it is removed. Orphaned files are removed in the background by Artifactory's garbage collection processes.

Upload, Download, and Replication

Before moving files from one location to another, Artifactory sends checksum headers. If the files already exist in the destination, they are not transferred even if they exist under a different path.

Filesystem Performance

Filesystem performance is greatly improved because actions on the filestore are implemented as database transactions, so there is no need to do a write-lock on the filesystem.

Checksum Search

Searching for a file by its checksum is extremely fast since Artifactory is actually searching through the database for the specified checksum.

Flexible Layout

Since the database acts as an indirection layer between the filestore and the displayed layout, any layout can be supported. This includes standard packaging formats like Maven1, Maven2, npm, NuGet, and custom layouts.