For large files, JFrog Artifactory implements a fast and reliable multi-part upload approach with the JFrog CLI. In case of upload failure, the multi-part approach has a retry mechanism that resumes uploads from the point of failure. This approach preserves all content that was uploaded before the point of failure. In contrast, with standard uploads, upload failure results in the loss of all data.
The following diagram shows the sequence of the events that occur when you perform a multi-part upload. This example uses AWS S3 storage but note that Google Cloud storage is also supported.
An upload request is submitted using JFrog CLI.
Artifactory authenticates the request.
JFrog CLI splits the file into parts and sends a request to Artifactory for five pre-signed URLs for the first five parts.
Note
The default number of parts that can be uploaded concurrently is five, but this number can be modified with the
--split-countvariable in the JFrog CLI.Artifactory sends the pre-signed URLs to JFrog CLI.
JFrog CLI concurrently uploads the first five parts directly to the S3 bucket.
Steps 3-5 are repeated until all parts are uploaded.
When all parts have been uploaded, JFrog CLI sends a
‘Complete Upload’trigger to the S3 bucket.The S3 bucket merges the parts into a single file (this merge can take some time). While the merge is happening, JFrog CLI polls Artifactory regarding the progress of the merge.
Artifactory verifies the checksum and notifies JFrog CLI that the upload has completed successfully.
Multi-Part Upload Settings
The JFrog CLI automatically uses multi-part upload for large files without intervention, according to the values of the following settings:
--min-split: Determines the minimum file size required for multi-part upload. The default is 200 MB.--split-count: Determines the number of parts that can be concurrently uploaded per file during a multi-part upload. The default is 5.--chunk-size: Determines the upload chunk size in MB of the parts that are concurrently uploaded. The default value is 20.
You can change the values when you run an upload command in the CLI. For example:
jf rt upload filename.zip /target-repo --min-split=150 --split-count=6 --chunk-size=25
After running a command with changed values, the values for these settings revert to the default. For more information on uploading files with the JFrog CLI, see Generic Files.
Multi-Part Upload Availability
Multi-part upload is available with the following software configurations:
Artifactory version 7.90.7 and later
JFrog CLI version 2.62.2 and later
Artifactory instances using AWS S3 or Google Cloud storage
Configuring Multi-Part Upload for Self-Managed Instances
Multi-part upload is available on self-managed instances of Artifactory using S3 Storage or Google Cloud Storage configured in binarystore.xml.
Multi-part upload is enabled for self-managed Artifactory instances using the system property artifactory.multipart.upload.enabled. For more information, see Artifactory System Properties.
Note
Multi-part upload is not supported on self-managed platforms with S3 sharding, or when S3 storage is configured with client-side KMS encryption (kmsClientSideEncryptionKeyId).
Multi-Part Upload with Encryption using a KMS Key
To perform multi-part upload with encryption using a KMS key, you must have permission for the kms:Decrypt and kms:GenerateDataKey actions on the key. These permissions are required because Amazon S3 must decrypt and read data from the encrypted file parts before it completes the multi-part upload.
If your IAM user or role is in the same AWS account as the KMS key, then you must have these permissions on the key policy. If your IAM user or role is not in the AWS account as the KMS key, then you must have permissions on both the key policy and your IAM user or role.