In Artifactory, when attempting to upload an artifact to Azure Blob Storage, you may encounter the following error in the Artifactory logs:
artifactory-service.log:
2022-11-03T02:41:31.241Z [jfrt ] [ERROR] [51b3889e9a4bd60 ] [n.NpmPackageMetadataIndexer:64] [p-nio-8081-exec-1152] - Failed index @gkuis/gkp-billing-ui
java.lang.RuntimeException: Failed to save stream to npm-local:.npm/@gkuis/gkp-bil-ui/package.json
....
Caused by: org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
at org.jfrog.type.azureblobstorage.v1.AzureBlobStorageBinaryProvider.saveToTempFileAndUpload(AzureBlobStorageBinaryProvider.java:491)
...
Caused by: java.io.IOException: Failed to put blob '5b65485960e061f01c8c7ea14905a867a6dcc8cf' to azure blob storage
at org.jfrog.type.azureblobstorage.v1.AzureBlobStorageBinaryProvider.addSmallStream(AzureBlobStorageBinaryProvider.java:297)
at org.jfrog.type.azureblobstorage.v1.AzureBlobStorageBinaryProvider.uploadStream(AzureBlobStorageBinaryProvider.java:269)
at org.jfrog.type.azureblobstorage.v1.AzureBlobStorageBinaryProvider.saveToTempFileAndUpload(AzureBlobStorageBinaryProvider.java:485)
... 104 common frames omitted
Caused by: java.io.IOException: The specified block list is invalid. Please see the cause for further information.
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:769)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.close(BlobOutputStreamInternal.java:327)
at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:899)
at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:743)
at org.jfrog.type.azureblobstorage.v1.AzureBlobStorageBinaryProvider.addSmallStream(AzureBlobStorageBinaryProvider.java:288)
... 106 common frames omitted
Caused by: com.microsoft.azure.storage.StorageException: The specified block list is invalid.
Cause
This issue is caused by Azure Blob Storage's block list limit. A block blob can contain up to 50,000 blocks.
Each uncommitted block must be committed to finalize the data upload.
Furthermore, a blob can have a maximum of 100,000 uncommitted blocks at any time. If the number of blocks exceeds this limit, Azure will throw an error similar to the one above: "The specified block list is invalid."
Solution
To resolve this issue, we recommend adjusting the multipart upload parameters in Artifactory’s binarystore.xml configuration for Azure Blob Storage.
This will reduce the number of blocks per file and avoid hitting the Azure limit.
Steps to Resolve
1. Update binarystore.xml Configuration: You will need to configure the following parameters in your binarystore.xml file to enable multipart uploads with larger chunk sizes:
Add or update these parameters in your binarystore.xml under the Azure Blob Storage configuration section:
E.g:
<config version="3">
<chain template="azure-blob-storage-v2-direct"/>
<provider type="cache-fs" id="cache-fs">
<cacheProviderDir>/var/opt/jfrog/artifactory/data/cache</cacheProviderDir>
<maxCacheSize>10000000000</maxCacheSize>
</provider>
<provider id="azure-blob-storage-v2" type="azure-blob-storage-v2">
<accountName>*****</accountName>
<accountKey>*****</accountKey>
<endpoint>https://artifactoryhaprod.xxxxxx.xxxxx.xxxxxxx.net/</endpoint>
<container>mycontainer</container>
<multiPartLimit>400000000</multiPartLimit>
<multipartElementSize>100000000</multipartElementSize>
</provider>
</config>
In the above example, multipartElementSize=100MB, which ensures that each artifact is split into chunks of 100MB or less and multiPartLimit=400MB, This allows the entire artifact to be uploaded in chunks of up to 400MB.
2. Restart Artifactory: After making these changes, you will need to restart Artifactory for the configuration to take effect.
3. Verify the Changes: Once Artifactory has restarted, attempt the upload again. You should see fewer blocks per file upload, which will help avoid hitting the 100,000 uncommitted block limit imposed by Azure.
External References: You can also review the following articles for additional information on troubleshooting and handling block list errors in Azure Blob Storage:
Conclusion
By configuring the multipartElementSize and multiPartLimit parameters as described above, you should be able to avoid the "The specified block list is invalid" error when uploading artifacts to Azure Blob Storage. If you continue to experience issues, please share your results with the JFrog support team for further investigation.