Configure Federated Repositories for Bulk Event Mirroring and Parallel Processing

JFrog Artifactory Documentation

ft:sourceType
Paligo

Bulk event mirroring, introduced in Artifactory version 7.63.2, helps to optimize Federation performance.

Before bulk event mirroring was introduced, artifacts were copied to target mirrors sequentially one event at a time per repository. This limited the number of events that could be processed in a reasonable amount of time and sometimes resulted in HTTP network bottlenecks, especially between geographically distant members where network latency can be a significant factor.

Bulk event mirroring bypasses potential HTTP network bottlenecks by collecting events into bulks, each of which is dedicated to a single repository, and sending them to the target mirror.

The target mirror assigns a dedicated thread pool to each bulk. Each pool contains multiple threads that upload artifacts to the local Artifactory in parallel. This parallel processing enables events to be uploaded much faster than was previously possible.

bulk-event-mirroring2.png

In addition to improving the standard event flow between Federation members, as described above, bulk events also improve the performance of Full Sync operations. Working in conjunction with a special cache mechanism (also introduced in 7.63.2), bulk mirroring helps to greatly reduce the load on Artifactory during Full Sync operations.

The following Event Queue properties can be configured for bulk event mirroring:

Property

Description

federated.mirror.events.bulk.exclude.repo.types=

Excludes specific package types from bulk event mirroring.

To exclude all package types:

federated.mirror.events.bulk.exclude.repo.types=*

To exclude specific package types, separate the types with a comma:

=docker,composer,npm

federated.mirror.events.bulk.max.pools

The number of available event processing thread pools (executors).

Default: 4

Maximum: 5

federated.mirror.events.bulk.threadsPerPool

The number of threads per pool (executor).

Default: 4

Maximum: 5

federated.mirror.events.batch.message.bulk.size

The number of Event Queue events sent with each replication message to the target mirror.

Default: 100

federated.mirror.events.bulk.executor.poll.timeout.minutes

The amount of time new requests will wait (in minutes) if all processing pools are occupied.

Default: 2

The following Full Sync event properties can be configured for bulk event mirroring:

Property

Description

federated.mirror.events.bulk.fullSync.threadsPerPool

The number of threads per pool. Full Sync events take priority in a single dedicated executor containing the number of threads defined here.

Default: 10

Maximum: Determined by the number of available processors.

federated.mirror.events.bulk.fullSync.executor.wait.timeout.minutes

The amount of time (in minutes) that Full Sync calls for other repositories wait while the pool works on the Full Sync of a particular repository.

Default: 2

artifactory.full.sync.batch.retry.count

The number of times to retry a failing batch (bulk) when performing Full Sync.

Default: 5

artifactory.full.sync.batch.retry.quiet.period.seconds

The amount of time to wait between batch (bulk) retries.

Default: 5 seconds

Note

This event property requires Artifactory 7.77.3 or above.