How Debian Repository metadata calculation works?

Overview of Debian Repository Structure

A Debian repository contains several directories organized to store and distribute software packages. Two of the most important directories are:

1. pool Directory

The pool directory is the central storage for all binary and source packages in the Debian repository.
Structure:

Packages are stored in subdirectories based on the first letter of their source package name (e.g., main, contrib, non-free).
Inside these, further subdirectories are organized alphabetically (e.g., a, b, c), and each directory contains the .deb file related to the package.
The pool directory is independent of distribution versions (e.g., stable, testing, unstable), preventing redundancy for packages shared across distributions.

2. dists Directory

The dists directory contains the metadata and organizational information about packages in the repository, structured by distribution and component.
Structure:

One subdirectory per distribution (e.g., bullseye, buster, sid).
Each contains:
- Component directories (main, contrib, non-free)
- Architecture-specific directories (amd64, i386, etc.)

Files in dists:

Release: Contains metadata about the distribution, such as the suite (e.g., stable, testing) and hash values for package lists.
Packages: Lists of packages available in the distribution, including details like name, version, and dependencies.

 Acts as the index of the repository, defining which packages are available for each distribution and architecture.

Local Repository Metadata Calculation

Event-Based Metadata Indexing
When a Debian package is deployed to a local repository, Artifactory adds an event to a queue to index the relevant distribution path.
Example:
For the repository debian-local and the distribution xenial, the distribution path would be: debian-local/dists/xenial
This queue is processed by dedicated Debian metadata workers (8 by default, configurable).
If the queue is empty and a worker is available, the event is picked up immediately, and metadata indexing begins.

To ensure consistency during processing, Artifactory applies a lock on the full distribution path, such as debian-local/dists/xenial.

High Availability (HA) Handling
In an HA cluster, multiple nodes can participate in processing the queue, but events for each path are handled sequentially to avoid conflicts.

In most environments, this process is fast enough that events are rarely queued for long — if at all.

Optional Metadata Compression Formats
Debian repositories support compressed metadata files to optimize bandwidth and storage usage.
Common formats include:

Packages.gz
Packages.xz
Packages.bz2

When a Debian client requests metadata, it selects the format based on its capabilities and preferences (e.g., the smallest or fastest to process).
In Artifactory, these formats can be configured under “Optional Index Compression Formats.”
By default, the Bzip2 (.bz2) format is enabled when creating a local Debian repository via the UI.
Tip: 
If you don’t need all formats, disabling unnecessary ones can significantly improve metadata calculation performance.


Debian Local Cache

Every Debian package whose metadata has been extracted is cached locally on the disk under $ARTIFACTORY_HOME/var/data/artifactory/.cache/debian folder. In this folder, you will find all the Debian repositories that are part of your Artifactory as directories, and these will include the metadata of the Debian packages that have been calculated. This is crucial in cases where you are running a synchronous calculation of the entire Debian repository. The presence of this folder will speed up the synchronous calculation significantly, as we would be retrieving metadata from the local disk instead of extracting it for each package again.

 

Virtual Repository Metadata Calculation

The virtual repository has its own metadata calculation process, which collects and combines the Packages and Release files from the selected aggregated repositories. Just like local repositories, it is indexed based on the distribution path and the virtual repository.
The implementation of metadata calculation for the virtual repository is similar to that of local repositories, using a dedicated queue and workers. By default, there are 5 workers assigned for this task, though this number can be adjusted as needed.
There are several important settings for the Debian virtual repository that can significantly affect the calculation time and performance:

Indexed Remote Architectures – As virtual repositories combine both local and remote repositories, specifying which architectures to index from the remote repositories can save a considerable amount of processing time. It’s advisable to index only what’s necessary.
Optional Index Compression Formats – This works in a similar manner to local repositories, but with even greater significance. Indexing only the required formats can help minimize calculation time.
Metadata Retrieval Cache Period – The default value is 2 hours. This determines how long the metadata cache remains valid. After the cache expires, a recalculation is required to download the metadata files. A longer cache period results in fewer recalculations, but may cause recently added items to be unavailable.

REST APIs

Calculate Debian Repository Metadata - Recalculate the metadata of an entire repository. Synchronous by default. Applicable to Local and Virtual repositories.

Calculate Cached Remote Debian Repository Coordinates - This API will add coordinates to cached Debian packages. This will make the packages resolvable if later moved to a local repository.


Tuning Debian Metadata Calculation

Heads Up! - Debian metadata calculation is based on a locking mechanism. Once a worker has started to index a specific combination of repository & distribution, another worker won’t start indexing another event for the same path while the lock exists. Increasing the workers’ allocation won’t help if all packages are deployed to the same repository and distribution.
The below-mentioned workers are part of the Artifactory async Thread Pool.
When changing the below configurations, you may need to consider increasing the total async Thread Pool.
Read more at: How do I tune Artifactory for heavy loads?

We will configure the mentioned properties in the $ARTIFACTORY_HOME/etc/artifactory.system.properties file:

Debian Local repository metadata calculation workers:

artifactory.debian.metadata.calculation.workers = 8

Debian Virtual repository metadata calculation workers:

artifactory.debian.virtual.metadata.calculation.workers = 5

Debian Cached Remote repository coordinates calculation workers:

artifactory.debian.coordinates.calculation.workers = 4

Troubleshooting Debian Metadata problems

Disclaimer: Changing log levels from their original value may result in performance degradation. Handle with care when applying to production systems.

To troubleshoot possible problems with the Debian Metadata calculation, we can add these loggers to $ARTIFACTORY_HOME/etc/logback.xml.
When applied, a "debian.log" file will appear in the $ARTIFACTORY_HOME/logs/ folder. A restart is not required for these changes to take effect.

Most of the Debian-related operations:
For Artifactory version 7.X:

<appender name="debian" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <File>${log.dir}/artifactory-debian.log</File>
    <rollingPolicy class="org.jfrog.common.logging.logback.rolling.FixedWindowWithDateRollingPolicy">
      <FileNamePattern>${log.dir.archived}/artifactory-debian.%i.log.gz</FileNamePattern>
      <maxIndex>10</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <MaxFileSize>25MB</MaxFileSize>
    </triggeringPolicy>
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
      <layout class="org.jfrog.common.logging.logback.layout.BackTracePatternLayout">
        <pattern>%date{yyyy-MM-dd'T'HH:mm:ss.SSS, UTC}Z [jfrt ] [%-5p] [%-16X{uber-trace-id}] [%-30.30(%c{3}:%L)] [%-20.20thread] - %m%n</pattern>
      </layout>
    </encoder>
  </appender>
  <logger name="org.jfrog.repomd.debian" additivity="false">
    <level value="debug"/>
    <appender-ref ref="debian"/>
  </logger>
  <logger name="org.artifactory.addon.debian" additivity="false">
    <level value="debug"/>
    <appender-ref ref="debian"/>
  </logger>
  <logger name="org.artifactory.addon.dpkgcommon" additivity="false">
    <level value="debug"/>
    <appender-ref ref="debian"/>
  </logger>

For Artifactory version 6.X:

<appender name="debian" class="ch.qos.logback.core.rolling.RollingFileAppender">
      <File>${artifactory.home}/logs/debian.log</File>
      <encoder>
         <pattern>%date ${artifactory.contextId}[%thread] [%-5p] \(%-20c{3}:%L\) - %m%n</pattern>
      </encoder>
      <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
         <FileNamePattern>${artifactory.home}/logs/debian.%i.log</FileNamePattern>
         <maxIndex>13</maxIndex>
      </rollingPolicy>
      <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
         <MaxFileSize>25MB</MaxFileSize>
      </triggeringPolicy>
</appender>

<logger name="org.jfrog.repomd.debian" additivity="false">
<level value="trace"/>
<appender-ref ref="debian"/>
</logger>
<logger name="org.artifactory.addon.debian" additivity="false">
<level value="trace"/>
<appender-ref ref="debian"/>
</logger>
<logger name="org.jfrog.repomd.dpkg" additivity="false">
<level value="debug"/>
<appender-ref ref="debian"/>
</logger>

Mover service (responsible for copy/move operations, being called during calculation finalization. This is a cross-Artifactory logger and not only Debian related):

<logger name="org.artifactory.repo.service.mover" additivity="false">
<level value="trace" />
<appender-ref ref="debian"/>
</logger>

Work queue (The core of the work queue used by Debian metadata calculations. This is a cross-Artifactory very verbose logger):

<logger name="org.artifactory.work.queue" additivity="false">
<level value="trace"/>
<appender-ref ref="debian"/>
</logger>

Information about read from local cache (This logger will help in troubleshooting issues with calculations of the entire repository):

<logger name="org.artifactory.addon.dpkgcommon" additivity="false">
<level value="trace"/>
<appender-ref ref="debian"/>
</logger>
<logger name="org.jfrog.repomd.dpkg.extractor.DpkgMetadataExtractor" additivity="false">
<level value="trace"/>
<appender-ref ref="debian"/>
</logger>