Puppet Repositories

Overview

Artifactory provides full support for managing Puppet modules, ensuring optimal and reliable access to Puppet Forge . By aggregating multiple Puppet repositories under a single virtual repository Artifactoryenables upload and download access to all your Puppet modules through a single URL.

As a fully-fledged Puppet repository, on top of its capabilities for advanced artifact management, Artifactory's support for Puppet provides:

The ability to provision Puppet modules from Artifactory to the Puppet command line tool for all repository types.
Calculation of Metadata for Puppet modules hosted in Artifactory's local repositories.
Access to remote Puppet repositories, such as https://forgeapi.puppetlabs.com/, using the Remote Repositories which provides proxy and caching functionalities.
Access to multiple Puppet repositories from a single URL by aggregating them under Virtual Repositories. This overcomes the limitation of the Puppet client which can only access a single registry at a time.
Support for flexible puppet repository layouts that allow you to organize your Puppet modules, and assign access privileges according to projects or development teams.

Puppet version support

Puppet does not support a context path up to version 4.9.1, we recommend using Artifactory with Puppet version 4.9.2 and above. Please see below if you are using Puppet 4.9.1 and below.

Configuration

Local Puppet Repository

To enable the calculation of Puppet module metadata in local repositories, from the Administration module go to Repositories |Repositories | Local set the Package Type to Puppet when you create the repository.

Repository Layout

Artifactory allows you to define any layout for your Puppet repositories. To upload a module according to your custom layout, you need to package your Puppet files using puppet module build.

This creates a .tar.gz file for your module which you can then upload to any path within your local Puppet repository.

Remote Puppet Repository

A Remote Repositories defined in Artifactory serves as a caching proxy for a repository managed at a remote URL such as https://forgeapi.puppetlabs.com/.

Artifacts (such as tar.gz files) requested from a remote repository are cached on demand. You can remove downloaded artifacts from the remote repository cache, however, you can not manually deploy artifacts to a remote Puppet repository.

To define a remote repository to proxy a remote Puppet resource follow the steps below:

In the Administration module, under Repositories | Repositories | Remote, click New Remote Repository.
In the New Remote Repository dialog, set the Package Type to Puppet, set the Repository Key value, and specify the URL to the remote repository in the URL field as displayed below.
Click Save & Finish.

Virtual Puppet Repository

A virtual repository, in Artifactory, aggregates modules from both local and remote repositories.

This allows you to access both locally hosted Puppet modules and those from remote proxied Puppet repositories from a single URL defined for the virtual repository.

To define a virtual Puppet repository, from the Administration module, select Repositories | Repositories | Virtual, set the Package Type to Puppet, and select the underlying local and remote Puppet repositories to include in the Basic settings tab.

Click Save & Finish to create the repository.

Using the Puppet Command Line

When accessing a Puppet repository through Artifactory, the repository URL path must be prefixed with api/puppet.

This applies to all Puppet commands including puppet module install and puppet module search.

For example, if you are using Artifactory standalone or as a local service, you would access your Puppet repositories using the following URL:

http://localhost:8081/artifactory/api/puppet/<REPO_KEY>

Or, if you are using Artifactory Cloud the URL would be:

https://<server name>.jfrog.io/artifactory/api/puppet/<REPO_KEY>

To use the Puppet command line you need to make sure Puppet is installed on your client.

Once you have created your Puppet repository, you can select it in the Application Module | Artifactory | Artifacts Tree Browser and click the Set Me Up button to get helpful code snippets. These allow you to change your Puppet repository URL in the puppet.conf file, and resolve modules using the Puppet command line tool.

Replacing the default repository

To replace the default repository with a URL pointing to a Puppet repository in Artifactory, add the following in your puppet.conf file:

Note: This example uses a repository with the key puppet.

[main]
module_repository=http://localhost:8080/artifactory/api/puppet/puppet

Tip: We recommend referencing a Virtual Repository URL as a repository. This gives you the flexibility to reconfigure and aggregate other external sources and local repositories of Puppet modules you deploy.

Note that if you do this, you can also use the --module_repository parameter to specify the local repository from which you want to resolve your module when using the Puppet module install command.

Once the Puppet command line tool is configured, every puppet module install command will fetch packages from the Puppet repository specified above. For example:

$ puppet module install --module_repository=http://localhost:8080/artifactory/api/puppet/puppet puppetlabs-mysql
Notice: Preparing to install into /Users/jainishs/.puppetlabs/etc/code/modules ...
Notice: Downloading from http://localhost:8080/artifactory/api/puppet/puppet ...
Notice: Installing -- do not interrupt ...
/Users/jainishs/.puppetlabs/etc/code/modules
└── puppetlabs-mysql (v3.10.0)

Using librarian-puppet

librarian-puppet is a bundler for your Puppet infrastructure. From version 5.4.5, you can use librarian-puppet with Artifactory as a Puppet repository to manage the Puppet modules your infrastructure depends on.

To configure librarian-puppet to fetch modules from Artifactory, add the following to your Puppetfile:

forge "http://<ARTIFACTORY_HOST_NAME>:<ARTIFACTORY_PORT>/artifactory/api/puppet/<REPO_KEY>"

For example, a Puppetfile that uses librarian-puppet could look something like this:

forge "http://localhost:8080/artifactory/api/puppet/puppet-local"

mod  'puppetlabs-mysql', '3.7.0'
mod  'puppetlabs-apache', '1.5.0'

To fetch and install the Puppet modules from Artifactory, run the following command:

librarian-puppet install --no-use-v1-api

Using r10k

r10k is a Puppet environment and module deployment tool. From version 5.4.5, you can use r10k to fetch Puppet environments and modules from an Artifactory Puppet repository for deployment.

To configure r10k to fetch modules from Artifactory, add the following to your r10k.yaml file:

forge: 
   baseurl: 'http://<ARTIFACTORY_HOST_NAME>:<ARTIFACTORY_PORT>/artifactory/api/puppet/<REPO_KEY>'

For example:

forge:
  baseurl: 'http://localhost:8080/artifactory/api/puppet/puppet-local'

To fetch and install the Puppet modules from Artifactory, run the following command:

r10k puppetfile install

Puppet Publish (Deploying Modules)

Setting Your Credentials

To support authentication, you need to add your Artifactory credentials to your puppet.conf file:

Note: your credentials should be formatted as username:password as a Base64 encoded strings

Your Artifactory credentials, formatted username:password as Base64 encoded strings.

For example:

[main]
module_repository=http://admin:AP7eCk6M6JokQpjXbkGytt7r4sf@localhost:8080/artifactory/api/puppet/puppet-local

Note

Make sure you have an Artifactory user in order to publish modules.

Deploying Your Modules

There are two ways to deploy packages to a local repository:

Using the Artifactory UI Once you have created your Puppet repository, you can select it in the Tree Browser and click Deploy to upload the Puppet module. Select your module(s), and click Deploy.
Using Artifactory REST API
For Example:
curl -uadmin:AP7eCk6M6JokQpjXbkGytt7r4sf -XPUT http://localhost:8080/artifactory/puppet-local/<TARGET_FILE_PATH> -T <PATH_TO_FILE>

Working with Artifactory without Anonymous Access

By default, Artifactory allows anonymous access to Puppet repositories. This is defined in the Administration module under Security | General. For details please refer to Allow Anonymous Access.

To be able to trace how users interact with your repositories, you need to uncheck the Allow Anonymous Access setting. This means that users will be required to enter their username and password as described in Setting Your Credentials above.

Puppet Search

Artifactory supports a variety of ways to search for artifacts. For details, please refer to Searching for Artifacts_OLD. However, a module may not be available immediately after being published, for the following reasons:

When publishing modules to a local repository, Artifactory calculates the search index asynchronously and will wait for indexing the newly published module.
Since a virtual repository may contain local repositories, a newly published package may not be available immediately for the same reason.
In the case of remote repositories, a new package will only be found once Artifactory checks for it according to the Retrieval Cache Period setting.

Tip

Artifactory annotates each deployed or cached Puppet module with two properties: puppet.name and puppet.version

You can use Property Search to search for Puppet packages according to their name or version.

Cleaning Up the Local Puppet Cache

The Puppet client saves caches of downloaded modules and their JSON metadata responses (called .cache.json).

The JSON metadata cache files contain the Puppet modules' metadata.

We recommend removing the Puppet caches, both modules and metadata, before using Artifactory for the first time. This is to ensure that your caches only contain elements that are due to requests from Artifactory and not directly from https://forge.puppet.com.

Viewing Individual Puppet Module Information

Artifactory lets you view selected Puppet module metadata directly from the UI.

Drill down in the Tree Browser and select the tar.gz file you want to inspect and view the metadata in the Puppet Info tab.

Screen Shot 2017-02-14 at 4.41.53 PM.png

Using Puppet 4.9.1 and Below

Up till version 4.9.1, the Puppet client does not support context path for remote Puppet Forge repositories. Therefore, we recommend using Artifactory with Puppet 4.9.2 and above.

If you need to use Puppet 4.9.1 and below you can use a workaround that uses NGINX or Apache to rewrite all requests from /v3/* to /artifactory/api/puppet/<repo-name>/v3/*.

For example, if you have a repository called puppet-virtual, and you are using Puppet 3.0, you would configure your proxy server to rewrite /v3/* to /artifactory/api/puppet/puppet-virtual/v3/*.

The following sections show sample configurations for NGINX and Apache for both the ports method and the sub-domain method to use a virtual repository named puppet-virtual.

## server configuration
server {
    listen 443 ssl;
    listen 80 ;
    server_name ~(?<repo>.+)\.artifactory-cluster artifactory-cluster;
    
    if ($http_x_forwarded_proto = '') {
        set $http_x_forwarded_proto  $scheme;
    }
    ## Application specific logs
    ## access_log /var/log/nginx/artifactory-cluster-access.log timing;
    ## error_log /var/log/nginx/artifactory-cluster-error.log;
    rewrite ^/$ /artifactory/webapp/ redirect;
    rewrite ^/artifactory/?(/webapp)?$ /artifactory/webapp/ redirect;
    rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2;
    rewrite ^/v3/(.*) /artifactory/api/puppet/$repo/v3/$1;
    chunked_transfer_encoding on;
    client_max_body_size 0;
    location /artifactory/ {
    proxy_read_timeout  900;
    proxy_pass_header   Server;
    proxy_cookie_path   ~*^/.* /;
    if ( $request_uri ~ ^/artifactory/(.*)$ ) {
        proxy_pass          http://artifactory/artifactory/$1;
    }
    proxy_pass          http://artifactory/artifactory/;
    proxy_next_upstream http_503 non_idempotent;
    proxy_set_header    X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port/artifactory;
    proxy_set_header    X-Forwarded-Port  $server_port;
    proxy_set_header    X-Forwarded-Proto $http_x_forwarded_proto;
    proxy_set_header    Host              $http_host;
    proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
    }
}

###########################################################
## this configuration was generated by JFrog Artifactory ##
###########################################################


## add HA entries when ha is configured
<Proxy balancer://artifactory>
    BalancerMember http://10.6.16.125:8080 route=14901314097097
ProxySet lbmethod=byrequests
ProxySet stickysession=ROUTEID
</Proxy>
<VirtualHost *:80>
    ProxyPreserveHost On
    ServerName artifactory-cluster
    ServerAlias *.artifactory-cluster
    ServerAdmin server@admin



    ## Application specific logs
    ## ErrorLog ${APACHE_LOG_DIR}/artifactory-cluster-error.log
    ## CustomLog ${APACHE_LOG_DIR}/artifactory-cluster-access.log combined
    AllowEncodedSlashes On
    RewriteEngine on
    RewriteCond %{SERVER_PORT} (.*)
    RewriteRule (.*) - [E=my_server_port:%1]
    ##  NOTE: The 'REQUEST_SCHEME' Header is supported only from apache version 2.4 and above
    RewriteCond %{REQUEST_SCHEME} (.*)
    RewriteRule (.*) - [E=my_scheme:%1]


    RewriteCond %{HTTP_HOST} (.*)
    RewriteRule (.*) - [E=my_custom_host:%1]


    RewriteRule ^/$                /artifactory/webapp/ [R,L]
    RewriteRule ^/artifactory(/)?$      /artifactory/webapp/ [R,L]
    RewriteRule ^/artifactory/webapp$   /artifactory/webapp/ [R,L]


    RequestHeader set Host %{my_custom_host}e
    RequestHeader set X-Forwarded-Port %{my_server_port}e
    ## NOTE: {my_scheme} requires a module which is supported only from apache version 2.4 and above
    RequestHeader set X-Forwarded-Proto %{my_scheme}e
    RequestHeader set X-Artifactory-Override-Base-Url %{my_scheme}e://artifactory-cluster:%{my_server_port}e/artifactory
    ProxyPassReverseCookiePath /artifactory /artifactory


    ProxyRequests off
    ProxyPreserveHost on
    ProxyPass /artifactory/ balancer://artifactory/artifactory/
    ProxyPassReverse /artifactory/ balancer://artifactory/artifactory/
    Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/artifactory/" env=BALANCER_ROUTE_CHANGED
</VirtualHost>


Listen 8001
<VirtualHost *:8001>
    ProxyPreserveHost On
    ServerName artifactory-cluster
    ServerAlias *.artifactory-cluster
    ServerAdmin server@admin
    ## Application specific logs
    ## ErrorLog ${APACHE_LOG_DIR}/artifactory-cluster-error.log
    ## CustomLog ${APACHE_LOG_DIR}/artifactory-cluster-access.log combined


    AllowEncodedSlashes On
    RewriteEngine on


    RewriteCond %{SERVER_PORT} (.*)
    RewriteRule (.*) - [E=my_server_port:%1]
    ##  NOTE: The 'REQUEST_SCHEME' Header is supported only from apache version 2.4 and above
    RewriteCond %{REQUEST_SCHEME} (.*)
    RewriteRule (.*) - [E=my_scheme:%1]


    RewriteCond %{HTTP_HOST} (.*)
    RewriteRule (.*) - [E=my_custom_host:%1]
    RewriteRule "^/v3/(.*)$" "/artifactory/api/puppet/puppet-virtual/v3/$1" [P]


    RewriteRule ^/$                /artifactory/webapp/ [R,L]
    RewriteRule ^/artifactory(/)?$      /artifactory/webapp/ [R,L]
    RewriteRule ^/artifactory/webapp$   /artifactory/webapp/ [R,L]


    RequestHeader set Host %{my_custom_host}e
    RequestHeader set X-Forwarded-Port %{my_server_port}e
    ## NOTE: {my_scheme} requires a module which is supported only from apache version 2.4 and above
    RequestHeader set X-Forwarded-Proto %{my_scheme}e
    RequestHeader set X-Artifactory-Override-Base-Url %{my_scheme}e://artifactory-cluster:%{my_server_port}e/artifactory
    ProxyPassReverseCookiePath /artifactory /artifactory


    ProxyPass /artifactory/ balancer://artifactory/artifactory/
    ProxyPassReverse /artifactory/ balancer://artifactory/artifactory/
    Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/artifactory/" env=BALANCER_ROUTE_CHANGED
</VirtualHost>

###########################################################
## this configuration was generated by JFrog Artifactory ##
###########################################################


## add HA entries when ha is configured
<Proxy balancer://artifactory>
    BalancerMember http://10.6.16.125:8080 route=14901314097097
ProxySet lbmethod=byrequests
ProxySet stickysession=ROUTEID
</Proxy>
<VirtualHost *:80>
    ProxyPreserveHost On
    ServerName artifactory-cluster
    ServerAlias *.artifactory-cluster
    ServerAdmin server@admin


    ## Application specific logs
    ## ErrorLog ${APACHE_LOG_DIR}/artifactory-cluster-error.log
    ## CustomLog ${APACHE_LOG_DIR}/artifactory-cluster-access.log combined


    AllowEncodedSlashes On
    RewriteEngine on


    RewriteCond %{SERVER_PORT} (.*)
    RewriteRule (.*) - [E=my_server_port:%1]
    ##  NOTE: The 'REQUEST_SCHEME' Header is supported only from apache version 2.4 and above
    RewriteCond %{REQUEST_SCHEME} (.*)
    RewriteRule (.*) - [E=my_scheme:%1]


    RewriteCond %{HTTP_HOST} (.*)
    RewriteRule (.*) - [E=my_custom_host:%1]


    RewriteCond "%{REQUEST_URI}" "^/(v1|v2|
    )/"
    RewriteCond "%{HTTP_HOST}" "^(.*)\.artifactory-cluster$"
    RewriteRule "^/v3/(.*)$" "/artifactory/api/puppet/%1/v3/$1" [PT]
    RewriteRule "^/(v1|v2)/(.*)$" "/artifactory/api/docker/%1/$1/$2" [PT]


    RewriteRule ^/$                /artifactory/webapp/ [R,L]
    RewriteRule ^/artifactory(/)?$      /artifactory/webapp/ [R,L]
    RewriteRule ^/artifactory/webapp$   /artifactory/webapp/ [R,L]


    RequestHeader set Host %{my_custom_host}e
    RequestHeader set X-Forwarded-Port %{my_server_port}e
    ## NOTE: {my_scheme} requires a module which is supported only from apache version 2.4 and above
    RequestHeader set X-Forwarded-Proto %{my_scheme}e
    RequestHeader set X-Artifactory-Override-Base-Url %{my_scheme}e://artifactory-cluster:%{my_server_port}e/artifactory
    ProxyPassReverseCookiePath /artifactory /artifactory


    ProxyRequests off
    ProxyPreserveHost on
    ProxyPass /artifactory/ balancer://artifactory/artifactory/
    ProxyPassReverse /artifactory/ balancer://artifactory/artifactory/
    Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/artifactory/" env=BALANCER_ROUTE_CHANGED
</VirtualHost>

Once you have your reverse proxy configured, you can install modules from Artifactory using the following commands:

Ports Method

puppet module install --module_repository http://localhost:8001 puppetlabs-apache

Subdomain Method

puppet module install --module_repository http://puppet-virtual.artifactory-cluster puppetlabs-apache

REST API

The following REST API endpoints are available to facilitate automation for configuration management with Puppet. Artifactory also uses these endpoints to support the librarian-puppet and r10k clients:

Get Puppet Modules: Returns a list of all Puppet modules hosted by the specified repository.Get Puppet Modules
Get Puppet Module: Returns information about a specific Puppet module.Get Puppet Module
Get Puppet Releases: Returns a list of all Puppet releases hosted by the specified repository.Get Puppet Releases
Get Puppet Release: Returns information about the specific Puppet module's release.Get Puppet Release