ARTIFACTORY: Quick Start Guide Docker-compose

Ino Choi
2022-01-13 21:29

What will you get?User-added image

This guide will outline how to install Artifactory 7 HA for a production ready environment using the Docker-Compose installation. In this guide, we will be walking through setting up a Postgres external database, multiple artifactory nodes, and a NGINX reverse proxy (which can act as a load balancer as well). At minimum, a server per Artifactory node is required. Note that you will need a license per Artifactory node. We also recommend having a type of cloud bucket storage (e.g. S3) ready. A good understanding of linux based operating systems, docker-compose and docker will be required to fully grasp the material throughout this guide.

Pre-requisites

Install Docker Engine

Install Docker Compose

OS that supports docker engine v18
https://docs.docker.com/engine/install/

The following ports will need to be available internally (within the container): 8081, 8082, 8040, 8045, 8048, 9092, 8070, 8086, 8046, 8047, 8049, 8091, 8061, 8062. 8081 and 8082 will additionally need to be reachable externally between nodes – and your LB/reverse proxy. These ports will be exposed below during container start up. They do not need to be exposed to the end user. The other ports just need to be available within the container and not consumed by another process – some of these ports are configurable via the system.yaml if it cannot be made available.

Environment details

Before installing Artifactory, refer to System Requirements for information on supported platforms, supported browsers and other requirements. We recommend using a machine that is easily scalable in case there is an increase of Artifactory usage. Generally a machine with 4-core CPU, 8GB RAM per node is enough starting out.

See for more node hardware details https://www.jfrog.com/confluence/display/JFROG/System+Requirements 

Database

We highly recommend using an external DB with Artifactory – the built-in derby database performance will degrade as you upload more artifacts, and does not support Artifactory HA. We recommend installing your database on a separate server, away from your Artifactory instance, but within minimal latency (same LAN). In this quick start guide, we will cover the Postgres setup.

PostgreSQL support:

  • 9.5 (EOL)
  • 9.6 (EOL soon)
  • 10.x
  • 11.x
  • 12.x
  • 13.x

External Postgres setup:

1. Connect to PostgreSQL and create Artifactory user and database

CREATE USER artifactory WITH PASSWORD 'password';
CREATE DATABASE artifactory WITH OWNER=artifactory ENCODING='UTF8';
GRANT ALL PRIVILEGES ON DATABASE artifactory TO artifactory;

2. Download the JDBC driver corresponding to your PostgreSQL version from the PostgreSQL JDBC Driver Download site and copy the downloaded jar file into – $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib directory.
Make sure your driver has read permissions for all users. Note that this directory will get created on installing the compose, or you may pre-create it.

3. Enable PostgreSQL connectivity from the Artifactory servers:

Add the following line to $POSTGRES_HOME/data/pg_hba.conf

host [artifactory_db_name] [artifactory_user] [cidr]    md5

Example:

host artifactory artifactory 123.456.78.90/32 md5

Add the following line to (if it already exists, update it)  $POSTGRES_HOME/data/postgresql.conf

listen_addresses='*'

We recommend having sizable storage space for the database, even if it is only storing metadata. It should be at least 2/10 of your expected file store size.

Artifactory supports a number of DB types, including mysql, oracle, mariaDB. You may find the links to other supported databases below in the glossary.

For more database recommendations, see https://jfrog.com/whitepaper/best-practices-for-managing-your-artifactory-database/

Filestore (S3)

By default, Artifactory is configured to use the local file system as its filestore. Artifactory offers flexible filestore management through the binarystore.xml configuration file located in the $JFROG_HOME/artifactory/var/etc/artifactory folder. Note that this file/directory will get created on installing the compose, or you may pre-create it.
By modifying this file you can implement a variety of different storage configurations. For ease of scaling, we recommend setting up a S3 bucket and utilizing that for unlimited scaling. Use the following cluster-s3-storage-v3 template to configure your S3 Cloud Storage.<config version="2">
<chain template="cluster-s3-storage-v3"/>
<provider id="s3-storage-v3" type="s3-storage-v3">
<endpoint>s3.amazonaws.com</endpoint>
<bucketName>bucketName</bucketName>
<path>pathPrefix</path>
<region>s3Region</region>
<identity>yourIdentity</identity>
<credential>yourCredentials</credential>
<usePresigning>true</usePresigning>
<signatureExpirySeconds>600</signatureExpirySeconds>
</provider>
</config>

We have other configuration templates for other cloud providers as well:

https://www.jfrog.com/confluence/display/JFROG/Configuring+the+Filestore#ConfiguringtheFilestore-ConfiguringShardingforHACluster

If you want to use local disk storage instead of S3, you can use:
 

<config version="2">
<chain template="cluster-file-system"/>
</config>

Download Location

Download the docker compose package from the Download JFrog Platform page.
Please make sure that your machine can connect to “releases-docker.jfrog.io” to pull the necessary images for Artifactory.

Steps to install

1. Extract the jfrog-artifactory-pro-<version>-compose.tar.gz file and go to the extracted folder.

$ tar -xvf jfrog-artifactory-pro-<version>-compose.tar.gz
$ cd artifactory-pro-<version>

2. Run the interactive script to set up Artifactory.

$ ./config.sh

3. It is recommended to define the Artifactory home directory into a directory that gives run and execute permissions to all users such as /opt.

Installation Directory (Default: /root/.jfrog/artifactory): /opt/jfrog/artifactory

4. Specify the IP address of the Artifactory machine.

For IPv6 address, enclose value within square brackets as follows : [<ipv6_address>]
Please specify the IP address of this machine (Default: <IP_ADDRESS>):

5. If you’re setting up an HA cluster and adding this machine to an existing Artifactory instance, select “y”, otherwise select “N”.

Are you adding an additional node to an existing product cluster? [y/N]: N

6. As we will use the external database we have configured, select “N” and select the database.

The installer can install a PostgreSQL database, or you can connect to an existing compatible PostgreSQL database
(https://service.jfrog.org/installer/System+Requirements#SystemRequirements-RequirementsMatrix)
If you are upgrading from an existing installation, select N if you have externalized PostgreSQL, select Y if not.
Do you want to install PostgreSQL? [Y/n]: N

Provide the type of your external database that you want to connect to.
Note : If you choose derby, it will be considered as an internal database and no further details will be asked
Enter database type, supported values [ postgresql mssql mariadb mysql oracle derby ]: postgresql

7. Provide the URL, username, and password.

Provide the database connection details
PostgreSQL url. Example: [jdbc:postgresql://<IP_ADDRESS>:<PORT>/artifactory]: postgresql://<IP_ADDRESS>:5432/artifactory

Database username (If your existing connection URL already includes the username, leave this empty): artifactory

Database password (If your existing connection URL already includes the password, leave this empty):

Database connection successful

8. Setup is complete and you can start Artifactory using the provided Docker-Compose command.

start: docker-compose -p rt up -d
stop: docker-compose -p rt down

9. Check that artifactory starts up by checking the logs for the following message:
 

$ docker logs -f artifactory
2021-09-20T20:44:34.610Z [jfrou] [INFO ] [30a6b3f102dc9b52] [local_topology.go:270 ] [main ] -
###############################################################
### All services started successfully in 86.544 seconds ###
###############################################################

10.  Once Artifactory comes up, the UI should be accessible at port 8082. Check that Artifactory is in HA mode by running the following REST API and look in the addon array for “ha”:

$ curl localhost:8082/artifactory/api/system/version -u admin:password
{
"version" : "7.25.7",
"revision" : "72507900",
"addons" : [ "ha",...

11. To install additional nodes, follow the installation steps above.
 a. For the step asking about adding the machine to an existing Artifactory instance, select “y”

Are you adding an additional node to an existing product cluster? [y/N]: y
12. For a new node to join a cluster, the nodes must connect to the same database and have the same Master Key.

a) Copy the master.key from the first node to the additional nodes located at $JFROG_HOME/artifactory/var/etc/security/master.key.If you would like to generate your own key ahead of time, you can follow the guide here to do so

b) Start the additional nodes.

c) Check the Artifactory log.

$ docker-compose -p rt logs

(*) For Docker installations, verify that the host's ID “shared.node.id” and IP “shared.node.ip” are added to the system.yaml. If these are not manually added, they are automatically resolved as the container's hostname and IP, meaning other nodes and services will not be able to reach this instance. 
 

shared:
extraJavaOpts: "-Xms512m -Xmx4g"
node:
id: "MyNodeID"
ip: "10.1.2.3"

Make sure to keep your YAML spacing consistent! We recommend either 2 or 4 spaces indentation, but ensure that it is consistent throughout.

SSL

In case SSL (HTTPS) connection to Artifactory is required, you can configure a reverse proxy such as Nginx or Apache in front of Artifactory. For Docker-Compose installation, you can find the Nginx Docker-Compose template in the “templates” folder under the extracted Docker-Compose folder. You can copy the template to the extracted folder as docker-compose.yaml. Please make sure to add port 443 under artifactory.ports.

version: '3'
services:
artifactory:
image: ${DOCKER_REGISTRY}/jfrog/artifactory-pro:${ARTIFACTORY_VERSION}
container_name: artifactory
environment:
- JF_ROUTER_ENTRYPOINTS_EXTERNALPORT=${JF_ROUTER_ENTRYPOINTS_EXTERNALPORT}
ports:
- ${JF_ROUTER_ENTRYPOINTS_EXTERNALPORT}:${JF_ROUTER_ENTRYPOINTS_EXTERNALPORT} #
- 8081:8081 # for artifactory communication
- 443:443 # for nginx
volumes:
- ${ROOT_DATA_DIR}/var:/var/opt/jfrog/artifactory
- /etc/localtime:/etc/localtime:ro
restart: always
logging:
driver: json-file
options:
max-size: "50m"
max-file: "10"
ulimits:
nproc: 65535
nofile:
soft: 32000
hard: 40000
nginx:
image: ${DOCKER_REGISTRY}/jfrog/nginx-artifactory-pro:${ARTIFACTORY_VERSION}
container_name: nginx
depends_on:
- artifactory
network_mode: service:artifactory
volumes:
- ${ROOT_DATA_DIR}/var/data/nginx:/var/opt/jfrog/nginx
- /etc/localtime:/etc/localtime:ro
environment:
- ART_BASE_URL=http://localhost:${JF_ROUTER_ENTRYPOINTS_EXTERNALPORT}
- NGINX_LOG_ROTATE_COUNT=${NGINX_LOG_ROTATE_COUNT}
- NGINX_LOG_ROTATE_SIZE=${NGINX_LOG_ROTATE_SIZE}
- SSL=true
restart: always
logging:
driver: json-file
options:
max-size: "50m"
max-file: "10"
ulimits:
nproc: 65535
nofile:
soft: 32000
hard: 40000

Sample NGINX template below:

###########################################################
## this configuration was generated by JFrog Artifactory ##
###########################################################

## add ssl entries when https has been set in config
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_certificate /etc/ssl/private/server.key;
ssl_certificate_key /etc/ssl/private/server.crt;
ssl_session_cache shared:SSL:1m;
ssl_prefer_server_ciphers on;
## server configuration
server {
listen 443 ssl;
listen 80 ;
server_name ~(?<repo>.+)\.artifactory_host artifactory_host;

if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
## Application specific logs
## access_log /var/log/nginx/artifactory_host-access.log timing;
## error_log /var/log/nginx/artifactory_host-error.log;
rewrite ^/$ /ui/ redirect;
rewrite ^/ui$ /ui/ redirect;
rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2;
chunked_transfer_encoding on;
client_max_body_size 0;
location / {
proxy_read_timeout 2400s;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_buffer_size 128k;
proxy_buffers 40 128k;
proxy_busy_buffers_size 128k;
proxy_pass http://localhost:8082;
proxy_set_header X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

location ~ ^/artifactory/ {
proxy_pass http://localhost:8081;
}
}
}

Reverse proxies can also be configured to handle load balancing traffic between nodes. In NGINX’s case, you can add this snippet at the top: 
 

upstream artifactory {
server 10.150.0.222:8082;
server <additional node IP>:8082;
}
upstream artifactory-direct {
server 10.150.0.222:8081;
server <additional node IP>:8081;
}

Then, modify the proxy_pass lines to be
 

proxy_pass http://localhost:8082; → proxy_pass http://artifactory;
proxy_pass http://localhost:8081; → proxy_pass http://artifactory-direct;

Otherwise you may use a dedicated load balancer to handle traffic balancing. 

Note that to support docker requests, you’ll need a reverse proxy or load balancer to handle request rewrites. Also, if you are planning on having a load balancer terminating SSL, and a reverse proxy, you’ll need the below headers to be hard coded to the details of your load balancer:

For NGINX:

proxy_set_header X-JFrog-Override-Base-Url https://<LBHOST>:<LBPORT>;
proxy_set_header X-Forwarded-Port <LBPORT>
proxy_set_header X-Forwarded-Proto https

Steps to upgrade

1. Stop the current server.

docker-compose -p rt stop

2. Extract the contents of the compressed archive and go to the extracted folder.

tar -xvf jfrog-artifactory-pro-<version>-compose.tar.gz
cd artifactory-pro-<version>

3. Copy the contents of the .env file in the previous installation (e.g. artifactory-pro-7.23.3 folder) to the newly created .env file in this archive (e.g. artifactory-pro-7.24.3 folder) without copying the versions, as this will affect the upgrade.

a) Previous installation .env file

## Docker registry to fetch images from
DOCKER_REGISTRY=releases-docker.jfrog.io

## Version of artifactory to install
ARTIFACTORY_VERSION=7.23.3

## The Installation directory for Artifactory. IF not entered, the script will prompt you for this input. Default [$HOME/.jfrog/artifactory]
ROOT_DATA_DIR=/opt/jfrog/artifactory

## Router external port mapping. This property may be overridden from the system.yaml (router.entrypoints.externalPort)
JF_ROUTER_ENTRYPOINTS_EXTERNALPORT=8082

## Nginx logrotation configuration
NGINX_LOG_ROTATE_COUNT=
NGINX_LOG_ROTATE_SIZE=

b) New installation .env file

## Docker registry to fetch images from
DOCKER_REGISTRY=releases-docker.jfrog.io

## Version of artifactory to install
ARTIFACTORY_VERSION=7.24.3

## The Installation directory for Artifactory. IF not entered, the script will prompt you for this input. Default [$HOME/.jfrog/artifactory]
ROOT_DATA_DIR=/opt/jfrog/artifactory

## Router external port mapping. This property may be overridden from the system.yaml (router.entrypoints.externalPort)
JF_ROUTER_ENTRYPOINTS_EXTERNALPORT=8082

## Nginx logrotation configuration
NGINX_LOG_ROTATE_COUNT=
NGINX_LOG_ROTATE_SIZE=

4. Run the config.sh script to set up folders with required ownership.

$ ./config.sh

5. Start up Artifactory using Docker-Compose command.

start: docker-compose -p rt up -d
stop: docker-compose -p rt down

Tuning Artifactory (Optional)

We have the following optional tuning section to optimize Artifactory for heavier loads – it is a good idea to keep these parameters in mind as your Artifactory instance takes on more load. 
https://jfrog.com/knowledge-base/how-do-i-tune-artifactory-for-heavy-loads/

1. javaOpts (heap size) in system.yaml – we recommend at least setting this one:
shared:
  extraJavaOpts: "-Xms512m -Xmx4g"shared:
extraJavaOpts: "-Xms512m -Xmx4g"

2. Customize database connections in system.yaml:

artifactory:
database:
maxOpenConnections: 200

access:
database:
maxOpenConnections: 200

metadata:
database:
maxOpenConnections: 200

As a rule of thumb, we require (upto) a number of DB connections based on the following formula:

Total number of connections = (number of nodes) * ((artifactory.database.maxOpenConnections * 2) + access.database.maxOpenConnections + metadata.database.maxOpenConnections) + 50

3. Tune Tomcat threads in system.yaml:

 

artifactory:
tomcat:
connector:
maxThreads: 400

access:
tomcat:
connector:
maxThreads: 100

When modifying the Access maxThreads, it is required to update the $JFROG_HOME/artifactory/var/etc/artifactory/artifactory.system.properties file with:

artifactory.access.client.max.connections = <VALUE>

4. Tune Async thread pool in the same file. Note the corePoolSize should not be more than 8x the number of CPU cores:

artifactory.async.corePoolSize = 32
artifactory.async.poolMaxQueueSize = 100000

Glossary

– System Requirements: https://www.jfrog.com/confluence/display/JFROG/System+Requirements
– JFrog Download: https://jfrog.com/download-jfrog-platform/
– External DB and Supported DBs: https://www.jfrog.com/confluence/display/JFROG/Configuring+the+Database
– Postgresql DB: https://www.jfrog.com/confluence/display/JFROG/PostgreSQL
– Filestore: https://www.jfrog.com/confluence/display/JFROG/Configuring+the+Filestore
– Amazon S3 Official SDK Template: https://www.jfrog.com/confluence/display/JFROG/Configuring+the+Filestore#ConfiguringtheFilestore-AmazonS3OfficialSDKTemplate
– SSL(HTTPS): https://www.jfrog.com/confluence/display/JFROG/HTTP+Settings
– Docker-Compose Installation: https://www.jfrog.com/confluence/display/JFROG/Installing+Artifactory#InstallingArtifactory-DockerComposeInstallation
– Docker-Compose Upgrade:
https://www.jfrog.com/confluence/display/JFROG/Upgrading+Artifactory#UpgradingArtifactory-DockerComposeUpgrade .