How to resolve failing propagation events?

Relevant versions: This information pertains to Artifactory HA versions 5.x through 6.3.4 and 6.4 through 6.7

There are high availability cluster situations, such as when a user modifies permission on one node, during which propagation from that node to other member nodes in a cluster will fail due to an invalid admin access token. Should this happen, your artifactory.log file for the propagating node will display the following error:

• in Artifactory HA versions 5.x through 6.3.4

2018-05-08 09:36:48,194 [art-exec-2] [WARN ] (o.a.a.h.p.HaPropagationServiceImpl:411) – Failed to propagate event 'supportBundleList' to 'A (https://172.16.78.163:8081/artifactory/api/support/internal/bundles)': 403:Forbidden

• in Artifactory HA versions 6.4 through 6.7

2018-12-04 16:12:33,102 [art-exec-12] [WARN ] (o.a.a.h.p.HaPropagationServiceImpl:511) – Invalid serviceId jfrt@01cx654e53fp5e11m2p16n1vfh

2018-12-04 16:12:33,102 [Thread-5] [ERROR] (o.a.a.h.p.HaPropagationServiceImpl:207) – Failed to propogate – sleeping…

2018-12-04 16:12:34,826 [http-nio-8082-exec-1] [WARN ] (o.a.a.h.p.HaPropagationServiceImpl:511) – Invalid serviceId jfrt@01cx654e53fp5e11m2p16n1vfh

2018-12-04 16:12:34,826 [http-nio-8082-exec-1] [ERROR] (o.a.a.h.r.HaRestAuthenticationFilter:73) – Error authenticating HA rest request from _system_@primary

To refresh or regenerate the admin access token, do the following:

• in Artifactory HA versions 5.x through 6.3.4

1. In your $ARTIFACTORY_HOME/etc folder, create a backup copy of the artifactory.config.latest.xml file and name it artifactory.config.latest.xml.copy.
2. Edit the original artifactory.config.latest.xml and remove the <adminToken> section (remove the tags including the content).
3. Remove the communication.token in your $ARTIFACTORY_HOME/etc/security folder.
4. Save these changes and change the name of the file to artifactory.config.import.xml, to make sure Artifactory reloads it while starting up.
5. Perform a rolling restart on all of your HA nodes.

• in Artifactory HA versions 6.4 through 6.7:

1. In $ARTIFACTORY_HOME/etc/security/access/access.admin.token, the adminToken is part of the file and it's present on all nodes in your cluster.
2. Remove the file from all nodes.
3. Perform a rolling restart on all of your HA nodes.
4. After restart the $ARTIFACTORY_HOME/etc/security/access/keys/service_id file would re-generate the token automatically.

If the Issue Persists, Check Your Access Properties

The access service has a settings file called access.properties, which is created in the $ArtifactoryHome/access/data directory during the startup of the Artifactory service. In some cases, when setting up an Artifactory HA cluster, if you copy the access directory from your primary or from any other node of the cluster, the access.properties is already present as a part of the directory copy and causes propagation issues since some of the fields in the access.properties file are meant to be unique between the nodes.

In this file the access.server.name property is the one that should be different on each node of an HA artifactory cluster, as this is how Artifactory will identify each separate service. To overcome this behavior, remove or move the access.properties file from the $ArtifactoryHome/access/data directory and restart the service and we should see a new access.properties file being created. This will guarantee that each node has a unique access server name.