How to resolve failing propagation events?

Nimer Bsoul
2019-05-21 08:43

Subject

There are situations in a High Availability cluster setup where could be propagation failures from one node to other member nodes in the cluster and it would be caused by a invalid access admin token.

Affected Versions

Artifactory HA versions 5.0 – 6.3.4 and 6.4 – 6.7

Details

If the version of the Artifactory HA instance is 5.0 – 6.3.4, please check the following details:

When a user modified a permission and that change needed to be propagated from the node that the change was made from to the other member nodes in the cluster.  
In the situation where the admin access token was no longer valid, the artifactory.log file, from the node that is doing the propagation, will display the following error:

2018-05-08 09:36:48,194 [art-exec-2] [WARN ] (o.a.a.h.p.HaPropagationServiceImpl:411) – Failed to propagate event 'supportBundleList' to 'A (http://172.16.78.163:8081/artifactory/api/support/internal/bundles)':
403:Forbidden

If the version of the Artifactory HA instance is 6.4 – 6.7, please check the following details:

When a user modified a permission and that change needed to be propagated from the node that the change was made from to the other member nodes in the cluster. In the situation where the admin access token was no longer valid, the artifactory.log file, from the node that is doing the propagation, will display the following error:

2018-12-04 16:12:33,102 [art-exec-12] [WARN ] (o.a.a.h.p.HaPropagationServiceImpl:511) – Invalid serviceId jfrt@01cx654e53fp5e11m2p16n1vfh

2018-12-04 16:12:33,102 [Thread-5] [ERROR] (o.a.a.h.p.HaPropagationServiceImpl:207) – Failed to propogate – sleeping…

2018-12-04 16:12:34,826 [http-nio-8082-exec-1] [WARN ] (o.a.a.h.p.HaPropagationServiceImpl:511) – Invalid serviceId jfrt@01cx654e53fp5e11m2p16n1vfh

2018-12-04 16:12:34,826 [http-nio-8082-exec-1] [ERROR] (o.a.a.h.r.HaRestAuthenticationFilter:73) – Error authenticating HA rest request from _system_@primary

Resolution

If the version of the Artifactory HA instance is 5.0 – 6.3.4, please check the following details:

In order to refresh or regenerate the admin access token, these are the steps that need to be taken:

1. Under the $ARTIFACTORY_HOME/etc folder, create a backup copy of the file artifactory.config.latest.xml, and call it artifactory.config.latest.xml.copy.

2. Edit the original artifactory.config.latest.xml, and remove the <adminToken> section (remove the tags including the content).

3. Remove the “communication.token” that is found under “$ARTIFACTORY_HOME/etc/security/”

3. Save the changes and change the name of the file to artifactory.config.import.xml, to make sure Artifactory reloads it while starting up.

4. Performing a rolling restart on your HA nodes.
 

If the version of the Artifactory HA instance is 6.4 – 6.7, please check the following details:

In order to refresh or regenerate the admin access token, these are the steps that need to be taken:

1. Under “$ARTIFACTORY_HOME/etc/security/access/access.admin.token” the adminToken is part of the file and it's present on all nodes in the cluster.

2. Remove the file from all nodes.

3. Perform a rolling restart and see if this resolves the problem.

4. After restart the $ARTIFACTORY_HOME/etc/security/access/keys/service_id file would re-generate the token automatically.

If Issues Persists Check Access Properties

The access service has a unique settings file called access.properties which is created during the start up of the Artifactory service under the $ArtifactoryHome/access/data directory.

In some cases of setting up a HA artifactory cluster if we had copied over the access directory from primary or from any other node of the cluster, the access.properties is already present as a part of the directory copy and causes propagation issues since some of the fields in the access.properties file are meant to be unique between the nodes.
In this file the access.server.name property is the one that should be different on each node of an HA artifactory cluster, as this is how Artifactory will identify each separate service.  

To over come this behavior, remove or move the access.properties file from the $ArtifactoryHome/access/data directory and restart the service and we should see a new access.properties file being created. This will guarantee that each node has a unique access server name.