ARTIFACTORY: How to fix Federated Monitoring Page showing N/A when "Monitoring is unavailable, migration in progress"

When you head into the Federated Monitoring page (i.e. Administration -> Monitoring -> Federation Status) you may see that one of the JPDs have N/A in all of its columns.

This can mean that there is an issue with reach that other jpd or that there is some migration taking place and as a result the information is not available. We will be covering on the latter situation when a JPD is reporting that "Monitoring is unavailable, migration in progress".

In this article we will first confirm that the issue encountered is from the unfinished migration. Then we will confirm that the migration is unfinished. Finally we will demonstrate a potential remedy.

Confirming N/A is a result of an unfinished migration

Run this endpoint from the Artifactory node that is trying to reach out to that particular JPD that is showing the N/A columns.

curl -H "Authorization: Bearer <TOKEN>" <JPD_URL>/artifactory/api/federation/status -vv

If you get “Monitoring is unavailable, migration in progress” continue on with the rest of this article.
If you see something like  “Internal Server Error” or some other message it potentially means that there is an issue with the upstream or reaching the upstream location. 

The “Monitoring is unavailable, migration in progress” message means that on the upstream JPD the “event-table-repo-key-migration” has not taken place or finished yet

Confirming that the migration has not finished or have not taken place

First check the node_events table (i.e. SELECT * FROM node_events LIMIT 1;). There should be a “repo” column. If there is not the migration has not taken place. The migration will add the “repo” column.

Second check the migration_status table. There should be a row with “event-table-repo-key-migration”. If there is not the migration has not taken place.
 

Enable more verbose logging for further troubleshooting

Go ahead and add this to the logback.xml  (i.e. <ART_HOME>/var/etc/artifactory/logback.xml) of the nodes in the cluster where the migration have not taken place so that more verbose logs be outputted. Place it right before the closing </configuration> tag. No restart is needed for the logger to take effect and you will see a log file called artifactory-event-migration.log be generated. Don’t forget to remove this snippet after you have finished troubleshooting so that the logs don’t keep getting outputted. 

<appender name="eventmigration-log" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <File>${log.dir}/artifactory-event-migration.log</File>
  <rollingPolicy class="org.jfrog.common.logging.logback.rolling.FixedWindowWithDateRollingPolicy">
    <FileNamePattern>${log.dir.archived}/artifactory-event-migration.%i.log.gz</FileNamePattern>
    <maxIndex>10</maxIndex>
  </rollingPolicy>
  <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
    <MaxFileSize>25MB</MaxFileSize>
  </triggeringPolicy>
  <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
    <layout class="org.jfrog.common.logging.logback.layout.BackTracePatternLayout">
      <pattern>%date{yyyy-MM-dd'T'HH:mm:ss.SSS, UTC}Z [jfrt ] [%-5p] [%-16X{uber-trace-id}] [%-30.30(%c{3}:%L)] [%-20.20thread] - %m%n</pattern>
    </layout>
  </encoder>
</appender>
 
<logger name="org.artifactory.event.migration" additivity="false">
  <level value="trace"/>
  <appender-ref ref="eventmigration-log"/>
</logger>

<logger name="org.artifactory.storage.db.event.dao.eventTableRepoKeyMigration" additivity="false">
  <level value="trace"/>
  <appender-ref ref="eventmigration-log"/>
</logger>

<logger name="org.artifactory.storage.db.event.dao.eventTableRepoKeyMigration.EventTableRepoKeyMigrationDao" additivity="false">
  <level value="trace"/>
  <appender-ref ref="eventmigration-log"/>
</logger>

<logger name="org.artifactory.event.EventsLogCleanUpService" additivity="false">
  <level value="trace"/>
  <appender-ref ref="eventmigration-log"/>
</logger>

<logger name="org.artifactory.storage.db.event.service" additivity="false">
  <level value="trace"/>
  <appender-ref ref="eventmigration-log"/>
</logger>

This will give some insight into if the migration is taking place or some other issue is preventing the migration.
Some logs to look out for:
"start execute events table repo key migration in the first time"
"start execute events table repo key again because its finished unsuccessfully"
"Inserting new migration status record '
"Migration with ID '{}' already exists."
"finish to add {} index on {}"
"successfully finished execute events table repo key migration"
"Migration for '{}' has finished."

The ‘{}’ are placeholders and will contain something else in the logs.

But if you don’t see any of these in the logs there check if you see items like the following.
“number of events that going to be deleted for repo key: ‘{}’ is ‘{}’”
This means that it is undergoing a cleanup job which must be completed before it starts the migration process. This cleanup can take some time and you must wait for this to completed.

Possible issue and remedy

If you do not see any of the above logs this may mean that the node you are looking at is not allowed to run the migration.
Check the system.yaml and see if you are still using the old setup of primary, secondary.
If you are still using this setup (instead of taskAffinity: any) make sure that you have at least one primary: true so that, that node will be able to start and perform the migration.