XRAY: Issue With Indexing Old Builds

Symptoms:

All the new build scans and artifact scans are working as expected. However, the old build scans are not working.

Amongst the 2 queues, especially indexExistingContent takes care of the indexing of existing artifacts triggered by reindexing a repository.

What might be the reason behind such cases?

• Communication between Artifactory and Xray could have been lost for some reason, which has caused the requests to get stuck in the index queue.   
• If the resources are not enough for the Xray server to process the huge number of concurrent scan requests.

Troubleshooting steps:

Step-01:

Since RabbitMQ takes care of the indexing tasks, it is worth verifying the status of the RabbitMQ database server to understand the root cause of the problem.

Check if the RabbitMQ message queue, especially indexExistingContent holds more messages despite having no active transactions between Artifactory and Xray. If the message count is more, probably the RabbitMQ server is stuck processing the requests. Perform a stop action against the Xray server and check if the RabbitMQ is still running. netstat -tulpn would reveal if the RabbitMQ is still listening on port 5672.

If the service is still listening on this port, we need to clear the process and perform a restart of the Xray service. This might help us to clear off the stuck messages. 

Step-02:

If the above doesn’t improve the behaviour, we need to check if there are stale entries still clogging the corresponding database table.

Login to the PostgreSQL database and verify the event_states table.
SELECT * from event_states;

To find the entries specific to a particular repository path, the following commands can be used.
SELECT * from event_states where art_path LIKE '<REPO_NAME>';

If there are no active scans invoked recently but the table displays entries representing concerned repositories and build references, it indicates an issue. We need to clear off the stuck entries and then invoke the new scans against the existing content. Prior to performing the deletion of such stale entries, ensure that the service is stopped (Xray) and no active scans are in progress.
DELETE from from event_states where art_path LIKE '%REPO_NAME%' and action = 'created' ;

IMPORTANT NOTE:
Even though we are executing this query to clear-off the stale processes, it is recommended to execute this query during a scheduled maintenance window [or] outside business hours, considering the query might take time to execute and the resource utilization.

Step-03:

If the issue persists despite clearing off the stale entries against the concerned builds, we need to check if the DB is having any dead tuples leading to the DB’s degraded performance. You may use the following command to identify the dead tuples (if any).

SELECT schemaname,
       relname,
       n_tup_ins,
       n_tup_upd,
       n_tup_del,
       n_live_tup,
       n_dead_tup,
       last_vacuum,
       last_autovacuum,
       last_analyze,
       last_autoanalyze
FROM   pg_stat_all_tables
WHERE  schemaname = 'public order by n_dead_tup desc;

If there are dead tuples, performing a vacuum task against the database would help. It is always recommended to perform such vacuuming task after consulting with the DBAs and during the non-business hours.