ARTIFACTORY: Understanding the Repeated Message "Currently Applying DB Migrations, Do Not Restart Until Finished

Introduction

You may wonder why you see the repeated message: "Currently applying db migrations, do not restart until finished," especially when you're not in the process of upgrading Artifactory.


Resolution

When you encounter the message "Currently applying db migrations, do not restart until finished," it raises questions about its meaning. This message is a default notification triggered during the startup of the metadata service, which checks if a database migration is required.

If the service is waiting to obtain locks due to an issue with the database, you may see this log entry. Specifically, this error occurs if obtaining the migration lock is delayed for more than 30 seconds. Therefore, it's important to interpret this message as an indication that there may be a problem with the database.

To investigate and verify further, you should examine the following two tables: md_lock and md_database_migrations. Pay particular attention to any stuck queries related to the md_lock table. 

Use the following SQL queries for your analysis:

1.  Check the lock table:

 SELECT * FROM md_lock; 

2. Review the database migrations:

 SELECT * FROM md_database_migrations;

3. If the above two tables do not immediately reveal the issue, check all currently active queries for signs of a problem by:

SELECT datname, substr(query, 1, 500) AS the_query, (now() - pg_stat_activity.query_start) AS takes_time, state, client_addr, pid 
FROM pg_stat_activity 
WHERE state != 'idle' 
ORDER BY takes_time DESC;

Conclusion and Next Steps

By running these diagnostic queries, you'll gain crucial insights into your database's current state, allowing you to quickly identify any potential locking or performance issues.

Action Based on Findings:

• If you identify prolonged long-running queries:
  
  • Shut down all Artifactory services immediately.
    
  • Re-check the active queries (Step 3 in the previous instructions).
    
• If queries persist after shutdown:
  
  • You must either manually kill the long-running queries (using pg_cancel_backend() or pg_terminate_backend() with the query's PID) or restart the database server itself.
    
• If the issue recurs:
  
  • If long-running queries return shortly after remediation (killing or restarting), gather the output of all diagnostic queries and reach out to JFrog Support for an in-depth analysis.