Introduction
If you encounter a 503 Service Unavailable error between high availability (HA) nodes, follow the steps below to identify and resolve the issue. An example log entry might look like this
2025-02-10T15:47:08.215Z [jfrt ] [WARN ] [8be12496dc855a7f] [p.HaPropagationServiceImpl:394] [5a7f|art-exec-131118] - Failed to propagate event 'syncStorageUsage' to 'Node2': 503: Service Unavailable
Resolution
Troubleshooting Steps:
1. Check Resource Contention:
Use the top command to check for any resource contention issues on the nodes. This will help you determine if there are any performance bottlenecks.
2. Verify Firewall Settings:
Ensure that there are no firewall rules blocking traffic between nodes. You can perform a health check by running the REST API from another node:
curl http://Other_Node_IP_OR_Hostname:8082/router/api/v1/system/health
3. Check Time Synchronization:
Verify the time difference between the nodes. Ensure that the NTP (Network Time Protocol) service is running and that the nodes are synchronized with a reliable time source.
4. Reproduce the Issue:
If you can reproduce the error by triggering a configuration change
5. Examine Internal IP Addresses:
Check for any internal IP addresses that may appear as 127.0.0.1 for each node by navigating to the Admin > Monitoring page. If you find that the IP addresses are incorrect, explicitly define the correct IP address for each node in the system.yaml configuration file as follows:
shared:
node:
ip: <Your_Node_IP>
If you still have the issue, please contact support@jfrog.com