In certain scenarios, you may encounter an issue where Artifactory fails to start, and the cause is a missing or unhealthy Mission Control (JFMC) service.
This typically occurs when the Mission Control service is not functioning properly, resulting in an unhealthy state.
To verify this, you can run the topology health command and check the status of the JFMC service.
Here’s an example of the output you might see when running the command below command from server where Artifactory is hosted.
curl -s http://localhost:8046/router/api/v1/topology/health
Example Output:
root@cec9e31b8264:~# curl -s http://localhost:8046/router/api/v1/topology/health
{
"nodes": {
"http://***.***.***.*:8082": {
"service_registry_state": "UNHEALTHY_PEER",
"last_heartbeat": 1730869879650,
"effective_state": "UNHEALTHY_PEER",
"health_response": {
"router": {
"message": "OK",
"node_id": "cec9e31b8264",
"required_service_types": [
"jfrt",
"jfac",
"jfmd",
"jffe",
"jfob",
"jfcon",
"jfmc",
"jfevt"
],
"state": "UNHEALTHY"
},
"services": [
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfac@01jbzxjhzfr2cf029x7kn515g8",
"state": "UNHEALTHY_PEER"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfcon@01jbzxjhzfr2cf029x7kn515g8",
"state": "UNHEALTHY_PEER"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfevt@01jbzxjhzfr2cf029x7kn515g8",
"state": "UNHEALTHY_PEER"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jffe@01jbzxjhzfr2cf029x7kn515g8",
"state": "UNHEALTHY_PEER"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfmc@01jbzxj8cnbw761p0ek2pj1468",
"state": "UHEALTHY"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfmd@01jbzxjex9gzhjbdttj9q8tf8r",
"state": "UNHEALTHY_PEER"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfob@01jbzxk2abzz8z1h0fgz3z0z1r",
"state": "UNHEALTHY_PEER"
},
{
"message": "OK",
"node_id": "cec9e31b8264",
"service_id": "jfrt@01jbzxk2abzz8z1h0fgz3z0z1r",
"state": "UNHEALTHY_PEER"
}
]
}
}
}
}
As you can see in the example output, the ‘jfmc’ service is marked as UNHEALTHY, which could be the reason Artifactory fails to start.
While inspecting the mc-service.log, you might not observe any explicit errors pointing to the root cause of the issue.
However, the tomcat-catalina.log located in /opt/jfrog/artifactory/var/log/tomcat will provide more insight as it captures most of the logs related to start up.
Example Error in tomcat-catalina.log:
08-Jul-2023 09:12:30.016 SEVERE [Catalina-utility-3] org.apache.catalina.startup.HostConfig.deployDescriptor Error deploying deployment descriptor [/opt/jfrog/artifactory/app/artifactory/tomcat/conf/Catalina/localhost/mc.xml]
java.lang.IllegalStateException: Error starting child
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:729)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:689)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1888)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mc]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
... 11 more
Caused by: java.lang.IllegalStateException: Could not create JFMC HOME directory structure
at org.jfrog.mc.environment.JfmcFileSystem.createDirectoriesIfNeeded(JfmcFileSystem.java:52)
at org.jfrog.mc.init.Initializer.run(Initializer.java:52)
at org.jfrog.mc.MissionControlApplication.runInitialization(MissionControlApplication.java:144)
at org.jfrog.mc.MissionControlApplication.beforeStartup(MissionControlApplication.java:109)
at org.jfrog.mc.MissionControlApplication.onStartup(MissionControlApplication.java:210)
at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:174)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5211)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 12 more
Caused by: java.io.IOException: Failed to list contents of /opt/jfrog/artifactory/var/work/mc/temp
at org.apache.commons.io.FileUtils.verifiedListFiles(FileUtils.java:2935)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:333)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1206)
at org.jfrog.mc.environment.JfmcFileSystem.createDirectoriesIfNeeded(JfmcFileSystem.java:48)
... 19 more
The error indicates an issue with creating the JFMC home directory structure due to insufficient permissions on the directory /opt/jfrog/artifactory/var/work/mc/temp.
This issue is commonly caused by incorrect folder permissions, particularly when the system undergoes an OS upgrade or Artifactory upgrade.
During these upgrades, the permissions of certain folders within the Tomcat webapps directory may be reset, potentially causing them to be owned by the root user instead of the appropriate artifactory user.
Resolution:
To resolve the issue, you need to correct the folder permissions for the directory /opt/jfrog/artifactory/var/work/mc/temp. The artifactory user should own this directory and all its contents.
1. Verify the current ownership and permissions of the directory:
ls -lrt /opt/jfrog/artifactory/var/work/mc/temp
If the directory is not owned by the artifactory user, you will need to update the permissions.
2. Change the ownership of the directory to the artifactory user:
chown artifactory:artifactory /opt/jfrog/artifactory/var/work/mc/temp -R
3. After updating the permissions, restart Artifactory. The JFMC service should now start correctly, and Artifactory should come up without any issues.
Conclusion:
By ensuring that the correct user permissions are applied to the Mission Control directory, you can resolve the startup issue caused by an unhealthy JFMC service.
Always verify folder ownership after system or application upgrades to prevent similar issues in the future.
If the issue persists even after fixing the folder permissions, further investigation into the system logs may be required to identify any other underlying causes.