This can be done, however we have not implemented this in the installService script so you will have to make some manual changes.
The quickest way is to edit the ${artifactory_home}/bin/installService.sh file and add the group name just before the ‘${ARTIFACTORY_USER}’ in line 169: chown -R -L ${GROUP_NAME}: ${ARTIFACTORY_USER}: ${ARTIFACTORY_HOME} || errorArtHome "Could not set permissions"
You can of course add some validations to the script to verify that the group actually exists before trying to install the service.