ARTIFACTORY: Configure Prometheus with Grafana using helm charts for Artifactory

This KB article will help to configure Prometheus with Grafana using helm charts for Artifactory.
JFrog supports integration with Prometheus for log monitoring. The JFrog Log Analytics and Metrics solution using Prometheus consists of three segments and we will be installing the following components in the cluster.

Prometheus - the component where metrics data gets ingested
Loki - the component where log data gets ingested
Grafana - the component where data visualization is achieved via prebuilt dashboards

In this KB article, we are configuring prometheus with grafana using helm charts for Artifactory in Google GKE

Pre-Requisites:

Recommended Kubernetes Version 1.25.2 and above
'kubectl' utility on the workstation which is capable of connecting to the Kubernetes cluster
HELM v3 Installed
Versions supported and Tested:

Jfrog Platform: 10.9.2
Artifactory : 7.46.10
Xray : 3.59.4
Prometheus:2.39.1
Grafana:9.2.3
Loki: 2.6.1

Installing Prometheus, Loki and Grafana on Kubernetes

The Prometheus Community kube-prometheus-stack helm chart allows the creation of Prometheus instances and includes Grafana. The Grafana Community grafana helm chart allows the creation of Loki instances and includes Grafana which can link to prometheus.

Once the Prerequisites are met, follow the below steps :

Create the Namespace required for Prometheus Stack deployment,We will use jfrog-plg as the namespace throughout this KB article:

kubectl create namespace jfrog-plg

Add the required Helm Repositories:

For Prometheus:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

For Grafana:

helm repo add grafana https://grafana.github.io/helm-charts

For loki: (While adding the this helm repository, we might get the below warning message and it can be ignored)
Error: looks like "https://grafana.github.io/loki/charts" is not a valid chart repository or cannot be reached: failed to fetch https://grafana.github.io/loki/charts/index.yaml : 404 Not Found

helm repo add loki https://grafana.github.io/loki/charts

Run the below helm command to get the latest information about charts from the respective chart repositories. Information is cached locally, where it is used by commands like 'helm search'.

helm repo update

Installing the required components from the above Helm Repositories:

Install Prometheus: (Deployment of prometheus might take few minutes to complete)
Note: "prometheus" here is the value that needs to be used against the value for "release_name" in the configuration section

helm install "prometheus" prometheus-community/kube-prometheus-stack -n jfrog-plg

Install loki:
Note:
1. "loki" will be the service name, the url to access loki as a datasource can be visualized as http://<service_name>.<namespace>:<port>
ex: http://loki.jfrog-plg:3100 will be the "loki_url" value

2. the --version v2.6.1 is the most recent loki version at the time of writing the document
if there is a need to deploy this exact version, then add "--version v2.6.1" to the command line.

helm upgrade --install "loki" grafana/loki-stack -n jfrog-plg

Deploying loki will fetch all the required sub-components for grafana from the grafana repository and that is the reason we will not deploy grafana explicitly.

Note: If prometheus and loki are already available you can skip the installation section and proceed with below steps.

To configure prometheus with grafana and metrics via Helm charts for Artifactory / Artifactory HA, please use the following sample YAML file:

global:     
  jfrog:    
    observability:      
      branch: master    
  prometheus:     
    loki_url: http:\/\/loki.jfrog-plg:3100      
    release_name: prometheus  
installerInfo: '{ "productId": "Helm_prometheus_artifactory/{{ .Chart.Version }}", "features": [ { "featureId": "ArtifactoryVersion/{{ default .Chart.AppVersion .Values.artifactory.image.version }}" }, { "featureId": "{{ if .Values.postgresql.enabled }}postgresql{{ else }}{{ .Values.database.type }}{{ end }}/0.0.0" }, { "featureId": "Platform/{{ default "kubernetes" .Values.installer.platform }}" },  { "featureId": "Channel/Helm_prometheus_artifactory" } ] }'
artifactory:
  openMetrics:    
    enabled: true 
  customSidecarContainers: |  
      - name: "artifactory-fluentd-sidecar"     
        image: "partnership-public-images.jfrog.io/fluentd/fluentd:1.18.0"    
        imagePullPolicy: "IfNotPresent"   
        command:  
          - 'sh'  
          - '-c'  
          - >     
            gem install fluent-plugin-prometheus --no-doc;  
            curl https://raw.githubusercontent.com/jfrog/log-analytics-prometheus/{{ .Values.global.jfrog.observability.branch }}/fluent_metrics.conf.rt -o fluentd_metrics.conf;   
            cp fluentd_metrics.conf fluent.tmp.conf;  
            sed -i -e 's/LOKI_URL/{{ .Values.global.prometheus.loki_url }}/g' fluent.tmp.conf;  
            mv fluent.tmp.conf fluentd.conf;    
            fluentd -v -c fluentd.conf    
        volumeMounts:   
          - mountPath: "{{ .Values.artifactory.persistence.mountPath }}"      
            name: artifactory-volume      
        env:      
          - name: JF_PRODUCT_DATA_INTERNAL      
            value: {{ .Values.artifactory.persistence.mountPath }}      
  replicaCount: 2 
  masterKeySecretName: my-masterkey-secret
  joinKeySecretName: my-joinkey-secret
  license:
    secret: artifactory-cluster-license
    dataKey: art.lic
postgresql:
  postgresqlPassword: ********
databaseUpgradeReady: true


## Allows to add additional kubernetes resources
## Use --- as a separator between multiple resources
## For an example, refer - https://github.com/jfrog/log-analytics-prometheus
additionalResources: |
  apiVersion: v1
  kind: Service
  metadata:
    name: {{ template "artifactory.fullname" . }}-metrics
    annotations:
      prometheus.io/path: "/metrics"
      prometheus.io/scheme: "http"
      prometheus.io/port: "24231"
      prometheus.io/scrape: "true"
    labels:
      app: {{ template "artifactory.name" . }}
      component: {{ template "artifactory.name" . }}
      chart: {{ template "artifactory.chart" . }}
      heritage: {{ .Release.Service }}
      release: {{ .Values.global.prometheus.release_name }}
      {{- with .Values.artifactory.labels }}
  {{ toYaml . | indent 4 }}
      {{- end }}
  spec:
    type: ClusterIP
    ports:
    - name: metrics
      port: 24231
      protocol: TCP
    selector:
      app: {{ template "artifactory.name" . }}
      component: {{ template "artifactory.name" . }}
  ---
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: servicemonitor-{{ template "artifactory.name" . }}
    labels:
      metrics: jfrog
      app: {{ template "artifactory.name" . }}
      chart: {{ template "artifactory.chart" . }}
      heritage: {{ .Release.Service }}
      release: {{ .Values.global.prometheus.release_name }}
  spec:
    selector:
      matchLabels:
        app: {{ template "artifactory.name" . }}
        component: {{ template "artifactory.name" . }}
    endpoints:
        - port: metrics
          interval: 15s
mc:
  enabled: true
nginx:
  enabled: true
  artifactoryConf: |
    {{- if .Values.nginx.https.enabled }}
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_certificate  {{ .Values.nginx.persistence.mountPath }}/ssl/tls.crt;
    ssl_certificate_key  {{ .Values.nginx.persistence.mountPath }}/ssl/tls.key;
    ssl_session_cache shared:SSL:1m;
    ssl_prefer_server_ciphers   on;
    {{- end }}
    ## server configuration
    server {
      {{- if .Values.nginx.internalPortHttps }}
      listen {{ .Values.nginx.internalPortHttps }} ssl;
      {{- else -}}
      {{- if .Values.nginx.https.enabled }}
      listen {{ .Values.nginx.https.internalPort }} ssl;
      {{- end }}
      {{- end }}
      {{- if .Values.nginx.internalPortHttp }}
      listen {{ .Values.nginx.internalPortHttp }};
      {{- else -}}
      {{- if .Values.nginx.http.enabled }}
      listen {{ .Values.nginx.http.internalPort }};
      {{- end }}
      {{- end }}
      server_name ~(?<repo>.+)\.vigneshs.support-testlab.com vigneshs.support-testlab.com
      {{- range .Values.ingress.hosts -}}
        {{- if contains "." . -}}
          {{ "" | indent 0 }} ~(?<repo>.+)\.{{ . }}
        {{- end -}}
      {{- end -}};
      if ($http_x_forwarded_proto = '') {
        set $http_x_forwarded_proto  $scheme;
      }
      ## Application specific logs
      ## access_log /var/log/nginx/artifactory-access.log timing;
      ## error_log /var/log/nginx/artifactory-error.log;
      rewrite ^/artifactory/?$ / redirect;
      if ( $repo != "" ) {
        rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2 break;
      }
      chunked_transfer_encoding on;
      client_max_body_size 0;
      location / {
        proxy_ssl_server_name on;
        proxy_read_timeout  2400;
        proxy_send_timeout 2400;
        proxy_pass_header   Server;
        proxy_request_buffering off;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_cookie_path   ~*^/.* /;
        proxy_pass          {{ include "artifactory.scheme" . }}://{{ include "artifactory.fullname" . }}:{{ .Values.artifactory.externalPort }}/;
        {{- if .Values.nginx.service.ssloffload}}
        proxy_set_header    X-JFrog-Override-Base-Url https://vigneshs.support-testlab.com;
        {{- else }}
        proxy_set_header    X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host:$server_port;
        proxy_set_header    X-Forwarded-Port  $server_port;
        {{- end }}
        proxy_set_header    X-Forwarded-Port  443;
        proxy_set_header    X-Forwarded-Proto https;
        proxy_set_header    Host              $http_host;
        proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
        add_header Strict-Transport-Security always;
        location /artifactory/ {
          if ( $request_uri ~ ^/artifactory/(.*)$ ) {
            proxy_pass       http://{{ include "artifactory.fullname" . }}:{{ .Values.artifactory.externalArtifactoryPort }}/artifactory/$1;
          }
          proxy_pass         http://{{ include "artifactory.fullname" . }}:{{ .Values.artifactory.externalArtifactoryPort }}/artifactory/;
        }
      }
    }
unifiedUpgradeAllowed: true

Accessing Prometheus and Grafana:

Use 'kubectl port-forwards' as mentioned in two terminal windows

 kubectl port-forward service/prometheus-operated 9090:9090 -n jfrog-plg
 kubectl port-forward service/prometheus-grafana 3000:80 -n jfrog-plg

Below screenshot for reference:
User-added image

Access Prometheus via UI: Go to the web UI of the Prometheus instance "http://localhost:9090" and verify "Status -> Service Discovery", the list shows the new ServiceMonitor for Artifactory or Xray or Both.

Access Grafana via UI: Go to Grafana "http://localhost:3000" and Default credentials for Prometheus grafana is -> admin/prom-operator (username/password)

User-added image

-> Add your Prometheus instance and Loki Instance as a datasource. While specifying datasource urls for Loki and Prometheus,please test and confirm that the connection is successful.
For Loki, specify url value as http://loki:3100
For Prometheus, specify url value as http://prometheus-kube-prometheus-prometheus.jfrog-plg:9090/

Go to Data Source:

User-added image

Add Loki as datasource, add the URL (http://loki:3100 ) and test

Add Prometheus as datasource, add the URL (http://prometheus-kube-prometheus-prometheus.jfrog-plg:9090/ ) and test

In the Grafana, import the Dashboards and select the appropriate sources. (Get the JSON from this link or import)

Go to Dashboards

User-added image

Import the JSON

We could see the Artifactory Log Analytics and System Metrics:

System Metrics:

Artifactory Log Analytics: