Sending Pipelines Nodes Agent Logs to Logstash

Pipelines Nodes have an agent running in them. This agent is responsible for fetching their work load from Pipelines, triggering the run at the node, and reporting everything that happened during the run to Pipelines. This agent produces logs while doing that work, which can be streamed to a Logstash service if available. This is especially useful for troubleshooting dynamic nodes that can be already gone when an error is detected, making it impossible to inspect the logs inside the node.

Logstash Requirements

Pipelines agent uses a TCP socket when writing logs to Logstash. For that to work, Logstash needs to have a TCP input enabled.

The following snippet shows an example of how to enable TCP input in Logstash:

input {
        tcp {
                port => 5000
                ssl_enable => true
                ssl_verify => true
                ssl_cert => "/usr/share/certs/logstash.crt"
                ssl_key => "/usr/share/certs/logstash.key"
                ssl_certificate_authorities => ["/usr/share/certs/rootca.crt"]
        }
}

Enabling Logstash Integration

To enable Pipelines Nodes Agent integration with Logstash, add the following entries to your Pipelines system yaml runtime section:Pipelines System YAML

runtime:
  logging:
    logstash:
      host: logstash.my_company.com
      port: 5000
      clientCertificatePath: /opt/jfrog/pipelines/var/etc/security/keys/trusted/node.ca.crt
      clientPrivateKeyPath: /opt/jfrog/pipelines/var/etc/security/keys/trusted/node.ca.key
      logstashCaCertificatePath: /opt/jfrog/pipelines/var/etc/security/keys/trusted/rootca.crt