ARTIFACTORY: Troubleshooting the Metadata Service

Shai Ben-Zvi
2023-01-22 11:10

Relevant versions: This information pertains to Artifactory version 7.x 

Written in Go, Metadata is a microservice that comes bundled with Artifactory 7.x. It's used to store metadata information packages from JFrog services, such as Artifactory and Xray, and has its own configuration settings and logs files. As Artifactory is written in a different language (Java), this can create several problems. Accordingly, when any issue arises with Metadata, start your troubleshooting by having a look in the $ARTIFACTORY_HOME/var/log directory at the following files:

  • metadata-service.log
  • metadata-request.log
  • console.log

One of the most common use cases, you might encounter is when Artifactory is providing a connection string (which occurs via the system.yaml configuration file). During startup Artifactory parses its Java connection string and parameters and then converts it to the Go format. However, from time to time, there are edge situations where this process can fail. Here are several other common issues, which may occur, along with suggestions regarding how to resolve them, and each error can be found inside the metadata-service.log:

If your Artifactory instance is running with an external Oracle DB, you might encounter this error:2020-03-19T18:26:17.366Z [jfmd ] [PANIC] [4302998e04479677] [database_bearer.go:51         ] [main                ] - Could not initialize database (db config: {oracle ARTIFACTORY_TEST/***@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=pepldr00237.corp.pep.pvt)(PORT=60003))(CONNECT_DATA=(SERVER=DEDICATED)(SID=stm1d)))}): ORA-00000: DPI-1047: Cannot locate a 64-bit Oracle Client library: "The specified module could not be found". See https://oracle.github.io/odpi/doc/installation.html#windows for help
error connecting to database. giving up
jfrog.com/metadata/services/common.(*databaseBearer).init
metadata/services/common/database_bearer.go:84
jfrog.com/metadata/services/common.NewDatabaseBearer
metadata/services/common/database_bearer.go:49

/database_bearer.go:51 +0x2a8
main.main()
metadata/metadata.go:29 +0x435

1. [database]
This means that there's a missing Libaio library, which is required to work with the Oracle DB. To resolve this issue, you'll need to install it by following the instructions available HERE.
2. An error occurred when converting a Java Database Connectivity (JDBC) connection string to a GO-style connection string. For example:

error reading configuration from system.yaml : reading db properties failed: could not derive db properties from system yaml: Type: , Url: : Could not set db properties to application configuration: Could not parse jdbc url's connect part: ldap://uoid.rsv.se:7389/ ZV3T001,cn=OracleContext,dc=rsv,dc=se
goroutine 1 [running]:

Here the connection string was as follows: jdbc:oracle:thin:@ldap://mydomain.nex.co:7389/ ZV3T001,cn=OracleContext,dc=rsv,dc=seMetadata tried but failed to convert the JDBC connection string to a Go connection string.
The workaround is to go inside your system.yaml and, in the metadata.database section, add a Go connection string with go: prefix, and then add a go-style connection string instead of JDBC connection string. The details on how to build a connection string for Oracle are available HERE. For example, if the connection string is oracle://user:passw@service_name, you'll need to set the value to go:oracle://user:passw@service_name. ֿ

3. Here's a different type of error:

2020-03-22T18:25:10.663Z [jfmd ] [ERROR] [4104ec4b9d67964e] [database_bearer.go:51         ] [main                ] - Could not initialize database (db config: {mysql artifactory:***@tcp(adim-mysql.cvooa7fn0wpy.us-west-2.rds.amazonaws.com:3306)/artdb?charset=utf8&tls=true&parseTime=true&clientFoundRows=true}): x509: certificate signed by unknown authority error connecting to database. giving up
jfrog.com/metadata/services/common.(*databaseBearer).init
/src/jfrog.com/metadata/services/common/database_bearer.go:84
jfrog.com/metadata/services/common.NewDatabaseBearer
/src/jfrog.com/metadata/services/common/database_bearer.go:49
main.main
/src/jfrog.com/metadata/metadata.go:29
runtime.main
/src/runtime/proc.go:203
runtime.goexit
 
/src/runtime/asm_amd64.s:1357

This indicates a missing SSL/TLS certificate. To fix this, on your Artifactory machine, you'll need to add the appropriate certificate file in your /etc/ssl/certs folder.

4. Another common error is when the database password inside of the Go connection string's metadata.database.url is not encrypted after startup. The error above occurs because it's impossible to encrypt only a part of the connection string safely, as there's no standard way of writing those in Go, as there is in the JDBC. To fix this, we've created a placeholder replacement for this configuration key only. Here's how to use it:

shared:
  database:
        driver: org.postgresql.Driver
        password: very_secret_password
        type: postgresql
        url: "jdbc:postgresql://mydatabase.net:5432/artifactory"
        username: artifactory
 
metadata:
    database:
        url: go:user='artifactory' password='${shared.database.password}' dbname=artifactory host=mydatabase.net port=5432 sslmode=disable
 
When reading the Go connection string, ${shared.database.password} will be read as very_secret_password (the file contents won't change) and very_secret_password will be encrypted in the physical file by the router service as usual. You may use any configuration key in the placeholder, even environment variables.

5. Master Data Services (MDS) fails with the following error:

2020-06-15T07:25:47.196Z [jfmd ] [INFO ] [60ebe5752c965a8d] [database_bearer.go:84         ] [main                ] - Connecting to (db config: {postgresql user='artifactory' password='***' dbname=artifactory host=127.0.0.1 port=5432 sslmode=disable}) [database]
2020-06-15T07:25:47.214Z [jfmd ] [PANIC] [60ebe5752c965a8d] [database_bearer.go:68         ] [main                ] - Could not initialize database (db config: {postgresql user='artifactory' password='***' dbname=artifactory host=127.0.0.1 port=5432 sslmode=disable}): error connecting to database
jfrog.com/metadata/services/common/db.(*databaseBearer).init
/src/jfrog.com/metadata/services/common/db/database_bearer.go:114
jfrog.com/metadata/services/common/db.NewDatabaseBearer
/src/jfrog.com/metadata/services/common/db/database_bearer.go:66
main.main/src/jfrog.com/metadata/metadata.go:38
runtime.main
/src/runtime/proc.go:203
runtime.goexit
 
/src/runtime/asm_amd64.s:1373
goroutine 1 [running]:
runtime/debug.Stack(0x38, 0xc000160300, 0xc00032c200)
/src/runtime/debug/stack.go:24 +0x9d
jfrog.com/jfrog-go-commons/pkg/log.(*standardLogger).Panicfc(0xc000554ac0, 0x16058a0, 0xc0004d9bf0, 0x1390d88, 0x32, 0xc00032c200, 0x2, 0x2)
/src/jfrog.com/go-commons/pkg/log/standard_logger.go:42 +0x6a
jfrog.com/metadata/services/common/db.NewDatabaseBearer(0x16058a0, 0xc0004d9bf0, 0x16064e0, 0xc000090b90, 0x160a760, 0xc0004e5810, 0x15fd860, 0xc0000bf2e8, 0x15e0aa0, 0xc000554a90, ...)
/src/jfrog.com/metadata/services/common/db/database_bearer.go:68 +0x2d4
main.main()
/src/jfrog.com/metadata/metadata.go:38 +0x5b7
 
 [database]

This occurs because the password to connect to the PostgreSQL database contained the backslash character, which failed to parse. There are a few possible workarounds:

  • Create a new password without removing the backslash character from the password (Note: This change must also be made on the DB side.)
  • Provide the password with an encoded character (instead of the '').
  • Similar to the workaround for issue #2 above, provide the Go connection string explicitly to the Metadata service, which is inside your YAML configuration file.
6. If you're the MSSQL/SQL server DB as your external DB, it's very common that the connection string will contain a backslash in the URL between the server name and the instance name. For example: JFrogsqlserver, which can also create an issue for the Metadata server. To resolve this, here are two options:
  • Modify your JDBC URL as follows:

url:jdbc:sqlserver://<your-db- dns>:<port>;instanceName=<instan-name>;sendStringParametersAsUnicode=false;databaseName=<database-name>
For example:url:jdbc:sqlserver://vtcFxzh@myartifactorydb:1385;instanceName=ARTIFACTORY;sendStringParametersAsUnicode=false;databaseName=artifactory
This is another version of the JDBC which doesn't contain the backslash and will be accepted by the Metadata service.

  • Another option is to provide an MSSQL Go connection string (as specified in the examples here) separately to the Metadata service.

Should you encounter any issue which has not been discussed here, open a ticket to Jfrog Support and we'll investigate further.

Published: Dec. 28, 2020
Last updated: May. 31, 2020

Keywords: metadata, database