Metadata Service Troubleshooting

Shai Ben-Zvi
2020-06-18 11:20

Subject 

Metadata Service Troubleshooting

Affected Versions

7.x

Description

The Metadata is a new micro service written in Golang, presented on Artifactory 7.x which holds the metadata of packages information from JFrog services like Artifactory and Xray.

This service has his own configurations section and logs files.
With any issue which is related to the metadata service, to start troubleshooting, please refer to the following log files which can be found under $ARTIFACTORY_HOME/var/log directory:

  • metadata-service.log
  • metadata-request.log files. 

Since  Artifactory is written in Java and Metadata is written in Golang, this can creates several issues.

Once of the most common use cases when working with Artifactory is to provide a connection string which is done through the system.yaml configuration file. 

When providing a connection string the connection string is in Java format, while the Metadata service requires a Go format which is different from Java.

Due to the above fact, the Artifactory application during startup is parsing the Java connection string and parameters and convert it to a Go format.
However from time to time, there are edge cases where this process can fail.
Here are some common issues which you may encounter and how to solve them, and each error can be found inside the metadata-service.log: 
   
1. In case your Artifactory is running with an external Oracle DB it is possible you may encounter this error:

2020-03-19T18:26:17.366Z [jfmd ] [PANIC] [4302998e04479677] [database_bearer.go:51 ] [main ] - Could not initialize database (db config: {oracle ARTIFACTORY_TEST/***@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=pepldr00237.corp.pep.pvt)(PORT=60003))(CONNECT_DATA=(SERVER=DEDICATED)(SID=stm1d)))}): ORA-00000: DPI-1047: Cannot locate a 64-bit Oracle Client library: "The specified module could not be found". See https://oracle.github.io/odpi/doc/installation.html#windows for helperror connecting to database. giving upjfrog.com/metadata/services/common.(*databaseBearer).init metadata/services/common/database_bearer.go:84jfrog.com/metadata/services/common.NewDatabaseBearer metadata/services/common/database_bearer.go:49…/database_bearer.go:51 +0x2a8main.main() metadata/metadata.go:29 +0x435 [database]

The above error means that there is a missing the Libaio Library which is required to work with Oracle DB.
To fix the issue, you will need to install it – please refer to this documentation which contains the required steps.

2. Some kind of error converting JDBC connection string to GO-style connections string, for example:

error reading configuration from system.yaml : reading db properties failed: could not derive db properties from system yaml: Type: , Url: : Could not set db properties to application configuration: Could not parse jdbc url's connect part: ldap://uoid.rsv.se:7389/ ZV3T001,cn=OracleContext,dc=rsv,dc=segoroutine 1 [running]:

Here the connection string was as follows: 

jdbc:oracle:thin:@ldap://mydomain.nex.co:7389/ ZV3T001,cn=OracleContext,dc=rsv,dc=se

Metadata tried to convert the JDBC connection string to GO connection string and failed.

The workaround for the above is to add inside the system.yaml under the metadata.database section a GO connection string with 'go:' prefix and then add go-style connection string instead of JDBC connection string.
You can find the details on how to build the connection string for oracle here.
For example if the connection string is oracle://user:passw@service_name  you need to the set value to go:oracle://user:passw@service_name

3. Another example of different kind of an error:2020-03-22T18:25:10.663Z [jfmd ] [ERROR] [4104ec4b9d67964e] [database_bearer.go:51 ] [main ] - Could not initialize database (db config: {mysql artifactory:***@tcp(adim-mysql.cvooa7fn0wpy.us-west-2.rds.amazonaws.com:3306)/artdb?charset=utf8&tls=true&parseTime=true&clientFoundRows=true}): x509: certificate signed by unknown authorityerror connecting to database. giving upjfrog.com/metadata/services/common.(*databaseBearer).init/src/jfrog.com/metadata/services/common/database_bearer.go:84jfrog.com/metadata/services/common.NewDatabaseBearer/src/jfrog.com/metadata/services/common/database_bearer.go:49main.main/src/jfrog.com/metadata/metadata.go:29runtime.main/src/runtime/proc.go:203runtime.goexit/src/runtime/asm_amd64.s:1357

The above error indicates on a missing SSL/TLS certificate.
To fix it, you will need to add the certificate file under /etc/ssl/certs in the Artifactory machine.

4. ERROR: 

The database password inside the go connection string metadata.database.url is not encrypted after startup.

The above error is because we cannot implement encryption of only a part of the connection string safely as there is no standard way of writing those in Golang like in JDBC.

To fix it – We have implemented a placeholder replacement ONLY for this configuration key. This is how you can use it:

shared: database: driver: org.postgresql.Driver password: very_secret_password type: postgresql url: "jdbc:postgresql://mydatabase.net:5432/artifactory" username: artifactorymetadata: database: url: go:user='artifactory' password='${shared.database.password}' dbname=artifactory host=mydatabase.net port=5432 sslmode=disable

 

When reading the GO connection string, “${shared.database.password}” will be read as “very_secret_password” (it won’t change the file content) and “very_secret_password” will be encrypted in the physical file by the Router service as usual. You can use any configuration key in the placeholder, even environment variables.

5. MDS fails with the following error:

2020-06-15T07:25:47.196Z [jfmd ] [INFO ] [60ebe5752c965a8d] [database_bearer.go:84 ] [main ] - Connecting to (db config: {postgresql user='artifactory' password='***' dbname=artifactory host=127.0.0.1 port=5432 sslmode=disable}) [database]2020-06-15T07:25:47.214Z [jfmd ] [PANIC] [60ebe5752c965a8d] [database_bearer.go:68 ] [main ] - Could not initialize database (db config: {postgresql user='artifactory' password='***' dbname=artifactory host=127.0.0.1 port=5432 sslmode=disable}): error connecting to databasejfrog.com/metadata/services/common/db.(*databaseBearer).init /src/jfrog.com/metadata/services/common/db/database_bearer.go:114jfrog.com/metadata/services/common/db.NewDatabaseBearer /src/jfrog.com/metadata/services/common/db/database_bearer.go:66main.main /src/jfrog.com/metadata/metadata.go:38runtime.main /src/runtime/proc.go:203runtime.goexit /src/runtime/asm_amd64.s:1373goroutine 1 [running]:runtime/debug.Stack(0x38, 0xc000160300, 0xc00032c200) /src/runtime/debug/stack.go:24 +0x9djfrog.com/jfrog-go-commons/pkg/log.(*standardLogger).Panicfc(0xc000554ac0, 0x16058a0, 0xc0004d9bf0, 0x1390d88, 0x32, 0xc00032c200, 0x2, 0x2) /src/jfrog.com/go-commons/pkg/log/standard_logger.go:42 +0x6ajfrog.com/metadata/services/common/db.NewDatabaseBearer(0x16058a0, 0xc0004d9bf0, 0x16064e0, 0xc000090b90, 0x160a760, 0xc0004e5810, 0x15fd860, 0xc0000bf2e8, 0x15e0aa0, 0xc000554a90, ...) /src/jfrog.com/metadata/services/common/db/database_bearer.go:68 +0x2d4main.main() /src/jfrog.com/metadata/metadata.go:38 +0x5b7 [database]

The cause of this error is because the password to connect to the Postgres database contained backslash '' sign which failed the parsing.
The possible workarounds are as follows:
1. Create a new password without Remove the backslash character from the password which requires also to change it on the DB side.
2. Provide the password with encoded character (instead of the ''). 
3. Same as the workaround for problem number 2 – Provide the GO connection string explicitly to the metadata service inside the configuration yaml file. 

In case you encounter any issue which is not included in this article, please open a ticket to the Support for further investigation.