JFrog’s Xray API at a Glance [swampUP 2021]

Hello everyone, welcome back to SwampUP 2021.
The topic of today is JFrog;s X-Ray API at a glance.
My name is Andres Almiray.
I’m currently employed by Oracle so many of you might know my name because I like to
talk about open source.
I am a Java champion alumni,
and a member of the groovy development team, the Apache groovy.
So I love to work with code, I love to work with open source.
And that’s one of the reasons why I decided to join Oracle. Again, this is my second time
at the database group. In this case, we have plenty of products that
touch the open source space. Yes, that is true.
And one of my capacities at this particular group is to establish bridges with
developers that will be interested to know what we are doing in the open source space.
So if you have any questions in that regard,
well, please let me know I’m very, very happy to talk to you about those things.
But today, we’re going to concentrate on JFrog’s X-Ray API.
Now, you might have seen these before,
JFrog has now published. While it’s known as the JFrog platform,
there is a series of products that are related to one another.
And then we can see that JFrog X-Ray is right in the middle of everything.
The other product that you may be familiar with is artifactory,
which is basically a place where we can put binaries of many different formats.
And based on the format, we gain different capabilities, such as if you were
Mavin, Palms and Yars,
when they will function your sub Maven compatible repository.
But it can also host Docker images, RPMs, DVM packages,
any kind of binaries.
Now, once you have these capabilities, this option of hosting binaries, another thing that you would like to have
is make sure that you don’t have any vulnerabilities in that particular binaries.
Now, if you happen to have some of those,
you also would like to be notified as soon as possible so you can
make changes, make adjustments, and make sure that your consumers are safe.
So well, you might have heard a lot about the JFrog platform and their products
today and in the past as well.
So the point that I want to make today
is that every single part of this platform, every single one of these products
are low, some sort of interaction with any different other capabilities with any other components.
And they do this by means of APIs.
So today, I would like to showcase a little bit of the X-Ray API,
and the reason why it will come quite apparent as we go through this particular session.
Now, the X-Ray API basically is a set of features that allow you to scan artifacts for vulnerabilities
or any other additional metadata such as
what kind of licenses the components have, and you can figure out if there are any
specific vulnerabilities of either there is the Docker image, one of the layers or there may be some Maven dependency
that has one of the issues or something else that is related to your particular environment.
You can set up pretty much CRUD operations for many of the capabilities exposed by the APIs,
for example, you can set up watches. So whenever an artifact gets a scan, you get notified immediately,
you can also set policies about those things,
you can get reports, you can create, you can delete those reports as much as you want.
And you can export and import the settings of your particular X-Ray instance.
So if you want to move to another instance or whatever is needed,
you have pretty much a lot of elbow room to move around.
And of course, there are other things that you can do with the X-Ray API.
The X-Ray again, it doesn’t work in the standalone mode,
it will be expected that you have your binaries hosted in an artifactory.
And so for that reason, today, we’re going to see both the artifactory API and the X-Ray API,
hand to hand.
Now these are the pages that are currently described what you can do with
each one of those APIs. The first one is for X-Ray. The second one is for artifactory.
If you browse to these particular URLs,
you will notice that artifactory has plenty of information about it because well, it is…
this is quite a big product in the sense that it has a lot of behavior,
it has a lot of capabilities,
whereas X-Ray is a little more recent than artifactory.
But also it doesn’t need to offer that much,
even though the behavior that he offers is is quite important.
Now, regardless of which way we want to browse the the APIs,
we will need to authenticate with one of those at some point.
There are many ways that we not authenticate
and the artifactory REST API
do mentions some of the ways that we can do it.
Before we get into that,
the X-Ray API,
it basically
follows the convention that we see directly on the screen,
it’s the name of the product, in this case is X-Ray API version one, I have an Asterix,
because currently X-Ray is migrating to version two, some of the…
the REST endpoints offered by version two are already working,
and not all of them have been migrated. So for the time being,
if you want to talk to every single capability exposed by this API, you will have to
dance a little bit between version one and version two.
Whereas in the case of artifactory, there is just one,
and the name of the product, artifactory, then /API, and then you can invoke pretty much anything else that you want to do.
Okay, so I was saying something about authentication.
There are a few ways that you can authenticate yourself when you are issuing a rest call
to any of these APIs.
The most basic one is to use basic authentication with a username and password.
Now the important key about it, this is something that tripped me on the first case is that
this is not your login password as is in plain text, you have to use
the encrypted version of your password.
If you use the plain version, you’re not going to have a good time because
the rest call is not going to be authenticated and it’s going to be a failure.
Now an alternative to using an encrypted password,
is that you can create an API key.
Now this works for both artifactory and X-Ray,
you can also use, you can create an access token
and also use basic authentication by passing the username.
Now the access token is going to be quite a long string of characters.
And I put here an Asterix
because when I was testing this out a few weeks ago,
I noticed that some of the endpoints that I was querying with
did not work correctly with this.
I don’t know exactly what I was doing, I tried a couple of times
and it would always come back with bad credentials.
So I had to switch to the other authentication mechanisms and just went back to working.
In this case, I was just testing out this one.
And if you still want to use an access token,
then the other alternative is that you use it as a bearer token.
So use pass in the authorization header bearer talking with the value
and off you go.
Now, you may be wondering, where do I get that encrypted password?
How do I get that value? Also, where do I get the IMA API key and where also
can I get a query and an access token?
Once you have logged in into your artifactory, or in this case, your JFrog platform instance,
navigate to your profile, look into administration. Once you log in and authenticate,
you see that in the section called authentication settings,
if you do not have an API key,
there will be a button right here that says generate an API key.
I already did it, that’s why it’s right there.
And you can copy and paste it,
you can regenerate it or you can revoke it and if you click on the little icon,
then you will see the value of the API key.
The same goes for the encrypted password.
There you can copy and paste it anywhere you want to or you can click on the icon and you will see the content.
This is the value that you must use
when you want to enter the gate using basic authentication with username and password.
Here is encrypted value of your password.
So what about access tokens?
If you see on the sidebar to your left,
there is a menu called Identity and Access.
If you click on it, there is a tab for access tokens and in this case,
I’m displaying the different access tokens that I have already.
Some of them have a limited lifetime
where they will expire very soon,
and some others will just live forever.
So when you click on the top right, you’ll see a button called generate admin token, and you click on it,
and then you’re getting greeted with this dialog. And it’s very important
that you select which product, you want to generate the token
Because if you select artifactory, then you’re going to talk to the artifactory API.
But if you want to use the same token and to talk to the X-Ray API,
you will notice that the API will not work, it will tell you bad credentials or some other kind of error.
The same happens with pipelines.
So if you want to use the three different products, you will have to generate three different access tokens.
You can also define how much time you want the token to live.
You may want not to expire it
or you might want set it to expire in so many hours.
Okay. So once you have this token, once you have the API key,
then you can safely just start talking to the API.
One thing that we can have is check
that the server that you want to talk to
is actually up and running.
There are two endpoints that we can call here,
we can call system ping
and they will return with a simple JSON response saying ponk.
Well, that’s simple, and notice that we don’t need any kind of authentication.
So this is a very, very trivial way to tell is the server up,
I can also query for its version, there is no need for authentication, either here.
That’s another way for you to tell, yeah, the server is up
and the version perhaps is the version number that you’re expecting.
So okay, if these match with your expectations, then we can certainly continue.
In the case of artifactory,
there is also a pink endpoint.
Now in this case, it just simply returns okay if everything is fine
and if there is a failure it will probably return on fail or failure or something else.
And if you want to query the version number,
notice that in this case, we do need authentication,
we need either the username or the password or the API key or the token.
The reason being is that not only does it display the version number of the server that you’re running,
but also certain capabilities that have been added to the server.
And you can notice that the last capability that this particular server is running is X-Ray.
So I have my artifactory instance tied up with X-Ray. So that’s pretty good.
Also, the last thing that I put out the redacted the license,
that you will see an actual number here, which somehow ties to your license. So
don’t share it if you can,
don’t do it.
Now,
the basis of artifactory is our repositories.
In my particular instance, I have three types of repositories.
Just playing with these things.
I have a Maven compatible repository,
I have a Docker repository, because I’ve just started playing with Docker images.
But for the nature of this presentation, I also created
just a generic repository.
Repositories of these type can handle any type of file,
any type of binary if you want to.
And if they happen to match the the expectations of the other…
of the other repositories, for example, maybe you are outdoors, and you have a loaded a Mavin jar
or a Docker image or an RPM or something else,
then use it when you’re browsing the near repository,
additional features will be available when you click on the file.
Otherwise, this is just like a bucket when you can
put in C files and tar files or anything,
this is something for you to upload binary bits.
Now, here we are seeing the display on the UI.
But we can get the same information from the API
when we issue this particular query.
So artifactory API repositories.
Also, this has to be authenticated, as you can see,
and we get a list of JSON values,
which is pretty much the same information that we got in the UI.
So we can summarize that what you see in the UI
is backed up by the artifactory API,
but it also will be backed up by the X-Ray API, and the other APIs exposed by the rest of the products
that are part of the JFrog platform.
So say that we have this binary repository
and what we need to do now is upload a binary.
And then once we do this, we’re going to scan it.
Now because I create a generic repository,
what I’m going to do is upload a zip file that contains, any kind of files, in this particular case,
it’s going to contain a jar file, because it happens to be a Java application.
But you don’t have to upload Java, you can upload anything you want to.
Say, if we want to use the API like this,
this is what we have to do.
We have to issue an output call,
and you pass in the full URL for your binary.
The important bit, sorry, is that after the binary, this is the name of your repository,
you can define the path for the file.
In this case, I said it’s going to be app/100, 100 is the version of my application,
and then the file,
or you can just not put app100 and just put in the file if you want to,
or do whatever, decide what path you want to do.
And then pass the file.
Now you see that also pass in additional values for the checks.
When I was testing this out,
did you just pass in the URL and the file
artifactory will automatically calculate the checksums for you.
And will have a button saying when you browse to the UI,
that the checksums were automatically calculated,
but you have to click on it to accept those.
But if you calculate the checksum on your own,
and you make sure that those are the actual ones, so you have a Sha1
and Sha256
and the last one is an MD file,
then what artifactory is going to do is still calculate those checksums on its own,
and will compare.
And when those matches, it will simply say, okay, fine,
I trust you, you have just uploaded the same thing that I just calculated on my own,
and we’re fine, we’re good.
If you were to browse the UI,
and you click on the button on the top right, that says set me up,
then there it gives you the options for uploading and also for downloading, that will be resolved.
Here, we see almost the same endpoint that we saw in the previous slide.
This one does not upload the calculated checksums it’s just the URL and file.
But also notice that artifactory has one additional authentication method that you can use,
you can pass in a header, a very specific header called x JFrog art API,
and the API key.
If you do this, then you will continue to work as if you were fully authenticated
with any other mechanism that we saw before.
Now, once we have uploaded the file,
if we navigate to the artifact, so you can find in the C file,
we have some metadata about it,
there it is, you can click on the different tabs and if we scroll down, you will see the checksums.
Everything checks out.
So it looks like our file, our binary is right there
in the right location, and you can double click on it, authenticate and download it.
That’s perfectly fine. So I guess that now’s the time to do a scan.
Okay, so for scanning,
you have to be really, really sure
that your repository has been configured for the scanning,
it does not happen by default.
And this is something that tripped me over the first time,
when I created the binary repository,
I missed something when I did so.
If we navigate to the repository section in administration,
click on my generic repository,
you’ll notice that there is a checkbox at the bottom,
enable indexing in X-Ray.
I have this one disable first I didn’t notice it.
And when I was trying out the API, I was getting lots of failures,
the component was not found,
the data that we’re sending was not correct. I was just doubting myself until I just
went back to figure out why was my repository not connected to X-Ray.
That was the last thing that I thought and
it was because I forgot to click on that particular checkbox.
So don’t do it. But even if you do, once you have created the repository,
you can go back to the administration and click on that particular checkbox and save.
The next thing that I did after enabling this checkbox
was look into the X-Ray settings
and then you see the index status.
After what I did, checking the checkbox, the index status of my repository was 0%,
I had two binaries, or 30C and a C file, so those were not indexed by X-Ray.
If you hover on that particular row,
you will see two additional buttons that not do not appear here on this screenshot,
the one is to calculate the index
or initiate an index for the data repository.
Another is for refreshing the index.
So this is the picture once I have calculated the indexes of my repository for the first time.
Now I also have a Docker image on my Docker repository,
that is why it says one artifact out of one, it’s 100%. I don’t have any JAR files right now on my Maven repository.
That’s why it is up to 0%.
Once I’ve done this, when I navigate back to my binary repository,
you see that there is now an additional tab that did not exist before.
This can tell me that I have my binaries, repositories connected with X-Ray.
And this particular artifact does not have any vulnerabilities, though.
So that’s quite fine given that
it’s a very simple Hello World application with no dependencies.
So there’s nothing that like, should fail with this component.
And I say should because that’s the whole point of doing what we’re doing.
It’s trying to figure out if we’re safe,
and it has detected one license.
The next thing that we want to do is query these components
using the API.
Now, we saw just that using the UI.
But in order to do it on the API, we have to know what is the component identifier.
And depending on the type of binary,
you have to use the different formats.
So if it were a Maven package, or like a jar file,
then we use the gap://[group ID] [artifact ID] version
but because we haven’t loaded binary file, a zip file,
then we have to use generic :// and then pass in the SHA 256, checksum
and then the name of the file,
it does not require the path.
So you might remember that I uploaded the C file as app100, then app100.c file
but in this case, I will just have to
copy and find the sha for that particular file, so I can proceed on the UI,
or the one that I just calculated.
And also path app-100.c file and that’s the only thing that I need to do.
So here’s what I do here.
You can do it on the command line, pass in curl username, password, because this has to be an authenticated operation,
we’re going to send JSON content to that particular operation and passing component ID.
And then the component.
notice that when I redacted the checksum for brevity,
but that should be the whole thing.
And just post it.
And that will tell me if it’s successful that a new scan is in progress,
like what is shown here.
If I messed up the generic URL of the component ID is not found for some reason,
then I will get some sort of error and then we’ll have to figure out how to correct this problem.
Once you have done a scan, we can query it.
Again, you can see it on the UI.
But if we want to access the API,
here’s how we can do it.
You can pass in a list of checksums. So here comes our good friend, the checksum again.
So you can query more than one component by passing more than one checksum
or you can also pass in the past, I decided to use the checksum way again, using a post,
we’;; now use another operation called summary artifact and as we saw
we get a bunch of JSON,
and the UI will get rendered nicely.
But as we start to have more metadata
that is exposed by other components,
just browsing through in the terminal is not really that nice.
We can tell here that the the response for X-Ray is that we have two components.
There is a zip component that has no license,
there’s no license files inside it.
And there is a Maven component,
which is the jar that contains a zip file.
And this jar does have a proper license and has a patch too so that’s pretty good.
When we navigate in the UI,
if I expand the zip file, then we can certainly see that there is a jar file inside.
And if I went to crack open the jar file,
I will find a license but I can also tell that x ray has found the license.
And there are no security issues, no violations, nothing like that,
again, because the jar file is very, very, very simple.
If we are getting tired of looking at the information in the terminal or in the UI,
and if you want to experiment with the API in other ways,
well, we can also use other tools such as postman.
The advantage of postman is you can save your authorization headers
and just reuse them as you play around with the API.
So in this case, I issued the same endpoint for finding the summary
of our particular artifact, the app100 zip file,
and we get the JSON result now nicely formatted, much easier to browse
and find so there is one license I get,
or non licensed, the zip file has nothing.
And the Apache License I get is for the jar file.
Okay, so now I’m getting into the reason why I got into X-Ray and artifactory.
And again, as a Java developer,
I like to have integration with different build tools
and well, Maven and Gradle being the most popular ones.
And what I want to do, actually, I created a tool
that allows you to create releases on GitHub and GitLab and Git.
And as part of those releases, you can upload binaries to anywhere you want to,
one of those places happens to be artifactory.
So once I have uploaded to artifactory, then that will be a great thing to also scan for any vulnerabilities.
Now, interacting with X-Ray API, now artifactory API using just plain Java
can be very, very verbose.
Thankfully for Java developers, there are a couple of options that we can use
to reduce the verbosity of dealing with low level HTTP codes.
And one of the options is called retrofit.
Now the idea of retrofit is that you define a Java interface
that contains the method definitions and some sort of mappings to your rest APIs, your rest operations.
And retrofit will automatically generate a proxy that matches that particular interface
and do all the HTTP low level stuff.
And you as a developer, the only thing that you’re going to deal with will be standard Java types,
your own domain types. In this case, you see there is an object called rapper,
this is just a domain object,
a simple pojo with getters and setters that contains the data
associated with a particular case or repository.
Under that list, is just a regular Java util. collections list.
And so retrofit was perhaps the too, the library that I like to use a lot,
but at some point, the makers of the library decided to go with,
well, they switch over from Java to kotlin. And I have some issues with that particular dependency.
And I would rather not have it as a dependency on my project.
So I had to search for an alternative
if there happens to be an alternative, and I’m happy to say that there is.
There is a project called open fame,
which the thing that was started almost at the same time as retrofit.
At some point, the team decided, okay, let’s not continue working on this,
because it looks like retrofit performs much better, it gives what we want,
but after the makers of retrofit decided to take the product into a different direction,
open frame came back into life.
And open frame works pretty much in the same way,
you define an interface and do mappings from object, from the Java types into the rest APIs.
The mapping perhaps is a bit more verbose.
The defaults also don’t give you access to the low level HTTP information.
So if you want to know about the headers of a particular call,
so that you can do pagination,
there’s a little bit more code, there’s a little bit more elaborate that you have to do
in order to make it happen with frame, with retrofit
it’s much more of a direct operation.
But the thing is that you can still make it happen.
In the end, it’s just plain Java.
So the tool that I was telling you that I created in order to make these things to create releases
and to upload to artifactory is called JReleaser.
Now this is a tool that is inspired by another one out there called GoReleaser
which allows you to publish and create binaries for Go projects.
JReleaser does it for Java.
Now, in the case of JReleaser when I wanted to upload a Binary to artifactory,
I could not use either Fein or retrofit.
The reason for this is that the current way that we upload binaries to artifactory
is through a simple put
but we have to supply a header called 100 continue,
and then just wait for the signal from the server,
so that the upload will proceed, now, it’s
definitely impossible right now to do that kind of mapping with Fein
so we just have to access the API directly.
And that’s exactly what I had to do.
So even though I already said that, yes, Fein and retrofit will reduce a lot of the verbosity,
there are times when you just have to use HTTP directly.
And that’s exactly what happens here.
There is a bunch of headers
that we add when we do an upload.
In this case, you notice that I am calculating the checksums
for a particular file, this is actually the code that was uploaded to C file
and the tar GC file that we saw before.
And the last bit is here,
this is just a standard low level HTTP client style.
This is actually just plain HTTP URL connection, or even JDK 11 HTTP deployment.
We’re just setting headers,
and then pass in the binary data
and just write to that output stream. And we just need to wait for 100 and continue header,
when it’s done, we can decode the response. And if everything is fine, then we’re good.
And if the status code is bigger or equal to 400,
then something didn’t go as we expected.
And that’s pretty much it.
Once we done this, the binaries are there,
you can issue another rest call through X-Ray
to indexes and find any vulnerabilities or any other issues that the binary may have.
So again, let me remind you, are the resources for the API.
These are the canonical references if you want to get to know more about X-Ray and artifactory
and how to talk rest to them.
And if you’re interested in creating releases for any kind of Java project,
then just have a look at what JReleaser is.
So with this, I thank you very much. Thank you very much for your time.
If you have any questions, well, you know where to find me online.
So thank you very much, and keep enjoying SwampUP.