She Promoted her Helm Chart… You Won’t Believe What Happened Next! – Simon Walton, Anaplan

So, here we go! So this is a talk about Artifactory plugins. And what we’re going to do is we’re going to talk about, why we wrote a plugin in the first place and we’re going to talk about, how we wrote the plugin. Okay? We essentially had a problem to solve at Anaplan and we wrote an ask(1.06) for your plugin to some of it. And along the way, we, kind of, had a, kind of, a journey of discovery on how to write plugins and I thought that I would share it with you. When I first pitched this talk to swampUP, it was, kind of, along the lines of, “Hey, we got a really cool plugin and let’s show the plugin off.” But then I realized, “You know what, the plugin is quite simple and plugins should be quite simple.” so instead, “How about I talk about, how to write a plugin. And along the way, I’ll show you how we wrote this particular plugin, how it works and how you might write a plugin as well.” So we wrote a plugin for Artifactory and it’s called Atomic-Promote. And it was designed to solve a set of problems that we had at Anaplan and problems all around CI/CD. So, we’re a Jenkins house, we use Jenkins for pretty much everything to do with bill tooling in the company, and we didn’t see a reason to change that. However, we did have a move towards more CI/CD ways of working.

So we had a brand new project that came along last year called the New User Experience that we wanted to ship to customers and we realized we couldn’t go through this kind of long winded release cycles that we currently have. So we decided, “You know what, this is the ideal use case for deploying a Kubernetes cluster.” I’m getting at least part of our infrastructure onto Kubernetes. So this was a really good use case for it. What we did was, we provisioned a Kubernetes cluster or a set of Kubernetes clusters and we decided to start, start with Jenkins, right? So we built this shared pipeline library in Jenkins. The previous talk in this room, actually, we’re doing this, exactly the same thing. So you know, you have a Jenkins all at the top, you say in port shared library and that shared library allows you to do various things. What we wanted to do was, we wanted this shared pipeline library to be able to deploy to a set of, kind of, untrusted repositories. Yes, you deploy your Helm chart and your Docker images to this kind of unstable state, and then you ask Artifactory, “Please promote everything up the chain, up to tested and up to production.” Finally. So this was all, well and good.

And the idea is that when you start promoting things, obviously, each cluster can only talk to each, kind of, scope or level. In other words, in the production Kubernetes cluster at the top there, that should not be able to pull Helm charts or Docker images from, say, Helm tested and Docker tested yet. And we lock things down with Artifactory permissions, for that reason. So briefly, this is how the pipeline works. You have a Jenkins file, you deploy to your development cluster, you publish your Helm shot and your Docker images to Artifactory, and those are associated with an Artifactory build, and then you ask Artifactory, “Please promote my build at the end,” and Artifactory duly promotes the build. At this point you may be thinking, why did these guys write a plugin? Because Artifactory can totally do promotions. It’s a good question.

And so we had a couple of things that we wanted to address with the way that Artifactory does promotions. We wanted to promote Helm charts and Docker images atomically, yeah? So what I mean by that is, if you promote one thing, the other thing goes with it or nothing happens at all. We either promote both of them or neither of them. If you remember that previous diagram we had, we had Helm repositories and Docker repositories and they go up together. That’s the kind of thing we wanted. So either the whole thing bails and the user gets an arrow, couldn’t promote it or we get both of them going up together. What we also wanted to do was, we wanted to enforce a variety of eligibility checks on that promotion. Because of the nature of Jenkins shared pipeline libraries, the user ultimately has the ability to call there as things… that maybe they shouldn’t be able to.

And what we wanted to do was to make sure that we could verify that that promotion was correct, that it matters, compliance and auditing regulations. So we do have a variety of things around that that will kick in when you initially commit to your repositories and it’ll tell you this is not going to pass compliance. However, we also wanted enforce that in the pipeline as well. So we had a phone call or a zoom call with the lovely people of JFrog France, and we said, “Is this a crap idea of what we’re trying to do?” And they said, “No. It’s actually quite a decent idea and we encourage you to develop a plugin and feed it back to the Greek community. You will come up with a couple of problems. First of all, there’s no way to do Docker promotions using the public API and I’ll show you how to do that later. And they also said, “Fundamentally, the way that you do build promotions on Docker promotions and Artifactory, they two different end points.” Yeah. They are two different ways of promoting things.

And also, when you promote things in Artifactory, you promote the build and if you are promoting in the sense of you are moving or copying something up the chain, then usually what you are doing is you’re saying, “I’ve got a source repository and a target repository.” So let’s say I’ve got a Maven build, I’m moving my Maven outputs, so say a set of jars, et cetera and some building firs, I’m moving them up the chain from one repository to the other. It’s not really geared around this idea of picking things together and moving them in a set. So with that in mind, I’m going to take you through how Artifactory plugins even work. Can I get a show of hands, who has written an Artifactory plugin before? Good, a number of people. Who wants to write one but didn’t know where to start. Okay, good. I’ve got a good selection of people here. So this is going to take you through how we wrote the plugin and I’m going to take you through exactly how to start writing the plugin as well.

Show of hands again, who really likes Groovy? Okay. Artifactory plugins are written in Groovy. Somewhat, unfortunately, I should be careful. I say that because I know the creative Groovy works for JFrog now. I personally have issues with it. I mean, GString, come on, GString! The number of times I’ve been called up saying GString out loud in my workplace, it’s crazy. I’ve been surprised, I’ve been sent to HR. So the way Artifactory plugins work is this, you have a Groovy file that expresses your plugin logic, okay? And you’ll deploy this into Artifactory. So you’ll have, MyCoolNewPlugin.groovy. It’ll sit in Artifactory, you’ll deploy to, etc plugins directory. If you’re in a HA configuration, they’ll be synchronized straight over to the other node. You’ll also need to call an end point called reload plugins. And that tells Artifactory, “Can you take a look at the plugins and just make sure that you’ve loaded them into the Artifactory, the JVM context.” You can tell Artifactory to automatically look for plugins on a polling basis. But JFrog explicitly say, “Do not do this in production.” It’s okay for development purposes, but if you do in production, you can end up in a weird case where you may reload a plugin halfway through and it’s kind of undefined behavior that results from that.

So you’ll be using the public API when you write a plugin for Artifactory and the public API, despite the name, is, kind of, like a private API. I don’t know why it’s called that. It sits the heart of Artifactory and allows you to interact with the internals. And when you write a plugin, you’ll be opening up a DSL block, which I’ll show you an example of. And inside that DSL, you will be opening a closure and you’ll be writing some Groovy that expresses how to interact with Artifactory. And some of the things you will do is, you will say, “Builds dot get build, repositories got dot get repository.” It also helps you have the ability to look at the security information, you can log out and things like that as well. So they’re for convenience really so that you don’t have to call the public REST end point or anything like that, which should be really long winded and difficult.

Artifactory plugins… There are, kind of, 10 different types of plugin. If you look on JFrog’s documentation, you’ll see there are 10 different types, each of which gives you a different DSL to work with. And they’re kind of three main ways that you can initiate or trigger a plugin. Some of them are based on Cron, some of them are scheduled, some of them are event based. So in other words, for example, a building for Jason is about to be saved or has been saved or our user has initiated a download and you want to overwrite some behavior. You want to modify some behavior. Anytime you want to, kind of, extend the way their Artifactory works, then you know, you’re gonna need a plugin. And the way that promotional plugins work is that, you have a dedicated REST end point from outside. Okay. I’ll show you how that works too. So they are user initiated from outside via the REST API.

If you want to start writing a plugin, this is where you start. You go to JFrog’s GitHub, type in user plugins and you’ll filter by the repositories that you really care about. So there’s two that you’ll care about. There’s the Dev environment, which is really good because it provides you a nice griddle project and that griddle project allows you to spin up a real Artifactory that you will write your tests against. And it provides you a lot of, kind of, convenient functions around that. There’s also Artifactory user plugins, which is the single most important resource that you can utilize when you’re trying to write an Artifactory plugin because that gives you a curated list of a set of Artifactory plugins that have been designed by JFrog themselves and also contributed by the community. JFrog are very welcoming for people to contribute plugins to.

The documentation for the Artifactory public API is a bit sparse in places, so I found that this is the best resource by far. So if you go to write a plugin, what you’ll do is, you’ll get Cron on the development environment and you’ll create your new plugin folder there. And your new plugin folder, it’ll contain two files. It’ll contain the plugin itself. So MyCoolNewPlugin.groovy will be sitting there. And then you’ll have this test file as well, and this test expresses a set of integration tests against your plugin, against the Artifactory running your plugin yet. So you will stress test your plugin in some way. And I’ll show you how those work as well. The development environment is actually a griddle project and the griddle project has a set of griddle tasks in it that are specific to developing Artifactory plugins.

So when you initially download it, you’ll say “Prepare Art-Pro” and what that will do is they’ll go off to JFrog’s website and it’ll curl in whatever version of our Artifactory that you want. So obviously, you were trying to match production or the version that you are about to upgrade to in production. So you might try to match your DR environment or something like that. And then you’d say, “Start Art-Pro” and what that would do is, it’s a griddle task, it’ll start Artifactory in the background, real Artifactory and it’ll wait for it to finish, and then hand you back controls with a running in the background for you. And the way that it can tell that it’s Artifactory has started, it’ll monitor the logs for specific messages. It’s quite clever. What I should say is it needs a license.

Yes, you need a real enterprise license. JFrog have no problem with you using a real license for this, integration tests and things like that. Seal there’s instructions on the, read me on how to license it properly as well. What you then say is, “Work on plugin” cause you’d say “Griddle work on plugin” and you’d give it the name of your plugin and then the griddle task will go off and it will try and find your plugin. By default, it will look one directory up. So in this case you say, griddle work on plugin my-cool-new-plugin and it’ll find this directory and it’ll symlink your plugin and it’ll symlink the test files straight into Artifactory plugins directory. And then you’re ready to go. You can run griddle test and then griddle test will run all of the tests that you have in your MyCoolNewPluginTest.groovy file against real Artifactory. So that’s really nice and it’s a real Artifactory. So when you spin this development environment up, you’ll see the really… I love this loading animation that JFrog put in, it’s fantastic. And you’ll see… It’s a real Artifactory that you can play around with via the UI. It’s available, the port is there. The one thing you will want to do when you start is enable logging. So in a production use case, you would have a log level of info or something like that. For development purposes, you’d probably want something a bit more… You’d want to go with debug or trace or something like that, but yeah, you just need to add a new log entry into the log back to XML file to get logging up, took us a long time to figure out.

So, how Atomic-Promote was written. I mentioned earlier that JFrog provide a set of DSLs with which you can write Artifactory plugins. Here’s what the one for promotion plugins looks like. So you have this opening promotions block that tells Artifactory, “I’m about to define a promotion plugin” and you name the promotion plugins. In our case, we just called it atomicPromote. And then we give it four parameters, in our case… There’s no real documentation for these parameters except for, in the examples in the GitHubs. If you’re looking for the documentation, looking the examples for one of the example promotion plugins, you will find it there. The first parameter tells Artifactory, “Here are the groups.” So from the point of view of the permission system, here are the groups that are allowed to execute this plugin, anything else and you will get permission denied.

We can also version the plugins, which is really nice because if we’re trying to diagnose issues with Artifactory and you have multiple Artifactories, it’s really nice to know which version of this plugin you are running so that you can diagnose issues when you’re looking at the logs and things like that. It also means that, for example, if you building this in the pipeline, you can bump that version every time you build it. We also then have parameters… So this is mandatory parameters that we are enforcing that the user passes in. In our case we are saying, they have to pass in a target score parameter, which we default to tested. You remember the levels that we have in our promotion, we have the sort of unstable, we have tested and then we have production. It defaults to assuming that you’re promoting from the first to the second layer and then we open up this closure.

The closure is the last parameter and the closure is essentially the thing that gets executed when the user calls this promotion. Yeah. So as a standard, when you execute a build promotion Artifactory, you have the buildName and the buildNumber. So that’s what you give to Artifactory and a set of parameters as well. So this is how we actually call it by the REST end point. As an authenticated user, the way that you normally call the build promotion Artifactory is you would call, pretty much what you have here, so your /artifactory/api/plugins/build/promote/ except onto the end of it, you’ll put the name of this promote plugin, yeah. You’ll put atomicPromote, which matches the function that we have inside the promotions block there.

Next you give it the buildName and the buildNumber with which you want to promote and these link to the buildName and the buildNumber that you see there, in the closure parameters. Then we open up a query string. Now the way that the query string works in Artifactories is, kind of, awkward because we have a query string which is parameters, but then we also might want to link some parameters into the parameters attribute that we see in the closure. And the way we do that is, we delineate with the, kind of, the UNIX pipe parameter that you see there. So in this case, I’ve said, ciUser=Jenkins|targetScope=production, so that’s how I’m passing my parameters in. So, this is just a refresher on what we’re trying to do. So what’s plugging attempts to do is, you give it a build and you say, “Go and look for the Helm charts in there. Go and find the Docker images that were pushed along with those Helm charts. I want you to get them both promoted at the same time. Okay. If you can’t do that, then I want you to just collapse in on yourself and just do nothing and just return an error.”

With that in mind, I’m going to show you some of the code for this promote plugin, so that you can get a feel for what the code looks like for plugins because some people may not have seen this code before. And also, you might be to see how I’ve written it. So let’s go to V S codes and let’s go full screen. Can everybody see that, okay? It’s fairly clear. I made sure that my line widths were quite tight just for this talk. And it’s good practice actually, you’re line width should be 80 characters if possible. And so this is the entry point, as you can see, we’ve already gone through it. This is where we open the DSL. We’re logging out some information. We’re assuming that everything is okay. HTTP(18.27) status is 200. And the functionality for this plugin is, kind of, broken into three parts. The first is some sanity checks.

So we’re grabbing the parameters that the users passed in and we’re running some sanity checks on them to make sure that, is what the user’s trying to do okay, from our point of view. Then what we’re doing is after that, we’re gonna ask Artifactory, “Give me the build metadata. Give me the objects from the public API representing the bills and give me the Docker images.” I’m going to go off and ask Artifactory, “Give me all the Docker images.” And then finally step three, we’re going to take all of the information and then we’re going to say, “Promote the Helm charts and the Docker images side by side, please.” And then exit. Nice and straightforward. So, getString property is just a method that we wrote to go into this Params HashMap that was passed in, that you get in the closure and just try to get a target scope out of it. And we’re saying true for mandatory. This getString property is in lots of the example plugins that you see on GitHub. It’s, kind of, a standard method that they tend to drop in there.

Then what we do is we calculate the source scope from the target scope. So the scope, remember is we’re saying, for Sap (19.40) for example, promote it to tested and so, the source scope with the level below. What we then do is, we figure out what the names of the repositories are. So we got the source and the target scope for the Helm on the Docker images. We just compute what these are and we do some sanity checks on those to make sure that they actually exist and it’s a valid scope. Or you’ll notice our Helm charts have -local on the end of it and that’s because we use the backing Helm repository. For those of you who’ve created Helm repositories in Artifactory, you will know that you need a local repository and a virtual one in order for Artifactory to act as a Helm repository. The virtual repository is the one that people actually access and then it’s backed by a local repository.

So then what we do is we attempt to populate this sourceBuild and detailedSourceBuild. And this is an example of how to getBuild information using the public API. So this is where we really start using this public API. So getBuild is a method that we’ve got that’s uninteresting, basically, it just says builds.getBuild behind the scenes and does a bunch of sanity checks. And then we convert that into a detailedBuilds. And the way you can think about these is a buildobject… Well, it’s actually called a BuildRun object and it’s in the public API. The BuildRun object is essentially what you get if you browse to a build in the Artifactory UI. The BuildRun is, kind of, like, the panel that you see at the top. And then the detailedBuild run is all of the tabs that you see below that sort things like, the list of modules, the release history of that build and so on. And if anything goes wrong at all, we bail.

The next thing we can do is… I remember I said earlier that we wanted to put eligibility checks into this plugin. So we wanted to say, “Hey, if the user is not eligible to promote this build or if something is wrong with the Artifacts in the build, like it doesn’t pass our compliance tests, then we want to bail immediately.” So this should we promote method, we’re going to be open sourcing this plugin and at the moment, that’s just a method that says, return true at the moment. So essentially, that’s to be filled in. In our example at Anaplan, we’ve got something… we extract our REST call in there. I don’t think it’s really advisable to call out in a plugin, but that’s what we do. We call it a service that just validates the commits and things like that to make sure they pass those standards. And then hopefully, you get a true back from that. So that’s that. And yeah, if the user is an admin, it’s nice and straight forward, right? We’ve got the user above by asking the security system, we’re saying if the user is, and I’ve been, then they can override that because you’ve always got to have an override mechanism in place in case things go wrong because things do go wrong.

The next function is quite interesting. This is called getDockerdigestsForArtifacts. There are a number of ways that you can associate Docker images with Helm charts, right? You can put the chart into the Helm chart. You can tell Helm, “Here’s the exact repository I’m going to and so on.” So it is possible that what we could do is we could download the Helm chart and we could go in there and we could figure out what we should be looking for. But we thought that was probably not a good strategy for the long run. What we decided to do instead was we decided to tag the Helm chart in Artifactory with attributes. And I’ll show you what that looks like in the UI. Some of you may not know that when… You can tag things in Artifactory with attributes and you can actually have lists as well as single properties. So in our case, what we do is we tag the Helm chart with a list of Docker digests. So we get the Docker digest strings and we tag them into the Helm chart attributes. All this method does… I’ll show you what it does.

So look(23.31), all it does is it says builds, which is again, that publicly available object that I mentioned earlier, you get all these builds and repositories and so on. So we say builds.getArtifactsfiles and you give it the build and then that’ll give you back all the files within that build. And we then go through them and we look for a property called atomic.docker.digests and then we flatten it, and then we… because if you tell Artifactory to store a list on a property, it’ll come back as a string that is comma separated. So we just split that, flatten it and then return all the ones where it’s no null. So it’s nice and straightforward. And then what we do is we convert that set of digests into a set of repoPath objects and a repoPath is as it suggests, is just to an object representing an exact path to something in Artifactory. So that’s what that does. And then we just do a sanity check to say, “Hey. Is the list of digests that we wanted to get the same size as the list of actual Docker images that we got back in return?” If we didn’t get them all, then there’s some problem. And in that case we would cancel the promotion with a 404. We’d say we didn’t find some of the Docker images.

Right. The next part. So the next thing that we want to do, we’ve got all of our information ready, the next thing we want to do is we want to say to Artifactory, “Please try a dry run of the Helm chart promotion and then a dry run of the Docker promotion.” And if everything goes well, then we know we are probably okay to go with the real promotions. So you can see this happening down here. So we do a dry run of Docker, dry run of Helm and then we do the real ones. And what these do is these call these closures that we have here. So here’s the Helm one. All the Helm one does is it creates this promotion object that’s available in the public API. So I’ve got this promotion in getPromotionInstance method that I’ve written that fills in the promotion object for me because it’s just a lot of parameters that go into a promotion. I don’t know if anybody’s called the promotion that implies a lot of things you can customize. And it passes this dryRun parameter that is also passed into the closure which means that we can run a dryRun first and then we promote the build by giving it this promotion object.

We also say if you are not doing a dryRun, then also under release status and a release status, I’ll show what that looks like in the UI after this. But the Release Status, sort of comes for free when you do a normal build promotion Artifactory but if you do custom promotions, you are the one that’s responsible for adding a Release Status. And it’s obviously really important from our auditability point of view that you have. You want to be able to say, you want me to click on the build and see, “Okay. This was promoted to the tested scope at some point by this user.” In our case, it’d be Jenkins that would appear as the person who promoted it, good old Jenkins.

Right. So now the Docker promotions. Now this is a bit more difficult because as I alluded to earlier, you don’t get, sort of, first class Docker promotion support available by the public API. You have to call the REST API, so the one everybody else uses. And that’s what we do here. So we say, dockerImages.each, promoteDockerImage for each one. Yeah. And if any of them fail, we immediately, sort of, bailout and remember that we can do a dry run of this first as well. Now unfortunately, when you try to ask Artifactory to perform a Docker promotion, you can’t do a dryRun like you can do with normal build promotions. It’s basically, you do it or you don’t do it. So in our case, what we do is we do a bunch of sanity checks. We say, “If dryRun, then do some sanity checks.” Like does the repository exist and so on. We could be doing a lot better here and we have improved this since, but that’s how it is right now.

And then what we do is we get the username and password of the current user executing the plugin and we literally just call the REST API internally. Right? So we’re running in Artifactory and we call Artifactory’s public REST API from inside itself. So 8081 is the Tomcat port. We just call Artifactory’s REST API. I didn’t expect this to all work when I wrote it, but it just did work, which is amazing. And then we look for this HTTP responseCode of 200 to make sure everything was successful at the end. So yeah, this is not as robust as I would like. I would like JFrog to introduce a dryRun of Docker promotions if possible. I don’t know if it is possible, be really nice because right now this is, kind of… It’s not as robust as I would like it. So at the very end then all we do is we just say, “Okay, if we got to this point, everything’s okay.” We log out, the plugin is ended because that’s where… You’re looking for those blocks. When you’ve written a plugin, you’re looking for, if anything goes wrong, you want to know when did the plugin start, when did it end, did it end successfully? We make sure the status is 200 and we write that, “OK”. And if you say, “OK”… When you call this endpoint, that’ll come back to you in the response body as well. Yeah. So it’s nice. You can see that in Jenkins. You can see, “OK”. We should probably be more descriptive than “OK”.

Tests. So tests in Artifactory plugins development are really, really interesting. If only because of how, kind of, unexpectedly elegant you can make them. So we’ll have a look. So this is my directory structure, is a bunch of sort of irrelevant stuff, but you’ll notice as I mentioned earlier, you’ve got this AtomicPromote.groovy, but we’ve also got AtomicPromoteTest.groovy as well. And this is where our tests live. Close that. These integration tests look like, well, they look, kind of, like integration tests as you’d expect. We have per-test setup and a per-test tear down. The per-test setup is telling Artifactory, “Create a bunch of repositories that I want to be there.” So it’ll create the Helm repositories, the Docker repositories. It’ll create this PromoteGroup that is eligible to perform the promotions that we can have a test whereby we removed that group and then we try running it and we should get a 403 in response.

The tests themselves, you can see it quite simple. I tried to make them as, kind of, plain English as possible by abstracting as much of it as I possibly could. This test is the most basic test you can run against this plugin. It’s verifying the very essence of the plugin. We’re saying, when I deploy three Alpine images and a Helm chart to unstable, and I execute a promotion to tested, so one level above, then I expect that the helm charts exist in tested, the level above. I also expect that the Docker images exist in tested, one level above. This is quite nice, right? Because if you… This is the standard way you should be writing tests, right? Is you abstract as much of it away as you can at the beginning so that when the next person comes along and writes a test, they realize, “Oh, this is very… Don’t repeat yourself.”

I’ve got nice to obstructions with which I can just verify this quick hunch that I’ve got. I’ve got some new functionality and you can properly red green Artifactory tests. We’ve got another test to… Promotion to production fails if service is not present and tested. So we can say deploy some Alpine images to tested, but we’re not deploying the Docker images there. What we’re going to do is we’re going to execute the promotion, and what we should get in response is a 404 because it can’t find the Docker images. Then we’re verifying crucially that neither thing happens. Neither the helm charts, which did exist, nor the Docker images, which didn’t even exist, have been promoted. That’s, kind of, the essence of the plugin. We’re verifying that if something goes wrong, then nothing happens.

So that’s how Artifactory tests work. If anybody wants to come and see me after we can go through the code together, I’d be very pleased to do that. I’m really only good with the promotion type plugins. There’s a wealth of plugins available on the inside, the user plugins repository on JFrogs GitHub. I really encourage you to have a look in there because there’s a huge number of things that you can extend in Artifactory. It’s really, really powerful. There’s some good examples of tests as well that you can look at. I think this is the most comprehensive test suite that I’ve seen. Certainly we’ve really gone to turn with these tests. I’m going to jump back to the presentation. Okay. Yeah.

As I was mentioning, this is what the Docker digest property looks like. See, you’ll see atomic.Docker.digest as the first property in the list. And you’ll see the values, it says “See list”. See you can click that in the Artifactory UI and it’ll open up a model box and it will show you the list in there. I do warn you, if you are setting these automatically via the rest end points, it’s kind of awkward. The character encodings kind of will trick you and the delineation character that you’ll use will trick you now and again. If you want to see examples of how I’ve done it, come and see me if you’ve been struggling with this as well. But you can store lists in properties which is really, really nice. This is what the release history looks like. When we perform a promotion, normally a build promotion, we expect to see this and we expect to see it as if we wrote our own promotion mechanism as well. Right? It’s important for auditing purposes. We want to see who performed the promotion let me say, and when it happened and we want to see why as well. You can add comments as well. Ours is just commenting promoted to target scope.

When we started writing this plugin, it was initially just me writing it. I had this idea that I wanted to try to plug in. I went ahead and I sort of had fun, lots of fun writing it. It came time to creating a PolarQuest and gave it to another member of the team to review. I created a PR and I said, “Sam, can you please review this for me?” He said, “Sure.” He checked out the brunch from GitHub enterprise and he said, “Oh, I see you’ve made a Read Me on how to run the test.” I said, “That’s right. I have been very thorough.” I think about six hours later he said, “If I still can’t get the test to run…” It turned out he’d missed one tiny step in this Read Me here because what you’ve got to do is you’ve got to get everything in exactly the right order directory wise. Then you’ve got to crawl all the Griddle commands in the correct order, and you’ve got to make a minor configuration change to Tomcat to match what’s in production so that the tests are pointing in the right place. You don’t have to change those and so on. You’ve got to make some change to the Docker. Remember that in our tests we are really using Docker. Those tests are literally using Docker, they’re Docker-pushing into the development Artifactory which is really cool. However, Docker will not push to a localhost repository. If you say, “Push to localhost repository,” Docker will say, “Nope, not doing it.” What you can do is you can say, “Okay, I’ll add something to my UTC hosts and I’ll say Artifactory is at 127.0.0.1.”

Then Docker will say, “That’s an insecure registry. I’m not pushing to that, are you crazy? Then you say, “Okay, I’ll add HTTP Artifactory to my list of insecure registry.” Then Docker is happy, it’ll push to that repository. I realized I missed that out of the reading as well. I’ve totally forgot that I even did that because I did it over a period of weeks and I realized this is not good. Because if the tests are this hard to run, then not only is it going to cost us a lot of development time, but also people, may end up just not running them. So I might give somebody a pull request and they might say, “Oh yeah, totally run the test, is brilliant. Why not just apply it?” Because they happen to think the Groovy code was okay because in the real world, right? Ideally integration test, they should be one click. Although integration testing is hard, it should still be one click. If you can possibly automate it, you should automate everything. I realized this wasn’t good enough, so I set out to automate these tests and I realized we had three concerns. But the plug in itself, we have Artifactory, sits in the middle. And then we have Docker, which we also want to control in some way.

So what we can do is we can Docker compose this, right? So we can say, “docker-compose up” and we can have a container that runs Artifactory inside it. We’ll say, “Prepare Artifactory Pro and download it, start it.” And then we can have our plugin periodically poll Artifactory’s ping and points which will eventually return your 200 with “OK” in it. So we can get the plugin to keep asking Artifactory, “Are you alive, are you alive, are you alive?” And then we can get the plugin to execute these griddle commands to say work on the plugin test. And there’s a bunch of other stuff that we’ll do as well to configure these tests exactly as we want them. All the things that somebody would manually forget to do, I would mess up in some way with typos and things. Because people are difficult, people don’t read sometimes and people are tired and it’s best to automate things. And this Artifactory is pushing and it’s pulling from a public Docker hub or actually it’s pulling from an Artifactory which mirrors out and it’s pushing back into Artifactory. It’s using this Docker that we control.

So this is the real official Docker in Docker image. Yup. And inside there all we’ve done in a Docker file is we’ve said, “From Docker and Docker.” And then we’ve customized to say that HTTP Artifactory is an insecure registry and that’s fine. And then Docker is totally happy and it means that we can have a one-click test setup for this plugin. If we say exit code from plugin, what that will mean is we have an environment where we can just run docker-compose up –exit-code-from plugin and docker-compose will run the full test suite. When the plugin exits, in other words, when all my tests have passed or maybe some of them failed, the whole environment will be brought down and control will be returned to the user, which is really nice. And then we can Docker CP the test results out of the Artifactory container. And if you are a human being, then you would CP the HTML version out. If you are a robot or you are a Jenkins, then you would get the XML version out.

In the spirit of designing for the person at the back of the room, here is a tiny text terminals that I’m going to show you that is running our integration tests. We’re saying runtests.SH, which is, kind of, like just a basket that runs docker-compose and then copies the tests out. You can see here at the bottom, Artifactory is still on unavailable, sleeping. Artifactory is trying to start up and the plugin is keeping asking Artifactory, “Are you awake? Are you awake?” Finally Artifactory starts up and our plugin says, “Artifactory is up, horray! Executing commands.” So know what the plugin does is the plugin container will set up Gradle and that were run all the tests. So this is massively sped up, by the way because I didn’t think we all wanted to sit here watching this. It’s executing all of the tests against Artifactory. The full theme takes about three minutes or so. It’s not, actually not too bad from start to finish. And then the plugging container is exited called zero. That’s good. And then the bass script opens up our test HTML and we can browse to see which tests passed and which ones failed. In this case, everything was good. Fantastic. So I realized once we had this, is that what I could do as well was, I could make Jenkins run these tests.

One of the standards we have in our team is that if you have tests, they should be running on PR hooks yet. So when you commit, when you raise a pull request, a web hook goes off to Jenkins and says to Jenkins, “Run all of the tests? So run Gradle tests.” And in our case we say, we have a Jenkins file that will run all of these integrations tests. So what we can have is we can have a Jenkins file and that Jenkins file just creates a pod. And inside that pod, there’s a Docker pod and a build pod. The build pod says, “docker-compose up –exit-code-from plugin.” And then the last thing that it does is it Docker CPs the J unit compatible XML out of the Artifactory container and gives it to the J unit plugin in Jenkins, which means that you can see the latest test results in Jenkins, um, for auditability and also means that, when you raise a PR hook and you look in Jenkins, you’ll see this spinning circle. Eventually you’ll see that Jenkins was okay with your build, all the tests pass.

Essentially what this is is because we have a Kubenetes powered Jenkins, the levels of abstraction here are a bit crazy because what we have is we have a Kubenetes powered Jenkins, which gives you an agent, which is inside a Docker container. And then you have a pod that defines two Docker containers, one of which is a Docker in Docker container, the other of which is a Docker container that spins up docker-compose with needs Docker in it. And then of the three of those containers that spin up with docker-compose, one of them as a Docker in Docker container. So all in all, I think it goes five levels of Docker deep. So it’s, I think we know we’ve reached peak Docker and some things may be a bit wrong when we go this far, but it works quite well for us actually.

So all in all, you can write Artifactory plugin and have fun and write them as a team and all collaborate together, if you have good automation. There are limitations, particularly for our use case because we need Docker, it makes things really difficult because it means, for example, I can’t easily PR this into a JFrogs official set of plugins because it would make their Jenkins file fail because it would expect Docker to be under our control, et cetera. So that’s not ideal. I’d like to improve that if I possibly could. You can have fun. So if you have good automation, we TDD part of this plugin, which… it’s amazing for something as esoteric as an Artifactory plugin. You can actually TDD it. We’d have one person sitting down, write a failing test, expressing the plugin functionality you want, hand the laptop to the other person, they would implement the functionality. And then you’d run the test suite and you’d hope that it passed again and then we switch roles. You can do the same kind of developments that you do with anything else, which is really, really nice.

And importantly, because you have integration tests, you can have confidence as well that when you deploy this plugin into production and you notice I haven’t covered deployment, because that’s, kind of, a separate issue. But when you do put this into production, you can have confidence that it will work, as you say it will. Obviously, for complete confidence, you would want to put this into your staging environment. Particularly, if you’re upgrading Artifactory, you’d want to know, does this plugin behave itself in the next version of Artifactory? So you’d still want to perform all those measures, but at least you know the plugin is behaving as you would want it to, logically. If you do write a plugin, I do encourage you to open source it because that’s the way that we all learn from each other.

We had that panel on community this morning. It’s really, really important that we all learn from each other and sort of improve the documentation around the place and give lots and lots of examples for how to do cool things with plugins and look out for atomic promote as well. I’m going through a process with Anaplan right now. What I’m trying to get this I’m open source to… It should be available pretty soon. Where am I gonna put it? I don’t know yet. But look out for it. If you go to github.com/Anaplaninc, it should be there, at some point soon. I believe there’s probably not any time for questions. However, I think I’m going to be led somewhere where people can ask me questions, so we’ll do that instead. Thanks very much everyone.