Use Case – Automated Distribution of Open Source and Private Components with JFrog Bintray

Talk Transcription:

[Allen] All right. So I’m Allen Reese with Yahoo. I’m here to present automated distribution of Open Source and Private Components with JFrog’s Bintray. My cohort Jack Weaver will talk about the open source part, I’m going to talk about the private party distributions we do at Bintray. First this is our safe harbor slide, everything I say is a lie.

[Allen] All right. We’re going to talk about how we use Bintray. How we distribute private party builds, deploying open source for free, and then deploying to Bintray with Travis CI, which, probably the most interesting part, then we’ll do a Q and A.

[Allen] So why would you want to use Bintray? We had somebody who wanted to use — who wanted to open a firewall hole for our internal Artifactory instance. It was for a single partner for a single build. We needed shared copies of a build of some application with these people. We turned them down. We’re trying to find a way to work around this. A few weeks later somebody, we ended up setting up with Bintray, and a few weeks later couple more of these came in and then somebody else was like, hey I want to publish open source with — on Bintray. Turns out Maven central setup is really hard to do.

[Allen] We have an unusual setup with Bintray. We have multiple people pushing open source and then we have multiple people sharing builds with contractors and third parties. We call these private party builds. They go to a specific customer, they’re generally watermarked, there’s only one or two downloads of them, they’re not for public consumption. They’re typically mobile apps.

[Allen] We have — we also have multiple teams that want to publish open source artifacts. Bintray requires you to use user credentials to push or if you have enterprise entitlements, and this has changed since we’ve wrote the slides. They’ve added a few new levels so there’s much lower level you can get entitlements at. So since we don’t want to be controlling the users and dealing with the various users, we want the teams to be able to do it themselves. We’ve decided to use entitlements.

[Allen] This is an example of some of the repos. You can see we have partner A. We have a bunch of JNI libraries everyone likes to use, we have Tachyon, our super-secret mobile apps, and the one’s that public is actually the Maven repo. That’s where you’ll find things like the Flurry SDK and the omit — omit libraries.

[Allen] So, what is a private party build? I kind of went over this, they’re non-public builds, they’re unpublished, and we can’t have people getting them. Because it would be leaking. So what we do is we download key entitlements. So the download keys are good for 24 hours and for one download.

[Allen] This actually works really well, we talked to someone who’s doing this with their license software. They send their customer a license key, it gives them four or five downloads, however many they need. And once those are downloaded it’s done, you can no longer download it again. You have to contact support for a new one. This is how we deal with it because there are builds we do not want leaked. We want one person to get it and then be done with it.

[Allen] We have one repo per consumer. We use signed URLs for every build. They expire after 24 hours. They’re valid for a single download, if you’re distributing licensed software, maybe you’ll say five downloads.

[Allen] We encrypt them with a shared key before we send them. Cause you don’t want somebody intercepting the email. For most of our use cases, it’s one repo per product. Depending on if you’re distributing private party builds or license software you might do it a little differently. You can control the entitlements. You can get pretty […] with the entitlements. You want to talk to support about that.

[Allen] And now we’re going to do open source distribution with Jack.

[Jack] Quick disclaimer. As of about two, three weeks ago, I left Yahoo, so I’m no longer there. In addition to what Allen says being a lie, everything I’m saying is definitely a lie.

[Jack] So I’m – I’m going to run into some quick things with open source and how we kind of manage that at Yahoo. We have the enterprise – enterprise license and setup so what we do is not going to be your typical use case, because we do employ the use of download keys and entitlements. And we’ve done that really early on when Bintray first introduced this. So, you know, gosh, right when it was kind of really cutting edge features of Bintray was — was introducing as download keys we decided to start using those. And we applied it into our open source.

[Jack] So the goal here is we kind of want automated releases right into Bintray with minimal hand to hand holding, right. Yahoo’s a big company. We’re a tools team. There’s only so many of us, we kind of want to empower the open source teams, those that own the open source projects, at GitHub dot com slash Yahoo, to go ahead and, you know, do releases as they need them but not have to come to — come to us all the time for everything.

[Jack] And what this means is that, well, there’s a few approaches. You want to on board them, so we’ve come up with kind of a stream line process. We have forms that they fill out, they go through, in order to get them on boarded. And in addition to that, we’re actually using a unique application of download keys, for every project and an entitlement for those projects. And so the way this works is if you go to Bintray dot com slash Yahoo right now, you’ll get a picture similar to this. We got our projects. They’re kind of, those four are visible right here. And we go through and we say, okay, you want to come on. You want to actually create your package on Bintray, you have your source code in GitHub, we’re going to create a download key for you, and we’re also going to create entitlements. And those entitlements are — there’s one per project. And then I’m going to scope your entitlement, your download key to that entitlement.

[Jack] And so what that means is that the keys that I’ve given you to access read and write into your repository is scoped, it won’t work for other projects. This gives us a lot of advantages. For one, there’s no shared account. So if you want to do this outside the use of download keys and entitlements you might have to share account credentials to gain access to published from Travis CI, for instance, which I’ll talk about in a minute. And two, I can roll these keys. If a lot of the maintainers of the project or someone’s whoever is a point of contact that owns that project leaves or, you know anything like that we can roll, all right.

[Jack] So, how do you go about doing this? Well, for one you want to create your key. Pretty simple, they’re — all this is documented in Bintray’s doc. I’m not going to talk too much about the APIs but I’ll show you some examples. You create your download key, it’s an easy post call. You’re basically are going to say, hey this is the name of my key, and in addition to that, you have to authenticate yourself to Bintray. And what that means is that, that right there is the base 64 encoded username and API key of my account on Bintray. So I’m on Bintray, I’m an admin and you know, here I am. I’m authenticating, it’s normal HTTP basic authentication. That’s all it is. You’re going to get back a response that going to give you your username. Don’t worry, this doesn’t exist. This key, this was just a test, I’ve deleted it since, so I’ve no problem showing the password up there. And that’s – that’s your key. Those are it, right. You’re good.

[Jack] Part two. You want to create the entitlement. Now that you have your key, you need to scope that key for access into some project. Pretty much the same in terms of the actual authentication, you also need to pass in your – your credentials. I got that basic cat encoded, you know, key off. In addition to that, you’re going to pass some json data. And you want to say rewrite or read or write. You want to give it some access. And then finally you’re going to say what key you want to apply that access to.

[Jack] You’ll notice in the URL, I’m actually down all the way to that hello Bintray. So packages, Yahoo Maven, hello Bintray. That hello Bintray part, that’s my package of Bintray. So we have some other, you know, customer or group within Yahoo that has an open source project on GitHub. What they do is we create that package for them, we — you know, hello Bintray — and then we go ahead and create these keys and scope them directly into that package. They have the download key. That username and the password in the prior step. We create this scope part that entitles that download key to go ahead and access with rewrite. And of course you’re going to get back, you know, your 201 response created, everything’s good to go.

[Jack] So, here’s the fun part. We got to make this work from CI. At Yahoo we have internal CI systems that we’ve been building and using in open source so the common use case is to use Travis. So Travis CI is pretty well known, it’s popular. What I’m going to walk through is kind of how to set that up with Maven. You can do all this stuff with Gradle just as easily. Maven’s just kind of the choice we have here.

[Jack] And then once it’s set up, there’s a couple of decisions you got to make. One is: Do you want to control your deployment process? So commits are coming into your GitHub repo. And you make a decision at some point in time to make a release. At that point you can, you know, use Maven release plugin. You run that command, it tags the code, it goes out to Travis, Travis sees that tag, Travis uses the credentials that we give it from our download key and it pushes that package out to Bintray. It’s kind of how this whole system works, and you’ll see in future slides.

[Jack] So one thing that you’re going to want to do is obviously give Travis CI your credentials. How do you do that? There’s — there’s two ways. This is a screenshot of Travis CI’s user interface. So if you go inside you can actually set Bintray user and Bintray API key. And it actually encrypts those at runtime. So now when your project runs, you’ll actually have environment variables that you can reference from Maven to get your user API.

[Jack] The second way to do it is to use the command line, the Travis CI command line tool. And you give it a command, you tell it what your username and your password is and it actually encrypts those and stores inside your Travis yaml file. And so you end up with those two little secure lines here at the bottom, they’re highlighted in yellow, it’s a little difficult to see on the slides. I recommend going the first way but, you know, you do know there are two ways of doing this.

[Jack] Another component is Maven needs to know what to do. Right. So, pretty easy, pretty straight forward. Now that you have those environment variables and they’re present at build time, you go ahead and reference them from your settings XML. So you create a settings dot XML file and you pass it into Maven. Env dot, that’s referencing the environment variable that’s available at build time from Travis. And it’s the same as what we’ve encrypted here prior. So Bintray user, Bintray API key, and Bintray user, Bintray API key right in the settings. And so what that means is your username and password is not exposed to anywhere. It’s secret, tucked away inside of Travis and of course we’ve already done the plumbing backend for Bintray so that key actually accesses the package we want in Bintray. And then the next thing is, you need to point your distribution. You got to point Maven to where the distribution’s at. I used a example project that we had at Yahoo called Omid. And the full, you know, it’s API dot Bintray dot com Maven Yahoo Maven Omid. Pretty straight forward.

[Jack] Now comes the fun part. You want to make a release. So you have a team of, you know, developers and they’ve gone through and they’ve accepted a lot of pull requests and they’ve gone back and forth. Maybe they’re working toward a milestone, whatever have you. Now they make the decision, hey I want to do a release. Well, what I’ve illustrated here so far is pretty good. Travis isn’t going to do a release. It doesn’t know that. All it knows is that it’s building. Right. Basic Maven, Maven clean install, basic build. You can tell Travis to do a Maven prepare release, Maven prepare and it will go ahead and release that into Bintray. You can do that, but then that means that you’re doing releases on every commit. You probably don’t want that.

[Jack] So what you do is you actually have your developer actually just run the Maven release plugin on their checkout. So one of the maintainers of the project has the code on their laptop, on a developer box, they come to a decision to make a new release, at that point in time, you can use the Maven release plugin. It’ll tag it, it’ll bump the version, and it will actually increment you to the next development cycle. So major-minor patch, that patch version will get bumped. And it sets you up nice and neat.

[Jack] When it gets to Travis, you can put settings in your — in your Travis file to say, hey do a Maven deploy whenever you see us get tagged. And by the way, use the settings file when you run that deploy. And what will happen is Travis will see that and it will grab whatever — whatever’s been built from that tag and it will go ahead and send that out to Bintray. That’s your release. That’s one way to do it. There’s a lot of ways to do this.

[Jack] What I have highlighted here is actually that command that Maven release prepare being run from Travis. That’s one way, there’s another way you can do it. Otherwise you would do release prepare, as a normal developer, right from your laptop and then, again, you configure Travis to match against that tag number. So you can actually give it a regex. There’s a lot you can do on Travis. A lot of ways to skin the cat.

[Jack] This is kind of what you end up with. This is a big, you know, boxes and lines, sort of picture. At the first, at the beginning, you got your developer. He’s going to go and run Maven release prepare from his laptop. Those commits are going to make their way out to Git. One of them is going to have that version number tagged in it. And what you tell Travis on the other end is, hey look, build it and after it’s successful, if the tag matches this regex, go ahead and do a Maven deploy. And then after it runs Maven deploy, all your binaries end up into Bintray on the project that you’ve done. And it knows to use your credentials, you’ve configured it already. And all of that’s scoped right inside of the package so you don’t have teams trying, you know, there’s no malicious sort of activity. Right? It would be very difficult for them to try and pull that off. On the consumer end of that, your consumer just configures their Pom to use it […].

[Jack] There’s a few parting thoughts here. One is that our use is a little unorthodox. We’re using enterprise features for open source. I know that a lot of open source organization are not going to have download keys or, they’re called access keys now, the names have changed. I told you when Yahoo first started using this, they were called download keys. We were using it from, you know, when they first released this feature. It was months and months ago. They’re not called access keys. Same thing.

[Jack] If you don’t have enterprise, you can still do this. There’s just a couple things you have to think about. One is you can create a dedicated account inside of Bintray and then you can just use that account credentials in your project and add that account into your Bintray organization. And you’re set up — you’re good to go. If it’s just you working on your open source there’s no reason why you can’t just put your own credentials in there. Encrypt it all with Travis the same way, everything’s kind of the same, and you’re all set up.

[Jack] What using download keys really buys you is this headless approach to trying to do your read-write access with your repos, which Travis needs in order to publish. The nice thing is you can fully automate this or can partially automate it. Like at any point in this — in that diagram that I’ve showed, you can make a decision to, you know, continue automation further or you can make putting human in the loop at that point.

[John] Bintray allows you to actually push pub — publish, quote unquote, packages or, you know, binaries into your Bintray package, but not yet expose them to the public. So as an option, if you wanted to, you can actually push in artifacts and only when you’re ready to do a release decide what ones you want to actually, you know, release or discard. So a lot of options here.

[John] What I – what I suggest is, I see a lot of open source projects, some are using, they’re actually using GitHub to store their artifacts in. I think Bintray is phenomenal because it’s really easy to use and on top of that, you can link to JCenter and that’s like your easy way into connecting up to everybody else so that there’s no special configuration that your consumer has to supply in their Pom xml if they’re using Maven or anything else. So this is actually a nice little system. I think it’s superior to currently what’s available out there, and you know, you can automate a lot of this end to end. So.

[Allen] All right. Thanks. Appreciate it.