Using DockerHub and Artifactory to Secure Open Source Packages

 Although most of my time has been in memory corruption of user space and kernel space targets, both in the defensive side of that, as well as on the vulnerability research kind of exploit development side of things. I’ve also done some time doing secure architecture and design of systems both on the review side and on the coming up with secure designs.

 Speaking with me today is Danish Nagarajan, he’s a partner at IBM Security Services, he’s got over 20 years in cybersecurity, and leads a big chunk of IBM Security Services. His focus areas are in application security, in DevSecOps transformations, and in cybersecurity services. And it’s also kind of cool, he’s out of the UK in London, so we get to work together, across the ocean there.

 Over to you Danish. Oh, sorry, we’re going to do the agenda first, then over to Danish. So first thing we’re going to do is talk about open source and application security kind of the state of the industry. So that’s where Danish is going to be providing his insight as one of the leaders in industry, from the DevSecOps space, from the application security space, talking about a bunch of different statistics around open source and application security there, and the implications that has on businesses. Then we’re going to talk about how we as RunSafe can deliver hardened open source software to help mitigate some of the open source security concerns. And the majority of the conversation is actually going to be talking about the challenges and solutions to harden to open source delivery, and that’s where artifactory comes in, as to how we’ve taken those problems and solve them using artifactory SAS offering. So I’ll hand it over to Danish to talk about open source and application security.

 Thanks, Shane. Thanks for the introduction. And we’ll go to the next slide. So I’m going to spend a few minutes talking about open source software, and the impact of open source software and also, in general, also the overall impact of application security.

 One of the things that a lot of people don’t realize is the element of pervasiveness of open source software in enterprise today. Some of the stats that that’s out here on this slide are quite eye opening in this context. Over 44% of the organization are planning to spend significant amount of increase in the use of open source software in the next couple of years. While they’ve started using open source, we’ve seen that this trend is increasing not just in, you know, small and medium sized companies. We’ve also seen large enterprises and global organizations serving consuming open source, and even regulated industries have started using open source software for their enterprise applications and business application platforms.

 One of the interesting thing in this as part of the open sources that over 60% of the organization believe that open source software is actually key for building their critical platforms and applications. So the open source software programs that most of these organizations use today are important and critical for, you know, supporting their key productivity and critical applications. So that just reflects the importance of open source that being used today in most of these organizations.

 While the increase of open source is increasing significantly, we’ve also seen that over 49% of organizations struggle to identify vulnerabilities in open source components. Open Source has a certain unique characteristics and that makes it slightly more challenging in regards to the security elements and identifying vulnerabilities. However, this is increasing significantly and a large number of clients today struggle to identify vulnerabilities and actually also remediate its vulnerabilities. So while the use of open source is increasing, there is the general state of application security, that also is a bit of a concern that we need to look at.

 Application security has become the most popular vector for cyber security threats today, a lot of these reason breaches that we’ve seen is basically exploiting application security vulnerabilities. Application, you know, layer and vectors presents to the attacker a lot more richer environment, and also and a lot more rich and attract factor for them to, you know, breach and attack and actually exfiltrate data, or bring down or break into a network.

 It’s no more, you know, challenging to break into a enterprise perimeter network application layer presents a lot more richer, and also more vulnerable and attractive threat vector space. Aside from, you know, the attacker increase over application security, there are also the over 63,000 reports of security incidents and the exploitation of enterprise applications. This is the current state on you know, the application landscape in a typical organization. And more importantly, one of the key stats aside from the new boundaries, is over a quarter of these vulnerabilities are not remediated before they go live. So these applications are running on production, with vulnerabilities that are not being patched or not being remediated and effectively being susceptible to attack and a breach of application software.

 As much as the increase in the vulnerability focus on the application there, we see that there’s going to be a huge year on year increase on the focus on open source software. As much as enterprises have started using open source software, it’s a very well known, you know, information for the threat actors that this is an area of space that they can focus on and using that, you know, a platform to break into applications and, you know, create a security bridge. One of the very well known open source vulnerabilities that impacted businesses was the Heartbleed, one which had impacted over 500,000 websites. So what’s the solution? We’ve seen the challenges with application security space, we also noticed the… and the problem with some pervasive names are open source.

 We felt from an IBM Security perspective that DevSecOps is an opportunity that can change this equation. With DevSecOps, you know, security being integrated into DevOps, organization can look to enforce security early in the development lifecycle. Security requirements, known security vulnerabilities can be identified and remediated before the software is released in production and also it can be tested during the process.

 While there are known vulnerabilities that are identified one of the ways of remediating security is to provide enough security capabilities for runtime application components, to make sure that the application when they are running, they are protected against known threats and vulnerabilities and the security threats are mitigated, and the risks are mitigated and the portion of the application doesn’t change.

 Once application that deployed using your dev sec Ops, you could also monitor the deployed software to make sure that the applications when they run, the security threats are identified and also mitigated on an ongoing basis. One of the key elements of using dev sec Ops, one of the key advantages of using DevSecOps is what we what we’ve seen is over 50% of application security is can be addressed by doing a secure by design principle within the DevSecOps mechanism.

 Excellent. So IBM security… Next slide please. IBM security has got a point of view on how DevSecOps can be integrated and automated in a continuous security perspective, there are some key components that needs to be considered for a very successful integration of security and DevOps practices, but fundamentally it forms of two key areas.

 Security in two key areas. One is the key elements required to develop securely within the DevOps lifecycle, but also monitoring and actually learning and having a learning cycle when the application is running on production. So there’s a very important element in this, again, is a continuous process. It’s not a one time fix. And we feel that using DevSecOps capability, you could address some of the challenges that we discussed today. And that’s, you know, pretty much a quick, you know, intro for myself on, you know, presenting a state of application security, and about the challenges of open source software.

 Thanks, Danish. And now before we get into why we harden and how we harden open source solutions, we’ll do a quick demo here where we’re going to walk through one of these exploits that Danish was just talking about, and see how those can impact the existing open source solutions that you’re running today, as well as the proprietary solutions in your environment. So what we have here is a demo environment in a browser, it’s being run on catacota. And so we’re going to walk through building a vulnerable application, exploiting it and then we’ll talk about the protections that RunSafe provides to open source solutions, show you that the exploit doesn’t work and then we’ll get into discussions around how we can deliver hardened open source solutions using Artifactory. So we’ve got three little windows here, on the left is kind of step by step text, we’re not going to worry about that, I’m not going to read that to you. And then at the top, we have an editor here, and right now we’ve got a Docker file that is going to build our application, we’re going to install, build essentials to our compiler, we’re going to install Python to actually do the exploit and then we’ve got some source code for a vulnerable application.

 We’re going to build it and then we’re going to be able to run it. So here we’re going to run this vulnerable application called verify password, it takes in a password and in this case, the password is super secure and it prints out a secret message for you, which is zero days or number. So now that we’ve gotten that we’re going to go through and we’ll quickly look at the exploit script here.

 This is in Python, and it uses some local stuff here to make a very reliable and robust exploit. So now we’re going to run the same program with our exploit. So we can see here that instead of printing zero days or number, we printed an error message that said, hey, this password is incorrect but then we ended up in a shell environment. So obviously, with our shell environments, we’re going to be at whatever level that program was running and we can do things like OS, we can modify files, we can pull files off the system, etc. So we’re going to exit out of this now. And we’re going to quickly modify the install here and we’re just going to have a couple changes, we’re going to add in a new stage to copy some files from, we’re going to copy those files in and then we’re going to add a script to build this securely. And now we’re going to rebuild it, just like we did before. And we’ll see that when we go to run the program normally everything is going to work just the same. And when we do this within the exploit environment, then we’re going to see that the exploit fails. So real quickly, we can see the unprotected version is still functional and we can see that the exploit still works. And now when we run the protected one, we’re going to see… and for some reason that didn’t go large size, there we go. So we see when we run it normally it’s going to work. And this is key here, when we talk about hardening open sources, what we don’t want to do is have customers users take that open source and it function very differently, or differently at all, while their application is running.

 But now we do the exploit, we see the same invalid password message, but we see a segfault, the program crashes so instead of the exploit succeeding and the attacker gaining control of the system, they actually don’t get control.

 So where do our products fit? You know, as we talk about securing open source solutions, you know, there’s a lot of places we get open source, we get open source software as source in some cases, people are using that during their build process as software dependencies.

 There’s some really interesting supply chain security problems there that you can look at and dig into and really kind of solve. And that’s where, you know, some of the things like JFrog Xray product can help you find those vulnerabilities in existing open source software. And then we also set the build time we can provide these protections.

 We’re going to be talking a little bit more on the repo side today, which is where we host repositories of open source solutions that are already hardened. And then we’re going to talk a little bit later about a new capability that we call Alkemists square that’s more on the monitoring side, but what’s going to be really cool about Flare is that it’s going to be able to integrate with your test environments and with Artifactory, to provide you some new insights that you may have never had before. So let’s get into some of the challenges in providing open source solutions to Artifactory or via Artifactory. So the first challenge is keeping software updated.

 And really, you know, this is problematic because Microsoft has Patch Tuesday, you know, where everybody knows on Tuesday, first Tuesday of the month, I think it is, there’s going to be patches that come out, I’ve got to go update my systems. But for open source solutions, or software, there’s not really a standard patch day, a standard day for new versions to be released. And so those schedules are sporadic.

 There’s no real web hooks out there. You know, so for people that are web developers and use the web hooks telling you about everything that’s changing, you don’t really have that in the open source space. So we need to be able to make sure that when we provide open source solutions, that the software is always updated, and that our customers are getting the latest and greatest, so they have the newest features as well as the newest security fixes. So once a day, we have a git lab CI pipeline that we run that rebuilds all the software in our offering, whether that’s for Docker, for Deb’s, for RPMs, we’ll probably soon have some APKs. And so we do all that daily to build software, but how do we make sure we’re getting the latest versions? Well, we do a couple different things.

 For Docker images, we actually pull the latest releases and all the latest tags from Docker Hub, and then use the GitHub repositories that back that to pull the latest Docker files, pull all that together, and then build the latest Docker image and push that to Artifactory to ship from there.

 For our devs and RPMs offering, we build the latest and greatest source RPMs and source tabs and push those to Artifactory.

 The next big challenge is distribution. So with Alkemist, we deliver our hardened open source as well as our main product offerings through… we offer them as APKs for Alpine, RPMs for CentOS, Fedora, Red Hat, devs for Debian and Ubuntu images, as well as Docker. And so maintaining all of that infrastructure, being able to constantly make sure that the infrastructure is providing the latest and greatest versions can be really challenging. So Artifactory was a natural fit for that, and then Artifactory software as a service offering was a really great fit for us as well, because we want to make sure that we always have the latest and greatest versions of artifactory and we don’t want to have to worry about you know, CDNs and worrying about how we’re going to make sure that our software is… our delivery repositories are secure.

 We can leverage Artifactory, SAS offerings for that. And then the other thing is, as we ship our hardened open source, we have dependencies on some of our main packages. So even if we were to run our own repositories, we have to constantly worry about the problem of, well, I need this package in 15 different repositories or 20 different repositories and so we would be doing a lot of work to keep duplicate copies of those files everywhere, we’d have huge, you know, ballooning of our disk storage usage. And so we do a mix of virtual and local repositories, we actually have 34 repositories total.

 That number changes and grows a little bit as we add more packages and more offerings. And so we use a mix of local and virtual repositories to allow us to concretely say, hey, here’s all the Apache packages, here’s all the node packages, here’s all the Redis packages. But then when we want to expose that to a customer, we just give them the virtual repository. So they always have the latest and greatest version of that. But then we can manage the versions that are available in local repositories a little easier. And then we also leverage those local repositories for development and testing. So that we’re not having to pull down software list for everything that we offer, we can just focus on one thing that we want to focus on for that development or testing effort.

 The next challenge that we have is discoverability. So if you go and look at Dabian, RPM package repositories, even Docker itself, there’s a discoverability problem, right? Docker Hub has a great search feature. But, you know, there’s limitations to that, when we’re looking at how do we provide a bunch of different software packages to people. And then with, you know, devs, and RPMs, you end up with kind of an HTTP listing, right? And it’s just the file structure, and you got to click all the way through and make sure you’re in the right directory.

 We didn’t want people to do that, we wanted to be able to show the latest and greatest versions with all the tags that we have. And right now through our current repo, we have eight different open source packages, and about 1098 Docker images, with all the different tags and all the different versions that we support across those packages. So we needed a way to make all of that discoverable, so that people that were looking for a specific tag release could see very quickly whether or not we supported that tag. And so what we have here on the right is kind of a beta version of our repo website that shows you all the different tags.

 You can see that we have the all different releases broken out by the image hash, and then all the tags that match that hash there. And so we leveraged Artifactory REST API to be able to dynamically generate this as users are browsing the website.

 So obviously, if you’re using App or Yum to install your packages, that’s going to be kind of invisible to you and you’re just going to install a package and call it a day but for Docker, we wanted to make that really visible. And so the rest APIs for listing all the tags, and all the releases in a given repository, were super easy for us to leverage into a front end that dynamically updates and is always current. So I mentioned earlier that we have some features that are coming soon. And so let’s just get into that.

 We have Alkemist: Flare as a product module that allows you to do monitoring of software process crashes as they occur. And so how does that fit into Artifactory?

 You know, how does this fit into this talk here? Well, what we found is that in open source solutions, there’s not a great way to monitor when those solutions crash and for people that are doing DevSecOps and are building software, deploying it, testing it all continuously, there’s a challenge of what happens when a test fails,

 Well, the first thing you have to do is go do root cause analysis on that testing failure. And so with JFrog Artifactory, trying to be the center of the kind of DevSecOps software development, production, release, testing lifecycle, we can leverage Flare data to post metadata about different binaries that are crashing. So when you go to run your tests, you’re going to see failures. And then you can leverage the metadata that we can provide the Artifactory to get an understanding about what the actual crash was, how it crashed, why it crashed, etc. And so not only now is Artifactory going to be able to you track your builds in your pipelines, in your testing, but also give you some data at the process crash time about what’s going wrong in that process that is going to help you leverage your engineering time more efficiently. So that’s coming soon, Alkemist: Flare is available today, but we don’t yet have the JFrog Artifactory integration, but that’ll be coming soon. So then you’ll be able to get all of that great functionality. So real quick, we’ll show you kind of what information would be available as part of that monitoring. And so now we’re going to go into my development environment. And we’re going to run a quick smoke test.

 Here, we watch a package, and it’s going to crash. And we’re going to see application crash data show up in Splunk. And we’re going to run this again real quick. And we see in real time where there was no data before, we’re going to see in real time, that there’s now crash data here. So we’re going to look at the signal event here and we can see that there’s a command run called signal test, we can get information about the arguments. So if you’ve got dynamic environments, where your tests are taking in a bunch of different arguments, you’ll be able to leverage this to see hey, this specific crash test… this specific test crash, sorry, these were the arguments that were passed to it, this was the environment variables that were used for that. And then you can also see things like the processor register state.

 Now normally, if you’re looking at this in GDB or in some other debugger, you’re going to see these values in text. Splunk provides those currently just as unsigned integers. But you can convert that pretty easily to see kind of the context, what was the instruction pointer, what was the processor doing. And then you can also get information about the host, you know, hostname, interfaces, etc. And then the signal info as well. So we can look at, as you’re trying to debug the root cause of a test failure, all this information is super useful to helping you diagnose that. So just want to say thank you to everyone for coming to our talk, and taking the time to learn about how RunSafe leverages Artifactory to secure open source solutions. And now I’m going to hand it over to Danish for any final comments.

 Thanks, Shane, for this opportunity to present with RunSafe.

 It’s a very interesting demo and presentation.

 Hopefully, people have learned something new on how to handle you know, security for open source.

 Thank you.