Understanding and tracking the impact of your ever-changing k8s deployments

Peter Bosch
Distinguished Engineer
As developers we’re not always fully aware of security implications stipulated from changes to our code whether these are done in the CI, CD or an artifact database.
It is always challenging to predict the impact of a changed 3rd party library, a security context or an RBAC permission, accessing a different network to the same resource or even using an API in a different way than we used to.
Understanding the impact immediately and being able to make a change without disrupting the pipeline is therefore an important requirement. This session will present best practices to cope with these day to day changes and will propose a set of tools to address them cohesively.

Video transcript

Hello, everybody, I am Peter Bosch. I am a Distinguished Engineer within Cisco’s ET&A, emerging technologies in incubation. And now for the next half hour or so I’m going to share some words on application security, what Cisco is doing in this space, why it’s important, making sure that and sharing with you in the industry at large why application security is such an important area for all of us to work in.

 First a word on who we are, we are the emerging technologies and incubation team. The things that we do within our team are the things that Cisco traditionally does not do, things that we should do, and things where we think that the future of the technology will bring us. We’re doing this together with the Corporate Strategy Team and the strategic alliances team. So that between the three groups, we actually cover quite a large area with respect to the future space, where organizations such as ourselves need to be in, but also for the larger ecosystem.

 The things that we look at are depicted on this diagram. So security, cloud, AI, ml, future of work are areas that we actively look at.

 Future of work, we have the WebEx team WebEx solutions, what else can we do? AI, ml, clearly, it’s going to be needed for understanding relations that are not obvious clouds. It’s kind of a given and security, that’s the one that I’m talking about here, and the way we’ve set it up, within ET&I is that we have a design partner program. So inside ET&I, we don’t build products, we build what we call Minimum Viable products, things that you can buy from Cisco, things that identify new market trends, build new solutions to meet our customers business needs to make sure that you actually build something that the customer wants, rather than saying this will be built., are you interested? And then specifically, in this space, we look at two early stage ventures.

 Application networking, what kind of networking stack do I need to glue together a wildly distributed application going between on prem and off prem solutions? And application security, which is what I’m going to be talking about here. And the benefits to being a design partner really is that as a design partner, you work with us to really scope out the process, user journeys, what do you want to work with, what kind of features to do need to be added to to a product, jointly collaborate on our roadmap but also jointly innovate.

 Things that we miss. And if we do that jointly with you can you actually give us a much better insight in what we should be building for our partners.

 Having said that, let’s switch gears into application security. So first, let us define what it is, why is application security important to begin with. And the way we’re thinking about this is that it really falls into two large categories.

 Application networking, protect people, in essence, and protecting applications and data protect those things that run in the cloud. So on the left hand side, let’s assume that we have a fictitious company called Acme Inc.

 Acme, Inc is really about making sure that… Acme had built up their network so that all of the applications that are part of Acme are on prem, there’s a picket fence around it, a firewall and IDs and IPS.

 We have VPN solutions, sassy solutions, whatever it is. And it gives you the illusion that once you connect into that network, you’re part of the secured area and once you’re inside that secured area, you can do whatever you want.

 That’s basically a model whereby you really go to on prem solutions. The new way of building applications is that you have an application that runs in the cloud.

 I have an application, I decompose it into its constituent parts. Acme.com is an application now being run by tens, if not hundreds of microservices all communicating with each other, running on a public cloud. There is no picket fence, there is no IDs, there is no IPS. And it’s really making sure that that bigger fence that used to be there is not there, which means that the application needs to fend for itself. So the thing that we’re looking into on the right hand side protection, protecting applications and data is about the techniques that you need to deploy to make sure that the application remains secure even though You don’t control the underlying network, you don’t control the underlying cloud system, you don’t control the firewall IDs, IPS, because they’re not there anymore. And then specifically, what we’re looking at is our Kubernetes environments, cloud native application security, in our mind is the place to be.

 We talked to a lot of our customers and you do see that notion where Kubernetes is more than just a technical revolution.

 At the heart of this, of course, Kubernetes is managing a bunch of processes, Unix processes, running on Linux with namespaces and whatnot but Kubernetes is more than that.

 Kubernetes is also the business revolution, because it does allow enterprises such as Acme that I was citing to quickly build up that the infrastructure needed for for hosting these born into Cloud type of companies. And what we are noticing is that even when we’re switching to our existing customers, some of whom have built out quite substantial on prem solutions, they’re switching into Kubernetes type of deployments as well, they are becoming the newborn in the cloud companies by themselves, they are simply leaving the on prem solution as is, they’re not going for lift and shift and they’re re implementing them inside the cloud as a born in the cloud company. And the key personas in these companies are really the cloud architects, cloud sec Ops, app sec engineering, and of course, the developer, the developer is at the heart of this. And these are not your traditional stakeholders, as we’ve seen them before.

 Traditional stakeholders, specifically for application security is of course IT.

 Here, it’s different, especially with that movement shift left, what we’re seeing is that many more decisions have now been made directly into the development community, by the developers themselves, they already decide on the operational side of things, they now also need to decide on the security side of things. So what is then the underlying problem that we want to solve?

 The underlying problem that we want to solve is really listed here. As these applications move to the clouds, they have different computer security requirements.

 Like I said, before, your monolithic application is now split into very many micro services, these micro services communicated up the wazoo with each other with external services, with internal services, we have serverless functions, we get infrastructure as code solutions out there and it’s a completely different way of building your application, relative to before.

 But then, with this movement to shift left, whereby the developer gets the responsibility to not only do the operational side of things, but also the security side of things, you start to realize that developers are in a tough spot, a developer, with the introduction of ci CD chains, has been given the tools to do that operational side of things.

 But from the security side of things, those tools are not there. Yes, DevOps, prod, a CI CD, continuous integration and continuous development… Continuous deployment, sorry. But that sec part is not there.

 There’s no such thing as ci CD plus continuous security, or an Integrated Security toolset that goes directly into the IDE.

 Really, what we’re after here is that we should be building a toolset that combines IDE, ci CD, and continuous security into one package, because otherwise devsecops is a tall order for developers. And then developers lack the security tools, like I just said before, and I call it a knowledge gap. Often the developer is being pushed to deliver as many features as they can as quickly as possible. Security is always an afterthought. And moreover, making sure that you build secure applications is a challenging task. So making sure that you help that developer pointing out common mistakes, often made mistakes, risky solutions, and pointing out where the vulnerabilities are is an important task for such a toolset. So then, first generation cloud application security really focused on deployment in front of vulnerability analysis, simple attacks, really.

 Really what we’re after here is that we should be going for more complex solutions, really that holistic view ID, ci CD, and continuous security in one package so that the developer really has the entire application from an operational side, from a development side and a security side in their fingertips. So let me introduce this by kind of an animated example. So this diagram presents on the left hand side and the right hand side a before and after. So let’s take a look at these three personas that are there.

 There’s the developer, there’s the application security persona or the secops persona in the middle and at the bottom, there’s the CISO.

 The developer’s task is to quickly cobble together a solution for their enterprise, they’re given a task saying implement feature X, Y, and Z for company Acme, and that developer will use whatever features are available to do this. So depicted in this diagram are of course, API services, there are serverless functions, there are libraries, SDKs and of course, other assets that they can use for building an application. And with that complex tool sets that has been given to the developer, basically, that developer will just sit down, build that entire application, and throw it over the fence towards app sec or SecOps, they commit the application and then sec ops and app sec really doesn’t have much to say about that deployment. And I’ve had conversations inside and outside our own organizations where sec ops and app sec really doesn’t know very much about what the application does.

 The application may have compromised image libraries, images layers, libraries, configurations, API services, serverless functions and whatnot and really, app sec doesn’t really know where these vulnerabilities are. And I’ve had heard anecdotal stories where app sec is dragging all of the developers into a conference room every 3 months and the developer needs to present what their external dependencies are. And you can be certain of two things.

  1. the developer doesn’t want to do this, so therefore, that presentation is going to be done as quickly as possible so that the developer came to their work, which means that there are going to be inaccuracies in that presentation and as soon as the developer goes back to work, they’ll change the solution anyway so that means that review is inaccurate to… only at that minute instance where it was shared with app sec and if I’m the CISO, the the person at the bottom of the diagram, I would be pulling out my hair, I would not know what to say to my customers.

 I would not know for a CISO to say to acme.com customers where the vulnerabilities are in these applications. And really the CISO now has the task of only when an application has been broken into to retroactively go back to these customers tell these customers that acme.com is terribly sorry, but their data leaked out. It doesn’t look well. And the solution is quite simple, which is what I’m presenting on the right hand side.

 The right hand side, we have the same three personas and rather than simply saying, you know what, we throw this over the fence by the developer, app sec and the CISO beforehand set guardrails. And these guardrails include things like as a developer, you can only use the following image layers, the following tested image layers, the following configurations, the following tool set to make sure that these assets that you stick into your application are secure, the following serverless functions and the following internal and external API service. And moreover, what we then do is, once we know all of these assets that developers can use, we’re going to test them beforehand.

 We already know beforehand what the vulnerabilities are going to be in terms of telemetry in terms of testing and whatnot, which means that if the developer is ready with their application, and presents this to app sec, app sec doesn’t have much to do because the application is within the guardrails, and you can simply roll that out.

 App sec is happy because they know exactly which artifacts went into what applications, which services were used, what images, what image layers, what configurations were used, and they can quickly sign off on that application.

 As a CISO, this is great too, because I now have that overview of all of the dependencies in my applications and I know exactly where I’m vulnerable and how I’m vulnerable, but as a sneak CEO, back then already said a vulnerability is only a vulnerability once it becomes a known vulnerability.

 It is important to constantly keep track of all these assets that go into your application, and by doing so, we know beforehand, or when a vulnerability is found, which applications are going to be impacted. And we use that knowledge to go back to app sec, or the developer.

 We will notify app sec that a new vulnerability is found in one of the applications and they need to fix it, either by way of remediating it, or by creating work items, work tickets in JIRA, or something of that sorts, to have the developer fix these problems. And that’s why on this right hand side, this diagram is really bi directional, it shows that applications move back and forth, you really want to have the continuous security, that CS part, now under the control. And as a CISO, even if things have been broken into now I can actually make sure that I can articulate this to my customers proactively.

 It’s a completely different story, when as a CISO, I’m engaging with my customers of acme.com, to tell them that we found a vulnerability, nothing’s leaked out yet and this is how we’re going to report it.

 That poor CISO on the left hand side, in contrast, will have to say, this is how your data leaked out.

 On the right hand side, this is how we’re going to remediate it before it leaks out. So it’s really about combining image metadata function and API scoring into development and runtime cycle. It’s really kind of like an insurance policy for devsecops.

 What I’d like to do next, is actually get into a quick demo. And what I’ll be showing in that quick demo is depicted on this diagram here.

 We picked up the Google hipster store and it shows that on the left hand side, and some poor programmer, in this particular case, I programmed it myself extended the functionality of the hipster store with some functionality to send an SMS whenever an item in the hipster store was bought. And as you can see in the yellow here, that poor programmer included all of the parts that are part of the functionality of the service and the tokens that were used directly into the code. So now we know that the accounts happens to be AC31C8, etc, etc, etc. And yes, it’s tied to some programmer’s…

 It used to be tied, let me put it that way, to some programmer’s credit card, and it has an authentication token. And then you engage with an external service, in this particular case Twilio, to send that SMS. And there’s another mistake there as well. Because that developer included the entire transaction for this.

 The entire transaction includes things like, we bought something on the online boutique store, this was the credit card number used, this was the CCV and this was the expiration date. Not how you want to do this, because that means that you’re now sharing PLL data to an external service and you don’t know what the ramifications are. What we do with our solution, secure CN and secure cloud native, is we make this visible through that through that console.

 Secure CN helps you start to manage that entire application, be that from a container perspective, an image perspective, an image layer perspective, an API perspective, the future is serverless function perspective, a token management perspective for these external services, for internal services and whatnot.

 That’s where we’re going with all of this. So really, that secure CN becomes a one stop shop for managing your application.

 Now, let me stop here, this recording. I’ll switch to a demonstration and I’ll be right back.

 Thank you.

Release Fast Or Die