So Happy Together – Making Life Better with a Real DevSecOps Culture

 I’ve been a hacker all my life. Currently, I am a Business Information Security Officer for S&P Global ratings. Now, if you don’t know what a business information security officer is, you want to hear about how that’s different from a chief information security officer, check out this blog I put together it’ll tell you all sorts of things about the difference between those two leadership roles.

 Now, you might be asking, why is a security leader here talking about DevSecOps at a conference like SwampUP? Well, I spent 10 years as a developer and then the last 15 of my career have been spent in various roles, I guess, within cybersecurity, including as a pen tester, but always really focused on application security. And I’ve worked with a number of organizations on how to develop DevSecOps practices, and certainly now with an S&P Global, we are going through our own DevSecOps and cloud transformation. So I’m going to share a lot of lessons learned from some of those different roles that I’ve had. And finally, I’m an author, blogger, podcaster.

 You can read more about that on my website, which I’ll share with you later on. But let’s start with a history lesson.

 You see, I got to thinking about this DevSecOps and more commonly just talked about as DevOps has existed for now 13 years. So I realized some of the people here probably weren’t even in the industry at the time when DevSecOps or DevOps got its start. So DevOps, if you didn’t know was really started by two gentlemen, Patrick Diwan and Andrew Schafer, they tried to connect at a conference, they wanted to talk about how do we alleviate the friction between our dev teams in our operations teams and break down those silos.

They didn’t end up connecting at the conference, but they got together later, they talked a whole lot, came up a bunch of different ideas, started to share it with the community and in 2008, or excuse me, 2009 rather, Patrick Diwan launched DevOps days and that’s the first time we heard that term DevOps. Now, DevOps got adopted by a lot of organizations really quickly, developers really liked it. But it wasn’t until three years later that security sort of realized, hey, we got to do something about this. And it was in 2012, at the RSA security conference in San Francisco, that Jean Kim and Josh Corman, who you see on the right here, had what is now kind of their their infamous talk about security in a DevOps world. And, you know, they kind of coined that term DevSecOps. So all this leads to when we think about DevOps, and we think about who it is that we are engaging with, the silos that we’re breaking down, there’s this interesting question about who’s responsible for security.

 Now, I used to work for a company called Snyka and in 2020, we did our state of open source security report. And one of the questions we asked our respondents was, who’s responsible for the security of your software and the security of your infrastructure in your organization.

 Now, what you see is 85% said developers are responsible for the security of software. Okay, that’s expected, but only 55% of security played a role in that and only 35% of operations did. And then you look at infrastructure, at least it evens off more but there’s still a lot of pressure being put on developers shoulders to secure all of these elements. And it’s an interesting dilemma, because we’re asking developers to do so much more these days than we ever did in the past and now we add on, hey, we want you to be responsible for everything.

 This doesn’t really jive with that idea of breaking down the silos and building true DevSecOps. So one of the big challenges we face in modern day development is this idea of just even knowing what’s in our software.

 Most of us leverage some form of open source in our development today. You can’t get away from it, right? You’re pulling packages from GitHub or wherever it is that you’re getting different functionality from the open source community. Now the problem is we don’t always know what’s involved in those packages. So one of the examples I love to use is this job application here. Now this Java application, it’s about 280 lines of code that this developer wrote.

 Okay, really small app. Now my days when you wrote 100,000 lines of code, that meant you had 100,000 line application, but we know it doesn’t work that way anymore, because as you can see here, this application defines eight dependencies. But each of those dependencies now in turn has its own sub dependencies. And so those eight dependencies turn into 89 sub dependencies, which takes our 280 lines of code up to 2.4 million, so just a slight increase. So how are developers supposed to even be able to keep track of this as we’re leveraging more and more open source and repeatable packages and libraries and things from that community? But it gets worse. I love this quote from Kelsey Hightower, I cite it all the time. So you want to roll your own application platform? Well, hey, you just need to know how to use all of these various technologies, you see Linux and Docker and Kubernetes and sto and Prometheus… and it just goes on and on and on. Right? And Isn’t this the world that you live in as developers today, when we think about our Ops, our SREs who are a part of this and all the expertise that they need to have.

 Well, it’s super complex world. But it gets worse. Because where do all these technologies live? Well, they’re all in that world of cloud native technology. Now, maybe you seen this chart before from the cloud native computing foundation.

 It’s mind numbing, right? Now, this version is I believe, about four or five months old when I took the screenshot, and you can already see how hard it is to read the names of the various tools and technologies that are on this list.

 You might recognize a lot of the logos, hopefully, you probably use a whole bunch of them. But now we’re asking our devs to be responsible for the security of this as well. And it’s just… that’s so much that we ask, so okay, but then we’ve got our security teams, our security teams are going to help us out, right?

 Well, they’ve got their own mess to deal with, essentially. I mean, look at this. They’re responsible for perimeter security, network security, and endpoint security, all those things that we hear about getting breached every day, then they can worry about application security, but they also got to think about data security. And of course, they got to make sure that we’ve got policies to manage all this, we need to make sure that we’re handling things on an operational basis.

 What are we doing with firewall rules? How are we configuring our cloud environments? All of these things fall to the security team. And so in their world is just as complex as it is for the devs, for the ops, the SREs.

 We’re all dealing with our own complex spaces. Sure, there’s lots of tools in security too, right? We have the same problem here we have in the cloud native world, look at all of these various technologies that we expect our security teams to be able to leverage. And in turn, if I’m saying that software developers are responsible for security, well, they’ve got to know all this too and understand all of these different vendors and the tools they offer and what niche space they fit into.

 It’s just, it’s unreal, the complexity that we throw at our modern data environments. Now, why does this happen? Because technology just keeps growing. And that’s a great thing. That’s what allows us to innovate and do so many cool creative new things with our applications. But in turn, it makes that threat landscape that we’re trying to deal with, and trying to manage so much more difficult. So how do we answer this problem?

 Well, it comes back to what Andrew Shafer and Patrick Dawbis we’re trying to do from the beginning. It’s that idea of breaking down silos, but how do we break down silos? It starts with culture. Now, how many of you when I say the word DevSecOps, or DevOps, if I asked you what is that all about, how many of you would start to think about tools and technologies?

 How you’re going to automate everything, you know, automate the world, that’s what everyone seems to think about when we talk about DevOps. But the reality is DevOps and therefore, DevSecOps are always meant to be a culture change, which means it’s not just about technology and tools, it’s about people, it’s about processes, it’s about governance.

 We need to provide the training. Yes, I did say governance, we’ll come back to that in a second.

 People, you need to provide training for the people, you need to provide processes that fit your paradigm and tie the tools and technologies together. People need to have those skills.

 And then I said that ugly word governance. Yeah, we need to be able to make sure that people are following the processes and the tooling, and everything that we put out there, we need to build a measure how effective that’s been. That’s where governance comes in. Every time I say governance, people think about audits and all these horrible questionnaires and things that they have to deal with.

When I’m talking about governance and DevSecOps, I’m just talking about the ability to measure the things that we do, and to make sure that we’re doing what we say we’re going to do, so that those measurements are accurate, because I can’t get better, I can’t get more mature in a DevSecOps culture, if I’m not measuring what it is that I’m doing, and making sure that my initiatives that I believe are right, are actually having the desired impact on my culture.

 So in this culture, what we’re seeking to do is break down those silos. This is where the idea of shared responsibility comes from. We have these silos of devs, and ops and security and typically, they each have their own priorities. And that’s where the problem begins with silos. So in DevSecOps culture, what we’re looking for is everybody is responsible for securing software, delivering it quickly, and making sure it’s stable when it’s deployed in our production environments. So that means security, yes, security, you’re responsible for making sure software gets delivered quickly, you’re responsible for making sure that it’s stable in the production environment.

 Devs, as we talked about before, you’re responsible for that stability, and the security of your code as well as getting it to production quickly. And Ops, you’re a part of this picture, too.

 This is what it’s all about, when we talk about DevSecOps, this is how we take all of that complexity and make it manageable, so that we’re all working together to make sure that that shared responsibility becomes a reality. Now, some of the challenges that we continue to have in our modern day pipelines in DevOps, when you think about DevOps, or DevSecOps, a lot of times people think about CI\CD, and we’ll get there. But let’s just think about the basics of a pipeline for a minute, we have these different phases of the pipeline, and we can lay them out linearly, we know that, you know, with the Infinity loop, yes, we continue to cycle through. But as we look at this security is doing this thing that they’ve always been doing.

We’ve been trying to push left. We’ve been talking about that for 20 years in security but what’s changed now are the conflicting motions, we’ve got developers pushing further and further right into the pipeline, with things like infrastructure as code and containers, all the things that we do, developers are defining the very infrastructure on which their software is going to run.

 So they’re pushing further and further towards the deploy side. And then we’ve got our ops teams, our SREs who… they’re pushing up the stack, because no longer can ops just be worried about bare metal servers and operating systems, they need to understand the code that is defining the infrastructure it’s going to run in our environments, most often cloud environments.

 Now, whoever we talked about in this yet, who kind of gets forgotten about?

 Well, that’s the business. The business has become more and more involved in a granular aspect within our pipelines than they ever have before.

 They’re part of the testing, they’re defining our user stories. They’re a part of every phase of this as well. And we want that, we need them to be a part of it. So they’re pushing down the stack, they’re getting more granular in terms of what they understand about the technology and how they are involved in building it.

 Now, so let’s talk about this idea of security in DevSecOps. I have been to I don’t even know how many, I can’t count talks given at conferences, about DevSecOps that were given by somebody who is in security. And so often, they do the traditional thing that we’ve done in security, we talk about this idea of gates.

 We want quality gates between each of the phases of the pipeline. So when we move out of the backlog, and we start to plan our sprint, we want to take this design time and build a big threat model, then we want to do static code analysis when we’re committing code. And then when we’re getting ready to test that code, and we’re moving towards deployment, well, then we want to do dynamic security testing.

 These don’t work in a modern DevOps paradigm, especially as we start to move into things like CI\CD.

 Gates break this model because gates threaten to stop us in each of these phases, and they threaten to push us backward. More on that in a minute. So what do we need to do from security perspective, we have to stop thinking about security as gates between our phases and instead, we have to look at how secure Yuri integrates into those phases.

 How do we do threat modeling, as part of the backlog? How do we bring software composition analysis so we can understand our open source exposures? How do we bring that into the coding and committing cycles? How do we make sure that static code analysis happens as part of the building and testing process? How do we make sure that container security is there when we deploy, and that monitoring and pentesting happen, not as gates to our ongoing monitoring, but they’re a part of our production environment.

 So when we think about this in a CI\CD world, we think about how to bring security and we start to build these gates, we think about this idea of build breaking.

 How many talks have you heard, how many videos or other things have you read about breaking the build when security tests fail? So what this looks like is we’ve got these development cycles, right?

 We’re coding and we’re committing, if we’re doing CI, that’s what we want, right? We want continuously, we’re just coding and committing. And when we’re ready, hey, we’re going to promote that we’re going to promote that up to an integration environment. And then we’re going to build that code, and we’re going to test it. And then when that’s ready to go, we probably have another test cycle, or a regression test, we get ready to deploy, we package it up and we deploy it.

 Well, in a traditional model of gates, when we do that security testing, what happens? We have a long feedback cycle, we push it off, and it goes runs in this code scanning tool and that tool comes back and says, oh my gosh, you have to fix vulnerabilities, we’re pushing you back to coding. And the same thing happens when we test those packages that are ready for deployment, oh, we found vulnerabilities, you got to go back and fix that in your code.

 This is what breaks DevSecOps, this is what breaks CI\CD. So when we want to get to a true CI\CD, we need to think about these things differently.

 When I think about testing an integration, I don’t want to have vulnerabilities that are so critical that I have to go back and fix those in a coding cycle, when I get to release, and I’m ready, and I’m testing that final package going through my regression test.

 Again, finding vulnerabilities at this stage have pushed me back all the way to the development process.

 This is what breaks the CI\CD.

 What we want to do is move away from that, we want these tests to identify vulnerabilities that we can simply add to the backlog. Now maybe add them as P ones, I hope you do, make them priority one, they’re going to get addressed in the next development cycle.

 Same thing, when we hit our regression test, we want those vulnerabilities to be of a nature that we can just add them to the backlog and as we move towards this model, we stop breaking builds with security, CI and CD happen because we’re not stopping the current flow in the pipeline, we’re just setting up the next run through to address the vulnerabilities that we discovered in this one. And as we get better at CI\CD, and we’re doing daily or even multi per day deploys, the risk of those vulnerabilities drops more and more and more. But how do I make sure that those vulnerabilities aren’t high risk that they force me to go fix them right away?

 Let’s talk about that. So in 2019, Circle CI and Puppet did their state of DevOps report. And one of the things they looked at were security practices. And they looked at specifically the frequency of how often we conduct those practices, versus how much of an impact they have on security posture. Now in the upper left, you can see the things that we do a lot.

 Static code analysis and penetration testing. But we see those don’t typically have a lot of impact on security posture.

 It’s down here in the lower right where we see collaboration, security, and ops, and dev teams working together on threat modeling, for instance, that collaboration is what helps us start to push further left and eliminate high severity vulnerabilities before they ever make it into our testing. Now I say threat modeling and you probably all think of things like this, big heavyweight process where you draw up DFD diagrams, and you use these different frameworks like stride and dread and you’re going to draw big attack maps, and you’re going to map it all to catback.

 Forget about all this, we need to think about threat modeling completely differently if we want to do it as part of our DevOps world. So how do we do that?

 I mentioned before, bringing threat modeling to the backlog. Imagine for a minute, instead of trying to threat model your entire system, you take in each individual user story and as that user story is being written, you just bring in basic thread information and make that a part of the user story. Identify the crucial assets that are critical to that particular user story, and then identify the threats not in terms of stride.

 I don’t care if it’s spoofing, tampering, repudiation, that’s meaningless. Let’s Talk about it in the terms at everybody from the business to the devs to the testers to production support can understand. Is there a threat of theft? Is there a threat of fraud? Is there a threat of stability or lack of service stability?

 These are the things that we want to document, now who writes your user stories? It’s the business people most often, right? So if they’re writing this, they already understand what the critical assets are. Is it PII, personally identifiable information?

 Data that we have to protect? Do we have trade secrets you have to protect? Do we have to protect the ability to deliver a critical service?

 Is that delivery the critical piece? So they can document this. This is how we bring them more into the process. So what does this look like and why is this important?

 Well, imagine I put this thread information in my user story. And now as I start to build out and plan my sprint, I take those user stories from the backlog as the developer or the archetype, I can look at that and I see those threats and I know what security requirements I need to define.

 Maybe I have and this is what we do at SMP, we have engineering standards that help us when we see specific types of threats, help us understand what those security requirements are, now I can document those. And then those flow into my building process, right, they’re security controls. So now my devs understand from the security requirements, these are the types of controls I need to build, and they build them. And this is just an inherent part of coding. And what happens next? Well, I have that threat information, I have those critical assets identified. So when it comes to my test cases, I can prioritize them in an automated sense, based on that information. And what about when we deploy?

 Well, when we deploy, now I have the information that tells me what it is that I need to be monitoring, and what I need to monitor for.

 I know what my critical assets are, and I know what threats they face. So this is where we start to make our life easier by having this culture where everybody is involved in this shared responsibility. And we’ve done it in this case, just by building threat information into our user story.

 But we need to go beyond. When we think about culture, culture comes from empathy. And this lack of empathy is one of the biggest challenges we have as we tried to deploy a DevSecOps culture and break down those silos. So how do I get my security people, my SREs to understand the world of the developer?

 How do I get my developers to understand their world as well? How do I make sure everybody that’s involved in this shared responsibility, understands and has appreciation for how their initiatives, their practices, impact the others?

 Start with walking a mile in their shoes, truly, watch a job shadowing program, where your security team members spend maybe a month working in the dev space and understanding that world, maybe you have your devs go sit in the security team for a while or work with your SREs and see what the ops world is like.

 Give them the time to build that empathy to really understand the challenges, all the conflicting motivations that their peers in these other traditional disciplines are facing on a daily basis. So they can see how what they do can make life easier. And don’t just send your junior candidates, don’t just send your junior developers because they’re expendable.

 We need those senior developers to be a part of this because they can lead from the top down, a lot of the senior folks may not want to listen to what a junior developer says after they come back from spending a month with security. But when your senior team members can lead by example, because they understand that this builds that culture.

 Second part of this does come down to technology and processes.

 How can we meet them where they live?

 As I bring in security practices, I’m looking for ways that I can integrate my security practices into existing processes and existing tooling.

 Now we do a lot of this already, we talk about the idea of automation. And a lot of times it’s being driven from the developer side, we need the security teams as we’re evaluating tooling to involve the SREs, the ops people, the developers, and make sure that the tools are going to fit into their processes, into their pipelines, and that things will automate properly so that they don’t introduce new friction.

 It’s all about that idea of frictionless enablement. And when I talk about enablement, I’m talking about paving the road.

 Putting the tools out there, but then creating this idea of accountable trust, where I have people within my development teams, within my SRE teams who I can trust, they know they’re accountable for that security thing that I want them to do and I trust them to do it. But that trust goes even further and this is where we truly pave the road, I make the secure answer the easy answer. But going beyond that, to make that really happen, I’m thinking about things like security champions program, something we’re building right now with an SMP.

 I’m thinking about how do I make sure that these people have not just the tooling, but the training that they need in order to make the decisions and then I enable them.

 This is where we come up with accountable trust. Again, I enable them to see to make a different decision.

 If they see a way to achieve security, but by to go a different route, make it happen, I want to enable them to do that and pave a new road and show us the way and bring that back and make it something repeatable, that creates this idea of continuous improvement. And then finally, I want mutual engagement.

 If I’m trying to build empathy, I want these teams working together on a day to day basis. They shouldn’t be silos.

 Do your developers go to your security status meetings? Do your security teams, your SREs come to your daily Scrum meetings?

 Who’s a part of those? Can we get our business people and our security teams and our SREs all involved in sprint planning?

 What about our retrospectives? Are they a part of that so we can all learn from the lessons that we we got from that last cycle? These are all the things that we can start to do and they start to break down those silos and form a more cohesive unit.

 We do this with an SMP, we have our weekly Scrum meetings where we have people from security, we have people from the development teams, from SRE, from our risk management teams, bring them all together, these have paid off exceptionally well for us and they give everybody a chance to share what they’ve learned and also to have visibility into what’s occurring.

 This is how we start to build a true DevSecOps culture that makes all of our lives easier, where we’re all marching toward that same shared responsibility. Now, as I wrap up, I want to share this quote with you from Henry Ford, it seems to fit perfectly.

 Coming together as the beginning, keeping it together as progress, but working together is success.

 We need to all be working together breaking down those silos every day working together toward that shared responsibility and that’s where we’ll find success that makes DevSecOps actually work and be something that makes our lives easier.

 Now it’s time for me to wrap up, but before I do, I invite you to continue the conversation. If you have questions, ideas, concerns, you want to dispute something I said, by all means reach out to me Twitter is the easiest but LinkedIn and my website are other ways that you can connect with me as well.

 All that information is here for you and I invite you, get in touch. Let’s talk, let’s share our ideas. And let’s see how we can continue to improve. So with that, I want to thank all of you for being here as part of SwampUP, for being here for this session.

 I want to thank JFrog for having me, I can’t tell you how much I appreciate it. And certainly thank you to my organization, S&P Global Ratings, for making it possible for me to be here today.

 I hope you enjoy the rest of the conference.

 I hope you gained some truly valuable insights from this session today and we’ll see you again really soon.

 Thank you and take care.