How to have the cake and eat it too: Software in a High Compliance Environment

Right here, and keep in mind that talks have lots of stuff in common so I’m hoping I’m going to be interesting and educating right after him. So good luck, to us. Let’s start. As I mentioned, my name is Liran Haimovitch and I’m a Co-Founder And CTO @ Rookout, I’m an advocate of modern software development methodologies such as Agile, DevOps and Lean, and before founding Rookout I spent about a decade doing cyber-security for the state of Israel.
I’ve done multiple roles as a cyber-security researcher, developer, team leader, project and product manager and moving into Rookout and I also got to know a lot about compliance. And at Rookout, we’ve actually got done a lot of compliance, especially for such a young company under 3 years old.
and along the way, we have picked up HIPAA SOC 2 Type 2 ISO 27001 as well as as well as GDPR and CCPA The reason we’ve gone through so much compliance is the way Rookout work with our clients to give you some context. Rookout is an SDK that gets deployed on our customer’s environment, whether that’s their staging on production and can be used to extract data on the fly, without having to write more code, redeploy or restart. Now, getting our customers to deploy us in those environments requires both a lot of trust and for us to meet their compliance and legal needs. and that’s how we got so deep into compliance so early Now, about 2 years ago, 2 and a half years ago as we were just starting our first compliance – SOC 2 Type 2 I was sitting down with our head of DevOps at the time and he shared with me his story from one of his previous employers where, as they were going through SOC 2 Type 2 compliance, it got so bad developers couldn’t do the work and he almost quit, and let’s say that wasn’t the most encouraging thing for me and as we all know, to get developers doing their job especially in modern, small, agile organisations you want them to be empowered.
I think the best way to take a look at it is through the state of the DevOps report, one of the most important stores of knowledge for DevOps today and this quote by Nicole and team says, “Our own research has found that an organisation of culture that optimises for information flow trust, innovation and risk sharing is predictive of software development organisational performance” And so we are trying to empower our developers, we’re trying to shift-left, we’re trying to give them as much as possible. At the same time, take a look at this quote about governance, this quote is by Read Deshler from Align Org solutions and what he’s saying is that in business, we often use the more politically astute term governance instead of control now, you can go ahead and read this article online and it does go and challenges that governance doesn’t have to mean control and there are different approaches to it, but traditionally traditional forms of compliance with change boards and all that’s involved has been about control, has been about you as an executive telling your employees what to do, how to do, when to do and we’re looking at Agile and DevOps and that’s not what we’re trying to achieve and not how we’re trying to achieve it.
And so, we have to figure out how to achieve governance without control and that’s actually possible, the two thoughts are not the same, the words are not the same.
You can achieve governance by defining the ground rules for your team for your team, you can empower them to achieve what there is needed to do, to keep your codes secure with compliance you just have to have the measurements in place, the boundaries in place and know when your team, when your engineers are going out of bounds you have to know if somebody is hacking your system, you have to know if something is going wrong, but you don’t have to control everything. You just have to lay the ground rules. And… There are so many principles of DevOps out there. For this lecture, I picked this one from the See The Whole blog.
It shows 6 different principles of DevOps Social Aspects, Automation, Quality Assurance Leaness, Sharing and Measurements and as you can see, some of those principles, such as social aspects, trust, and sharing are not well-aligned with control because you’re trying to empower you’re trying to trust, you’re trying to enable on the other hand, a lot of the other stuff such as quality assurance, measurements and to a certain degree, automation actually work hand-in-hand with governance and so, you have to deescalate this conflict. You achieve governance through DevOps kind of like what John said without trying to achieve control and throughout this process I had my own little framework of trying to figure out how to do that and I want to share it with you today So, this is my way of implementing governance for existing DevOps which has served us for multiple compliance cycles so, whenever you’re doing compliance whether it is from an external auditor or some internal auditor or some instruction from your boss you’re going to get the requirement that requirement is going to ask and going to require you to ensure that something is being done and governed it in a predictable way you have to ask yourself is this currently met to an existing DevOps process within my team, within my group if this requirement is currently met then you’re golden.
All you have to do is document the way you’re meeting that requirement through an existing process. And that’s it. On the other hand, you may find that this requirement is not currently being met by your existing DevOps process. And so, that raises a question Why? I mean, DevOps is all about continuous learning Now, think of it as a learning opportunity somebody is telling you this requirement is required, this part is necessary to avoid a business risk to avoid a business risk for your business to avoid a business risk for your clients do you agree with them? Do you think this poses a significant risk? If so, then you’ve just learned something. There’s a new risk, maybe it’s the result of a new market or geography you’re targeting and you have to meet this risk, but you can do it the DevOps way.
Don’t mitigate the risk by adding a change board or manual process mitigate the risk by automating by shifting left the responsibility onto the developers by measuring for the risk and so, once you’ve met the risk this way all you have to do is document it and share it with the auditors Now, sometimes once you understand a risk, you might find that this risk is not significant for your business or your product or your platform and so, in many cases, you can just go ahead and document it as an acceptable governance risk I understand what the risk we’re trying to govern I understand the control that’s being required, and I feel that the existing condition is good enough Now, you’d be surprised but in many cases once you’re proficient, once you understand the compliance when you start to understand your processes in the business = then you can quite often convince the auditors and your clients that the risk truly is acceptable and is not worth the time and effort to fix it unfortunately in some cases that’s not gonna go so well and at Rookout there has been a handful of cases where we did have to implement controls within its areas we didn’t feel it was precisely necessary just to meet the governance controls as an example, we’ve always felt that physical security at our offices were not significant for the security of our services all the laptops were encrypted, password protected we’re using MFAs to login and so on unfortunately for us one of the compliance auditors we’ve met didn’t agree to budge on this requiremnet and so we’ve had to upgrade our physical security to meet the higher standard even though, for us, it wasn’t our best judgment but at the end of the day throughout this entire process developers at Rookout were almost not impacted at all by the compliance processes we went through now through this process through this framework we’ve ended up achieving compliance through 3 main pillars let me share it with you, with the first being automation now, we’re in a DevOps conference and everybody is going to be speaking about the value of automation today and they’re going to talk about how it saves storage, how it speeds things up, how it provides more consistent results, and all that is true in our case, from a security and governance perspective the most important thing is that everything we’ve said about the difficulty of empowering people and trusting them, setting boundaries keeping it at balance, well that’s not true with machines machines can be very easily governed through control you write a script, you control it and so machines are very predictable and allow you to very easily be goverened on top of that, when you automate for security think about how can you remove privileges from employees, from engineers and stop giving people access to production environments you can often just have scripts do the job for them and so you don’t have to give as many privileges to people and so, this is the way we’ve done it at Rookout we grant all our employees permissions to an automation portal in our case the automation is a Jenkins cluster your milage may differ and each employee gets a login to that portal and gets access to various jobs based on his needs anything that’s being automated is done from the portal whether that’s provisioning environment for Kubernetes, scaling up and down cloud resources spinning up machines backuping databases and so on and so forth.
Now everything that’s been happening throughout this process employees logging in, employees triggering jobs as well as the actual work of the jobs everything is being audited to logs and so, we’ve put governance over the process and at the same time we don’t have to give our employees access through SSH through administrator access for privileges to production environments but the best of it is that employees don’t see it as something that’s there to depower or hurt them or prevent them from doing their jobs.
On the other hand, this is empowering employees to do their real job and save on toil Now, the whole bunch of automation candidates deployments and releases, provisioning and deprovisioning environments system and integration test, maintenance and cleanups and backups and restores that’s not different than stuff you would automate for Dev and Ops perspective the only thing you have to think through is whether to automate so that I can give less permissions to less people and by giving less permissions you are improving your governance now, the second pillar at Rookout is CI/CD and GitOps essentially taking automation to the next level baking it directly into the work processes at the company. Now, as I mentioned, it’s a DevOps conference so people are going to be speaking about CI/CD all day long there’s even this new CI/CD offering from JFrog which I’m sure that you’re going to hear a lot about today so, I’m going to let other people discuss it I do want to share with you the value for CI/CD in governance and there’s quite a bit of it first, you get a single source of truth by putting everything in Git, then you can always go there and see what was the configuration, what was the code, what was deployed, what and where and how things are going second, your shifting left instead of adding all those governance requirements on the application after it’s been written, after it’s been tested after it’s ready for deployment you shift left as much as possible require your developers to build a secure governed application from day 0 put every requirement you can on them and challenge them to figure out how to write the code and how to test it in a way that ensures it’s being governed and you actually already have a native mechanism to do that pull requests by default include both a human and machine processes for evaluating the code and evaluating the change and just like you test for bugs any test for readability of the code you can test for any security requirements through the pull request and you actually have… everything is recorded you actually know who opened a pull request who approved it, how was it being tested, and making sure everything looks good. and best of all all of those existing work processing and control are already in place most of your teams already know how to work this way, how to use it and you’re not making any changes in their day-to-day lives they’re just required to do a few extra steps through the pull requests which are mostly automated.
And as I mentioned, everything is audited both the changes in the Git in the source repositories and the changes being done to CI/CD everything is audited, you know exactly what happened, why, who triggered it and so you can easily provide it to your auditors as proof Now, the third pillar for achieving compliance is monitoring and observability now, there are two reasons for that.
The first reason, which I’m not going to spend much time discussing is that to ensure compliance you actually have to know what’s going on you have to collect the information from your environments to know for instance, if your governance required uptime monitoring you have to monitor for uptime if you’re making other guarantees then you have to monitor for those as well but even more important than that, you have a production environments where everybody in your organization wants to know what’s going on in that production environments you have business people sales and marketing looking to know how many people sign up how much they are paying you have product and user experience folks who are trying to learn how your users are using your system what’s being used more, what’s being used less what’s more intuitive or not so much you have Ops and security people trying to make sure the system is up secure, working as expected and last and definitely not least, you have developers which are often your biggest source of danger here because developers require different pieces of data everyday wether it’s to fix a bug or develop a new feature Developers daily data requirements change while all the other groups I’ve mentioned tend to require the same data, day-in-day-out developers needs change based on the task they’re working on which often leads many of us to grant them excessive access to production just to see into the code and by doing that we are hurting our governance into production and so, make sure you have monitoring tools in place to ensure everybody gets the data they need without having to access production directly in an ungoverned way make sure you have business intelligence tools for the business folks make sure you have logging and APMs for security and ops people.
Make sure to have error tracking for engineers and send as much data as you can from your application to whoever needs it obviously you have to make sure that all those monitoring tools are also secure, have access control and so on one of the nice things about Rookout is that we use our own products to allow our developers access to the production environments to collect data as needed in a secure and audited manner without granting them direct access to the production servers themselves.
Now, building your observability and monitoring is a big topic and a lot can be said and written about it there might even be a few talks today about it and there are definitely others out there I have even given a few I do want to give a few pointers especially those focusing around security So start by mix and matching tooling there isn’t a single tool out there that’s going to provide everything you need you will probably acquire a mix of those and you’re going to have to keep an eye out on the scope of that, from a security perspective as well try to get as much as you can off the shelf there’s plenty of commercial and open source software that you can use but at the end of the day, you sometimes have to do it yourself whether to put some duct tape around it to bind it together or just develop a little something for yourself try not to do it yourself too much unless it’s truly necessary make sure to redact data you’ve just went through an entire ordeal to keep your production secure to keep your production governed and make sure you know what’s going on there so make sure that the data coming out of it is being redacted based on its sensitivity and where is it going so that you’re not accidentally sending data under compliance to non-compliant systems and last but not least make sure to have retrospectives at the end of the day, there is no better way to figure out what you need than by seeing what you needed in the past now especially when you have production issues and data is needed then everybody needs to step in use the permissions they have and extract the data from production to fix it but once the dust settles down have a retrospective, think through it what data was missing? what data did you have to risk your activities in Ops or security perspective to get and how can you make sure that next time that data will be available faster, easier and to people who might not have as many privileges as the Ops people? Now, that’s it.
That’s all I have for today. As I mentioned, I’m right here with you in the channel so feel free to ask more questions check us out at rookout.com and feel free to follow me on Twitter @liran_last