Software Bill of Materials and its impact on Enterprise DevOps @DevOps Vancouver Meetup

January 25, 2022

3 min read

Bill Manning from JFrog introduced us to the Software Bill of Materials and its impact on Enterprise DevOps.

Bill Manning from JFrog will introduce us to the Software Bill of Materials and its impact on Enterprise DevOps. With the White House’s cybersecurity executive order in May 2021, has the Software Bill of Materials (aka SBOMs), graduated from being a “nice to have” to a “must-have” global standard when developing and deploying secure software from the cloud? In a nutshell, SBOMs provides visibility into which components make up a piece of software and detail how it was put together, so it’s easy to determine if it contains security and compliance issues. In this talk, we’ll discuss:

• What exactly is an SBOM?
• Securing your Software Supply Chain
• Why SBOM must be a key element of your software development life cycle’s (SDLC) security and compliance approach
• The misconceptions that exist around SBOMs
• Insights and best practices on SBOM creation and usage.

Bill is a Solutions Architect with JFrog. He is also a mentor with TechStars, Matter, and NestGSV. He has successfully exited 3 companies and took one public in Australia. He is also currently helping various startups as an advisor. In his spare time, he likes to travel with his wife and two boys. He also plays guitar, loves the ocean, and is an avid cyclist.

In the second part Willy Schaub will talk about his experience in setting up common engineering system at WorkSafeBC.
Explore the trajectory of our Common Engineering ecosystem

In recent Meetups, we introduced our efforts to establish a common engineering system to empower our engineering teams and strive for consistency, standardization, security, and continuous innovation. After we operated like the Rebel Alliance on the ice planet Hoth, I received the opportunity to create and lead a common engineering team that operates at the heart of our organization. We have an ambitious goal to empower every engineer by standardizing our engineering practices and tools and enabling continuous delivery of value to delight end-users. In this brief overview, you will get an update on where we are on our thrilling roadmap.

Willy-Peter Schaub started his IT career when computer memory was measured in kilobytes and storage in megabytes. He is a software engineer who strives for simplicity and maintainability, continuously experimenting, failing, learning, and innovating to foster healthy DevOps mindsets and empower communities and fellow engineers. Explore some of his publications, and connect with him on and

View Slides Here


Bill Manning

Bill is a Solutions Engineering Manager with JFrog. He is also a mentor with TechStars (Nike Incubator), Matter, and NestGSV. He has successfully exited 3 companies and took one public in Australia. He also currently helping various startups as an advisor. In his spare time, he likes to travel with his wife and two boys. He also plays guitar, lives for the beach and rides skateboards.

Video Transcript

okay all right so let’s start um so uh thank you very much uh good evening my name is
andre kaminski i’m organizer of this meetup and this is the first meetup uh of this year
end of last year we introduced a couple of changes i wanted to thank alison bennett who was
the co-organizer for the last 18 months and she was helping me with with leading those sessions
and i would like to welcome ilsa bishop and willie shalp who will be helping me this year
our intention this year is to continue finding interesting speakers companies and products and try to experiment with
different delivery formats today we are going to have a two interesting presentations but first i
want to say a couple of housekeeping items so first of all this session is recorded please keep your mics muted
after each presentation will have 10 minutes for questions you can type questions anytime in the in the chat or
unmute your microphone if you want to ask questions directly all right so as you know devops has
different flavors and sometimes some people refer also to concept of death sec ops and and to me
this doesn’t really make a lot of sense because honestly speaking security is part of
delivery one way or another right this is not negotiable part of non
of of functional requirements non-functional requirements right and um
as a devops we build applications and we operate those applications and we often assemble applications from various
various vendors and some of those applications of those components
are including for example sometimes open source directly or indirectly if we implement a component
from a vendor that might be using some of those open source components we might
not immediately see those but since most of the applications um currently are really in the cloud
delaying updates to those applications is no more an option um applications
becoming more and more complicated and either because we try to keep those
applications secure or stable due to uh changing underlying software software frameworks
we need to keep that track of those components so today our first topic will be related to
exactly to the subject and i would like to introduce and welcome ari waller and bill bunning from
ari over to you thank you so much i really appreciate it hi everybody my name is ari waller and i
am the meetup event manager for jfrog and we really really appreciate the warm welcome uh tonight here at the vancouver
devops uh meetup community um i’ll share just a little bit of who we are uh jfrog is a devops software
company known best for artifactory which is considered by many to be a gold standard for managing your artifacts and
dependencies um in fact one thing a lot of people don’t know yet is that jfrog has a free cloud version of our
artifactory product for the meetup community it’s not a trial version that runs out but a free version
it’s great especially if you’re working with docker as i know many of you are jfrog’s cloud subscription is capable of
functioning as a pull through cache for docker hub and because of the partnership we have with docker you’ll
be exempt from any rate limits on free or anonymous uh free pulls from anonymous accounts so that’s something
that could be valuable now i want to share my screen with you we have a something special for uh
the vancouver devops community tonight um let me get to oh let me get to these where’s my
spy that’s not good uh one moment please
you know you think you would think that after all of uh you think that after all of this
this experience we have now with all the virtual meetups here i go uh this would
be so simple um but uh it’s not so here we go hopefully
everyone can see my screen okay now um but uh let me just grab one thing here and you
should still see it but um see everyone see the slide okay yes
great a bonus for you tonight is since our topic uh is um around um
something technical we want to of course give you a technical treat so jfrog is going to give you a chance to enter to
win a raspberry pi 4 which if you try to procure these days it’s not the easiest thing to procure i have to order them
only there’s only a limit of one if you can even find one um so this is one that you can get your hands on and play with
a little bit um you can enter with the qr code or the bitly link that you see and i’m also going to drop it in the
chat i can’t do live sweepstakes uh online for compliance reasons however we will
randomly select the winner within two business days we will contact that winner via email and once the winner
claims the prize officially we will go ahead and share the winner with your meetup community as well but we want
everyone to be a winner tonight or ever almost everyone at least to have the opportunity so for the first 20 people
that do enter um that enter the sweepstakes we’re going to give them a
jfrog t-shirt and a liquid software book uh we’ll send it to you um just uh just
for just for entering to the first 20 people so um i’ll go ahead and drop that link in the chat and uh
i’m looking forward to hearing bill speak tonight um and i’ll let him do further introductions but thank you uh
andre and uh team for having uh having us in your community tonight we really
appreciate it bill i’m gonna turn it over to you and stop sharing excellent hey guys how you doing um i’m
bill manning i’m gonna share my screen here and let’s just kick this off uh let’s get this shin dig go in here uh
let me move the little uh annoying zoom window out of the way so it’s not covering up all my stuff um all right so
how you doing guys so tonight um i’m gonna introduce you guys to s-bombs and uh you know software build of materials
and uh basically the impact on on enterprise devops right this is a become a huge talking point uh especially for
me over the past year um with for a lot of different reasons um i’ve done this talk um not this exact talk but similar
talks to this for multiple times uh both live at like reinvent and some other security things and uh today i’m going
to present it to you so um i am uh one of the solution architects slash also the manager of uh
solution engineering for the americas for jfrog um as ari stated we are a uh you know we’re an international global
company uh we have over 6 300 customers 70 of the fortune 100 we went ipo last
year um you know we are involved in everything from uh we’re officially like a cve provider we’re also members of the
cloud native foundation um in terms of governance so working with a lot of you know under you know basically coming up
with the standard that most companies can utilize to do sort of the cloud native approach um there’s a multitude
of things that we’re involved in i’ve been with the company for about five years so you can follow us at jfrog on twitter or even follow myself at uh
william manning um but today you know the importance of this is is not only talking about software build materials
but what really what really led up to it and and really what it means uh in terms of organization i’m also going to give a
demonstration and show you like how we handle software build materials and i’m also going to talk about the two
different formats that are currently being utilized right now and i’m actually going to demonstrate some of those formats to you so you understand
what it entails so we all see the news all the time right i mean really the software build
materials has been a thing for a while but it’s really become more apparent when we see alerts like this you know
something from hacker news you know that’s always a place where i always go to make sure to see you know what’s interesting out there you know what
exploits are happening um but the thing is though is what we’re going to deal with today is we’re going to talk about supply chain security right because
supply chain security in our software development is really what came about to really drive the software build
materials idea and concept forward especially in the united states i mean we see these headlines all the time um
there’s nothing new here right i mean it’s constantly even a couple weeks ago i should have added in here a lot like what two weeks ago the massive log for j
thing right i mean one of the most commonly used libraries across the planet also happened to be one of the
most susceptible right and the thing is is that last year alone in 2021 there was a 650
increase in supply chain attacks i mean that’s insane when you think about it right i mean the thing is is that you
know most people’s exposure to software supply chain attacks you know they’ve been going on for a long time it’s just
that it became more publicly acknowledgeable um when it came down to solar winds right solar ones was really
the catalyst for a lot of countries and a lot of corporations and a lot of industries to really reevaluate the way
software supply chain is actually handled i mean when you look at saw you know solarwinds itself it affected 18
000 customers globally right it was a multi-billion dollar remediation scale and the thing is is that it actually
broke a trust model and the trust model is is that when we start talking about being developers or devon you know deaf
engineers or doug you know uh you know devsecops engineers release managers qa engineers the thing is is that when we
go down to his lowest factor the one thing that’s important to recognize is is like i mean actually you see from the
white of my face here you know i’ve been doing this for a long time you know i started off as a software engineer and
did i’ve done a multitude of things from being a software engineer to cto and founder to vc to what i’m doing now but
the idea is is that when you’re looking at this we have this inherent trust model that the things that we use to
build the software we produce uh was a trust it was a blind trust you know we would bring in these libraries that we
need to do our job easier um you know we need a function to go ahead and you know do something with a string you know go
find something that does that and the thing is though is when you bring in those libraries and we start talking
about this remember that library comes in with all of its friends and you know it’s like throwing a party and you
invite one or two people and next thing you know there’s a hundred people at your house and you have no idea who they are or how they got there
but solarwinds was really was the catalyst because in the united states alone it affected the department of defense department of homeland security
the federal reserve bank you know these this infected so many systems you know out there and it was a third-party
transit of dependency attack and it wasn’t even a direct dependency attack it was an indirect dependency attack
meaning was the dependency of a dependency of a dependency that came in and why are software supply chain
attacks so on the prominent rise right when we talk about this and the reason why inherently the model is broken and
the things need to be put in place to ensure safety and security not only for
you know what you do as a job but for your company and the people that use your products and stuff because let’s
face it no company wants to be a headline right no company wants to be like x company got in you know
infiltrated based on a software attack you know that was you know blah blah blah and you know next thing you know
there’s there’s you know people are running and panicking and trying to figure out what happened right and the
reason why is is that number one is super low effort right it’s completely low effort compared to most coordinated
you know other attacks that you might have on like you know ddos or you might have like a network infiltration or you
know all the typical sql injections and all that because those take a lot of effort what this doesn’t really take a
lot even the technical skill required is pretty low i mean the thing is is that associating yourself to uh you know uh
you know putting in some sort of malicious code into some sort of third-party transit dependency that gets
the site you know basically loaded in like a side car is insane and the thing is it’s super high speed to spread
that’s the insane part because the thing is is that once you’ve actually introduced it into the system you know
it’s there and the thing is is that once again it abuses the relationship uh between you know companies trust and
this you know the things that they produce and there’s always this little nagging thing in the back your head after you know you’re reading these
things that says am i really i mean can i really you know cause something completely terrible to happen with my
corporation or maybe even just a project i’m working on because the thing is is that the thing
is is that these hackers you know these people who write this malicious component you know they just blend into
the community they’re trying to become part of the community they add to it they contribute to it and somewhere down
the line they’ve introduced something that could be potentially threatening so when we look at the way this happens
is that as a developer and like i said this is all leading up to the reason why accountability is so important is is
that you know as a developer you go and you have to code something you go ahead and no matter non-dependent on what
language you do you start pulling in a bunch of transit dependencies that help you do your job better right you produce
your code and you know what happens if one of the packages you have in here is malicious you know next thing you know
it’s it’s part of what you do it’s part of what you distribute and next thing you know you’re putting it into a web
service or you’re installing it on the device i actually give another talk i gave a couple like a year and a half ago
it was called like if code could kill you know basically like bad programming and bad libraries that cause
unforeseeable tragedies you know on things that have happened but the thing is though is this is not only an erosion
of the actual software component itself but also you know the reputation of a company but most of the time it’s not a
direct dependency attack that actually causes the problem the dependencies you have depend on other things and if one
of those components is nefarious it gets dragged along with that same you know that same level of expertise and next
thing you know you’re at the same place right you’ve introduced something into the stream of what you do and like i
said you know developers are inherently trustworthy we’re kind of blind trustworthy in this case by saying you
know these are the things i need to accomplish my tasks meet my kpis meet the deadline in my sprint i mean i can
go through the all the different things right but the thing is is that these are supposed to make our jobs easier now
there’s a level of doubt associated to it because 85 85 to 90 percent of your
software is someone else’s code base right it really is i mean the thing is is that when you think you’re developing
software and you’re building it 85 to 95 percent of that you know 90 of that is someone else’s
99 of that contains you know is mostly open sources 75 that contains at least
one vulnerability some vulnerability that could potentially affect the way that your your software operates and
also the security behind it on top of that 49 of that code base that was analyzed has one
high risk component and when i say high risk opponent if you’re familiar with cve which is kind of the standard or
cvss score which is also the other standard is version two and version three um you know there’s a multitude of
different ways of factoring in the viability you know the vulnerabilities behind this contain at least one item
and then the thing is though is 90 of the applications out there have that pieces in it that are either old
out of date or completely abandoned so we’re just saying oh you know what this thing hasn’t been updated since 2014 i’m
sure it’s fine um you know i mean that’s just the way it is you know we bring it in and most of the time we just check to
see and you know maybe if there’s a problem we go check stack overflow and we look for a bad example we try to
retrofit into what we’re trying to do we spend more time messing around with that i mean that’s just the way it happens
but the thing is is though 74 of this can be actually alleviated by just simple updates simple updates right so
in other words taking a little more care in the actual transit of dependencies that we utilize as developers and this
carries over into ci processes because now with the modern day you know when now with dev apps and devsecops i mean
you can use them interchangeably you know the idea is you know a lot of these people do bin ops or git ops that
whatever your organization does you know a lot of times now it’s more automated processes so even it happens even out of
the purview of the actual developer doing the job that they do you know they work on their code they compile it they
send it off to a ci server maybe run some unit tests brings back some data whatever and a lot of times we’re like
you know what i don’t want to update that library because potentially it could ruin what i’m doing so what software attacks is you know
there’s a lot of different things out there a big one is you know typo swatting being more responsible for the
things that you do because there are people out there that create code and your libraries for your code that might
be an i before e except after c you know they might just slip around one of the letters or one you know and the thing is
is that you accidentally bring in something you thought was an actual library that you wanted to use but it
was a typo and next thing you know you’re introducing something malicious into it another one is dependency confusion
dependency confusion is so prominent it is unbelievable the thing is is that we
go out there we do a search and we say you know hey this is
oh there we go um so um you know the thing is with dependency confusion the idea here is is that you know there are
these private repositories out there you know like netflix has them right here’s some helpful libraries you know paypal
for authentication purposes maybe some other high security you know apple you know there’s all these ones out there
and how do you know if they’re the true thing right how do you know if they’re truly from the actual source itself we
have an example that i did where you know somebody was like we went through it and the thing is is that for what
they were trying to do for their software they’re like hey paypal has an off thing and it has an analytics piece
i’m going to utilize it you know it’s got a write-up these are fake these are not real somebody actually went in and
said they’re going to go ahead and use you know they created these paypal um style dependencies you bring it in and
suddenly there’s this massive you know of data leakage that could be happening you know we found this out with the actual off paypal right it wasn’t the
true auth it actually took credentials and sends it off somewhere randomly that could be this extracted and turned to
the next wave of attacks so the united states government and this is actually one of the direct things
that have happened this is actually now going globally was back in may of 2021
the biden administration signed the executive order improving the nation’s cyber security yes i had the unfortunate
task of having to go in and read this um and i will say i i fell asleep a couple times definitely because it’s not
exactly the most enticing uh thing you’re gonna read but one part of it really stood out and the part that stood
out was of course section four enhancing software supply chain security this is huge this meant that the
government had actually taken the time and effort to take this seriously right and this is actually you know you know
canada is going through the same process uh the eu is going through it uh it’s going through in britain this is
actually becoming a standard now globally but the important part is is that if you deal with the government the
government agency and this is now by the way avi the avionics industry health and medical
financial institutions are now following suit not only the governments and what you have to do is if you produce
software any software that you sell somebody can request a software bill of
materials and today i’m going to explain to you the reasoning behind this and how you can actually utilize this
so first of all it’s just what i’m showing here first of all software builder materials is boring in a way and
so we’re going to talk about cake today instead so first of all when you look at
about you know a cake box or any sort of food you eat one of the important things is you get a list of ingredients and you
also get a list of warnings and the reason why i’m bringing this up is is because actually the company the
actual group that proposed the software bill of materials was twofold
it actually has to do with the fact is you need to know what’s inside your your cake that you’re eating right you know
it has a whole bunch of ingredients right and this is really what a sovereign bill of materials is it’s the
list of ingredients because the national telecommunication information administration
in conjunction with the fda and then the associated relevant industries in other countries got together and said this is
back in 2018 and said we need to have accountability for medical devices we do do this for food
and the fda in america said we need to come up with the food and drug administration so we need to come up with an accountability aspect of the
software that gets utilized in medical devices think about it right you don’t want to have some sort of nefarious
codes sitting inside your pacemaker or your ventilator these days for the past two years god i mean right i mean these
are things that you need to have so this became a standard they started proposing what would it take and what does it mean
to actually have something that is safe secure and accountable
so basically what it is it’s just a list of ingredients that makes up your software this includes all the libraries
used whether it’s paid or proprietary right it also talks about access what kind of access controls does it have you
know this kind of level of information additional information that is beneficial and i’m going to talk about
that today is also to what tooling was used you know what environmental and system of variables were used when you
were constructing it are there specific settings that you have are the specific versions that you’re utilizing when
constructing your software basically it’s a map to how all the ingredients
that you have in your software and it’s a way to provide it to your customers to have that accountability aspect i know
there’s tools out there that parse through the information and we’ll talk about that soon well what is it used for right
so in a lot of cases it’s actually used for maybe maintenance right and i’m going to show you an example of this
today like you know we talked about log4j we talked about you know these kind of components and how you can utilize this information to help you
know safeguard your customers against things so in other words say there is something that varies you can provide
them with a level of information to say that by the way yes we shipped you code and we’re sorry but it contained this
information those companies too when they read those warnings can go in and search and
contact your company to say hey by the way we read an article that said xyz library
is potentially threatening because it actually can go ahead and leak all of our customer data and according to
software build materials you guys have been utilizing it it’s a safeguard it’s also a level of communication that’s
never been done before between the customer and the provider which is actually very important to increasing
the level of awareness of this sort of level of attack on top of that it’s also used for
pre-purchasing and negotiations in a lot of cases you could go through a legal process selling your software to a
company and they go through it and everybody goes through and looks at the you know the terms and conditions and
all the stuff this will be that next level to say hey maybe we are purchasing your software and we have a certain set
of safeguards in place of certain things that we don’t want to use or say you’re a company you’re a small startup you’re
getting acquired uh by another one you need to provide a software build of materials for the software you’re using
i actually had to go through this where i actually had to spend six weeks with the guy i was working with this is like
back in 2010 we were getting acquired and we had to go through and detail out all the things that we used to build our
software and then we also found out that we had to make changes because that company didn’t accept what we were doing
on top of that it’s also a way to also involve management and other asset managers and legal teams and stuff all
on the possibility of these inclusions leaks and potential threats that might happen and it’s also a way to ensure a
level of responsibility in terms of insuring yourself you know against the companies that could potentially even throw lawsuits at
you because it’s one of those we told you what was in the software now the benefits of it is is the way to
identify mitigate and avoid right these are things i like to always talk about that these should be things that you’re
doing also to in terms of security anyway around software development right it’s also a ways in a lot of legal teams
how many times have you guys worked for a software company and i i’ve had this before where the legal team says hey we
need a list of all the licensing we have we need to do a license review of the software to make sure that we’re not
using any licenses that are not approved by our legal department right there’s also a way for you to go
in there and also look at the inherent risks of some of the interdependencies on the software because it also gives
you a broad view of not only the transit of the presidencies that you implicitly stated as part of your development but
also the indirect dependencies that came with it i think i see a bunch of chat information here um i actually would
like to uh i’ll address those in a little bit if you guys don’t mind um and
also too there’s also environmental and system information because wouldn’t you like to know if you’re producing
software and you go to do a recap and say you know say you’re deploying version 1.1 and 1.2 is now about to go
into production you put 1.2 in place suddenly um it’s performing terrible
comparably and 1.1 was actually better and you don’t want to roll back you want to move forward you want to see what’s
different you know what i i bet you once again phil left the debug flag on when he was compiling and it’s accumulating
stuff in the logs and it’s constantly writing and it slowed us down and damn it we got to go back and change it again and go to the next one right this is
stuff though but to do that you know in a normal environment would take you a while where you can go in and see this
information firsthand you can even go ahead and do a basically a diff and say oh yeah you know what phil did leave it
on ah you know and then you can scream and yell and go make fun of them for a couple weeks whatever
but then at the same time it also helps lower some of the operating costs because the thing is is that that
accountability aspect is retained in a standardized format and we’re going to talk about those formats today one is
called spdx which was approved by the us government and by the way canada is also looking at the same format and then
there’s cyclone dx which is another one there’s been once again it’s the format battles right on who decides on what
formats best now saying that we know software has a lot
of stuff now let’s get into it right so this is the part that i want to show and i’m going to show you a couple of examples of it too right but like i said
i don’t want to talk about software i want to talk about cake i’m actually kind of hungry right now anyway so this will kind of go along with it
so first of all this is a lovely cake and you know some stuff about it right this is what you normally would get in software you know it’s a cake or it
might be part of a larger cake maybe it’s a topper right you know it was made somebody you know it was mixed somehow
you know it was put into one oven or your hope was put into an oven or you know it was cooked some other method it
might be tasty you never know right these are all kind of things that make up the attributes but we also know the
number one thing is used ingredients so when we look at this we know the ingredients that it has right we know
that these are all the pieces that make up the software oh somebody’s uh there we go we know all the software
that’s part of it and you know like these are all the pieces that make it up but also too there’s a whole set of
instructions right on how to prepare this just because i give you a box of ingredients i’m sure you guys have all
seen the great british bake off right i mean it’s that kind of idea and there’s that whole blind thing you kind of have to figure it out but most most of the
time most of us wouldn’t know how to do that we would actually have to go away most people at least have some you know practice behind it this is
actually a way to say not only this is how you did it but also too this is how you actually made it
so when we look at it what if what happened if we changed the ingredient right the thing is is that maybe the library changed but in this case i’m
using baking soda the recipe says baking soda what if i go and actually substitute that for baking powder i’ve
done this i’ll admit it i’m terrible um but what if you did for baking powder you’re right you may be the result they
sound baking powder soda what’s the difference well there’s a big difference you know the thing is and this actually
is also with your software you know you know interchangeable components you know maybe one change versus the other how do
you know and this will allow you to look at the two different recipes or in this case the software build of materials and
allow you to have a diff between them to say oh yeah we did accidentally use baking powder instead of baking soda we
did leave that debug flag on oh we did change you know what somebody just introduced a new library into this and
i’m going to show you some examples of this too so what’s inside the accounts right so once again you know i’m a developer i’m
going through i’m building my software i’m doing my thing you know and when i bring in all these transitive
dependencies i want to make sure all that information goes into the software build materials
at the same time i can provide that to my customer base and if something changes i can notify them in the
software machine materials that something changed between the versions that we have
if something is nefarious in place i can notify my customers and they have that list of that information as a parent to
them it’s also an internalized list that you can maintain and look at yourself so this way you can notify the customers
based on this so that means if you do find a component like this you can go ahead and find all the versions of the
software that contain that component based on the information of the software bill of materials
this also means the thing is what about cakes with lots of layers right that was like an individual soft for instance
what about things you know that make up you know made up of various components like web services and things like that
right your interdependency models where you have one program that depends on another or you know functionality you
know you might have light you know and like a self-driving car you might have a lidar that depends on an ai function
which depends on you know multitude of other facets right each one of those is different each one of those is made
differently each has different components you know together they work together they might be delivered in
different locations um you know it depends on their purpose so when we look at something like a web
service you can think of it as a multi-share cake like i said we do we’re going to talk about cake or you know here we go right so in this case maybe
the from is the hot you know the helm is the frosting that keeps the web service together but each layer of that cake can
have its own set of attributes its own software bill and materials so when we look at something like a
docker container even you know we start looking and there’s an application layer there’s a run time right there’s an os
this itself can be a software build of materials when we start talking about a web service same idea a whole bunch of
docker images with their own software build materials a helm chart that can have all the information of how the pods
are structured these can be individuals cases also and if there’s say one
component in it you can go ahead and flag that component so the whole thing is i’m trying to make sure we have
enough time because i want to give willie enough time for him to do his spiel too um but the whole thing here is is that you know additional information
what’s the icd tooling are you using you know where we know when was the software built you know what stages you know of
the sdlc that they go through the software development life cycle you want to know did they go through qa did it go
through staging they go into production and when you know with the qa team you know let’s go look at the testing
results that happened you know you want some way of accountability and this is these are things that you can include
into it you also need to know what foss process right so in other words did you pre-evaluate some of these free and open
source libraries right once again it’s uh you know free isn’t speech not as in beer um you know kind of idea of when it
comes down to you know free and open source licensing you know libraries that you use which make up 85 to 90 percent
of your software also too what environment was it used you know was it built in you know the thing is is that
maybe you have a ci system that’s dependent on specific build versions of these kind of runtimes
at the same time one environmental system once again going back to the debug flag did somebody go in and set
you know the java heap value too high or something like that and then also too
where is a is there any security and vulnerabilities that are attached inside
so saying that you know i’m actually going to skip this part because i’m actually going to show you what it looks
like i’m actually going to break it down i’m actually instead i was going to show you some some screenshots but i’m going
to hold off on that um because i’m actually going to skip ahead to the next piece because i’m going to actually show you what it looks like
let’s go look at the software bill of materials all right and i’m going to show you the two formats so this is our product and
in this case this is our jfrog platform but for right now we’re going to concentrate on artifactory so
artifactory is our universal binary repository manager it’s a mouthful to say but it provides a lot of functions
we support over 30 you know package types directly out of the box but let’s go look at a build and for this build
i’m actually going to go in and i’m going to pick docker i like picking docker docker’s super complex right so
with docker you actually and i’m going to show you the fact that i have an application that has its own software bill of materials on top of the docker
software bill of materials so let’s go look at one this one happens to be built by a jenkins server right so if i show
you up here i’ve got a jenkins server that builds my pipelines and actually publishes a build into artifactory i
have a whole bunch of transit dependencies that i need well first of all i’m going to show you like this build here right so first of
all here we go here’s your your good fun exciting obfuscated you know docker image layers you do a docker
pull you do a docker run and the magic happens well you want to know what’s inside so if i look in here i can show you that
this actual docker image itself is actually running a node front end of 3.0-28
and it’s running a java back-end service of you know 2.0.47
right i actually know what version of the actual software is running inside this container it’s just not you know
container tag number 82 right you know that’s the thing this actually has quantifiable information inside of it
i can even go in and say this is remember i was talking about bob you know screwing up in action
know screwing up and actually you know say phil screwed up and stuff i
could actually go in here and actually do a diff between two images to say hey by the way it looks like the java it
looks like the note front end stayed the same but the java back end changed
i could show you all the environmental and system information right on how it was constructed
um i gotta warn you i’m loading up our vulnerability data and all this and just to show you this by the way my container
that i’m showing you i probably should have picked a different one this thing is the worst thing i’ve ever created it has every major vulnerability you could
ever imagine in it 1630 vulnerabilities um but the thing is though is these are
all the component layers this is the application this is the os this is the run time right i have security information on how
it was actually constructed and you know all the pieces that are in it i know all the licensing information around this
right this is just how all the stuff that we collect in our case but i can export all this information too
so i can go in here in our case and you see i’ve export s-bomb to spdx and to cycle and dx formats
and just to kind of exemplify something quickly is remember i talked about how you could have software build materials
that represent software that runs in other things right multi-tiered applications um you know web services
dependency models even if i go back here i’ll just show you quickly is i can go in here and in this case you know here’s
a build i produced i’ll go look at the build that actually produced it if i click in here this is actually the build
i have this is that node front end i’m doing i can show you here here’s the targe z that i outputted here’s all 482
transit dependencies i have um that make this up right so this is all that multi-complex layer of software by the
way this build for this ui has five dependencies defined and it brings in
482 transit of direct and indirect penalties how can you attribute to all of that
but at the same time i can also export its own software builder materials right because it has its own also
let’s go look at this right so i’ve already exported these before because i i had to do it um
because you know anytime you have any sort of security flaw or any sort of security issue you know one of the
things is accountability so i can even show you in here like in our product we’ve actually built this in where i
could say cve take a cve do a search on it right and this will actually go ahead
and find all the versions of the software that reference the cve so i can use the software build materials to say
oh wow you know this issue has affected all these builds quickly and rapidly right it allows me to i mean within
seconds i can go in there and find all the places that something has been used of a potential threat
so knowing that let’s go in and i’m going to share my vs code for a second guys and i’m going to show you the two
formats so i didn’t export so the one i have here is the spdx format it’s uh it
comes you could actually do it in a couple different ways actually i should probably show you that by the way so there’s a couple of applicable ways that
you can go ahead and do this and the thing is is that the four mess under the standard are the interesting part let me
go find uh here i’ll go ahead and do this one um let me just grab a random version
um i’ll just show you because some companies will expect others uh versions so if i do spdx you can either do it as
a tagged value right which is the is one of the tagged values so sometimes i’ll ask for that you could also use
doing just excel right so just do it as a spreadsheet if you want to do it for legal purposes and also or for json now
for our purposes i’m going to show you the json version because this is actually more of the standard because there are a lot of management tools for
software build materials now there’s a there’s a bunch out there so most of the time it’s going to be json they’re going
to request it you’re going to ingest it into a system and then they’re going to have there’s all these companies springing up now that are doing that
um also too you can also do cyclone dx it’s either xml or json right so it’s
either or so we actually do export both of these but i wanted to show you what it looks like because you know you’re
going to wonder well if you look here you can actually see like the spdx format shows all the
creation info right so it basically says like what it is and you know the organization and and me in this case the
bill m that’s me but inside this is the list of all the dependencies right so there’s initially
just a straight list of every single one i probably should have picked something a little smaller uh than this to kind of
show you because this is going to be scrolling for a while um because there are a lot of components here i mean let’s see this just to show
you this docker image is about 900 says 982 if you take out the extraction in the
front this has about 960 components that make up this container
this is 960 pieces of software that are utilized to actually construct that one docker image
all right now picture trying to do this in some other respect but then it breaks it down for this so in other words
here’s a link directly in our case to ours right so this asks for the document namespace and then it breaks down the
packages even further so it actually grabs all the information from all those
components and actually pulls out any information that could be there including things like checksum um you
know the name of it um like here is like you know here’s a you know ant jfx jar right so this is actually all the
information that gets cataloged once it’s catalog it becomes searchable and that’s one of the major factors of why
software with materials was brought about
now s now cycle and dx is the other but if you look here you can see that it’s
different there are distinct differences and the thing is like the us government decided on site on spdx canada followed
suit i know that in eu they’re going with um spdx i know the britain because
they want to be different with cyclone dx um right it’s just you know but both formats are out there and i will say i
think spdx i think is probably going to grab more prominence because i think it’s an easier ingest method i think
cyclone dx is another format to that but just let you know both are out there they both serve the same purpose
but if you look here you can actually see you know like once again it’s that same level of information you know it’s the name the creator um you know this
actually has you know the type in this case um and then once again it just starts breaking down all the pieces but
instead of having an index at the top it’s actually individual components that are listed out instead
now saying that understanding it and having that level of information really helps
you so when you’re when you are doing your builds you know make sure that you know some of the even if you’re not using our product there’s the thing is
is that you can even there’s tools out there for things like jenkins there’s one that takes like all that information
some of that some of that information is collected inside of jenkins you can export there’s tools out there that will
go ahead and you know you can point it we’re actually gonna we have one that’s kind of like this you can point at a
piece of software that you’re doing and you know and be able to go in and and look at the components and build a
component graph is what it is you’re building a component graph of all the pieces and then you have to export it
into a format that you know works for you know what you’re doing or what your regulation is or what your customer
expects but what you’re doing is it’s all about accountability it’s all about understanding and it’s all about you
know like you know like i said you know when we’re looking at something like you know like a lot you know the log for j
issue right i can go in here and i can say i think i actually have it already set up queued up here somewhere like i
can go in here and say you know like you know log you know in our product we just say log you know for j you know i can go
ahead or i was allowed for j um oh i’m looking for artifacts sorry let me go in here or actually you know we’ll go to
packages we’ll do it from here instead um you know i can go in here and say okay let’s go look for you know all the
pieces are if i was actually doing things correctly today whatever um i’m gonna go back to packages here we go um
but i can do searches on the things that matter to me or actually i’m not doing it right right now i’m actually keep
going back to there i don’t know what i’m doing wrong i messed up before i think i or i said something different
but understanding you know like where things are being utilized like i’ll actually i’ll show my no works here hold
on here um this is my test system i always feel like i’m inviting you into my house and i haven’t cleaned up so
that i’ve got a lot of garbage in my system but you know being able to actually go ahead and have accountability of like here’s like a
battle 6.263 um being able to say where is this being
utilized right i can show you every single touch point in which it’s actually been used but then i can also
go ahead and you know you want to understand well yeah that’s great you know what this dependency that i have as part of
my build system here well it also depends on all these 19 other indirect
transitive dependencies for the model i’m building so the understanding you know what
you’re bringing in and how you’re utilizing is very key to what you’re doing and the software building materials allows you to have
accountability a little safety from your organization by informing your customers of what you’re using how they’re using
it and the software that they purchased especially you don’t want to be that company that you know becomes a headline
so the big takeaway from this too is is also two number one misconceptions we hear this
all the time you know can it be used as a roadmap for an attacker not really i mean the thing is is that yes they can
get an idea what you’re utilizing maybe some versions but they can go in and infiltrate the community but that’s
outside of you that just happens right protecting yourself is essential
don’t have to require any source code there’s no source code required you do not have to do it unless it’s implicitly
stated by the legal teams of that company and that’s a negotiation that your legal team and their legal team
have to have together to say you know we’re going to put this the your source code in escrow we’re going to keep a
copy of it blah blah blah i’ve heard this numerous times with some of those pieces and you’re not going to expose any
intellectual property the thing is you don’t really you know the stuff that you’re building is the stuff you’re building it’s like like i said it’s like
having a list of ingredients without the instructions on how to actually put those together sure you could probably
try to figure it out but if you have all your algorithms protected and you follow safety protocols there’s nothing you
have to worry about so at the end of the day if you want to work the us government or a multitude of
governments in the industries you know regulated industries are now adopting this methodology you need this if you’re
going through and you want a complete list of all the free and open source stuff that you’re using when you build your software you need it you know if
you want to understand how what and where and why things were made this gives you the ability to go in and even
track things like velocity you know how much change are we doing between versions it allows complete
accountability at the end of the day it also adheres to software and license compliance that you might get from legal
teams um you know outside third parties auditors m as
you name it these are things that you want to be able to easily present if you’re required to
so that was a lot to throw at you in an hour and i’m sorry or whatever time i just did that in but i just want to let
you know that this is kind of the level of information on what a software bill of materials is um i don’t know if
there’s any questions or anything like that um let me see here oh i just see a
lot of people at greetings and free software but materials there we go is there any questions or anybody have any
questions for me in this respect i can go stop sharing so i can see faces so maybe i can just comment i noticed at
the beginning of your presentation you had the gitlabs hack and message was from ravi lakshman and by the way
actually ravi was presenting on last year actually in this meetup so
small world very cool but so yeah so that’s basically um you
know ways to utilize it ways to do it like i said protecting your customers protecting the software you produce and
then this is just an accountability aspect behind it so the software you know these are the formats that are out there and really it’s up to you as
developers devops engineers devstock engineers release managers of course to ensure security at different levels you
know the thing is is that like for us at jfrog we have our product called x-ray and actually it was just stored on
46:42 as being the top security product for this stuff i’ll take that as a yes that was pretty rad um but the thing is is that you know
being able to implore tools actually you probably saw it one of the things here is is that you know as a developer in
here i’m actually working on a uh oh here’s one of my cats i was talking about you might see its tail in a second um
but um you know these are like here’s a here’s a bunch of libraries i’m using right and being able to actually identify these
libraries you know as a developer this is the front line defense this is that idea shift left right here shift left
and ship right so this is really that idea of being able to say oh hey my software is utilizing these pieces and
as a developer i have a responsibility to go in this is a critical issue right this is a critical cve
or even being able to even go in and use a tool like this where we even have our like for us we have our jfrog cli and i
can say audit right and if i can spell audit today and in this case this is going to go through the project i’m
doing it’s going to pull in all those transit dependencies as a model right so it’s not actually bringing us bringing
them as a model and i can take this and i can display any potential threats and the thing is is that with a soccer build
materials and first you know shift left defense this is a great place to do it also integrating this level of security
into your ci process into your release process even into your runtime like we’ll be introducing a runtime analysis
down the road to say this component is running in your runtime right now but the idea is is that this accountability
is the you know the thing is you want to be able to map all the dependencies uh direct and indirect that you utilize as
an organization and they have a format for it so i hope
this has been helpful guys yeah hey absolutely yeah hey this is ahmed owen i work at adp
and i have a question i wanted to know what’s the best way to distinguish a vulnerability that came from
your base image versus uh your app code in a doc ah so when you’re looking at that right so
one way to do it is is actually to go in and in some cases the best way to do it
is actually i i’ve actually done an experiment like this where i’ve actually used two different tag versions right to
say like here’s a base image one here’s this or if you’re doing a docker compose you want to look at it so like in our
product just you know i know i know i’m plugging the product whatever um we have the ability to actually scan that base
level image before he is put into the docker compose right and there’s tools out there that you can utilize to scan
these images before you utilize them right so in our case we use our x-ray product to say i’m doing a docker
compose i want to scan this docker you know this docker base image before i even compo you know build it i can go in
and determine whether or not there are nefarious components in there but then also too you can also look based on the
base image and abstract that out and give it its own software build materials just so you know you can actually pull
the information out if you compose that and you actually could see all the baseline imaging in here like i’ll just
show you really quick if you don’t mind um let me um where’s my share there it is
um here let’s go back and look at a docker image for a second right let’s go back and look at my my docker image i
have here i’m going to show you my favorite view um yes i am plugging again and i’m going
to show you my favorite nerdtastic view that we show inside of our product but here’s an example here like here’s all
those base level here’s you know here’s my docker image here’s all those layers of the actual docker image itself
i could actually go in here and show you in our case i could actually abstract
all the actual you know information from that base in this case um i could actually show you all like
and this is a debian uh container um you know i can actually show you like the debian components that actually made up
this one and in this case you can even see here where i’m actually i actually have something nefarious in here with an issue
so there’s ways to do this but you gotta there’s certain you know only certain tools will allow you to have that insight also too depending on how you
actually build the docker compose you can also use the log function and have it show when you’re doing it and it’s
actually constructing this stuff you can actually pull that in for a while also there’s way there’s multiple different ways to do it
so bill could you then essentially like have the default behavior be that it will not build if
there is a vulnerability uh discovered in one of the dependencies yes in our products you can we actually
have the ability to create rules that will actually terminate a build even on ingest to say and flag a container as
actually being non-viable so and where is kind of the threat intel coming from with regards to the
vulnerabilities ah so in most cases it depends right so like you have all the standard nist
sources nist is actually this the relative source is out there for most of the software out there there’s actually
vulnerability lists that are out there um there’s also companies like rbs risk-based security called vol they have
you can actually go and search for it and you can actually look up a component and it can bring back those lists um in
our case we actually algorithmically and manually curate it and we also have um a team um that we have that we you know is
our security team that allows us to create cves because we actually create cves uh publicly um we’re actually one
of the one of the few companies i think there’s in total i think there’s just under 80 companies in the world that can produce cves and we’re one of them um
but the thing is is that you know that level of information on vulnerabilities like i’ll just i’ll just show you an
example um let me see here let me see i can find an example on like where those sources come from because i actually
could show you the sources themselves um you know what let’s go to another build um
yeah you know what actually let’s go back here um you know using sources like nist provides a certain level of neutrality where people can’t start
pointing fingers exactly so the thing is is that like i’ll just show you quickly let’s go bring up one right so we
actually have like you know here’s an example of the actual you know issue itself right here’s the link you know
the cve um you know and it depends i mean in some cases there’s really not much detail that you know that’s provided
like this is actually pretty terrible uh but the reference materials it depends you know in some cases you know it comes
from say like in this case like here’s you know issues that apache right this is actually uh you know this is actually
the issue that caused it right these are actually some of the publicly uh available components um you know there’s
other ones here you know from like you know maybe a source from github that you know goes ahead and says yes this is the
actual issue that it is here’s the cve component itself you can go ahead and look it up so it depends though i mean
every company handles it differently but there’s also sources out there and products that allow you to go ahead and
do the searching on this and also associate those binaries to those actual potential threats
and by the way always be cautious because you should always go ahead and look at the actual
issue itself and when you look at the issue itself there’s a lot of the time is that whole baby in the bathwater thing i’ll be able to say you know maybe
there’s a hundred functions in a library is this applicable to the usage that i’m
actually doing because in some and then you have to evaluate too is this an internal product or an external product
right and you know there’s a lot of things to take into consideration you know the thing is that most of the time
most of the most of the time most people are like it’s got a vulnerability check it out but what if it’s an external
remote source vulnerability and you’re using this on an internalized product or what if the fact is is that the actual
component defined in the csv is something that you’re not actually even using as a function so there’s also a
level of expectation in terms of developers to do a little research into this if something is found to say hey by
the way this really isn’t applicable to the way we should do it we should label it as such
i mean what i’m getting at is that this is really about visibility you know people have fallen the other way for way
too long saying i don’t know they shrugged their shoulders right so that’s right you know it’s right there it’s
right in front of you you had to intentionally bypass that flag or that warning to to complete this build
absolutely and that’s the thing right and that’s one of the things that we try to protect against but also too like you said there’s a there’s a
the you know the thing is is that there’s always that knee-jerk reaction of just disposal right um and that’s the
thing is like i said there is there are things that you know research that developers should do um
and things like that and like you said bypassing it you should be able to justify the reason right if you don’t
justify it then yes of course the you know uh you know they then you know they should be you know reprimanded or
whatever to say you know don’t do that you know whatever but oh i see jacob you have your hand raised
or virtually hand raised you’re on youtube oh you gotta mute
yourself i’m terrible at reading lips
okay there we go okay yes i am i have a question okay sure
okay now this this program this this program has been used for security right
well well part of it i mean our the stuff that we were doing yeah i mean our our product is actually more binary
management and security is actually just a feature of our solution
oh right okay but it is i mean it hasn’t gotten to the point
where like secure like the whole spectrum between your lock out your customers
uh the spectrum between the security and safety has has has
has a veered over to the level of security like um
is it do do you find that i mean because because i mean
i mean you you were all um you know very
uh technically uh your your technical prowess is strong but
for people who have um who who are having trouble getting locked over their bank accounts seniors
isn’t isn’t isn’t this isn’t this an issue in programming for excessive
security um no so that that access
there’s application level security right you know actually running the application and providing security for
the consumer right that’s more of like locking you out of your accounts and and things like that this is actually more
development level security um more you know security around not even getting to that point right
this is actually potential threats inside the software that might cause a problem right so like um you know you’ll
read things of like uh you know so and so organization leaked 450 million accounts right that was more
of an application level security um in terms of actual accessibility to the
contents or the actual data in the software locking out people from their accounts
and stuff is more of a safety measure from accessibility but what do you mean by leaked well
defined leap um so dependency confusion attacks right so
or sql injection where you go to a website you look at the website there’s a whole bunch of input fields on it in
some cases a lot of hackers what they’ll do is they’ll use something like net you know like uh there’s a lot of areas like
sql injector there’s a whole bunch of black hat tools that you can use trust i i’m not saying i use them but i’m just
saying that they’re out there but you can point them at a site and say show me a field you know show me a data field
and let me see if i can inject um a query into it so log for j why was logs
for j so dangerous this one of the biggest libraries used in the entire world it’s been used
for decade you know over a decade why was it so dangerous because you can inject information into a field right
that would go in into the log4j platform and then if one of this one well the
scary one that anybody can run is you can actually drop into the query field with the proper flag and say
in this case for unix command you could say rm dash rf slash
and you can throw that in and then what it would do is it would execute on the runtime environment and just wipe out
the file system right so you can actually have it go in with the proper pseudo credentials and actually it would
just wipe out it would just wipe out the directory just gone bye-bye right and this came with android phones by the way
so you could actually do this to an android phone um that’s why google was scrambling to put out a patch to
alleviate this yeah sorry i will need to jump in i know there are there is one question i don’t
know if you keep your your capable of answering it but okay
now the the hackers jacob could i just stop you for it for a moment
three three different categories the the checkup sorry we’ve got another
presentation so could i um so uh bill you you’re still here right can we just wait another i would say
30 minutes and we there’s a couple of other questions as well so maybe we can just do it after the
other presentation yeah i’ll stick around i’m actually gonna go get a refill of water will will he do his thing lily i’m looking forward to
hearing what you got to say buddy um i’m gonna i’m gonna go mute guys um and i’ll come back later i’ll be back i’ll be
here so thanks thank you very much all right so uh we are moving to our second speaker and
that’s willie shop and uh willie is not a stranger to this meetup uh i have a
pleasure of working with willie at worksafebc and he’s the manager of common engineering practices
willie knows um all elon musk’s quotes and uh both of them
share common interests so today he will be talking about his experience in setting up common
engineering practice at workcpc will he over to you thanks andre and can you just allow
sharing for me as well somehow that just disappeared oh really okay disabled yeah
let me just do that
all right let me just try to do that interesting sharing it that doesn’t work for you
no it’s his host disabled screen sharing oh that’s not what i was supposed to be
okay well it gives bill time to get his water
it’s all part of the script okay here we go
can you try now
here we go okay here we go you should see the screen now yes
and i assume bill is back great so good evening and afternoon
morning and welcome from wherever you’re joining us at today’s devops meetup and a special shout out to you initiates
from brazil it’s really great to see you it’s it’s my pleasure to to give you a
quick update on our efforts to establish a common engineering ecosystem at
worksafebc so let us start by taking a peek at our
roadmap and what you’ll see is as shown at the bottom is it’s based on
quite an ambitious vision to empower in every engineer by standardizing our
engineering practices and tools and enable continuous delivery of value
to delighted end users so from the early 2019s
we operated undercover i heard star wars today so we operated
like the rebel alliance on the ice planet hearth kind of out of sight out of trouble
and we focused on creating manifestos and guardrails that enabled engineering to build
consistent secure and simple solutions and you can actually refer to
the common engineering system at worksafebc meetup session that we delivered back in april 2000
for more details on those ic times we also innovated our continuous
integration and delivery pipelines which was another meetup session we
shared last year in june and released our works fbc technical blog which now allows us to share all
our failures learnings and innovation with the community
back in september everything changed for us because we got the opportunity to launch
a common engineering team with a leadership mandate to empower every engineer by standardizing practices and
products and stewarding guard rails again to promote engineering consistency
enablement security and most importantly simplicity
so for the first 90 days we focused on establishing an efficient system of highways and that
enabled both our business and engineering and ensure that our common engineering
ecosystem had a common and sound vision a vibrant collaboration and an
atmosphere that encourages failure learning and continuous innovation
and we achieved three major milestones
as shown the cell service automation which we often refer to as the walking
skeleton it creates an azure repo injects an app-type sample
configures and injects a yammer-based azure pipeline and cues the pipeline as good measure
for final validation of the setup all in less than 10 seconds
if that doesn’t make the developers happy i don’t know what will
we looked at working agreements also known as cheat sheets to guide how to work together and to
create positive productive processes and
we started a war on on a little bit too fast we started a war on waste and silos
again by fostering collaboration through what we refer to as centers of enablement
working groups dojos community of practices all of which you can read up on on our technical blog
now for the next 90 days or so we plan to switch to implementation mode
we have the highways now let’s start implementing and our plan is to
enable and foster steward guard rails and collaboration so that for us is
foundation it’s all about collaborating with each other we want to monitor and learn from
baseline metrics and we want to open source our application type yaml based
pipeline blueprints and automate everything automatable
thereafter i think let your imagination saw we are looking at things like chaos
engineering which will scare operations to no end but that is
fine the distance we should be getting ready for chaos engineering as well
so that’s our road map that we’ve set up for ourselves
here’s a quick snapshot of the extent of collaboration we actually achieved during the first 90 days with our
centers of enablement working groups relentless banging on virtual walls
twisting of rubber arms and stubborn push for complete transparency
so if you look at the diagram the variety of teams
the vibrance of collaboration we achieved and the focus on outcomes gives
me the reassurance that we’re on the right track and hope that we can actually achieve the unthinkable as an organization
erasing the silos avoiding them to regrow like weeds
establishing trust and getting all the stakeholders to collaborate
has been will probably be one of the biggest tests for our common engineering ecosystems today
which brings me to my personal learnings from the challenge of launching such a team
especially when you are like i a software engineer at heart and you want to remain involved hands-on with the
continuous research the learning and the improvements so apart from the people management
dealing with ridgid at times mind numbing and time-consuming processes
um a lack of skilled resources in the local market which we’re probably all feeling
our team also has to remember our responsibility of operational support
and quality assurance so those are two other pillars that we can never forget
so balancing these with our quest for bold change and continuous innovation
has and will be challenging so creating a harmonious team
where everyone takes responsibility for their action where we have a work-life balance and
where everyone is a leader is definitely not a walk in the park
i am constantly worried about everyone in my team who are we are like you in tough and
trying times where remote work isolation and the lack of social contra contact
is not only taking a toll on every one of us but in my opinion maybe rewriting our
rule books for business engineering and work psychology for the future
an anomaly that keeps me awake at night there’s many but this specific one
is how to ensure that everybody has an innovation mindset
has the time and the grit to take a chance i find that many engineers can spend
hours talking about a problem but often they have no time to actually solve the problem so they will spend
three hours telling you what the problem is all about and when you say let’s solve it they say
i haven’t got any time now i have to go back and write out so hopefully we as a community and i’m
hoping we can do this as a devops community can collaborate to create some helpful guidance for for the community
around taking a chance being open for failure
and also this new working environment which some enjoy
and some find extremely stressful
so my personal mission statement which i stare at every morning is to encourage
everyone’s creativity passion purpose and strengths
i’m also experimenting with a variety of guidance for example from
david markey carrying around his book then turn the ship around and you can actually ah
there you can see it like this is my my holy book that i carry around all over the place
like wherever i go because it’s a real leadership success story that is really really worth to
read i also remind myself to foster autonomy mastery and purpose as
just discussed by daniel pink and as mentioned by andre i’m i’m
inspired by the spacex phenomenal innovation and
i use kind of elon musk’s rules to simplify everything
encourage my team to automate everything automatable to question every process
and we often provide colleagues by asking the five ways or simply just ripping out a part of an
inefficient process to see what happens and last but not least i remember or
remind everyone to to have fun
and that is to combat today’s stressful days of isolated and remote engineering
and to remind us of the need to balance work and life
and it’s probably a reason that our most recent quarterly update looked more like a newspaper than a
formal report we wanted it to make make it fun and i think it worked
um andre you can say later on whether you agree or not but we got some really positive feedback on on that report so
we’ll have to find other cool things to do i also work hard to create a mindset to
continuously experiment and embrace failure as an opportunity to learn and to
innovate and you’ll see soon see this round red sticker appearing on all our
laptops and that’s really to instill the courage to fail as one of the pillars of our healthy
devops mindset now what i’ve learned is the challenge is to avoid the fear of failure amongst
engineers all stakeholders in an organization must trust the engineering process
and embrace failure and as an opportunity to innovate
and the best antidote to toxic fear of failure or leaders who are supportive
and inspiring of which we have many plus a collective blameless fail force
mindset if you have that in place we can start experimenting and we can start
learning or as elon musk says and i quote failure is an option here
if things are not failing you are not innovating enough and that came straight from the spacex
launch site where they are truly experimenting
so to be successful we need a dream team that is continuously invested in
and focused on operational support quality assurance and innovation
so we have a team of engineers focused on operational support and innovation
and another team of engineers focused on quality assurance and innovation
now this is where my next challenge is coming in to create one team mindset
aligning the innovation of both those teams within the team
and i’m pretty sure it will continue to be a challenge again in today’s remote and isolated working
model because you can’t just walk over to a desk and say hi bill how you doing today
house quality assurance the day of quality assurance going you cannot do that anymore
so i believe that the secret sauce that holds us together especially during these trying times is
distrust so you have to trust each other you have
to trust your leadership your stakeholders your end users and vice versa
so it’s the whole chain and back
so as mentioned on our technical works fbc blog and i’ve mentioned brief
before as well we live and breathe by our working agreements
with an emphasis on optimizing meetings which is another one of elon musk’s muses
and inspire an agile kanban and devops mindset as well as the five-day obscure
values so the working agreement pinned to most of our walls
reminds us daily what we value steward optimize and inspire as one team
and our team working agreement has actually triggered a range of other working agreements such as starting a
center of enablement or even rotating through our company engineering team
and again i invite you to have a look at our technical works fbc blog or ping me if you have any questions on
any of our working agreements
all right so to wrap up this quick overview i just wanted to put all the links of
information i’ve referenced in today’s update i hope that our journey inspires you
and that you’ll actively collaborate with us on twitter and linkedin you’ll find us in both
places and with that thank you for listening
any thoughts or questions
oh yeah i will say man i i did love the fact that you didn’t quote the uh elon mustang another one of my favorite
tournament one of my favorite quotes is from marcus aurelius um the obstacle is the way
right so always look at every obstacle in this place as not a uh you know
something as a failure but look at it as more as a learning experience for a better solution right so whenever you
come up against that the stoics had it right in that respect of being able to say that or you know as i say the people
on my team is nothing’s impossible it’s only improbable yeah
right yeah really i’m curious to hear how you uh first of all i really want to apprec
i really want to mention how much i appreciate you recognizing the value of failure you know and that failure is a necessity
in order to succeed um you can’t have light without darkness etcetera etcetera um so how did you then
sell the value of failure to leadership
oh still busy selling the value but we actually as i said is we actually have great leaders who are supporting us
um with for example failure and what we started doing is we literally just
started putting the quotes on our back on our
laptops it used to be great when you were actually at the office and you could have all the stickers of
importance on the laptop but that’s where we actually started advertising the need to fail
i started kind of quoting the likes of elon musk with our leadership
um anja you know that very often in the morning on teams when you all say hi to each other we i have another quote from
elon musk and a lot of his quotes are actually around innovation
i guess for a reason and he’s strong push for failure as being a necessity
to innovate um so it’s we’ve been pushing that message
yes with some managers we are still pounding our heads against the wall because you just have to get through
that wall or through that silo but i think once engineers actually realize
that it’s great to fail because now we can learn out of it and work together
you actually see a change in the engineers so i don’t have a secret source other
than share that message as much as you can and support your team
if there is actually a failure don’t blow up and say we have a disaster
no great we have failed so it’s the kind of the way you respond i love it we have
failed so what can we learn from this let’s get together let’s have a virtual coffee and let’s discuss what happened
so it’s again how you work with your team and kind of how you react to
to the failures really it’s really quick i’m sorry i’m just rejecting it again so i’m actually
i saw i’m a mentor at tech stars uh 500 startups y combinator um i’ve actually
been work i work with a lot of companies uh over time um i’ve been doing this for a couple of years i’ve had three exits
i’ve actually taken two companies public all these kind of things and so i spend my time talking with you know basically
you you don’t know if you don’t try and if you fail you reassess and move
forward right do not look at the hindrance but always go forward and the thing is that actually you learn your
best viability from those failures themselves right you know they actually you know if you succeed all the time the
problem with that is is that you’re probably not doing it right to be honest you’re not taking the risk you’re not taking the chance and in a lot of cases
if you do fail um you know the thing is like you said you brush yourself off you move forward and you say what did i
learn from that and actually failure is always the best teacher in my opinion um and when i i actually i was down like
you said the elon musk thing so i did a bunch of work in hawthorne um at the spacex manufacturing facility uh where
they make the rockets and um i was working with their teams down there and they have this work all those wordings
spread all throughout the facility everywhere you look there’s words of of that you know quotes and stuff and we do
the same thing at the jfrog office our jfrog office is filled with all these quotes to kind of reinforce the idea
that we’re collective that we do fail and when we fail we just brush ourselves off and move forward um you know as a
team collectively and our job is to pick each other up and work with each other to move forward
i guess and loads of words of wisdom sorry didn’t mean to interject like that
but i just wanted to say it’s like i feel everything that you’re saying and i totally wholeheartedly
undeniably a billion percent agree and i’m so jealous building that you were actually at the spacex center
so you’re one you’re a gazillion miles ahead of us the other thing i want you to just um
throw in there is with failure as well if we actually combine it with experimentation you’ve
got the winning formula i think selling experimentation to an organization is much easier than selling failure
and then kind of again like on the ice planet health you do it undercover you actually
explain to to the stakeholders that yes we’ll do an experiment and then we’ll get to a point where we’ll decide if
this is okay or not if it’s not okay well we failed we’ll rotate around and do another experiment
and you continuously doing experiments failing or succeeding and that is another way to actually sell
the value of failure is combined with experiments
excellent thank you as a matter of fact there is a lot of other quotes in chat i think what we we
are going to do we are going to collect all of them and we are going to post them somewhere so you know that’s
actually a a great event today because it shows the right mindset right actually everybody
here in this uh in this meetup has a right mindset which is which is great to great to hear
okay do we have another question for willie
if not then i would like to close the official part of the of the meeting and
uh bill if you don’t mind a couple of minutes in willy there is there there are a couple of other questions as well
so uh so we can maybe have a quick conversation but first of all i would like to say thank you ari are you
already left but bill this was really fantastic presentation thank you very much and
willie obviously uh you’re doing great job with implementing and and pushing the envelope which is extremely hard i
know it’s a cultural change in organizations so uh thank you very much for this and thank you to all the
participants as always if there is any topic that you would like to talk about you’d like to share your experience um
anything that is related to devops and the product that that you have please do not hesitate to contact us and
we’ll schedule you for one of the next meetups so thank you very much everyone
and uh have a great evening thank you thank you everyone for having
me i appreciate it really and andre guys thank you so much willie amazing presentation thank you thank you so much
for your your time in dealing with me [Laughter]
thank you so um uh lorena i think you had a question right
one of the questions yeah you’re looking to say is there a way to go find log for j components right is that what we’re
saying lorena so yes there are different multitudes of ways to do this right of course um i mean i’ll just show you an
example uh when this came out we were one of the first responders uh to this so like we even went out and we actually
produced a free open source tool for companies to go ahead and actually you know point this to the software that
they’re utilizing i’ll put this in the chat for everybody just to kind of see but we actually released this free to
the public without even our customer base because we believe in the fact that all software should be safe and secure
whether you use this or not we don’t care um i mean we care of course we care i mean we like customers you know we
like rev you know we’re a company right um but at the same time and in addition
to that there are other ways to do this too so like even in our product too there’s also search factors you can go
ahead like i showed before you know every touch point that it came into in this case this was just a cli tool
that we released that you can point at your software and it’ll build a component graph around it so you can find out whether or not um that is part
of the solution that you have all right um i think there’s some other questions
too right yeah ahmed isaac hey guys oh hey thank you andre for listening i
just wanted to ask if jfrog had the ability to mark a vulnerability as risk accepted
yes so you can actually go ahead and i’ll actually i might as well show i must keep my screen up uh let’s go look at
you know like say any of these builds here i’ll just go look at a you know any sort of build that might be here with an
exception um there’s a couple different ways hold on let me go in here let me bring up any of these so there’s a
couple different ways you can handle it in jfrog you can either say create an ignore rule or you can say i want to
ignore it based on the license it was a licensing one sorry it could be vulnerability or license compliance uh
the component the build the watch the watch is how you actually detect it it’s a rule set you define
uh you add in the note and then you can also say you want to have it for an x number of days right so there might be
times where you actually say you know what we want to continue on this journey we have a release coming up we’re not going to address it just yet we’re going
to ignore it now but we want it to come back in effect so i want to come back on february 4th i want to be notified
that’s one way another way that you can utilize the two is that you know when you have any of
these components like i’ll just go find just want to show you a high level component in this case oops
i’m using a beta version so i probably should have picked one that was actually i’m in the middle of debugging a version
oh let me turn off the debug i’ll i’ll do it afterwards i don’t need to do it right now um but let’s go in and uh look
at you know any of the other components that might be in here do i’m not scanning that one of course let me find one i’m scanning um we’ll go back here
let me go in and yeah i’ll do this one this will be good i’ll go find the descendant let’s go in and look at like
say a vulnerability of any type even one here whatever here so i can go in here and look at a component and i’m just
doing it from the easiest place possible i can say okay i want to assign a custom issue now this custom issue allows you to you
know give it a reasoning behind it uh you could change its severity if you want you can give some sort of type
justification you can even save for the single version of it or a range of versions in which you want to do it and
then you can go in and add additional metadata properties like uh you know we we’re going to accept this right now but
let’s go ahead and um let’s let’s flag it right let’s label it and then i can query on it later and say um show me all
the you know in this case we can go to like artifacts here i can say i want to do a search i want to do a search on
properties i want to do a search on key value pairs where i say you know review later
true right i can spell true um do a search and that will find me all the you know
components that i flag with this metadata property and then i can go in and you know from in here i can even
throw exceptions in also so there’s ways to do this and i can also produce reports on whether or not i had you know
any violations that i’ve gone and ignored and you could also do the same thing from here also by the way um of
course i picked one with no violations of it but um you know the ability for you to go in here and and actually say
you’ve accepted this and then you can query on it afterwards and you can do that from the cli also
you can actually go and use our cli if you don’t want to go into our user interface everything i show you is api-able too
ours is just a simple view interface on top of our api i’m not gonna there’s no magic
thank you no worries um let’s see i think there was any more
i’ve only got a minute or two left unfortunately guys i do need to go uh i eventually need to eat um
i’ve been i’ve been on i’ve been on the phone since 6 30 a.m that’s the problem with having international i actually have a phone call tonight at 11 pm with
my my office in tel aviv um so let’s see here um
yeah uh any other questions or anything guys if you want to ask me ask me now yeah maybe last one question
sure we all have to eat i’ll song and dance for you guys if you want a little salsa
no we’re good if not great thank you very much again thank you bill thank you willie thank you
everyone thank you guys okay thank you bye-bye be safe