Software Bill of Materials and its impact on Enterprise DevOps @DevOps Vancouver Meetup

0:00
okay all right so let’s start um so uh thank you very much uh good evening my name is
0:06
andre kaminski i’m organizer of this meetup and this is the first meetup uh of this year
0:12
end of last year we introduced a couple of changes i wanted to thank alison bennett who was
0:18
the co-organizer for the last 18 months and she was helping me with with leading those sessions
0:26
and i would like to welcome ilsa bishop and willie shalp who will be helping me this year
0:32
our intention this year is to continue finding interesting speakers companies and products and try to experiment with
0:39
different delivery formats today we are going to have a two interesting presentations but first i
0:45
want to say a couple of housekeeping items so first of all this session is recorded please keep your mics muted
0:53
after each presentation will have 10 minutes for questions you can type questions anytime in the in the chat or
1:00
unmute your microphone if you want to ask questions directly all right so as you know devops has
1:07
different flavors and sometimes some people refer also to concept of death sec ops and and to me
1:15
this doesn’t really make a lot of sense because honestly speaking security is part of
1:20
delivery one way or another right this is not negotiable part of non
1:26
of of functional requirements non-functional requirements right and um
1:32
as a devops we build applications and we operate those applications and we often assemble applications from various
1:39
various vendors and some of those applications of those components
1:44
are including for example sometimes open source directly or indirectly if we implement a component
1:51
from a vendor that might be using some of those open source components we might
1:56
not immediately see those but since most of the applications um currently are really in the cloud
2:03
delaying updates to those applications is no more an option um applications
2:09
becoming more and more complicated and either because we try to keep those
2:15
applications secure or stable due to uh changing underlying software software frameworks
2:21
we need to keep that track of those components so today our first topic will be related to
2:29
exactly to the subject and i would like to introduce and welcome ari waller and bill bunning from
2:35
ari over to you thank you so much i really appreciate it hi everybody my name is ari waller and i
2:43
am the meetup event manager for jfrog and we really really appreciate the warm welcome uh tonight here at the vancouver
2:50
devops uh meetup community um i’ll share just a little bit of who we are uh jfrog is a devops software
2:57
company known best for artifactory which is considered by many to be a gold standard for managing your artifacts and
3:03
dependencies um in fact one thing a lot of people don’t know yet is that jfrog has a free cloud version of our
3:09
artifactory product for the meetup community it’s not a trial version that runs out but a free version
3:14
it’s great especially if you’re working with docker as i know many of you are jfrog’s cloud subscription is capable of
3:20
functioning as a pull through cache for docker hub and because of the partnership we have with docker you’ll
3:26
be exempt from any rate limits on free or anonymous uh free pulls from anonymous accounts so that’s something
3:32
that could be valuable now i want to share my screen with you we have a something special for uh
3:39
the vancouver devops community tonight um let me get to oh let me get to these where’s my
3:46
spy that’s not good uh one moment please
3:52
you know you think you would think that after all of uh you think that after all of this
3:59
this experience we have now with all the virtual meetups here i go uh this would
4:04
be so simple um but uh it’s not so here we go hopefully
4:09
everyone can see my screen okay now um but uh let me just grab one thing here and you
4:16
should still see it but um see everyone see the slide okay yes
4:22
great a bonus for you tonight is since our topic uh is um around um
4:28
something technical we want to of course give you a technical treat so jfrog is going to give you a chance to enter to
4:34
win a raspberry pi 4 which if you try to procure these days it’s not the easiest thing to procure i have to order them
4:40
only there’s only a limit of one if you can even find one um so this is one that you can get your hands on and play with
4:46
a little bit um you can enter with the qr code or the bitly link that you see and i’m also going to drop it in the
4:52
chat i can’t do live sweepstakes uh online for compliance reasons however we will
4:59
randomly select the winner within two business days we will contact that winner via email and once the winner
5:05
claims the prize officially we will go ahead and share the winner with your meetup community as well but we want
5:12
everyone to be a winner tonight or ever almost everyone at least to have the opportunity so for the first 20 people
5:18
that do enter um that enter the sweepstakes we’re going to give them a
5:24
jfrog t-shirt and a liquid software book uh we’ll send it to you um just uh just
5:30
for just for entering to the first 20 people so um i’ll go ahead and drop that link in the chat and uh
5:37
i’m looking forward to hearing bill speak tonight um and i’ll let him do further introductions but thank you uh
5:43
andre and uh team for having uh having us in your community tonight we really
5:48
appreciate it bill i’m gonna turn it over to you and stop sharing excellent hey guys how you doing um i’m
5:55
bill manning i’m gonna share my screen here and let’s just kick this off uh let’s get this shin dig go in here uh
6:01
let me move the little uh annoying zoom window out of the way so it’s not covering up all my stuff um all right so
6:08
how you doing guys so tonight um i’m gonna introduce you guys to s-bombs and uh you know software build of materials
6:15
and uh basically the impact on on enterprise devops right this is a become a huge talking point uh especially for
6:22
me over the past year um with for a lot of different reasons um i’ve done this talk um not this exact talk but similar
6:29
talks to this for multiple times uh both live at like reinvent and some other security things and uh today i’m going
6:36
to present it to you so um i am uh one of the solution architects slash also the manager of uh
6:42
solution engineering for the americas for jfrog um as ari stated we are a uh you know we’re an international global
6:48
company uh we have over 6 300 customers 70 of the fortune 100 we went ipo last
6:54
year um you know we are involved in everything from uh we’re officially like a cve provider we’re also members of the
7:01
cloud native foundation um in terms of governance so working with a lot of you know under you know basically coming up
7:08
with the standard that most companies can utilize to do sort of the cloud native approach um there’s a multitude
7:13
of things that we’re involved in i’ve been with the company for about five years so you can follow us at jfrog on twitter or even follow myself at uh
7:20
william manning um but today you know the importance of this is is not only talking about software build materials
7:26
but what really what really led up to it and and really what it means uh in terms of organization i’m also going to give a
7:32
demonstration and show you like how we handle software build materials and i’m also going to talk about the two
7:38
different formats that are currently being utilized right now and i’m actually going to demonstrate some of those formats to you so you understand
7:44
what it entails so we all see the news all the time right i mean really the software build
7:50
materials has been a thing for a while but it’s really become more apparent when we see alerts like this you know
7:55
something from hacker news you know that’s always a place where i always go to make sure to see you know what’s interesting out there you know what
8:01
exploits are happening um but the thing is though is what we’re going to deal with today is we’re going to talk about supply chain security right because
8:08
supply chain security in our software development is really what came about to really drive the software build
8:14
materials idea and concept forward especially in the united states i mean we see these headlines all the time um
8:21
there’s nothing new here right i mean it’s constantly even a couple weeks ago i should have added in here a lot like what two weeks ago the massive log for j
8:28
thing right i mean one of the most commonly used libraries across the planet also happened to be one of the
8:34
most susceptible right and the thing is is that last year alone in 2021 there was a 650
8:41
increase in supply chain attacks i mean that’s insane when you think about it right i mean the thing is is that you
8:47
know most people’s exposure to software supply chain attacks you know they’ve been going on for a long time it’s just
8:53
that it became more publicly acknowledgeable um when it came down to solar winds right solar ones was really
9:00
the catalyst for a lot of countries and a lot of corporations and a lot of industries to really reevaluate the way
9:07
software supply chain is actually handled i mean when you look at saw you know solarwinds itself it affected 18
9:14
000 customers globally right it was a multi-billion dollar remediation scale and the thing is is that it actually
9:21
broke a trust model and the trust model is is that when we start talking about being developers or devon you know deaf
9:28
engineers or doug you know uh you know devsecops engineers release managers qa engineers the thing is is that when we
9:34
go down to his lowest factor the one thing that’s important to recognize is is like i mean actually you see from the
9:41
white of my face here you know i’ve been doing this for a long time you know i started off as a software engineer and
9:46
did i’ve done a multitude of things from being a software engineer to cto and founder to vc to what i’m doing now but
9:52
the idea is is that when you’re looking at this we have this inherent trust model that the things that we use to
9:57
build the software we produce uh was a trust it was a blind trust you know we would bring in these libraries that we
10:03
need to do our job easier um you know we need a function to go ahead and you know do something with a string you know go
10:09
find something that does that and the thing is though is when you bring in those libraries and we start talking
10:14
about this remember that library comes in with all of its friends and you know it’s like throwing a party and you
10:20
invite one or two people and next thing you know there’s a hundred people at your house and you have no idea who they are or how they got there
10:26
but solarwinds was really was the catalyst because in the united states alone it affected the department of defense department of homeland security
10:33
the federal reserve bank you know these this infected so many systems you know out there and it was a third-party
10:40
transit of dependency attack and it wasn’t even a direct dependency attack it was an indirect dependency attack
10:46
meaning was the dependency of a dependency of a dependency that came in and why are software supply chain
10:51
attacks so on the prominent rise right when we talk about this and the reason why inherently the model is broken and
10:58
the things need to be put in place to ensure safety and security not only for
11:03
you know what you do as a job but for your company and the people that use your products and stuff because let’s
11:08
face it no company wants to be a headline right no company wants to be like x company got in you know
11:15
infiltrated based on a software attack you know that was you know blah blah blah and you know next thing you know
11:21
there’s there’s you know people are running and panicking and trying to figure out what happened right and the
11:26
reason why is is that number one is super low effort right it’s completely low effort compared to most coordinated
11:32
you know other attacks that you might have on like you know ddos or you might have like a network infiltration or you
11:39
know all the typical sql injections and all that because those take a lot of effort what this doesn’t really take a
11:44
lot even the technical skill required is pretty low i mean the thing is is that associating yourself to uh you know uh
11:51
you know putting in some sort of malicious code into some sort of third-party transit dependency that gets
11:56
the site you know basically loaded in like a side car is insane and the thing is it’s super high speed to spread
12:02
that’s the insane part because the thing is is that once you’ve actually introduced it into the system you know
12:08
it’s there and the thing is is that once again it abuses the relationship uh between you know companies trust and
12:14
this you know the things that they produce and there’s always this little nagging thing in the back your head after you know you’re reading these
12:20
things that says am i really i mean can i really you know cause something completely terrible to happen with my
12:27
corporation or maybe even just a project i’m working on because the thing is is that the thing
12:33
is is that these hackers you know these people who write this malicious component you know they just blend into
12:39
the community they’re trying to become part of the community they add to it they contribute to it and somewhere down
12:45
the line they’ve introduced something that could be potentially threatening so when we look at the way this happens
12:50
is that as a developer and like i said this is all leading up to the reason why accountability is so important is is
12:57
that you know as a developer you go and you have to code something you go ahead and no matter non-dependent on what
13:02
language you do you start pulling in a bunch of transit dependencies that help you do your job better right you produce
13:09
your code and you know what happens if one of the packages you have in here is malicious you know next thing you know
13:14
it’s it’s part of what you do it’s part of what you distribute and next thing you know you’re putting it into a web
13:20
service or you’re installing it on the device i actually give another talk i gave a couple like a year and a half ago
13:26
it was called like if code could kill you know basically like bad programming and bad libraries that cause
13:32
unforeseeable tragedies you know on things that have happened but the thing is though is this is not only an erosion
13:38
of the actual software component itself but also you know the reputation of a company but most of the time it’s not a
13:45
direct dependency attack that actually causes the problem the dependencies you have depend on other things and if one
13:52
of those components is nefarious it gets dragged along with that same you know that same level of expertise and next
13:59
thing you know you’re at the same place right you’ve introduced something into the stream of what you do and like i
14:05
said you know developers are inherently trustworthy we’re kind of blind trustworthy in this case by saying you
14:12
know these are the things i need to accomplish my tasks meet my kpis meet the deadline in my sprint i mean i can
14:18
go through the all the different things right but the thing is is that these are supposed to make our jobs easier now
14:23
there’s a level of doubt associated to it because 85 85 to 90 percent of your
14:29
software is someone else’s code base right it really is i mean the thing is is that when you think you’re developing
14:35
software and you’re building it 85 to 95 percent of that you know 90 of that is someone else’s
14:40
99 of that contains you know is mostly open sources 75 that contains at least
14:46
one vulnerability some vulnerability that could potentially affect the way that your your software operates and
14:52
also the security behind it on top of that 49 of that code base that was analyzed has one
14:59
high risk component and when i say high risk opponent if you’re familiar with cve which is kind of the standard or
15:05
cvss score which is also the other standard is version two and version three um you know there’s a multitude of
15:11
different ways of factoring in the viability you know the vulnerabilities behind this contain at least one item
15:17
and then the thing is though is 90 of the applications out there have that pieces in it that are either old
15:24
out of date or completely abandoned so we’re just saying oh you know what this thing hasn’t been updated since 2014 i’m
15:30
sure it’s fine um you know i mean that’s just the way it is you know we bring it in and most of the time we just check to
15:36
see and you know maybe if there’s a problem we go check stack overflow and we look for a bad example we try to
15:42
retrofit into what we’re trying to do we spend more time messing around with that i mean that’s just the way it happens
15:48
but the thing is is though 74 of this can be actually alleviated by just simple updates simple updates right so
15:56
in other words taking a little more care in the actual transit of dependencies that we utilize as developers and this
16:02
carries over into ci processes because now with the modern day you know when now with dev apps and devsecops i mean
16:08
you can use them interchangeably you know the idea is you know a lot of these people do bin ops or git ops that
16:13
whatever your organization does you know a lot of times now it’s more automated processes so even it happens even out of
16:19
the purview of the actual developer doing the job that they do you know they work on their code they compile it they
16:24
send it off to a ci server maybe run some unit tests brings back some data whatever and a lot of times we’re like
16:30
you know what i don’t want to update that library because potentially it could ruin what i’m doing so what software attacks is you know
16:37
there’s a lot of different things out there a big one is you know typo swatting being more responsible for the
16:42
things that you do because there are people out there that create code and your libraries for your code that might
16:48
be an i before e except after c you know they might just slip around one of the letters or one you know and the thing is
16:54
is that you accidentally bring in something you thought was an actual library that you wanted to use but it
17:00
was a typo and next thing you know you’re introducing something malicious into it another one is dependency confusion
17:06
dependency confusion is so prominent it is unbelievable the thing is is that we
17:11
go out there we do a search and we say you know hey this is
17:21
oh there we go um so um you know the thing is with dependency confusion the idea here is is that you know there are
17:28
these private repositories out there you know like netflix has them right here’s some helpful libraries you know paypal
17:34
for authentication purposes maybe some other high security you know apple you know there’s all these ones out there
17:39
and how do you know if they’re the true thing right how do you know if they’re truly from the actual source itself we
17:46
have an example that i did where you know somebody was like we went through it and the thing is is that for what
17:52
they were trying to do for their software they’re like hey paypal has an off thing and it has an analytics piece
17:58
i’m going to utilize it you know it’s got a write-up these are fake these are not real somebody actually went in and
18:04
said they’re going to go ahead and use you know they created these paypal um style dependencies you bring it in and
18:11
suddenly there’s this massive you know of data leakage that could be happening you know we found this out with the actual off paypal right it wasn’t the
18:18
true auth it actually took credentials and sends it off somewhere randomly that could be this extracted and turned to
18:24
the next wave of attacks so the united states government and this is actually one of the direct things
18:30
that have happened this is actually now going globally was back in may of 2021
18:35
the biden administration signed the executive order improving the nation’s cyber security yes i had the unfortunate
18:41
task of having to go in and read this um and i will say i i fell asleep a couple times definitely because it’s not
18:47
exactly the most enticing uh thing you’re gonna read but one part of it really stood out and the part that stood
18:53
out was of course section four enhancing software supply chain security this is huge this meant that the
19:00
government had actually taken the time and effort to take this seriously right and this is actually you know you know
19:06
canada is going through the same process uh the eu is going through it uh it’s going through in britain this is
19:12
actually becoming a standard now globally but the important part is is that if you deal with the government the
19:18
government agency and this is now by the way avi the avionics industry health and medical
19:24
financial institutions are now following suit not only the governments and what you have to do is if you produce
19:31
software any software that you sell somebody can request a software bill of
19:36
materials and today i’m going to explain to you the reasoning behind this and how you can actually utilize this
19:43
so first of all it’s just what i’m showing here first of all software builder materials is boring in a way and
19:49
so we’re going to talk about cake today instead so first of all when you look at
19:55
about you know a cake box or any sort of food you eat one of the important things is you get a list of ingredients and you
20:01
also get a list of warnings and the reason why i’m bringing this up is is because actually the company the
20:07
actual group that proposed the software bill of materials was twofold
20:12
it actually has to do with the fact is you need to know what’s inside your your cake that you’re eating right you know
20:18
it has a whole bunch of ingredients right and this is really what a sovereign bill of materials is it’s the
20:24
list of ingredients because the national telecommunication information administration
20:29
in conjunction with the fda and then the associated relevant industries in other countries got together and said this is
20:36
back in 2018 and said we need to have accountability for medical devices we do do this for food
20:44
and the fda in america said we need to come up with the food and drug administration so we need to come up with an accountability aspect of the
20:51
software that gets utilized in medical devices think about it right you don’t want to have some sort of nefarious
20:57
codes sitting inside your pacemaker or your ventilator these days for the past two years god i mean right i mean these
21:04
are things that you need to have so this became a standard they started proposing what would it take and what does it mean
21:10
to actually have something that is safe secure and accountable
21:16
so basically what it is it’s just a list of ingredients that makes up your software this includes all the libraries
21:21
used whether it’s paid or proprietary right it also talks about access what kind of access controls does it have you
21:28
know this kind of level of information additional information that is beneficial and i’m going to talk about
21:33
that today is also to what tooling was used you know what environmental and system of variables were used when you
21:39
were constructing it are there specific settings that you have are the specific versions that you’re utilizing when
21:45
constructing your software basically it’s a map to how all the ingredients
21:50
that you have in your software and it’s a way to provide it to your customers to have that accountability aspect i know
21:56
there’s tools out there that parse through the information and we’ll talk about that soon well what is it used for right
22:03
so in a lot of cases it’s actually used for maybe maintenance right and i’m going to show you an example of this
22:09
today like you know we talked about log4j we talked about you know these kind of components and how you can utilize this information to help you
22:16
know safeguard your customers against things so in other words say there is something that varies you can provide
22:21
them with a level of information to say that by the way yes we shipped you code and we’re sorry but it contained this
22:28
information those companies too when they read those warnings can go in and search and
22:34
contact your company to say hey by the way we read an article that said xyz library
22:40
is potentially threatening because it actually can go ahead and leak all of our customer data and according to
22:45
software build materials you guys have been utilizing it it’s a safeguard it’s also a level of communication that’s
22:52
never been done before between the customer and the provider which is actually very important to increasing
22:58
the level of awareness of this sort of level of attack on top of that it’s also used for
23:04
pre-purchasing and negotiations in a lot of cases you could go through a legal process selling your software to a
23:09
company and they go through it and everybody goes through and looks at the you know the terms and conditions and
23:15
all the stuff this will be that next level to say hey maybe we are purchasing your software and we have a certain set
23:22
of safeguards in place of certain things that we don’t want to use or say you’re a company you’re a small startup you’re
23:28
getting acquired uh by another one you need to provide a software build of materials for the software you’re using
23:33
i actually had to go through this where i actually had to spend six weeks with the guy i was working with this is like
23:39
back in 2010 we were getting acquired and we had to go through and detail out all the things that we used to build our
23:45
software and then we also found out that we had to make changes because that company didn’t accept what we were doing
23:52
on top of that it’s also a way to also involve management and other asset managers and legal teams and stuff all
24:00
on the possibility of these inclusions leaks and potential threats that might happen and it’s also a way to ensure a
24:06
level of responsibility in terms of insuring yourself you know against the companies that could potentially even throw lawsuits at
24:13
you because it’s one of those we told you what was in the software now the benefits of it is is the way to
24:20
identify mitigate and avoid right these are things i like to always talk about that these should be things that you’re
24:25
doing also to in terms of security anyway around software development right it’s also a ways in a lot of legal teams
24:32
how many times have you guys worked for a software company and i i’ve had this before where the legal team says hey we
24:38
need a list of all the licensing we have we need to do a license review of the software to make sure that we’re not
24:44
using any licenses that are not approved by our legal department right there’s also a way for you to go
24:50
in there and also look at the inherent risks of some of the interdependencies on the software because it also gives
24:55
you a broad view of not only the transit of the presidencies that you implicitly stated as part of your development but
25:03
also the indirect dependencies that came with it i think i see a bunch of chat information here um i actually would
25:09
like to uh i’ll address those in a little bit if you guys don’t mind um and
25:14
also too there’s also environmental and system information because wouldn’t you like to know if you’re producing
25:20
software and you go to do a recap and say you know say you’re deploying version 1.1 and 1.2 is now about to go
25:27
into production you put 1.2 in place suddenly um it’s performing terrible
25:32
comparably and 1.1 was actually better and you don’t want to roll back you want to move forward you want to see what’s
25:38
different you know what i i bet you once again phil left the debug flag on when he was compiling and it’s accumulating
25:45
stuff in the logs and it’s constantly writing and it slowed us down and damn it we got to go back and change it again and go to the next one right this is
25:52
stuff though but to do that you know in a normal environment would take you a while where you can go in and see this
25:57
information firsthand you can even go ahead and do a basically a diff and say oh yeah you know what phil did leave it
26:03
on ah you know and then you can scream and yell and go make fun of them for a couple weeks whatever
26:09
but then at the same time it also helps lower some of the operating costs because the thing is is that that
26:14
accountability aspect is retained in a standardized format and we’re going to talk about those formats today one is
26:19
called spdx which was approved by the us government and by the way canada is also looking at the same format and then
26:25
there’s cyclone dx which is another one there’s been once again it’s the format battles right on who decides on what
26:31
formats best now saying that we know software has a lot
26:37
of stuff now let’s get into it right so this is the part that i want to show and i’m going to show you a couple of examples of it too right but like i said
26:44
i don’t want to talk about software i want to talk about cake i’m actually kind of hungry right now anyway so this will kind of go along with it
26:50
so first of all this is a lovely cake and you know some stuff about it right this is what you normally would get in software you know it’s a cake or it
26:57
might be part of a larger cake maybe it’s a topper right you know it was made somebody you know it was mixed somehow
27:02
you know it was put into one oven or your hope was put into an oven or you know it was cooked some other method it
27:08
might be tasty you never know right these are all kind of things that make up the attributes but we also know the
27:13
number one thing is used ingredients so when we look at this we know the ingredients that it has right we know
27:20
that these are all the pieces that make up the software oh somebody’s uh there we go we know all the software
27:26
that’s part of it and you know like these are all the pieces that make it up but also too there’s a whole set of
27:32
instructions right on how to prepare this just because i give you a box of ingredients i’m sure you guys have all
27:37
seen the great british bake off right i mean it’s that kind of idea and there’s that whole blind thing you kind of have to figure it out but most most of the
27:43
time most of us wouldn’t know how to do that we would actually have to go away most people at least have some you know practice behind it this is
27:50
actually a way to say not only this is how you did it but also too this is how you actually made it
27:56
so when we look at it what if what happened if we changed the ingredient right the thing is is that maybe the library changed but in this case i’m
28:03
using baking soda the recipe says baking soda what if i go and actually substitute that for baking powder i’ve
28:09
done this i’ll admit it i’m terrible um but what if you did for baking powder you’re right you may be the result they
28:15
sound baking powder soda what’s the difference well there’s a big difference you know the thing is and this actually
28:21
is also with your software you know you know interchangeable components you know maybe one change versus the other how do
28:27
you know and this will allow you to look at the two different recipes or in this case the software build of materials and
28:33
allow you to have a diff between them to say oh yeah we did accidentally use baking powder instead of baking soda we
28:39
did leave that debug flag on oh we did change you know what somebody just introduced a new library into this and
28:46
i’m going to show you some examples of this too so what’s inside the accounts right so once again you know i’m a developer i’m
28:52
going through i’m building my software i’m doing my thing you know and when i bring in all these transitive
28:58
dependencies i want to make sure all that information goes into the software build materials
29:03
at the same time i can provide that to my customer base and if something changes i can notify them in the
29:09
software machine materials that something changed between the versions that we have
29:14
if something is nefarious in place i can notify my customers and they have that list of that information as a parent to
29:20
them it’s also an internalized list that you can maintain and look at yourself so this way you can notify the customers
29:26
based on this so that means if you do find a component like this you can go ahead and find all the versions of the
29:33
software that contain that component based on the information of the software bill of materials
29:39
this also means the thing is what about cakes with lots of layers right that was like an individual soft for instance
29:45
what about things you know that make up you know made up of various components like web services and things like that
29:50
right your interdependency models where you have one program that depends on another or you know functionality you
29:56
know you might have light you know and like a self-driving car you might have a lidar that depends on an ai function
30:01
which depends on you know multitude of other facets right each one of those is different each one of those is made
30:07
differently each has different components you know together they work together they might be delivered in
30:12
different locations um you know it depends on their purpose so when we look at something like a web
30:18
service you can think of it as a multi-share cake like i said we do we’re going to talk about cake or you know here we go right so in this case maybe
30:24
the from is the hot you know the helm is the frosting that keeps the web service together but each layer of that cake can
30:30
have its own set of attributes its own software bill and materials so when we look at something like a
30:36
docker container even you know we start looking and there’s an application layer there’s a run time right there’s an os
30:43
this itself can be a software build of materials when we start talking about a web service same idea a whole bunch of
30:49
docker images with their own software build materials a helm chart that can have all the information of how the pods
30:54
are structured these can be individuals cases also and if there’s say one
31:00
component in it you can go ahead and flag that component so the whole thing is i’m trying to make sure we have
31:05
enough time because i want to give willie enough time for him to do his spiel too um but the whole thing here is is that you know additional information
31:12
what’s the icd tooling are you using you know where we know when was the software built you know what stages you know of
31:18
the sdlc that they go through the software development life cycle you want to know did they go through qa did it go
31:24
through staging they go into production and when you know with the qa team you know let’s go look at the testing
31:30
results that happened you know you want some way of accountability and this is these are things that you can include
31:35
into it you also need to know what foss process right so in other words did you pre-evaluate some of these free and open
31:42
source libraries right once again it’s uh you know free isn’t speech not as in beer um you know kind of idea of when it
31:48
comes down to you know free and open source licensing you know libraries that you use which make up 85 to 90 percent
31:54
of your software also too what environment was it used you know was it built in you know the thing is is that
32:00
maybe you have a ci system that’s dependent on specific build versions of these kind of runtimes
32:06
at the same time one environmental system once again going back to the debug flag did somebody go in and set
32:12
you know the java heap value too high or something like that and then also too
32:17
where is a is there any security and vulnerabilities that are attached inside
32:22
so saying that you know i’m actually going to skip this part because i’m actually going to show you what it looks
32:28
like i’m actually going to break it down i’m actually instead i was going to show you some some screenshots but i’m going
32:33
to hold off on that um because i’m actually going to skip ahead to the next piece because i’m going to actually show you what it looks like
32:39
let’s go look at the software bill of materials all right and i’m going to show you the two formats so this is our product and
32:46
in this case this is our jfrog platform but for right now we’re going to concentrate on artifactory so
32:51
artifactory is our universal binary repository manager it’s a mouthful to say but it provides a lot of functions
32:57
we support over 30 you know package types directly out of the box but let’s go look at a build and for this build
33:03
i’m actually going to go in and i’m going to pick docker i like picking docker docker’s super complex right so
33:09
with docker you actually and i’m going to show you the fact that i have an application that has its own software bill of materials on top of the docker
33:16
software bill of materials so let’s go look at one this one happens to be built by a jenkins server right so if i show
33:22
you up here i’ve got a jenkins server that builds my pipelines and actually publishes a build into artifactory i
33:28
have a whole bunch of transit dependencies that i need well first of all i’m going to show you like this build here right so first of
33:34
all here we go here’s your your good fun exciting obfuscated you know docker image layers you do a docker
33:41
pull you do a docker run and the magic happens well you want to know what’s inside so if i look in here i can show you that
33:47
this actual docker image itself is actually running a node front end of 3.0-28
33:54
and it’s running a java back-end service of you know 2.0.47
33:59
right i actually know what version of the actual software is running inside this container it’s just not you know
34:05
container tag number 82 right you know that’s the thing this actually has quantifiable information inside of it
34:13
i can even go in and say this is remember i was talking about bob you know screwing up in action
34:18
know screwing up and actually you know say phil screwed up and stuff i
34:25
could actually go in here and actually do a diff between two images to say hey by the way it looks like the java it
34:31
looks like the note front end stayed the same but the java back end changed
34:36
i could show you all the environmental and system information right on how it was constructed
34:41
um i gotta warn you i’m loading up our vulnerability data and all this and just to show you this by the way my container
34:48
that i’m showing you i probably should have picked a different one this thing is the worst thing i’ve ever created it has every major vulnerability you could
34:54
ever imagine in it 1630 vulnerabilities um but the thing is though is these are
35:00
all the component layers this is the application this is the os this is the run time right i have security information on how
35:07
it was actually constructed and you know all the pieces that are in it i know all the licensing information around this
35:13
right this is just how all the stuff that we collect in our case but i can export all this information too
35:20
so i can go in here in our case and you see i’ve export s-bomb to spdx and to cycle and dx formats
35:27
and just to kind of exemplify something quickly is remember i talked about how you could have software build materials
35:32
that represent software that runs in other things right multi-tiered applications um you know web services
35:39
dependency models even if i go back here i’ll just show you quickly is i can go in here and in this case you know here’s
35:46
a build i produced i’ll go look at the build that actually produced it if i click in here this is actually the build
35:52
i have this is that node front end i’m doing i can show you here here’s the targe z that i outputted here’s all 482
36:00
transit dependencies i have um that make this up right so this is all that multi-complex layer of software by the
36:06
way this build for this ui has five dependencies defined and it brings in
36:11
482 transit of direct and indirect penalties how can you attribute to all of that
36:17
but at the same time i can also export its own software builder materials right because it has its own also
36:25
let’s go look at this right so i’ve already exported these before because i i had to do it um
36:31
because you know anytime you have any sort of security flaw or any sort of security issue you know one of the
36:36
things is accountability so i can even show you in here like in our product we’ve actually built this in where i
36:41
could say cve take a cve do a search on it right and this will actually go ahead
36:47
and find all the versions of the software that reference the cve so i can use the software build materials to say
36:54
oh wow you know this issue has affected all these builds quickly and rapidly right it allows me to i mean within
37:00
seconds i can go in there and find all the places that something has been used of a potential threat
37:07
so knowing that let’s go in and i’m going to share my vs code for a second guys and i’m going to show you the two
37:13
formats so i didn’t export so the one i have here is the spdx format it’s uh it
37:19
comes you could actually do it in a couple different ways actually i should probably show you that by the way so there’s a couple of applicable ways that
37:25
you can go ahead and do this and the thing is is that the four mess under the standard are the interesting part let me
37:30
go find uh here i’ll go ahead and do this one um let me just grab a random version
37:36
um i’ll just show you because some companies will expect others uh versions so if i do spdx you can either do it as
37:44
a tagged value right which is the is one of the tagged values so sometimes i’ll ask for that you could also use
37:50
doing just excel right so just do it as a spreadsheet if you want to do it for legal purposes and also or for json now
37:57
for our purposes i’m going to show you the json version because this is actually more of the standard because there are a lot of management tools for
38:04
software build materials now there’s a there’s a bunch out there so most of the time it’s going to be json they’re going
38:10
to request it you’re going to ingest it into a system and then they’re going to have there’s all these companies springing up now that are doing that
38:16
um also too you can also do cyclone dx it’s either xml or json right so it’s
38:22
either or so we actually do export both of these but i wanted to show you what it looks like because you know you’re
38:28
going to wonder well if you look here you can actually see like the spdx format shows all the
38:34
creation info right so it basically says like what it is and you know the organization and and me in this case the
38:41
bill m that’s me but inside this is the list of all the dependencies right so there’s initially
38:48
just a straight list of every single one i probably should have picked something a little smaller uh than this to kind of
38:53
show you because this is going to be scrolling for a while um because there are a lot of components here i mean let’s see this just to show
39:01
you this docker image is about 900 says 982 if you take out the extraction in the
39:07
front this has about 960 components that make up this container
39:13
this is 960 pieces of software that are utilized to actually construct that one docker image
39:21
all right now picture trying to do this in some other respect but then it breaks it down for this so in other words
39:28
here’s a link directly in our case to ours right so this asks for the document namespace and then it breaks down the
39:34
packages even further so it actually grabs all the information from all those
39:40
components and actually pulls out any information that could be there including things like checksum um you
39:45
know the name of it um like here is like you know here’s a you know ant jfx jar right so this is actually all the
39:52
information that gets cataloged once it’s catalog it becomes searchable and that’s one of the major factors of why
39:58
software with materials was brought about
40:03
now s now cycle and dx is the other but if you look here you can see that it’s
40:08
different there are distinct differences and the thing is like the us government decided on site on spdx canada followed
40:15
suit i know that in eu they’re going with um spdx i know the britain because
40:20
they want to be different with cyclone dx um right it’s just you know but both formats are out there and i will say i
40:26
think spdx i think is probably going to grab more prominence because i think it’s an easier ingest method i think
40:32
cyclone dx is another format to that but just let you know both are out there they both serve the same purpose
40:39
but if you look here you can actually see you know like once again it’s that same level of information you know it’s the name the creator um you know this
40:46
actually has you know the type in this case um and then once again it just starts breaking down all the pieces but
40:52
instead of having an index at the top it’s actually individual components that are listed out instead
40:59
now saying that understanding it and having that level of information really helps
41:04
you so when you’re when you are doing your builds you know make sure that you know some of the even if you’re not using our product there’s the thing is
41:11
is that you can even there’s tools out there for things like jenkins there’s one that takes like all that information
41:17
some of that some of that information is collected inside of jenkins you can export there’s tools out there that will
41:22
go ahead and you know you can point it we’re actually gonna we have one that’s kind of like this you can point at a
41:27
piece of software that you’re doing and you know and be able to go in and and look at the components and build a
41:33
component graph is what it is you’re building a component graph of all the pieces and then you have to export it
41:38
into a format that you know works for you know what you’re doing or what your regulation is or what your customer
41:44
expects but what you’re doing is it’s all about accountability it’s all about understanding and it’s all about you
41:50
know like you know like i said you know when we’re looking at something like you know like a lot you know the log for j
41:56
issue right i can go in here and i can say i think i actually have it already set up queued up here somewhere like i
42:02
can go in here and say you know like you know log you know in our product we just say log you know for j you know i can go
42:08
ahead or i was allowed for j um oh i’m looking for artifacts sorry let me go in here or actually you know we’ll go to
42:13
packages we’ll do it from here instead um you know i can go in here and say okay let’s go look for you know all the
42:18
pieces are if i was actually doing things correctly today whatever um i’m gonna go back to packages here we go um
42:25
but i can do searches on the things that matter to me or actually i’m not doing it right right now i’m actually keep
42:30
going back to there i don’t know what i’m doing wrong i messed up before i think i or i said something different
42:35
but understanding you know like where things are being utilized like i’ll actually i’ll show my no works here hold
42:41
on here um this is my test system i always feel like i’m inviting you into my house and i haven’t cleaned up so
42:47
that i’ve got a lot of garbage in my system but you know being able to actually go ahead and have accountability of like here’s like a
42:53
battle 6.263 um being able to say where is this being
42:59
utilized right i can show you every single touch point in which it’s actually been used but then i can also
43:04
go ahead and you know you want to understand well yeah that’s great you know what this dependency that i have as part of
43:11
my build system here well it also depends on all these 19 other indirect
43:17
transitive dependencies for the model i’m building so the understanding you know what
43:22
you’re bringing in and how you’re utilizing is very key to what you’re doing and the software building materials allows you to have
43:27
accountability a little safety from your organization by informing your customers of what you’re using how they’re using
43:33
it and the software that they purchased especially you don’t want to be that company that you know becomes a headline
43:40
so the big takeaway from this too is is also two number one misconceptions we hear this
43:45
all the time you know can it be used as a roadmap for an attacker not really i mean the thing is is that yes they can
43:52
get an idea what you’re utilizing maybe some versions but they can go in and infiltrate the community but that’s
43:57
outside of you that just happens right protecting yourself is essential
44:02
don’t have to require any source code there’s no source code required you do not have to do it unless it’s implicitly
44:08
stated by the legal teams of that company and that’s a negotiation that your legal team and their legal team
44:13
have to have together to say you know we’re going to put this the your source code in escrow we’re going to keep a
44:19
copy of it blah blah blah i’ve heard this numerous times with some of those pieces and you’re not going to expose any
44:26
intellectual property the thing is you don’t really you know the stuff that you’re building is the stuff you’re building it’s like like i said it’s like
44:32
having a list of ingredients without the instructions on how to actually put those together sure you could probably
44:37
try to figure it out but if you have all your algorithms protected and you follow safety protocols there’s nothing you
44:42
have to worry about so at the end of the day if you want to work the us government or a multitude of
44:47
governments in the industries you know regulated industries are now adopting this methodology you need this if you’re
44:53
going through and you want a complete list of all the free and open source stuff that you’re using when you build your software you need it you know if
45:00
you want to understand how what and where and why things were made this gives you the ability to go in and even
45:06
track things like velocity you know how much change are we doing between versions it allows complete
45:11
accountability at the end of the day it also adheres to software and license compliance that you might get from legal
45:16
teams um you know outside third parties auditors m as
45:21
you name it these are things that you want to be able to easily present if you’re required to
45:28
so that was a lot to throw at you in an hour and i’m sorry or whatever time i just did that in but i just want to let
45:34
you know that this is kind of the level of information on what a software bill of materials is um i don’t know if
45:41
there’s any questions or anything like that um let me see here oh i just see a
45:46
lot of people at greetings and free software but materials there we go is there any questions or anybody have any
45:53
questions for me in this respect i can go stop sharing so i can see faces so maybe i can just comment i noticed at
46:00
the beginning of your presentation you had the gitlabs hack and message was from ravi lakshman and by the way
46:06
actually ravi was presenting on last year actually in this meetup so
46:11
small world very cool but so yeah so that’s basically um you
46:18
know ways to utilize it ways to do it like i said protecting your customers protecting the software you produce and
46:24
then this is just an accountability aspect behind it so the software you know these are the formats that are out there and really it’s up to you as
46:30
developers devops engineers devstock engineers release managers of course to ensure security at different levels you
46:37
know the thing is is that like for us at jfrog we have our product called x-ray and actually it was just stored on
46:42
devops.com as being the top security product for this stuff i’ll take that as a yes that was pretty rad um but the thing is is that you know
46:49
being able to implore tools actually you probably saw it one of the things here is is that you know as a developer in
46:54
here i’m actually working on a uh oh here’s one of my cats i was talking about you might see its tail in a second um
47:00
but um you know these are like here’s a here’s a bunch of libraries i’m using right and being able to actually identify these
47:06
libraries you know as a developer this is the front line defense this is that idea shift left right here shift left
47:12
and ship right so this is really that idea of being able to say oh hey my software is utilizing these pieces and
47:18
as a developer i have a responsibility to go in this is a critical issue right this is a critical cve
47:24
or even being able to even go in and use a tool like this where we even have our like for us we have our jfrog cli and i
47:30
can say audit right and if i can spell audit today and in this case this is going to go through the project i’m
47:36
doing it’s going to pull in all those transit dependencies as a model right so it’s not actually bringing us bringing
47:41
them as a model and i can take this and i can display any potential threats and the thing is is that with a soccer build
47:47
materials and first you know shift left defense this is a great place to do it also integrating this level of security
47:53
into your ci process into your release process even into your runtime like we’ll be introducing a runtime analysis
47:59
down the road to say this component is running in your runtime right now but the idea is is that this accountability
48:05
is the you know the thing is you want to be able to map all the dependencies uh direct and indirect that you utilize as
48:11
an organization and they have a format for it so i hope
48:17
this has been helpful guys yeah hey absolutely yeah hey this is ahmed owen i work at adp
48:22
and i have a question i wanted to know what’s the best way to distinguish a vulnerability that came from
48:30
your base image versus uh your app code in a doc ah so when you’re looking at that right so
48:36
one way to do it is is actually to go in and in some cases the best way to do it
48:41
is actually i i’ve actually done an experiment like this where i’ve actually used two different tag versions right to
48:46
say like here’s a base image one here’s this or if you’re doing a docker compose you want to look at it so like in our
48:52
product just you know i know i know i’m plugging the product whatever um we have the ability to actually scan that base
48:58
level image before he is put into the docker compose right and there’s tools out there that you can utilize to scan
49:04
these images before you utilize them right so in our case we use our x-ray product to say i’m doing a docker
49:10
compose i want to scan this docker you know this docker base image before i even compo you know build it i can go in
49:17
and determine whether or not there are nefarious components in there but then also too you can also look based on the
49:23
base image and abstract that out and give it its own software build materials just so you know you can actually pull
49:29
the information out if you compose that and you actually could see all the baseline imaging in here like i’ll just
49:34
show you really quick if you don’t mind um let me um where’s my share there it is
49:40
um here let’s go back and look at a docker image for a second right let’s go back and look at my my docker image i
49:46
have here i’m going to show you my favorite view um yes i am plugging again and i’m going
49:52
to show you my favorite nerdtastic view that we show inside of our product but here’s an example here like here’s all
49:58
those base level here’s you know here’s my docker image here’s all those layers of the actual docker image itself
50:04
i could actually go in here and show you in our case i could actually abstract
50:10
all the actual you know information from that base in this case um i could actually show you all like
50:16
and this is a debian uh container um you know i can actually show you like the debian components that actually made up
50:22
this one and in this case you can even see here where i’m actually i actually have something nefarious in here with an issue
50:29
so there’s ways to do this but you gotta there’s certain you know only certain tools will allow you to have that insight also too depending on how you
50:36
actually build the docker compose you can also use the log function and have it show when you’re doing it and it’s
50:42
actually constructing this stuff you can actually pull that in for a while also there’s way there’s multiple different ways to do it
50:51
so bill could you then essentially like have the default behavior be that it will not build if
50:57
there is a vulnerability uh discovered in one of the dependencies yes in our products you can we actually
51:04
have the ability to create rules that will actually terminate a build even on ingest to say and flag a container as
51:10
actually being non-viable so and where is kind of the threat intel coming from with regards to the
51:16
vulnerabilities ah so in most cases it depends right so like you have all the standard nist
51:23
sources nist is actually this the relative source is out there for most of the software out there there’s actually
51:28
vulnerability lists that are out there um there’s also companies like rbs risk-based security called vol they have
51:36
you can actually go and search for it and you can actually look up a component and it can bring back those lists um in
51:42
our case we actually algorithmically and manually curate it and we also have um a team um that we have that we you know is
51:49
our security team that allows us to create cves because we actually create cves uh publicly um we’re actually one
51:56
of the one of the few companies i think there’s in total i think there’s just under 80 companies in the world that can produce cves and we’re one of them um
52:03
but the thing is is that you know that level of information on vulnerabilities like i’ll just i’ll just show you an
52:08
example um let me see here let me see i can find an example on like where those sources come from because i actually
52:14
could show you the sources themselves um you know what let’s go to another build um
52:20
yeah you know what actually let’s go back here um you know using sources like nist provides a certain level of neutrality where people can’t start
52:26
pointing fingers exactly so the thing is is that like i’ll just show you quickly let’s go bring up one right so we
52:32
actually have like you know here’s an example of the actual you know issue itself right here’s the link you know
52:38
the cve um you know and it depends i mean in some cases there’s really not much detail that you know that’s provided
52:44
like this is actually pretty terrible uh but the reference materials it depends you know in some cases you know it comes
52:51
from say like in this case like here’s you know issues that apache right this is actually uh you know this is actually
52:57
the issue that caused it right these are actually some of the publicly uh available components um you know there’s
53:03
other ones here you know from like you know maybe a source from github that you know goes ahead and says yes this is the
53:09
actual issue that it is here’s the cve component itself you can go ahead and look it up so it depends though i mean
53:16
every company handles it differently but there’s also sources out there and products that allow you to go ahead and
53:21
do the searching on this and also associate those binaries to those actual potential threats
53:29
and by the way always be cautious because you should always go ahead and look at the actual
53:34
issue itself and when you look at the issue itself there’s a lot of the time is that whole baby in the bathwater thing i’ll be able to say you know maybe
53:40
there’s a hundred functions in a library is this applicable to the usage that i’m
53:46
actually doing because in some and then you have to evaluate too is this an internal product or an external product
53:52
right and you know there’s a lot of things to take into consideration you know the thing is that most of the time
53:57
most of the most of the time most people are like it’s got a vulnerability check it out but what if it’s an external
54:03
remote source vulnerability and you’re using this on an internalized product or what if the fact is is that the actual
54:09
component defined in the csv is something that you’re not actually even using as a function so there’s also a
54:14
level of expectation in terms of developers to do a little research into this if something is found to say hey by
54:20
the way this really isn’t applicable to the way we should do it we should label it as such
54:25
i mean what i’m getting at is that this is really about visibility you know people have fallen the other way for way
54:30
too long saying i don’t know they shrugged their shoulders right so that’s right you know it’s right there it’s
54:36
right in front of you you had to intentionally bypass that flag or that warning to to complete this build
54:43
absolutely and that’s the thing right and that’s one of the things that we try to protect against but also too like you said there’s a there’s a
54:50
the you know the thing is is that there’s always that knee-jerk reaction of just disposal right um and that’s the
54:56
thing is like i said there is there are things that you know research that developers should do um
55:01
and things like that and like you said bypassing it you should be able to justify the reason right if you don’t
55:07
justify it then yes of course the you know uh you know they then you know they should be you know reprimanded or
55:12
whatever to say you know don’t do that you know whatever but oh i see jacob you have your hand raised
55:18
or virtually hand raised you’re on youtube oh you gotta mute
55:23
yourself i’m terrible at reading lips
55:30
okay there we go okay yes i am i have a question okay sure
55:36
okay now this this program this this program has been used for security right
55:44
well well part of it i mean our the stuff that we were doing yeah i mean our our product is actually more binary
55:50
management and security is actually just a feature of our solution
55:56
oh right okay but it is i mean it hasn’t gotten to the point
56:02
where like secure like the whole spectrum between your lock out your customers
56:08
uh the spectrum between the security and safety has has has
56:15
has a veered over to the level of security like um
56:21
is it do do you find that i mean because because i mean
56:26
i mean you you were all um you know very
56:33
uh technically uh your your technical prowess is strong but
56:38
for people who have um who who are having trouble getting locked over their bank accounts seniors
56:45
isn’t isn’t isn’t this isn’t this an issue in programming for excessive
56:50
security um no so that that access
56:58
there’s application level security right you know actually running the application and providing security for
57:04
the consumer right that’s more of like locking you out of your accounts and and things like that this is actually more
57:11
development level security um more you know security around not even getting to that point right
57:17
this is actually potential threats inside the software that might cause a problem right so like um you know you’ll
57:24
read things of like uh you know so and so organization leaked 450 million accounts right that was more
57:32
of an application level security um in terms of actual accessibility to the
57:38
contents or the actual data in the software locking out people from their accounts
57:43
and stuff is more of a safety measure from accessibility but what do you mean by leaked well
57:49
defined leap um so dependency confusion attacks right so
57:54
or sql injection where you go to a website you look at the website there’s a whole bunch of input fields on it in
58:01
some cases a lot of hackers what they’ll do is they’ll use something like net you know like uh there’s a lot of areas like
58:06
sql injector there’s a whole bunch of black hat tools that you can use trust i i’m not saying i use them but i’m just
58:12
saying that they’re out there but you can point them at a site and say show me a field you know show me a data field
58:18
and let me see if i can inject um a query into it so log for j why was logs
58:24
for j so dangerous this one of the biggest libraries used in the entire world it’s been used
58:31
for decade you know over a decade why was it so dangerous because you can inject information into a field right
58:40
that would go in into the log4j platform and then if one of this one well the
58:45
scary one that anybody can run is you can actually drop into the query field with the proper flag and say
58:52
in this case for unix command you could say rm dash rf slash
58:58
and you can throw that in and then what it would do is it would execute on the runtime environment and just wipe out
59:04
the file system right so you can actually have it go in with the proper pseudo credentials and actually it would
59:10
just wipe out it would just wipe out the directory just gone bye-bye right and this came with android phones by the way
59:17
so you could actually do this to an android phone um that’s why google was scrambling to put out a patch to
59:22
alleviate this yeah sorry i will need to jump in i know there are there is one question i don’t
59:28
know if you keep your your capable of answering it but okay
59:33
now the the hackers jacob could i just stop you for it for a moment
59:39
three three different categories the the checkup sorry we’ve got another
59:44
presentation so could i um so uh bill you you’re still here right can we just wait another i would say
59:52
30 minutes and we there’s a couple of other questions as well so maybe we can just do it after the
59:58
other presentation yeah i’ll stick around i’m actually gonna go get a refill of water will will he do his thing lily i’m looking forward to
1:00:04
hearing what you got to say buddy um i’m gonna i’m gonna go mute guys um and i’ll come back later i’ll be back i’ll be
1:00:10
here so thanks thank you very much all right so uh we are moving to our second speaker and
1:00:16
that’s willie shop and uh willie is not a stranger to this meetup uh i have a
1:00:22
pleasure of working with willie at worksafebc and he’s the manager of common engineering practices
1:00:28
willie knows um all elon musk’s quotes and uh both of them
1:00:34
share common interests so today he will be talking about his experience in setting up common
1:00:40
engineering practice at workcpc will he over to you thanks andre and can you just allow
1:00:46
sharing for me as well somehow that just disappeared oh really okay disabled yeah
1:00:52
let me just do that
1:01:00
all right let me just try to do that interesting sharing it that doesn’t work for you
1:01:07
no it’s his host disabled screen sharing oh that’s not what i was supposed to be
1:01:13
okay well it gives bill time to get his water
1:01:19
it’s all part of the script okay here we go
1:01:30
can you try now
1:01:39
here we go okay here we go you should see the screen now yes
1:01:44
and i assume bill is back great so good evening and afternoon
1:01:50
morning and welcome from wherever you’re joining us at today’s devops meetup and a special shout out to you initiates
1:01:57
from brazil it’s really great to see you it’s it’s my pleasure to to give you a
1:02:04
quick update on our efforts to establish a common engineering ecosystem at
1:02:09
worksafebc so let us start by taking a peek at our
1:02:16
roadmap and what you’ll see is as shown at the bottom is it’s based on
1:02:23
quite an ambitious vision to empower in every engineer by standardizing our
1:02:28
engineering practices and tools and enable continuous delivery of value
1:02:33
to delighted end users so from the early 2019s
1:02:40
we operated undercover i heard star wars today so we operated
1:02:45
like the rebel alliance on the ice planet hearth kind of out of sight out of trouble
1:02:51
and we focused on creating manifestos and guardrails that enabled engineering to build
1:02:58
consistent secure and simple solutions and you can actually refer to
1:03:04
the common engineering system at worksafebc meetup session that we delivered back in april 2000
1:03:12
for more details on those ic times we also innovated our continuous
1:03:19
integration and delivery pipelines which was another meetup session we
1:03:24
shared last year in june and released our works fbc technical blog which now allows us to share all
1:03:31
our failures learnings and innovation with the community
1:03:37
back in september everything changed for us because we got the opportunity to launch
1:03:43
a common engineering team with a leadership mandate to empower every engineer by standardizing practices and
1:03:51
products and stewarding guard rails again to promote engineering consistency
1:03:57
enablement security and most importantly simplicity
1:04:03
so for the first 90 days we focused on establishing an efficient system of highways and that
1:04:12
enabled both our business and engineering and ensure that our common engineering
1:04:18
ecosystem had a common and sound vision a vibrant collaboration and an
1:04:24
atmosphere that encourages failure learning and continuous innovation
1:04:31
and we achieved three major milestones
1:04:36
as shown the cell service automation which we often refer to as the walking
1:04:41
skeleton it creates an azure repo injects an app-type sample
1:04:47
configures and injects a yammer-based azure pipeline and cues the pipeline as good measure
1:04:54
for final validation of the setup all in less than 10 seconds
1:05:00
if that doesn’t make the developers happy i don’t know what will
1:05:05
we looked at working agreements also known as cheat sheets to guide how to work together and to
1:05:11
create positive productive processes and
1:05:17
we started a war on on a little bit too fast we started a war on waste and silos
1:05:26
again by fostering collaboration through what we refer to as centers of enablement
1:05:31
working groups dojos community of practices all of which you can read up on on our technical blog
1:05:40
now for the next 90 days or so we plan to switch to implementation mode
1:05:47
we have the highways now let’s start implementing and our plan is to
1:05:52
enable and foster steward guard rails and collaboration so that for us is
1:06:00
foundation it’s all about collaborating with each other we want to monitor and learn from
1:06:06
baseline metrics and we want to open source our application type yaml based
1:06:13
pipeline blueprints and automate everything automatable
1:06:20
thereafter i think let your imagination saw we are looking at things like chaos
1:06:27
engineering which will scare operations to no end but that is
1:06:32
fine the distance we should be getting ready for chaos engineering as well
1:06:38
so that’s our road map that we’ve set up for ourselves
1:06:44
here’s a quick snapshot of the extent of collaboration we actually achieved during the first 90 days with our
1:06:51
centers of enablement working groups relentless banging on virtual walls
1:06:57
twisting of rubber arms and stubborn push for complete transparency
1:07:03
so if you look at the diagram the variety of teams
1:07:08
the vibrance of collaboration we achieved and the focus on outcomes gives
1:07:14
me the reassurance that we’re on the right track and hope that we can actually achieve the unthinkable as an organization
1:07:23
erasing the silos avoiding them to regrow like weeds
1:07:28
establishing trust and getting all the stakeholders to collaborate
1:07:34
has been will probably be one of the biggest tests for our common engineering ecosystems today
1:07:45
which brings me to my personal learnings from the challenge of launching such a team
1:07:52
especially when you are like i a software engineer at heart and you want to remain involved hands-on with the
1:07:58
continuous research the learning and the improvements so apart from the people management
1:08:05
dealing with ridgid at times mind numbing and time-consuming processes
1:08:10
um a lack of skilled resources in the local market which we’re probably all feeling
1:08:16
our team also has to remember our responsibility of operational support
1:08:22
and quality assurance so those are two other pillars that we can never forget
1:08:28
so balancing these with our quest for bold change and continuous innovation
1:08:34
has and will be challenging so creating a harmonious team
1:08:40
where everyone takes responsibility for their action where we have a work-life balance and
1:08:46
where everyone is a leader is definitely not a walk in the park
1:08:53
i am constantly worried about everyone in my team who are we are like you in tough and
1:09:00
trying times where remote work isolation and the lack of social contra contact
1:09:08
is not only taking a toll on every one of us but in my opinion maybe rewriting our
1:09:13
rule books for business engineering and work psychology for the future
1:09:20
an anomaly that keeps me awake at night there’s many but this specific one
1:09:27
is how to ensure that everybody has an innovation mindset
1:09:32
has the time and the grit to take a chance i find that many engineers can spend
1:09:39
hours talking about a problem but often they have no time to actually solve the problem so they will spend
1:09:46
three hours telling you what the problem is all about and when you say let’s solve it they say
1:09:52
i haven’t got any time now i have to go back and write out so hopefully we as a community and i’m
1:09:58
hoping we can do this as a devops community can collaborate to create some helpful guidance for for the community
1:10:05
around taking a chance being open for failure
1:10:11
and also this new working environment which some enjoy
1:10:16
and some find extremely stressful
1:10:23
so my personal mission statement which i stare at every morning is to encourage
1:10:28
everyone’s creativity passion purpose and strengths
1:10:33
i’m also experimenting with a variety of guidance for example from
1:10:39
david markey carrying around his book then turn the ship around and you can actually ah
1:10:45
there you can see it like this is my my holy book that i carry around all over the place
1:10:51
like wherever i go because it’s a real leadership success story that is really really worth to
1:10:56
read i also remind myself to foster autonomy mastery and purpose as
1:11:03
just discussed by daniel pink and as mentioned by andre i’m i’m
1:11:09
inspired by the spacex phenomenal innovation and
1:11:15
i use kind of elon musk’s rules to simplify everything
1:11:21
encourage my team to automate everything automatable to question every process
1:11:28
and we often provide colleagues by asking the five ways or simply just ripping out a part of an
1:11:34
inefficient process to see what happens and last but not least i remember or
1:11:42
remind everyone to to have fun
1:11:47
and that is to combat today’s stressful days of isolated and remote engineering
1:11:53
and to remind us of the need to balance work and life
1:11:59
and it’s probably a reason that our most recent quarterly update looked more like a newspaper than a
1:12:04
formal report we wanted it to make make it fun and i think it worked
1:12:10
um andre you can say later on whether you agree or not but we got some really positive feedback on on that report so
1:12:18
we’ll have to find other cool things to do i also work hard to create a mindset to
1:12:24
continuously experiment and embrace failure as an opportunity to learn and to
1:12:30
innovate and you’ll see soon see this round red sticker appearing on all our
1:12:37
laptops and that’s really to instill the courage to fail as one of the pillars of our healthy
1:12:43
devops mindset now what i’ve learned is the challenge is to avoid the fear of failure amongst
1:12:50
engineers all stakeholders in an organization must trust the engineering process
1:12:57
and embrace failure and as an opportunity to innovate
1:13:03
and the best antidote to toxic fear of failure or leaders who are supportive
1:13:08
and inspiring of which we have many plus a collective blameless fail force
1:13:15
mindset if you have that in place we can start experimenting and we can start
1:13:21
learning or as elon musk says and i quote failure is an option here
1:13:26
if things are not failing you are not innovating enough and that came straight from the spacex
1:13:33
launch site where they are truly experimenting
1:13:41
so to be successful we need a dream team that is continuously invested in
1:13:47
and focused on operational support quality assurance and innovation
1:13:54
so we have a team of engineers focused on operational support and innovation
1:13:59
and another team of engineers focused on quality assurance and innovation
1:14:05
now this is where my next challenge is coming in to create one team mindset
1:14:10
aligning the innovation of both those teams within the team
1:14:15
and i’m pretty sure it will continue to be a challenge again in today’s remote and isolated working
1:14:22
model because you can’t just walk over to a desk and say hi bill how you doing today
1:14:27
house quality assurance the day of quality assurance going you cannot do that anymore
1:14:34
so i believe that the secret sauce that holds us together especially during these trying times is
1:14:40
distrust so you have to trust each other you have
1:14:45
to trust your leadership your stakeholders your end users and vice versa
1:14:52
so it’s the whole chain and back
1:15:00
so as mentioned on our technical works fbc blog and i’ve mentioned brief
1:15:06
before as well we live and breathe by our working agreements
1:15:11
with an emphasis on optimizing meetings which is another one of elon musk’s muses
1:15:17
and inspire an agile kanban and devops mindset as well as the five-day obscure
1:15:23
values so the working agreement pinned to most of our walls
1:15:29
reminds us daily what we value steward optimize and inspire as one team
1:15:38
and our team working agreement has actually triggered a range of other working agreements such as starting a
1:15:45
center of enablement or even rotating through our company engineering team
1:15:50
and again i invite you to have a look at our technical works fbc blog or ping me if you have any questions on
1:15:56
any of our working agreements
1:16:03
all right so to wrap up this quick overview i just wanted to put all the links of
1:16:09
information i’ve referenced in today’s update i hope that our journey inspires you
1:16:16
and that you’ll actively collaborate with us on twitter and linkedin you’ll find us in both
1:16:21
places and with that thank you for listening
1:16:28
any thoughts or questions
1:16:36
oh yeah i will say man i i did love the fact that you didn’t quote the uh elon mustang another one of my favorite
1:16:42
tournament one of my favorite quotes is from marcus aurelius um the obstacle is the way
1:16:47
right so always look at every obstacle in this place as not a uh you know
1:16:52
something as a failure but look at it as more as a learning experience for a better solution right so whenever you
1:16:59
come up against that the stoics had it right in that respect of being able to say that or you know as i say the people
1:17:05
on my team is nothing’s impossible it’s only improbable yeah
1:17:11
right yeah really i’m curious to hear how you uh first of all i really want to apprec
1:17:18
i really want to mention how much i appreciate you recognizing the value of failure you know and that failure is a necessity
1:17:24
in order to succeed um you can’t have light without darkness etcetera etcetera um so how did you then
1:17:31
sell the value of failure to leadership
1:17:37
oh still busy selling the value but we actually as i said is we actually have great leaders who are supporting us
1:17:44
um with for example failure and what we started doing is we literally just
1:17:50
started putting the quotes on our back on our
1:17:55
laptops it used to be great when you were actually at the office and you could have all the stickers of
1:18:01
importance on the laptop but that’s where we actually started advertising the need to fail
1:18:08
i started kind of quoting the likes of elon musk with our leadership
1:18:15
um anja you know that very often in the morning on teams when you all say hi to each other we i have another quote from
1:18:22
elon musk and a lot of his quotes are actually around innovation
1:18:28
i guess for a reason and he’s strong push for failure as being a necessity
1:18:35
to innovate um so it’s we’ve been pushing that message
1:18:41
yes with some managers we are still pounding our heads against the wall because you just have to get through
1:18:47
that wall or through that silo but i think once engineers actually realize
1:18:53
that it’s great to fail because now we can learn out of it and work together
1:19:00
you actually see a change in the engineers so i don’t have a secret source other
1:19:05
than share that message as much as you can and support your team
1:19:11
if there is actually a failure don’t blow up and say we have a disaster
1:19:17
no great we have failed so it’s the kind of the way you respond i love it we have
1:19:22
failed so what can we learn from this let’s get together let’s have a virtual coffee and let’s discuss what happened
1:19:29
so it’s again how you work with your team and kind of how you react to
1:19:34
to the failures really it’s really quick i’m sorry i’m just rejecting it again so i’m actually
1:19:39
i saw i’m a mentor at tech stars uh 500 startups y combinator um i’ve actually
1:19:45
been work i work with a lot of companies uh over time um i’ve been doing this for a couple of years i’ve had three exits
1:19:51
i’ve actually taken two companies public all these kind of things and so i spend my time talking with you know basically
1:19:58
you you don’t know if you don’t try and if you fail you reassess and move
1:20:04
forward right do not look at the hindrance but always go forward and the thing is that actually you learn your
1:20:09
best viability from those failures themselves right you know they actually you know if you succeed all the time the
1:20:16
problem with that is is that you’re probably not doing it right to be honest you’re not taking the risk you’re not taking the chance and in a lot of cases
1:20:23
if you do fail um you know the thing is like you said you brush yourself off you move forward and you say what did i
1:20:29
learn from that and actually failure is always the best teacher in my opinion um and when i i actually i was down like
1:20:35
you said the elon musk thing so i did a bunch of work in hawthorne um at the spacex manufacturing facility uh where
1:20:42
they make the rockets and um i was working with their teams down there and they have this work all those wordings
1:20:48
spread all throughout the facility everywhere you look there’s words of of that you know quotes and stuff and we do
1:20:53
the same thing at the jfrog office our jfrog office is filled with all these quotes to kind of reinforce the idea
1:20:59
that we’re collective that we do fail and when we fail we just brush ourselves off and move forward um you know as a
1:21:05
team collectively and our job is to pick each other up and work with each other to move forward
1:21:12
i guess and loads of words of wisdom sorry didn’t mean to interject like that
1:21:19
but i just wanted to say it’s like i feel everything that you’re saying and i totally wholeheartedly
1:21:24
undeniably a billion percent agree and i’m so jealous building that you were actually at the spacex center
1:21:30
so you’re one you’re a gazillion miles ahead of us the other thing i want you to just um
1:21:37
throw in there is with failure as well if we actually combine it with experimentation you’ve
1:21:42
got the winning formula i think selling experimentation to an organization is much easier than selling failure
1:21:49
and then kind of again like on the ice planet health you do it undercover you actually
1:21:55
explain to to the stakeholders that yes we’ll do an experiment and then we’ll get to a point where we’ll decide if
1:22:02
this is okay or not if it’s not okay well we failed we’ll rotate around and do another experiment
1:22:08
and you continuously doing experiments failing or succeeding and that is another way to actually sell
1:22:14
the value of failure is combined with experiments
1:22:21
excellent thank you as a matter of fact there is a lot of other quotes in chat i think what we we
1:22:26
are going to do we are going to collect all of them and we are going to post them somewhere so you know that’s
1:22:32
actually a a great event today because it shows the right mindset right actually everybody
1:22:39
here in this uh in this meetup has a right mindset which is which is great to great to hear
1:22:46
okay do we have another question for willie
1:22:51
if not then i would like to close the official part of the of the meeting and
1:22:56
uh bill if you don’t mind a couple of minutes in willy there is there there are a couple of other questions as well
1:23:02
so uh so we can maybe have a quick conversation but first of all i would like to say thank you ari are you
1:23:10
already left but bill this was really fantastic presentation thank you very much and
1:23:16
willie obviously uh you’re doing great job with implementing and and pushing the envelope which is extremely hard i
1:23:23
know it’s a cultural change in organizations so uh thank you very much for this and thank you to all the
1:23:29
participants as always if there is any topic that you would like to talk about you’d like to share your experience um
1:23:36
anything that is related to devops and the product that that you have please do not hesitate to contact us and
1:23:44
we’ll schedule you for one of the next meetups so thank you very much everyone
1:23:50
and uh have a great evening thank you thank you everyone for having
1:23:55
me i appreciate it really and andre guys thank you so much willie amazing presentation thank you thank you so much
1:24:01
for your your time in dealing with me [Laughter]
1:24:06
thank you so um uh lorena i think you had a question right
1:24:13
one of the questions yeah you’re looking to say is there a way to go find log for j components right is that what we’re
1:24:19
saying lorena so yes there are different multitudes of ways to do this right of course um i mean i’ll just show you an
1:24:25
example uh when this came out we were one of the first responders uh to this so like we even went out and we actually
1:24:32
produced a free open source tool for companies to go ahead and actually you know point this to the software that
1:24:37
they’re utilizing i’ll put this in the chat for everybody just to kind of see but we actually released this free to
1:24:43
the public without even our customer base because we believe in the fact that all software should be safe and secure
1:24:49
whether you use this or not we don’t care um i mean we care of course we care i mean we like customers you know we
1:24:55
like rev you know we’re a company right um but at the same time and in addition
1:25:00
to that there are other ways to do this too so like even in our product too there’s also search factors you can go
1:25:06
ahead like i showed before you know every touch point that it came into in this case this was just a cli tool
1:25:12
that we released that you can point at your software and it’ll build a component graph around it so you can find out whether or not um that is part
1:25:19
of the solution that you have all right um i think there’s some other questions
1:25:25
too right yeah ahmed isaac hey guys oh hey thank you andre for listening i
1:25:32
just wanted to ask if jfrog had the ability to mark a vulnerability as risk accepted
1:25:39
yes so you can actually go ahead and i’ll actually i might as well show i must keep my screen up uh let’s go look at
1:25:46
you know like say any of these builds here i’ll just go look at a you know any sort of build that might be here with an
1:25:52
exception um there’s a couple different ways hold on let me go in here let me bring up any of these so there’s a
1:25:57
couple different ways you can handle it in jfrog you can either say create an ignore rule or you can say i want to
1:26:04
ignore it based on the license it was a licensing one sorry it could be vulnerability or license compliance uh
1:26:10
the component the build the watch the watch is how you actually detect it it’s a rule set you define
1:26:16
uh you add in the note and then you can also say you want to have it for an x number of days right so there might be
1:26:21
times where you actually say you know what we want to continue on this journey we have a release coming up we’re not going to address it just yet we’re going
1:26:27
to ignore it now but we want it to come back in effect so i want to come back on february 4th i want to be notified
1:26:34
that’s one way another way that you can utilize the two is that you know when you have any of
1:26:39
these components like i’ll just go find just want to show you a high level component in this case oops
1:26:44
i’m using a beta version so i probably should have picked one that was actually i’m in the middle of debugging a version
1:26:50
oh let me turn off the debug i’ll i’ll do it afterwards i don’t need to do it right now um but let’s go in and uh look
1:26:57
at you know any of the other components that might be in here do i’m not scanning that one of course let me find one i’m scanning um we’ll go back here
1:27:06
let me go in and yeah i’ll do this one this will be good i’ll go find the descendant let’s go in and look at like
1:27:13
say a vulnerability of any type even one here whatever here so i can go in here and look at a component and i’m just
1:27:20
doing it from the easiest place possible i can say okay i want to assign a custom issue now this custom issue allows you to you
1:27:26
know give it a reasoning behind it uh you could change its severity if you want you can give some sort of type
1:27:32
justification you can even save for the single version of it or a range of versions in which you want to do it and
1:27:38
then you can go in and add additional metadata properties like uh you know we we’re going to accept this right now but
1:27:45
let’s go ahead and um let’s let’s flag it right let’s label it and then i can query on it later and say um show me all
1:27:53
the you know in this case we can go to like artifacts here i can say i want to do a search i want to do a search on
1:28:00
properties i want to do a search on key value pairs where i say you know review later
1:28:06
true right i can spell true um do a search and that will find me all the you know
1:28:12
components that i flag with this metadata property and then i can go in and you know from in here i can even
1:28:18
throw exceptions in also so there’s ways to do this and i can also produce reports on whether or not i had you know
1:28:25
any violations that i’ve gone and ignored and you could also do the same thing from here also by the way um of
1:28:31
course i picked one with no violations of it but um you know the ability for you to go in here and and actually say
1:28:38
you’ve accepted this and then you can query on it afterwards and you can do that from the cli also
1:28:45
you can actually go and use our cli if you don’t want to go into our user interface everything i show you is api-able too
1:28:53
ours is just a simple view interface on top of our api i’m not gonna there’s no magic
1:28:59
thank you no worries um let’s see i think there was any more
1:29:05
i’ve only got a minute or two left unfortunately guys i do need to go uh i eventually need to eat um
1:29:14
i’ve been i’ve been on i’ve been on the phone since 6 30 a.m that’s the problem with having international i actually have a phone call tonight at 11 pm with
1:29:20
my my office in tel aviv um so let’s see here um
1:29:26
yeah uh any other questions or anything guys if you want to ask me ask me now yeah maybe last one question
1:29:32
sure we all have to eat i’ll song and dance for you guys if you want a little salsa
1:29:40
no we’re good if not great thank you very much again thank you bill thank you willie thank you
1:29:46
everyone thank you guys okay thank you bye-bye be safe