A Look At The Kubernetes SLSA Compliance Project – Kubernetes Philly Meetup

0:04 / 1:01:40

KubePhilly March 2022- A Look At The Kubernetes SLSA Compliance Project
45 viewsMar 11, 2022

3

DISLIKE

SHARE

SAVE

Kubernetes Philly
59 subscribers
Presentation:

The Kubernetes Release Engineering team is working towards higher levels of SLSA compliance in its release process. SLSA, or Supply chain Levels for Software Artifacts, is a security framework to gradually increase the security of software releases.

In this talk, Adolfo will go through the changes that the Kubernetes organization is currently undergoing, like signing images and artifacts, to reach higher SLSA levels. Join this session to understand how these improvements will directly impact Kubernetes users and other projects under the k8s organization.

Presenter:

Adolfo García Veytia (puerco) is a software engineer with
Chainguard, Inc where he works helping open source projects
achieve better levels of security in their release processes.

He is also a Technical Lead with Kubernetes SIG Release. He
actively works on the Release Engineering team, specializing
in improvements to the software that drives the automation
behind every Kubernetes release.

Adolfo is passionate about writing software with friends,
helping new contributors, and amplifying the Latinx presence
in the Cloud Native community.
0 Comments
Ari Waller
Add a comment…
Transcript

0:02
hello everyone this is james strong and this is team philly for march 10th 2022.
0:08
as always this is a cncf community event so that means it adheres to the cncf code of conduct if you have any issues
0:15
with that plea with any issues please report that to me or someone from the code of conduct staff so today we have a
0:23
really awesome meet up for two reasons one we’re announcing our sponsorship from jfrog so ari from jfrog is here to
0:30
tell us all about that and we have the sig release lead from kubernetes talking about salsa so we’re going to learn
0:36
about what salsa is it’s not just something delicious to put on your chips and
0:42
we’ll talk about how kubernetes is working to integrate all of that so with that um i think so like i said
0:49
generally we start with uh opens uh anyone who wants to introduce themselves new to the group um any projects you’re
0:57
working on if you are looking to be hired or are you looking to hire um
1:02
open the floor to the community for any announcements that they would like to talk about
1:09
no pressure actually actually i’ll put one out there james uh at jfrog we are looking for a
1:15
director of developer advocacy um rvp um is looking to hire somebody who we have
1:22
a great developer advocate advocacy team but we are looking for a developer i was gonna be a director so if anyone has
1:28
experience in the developer advocacy space and developer relations would like to take on a role like that uh let me
1:34
know awesome thanks for that all right well if no one has any
1:40
announcements i think the only real announcement i have is that i know the kubecon schedule for
1:46
eu just got posted so that’s out there look and uh see if there’s any interesting topics of course if you
1:53
can’t make it to valencia spain that’s pretty high bar you can still join virtually i know that at kubecon
2:02
north america in la they were doing the multicast of doing lots of
2:07
presentations in person but also broadcasting them so you can still join virtually um
2:13
so still something for everybody and i know i’m gonna you know
2:18
put this announcement out there i know adolfo got one accepted i unfortunately did not get my talk accepted that
2:24
happens you know with all of the talks so pretty exciting from that perspective
2:29
um with that i don’t think there was any other announcements that i saw i’ll go ahead and we’ll hand this off to
2:36
ari our new sponsor from drayfrog so ari tell us all about uh jfrog and what you guys are
2:42
doing to help us okay awesome well first of all thank you uh james for the opportunity i’m gonna go ahead and share a slide with
2:49
everybody one of the benefits to having a maybe a small attendance tonight is actually somebody here who really has some great
2:55
odds in winning our raffle tonight but uh just in brief my name is ari waller and i am the meet up event manager for
3:02
jfrog yeah it’s a real title and i and i got it so i’m i’m very very pleased with that it’s uh pretty exciting to work
3:08
with meetups for a living especially uh even though it’s been virtual for the last two years we are
3:13
getting back at the space meetups and uh i’m very very excited about that uh happy to be part of the philly
3:19
kubernetes community while i live in atlanta i am actually a temple graduate so um i graduated from temple university
3:25
way back a long time ago um so i always feel like philly is a home place for me i’ll share just a little bit about who
3:31
we are um jfrog is the devops software company best known for artifactory
3:36
um all right before you keep going does the slide look look all right for everybody it looks like it’s pretty cut
3:42
off i can only see like the top slide i can only really is that better or no
3:48
there we go yeah that’s better oh okay cool i’m glad you told me about that thank you
3:53
um yeah i was wondering maybe i you know i may have pulled it uh oh i see what happened okay
3:58
everyone can see it now yeah all right good deal um i’ll share just a little bit about who uh who we are jfrog
4:05
is the devops software company best known for artifactory which is considered to be many be the gold
4:10
standard for managing your artifacts and dependencies but um besides the raffle i’ll tell you about just in a few
4:16
moments uh we want uh jfrog actually has a free version of our cl of our cloud
4:22
software and a free version of artifactory available to the community now it’s not a trial version you know
4:27
it’s not something that’s going to run out and suddenly you’re oops you’re a customer it’s uh something that if you play with you um you know you can
4:33
continue to use it no um no credit card is required to use it there is one portion of it from a pipeline standpoint
4:40
that is requires a credit card that won’t be charged just to prevent fraud
4:45
um with uh but other than that it’s uh yours to keep and play with as long as you would
4:52
like so uh that’s something there and if you were interested in that you can click you can scan the qr code or use
4:57
the bitly link which i’ll drop in the chat in just a few moments now also if you’re working with docker like i know a
5:02
lot of you are uh you can use artifactory uh as a pull through as a
5:08
pull through cache and you’ll be exempt from any rate limits on polls from free or anonymous accounts um that’s because
5:14
of a partnership that we set up with docker as soon as they put those in place uh jfrog was one of the first to
5:20
set up that arrangement with them and that will exist today so that could be a value uh to you and even does work in
5:27
our free version so um there you go uh tonight uh tonight’s raffle is uh
5:33
going to be for uh oculus quest 2. so someone will have a really cool opportunity to win that now we don’t
5:40
unfortunately i can’t because of compliance purposes do online sweepstakes as a public company uh for
5:46
various reasons i mean uh but we do pick a random we do pick someone randomly within two business days we
5:53
contact you by email and once you formally accept the prize um we would be
5:59
able to um share we will of course share the winner with the entire community
6:04
so we can all celebrate with you we’ll even send a tweet out on your behalf uh as as the winner of that so you can scan
6:10
the qr code there and use the bitly link also to take part in that and i’ll also
6:16
drop those two links in the chat but uh all in all we’re just happy to be part
6:22
of uh part of what you have going on there i’m excited i can’t wait to get up there in
6:27
person when you start holding them in person james and uh really looking forward to adolfo’s talk today so thank
6:33
you awesome that’s uh that’s awesome um definitely we’ll use that uh that free
6:39
cloud subscription it’s always nice to test out tools and things that we’re trying so i appreciate that that’s
6:44
awesome pleasure good deal all right and with that the uh the man
6:50
of the hour adolfo go ahead and uh um introduce yourself and uh we’ll uh
6:56
let you uh teach us all about salsa awesome uh just let me
7:03
share the presentation and please let me know if you can see it
7:12
yep i see that i see the little squirt down an octopus i’m sorry
7:17
exactly um yeah so first of all thank you james for having
7:23
me i’m really excited to share some of the work that we’ve been doing to to make the kubernetes release process a
7:31
little bit more compliant with the salsa framework which we’re going to be talking about in a
7:37
little bit so first of all let me tell you a little bit about
7:42
myself uh my name is adolfo garcia and i work at a happy place called
7:49
changan which is a supply chain security company uh i am also during the night times mostly
7:57
well now now it’s my daytime job as well but uh for years i’ve been working i’m
8:02
most lighting and over the weekends as a technical lead for kubernetes release
8:09
i have two girls and uh i like to go around the world and my bike whenever i can at least i used to before when i
8:15
make it and uh you can find me us at puerco on most
8:21
twitter slack kubernetes etc so
8:26
uh before we start i will i would like to just uh so if i have some takeaways uh i would
8:34
like to to to the so some turkey what i would like to to leave you today is
8:39
uh first um i want to share the assessments uh the
8:44
way we did the assessment to see how the kubernetes release process stands in light of the salsa framework
8:50
i would like to share some of the successes that we already had with with this effort and which are already
8:57
working for you as kubernetes end users or repackagers even and also some of the
9:05
challenges that we’ve faced so far and that we are still currently facing
9:10
um okay so first before we start i would like to do a little bit of a salsa introduction
9:17
uh so salsa which is an algorithmic an acronym that means supply chain levels for software
9:24
artifacts it’s a framework to uh ensure
9:29
that you are that attempts to guide you to secure your software releases
9:35
and if you think about [Music] a release pipeline for example
9:42
it’s there are many attack vectors that can compromise the artifacts that you’re
9:47
putting out so it’s a complex thing to secure uh your release process and it’s also an
9:54
even bigger and even bigger uh even more complex uh fact uh
10:00
then an even more complex task to secure your software releases if you
10:07
think of them inside of a broader supply chain so salsa attempts to
10:15
give some recommendations on how to secure your releases by giving you gradual uh
10:23
levels that you can start applying uh and that also will give you a better
10:28
understand understanding of your of your release process um
10:35
okay so the next thing i would like to briefly touch is uh how
10:40
kubernetes is released um the kubernetes release process is
10:46
powered by a tool called krell which is an acronym or cellular acronym for kubernetes release tool
10:53
and this this tool uh is designed it does many things not only
10:58
releasing kubernetes uh it’s also used for uh generating the release notes for
11:03
kubernetes it’s also used for publishing cve information whenever there’s a vulnerability in kubernetes
11:10
and it also builds um uh it also has a mode to build kubernetes daily so that end-to-end
11:17
tests can run and several other things but it also the the the main part that concerns us today
11:24
is that it runs the kubernetes release process uh so
11:30
krell has two main modes of operation first staging and release so the idea is that we
11:35
every time we do a kubernetes release we stage everything and make sure that the artifacts look correct and that
11:43
everything is that looks like as expected then we do a promotion of those artifacts and
11:50
then release them the promotion step in the middle what it does is that uh we
11:57
let’s say we open a pull request for uh
12:02
for uh promo uh proposing the images that are going to be released
12:08
uh and then it is signed off by some parties the release managers generally
12:13
and after approval it goes they they get promoted and uh they’re available to the public
12:20
so some quick facts about the releases we produce over 30 container images of
12:25
all the kubernetes components for all the architectures we produce over 50 binaries of different kinds
12:31
also for different architectures and operating systems we produce a set of tables not only the
12:37
source code terrible but there are some terribles that contain some parts of the of the systems
12:43
uh for example uh if you think there is a table that contains all of the binaries for the node that they
12:50
so those are in there we also produce and a set of s-bombs uh
12:56
a software build of materials and some provenance metadata these two items we’re gonna be uh explaining a
13:02
little bit more later today and the release process also handles some
13:08
atom automation that updates the the github page produces a release notes
13:14
it sends out the announcements that people overshoot to the mailing list and so on and then a final
13:21
part of the release is that we produce rpms and dev packages
13:26
and these are not done by us they are they are
13:32
built by a team inside of google because of they have some special access
13:38
that is in it to build them and uh which is an important part that we’re going to talk about in a little
13:45
bit as well so uh the main effort about making the
13:52
kubernetes release process salsa compliant is tracked by a cap which is a kubernetes enhancement proposal
13:59
which is kept 30 to i’m going to have links in this presentation which maybe
14:04
we can share at the end uh for all of this for all these uh
14:09
things that we were talking about uh so the cap proposes a schedule of enhancements that
14:17
we have to do to to meet the highest uh level of salsa uh that we can
14:22
so if you remember from from for the new for the folks who might not
14:28
know can you explain what a cap is oh yeah yeah so the cap is um it’s a public
14:33
public document that proposes an enhancement to kubernetes uh specifically to to
14:40
whenever an enhancement um affects in end users you have to uh write up a cap and uh all
14:48
of the all of the stakeholders generally other sikhs
14:53
will go and comment and you can discuss uh how the enhancement will be handled and
14:59
the schedule and all of the infrastructure needs that it has and uh and if it if it will have an
15:07
effect on downstream consumers and also the documentation that is going to be put
15:13
out and then after discussing some of that it merges and then you can use that
15:18
as a as a guiding document for for the effort that you’re conducting so it’s not needed for every pull
15:25
request in the kubernetes just for changes that will affect end users exactly so if you propose for example
15:32
changing a flag inside of the cubelet you will have to put out a cap discussing and proposing and explaining
15:39
why you think it’s a good idea and others will comment on that and if it gets accepted it merges and then you you
15:44
do that so the the cap that we uh that we open for salsa compliance is
15:50
is the step number 3027 which which is titled uh the the full title is
15:58
uh salsa level three compliance in the kubernetes release process so if you think about this the levels
16:04
that we just saw salsa goes up to level four but there
16:09
the the the challenges to comply with salsa get harder as you move up
16:14
so it was uh it was decided to to scope the cap only until salsa 3 because there are
16:22
some challenges that we’re going to see later that we think that we cannot uh we
16:28
cannot comply with so we sculpted to level three so
16:34
uh okay so last year we started the salsa
16:39
challenge uh this is a photo of me uh with a couple of salsas i am i am mexican and
16:46
i live in mexico so that means i was born with a professional uh degree of to
16:52
be able to talk about salsas and tacos and all of that so uh this is also going to be salsa
16:57
thing so we get get ready so last year around the mid yeah it was about one
17:04
year ago we started the process too and doing an analysis of the kubernetes
17:09
release process and making sure well first doing an assessment of where
17:15
it stands and matching trying to match the kubernetes release process with the
17:20
the the salsa spec that was out at that time with trust 0.1
17:26
and we built this this spreadsheet where uh we made it public and everybody could go
17:32
and discuss and comment about each of the requirements of salsa we’re going to
17:37
go level by level and trying to and i’m going to explain some of the challenges
17:42
that we’re facing in how those were sold some of them we didn’t have to solve
17:48
them because as the release process runs inside of google cloud build so some of the some
17:54
of the requirements just come for us for free via the field service we use
18:00
uh but we’re going to to go step by step and see each of them so
18:06
the first one uh salsa level one uh salsa level one is titled
18:11
documentation of the build process and it involves uh the following uh it reposes the
18:19
following requirements so the first one is that your source code
18:24
has to be version control which most modern projects are anyway because most people handle them in git so that that
18:30
is that’s a given then your bill has to be scripted in the
18:36
sense that uh you cannot be issuing commands to do your build by hand you
18:41
have to have a script that runs it then you have to have uh provenance metadata available which
18:48
is uh a format uh provenance what it means uh what in what it means
18:54
in in the in the in the context of supply chain security is
19:01
some metadata that your build process produces that allows you to track
19:06
uh [Music] each of the artifacts allows you to verify that they were produced at some points
19:13
in the real process we’re going to see an example of this in a little bit
19:18
then the provenance should identify the artifacts identified
19:23
the builder that built the things and it it also should list the actual
19:30
uh instructions that went into building your software release so we
19:36
took this the to to make sure that we comply with salsa level one we
19:41
went back to the to the spreadsheet did the analysis and determine that we were uh at like this
19:49
so at that point we had version control and a scripted build but we in order to
19:54
comply with salsa level one we had no problems so we had to to to wheel that
20:00
into the release process which we did and we actually modified
20:05
the release process in order to ensure that when we stage
20:11
the build it produces a prominence attestation and that provenance at the station gets checked before releasing to
20:18
ensure that the artifacts look and have the integrity that the proven
20:24
assad station specifies so we did that and we announced it during kubecon
20:31
in los angeles this was a moment when where where we announced that the kubernetes process
20:37
was salsa level one compliance so we’ve before we go further uh to the
20:44
next levels i would like to show you just briefly what the
20:49
the actual provenance looks like uh let me switch my screen
20:55
and share my terminal oh you’re doing that has anybody heard
21:00
about salsa or worked with um write about salsa before today’s meetup
21:12
okay so probably so ah go ahead
21:18
no i was just waiting for folks to answer the question are you good
21:24
yeah so i think the important part to to to to say here is salsa is a project
21:31
that is now under the open ssf foundation the open source security foundation
21:38
and it has a very lively community so far
21:44
where the uh there are folks involved from
21:49
many many companies and it’s it’s uh you can
21:54
join the calls there every other thursday and i think i i have a link at the end
22:02
of the day presentation and when we share the presentation there you can you can
22:07
follow the links there so can you see my terminal
22:16
yes okay is it big enough yes okay cool
22:22
so let me
22:28
let me show you inside of the pocket of a kubernetes release
22:35
so this is the bucket where the 1 123 4 version is located and you can see
22:44
so this is everything that is inside of the release uh some of the terribles
22:49
that are here there are this is the bill of materials
22:55
there’s a over here the directory with the binaries and so on
23:00
so let me get
23:08
uh first of all let me show you the one of the s-bones
23:13
so that you can see the information that we are producing
23:23
so this is uh i’m not going to go into what an s bomb is very deeply just
23:30
to say that an s bomb is like a file that lists everything that we are putting out in our release it lists all
23:37
of the images all of the all of the binaries the dependencies that go into the into the code so that the idea of an
23:44
ice bomb is that whenever you if a vulnerability is announced for example you can take the s1 method
23:50
software you are running and ensure that that vulnerability is not or
23:56
check if the other vulnerability applies to you so an s1 looks
24:02
like this it’s a it’s um it lists all of the all of the files
24:07
there their hashes and so on the licensing information everything is in there uh
24:12
there are um again if someone is interested i can share some presentations that we’ve done
24:18
about this bombs in the past but i can show you uh
24:24
what this looks like in there yeah i just linked them your blog post about what what is an ass ball okay cool
24:32
so this is let me show you
24:37
some maybe okay so let me show you what it contains
24:44
inside of the of the s-bone so this s1 describes the kubernetes release so you can see here these are the images
24:52
uh it’s uh this is this is done by a tool that we produce in sig release for
24:59
writing and working with s-forms it allows you to see inside of the s1 to check the
25:05
what what’s what’s contained in the document because if you remember the the source code of the of the s bomb
25:11
is very hard for people to to actually parse the information and this allows you to visualize it a little
25:17
bit so if you see the s1 it contains information about the images all of the
25:22
layers then further down below
25:28
so these these are the actual files that are included in the release like each of the binaries for
25:35
for the cubelet the api server and all of the cubectl all of those are
25:41
contained in there so keep in mind that one because the s1 is going to play uh
25:46
some part in the in the south side compliance effort and the other one that i wanted to show you
25:53
was the problems at the station which is
25:59
this one should be
26:09
let’s see if this okay yeah okay so this is a provenance attestation uh the provenance of the station like i was
26:15
saying before uh lists all of the files uh that were produced during staging
26:22
with some uh hashes to ensure their their integrity and uh it allows you to check that
26:30
one step of a build matches uh
26:36
so that one step of the build receives what it’s expecting from the previous step
26:41
so uh this lists all of the artifacts that you produce during the build but the the
26:46
the part that i wanted to show you is here so in the end uh this part of the build lists um
26:54
where it was run so it’s in cloud build it lists the the the configuration that we
27:01
use to run the build it run it lists the the actual git commit that where we were building so if
27:08
you go back in the kubernetes source code history you can go to this commit and ensure that that’s where version
27:15
123.4 was was uh was caught um and then there
27:21
there is other information about the release process itself so the date that the invocation
27:27
so this invocation id you can actually go and check inside of the google cloud
27:33
build uh logs and that’s when when we run and the uh
27:39
here is the they will comment again so i just wanted to show you this because i i didn’t want to be just like a
27:46
conceptual thing i wanted you to see the the actual file so let me switch back to the slides
27:54
to continue on the preventation
28:00
okay where’s the window okay here okay
28:07
so where we stand this at salsa level one that’s uh that’s the part that we did
28:14
last year and we’ve been working this year to comply with uh salsa level two
28:19
so just um so the way we schedule the cap is that we wanted to comply with
28:25
salsa level one for kubernetes 123 which is done then salsa level two we’re
28:30
attempting to get it ready for uh kubernetes 124 and
28:36
maybe if we can reach salsa level 3 by kubernetes 125.
28:41
each of the releases takes around four months and
28:46
it involves uh work that also applies to other caps which we are going to see right now so
28:53
uh giving another an overview of salsa level two
28:58
these are the requirements that are on top of of those of salsa one so the
29:03
first one is that uh builds should run in a build service
29:09
they should be authenticated service generated means that that the
29:15
actual build service should give you the prominence and then
29:20
it should identify some of the the build point that you that you were running when you
29:27
got the release so this is the way that we consider it to be so we are running in the build
29:32
service which is google cloud build the the build is authenticated and
29:38
so not um so the build is authenticated it means that
29:45
you can actually verify it uh via the other stations that the the bill was run by someone
29:53
but but that that the provenance actually was generated by by the the build service this is this is
30:00
not in place yet and also uh so the idea is that we are currently
30:06
working on a big signing effort of all the kubernetes artifacts so we are
30:11
working on establishing uh with a six-year project establishing
30:17
um a delegation of trusts from six store
30:22
if you’re not familiar with sixer it’s a it’s another project of the open ssf uh
30:28
which allows you to do easy signing of software and you can
30:35
via a very easy to use a set of tools sign your container images sign your
30:41
binaries and the idea is that we are building all of the six store tools into the
30:47
kubernetes organization so all of the projects that go that are
30:53
release artifacts built as part of the kubernetes organization are going to get signed
31:00
and one of the things that we need to sign is the provenance attestations uh are
31:06
the problems at the station so once we get the signing effort into place we will be able to meet the point number
31:13
two here which is authenticating the the provenance of the stations
31:18
then the third point is uh if the if if if the provenance so the the
31:26
the the language there is a little bit big so the build service is uh is is generating
31:34
uh so the build service should generate the at the station so if you think about uh
31:41
running for example uh your builds in github actions there is a chance that github
31:47
actions could at some point give you a provenance attestation of your artifacts and without you doing anything
31:55
uh the same could be true for cloud build it could run and give you the attestations that you need
32:01
to comply with this point those two currently do not give you that but they
32:07
when we run the builds our build system generates those attestations so there is
32:13
currently some doubt if this counts to to meet this point or not
32:19
because we do get the provenance at the station without any human intervention and runs and gets produced inside of the
32:26
service so uh there’s still some discussion if if that is valid to ensure this from
32:32
this uh this point here salsa is still currently under development so the current version is
32:38
0.2 of the specification and people are bringing their feedback all the time and
32:44
it gets incorporated into the specs all the time so it’s rapidly changing
32:49
so our seats as it stands right now we believe that we may be able to comply
32:54
but it’s open for for interpretation and then finally the
33:00
identified source code as we saw in the in the example file of the of the provenance that the kubernetes field
33:06
process produces we do list the the the git commit where it
33:12
was uh produced okay so
33:17
next point salsa level three so salsa level three is uh so the the first if you think
33:26
about the evolution of the salsa level so the first one allows you to know what’s going on
33:31
inside of your build process then the second level allows you to make
33:36
sure that that information that you’re getting from your build process cannot be falsified and
33:42
is actually signed and so that there’s no no question of its
33:48
authenticity and salsa level 3 attempts to plug some of the common thread vectors for
33:55
for build processes so these are some of the
34:00
these are the the the next requirements to meet with salsa 3.
34:06
so the first one is the verified history means that uh
34:13
each of the changes to the source code has to be strongly authenticated so it has to be
34:19
uh well strongly authenticated it’s also uh open
34:25
uh for for uh interpretation uh we in kubernetes uh
34:31
well let me jump to the to the next slide which uh
34:37
allows uh which is colored so that we can discuss each one so if you ever contributed to kubernetes
34:44
all of the all of the whenever you open a pull request
34:49
you have to to send that uh with uh with your credentials
34:54
and also whenever those requests are authorized
34:59
and approved and lgtm by someone uh they
35:04
have they get an identity which we consider it matches this point then the next point build us code let’s
35:11
leave the two red ones until the end i was going to say one of the other points in the in the kubernetes
35:17
community is that you have to have multi-factor authentication turned on to your github account to order to put in a
35:24
pull request so that’s one of the things from a strongly authenticated perspective
35:29
and now and also the reviewers have to have to match those requirements so uh we think that we’re good on that
35:38
side uh let’s leave the red ones for this question at the end um
35:43
then the wheel series we have a we have a question in the chat
35:48
how are you ensuring dependencies are not vulnerable during the build process
35:54
so we we so we we don’t have a way to of knowing that just now uh whenever so the there
36:03
are some efforts in the um in seek security to implement scanning of code there are
36:11
some efforts also to to automate the incorporation of cvs into into the
36:19
the base images that we use in the dependencies but we simply run and
36:25
pull the dependencies we do have uh turned on for example dependable on the some of the
36:32
repos but not all repositories have it and whenever
36:38
whenever we put out a release we attempt to we
36:43
do like uh like strong attempts to to to ensure that so we have
36:51
the dependencies up to date but uh but those are up to date uh
36:57
as the dependency uh admins i think they’re called so if
37:03
you see the code in kubernetes kubernetes dependencies are tracked by a specific
37:08
group uh that handle that part of the code not everyone so if you for example if you attempt to put a pull request
37:15
that updates a dependency it has to be uh authorized by a set of uh of people
37:21
that handle that there’s there’s a document somewhere that that that has a dependency policy that uh
37:29
that establishes that that project so we on the secretly side
37:35
just make sure that all of the dependencies we are pulling match those uh set up by the code
37:41
and then trust that from the from the dependency administrators
37:48
i don’t know if that means that sorry it’s currently in process
37:54
okay all right so uh okay the then going back to the
38:00
requirements uh the isolated requirements of the build process
38:06
that one i signed it for us for free by google cloud build um so whenever you run a
38:12
google cloud build it will launch your build in an isolated environment so
38:17
that that is uh complied with the non-falsifiable
38:22
um oh okay so this one okay so the non-falsifiable uh aspect
38:31
what it means is that whenever you put a prominence file
38:36
your your users should not be able to falsify that we ensure that at well
38:41
it is not yet in place but once we start signing it we’re going to be able only
38:47
the release process service account is able to produce and sign that those those
38:54
those files so we we consider that to be it will be handed down by the signing
39:00
efforts on salsa level two uh when we comply with salsa level two
39:06
then the entry point uh is the the same part of the the same part of the of
39:13
establishing the actual build point of the git commit that we’re running the build from then uh all the build parameters if you
39:20
remember the satellite stations that we’re putting out right now they already have the they built
39:25
parameters parameters in there so we we are going on those now on the on the part of the end of the
39:33
upcoming efforts to comply with salsa level three the first one is build as code
39:39
so whenever the buildups as code requirements there is a there is a
39:45
project which is in just a just a plan there is no code yet
39:51
for it is that the idea is that we will be able to have inside of the of one of the the
39:58
kubernetes or release engineering repositories we’re going to be able to have
40:03
a definition of the bill that we want to run where we will list
40:09
for example the the the build point we will list the the
40:14
actual platforms that we expect to build and things like that and that is going to get checked into the repository
40:20
and the build is going to run configured configured by those
40:26
the idea of this is that currently the way the the the build process runs is
40:31
that one of the release managers executes the
40:36
the build process from their computer the the build doesn’t run in the in the
40:42
in the that person’s laptop it runs in google cloud bit but it is launched by uh the release manager
40:50
it does some some things to to to ensure that uh for example
40:56
a release manager cannot specify the build commit where the build process runs it it you
41:02
know it all gets uh determined by the actual release
41:07
process uh but when when for example when releasing
41:14
you you can you can there is there is a chance that you can specify some uh
41:21
another another viewpoint if you because sometimes we do several stages
41:27
if something fails for example so when you release you tell the release process to to to use as for example the next one
41:35
is going to be i think 123.5 you can specify you you you’re you have
41:40
several builds staged in the in the in the pockets and you when when we release kubernetes we tell okay so for
41:48
kubernetes 123.5 use that this specific stage field
41:55
and the idea of having the build and release as code is that
42:01
a file gets checked into the release uh into one of the secret list repositories
42:07
that is uh validated by some of the release managers and then we released based on
42:13
the information on that file so that they’re going to be uh either error or
42:22
malice when when releasing them so we have to with that that project is still
42:28
just planned uh if someone wants to help building them and you are more than
42:34
welcome to come and help building the build that’s code for kubernetes and then
42:39
finally the ephemeral environment okay so that that i i made a mistake so the female
42:47
environment is in place and we get that handed down by google cloud build as well so
42:54
when you the idea of that point is that you run your builds and it runs either in a
43:00
container or runs in a virtual machine that runs executes the build and gets disappeared so that you don’t
43:07
that there’s not like no risk of getting a compromised environment handed
43:13
down by a previous process that run in it uh that that one should be you should be green because it’s it’s already
43:20
uh in place okay so now uh the final
43:27
level of of salsa salsa level four this is the hardest
43:32
level to to to to comply with and we decided to leave this out because
43:39
it involves a lot of human human efforts that have to to go in
43:45
place which we’re going to talk about briefly so these are the first ones let me
43:50
switch to the color slide so the first one parameter less
43:56
like we said currently a race manager can pass some parameters to the build system and may be able to
44:03
do some tampering in there the idea is that whenever you run your builds they don’t have any parameters
44:09
it all gives get gets defined inside of the build as code that we
44:14
looked at before before the slide then the hermetic builds we don’t comply with
44:21
them because the idea of a hermetic build system is that it doesn’t have any access to the
44:26
network so if it gets compromised uh you may be able to scan your build
44:32
system see nothing there but if it gets compromised at some point of the build there if you have it aromatic it doesn’t
44:39
have a way to reach out to the internet and download malware patches or whatever we don’t comply with that one the builds
44:46
run with full access to the internet because we pull all of all sorts of things right
44:51
now uh if you think when when a build runs it holds down dependencies it runs
44:57
people’s and licensing information it talks to github for several things to
45:02
to docker to pull some base images and so on so that we don’t play with that one then reproducible uh so reproducible
45:09
builds are a subject of their own and could be uh could be
45:15
the subject of a talk by itself so the idea of a reproducible build is that if
45:21
you run your build at the same commit point over and over you get the same byte for byte
45:29
files every time you run it so it doesn’t did it doesn’t so kubernetes doesn’t have
45:35
that yet i doubt that we will ever be able to have
45:40
a reproducible because there’s entropy in there of all sorts from
45:47
uh from from from the dependencies and from from some of the
45:52
some of the processes that we run inside from signing things generating hashes and so on
45:59
then um so in the last the below two we do
46:05
comply with uh so if you saw the s bomb that gets produced with the kubernetes
46:10
releases currently it includes all of the dependencies of the software and it includes all of the
46:17
transitive dependencies as well so the difference between those two are the
46:22
dependencies are all of the go dependencies that kubernetes uses and the transitive dependencies are all of
46:28
those that the the dependencies depend on so the two levels of of dependencies
46:34
those are included in the in the s1 that get through so we already complied with that
46:40
so these are more technical uh requirements and there
46:46
they may be that there may be a way to comply with with them but the hard ones
46:51
are the next ones so these are more human uh
46:57
human uh uh requirements that are harder so well the first one though the the first one
47:03
is the the information to reproduce the the the build as we said
47:08
we don’t have reproducible bills right now in kubernetes so there’s no information to to run the build and make
47:14
it and reproduce it but then security so the the security
47:20
the idea of that one is that your build process is certified by some
47:26
standard or or some underground some
47:32
security of it while there is a chance that we may be able to
47:38
do that there is there is still an ongoing debate in the
47:44
salsa community of how to actually express that in your in a machine learning way so
47:51
the expectation is that at some point uh there will be a security say some security auditors that will be able to
47:58
certify uh your building in in a way that satisfies satisfies the
48:03
salsa uh a spec and then produce some sort of an electronic
48:09
attestation that your build is compliant so that one is still in progress and we
48:14
are far from reaching salsa level four so we don’t comply and we currently don’t do not uh
48:21
we are not attempting to to comply with that one then uh
48:26
there’s the factor of access and the super users so these two have
48:33
a problem with kubernetes so some of the infrastructure
48:39
where kubernetes gets released is owned by google
48:44
as we were saying at the beginning of the talk for example the
48:50
rpms and the dev packages are cut in every release by a googler so all of
48:57
the volunteers that take part in sig release can well not not so all of the not all
49:03
of the volunteers but the release managers can any of them can run a kubernetes release they have
49:09
the necessary access they can run the process and and generate the images and generate the binaries but when it gets
49:17
to the rpms and depth packages those can only be done by googlers
49:23
and all they have to do so there is a
49:30
team of volunteers inside of google that handles this part but in in theory any googler can not can request access
49:37
um to cut the the kubernetes packages so if you consider it inside of the
49:44
kubernetes organization we have like an and no unknown on the google side
49:50
because we don’t really know who can access or or has
49:56
at any point access to to to the keys necessary to to sign the packages
50:02
so we consider that to be unachievable at the moment because we depend on a third party which the
50:08
we cannot ensure that we can restrict the access to
50:14
and the same thing as super users so
50:19
the some of the projects were so the kubernetes releases run on google
50:24
cloud and those projects are owned by google and some of them
50:31
are the owners of those are groups and those groups have access as super
50:37
users to the to the to the projects uh that we don’t control so that’s also one
50:45
that we cannot uh ensure that um
50:50
or restrict access to the to the actual project owners so
50:55
if you if you think of those of those factors uh we decided not to pursue salsa level four at the moment because
51:03
uh there are things that are outside of the release engineering team so
51:08
we’ll once we reach level three we may be able to start discussing some of those things
51:14
um and maybe attempt to get some of the salsa level four
51:20
requirements in place but so far it’s uh simply not possible
51:26
and yeah i think the the next and final thing is that i wanted to
51:32
invite anyone who wants to who who likes this sort of thing and
51:38
likes um exploring some of the edges of supply chain security
51:44
and because we are really working with the specs as they go to join and to join the the the effort
51:52
um we invite you to come to to uh to the sick release channels uh
51:59
which are sick release and release management on the kubernetes slack these are links for the for the caps uh
52:06
for the salsa level three compliance for the signing of all the kubernetes artifacts
52:12
which are we are currently building and uh some links i’m leaving here some
52:17
links for the repositories in secretly so that you can take a look at the tooling and and
52:23
my contact information if you want to to ask any questions or ensure that
52:30
then that or you or your all your doubts are are covered uh i hope
52:36
uh it it was kind of clear what i did and with that thank you so much and happy to answer
52:43
any questions around the the topics that judges cover
52:49
that was a lot of information i definitely will be going back over that recording and be asking you more
52:56
questions since i’m a co-worker and i can bug you all the time but this this is a lot of information
53:03
it was really helpful to go through all of that um folks do you have any
53:08
questions is was that just a deluge of information i’ll have to invite um adolfo back when
53:16
125 gets cut and you know check up on the progress
53:24
yeah at any time we can do like an update uh and
53:30
also so the i i think the the next big one is going to be when we start
53:36
signing the images because you’ll be able to to one verify
53:41
any of the artifacts that you download from kubernetes but also it will enable
53:47
tooling downstream uh that checks and what what if you if you build
53:54
any tool that consumes any of the kubernetes uh binaries or images
53:59
they will be able to ensure that what they’re pulling from the network is actually a validator effect from from
54:05
from the governor’s organization gotcha we got a question in chat from will the the saw suspect with the s bomb
54:12
spec come out first um that is a question i don’t know the answer to
54:18
um one of our other co-workers is on here i don’t know tracy do you know the answer that i’m gonna put on the spot
54:24
i saw you sneak in here yeah can you hear me
54:29
yeah yeah yeah and i think s-bombs have been around for uh quite a long time so i
54:35
would say they came first though salsa well fairly recent um it is based on an
54:41
internal google system which has been around for about eight years so
54:46
in its current incantation as an open source thing it’s it’s relatively new but it is based on like eight years of
54:53
experience from google yeah gotcha awesome thank you for that um also folks
54:59
did ari did place in there the uh things for the raffle so go ahead and make sure to
55:05
follow those links and sign up for that free subscription
55:11
uh well another question will salsa be merged with any fips or fedramp requirements
55:17
um i don’t know about being merged with those i know nist just came out with a new
55:22
framework uh ssdf so secure software development framework to help
55:28
augment sdlc stuff so they have requirements on there about how to do secure software development
55:35
i don’t know if anyone’s done a match-up between salsa requirements and the ssdf
55:41
framework so that’d be an interesting blog post to write
55:46
yeah indeed and also well one of the benefits of salsa is that it allows you to to do this
55:52
gradually so uh sometimes when specs come out they just there are lists of requirements
55:58
that sometimes are hard to match but uh salsa allows you to start
56:03
gradually and tackle the the most uh pressing
56:08
threats to your pipelines uh first
56:13
i have a question at alpha how many folks are they on the release team that
56:19
you’re coordinating across and using salsa as a way to focus everybody
56:25
uh so the the release management theme there are two two levels of release managers the first
56:32
one are the full loan release managers and we are
56:38
i mean i say we because i’m part of those i think there are seven no five
56:45
yeah five and then we have i think another seven release manager associates which are in the
56:51
process of learning and which at some point are are going to to be to become full-blown release
56:59
managers uh the difference between the two is that uh the the
57:04
full-blown release managers can at any time launch a kubernetes release and the associates they need to
57:11
get access first before they can trigger one of the releases
57:17
yeah and those and that secondary team that changes for every release cycle right because you can nominate yourself
57:23
to be part of that team no no no no that’s the release team so the release team okay is is like another part of the
57:29
google releases and they handle uh they don’t they don’t actually run the bills what they do is uh they handle
57:37
uh i think it’s five teams that they handle different aspects of the release from communications they handle the
57:43
release notes they handle the enhancements like making sure that
57:48
all of the proposed enhancements meet the deadlines to make it into the release we have a doctory ash team and there’s
57:56
also a ci signal that ensures that the tests are all green when running and uh
58:02
yeah and anyone can participate in those as well you can there’s a survey that opens up whenever uh a few
58:09
like a month before the so it usually goes out when um for the next release when one release
58:15
gets caught and you can sign up for that uh it’s it’s really hard to to get a
58:20
spot in there because we only so there are only like available spots
58:26
for around 13 to 15 percent of the of the applicants uh there is team is
58:32
somewhat large because it’s like it involves like 30 people but but uh we still get hundreds and
58:39
hundreds of applicants so it’s hard to to to get a spot but i mean you can try
58:44
and and uh and get involved usually uh so it’s it’s it’s
58:51
the release each of the of the leads of the of the release team is supposed to
58:56
there are some criteria to picking up the the the folks then then going to the team
59:02
and we try to ensure like uh the diversity both in
59:08
and origin of the people but also in a global time zone uh aspects
59:15
and um and yeah make sure and to go there whenever whenever we put out the
59:20
call you can go and ask any questions there are the guys for each of the team published in in the siege release um
59:29
the secretary’s repositories as well if you want to to get more information of the roles
59:34
awesome thanks for that alright folks do we have any other questions um i know there was a lot in
59:41
chat do you have any questions um before we break again make sure to fill out the survey so that you can get a
59:47
chance to win the oculus and jprog and the free class description
59:53
yeah actually uh one thing also i forgot to mention is that uh you know one of the reasons i joined
59:59
is because i’m not a technologist it wasn’t because of the devops software for me personally it was always the t-shirts
1:00:05
uh and uh for and actually and what i forgot to mention is that i wanted to let you know anyone that enters the
1:00:11
raffle there really are no losers if you want a t-shirt there should be a place in there for your address and your
1:00:16
church size um i’m not gonna come visit you or anything like that all i’m gonna do is put a t-shirt in the
1:00:22
mail for you with our black panther frog t-shirt which i’m not wearing um today but uh just want to
1:00:29
make sure that everyone here today will get one so if you want one it’s yours thanks that was a great talk adolfo thank you
1:00:35
for uh thank you thank you for sharing oh thank you
1:00:40
man thanks for that all right awesome well thanks everyone we don’t have an april talk scheduled
1:00:46
just yet as always i’m always looking for speakers if you have any projects or anything related to the cncf um
1:00:54
please let me know and we’ll get that scheduled
1:00:59
all right well thanks folks thanks adolfo again that was that was awesome yeah and also just to emphasize uh if
1:01:07
you want if you are interested in in all of the releasing of kubernetes come join us
1:01:14
join our calls are every tuesday uh in the american morning uh so uh
1:01:20
please come say hi and we’ll be happy to host you there
1:01:26
yeah i post a lot of links and i’ll get the recording posted as soon as it’s done rendering
1:01:33
thanks everyone thank you