Distroless Docker Images @ CNCF Eastern Canada Meetup

going up in a few seconds we’ll start
hello youtube
hello okay
so let’s start it for today’s meetup
or we wait one more minute
i think we can get going
yeah
yeah dude it’s still climbing like
i guess attendees are late today
i think we’re a good number now so
that’s that’s uh you know that’s a good
point to start 50.
okay
bus 50 let’s start that so hello hello
everybody and thanks for joining us
today
uh
i guess everything is done here let’s go
to the agenda
so today’s agenda we have a little bit
of housekeeping
we have the kipcon wrap up so we’re
going to present you a few of the talks
that our organizers
like the most
and then we have a great talk of about
this trolleys so how to secure docker
images using digital lines and the stock
is from melissa mckee from jfrog
and we have huge prizes today and thanks
to jfrog we’ll get to that later on
so our host today myself uh julia and
archie as always anthony
uh will also talk about his speaks of uh
um the cncf and kubernetes organizers
it’s a big family and it’s growing
almost every time
so thanks everybody for being part of it
so we have now two slides for the
organizers so thank you all
um
we have sponsors and and so thanks again
to elastic and jfrog for helping us
organizing all that and providing swag
and and whatever we need
that’s a great support
you can become a sponsor we have a qr
code here and you can find that in the
slides that we’re going to share later
on
uh so if your company wants to be part
of it you’re welcome and you will have
like time slots for speaking if you have
like great things to share with us so
let’s do that
uh so
let’s go
we already talked about that so cncf
update and the melisa’s talk i want to
mention that we also have a survey at
the end
and this is helping us to better
organize and and present things for you
so
please
fill the survey at the end
so as always
we there is different ways to reach to
us uh slack linkedin
and twitter uh please come to the select
channel if you have some questions
there’s great people there bringing a
lot of information and news about
currencies and cncf projects so
please join
we have cfps
so if you want to participate if you
have a talk to submit to us
the links are here
please
come to the slack channel or use these
links to submit your your talks
cncf announcement so
there was kubcon last not actually last
month this month
two weeks ago uh we have some stats
about this con it was very special it
was the first in the last two years
uh it was a great event so
a lot of attendees actually
virtual mostly virtual sadly
great stats
canada represents a lot in description
because it’s the third
overall
country of participants
uh
we’re very proud of that uh so for in
person 3500
and something
uh that’s great
uh that’s great for a first one in in
you know coffee time i can tell you that
everything was secure a lot of checks
everybody’s was wearing masks inside
uh
it
doesn’t feel like covet at all except
for like having face masks
and i’m sure
like archie or or or julia will be able
to comment on that but i had a great
impression
uh
and almost uh 20 000 attendees online
uh
what i can see it’s a lot of men and and
still we have very few women uh
to this kind of technical events
so uh
join us join us every time and and once
again here even virtual canada is the
third country of attendees so that’s
great we’re a great crowd
a few pictures
so you know capcom it’s keynotes
sessions
and parties actually and and meeting
people so that’s also a great part of
the con and and we have a big picture in
the middle all canadian folks
there was a lot more than that
but you’re hiding
so
the pigs
so
i only have one
because at the beginning we said like
everybody is going to pick one so i
picked one
it’s about salesforce explaining how
they build
a paas so platform as a service using
crds
so it’s the new
oam model uh where you define your
applications in operators and you have
everything inside the cluster taking
care of the ci cd
and bringing up everything that is
needed so the the users can
just describe the application and
everything is taken care by
by automation in the cluster that was a
very great
talk very inspiring
so that’s my take
uh
[Music]
anthony
yes sure hello everybody uh thanks
sebastian
so yeah i was over here uh attendee i
was part of 20 000 uh joining online and
yeah uh well first of all it was uh it
was a pretty decent experience actually
uh whether it was uh you know so there
were people presenting live and they
were being of course are live streamed
oh so there were there were also a lot
of recording talks i guess it was 50 50
um as um
to my opinion but i don’t have the
official uh stats anyways it was a very
pleasant experience following from uh
from my living room
so uh of course i still um well i have
picked uh three or three themes that i
really liked and that were super well
represented at kubecon first of all the
developer experience
the developer experience was really put
into in focus this year
many people many speakers were talking
about telepresence you know this dark
magic from ambassador labs but allows
you to actually have local processes on
your machine being exposed uh to your
remote cluster so a very nice tool that
is getting actually a 2.0 ladies i think
it was this year so it’s a it’s a really
nice tool to check out i highly
recommend it to you
new to me was bridge to kubernetes
something very similar from microsoft
that is well integrated with vs code and
same thing it allows also to send some
uh
some remote uh traffic to your local
process
a nice tool that i didn’t know about as
well was dapper adapter from microsoft
as well so if you’re familiar with istio
architecture it’s kind of the same thing
but focus for developers this time
meaning that you will have uh you will
be able with dapper to inject some
sidecar containers that will do things
such as resolving network uh mounting
volumes uh and so on and so forth so all
those you know those aspects of
configuration that can be how to do with
kubernetes then dapper is going to
actually um remove all the configuration
from your application and put it in the
centralized dapper place
so that definitely seems to be a good
tool to check out
porter also was new it’s kind of a
packaging for kubernetes i’m not sure
where that’s
going to go but the idea was you know to
be able to safely package uh some
deployments uh to kubernetes using this
tool
so that’s it for my developer experience
but as you can see uh below on the slide
apparently archie also picked up some
interesting tools so more to come
then
on the observability side open telemetry
everywhere this is the you know this is
the merge of open sensors open matrix
and open tracing this is the way to go
open telemetry is ga for traces there’s
i mean this is your go-to tool if you
want to do a summer choices
instrumentation for sure they even have
metric support now although it’s only in
beta and also they envision bringing
logs support as well
interestingly enough
the open source jaeger was
seems to be evolving into a full apm
tool so there were only about traces
before but now they really want to
encompass more observability pillars
finally github’s
guitars everywhere thanks to flux and
most certainly argo cd argo cd is now
so mature that there are a lot of
integration that appeared with it such
as argo cd application set where you can
actually bundle several applications
into argo cd easily i’ve also seen some
interesting plugins to integrate with
vault and overseas hat managers so yeah
go cd is really um
is really big now and uh superficial
that’s it for my text i will also place
some links
in this chat as well as on slack with
many uh with uh some
blog posts about
what happened at kubecon
thanks anthony
i actually want to add a few few more
words about argo cd that we had a last
meet up around argo cd and at kubecon i
saw two new companies that are packaging
argo cd and they’re hiring people so if
you’re interested to work in argo cd
space there are companies actually
hiring now and you can join them in a
different position starting from
software developer to developer
developers like melissa for example so
check it out and if you’re looking for
that uh let me know i’ll be happy to
connect it
but honestly for myself i was just happy
to be in person at keep going away and
to see all friends uh need the community
go to the rooftop parties and you know
have lots of swag so i had like two bags
of swag that i brought with me from uh
los angeles uh at cubecon i was a track
host so i was representing different
speakers before they present and i was
very lucky to represent uh katrina veray
from canada uh she’s working today at
apple and she’s a six-year-old lead so
she was presenting at custom around
customized
but most of the time i usually spend at
the booth so i go check my old uh
friends like a company like jfrog or see
the new companies who are there up there
and just trying to understand you know
what’s where the industry is going and
what’s happening and you know get some
swag
uh and obviously for me the main goal is
to find new talks and new sponsors for
meetups
so um i had to agree with anthony on
that the developer experience with
kubernetes is becoming one of the major
focus in this kubernetes space um
without going to the talks i was at the
booth
uh i checked few interesting companies
that i haven’t seen before that uh you
know appeared post-covered um so one
company is called garden and they have a
you know project garden that lets
developers not only have a developer
kubernetes loop locally but also
developers can run part of the ci
locally so that kind of helps developers
to have end to end loop
to understand their
software much better
obviously project tilt from cncf which
is still in sandbox but is getting a lot
of traction if you know scaffold a
project from google this is very very
similar to that
and i’ve met the devrel so they’re
probably going to come to present at our
meetup shortly
another project very cool is devspace um
it’s
for coming from company called aloft so
they allow you to easily deploy
kubernetes application using like a
docker compose style uh approach
and then finally a very funny name in
french project cui
uh so let’s build the framework for
enhancing clies
with graphics
i i just really like the name so i
brought it up
um the second trend that i’ve seen uh at
the event and i think most of you agree
as well is like around kubernetes
security and ebpf
uh so i really like project loft it
provides
tools to solve multi-tenancy problems
with kubernetes which i think everyone
is agree that is very difficult so they
have a project called vcluster that lets
you create virtual cluster on the same
kubernetes cluster and kind of provides
isolation for developers to run their uh
workloads and then a very interesting
project called gs policy so if you’re
familiar with opa and caverno this is a
very similar tool but uh you you kind of
creating the policies in javascript so
if your company have lots of javascript
developers they’re going to love it
but i think like one of the coolest
projects that i’ve seen overall it was
project called pixi which is sandbox
project in cncf that is powered by edpf
and provides you observability and debug
debugging for developers without any
instrumentation so it just magically
starts and then you can see all the you
know your services connections your
debugging your matrixes so it’s pretty
cool to check out you can install it try
it out i highly recommend has a nice ui
as well
uh so that’s that’s i think one of the
coolest things that i’ve seen
and finally i think one area that had
huge traction uh including the keynotes
uh was security supply chain and i think
it’s thanks to my friends in russia
that recently hacked solarwinds uh it’s
it’s kind of becoming a key key focus
for us
so
even in a kubernetes community we want
to that kubernetes uh
development and secure and the one that
kubernetes application is securely
deployed right
so the first initiative that you see
there it’s called six store uh it’s
initially started by red hat and google
that provides you new standard for
signing and verifying and protecting
your software
and it’s funny enough there’s few google
employers that left the google and
started a new startup called chainguard
so check it out the hiring right now so
if you want to work in the security
supply chain this is very cool company
to check out
and
another thing that has been talked at
keynote uh something called s bomb
software deal of materials uh this is
essentially you know we’re trying to to
make sure that even kubernetes pipeline
has you know all the all the
software that is dependencies that all
secure
so that was my kind of uh
very short
recap but honestly i haven’t go too many
talks i’m still looking forward to to
see the recordings that are available on
the platform or they’re going to be
available in youtube shortly uh maybe in
a couple of weeks
sebastian back to you
or julia yeah back to julia yay
um hi everybody i just wanted to take a
moment to um
i guess talk about my experience so my
favorite talk was my talk
um because it’s about a topic that is
very important to me and near and dear
to my heart
it was about my burnout and depression
and mental health um
journey and experience a couple of years
ago so i was really felt very fortunate
and i was very honored to be able to do
this talk in person
i done it i’ve done it a couple of times
online and it was a much more
nerve-wracking experience to do it in
person but it was very well attended and
our canadian community was
so supportive and put my little burnout
ribbons on their tags and i was really
really happy to have you all there and
all of you
who sent me messages after and who got
in touch with me it really um
i can’t express how much it means to me
so thank you so much um for
you know supporting my not just my talk
in the topic but also
um i guess my journey from you know
being a non-technical contributor to
being able to do this it was it was
really special for me
so um enough about enough about me um my
favorite talk at uh kubecon was one of
the keynotes actually because it was
about people um and i guess also the
developer experience but it was a
keynote by robert duffy and he talks a
lot about um how how they work together
and how meaningful that is so if you do
have a chance to check it out um i
highly recommend it was actually in a
bit of a series with the other co-chairs
of the conference so constance and
jasmine um and really kind of talking
about the human element which obviously
as you can see is my jam so i really
appreciated that uh this is starting to
become more of a topic and more of a
focus for
for these conferences and as archie
mentioned there will be talks available
it’s actually this friday as of the 29th
so all the talks from kubecon will be
available on the cncf youtube channel
as of this friday so you can check those
all out
and if you just want to switch yeah so
in vain of all this um a wonderful new
friend who’s actually here attending
thomas reached out to me and asked if we
would create a peer support group so
that’s what we’re doing so if anybody
wants to talk about uh burnout or just
how they’re doing and any struggles they
might be having or just wants to connect
with people in a different way
please join the cncf slack
workspace so the link is here
slack.cncf.io
and there’s a channel in there called
burnout and we are going to be doing
weekly huddles in slack in that space in
that workspace
so right now they’re at their thursdays
at 12 p.m eastern so if you’d like to
join us we’re going to be doing them
weekly for the first little bit and
we’ll see how how it goes and how people
like it but i just wanted to
give thomas a shout out and also support
or promote our little uh initiative here
so um hope to see you there i’ll pass it
back to you sebastian thank you yes
thank you i want to add that i was at
the julia’s talk it was awesome it was
crowdy
and actually it was a lot of men
usually
like
men we don’t we’re strong we don’t talk
about that and that was very great to
see how many they were and and i want to
relate that to to
uh of course uh carrie price experience
uh uh
like we as canadian have to to work
around burnouts and be be honest about
it and share
our thoughts and so that was a great
experience and thank you again julia for
bringing that
to everybody and supporting this
next evans actually so
in the coming months uh we have the the
in december actually china kucon
and then next year you have the dates uh
valencia spain and then next year forest
north america in detroit so very close
to canada for once
um
and then archie it’s your turn
all right
um i just want to mention before we’re
going to start with the presenter um we
going to have at the end of our meetup a
post event survey that will have a price
draw sponsored by jfrog so we’ll have um
kubernetes mask we have also
titanium key that helps you to protect
your secure supply chain from phishing
and
most importantly the best price is a
raspberry pi 4 desktop kit so please
stick with us until the end don’t drop
and get a chance to win the amazing
prizes from
a nice jfrog
company
and don’t forget to fill the survey
because you have to fill the survey to
be a person yeah yeah exactly yeah it’s
just a survey
and then uh finally
um
i i want to invite melissa mckay from
jfrog
so i was looking for her all the time at
kipcom but i couldn’t find her um and i
just find the fun fact about her she
never visited canada so we’ll try to
make sure we’re going to fix that
problem
and she will be able to come us and the
topic of today’s presentation is
actually very close to my heart i have a
lot of users and people asking me how to
run those you know docker images
securely on kubernetes cluster
so uh and i i heard about the project
that i think started by
google jfrog and few others um around
distroless docker images so when i when
i saw that melissa’s presenting around
this topic i was very excited and i
invited we invited her to present at our
meetup
melisa
welcome
i think you muted but uh yeah
yeah thanks for the introduction and for
the invite to canada i’ll definitely be
up there as soon as i can
uh one thing i wanted to say about going
to kubecon was
um you know i didn’t i kind of think of
myself as more of an introvert i thought
i was just fine not being around people
but i tell you what i mean going to that
conference fed my soul
in ways i didn’t expect i didn’t even
know i was hungry that hungry for
in-person events again so i’m looking
forward to the next one hoping things
improve across the country and we’ll be
able to continue doing that
all right
let’s go ahead and get started
so uh this is a talk that i’ve somewhat
modified to include um a discussion
about distrolus it’s a talk i’ve given a
few times and i have had a lot of
interest in and it basically it goes
back to the basics all the way back to
the developer when you are actually
building your images and when you’re
putting your application together so
this falls right in line with supply
chain security um you know these issues
have been around a long time and i’m
actually surprised that this hasn’t
caused us more problems
but obviously we’re hearing more and
more in the media about um
security problems with certain
applications in certain packages that
cause some
you know a lot of media attention and a
lot of problems for some companies i
feel bad that solarwinds is now the
example that we all give for how this
kind of thing can happen
so just to set some expectations up
front there is a lot of information
about docker in this talk but this isn’t
really a tutorial on docker or a deep
dive in docker commands or anything like
that
when i put this talk together i wanted
to do something a little bit different i
did a lot of background research to
focus more on how we got here and to get
some answers to some of the big why
questions around containers in general
and my hope is you’ll come away with a
better understanding of the history
behind containers how they actually work
on your system and some of what is
really going on under the covers
obviously this is a few steps back from
actually deploying your containers
using kubernetes this is all the way
back to building the initial images to
begin with that your containers are
based on so we need to shift that
security left all the way back to when
they’re first built
and and then i’ll lead into a discussion
about distrolus containers and what that
means
all right so let’s go
um
i have we do have an extra
chance for a t-shirt and a liquid
software book provided by jfrog
these are available at the link that you
see here on the screen
um
actually let me try
posting that link so that it’s a little
bit easier for you
okay i posted that in the chat um my
slides will be posted there the video
will be there eventually but obviously
this meetup group will probably post
stuff as well um the first 15 people
that sign up for this raffle will get a
t-shirt and a book so please go there
and you can also rate my talk there
all right um i was already introduced i
am melissa mckay i am a developer
advocate with jfrog real quickly uh my
background is actually as a developer
i’ve been a developer for 20 years all
the way from an intern to principal
engineer
but about a year and a half ago i
decided that i really really enjoyed
public speaking i enjoyed teaching and
it was getting to be a little bit too
much to do that on top of my full-time
job as an engineer so i was looking
around for opportunities to actually get
paid to do this kind of thing and jfrog
came about and i was pretty thrilled
it’s my dream job in a way i’m working
with some people i’ve known for years
and years and i’m really enjoying um how
this the developer relations program has
grown and started with jfrog
i am a java champion um primarily my
background is java server-side coding
and i’m also a doctor
doctor captain
you can reach me
on twitter on linkedin i’ve got my
contact information here
all right so like i said we’re going to
learn about history because sometimes we
just need to you know remind ourselves
how we got here to begin with
we’ll look a little bit about
look at containers in real life how they
look on your system specifically
my macbook because that’s that’s what i
used
and then container gotchas things that i
experienced when i first started using
containers and that will lead right into
distro-less images and what those are
all about
all right i have several talks now where
i have a theme and my favorite theme
right now for docker um
images and containers is bananas i
absolutely refuse to use
shipping container pictures and any more
talks or anything i’m tired of seeing
those so we get bananas and and i’m
sorry somebody says they oh sebastian
you hate bananas that’s too bad
here they are you’re gonna get a lot of
bananas in this talk
all right um
so in the beginning when all of this
started this whole idea of using
containers
there were obvious limits on time and
resources that created bottlenecks and
inefficiency with um just the physical
hardware even and i’m just being able to
share a workstation was not enough there
needed to be a way to share without
getting in each other’s way or having
one person inadvertently cause an entire
system to crash for everyone
fast forward to the future now we’re
looking at huge
infrastructures where we do you know
blue green deployments and we’re really
working hard on not destroying an
application for all users when we make
little changes
and want to update our stuff
but um back then
way back then i’m aging myself a little
bit
we needed better strategies in sharing
compute resources and that actually
started this path of innovation that we
see massive benefits from today
there’s some key points that brought us
to the state we are in
and i’m going to begin our container
history lesson with charute
that was born in 1979 during the
development of the seventh edition of
unix and was added to the berkeley
software distribution in
1982 being able to change that apparent
root directory for a process and its
children which tarot allowed you to do
resulted in a bit of isolation in order
to provide an environment for
testing and distribution for example in
fact that’s how it was introduced was as
a test utility which grew into something
more
so churu was a great idea solved some
specific problems but more was needed
and the jail command was introduced by
freebsd in 2000
jail is more sophisticated than true
root in that its additional features
help further isolate file systems users
and networks and has the ability to
assign an ip address to each jail
in 2004 solaris zones brought us ahead
even further giving an application full
user process and file system space and
access to system hardware solaris zones
also introduced that idea of being able
to snapshot a file system or made it
popular anyway obviously this is
something super important uh when you’re
talking about vms um snapshotting a vm
and also of you know creating an image
in 2006 google jumped in with their
process containers
those were later named c groups which
might be more of a familiar term for you
that centered around isolating and
limiting the resource usage of a single
process
and moving right along in 2008 c groups
were merged into the linux kernel and
along with linux namespaces that led to
ibm’s development of linux containers
so um just right away i just just
noticed containers are not a new thing
they’ve been around a while
all right 2013 was a big year that’s
when docker came on the scene they
brought their ability to package
containers and move them from one
environment to the to another and that
same year google open sourced their let
me container that for you project which
provided applications the ability to
create and manage their own sub
containers and from here we see the use
of containers and docker specifically
explode
in 2014 docker chose to swap out their
use of the lxe toolset for launching
containers with lib container in order
to utilize a native golang solution
that was something interesting when i
first started using containers realizing
that docker was written in go
and i’m almost done with this history
lesson
but
i’m going to skip over some details but
there is an important
date that i want to
get to that happened in june 2015
and that was the open container project
initiative
uh this event is really important to
know about because it’ll give you
insight into some of the activity and
innovation or motivations behind shifts
in the market
and
this organization was established under
the linux foundation it includes members
from many major stakeholders including
docker with the goal of creating open
standards for container runtimes and
image specs
all right well all of this is happening
in the container world there’s a couple
of other dates i’m just going to throw
out there because i’m a java person and
i’m selfish so uh java 7 was released in
2011
work was started on java 8 which was
released in march of 2014
keep that in mind because when you start
containerizing java applications this
little bit of history is important to
know um
it’ll come up again later
just note that
containers and the the whole environment
and docker was in the middle of
making stuff work for everyone and java
was still continuing on its own way as
well
all right that’s it for our history
lesson thanks for paying attention i
know that was a lot on one slide but um
i think it’s good to have that
background just to understand how we got
here
so what exactly is a container i always
find it interesting when i discuss these
with people um
you know getting the getting the terms
correct
about you know containers versus images
um i’m sure all of you in this audience
know the difference but it’s sometimes
frustrating to me to hear them being you
know used in place of each other and
image is actually
you know the blueprint that a container
uses to
to run on your system um for those that
don’t know that that’s that’s the main
difference
um so i want to just show you what a
real-life container looks like when it’s
actually run on your system and i’ll
show you how it’s actually stored what’s
happening under the covers and you’ll
discover that it’s really not all that
magical
all right so
i um my first experience with containers
was on a developer with a tight deadline
of course i think that’s pretty typical
and um that’s probably a good
description of most projects
the best course of action for me was
just to jump in and start getting
something up and running on my local
machine and i learned best by doing and
the docker documentation is actually
really good so i mean that really helped
me a lot if you find yourself in a
similar position definitely go to their
get started docs i’ve linked those here
and going through that guide will get
you comfortable with the docker commands
that you need and all of that but the
first thing to note is a docker
image is really just a tarball of a
complete file system and when an image
is unpacked it’s just thrown into its
own directory which becomes its root
file system
the second is that processes that are
involved in running the containers are
just regular linux processes on top of
that they’re just a few linux features
that are used together in a way to
achieve that isolation that we want from
containers
and here um
you can see i i had some running on my
system
namespaces are important ingredient this
is what’s used to provide the virtual
separation between containers and how
the processes inside a container
don’t interfere with the host or
processes inside another container
so these are just some of the namespaces
that were set up for a postgresql
container that i had running on my box
the c groups functionality is essential
to constraining how much a container can
use things like cpu memory network
bandwidth etc and i can set these
constraints by including options on the
docker run command when launching an
image on my local machine as a developer
and here you can see i’ve constrained
the memory usage limit on one of my
containers now a lot of orchestration
will handle this for you there are
settings you know in your yaml files
that you can set to set these
constraints
all right i’m going to quickly gloss
over so just some file system details
that are interesting uh where containers
and images are actually stored on your
file system uh first off after you’ve
installed docker running the command
docker info will spit out a bunch of
information about your installation
including the docker root directory and
this is where most everything you’re
going to care about regarding your
docker images and containers will be
stored
and if you’re on a mac note that your
containers are actually running in a
tiny vm so you’re going to need to use a
screen or something like that to get in
there and get the docker root directory
to look around
if you’re not familiar with the screen
command definitely google that get
familiar with it first because you’re
it’s pretty easy to mess up your text
display if you don’t enter an exit
screen in the right way
all right this slide shows how you get
information about the image the images
that are stored on your system i listed
my available images using the docker
images command and i have several
installed but only the first are a
couple displayed here
using the docker inspect command i can
inspect any image i like using its image
id which spits out a ton of information
but i want to highlight here the graph
driver section which contains the paths
to the directories where all of the
layers that belong to this image live
docker images are composed of layers
which represent instructions in the
docker file that was used to build the
image originally these layers translate
into directories and these layers can be
shared across images in order to save
space pretty clever
the lower der mergeder and upper
sections are
especially important um the lower dirt
directory contains all of the
directories or layers that were used to
build the original image
this is going to be important when we
talk about distrolus images later
these are read-only the upper dir
directory contains all of the content
that has been modified while the
container is running if modifications
are needed for a read-only layer and
loader then that layer is copied into
the upper door where it can be written
to
this is called a copy on right operation
it’s important to remember
the data in the upper dir is ephemeral
data it will only live as long as the
container lives in fact if you have data
that you intend to keep you should
utilize the volume features and mount a
location that will stick around after
the container dies
this is how most containers running a
database are run for example or that
rely on a database
lastly the mergeder is kind of like a
virtual directory that combines
everything from lower dir and upper dar
and the way the union file system works
is that any edited layers that were
copied into upper jar will overlay
layers in the lower so if you were to
exact into a container for example you
would see
the results of that merge
all right this is another slide showing
a couple of containers currently running
on my system one of them is my local
jfrog container registry installation
that includes a container for
artifactory and a container for a
postgres database and the other simple
is just a test container that i was
playing around with
note that the container ids of the
running containers match up with the
container subdirectory names and
something to remember here is if you
stop a container that corresponding
directory doesn’t automatically go away
until the containers actually removed
with the docker remove command
so if you have stopped containers lying
around that never get cleaned up you
might see your available space on the
system start to dwindle
there is a docker system prune command
you can run every now and then to help
clean things up or you can launch a
container with the flag to indicate that
it should be removed when it’s finished
running now obviously this is for
development maybe perhaps test
environments your orchestration like
kubernetes can handle this for you as
well
all right
uh container gotchas okay this is my
favorite part of this presentation
there were several things that i ran
into when i first started using
containers they are very easy to set up
and get running but there are a lot of
ways to put your images together um that
are going to lead to some consequences
that you’re going to end up cleaning up
later i guarantee
so
the tool sets around building and
running images and containers have made
things so easy it’s also easy to shoot
yourself in the foot
and it’s deceptively simple
i’m just going to go over a few of the
common gouches including some jvm
specific gotchas that i ran into almost
immediately
another thing i want to note here is i
do say a lot of docker and that that’s
my focus right now but this also applies
to oci images as well
oci is you know the
the standards
that docker follows
but just note that
when i say images i don’t only mean
docker images
or even the tool set there are a lot of
other tools now to build and deploy your
images as well
all right so the first is uh running a
containerized application as the root
user
um i think this is pretty common well
known but i’ll be honest when i was
initially getting stuff up and running i
was so excited about
it even working that it was a while
before i took it seriously and now that
you know that
processes that are running inside a
container are just like any other
process on the system
albeit just a few constraints it’s scary
now to run as root inside a container
doing that opens up the possibility of a
process escaping those intended confines
of the container and gaining access to
host resources
reduce your attack surface of your
container by following the principle of
least privilege and although containers
are designed not to affect other running
containers if someone were to gain
access to your container and immediately
have root privileges they can wreak
havoc not only on your application but
on the host and everything else running
on the host
so how do we mitigate this problem
the best thing to do is create a user
you can use the user command inside of a
docker file when the container is built
in order to run processes as that you
as that user
and there’s a way to specify a user when
the docker run command is used but that
obviously leaves open the possibility of
forgetting to do that so it’s nice if
your production image is just set up by
default not to run as root
also
pay attention to official images that
you pull from docker hub
whether or not they run as root or if
they leave that up to you
to fix
all right even though docker provides
you with the ability to set resource
limits on your container it doesn’t
automatically do that for you
in fact the default settings are free
for all
so make sure you understand the resource
needs of your application
too little your container will die from
starvation too much in your container
could smother others on the system so
just running containers does not give
you a license to be lazy and not
understand the resource needs of your
application
the resource usage of your containers is
something you’re going to want to
monitor over time as well and adjust as
needed it’s a good way to determine if
something’s going wrong
or if load on your system has changed
for some other reason you could also use
that monitoring information to determine
if your application needs to be split up
further with certain actions scaled
differently um the most obvious is front
end and and back-end activities
all right never updating so this is a
pretty big security issue and now we’re
getting into the guts of this
it’s easy to get complacent and not pay
attention to what is actually getting
pulled in when you build images so not
only do you need to be aware of outdated
versions of
libraries and packages that you specify
in your own docker file but you need to
pay attention to that base image and
where it’s coming from
not updating those packages and
libraries inside your container can lead
to some pretty embarrassing results
especially when there are tools
available now to alert you when security
issues have been discovered i’ll
actually show you what that looks like
in a moment
and even ensuring that you’re running
containers with a non-privileged user
has risk when there are known
vulnerabilities that exist within your
container or even on the kernel of the
host for that matter
from time to time these exploits are
found that enable attackers to
potentially escape a container
so keep up with those security updates i
know
i’ve been on teams where that has not
been a priority priority because of the
primarily because of the fear of
breaking a product or a service that’s
already running
but that’s a symptom of a different
problem that really does need to be
solved it’s much worse to leak private
data or potentially be the start of a
domino effect that can bring an entire
system down
so definitely get those
updates
all right this is very java specific
again i
am a java programmer and i’m selfish
so
um
so i mentioned earlier about the dates
when java 7 and java 8 came out
obviously we’re all the way to java 17
now and 11 is out
but there are still for various reasons
a lot of
enterprise applications that are still
running java 8.
so considering that timeline of docker
and java releases
java 7 and early versions of java 8 are
not fully container aware this means
that your java application won’t
necessarily obey memory and cpu
constraints that you put on your
container and you may end up with some
surprise out of memory killer activity
the reason for that is the mechanisms
that jbm uses in these older versions to
retrieve the resources that are
available come from the actual host
machine and not from the c group limits
that you would expect
so there obviously this has been fixed
now there were some improvements around
container awareness that were introduced
in java 8 update 131
and there were further improvements in
later versions but to really get all of
the benefits of container awareness you
really should get to java 11
and
that’s you know the latest
release that is container aware
all right image bloat so it was a long
time before i realized i needed to start
using a docker
ignore file
this file is much like git ignore
when you build your images you are
sending a context the entire context to
the doctor daemon to and it uses that
context to build your image and
sometimes if you’re not careful and you
do a
you know wild card copy from that
context into the image you’re going to
get a lot of stuff in your image that
you may not want there you could have
secrets in there you could have
infrastructure information in there that
you do not want included in the
container you could have um
you know unit test stuff that just just
ends up
leaving you with a large
large container that just has too much
stuff in it we already discussed
you know shrinking your uh attack space
so that there’s just less opportunity
for hackers to get in and mess stuff up
for you
um so yeah things when i see a docker
file where the entire source tree is
being copied over in one line
that’s a problem also when you do that
that’s one layer
each command is a layer
that’s what it results in so any code
change at all is going to result in
having to regenerate that layer
that’s just a waste of time really so
it’s good to organize your docker files
to have the least
the least often updated stuff closer to
the top of the file
um another thing that is interesting you
could include your git directory i i
didn’t even think about this and i’m
pretty sure there are containers running
out there that include git directory
information um the docker ignore file
looks much like the git ignore file and
the format is pretty much the same you
shouldn’t have to get too fancy about
filtering or you know whatever um if you
do you might want to think about
reorganizing your code base but it’s
pretty simple to use just make sure
you’re actually using it
all right one other thing
if you’re pulling for example your java
source or any other source into the
docker file you might also be including
the build tool like maven or npm
as well
um learn to use docker’s multi-stage
docker builds so that you don’t end up
with all of that extra stuff in your
production image
that problem is not limited to java
projects that’s anything any of those
package managers or you know build tools
you don’t need those in your production
image
all right i’m glad you guys are loving
these bananas i really enjoyed putting
this together
all right so now we have finally
come down to distroless
containers images containers
what are they um basically the idea is
to
not have all of the extra fluffy stuff
that you don’t need the idea is to only
include what you absolutely need for
your application
this reduces your attack vector
so that you have less opportunity for
problems or vulnerabilities to arise
you don’t need the cherry on top
um so uh one thing to recognize too i’m
going to mention again that docker hub
has
official images
these are a curated set of docker repos
of docker repositories that are based on
docker hub
just because they have that
you know official label doesn’t mean
that they are the best choice for you to
use especially if you’re concerned for
size or for limiting your risk
so pulling in those large parent or base
images that have a bunch of stuff you
don’t need just increases your risk it
increases your attack surface area
and also
just in a general
you know just every day working shipping
those large images around and storing
them can become pretty clumsy slow and
expensive
uh one of the biggest
things we already discussed not
including package managers um
that was in the context of of your code
but what about package managers like
apt-get you know those kinds of things
that usually come along with an os you
don’t need those either so that’s one
thing that uh distro list containers try
to try to give you
those the images that those containers
are based on
don’t contain a shell even
certainly not package managers but also
not a shell
now that’s an interesting um
fact because i was in a situation where
and this is horrible um don’t judge me
but i was in a situation where
you know there was some some issue with
production um containers that were
running and one of the
ideas was to exec into those containers
and figure out what was going on
you shouldn’t be doing that
you don’t need a shell in your
production container if you find that
you do need a shell there there are
other things that you need to put in
place to avoid that kind of
troubleshooting in production um
that is meant for you know debug
purposes things like that um you
certainly shouldn’t be doing that in
production
so don’t feel
sorry to see the shell go when we talk
about distro-less containers
all right so how do you actually build
these things um i have a link here
there are google
distro list containers
uh let me i’m just gonna post that link
in the chat so that you can pull it up
and take a look at it
google has uh
several
containers that uh have been hardened
for production and um
they do not contain shells they just
contain the bare minimum of what you
need for certain development
environments
or certain you know code environments
here’s an example of one you know being
built for java this is an example of a
multi-stage build as well so that first
section beginning with from is actually
pulling a
um pulling an image from docker hub the
slim jdk 11 image
and
the important part is um
you know that as build environment part
that indicates that we’re going to
reference everything that happened in
this section later
we go ahead and build a simple
application create a jar file out of it
and then the second part actually pulls
in one of the distroless containers that
are available from google
and we just copy the results of our
build into that
and then we
specify the jar file that we want as our
command um the
the entry
point is for this particular
distrolus images is java.jar so we just
need to provide the command of the
of the jar file that we want to run so
pretty cool pretty simple
definitely use multi-base images if
you’re not
if or multi-stage builds if you’re not
doing that already
pretty important
all right now we are not ones to just
believe everything we’re told so i’m
going to pull up
my terminal here
in a moment
and i want to show you
the difference between
a couple of the jdk images that i’ve
pulled
from docker hub compared to that
distrolus image
all right you should be able to see my
terminal now
just make sure okay um so i’ve i’ve
cleaned up my system so you don’t have a
lot to look through
but i’m just going to pull up the images
that i have right now that are existing
on my system
and ignore the ones right now that start
with mm11 i’ll explain what those are in
a second but what we care about are
the images that came directly from
docker hub so we’ve got this open jdk
tagged 11 jdk slim and look at the size
of this
429 megs
okay
now let’s look at the one that’s not
slim
659 megs
okay
and then check out the distrolus from
google
206.
so you can see there really is a
difference in the size
now
that’s that’s good but i i want to know
if that really benefits me at all i mean
obviously not having a huge image that i
have to ship around that’s great but
does this benefit me in any other way
and yeah i noticed that too um this
created date 51 years ago i thought that
was pretty interesting too that’s
something i want to look at
um
later but uh yeah
so um
anyway i want to pull up um images that
i have pushed into my personal
jfrog platform so that i can show you
the number of vulnerabilities that are
in each of these images
[Music]
all right so um i’ve got
the packages here
i’ve pushed my docker packages up here
there’s two for open jdk i’ve got the
slim version and the 11 the the regular
version so let’s start with that one and
just see how many vulnerabilities we
have i’m not going to go deep into what
they are i just want to see the number
just for you know a good a good metric
okay so here we have 67 so remember
67
all right i’m gonna go back to the slim
version and see how many are in there
okay i have 36.
that’s pretty good all right
now let’s go into the
distroless one
and see what this one looks like
i see 51
here so did we really improve anything
does that um built 51 years ago
mean anything
these are things you know like i said
don’t just believe everything you hear
you still need to pay attention to what
packages are actually being included in
your base images
one thing that can trip people up is
especially when you’re on an internal
project you may not even have the
you know because of the hierarchy of
base images the base images that you’re
using might have its own base image
you’ve got to go all the way up to the
that layer 1
and check out and see what everything
is being used in your your build
all right so uh definitely check out
those distro-less images compare them to
other images that you are using today
and see if they are beneficial to you
lastly
i did write this
d-zone article i’m going to share that
link with you as well
that just gives you some more
information about choosing a doctor
registry
um or any registry for that matter
whether it supports oci or docker or
both um it’s it’s meant to just give you
an education on where your images are
being stored now that you have them
built where you put them to make sure
that they are protected and
that they don’t you know you don’t get
something unexpected
and then also if you’re interested in of
course just playing around with the
vulnerability checking and pushing those
images up to
the jfrog platform there’s a link here
to use
to get one set up it’s really fast
really easy to use
i’ll put that link here
as well if you want to do your own
experimentation
all right and that is all i have for you
um don’t forget to go fill out the
survey to get your chance to win a shirt
yeah thank you melissa we have a few
questions here
yes uh
i’ll start with the biggest concern
okay it’s about getting rid of the shell
so
um
for developers it’s a big barrier to for
adoption
they feel like it will be impossible to
investigate word issues and things like
that
what would you answer a developer about
that
well i mean depending on what you’re
using
like i said you should never be going
into a production
image
um to
to do troubleshooting of that sort um
you do you can use logging you can you
know send all the logging out into some
other system in order to check things
that way
and also in your development environment
though you have the opportunity to con
you don’t have to use in fact you
probably wouldn’t want to use a
distro-less image
with your development because you do
need to be able to get in there and
check things out
um so this is i i would reserve distro
less images for production
you want to use you want that same image
that ends up in production to go through
your testing though
uh thoroughly so um
because you want to be able to just
promote the same image you don’t want to
have all of your testing done on the
development image
and then turn and use a
distro-less image because there could be
things missing there that you don’t
realize
so you definitely want all of your
testing and stuff to be done on the
distrolus image as well but yeah for
development purposes i wouldn’t use a
distro-less image
i’ll check out the project pixie and
cncf as well which is helping you to
instrument uh quickly get all of this
observability and
tracing and debugging
um quickly
about the java 7 version which is not
container aware
what do you mean about that
so what i mean by not being container
aware is when you try to set limits like
on cpu usage it will actually use the
resources that are on the host
not in the c group for that container
so not a huge issue if you only have one
container running on the system but if
you have other things running on the
system you might find yourself
starving the container or starving other
processes that are running on the host
melissa can i share my screen at the
same time while we answering the
question we want to keep it
survey
thank you
well there is
another question from jamil noor
it’s getting very technical so maybe
jamil uh if you’re on slack uh maybe we
could answer you offline
about this one
uh there was a quick question melissa
about
your usage of pre-doctor containers
uh how and where was that
my usage of of
what
the question is from
thomas uh i might have missed it but oh
please
containers that you were using
pre-docker containers
like linux
containers
um
those types so yeah um dockers made
containers
famous but containers existed long
before docker
for sure
um docker has a bunch it’s a whole suite
of uh capabilities right it has the
ability to
build to
um
deliver you’d uh publish to doctor hub
and stuff like that share with other
people it has the ability to launch and
run containers all of that was put
together in a single tool that
developers can use that made it really
easy for them
but not to say that containers didn’t
exist before docker did
and there was some tooling around that
before the occurred to put strap and
elixir set up name spaces and everything
yeah okay well um
maybe we have another uh
more technical question so maybe i’ll
reach out to you and we’ll share that
with the rest of the folks uh later on
uh we’re
way over time
thank you melissa was really nice to
meeting you virtually
and we’re hoping that you’ll have a
chance to come to canada uh once we’re
gonna be back in person uh we’re
probably gonna do a survey soon just to
asking if our users are ready for in
person and we’re going to be probably
start looking for events and venues to
starting to organize because we also
missed it a lot
all right um
so just before we’re going to close out
we have to uh not have to but like you
you can do a post-event survey um and uh
have a chance to win a prize uh thanks
to jfrog who is uh donating today a
raspberry pi 4 desktop kit
and also
titanium usb key uh that helps to secure
supply security
so
i have right now 31 respondents i will
probably make a last call uh just
32
just few more seconds
uh
okay so people are filling up still
that’s good
um
julia you want to add anything uh before
we gonna go for the draw
um no i think i’m good i just wanted to
give a shout out to melissa again thank
you so much for um as i mentioned in the
chat making this really fun um with
bananas and monkeys uh you can’t go
wrong so thank you for that and um for
those of you who are interested in uh
chatting about burnout again uh it’s
thursday um in the cncf
slack workspace in the burnout channel
join us there and i think um we’re ready
to announce the prizes i’m actually just
gonna
stop the recording now
perfect bye
so i have 36 37 okay