Distroless Docker Images @ CNCF Eastern Canada Meetup

October 27, 2021

< 1 min read

Distroless Docker Images

Check out this talk by Melissa as she speaks at the CNCF Eastern Canada  Meetup!  Securing a Docker image does not just stop at securing the software you put into it, you also have to deal with the base image security issues. This is where a Distroless Docker image helps you, by reducing the potential attack vectors of your Docker image, you will end up with a more secure and scanner-friendly product. In this talk, Melissa will show how you can build these “Distroless” Docker images and the advantages of doing so.

View Slides Here

Speakers

Melissa McKay

    Melissa McKay

    Melissa is a long-time developer/software engineer turned international speaker and is currently a Developer Advocate on the JFrog Developer relations team, sharing in the mission to improve the developer experience with DevOps methodologies. Her background and experience as a software engineer span a slew of languages, technologies, and tools used in the development and operation of enterprise products and services. She is a mom, Java Champion, Docker Captain, co-author of the upcoming book DevOps Tools for Java Developers, a huge fan of UNconferences, and is always on the lookout for ways to grow and learn. She has spoken at Kubecon, DockerCon, CodeOne, JFokus, Java Dev Day Mexico, the Great International Developer Summit, and is part of the JCrete and JAlba UNconference teams. Given her passion for teaching, sharing, and inspiring fellow practitioners, you are likely to cross paths with her in the conference circuit — both online and off!

    Video Transcript

    going up in a few seconds we’ll start
    hello youtube
    hello okay
    so let’s start it for today’s meetup
    or we wait one more minute
    i think we can get going
    yeah
    yeah dude it’s still climbing like
    i guess attendees are late today
    i think we’re a good number now so
    that’s that’s uh you know that’s a good
    point to start 50.
    okay
    bus 50 let’s start that so hello hello
    everybody and thanks for joining us
    today
    uh
    i guess everything is done here let’s go
    to the agenda
    so today’s agenda we have a little bit
    of housekeeping
    we have the kipcon wrap up so we’re
    going to present you a few of the talks
    that our organizers
    like the most
    and then we have a great talk of about
    this trolleys so how to secure docker
    images using digital lines and the stock
    is from melissa mckee from jfrog
    and we have huge prizes today and thanks
    to jfrog we’ll get to that later on
    so our host today myself uh julia and
    archie as always anthony
    uh will also talk about his speaks of uh
    um the cncf and kubernetes organizers
    it’s a big family and it’s growing
    almost every time
    so thanks everybody for being part of it
    so we have now two slides for the
    organizers so thank you all
    um
    we have sponsors and and so thanks again
    to elastic and jfrog for helping us
    organizing all that and providing swag
    and and whatever we need
    that’s a great support
    you can become a sponsor we have a qr
    code here and you can find that in the
    slides that we’re going to share later
    on
    uh so if your company wants to be part
    of it you’re welcome and you will have
    like time slots for speaking if you have
    like great things to share with us so
    let’s do that
    uh so
    let’s go
    we already talked about that so cncf
    update and the melisa’s talk i want to
    mention that we also have a survey at
    the end
    and this is helping us to better
    organize and and present things for you
    so
    please
    fill the survey at the end
    so as always
    we there is different ways to reach to
    us uh slack linkedin
    and twitter uh please come to the select
    channel if you have some questions
    there’s great people there bringing a
    lot of information and news about
    currencies and cncf projects so
    please join
    we have cfps
    so if you want to participate if you
    have a talk to submit to us
    the links are here
    please
    come to the slack channel or use these
    links to submit your your talks
    cncf announcement so
    there was kubcon last not actually last
    month this month
    two weeks ago uh we have some stats
    about this con it was very special it
    was the first in the last two years
    uh it was a great event so
    a lot of attendees actually
    virtual mostly virtual sadly
    great stats
    canada represents a lot in description
    because it’s the third
    overall
    country of participants
    uh
    we’re very proud of that uh so for in
    person 3500
    and something
    uh that’s great
    uh that’s great for a first one in in
    you know coffee time i can tell you that
    everything was secure a lot of checks
    everybody’s was wearing masks inside
    uh
    it
    doesn’t feel like covet at all except
    for like having face masks
    and i’m sure
    like archie or or or julia will be able
    to comment on that but i had a great
    impression
    uh
    and almost uh 20 000 attendees online
    uh
    what i can see it’s a lot of men and and
    still we have very few women uh
    to this kind of technical events
    so uh
    join us join us every time and and once
    again here even virtual canada is the
    third country of attendees so that’s
    great we’re a great crowd
    a few pictures
    so you know capcom it’s keynotes
    sessions
    and parties actually and and meeting
    people so that’s also a great part of
    the con and and we have a big picture in
    the middle all canadian folks
    there was a lot more than that
    but you’re hiding
    so
    the pigs
    so
    i only have one
    because at the beginning we said like
    everybody is going to pick one so i
    picked one
    it’s about salesforce explaining how
    they build
    a paas so platform as a service using
    crds
    so it’s the new
    oam model uh where you define your
    applications in operators and you have
    everything inside the cluster taking
    care of the ci cd
    and bringing up everything that is
    needed so the the users can
    just describe the application and
    everything is taken care by
    by automation in the cluster that was a
    very great
    talk very inspiring
    so that’s my take
    uh
    [Music]
    anthony
    yes sure hello everybody uh thanks
    sebastian
    so yeah i was over here uh attendee i
    was part of 20 000 uh joining online and
    yeah uh well first of all it was uh it
    was a pretty decent experience actually
    uh whether it was uh you know so there
    were people presenting live and they
    were being of course are live streamed
    oh so there were there were also a lot
    of recording talks i guess it was 50 50
    um as um
    to my opinion but i don’t have the
    official uh stats anyways it was a very
    pleasant experience following from uh
    from my living room
    so uh of course i still um well i have
    picked uh three or three themes that i
    really liked and that were super well
    represented at kubecon first of all the
    developer experience
    the developer experience was really put
    into in focus this year
    many people many speakers were talking
    about telepresence you know this dark
    magic from ambassador labs but allows
    you to actually have local processes on
    your machine being exposed uh to your
    remote cluster so a very nice tool that
    is getting actually a 2.0 ladies i think
    it was this year so it’s a it’s a really
    nice tool to check out i highly
    recommend it to you
    new to me was bridge to kubernetes
    something very similar from microsoft
    that is well integrated with vs code and
    same thing it allows also to send some
    uh
    some remote uh traffic to your local
    process
    a nice tool that i didn’t know about as
    well was dapper adapter from microsoft
    as well so if you’re familiar with istio
    architecture it’s kind of the same thing
    but focus for developers this time
    meaning that you will have uh you will
    be able with dapper to inject some
    sidecar containers that will do things
    such as resolving network uh mounting
    volumes uh and so on and so forth so all
    those you know those aspects of
    configuration that can be how to do with
    kubernetes then dapper is going to
    actually um remove all the configuration
    from your application and put it in the
    centralized dapper place
    so that definitely seems to be a good
    tool to check out
    porter also was new it’s kind of a
    packaging for kubernetes i’m not sure
    where that’s
    going to go but the idea was you know to
    be able to safely package uh some
    deployments uh to kubernetes using this
    tool
    so that’s it for my developer experience
    but as you can see uh below on the slide
    apparently archie also picked up some
    interesting tools so more to come
    then
    on the observability side open telemetry
    everywhere this is the you know this is
    the merge of open sensors open matrix
    and open tracing this is the way to go
    open telemetry is ga for traces there’s
    i mean this is your go-to tool if you
    want to do a summer choices
    instrumentation for sure they even have
    metric support now although it’s only in
    beta and also they envision bringing
    logs support as well
    interestingly enough
    the open source jaeger was
    seems to be evolving into a full apm
    tool so there were only about traces
    before but now they really want to
    encompass more observability pillars
    finally github’s
    guitars everywhere thanks to flux and
    most certainly argo cd argo cd is now
    so mature that there are a lot of
    integration that appeared with it such
    as argo cd application set where you can
    actually bundle several applications
    into argo cd easily i’ve also seen some
    interesting plugins to integrate with
    vault and overseas hat managers so yeah
    go cd is really um
    is really big now and uh superficial
    that’s it for my text i will also place
    some links
    in this chat as well as on slack with
    many uh with uh some
    blog posts about
    what happened at kubecon
    thanks anthony
    i actually want to add a few few more
    words about argo cd that we had a last
    meet up around argo cd and at kubecon i
    saw two new companies that are packaging
    argo cd and they’re hiring people so if
    you’re interested to work in argo cd
    space there are companies actually
    hiring now and you can join them in a
    different position starting from
    software developer to developer
    developers like melissa for example so
    check it out and if you’re looking for
    that uh let me know i’ll be happy to
    connect it
    but honestly for myself i was just happy
    to be in person at keep going away and
    to see all friends uh need the community
    go to the rooftop parties and you know
    have lots of swag so i had like two bags
    of swag that i brought with me from uh
    los angeles uh at cubecon i was a track
    host so i was representing different
    speakers before they present and i was
    very lucky to represent uh katrina veray
    from canada uh she’s working today at
    apple and she’s a six-year-old lead so
    she was presenting at custom around
    customized
    but most of the time i usually spend at
    the booth so i go check my old uh
    friends like a company like jfrog or see
    the new companies who are there up there
    and just trying to understand you know
    what’s where the industry is going and
    what’s happening and you know get some
    swag
    uh and obviously for me the main goal is
    to find new talks and new sponsors for
    meetups
    so um i had to agree with anthony on
    that the developer experience with
    kubernetes is becoming one of the major
    focus in this kubernetes space um
    without going to the talks i was at the
    booth
    uh i checked few interesting companies
    that i haven’t seen before that uh you
    know appeared post-covered um so one
    company is called garden and they have a
    you know project garden that lets
    developers not only have a developer
    kubernetes loop locally but also
    developers can run part of the ci
    locally so that kind of helps developers
    to have end to end loop
    to understand their
    software much better
    obviously project tilt from cncf which
    is still in sandbox but is getting a lot
    of traction if you know scaffold a
    project from google this is very very
    similar to that
    and i’ve met the devrel so they’re
    probably going to come to present at our
    meetup shortly
    another project very cool is devspace um
    it’s
    for coming from company called aloft so
    they allow you to easily deploy
    kubernetes application using like a
    docker compose style uh approach
    and then finally a very funny name in
    french project cui
    uh so let’s build the framework for
    enhancing clies
    with graphics
    i i just really like the name so i
    brought it up
    um the second trend that i’ve seen uh at
    the event and i think most of you agree
    as well is like around kubernetes
    security and ebpf
    uh so i really like project loft it
    provides
    tools to solve multi-tenancy problems
    with kubernetes which i think everyone
    is agree that is very difficult so they
    have a project called vcluster that lets
    you create virtual cluster on the same
    kubernetes cluster and kind of provides
    isolation for developers to run their uh
    workloads and then a very interesting
    project called gs policy so if you’re
    familiar with opa and caverno this is a
    very similar tool but uh you you kind of
    creating the policies in javascript so
    if your company have lots of javascript
    developers they’re going to love it
    but i think like one of the coolest
    projects that i’ve seen overall it was
    project called pixi which is sandbox
    project in cncf that is powered by edpf
    and provides you observability and debug
    debugging for developers without any
    instrumentation so it just magically
    starts and then you can see all the you
    know your services connections your
    debugging your matrixes so it’s pretty
    cool to check out you can install it try
    it out i highly recommend has a nice ui
    as well
    uh so that’s that’s i think one of the
    coolest things that i’ve seen
    and finally i think one area that had
    huge traction uh including the keynotes
    uh was security supply chain and i think
    it’s thanks to my friends in russia
    that recently hacked solarwinds uh it’s
    it’s kind of becoming a key key focus
    for us
    so
    even in a kubernetes community we want
    to that kubernetes uh
    development and secure and the one that
    kubernetes application is securely
    deployed right
    so the first initiative that you see
    there it’s called six store uh it’s
    initially started by red hat and google
    that provides you new standard for
    signing and verifying and protecting
    your software
    and it’s funny enough there’s few google
    employers that left the google and
    started a new startup called chainguard
    so check it out the hiring right now so
    if you want to work in the security
    supply chain this is very cool company
    to check out
    and
    another thing that has been talked at
    keynote uh something called s bomb
    software deal of materials uh this is
    essentially you know we’re trying to to
    make sure that even kubernetes pipeline
    has you know all the all the
    software that is dependencies that all
    secure
    so that was my kind of uh
    very short
    recap but honestly i haven’t go too many
    talks i’m still looking forward to to
    see the recordings that are available on
    the platform or they’re going to be
    available in youtube shortly uh maybe in
    a couple of weeks
    sebastian back to you
    or julia yeah back to julia yay
    um hi everybody i just wanted to take a
    moment to um
    i guess talk about my experience so my
    favorite talk was my talk
    um because it’s about a topic that is
    very important to me and near and dear
    to my heart
    it was about my burnout and depression
    and mental health um
    journey and experience a couple of years
    ago so i was really felt very fortunate
    and i was very honored to be able to do
    this talk in person
    i done it i’ve done it a couple of times
    online and it was a much more
    nerve-wracking experience to do it in
    person but it was very well attended and
    our canadian community was
    so supportive and put my little burnout
    ribbons on their tags and i was really
    really happy to have you all there and
    all of you
    who sent me messages after and who got
    in touch with me it really um
    i can’t express how much it means to me
    so thank you so much um for
    you know supporting my not just my talk
    in the topic but also
    um i guess my journey from you know
    being a non-technical contributor to
    being able to do this it was it was
    really special for me
    so um enough about enough about me um my
    favorite talk at uh kubecon was one of
    the keynotes actually because it was
    about people um and i guess also the
    developer experience but it was a
    keynote by robert duffy and he talks a
    lot about um how how they work together
    and how meaningful that is so if you do
    have a chance to check it out um i
    highly recommend it was actually in a
    bit of a series with the other co-chairs
    of the conference so constance and
    jasmine um and really kind of talking
    about the human element which obviously
    as you can see is my jam so i really
    appreciated that uh this is starting to
    become more of a topic and more of a
    focus for
    for these conferences and as archie
    mentioned there will be talks available
    it’s actually this friday as of the 29th
    so all the talks from kubecon will be
    available on the cncf youtube channel
    as of this friday so you can check those
    all out
    and if you just want to switch yeah so
    in vain of all this um a wonderful new
    friend who’s actually here attending
    thomas reached out to me and asked if we
    would create a peer support group so
    that’s what we’re doing so if anybody
    wants to talk about uh burnout or just
    how they’re doing and any struggles they
    might be having or just wants to connect
    with people in a different way
    please join the cncf slack
    workspace so the link is here
    slack.cncf.io
    and there’s a channel in there called
    burnout and we are going to be doing
    weekly huddles in slack in that space in
    that workspace
    so right now they’re at their thursdays
    at 12 p.m eastern so if you’d like to
    join us we’re going to be doing them
    weekly for the first little bit and
    we’ll see how how it goes and how people
    like it but i just wanted to
    give thomas a shout out and also support
    or promote our little uh initiative here
    so um hope to see you there i’ll pass it
    back to you sebastian thank you yes
    thank you i want to add that i was at
    the julia’s talk it was awesome it was
    crowdy
    and actually it was a lot of men
    usually
    like
    men we don’t we’re strong we don’t talk
    about that and that was very great to
    see how many they were and and i want to
    relate that to to
    uh of course uh carrie price experience
    uh uh
    like we as canadian have to to work
    around burnouts and be be honest about
    it and share
    our thoughts and so that was a great
    experience and thank you again julia for
    bringing that
    to everybody and supporting this
    next evans actually so
    in the coming months uh we have the the
    in december actually china kucon
    and then next year you have the dates uh
    valencia spain and then next year forest
    north america in detroit so very close
    to canada for once
    um
    and then archie it’s your turn
    all right
    um i just want to mention before we’re
    going to start with the presenter um we
    going to have at the end of our meetup a
    post event survey that will have a price
    draw sponsored by jfrog so we’ll have um
    kubernetes mask we have also
    titanium key that helps you to protect
    your secure supply chain from phishing
    and
    most importantly the best price is a
    raspberry pi 4 desktop kit so please
    stick with us until the end don’t drop
    and get a chance to win the amazing
    prizes from
    a nice jfrog
    company
    and don’t forget to fill the survey
    because you have to fill the survey to
    be a person yeah yeah exactly yeah it’s
    just a survey
    and then uh finally
    um
    i i want to invite melissa mckay from
    jfrog
    so i was looking for her all the time at
    kipcom but i couldn’t find her um and i
    just find the fun fact about her she
    never visited canada so we’ll try to
    make sure we’re going to fix that
    problem
    and she will be able to come us and the
    topic of today’s presentation is
    actually very close to my heart i have a
    lot of users and people asking me how to
    run those you know docker images
    securely on kubernetes cluster
    so uh and i i heard about the project
    that i think started by
    google jfrog and few others um around
    distroless docker images so when i when
    i saw that melissa’s presenting around
    this topic i was very excited and i
    invited we invited her to present at our
    meetup
    melisa
    welcome
    i think you muted but uh yeah
    yeah thanks for the introduction and for
    the invite to canada i’ll definitely be
    up there as soon as i can
    uh one thing i wanted to say about going
    to kubecon was
    um you know i didn’t i kind of think of
    myself as more of an introvert i thought
    i was just fine not being around people
    but i tell you what i mean going to that
    conference fed my soul
    in ways i didn’t expect i didn’t even
    know i was hungry that hungry for
    in-person events again so i’m looking
    forward to the next one hoping things
    improve across the country and we’ll be
    able to continue doing that
    all right
    let’s go ahead and get started
    so uh this is a talk that i’ve somewhat
    modified to include um a discussion
    about distrolus it’s a talk i’ve given a
    few times and i have had a lot of
    interest in and it basically it goes
    back to the basics all the way back to
    the developer when you are actually
    building your images and when you’re
    putting your application together so
    this falls right in line with supply
    chain security um you know these issues
    have been around a long time and i’m
    actually surprised that this hasn’t
    caused us more problems
    but obviously we’re hearing more and
    more in the media about um
    security problems with certain
    applications in certain packages that
    cause some
    you know a lot of media attention and a
    lot of problems for some companies i
    feel bad that solarwinds is now the
    example that we all give for how this
    kind of thing can happen
    so just to set some expectations up
    front there is a lot of information
    about docker in this talk but this isn’t
    really a tutorial on docker or a deep
    dive in docker commands or anything like
    that
    when i put this talk together i wanted
    to do something a little bit different i
    did a lot of background research to
    focus more on how we got here and to get
    some answers to some of the big why
    questions around containers in general
    and my hope is you’ll come away with a
    better understanding of the history
    behind containers how they actually work
    on your system and some of what is
    really going on under the covers
    obviously this is a few steps back from
    actually deploying your containers
    using kubernetes this is all the way
    back to building the initial images to
    begin with that your containers are
    based on so we need to shift that
    security left all the way back to when
    they’re first built
    and and then i’ll lead into a discussion
    about distrolus containers and what that
    means
    all right so let’s go
    um
    i have we do have an extra
    chance for a t-shirt and a liquid
    software book provided by jfrog
    these are available at the link that you
    see here on the screen
    um
    actually let me try
    posting that link so that it’s a little
    bit easier for you
    okay i posted that in the chat um my
    slides will be posted there the video
    will be there eventually but obviously
    this meetup group will probably post
    stuff as well um the first 15 people
    that sign up for this raffle will get a
    t-shirt and a book so please go there
    and you can also rate my talk there
    all right um i was already introduced i
    am melissa mckay i am a developer
    advocate with jfrog real quickly uh my
    background is actually as a developer
    i’ve been a developer for 20 years all
    the way from an intern to principal
    engineer
    but about a year and a half ago i
    decided that i really really enjoyed
    public speaking i enjoyed teaching and
    it was getting to be a little bit too
    much to do that on top of my full-time
    job as an engineer so i was looking
    around for opportunities to actually get
    paid to do this kind of thing and jfrog
    came about and i was pretty thrilled
    it’s my dream job in a way i’m working
    with some people i’ve known for years
    and years and i’m really enjoying um how
    this the developer relations program has
    grown and started with jfrog
    i am a java champion um primarily my
    background is java server-side coding
    and i’m also a doctor
    doctor captain
    you can reach me
    on twitter on linkedin i’ve got my
    contact information here
    all right so like i said we’re going to
    learn about history because sometimes we
    just need to you know remind ourselves
    how we got here to begin with
    we’ll look a little bit about
    look at containers in real life how they
    look on your system specifically
    my macbook because that’s that’s what i
    used
    and then container gotchas things that i
    experienced when i first started using
    containers and that will lead right into
    distro-less images and what those are
    all about
    all right i have several talks now where
    i have a theme and my favorite theme
    right now for docker um
    images and containers is bananas i
    absolutely refuse to use
    shipping container pictures and any more
    talks or anything i’m tired of seeing
    those so we get bananas and and i’m
    sorry somebody says they oh sebastian
    you hate bananas that’s too bad
    here they are you’re gonna get a lot of
    bananas in this talk
    all right um
    so in the beginning when all of this
    started this whole idea of using
    containers
    there were obvious limits on time and
    resources that created bottlenecks and
    inefficiency with um just the physical
    hardware even and i’m just being able to
    share a workstation was not enough there
    needed to be a way to share without
    getting in each other’s way or having
    one person inadvertently cause an entire
    system to crash for everyone
    fast forward to the future now we’re
    looking at huge
    infrastructures where we do you know
    blue green deployments and we’re really
    working hard on not destroying an
    application for all users when we make
    little changes
    and want to update our stuff
    but um back then
    way back then i’m aging myself a little
    bit
    we needed better strategies in sharing
    compute resources and that actually
    started this path of innovation that we
    see massive benefits from today
    there’s some key points that brought us
    to the state we are in
    and i’m going to begin our container
    history lesson with charute
    that was born in 1979 during the
    development of the seventh edition of
    unix and was added to the berkeley
    software distribution in
    1982 being able to change that apparent
    root directory for a process and its
    children which tarot allowed you to do
    resulted in a bit of isolation in order
    to provide an environment for
    testing and distribution for example in
    fact that’s how it was introduced was as
    a test utility which grew into something
    more
    so churu was a great idea solved some
    specific problems but more was needed
    and the jail command was introduced by
    freebsd in 2000
    jail is more sophisticated than true
    root in that its additional features
    help further isolate file systems users
    and networks and has the ability to
    assign an ip address to each jail
    in 2004 solaris zones brought us ahead
    even further giving an application full
    user process and file system space and
    access to system hardware solaris zones
    also introduced that idea of being able
    to snapshot a file system or made it
    popular anyway obviously this is
    something super important uh when you’re
    talking about vms um snapshotting a vm
    and also of you know creating an image
    in 2006 google jumped in with their
    process containers
    those were later named c groups which
    might be more of a familiar term for you
    that centered around isolating and
    limiting the resource usage of a single
    process
    and moving right along in 2008 c groups
    were merged into the linux kernel and
    along with linux namespaces that led to
    ibm’s development of linux containers
    so um just right away i just just
    noticed containers are not a new thing
    they’ve been around a while
    all right 2013 was a big year that’s
    when docker came on the scene they
    brought their ability to package
    containers and move them from one
    environment to the to another and that
    same year google open sourced their let
    me container that for you project which
    provided applications the ability to
    create and manage their own sub
    containers and from here we see the use
    of containers and docker specifically
    explode
    in 2014 docker chose to swap out their
    use of the lxe toolset for launching
    containers with lib container in order
    to utilize a native golang solution
    that was something interesting when i
    first started using containers realizing
    that docker was written in go
    and i’m almost done with this history
    lesson
    but
    i’m going to skip over some details but
    there is an important
    date that i want to
    get to that happened in june 2015
    and that was the open container project
    initiative
    uh this event is really important to
    know about because it’ll give you
    insight into some of the activity and
    innovation or motivations behind shifts
    in the market
    and
    this organization was established under
    the linux foundation it includes members
    from many major stakeholders including
    docker with the goal of creating open
    standards for container runtimes and
    image specs
    all right well all of this is happening
    in the container world there’s a couple
    of other dates i’m just going to throw
    out there because i’m a java person and
    i’m selfish so uh java 7 was released in
    2011
    work was started on java 8 which was
    released in march of 2014
    keep that in mind because when you start
    containerizing java applications this
    little bit of history is important to
    know um
    it’ll come up again later
    just note that
    containers and the the whole environment
    and docker was in the middle of
    making stuff work for everyone and java
    was still continuing on its own way as
    well
    all right that’s it for our history
    lesson thanks for paying attention i
    know that was a lot on one slide but um
    i think it’s good to have that
    background just to understand how we got
    here
    so what exactly is a container i always
    find it interesting when i discuss these
    with people um
    you know getting the getting the terms
    correct
    about you know containers versus images
    um i’m sure all of you in this audience
    know the difference but it’s sometimes
    frustrating to me to hear them being you
    know used in place of each other and
    image is actually
    you know the blueprint that a container
    uses to
    to run on your system um for those that
    don’t know that that’s that’s the main
    difference
    um so i want to just show you what a
    real-life container looks like when it’s
    actually run on your system and i’ll
    show you how it’s actually stored what’s
    happening under the covers and you’ll
    discover that it’s really not all that
    magical
    all right so
    i um my first experience with containers
    was on a developer with a tight deadline
    of course i think that’s pretty typical
    and um that’s probably a good
    description of most projects
    the best course of action for me was
    just to jump in and start getting
    something up and running on my local
    machine and i learned best by doing and
    the docker documentation is actually
    really good so i mean that really helped
    me a lot if you find yourself in a
    similar position definitely go to their
    get started docs i’ve linked those here
    and going through that guide will get
    you comfortable with the docker commands
    that you need and all of that but the
    first thing to note is a docker
    image is really just a tarball of a
    complete file system and when an image
    is unpacked it’s just thrown into its
    own directory which becomes its root
    file system
    the second is that processes that are
    involved in running the containers are
    just regular linux processes on top of
    that they’re just a few linux features
    that are used together in a way to
    achieve that isolation that we want from
    containers
    and here um
    you can see i i had some running on my
    system
    namespaces are important ingredient this
    is what’s used to provide the virtual
    separation between containers and how
    the processes inside a container
    don’t interfere with the host or
    processes inside another container
    so these are just some of the namespaces
    that were set up for a postgresql
    container that i had running on my box
    the c groups functionality is essential
    to constraining how much a container can
    use things like cpu memory network
    bandwidth etc and i can set these
    constraints by including options on the
    docker run command when launching an
    image on my local machine as a developer
    and here you can see i’ve constrained
    the memory usage limit on one of my
    containers now a lot of orchestration
    will handle this for you there are
    settings you know in your yaml files
    that you can set to set these
    constraints
    all right i’m going to quickly gloss
    over so just some file system details
    that are interesting uh where containers
    and images are actually stored on your
    file system uh first off after you’ve
    installed docker running the command
    docker info will spit out a bunch of
    information about your installation
    including the docker root directory and
    this is where most everything you’re
    going to care about regarding your
    docker images and containers will be
    stored
    and if you’re on a mac note that your
    containers are actually running in a
    tiny vm so you’re going to need to use a
    screen or something like that to get in
    there and get the docker root directory
    to look around
    if you’re not familiar with the screen
    command definitely google that get
    familiar with it first because you’re
    it’s pretty easy to mess up your text
    display if you don’t enter an exit
    screen in the right way
    all right this slide shows how you get
    information about the image the images
    that are stored on your system i listed
    my available images using the docker
    images command and i have several
    installed but only the first are a
    couple displayed here
    using the docker inspect command i can
    inspect any image i like using its image
    id which spits out a ton of information
    but i want to highlight here the graph
    driver section which contains the paths
    to the directories where all of the
    layers that belong to this image live
    docker images are composed of layers
    which represent instructions in the
    docker file that was used to build the
    image originally these layers translate
    into directories and these layers can be
    shared across images in order to save
    space pretty clever
    the lower der mergeder and upper
    sections are
    especially important um the lower dirt
    directory contains all of the
    directories or layers that were used to
    build the original image
    this is going to be important when we
    talk about distrolus images later
    these are read-only the upper dir
    directory contains all of the content
    that has been modified while the
    container is running if modifications
    are needed for a read-only layer and
    loader then that layer is copied into
    the upper door where it can be written
    to
    this is called a copy on right operation
    it’s important to remember
    the data in the upper dir is ephemeral
    data it will only live as long as the
    container lives in fact if you have data
    that you intend to keep you should
    utilize the volume features and mount a
    location that will stick around after
    the container dies
    this is how most containers running a
    database are run for example or that
    rely on a database
    lastly the mergeder is kind of like a
    virtual directory that combines
    everything from lower dir and upper dar
    and the way the union file system works
    is that any edited layers that were
    copied into upper jar will overlay
    layers in the lower so if you were to
    exact into a container for example you
    would see
    the results of that merge
    all right this is another slide showing
    a couple of containers currently running
    on my system one of them is my local
    jfrog container registry installation
    that includes a container for
    artifactory and a container for a
    postgres database and the other simple
    is just a test container that i was
    playing around with
    note that the container ids of the
    running containers match up with the
    container subdirectory names and
    something to remember here is if you
    stop a container that corresponding
    directory doesn’t automatically go away
    until the containers actually removed
    with the docker remove command
    so if you have stopped containers lying
    around that never get cleaned up you
    might see your available space on the
    system start to dwindle
    there is a docker system prune command
    you can run every now and then to help
    clean things up or you can launch a
    container with the flag to indicate that
    it should be removed when it’s finished
    running now obviously this is for
    development maybe perhaps test
    environments your orchestration like
    kubernetes can handle this for you as
    well
    all right
    uh container gotchas okay this is my
    favorite part of this presentation
    there were several things that i ran
    into when i first started using
    containers they are very easy to set up
    and get running but there are a lot of
    ways to put your images together um that
    are going to lead to some consequences
    that you’re going to end up cleaning up
    later i guarantee
    so
    the tool sets around building and
    running images and containers have made
    things so easy it’s also easy to shoot
    yourself in the foot
    and it’s deceptively simple
    i’m just going to go over a few of the
    common gouches including some jvm
    specific gotchas that i ran into almost
    immediately
    another thing i want to note here is i
    do say a lot of docker and that that’s
    my focus right now but this also applies
    to oci images as well
    oci is you know the
    the standards
    that docker follows
    but just note that
    when i say images i don’t only mean
    docker images
    or even the tool set there are a lot of
    other tools now to build and deploy your
    images as well
    all right so the first is uh running a
    containerized application as the root
    user
    um i think this is pretty common well
    known but i’ll be honest when i was
    initially getting stuff up and running i
    was so excited about
    it even working that it was a while
    before i took it seriously and now that
    you know that
    processes that are running inside a
    container are just like any other
    process on the system
    albeit just a few constraints it’s scary
    now to run as root inside a container
    doing that opens up the possibility of a
    process escaping those intended confines
    of the container and gaining access to
    host resources
    reduce your attack surface of your
    container by following the principle of
    least privilege and although containers
    are designed not to affect other running
    containers if someone were to gain
    access to your container and immediately
    have root privileges they can wreak
    havoc not only on your application but
    on the host and everything else running
    on the host
    so how do we mitigate this problem
    the best thing to do is create a user
    you can use the user command inside of a
    docker file when the container is built
    in order to run processes as that you
    as that user
    and there’s a way to specify a user when
    the docker run command is used but that
    obviously leaves open the possibility of
    forgetting to do that so it’s nice if
    your production image is just set up by
    default not to run as root
    also
    pay attention to official images that
    you pull from docker hub
    whether or not they run as root or if
    they leave that up to you
    to fix
    all right even though docker provides
    you with the ability to set resource
    limits on your container it doesn’t
    automatically do that for you
    in fact the default settings are free
    for all
    so make sure you understand the resource
    needs of your application
    too little your container will die from
    starvation too much in your container
    could smother others on the system so
    just running containers does not give
    you a license to be lazy and not
    understand the resource needs of your
    application
    the resource usage of your containers is
    something you’re going to want to
    monitor over time as well and adjust as
    needed it’s a good way to determine if
    something’s going wrong
    or if load on your system has changed
    for some other reason you could also use
    that monitoring information to determine
    if your application needs to be split up
    further with certain actions scaled
    differently um the most obvious is front
    end and and back-end activities
    all right never updating so this is a
    pretty big security issue and now we’re
    getting into the guts of this
    it’s easy to get complacent and not pay
    attention to what is actually getting
    pulled in when you build images so not
    only do you need to be aware of outdated
    versions of
    libraries and packages that you specify
    in your own docker file but you need to
    pay attention to that base image and
    where it’s coming from
    not updating those packages and
    libraries inside your container can lead
    to some pretty embarrassing results
    especially when there are tools
    available now to alert you when security
    issues have been discovered i’ll
    actually show you what that looks like
    in a moment
    and even ensuring that you’re running
    containers with a non-privileged user
    has risk when there are known
    vulnerabilities that exist within your
    container or even on the kernel of the
    host for that matter
    from time to time these exploits are
    found that enable attackers to
    potentially escape a container
    so keep up with those security updates i
    know
    i’ve been on teams where that has not
    been a priority priority because of the
    primarily because of the fear of
    breaking a product or a service that’s
    already running
    but that’s a symptom of a different
    problem that really does need to be
    solved it’s much worse to leak private
    data or potentially be the start of a
    domino effect that can bring an entire
    system down
    so definitely get those
    updates
    all right this is very java specific
    again i
    am a java programmer and i’m selfish
    so
    um
    so i mentioned earlier about the dates
    when java 7 and java 8 came out
    obviously we’re all the way to java 17
    now and 11 is out
    but there are still for various reasons
    a lot of
    enterprise applications that are still
    running java 8.
    so considering that timeline of docker
    and java releases
    java 7 and early versions of java 8 are
    not fully container aware this means
    that your java application won’t
    necessarily obey memory and cpu
    constraints that you put on your
    container and you may end up with some
    surprise out of memory killer activity
    the reason for that is the mechanisms
    that jbm uses in these older versions to
    retrieve the resources that are
    available come from the actual host
    machine and not from the c group limits
    that you would expect
    so there obviously this has been fixed
    now there were some improvements around
    container awareness that were introduced
    in java 8 update 131
    and there were further improvements in
    later versions but to really get all of
    the benefits of container awareness you
    really should get to java 11
    and
    that’s you know the latest
    release that is container aware
    all right image bloat so it was a long
    time before i realized i needed to start
    using a docker
    ignore file
    this file is much like git ignore
    when you build your images you are
    sending a context the entire context to
    the doctor daemon to and it uses that
    context to build your image and
    sometimes if you’re not careful and you
    do a
    you know wild card copy from that
    context into the image you’re going to
    get a lot of stuff in your image that
    you may not want there you could have
    secrets in there you could have
    infrastructure information in there that
    you do not want included in the
    container you could have um
    you know unit test stuff that just just
    ends up
    leaving you with a large
    large container that just has too much
    stuff in it we already discussed
    you know shrinking your uh attack space
    so that there’s just less opportunity
    for hackers to get in and mess stuff up
    for you
    um so yeah things when i see a docker
    file where the entire source tree is
    being copied over in one line
    that’s a problem also when you do that
    that’s one layer
    each command is a layer
    that’s what it results in so any code
    change at all is going to result in
    having to regenerate that layer
    that’s just a waste of time really so
    it’s good to organize your docker files
    to have the least
    the least often updated stuff closer to
    the top of the file
    um another thing that is interesting you
    could include your git directory i i
    didn’t even think about this and i’m
    pretty sure there are containers running
    out there that include git directory
    information um the docker ignore file
    looks much like the git ignore file and
    the format is pretty much the same you
    shouldn’t have to get too fancy about
    filtering or you know whatever um if you
    do you might want to think about
    reorganizing your code base but it’s
    pretty simple to use just make sure
    you’re actually using it
    all right one other thing
    if you’re pulling for example your java
    source or any other source into the
    docker file you might also be including
    the build tool like maven or npm
    as well
    um learn to use docker’s multi-stage
    docker builds so that you don’t end up
    with all of that extra stuff in your
    production image
    that problem is not limited to java
    projects that’s anything any of those
    package managers or you know build tools
    you don’t need those in your production
    image
    all right i’m glad you guys are loving
    these bananas i really enjoyed putting
    this together
    all right so now we have finally
    come down to distroless
    containers images containers
    what are they um basically the idea is
    to
    not have all of the extra fluffy stuff
    that you don’t need the idea is to only
    include what you absolutely need for
    your application
    this reduces your attack vector
    so that you have less opportunity for
    problems or vulnerabilities to arise
    you don’t need the cherry on top
    um so uh one thing to recognize too i’m
    going to mention again that docker hub
    has
    official images
    these are a curated set of docker repos
    of docker repositories that are based on
    docker hub
    just because they have that
    you know official label doesn’t mean
    that they are the best choice for you to
    use especially if you’re concerned for
    size or for limiting your risk
    so pulling in those large parent or base
    images that have a bunch of stuff you
    don’t need just increases your risk it
    increases your attack surface area
    and also
    just in a general
    you know just every day working shipping
    those large images around and storing
    them can become pretty clumsy slow and
    expensive
    uh one of the biggest
    things we already discussed not
    including package managers um
    that was in the context of of your code
    but what about package managers like
    apt-get you know those kinds of things
    that usually come along with an os you
    don’t need those either so that’s one
    thing that uh distro list containers try
    to try to give you
    those the images that those containers
    are based on
    don’t contain a shell even
    certainly not package managers but also
    not a shell
    now that’s an interesting um
    fact because i was in a situation where
    and this is horrible um don’t judge me
    but i was in a situation where
    you know there was some some issue with
    production um containers that were
    running and one of the
    ideas was to exec into those containers
    and figure out what was going on
    you shouldn’t be doing that
    you don’t need a shell in your
    production container if you find that
    you do need a shell there there are
    other things that you need to put in
    place to avoid that kind of
    troubleshooting in production um
    that is meant for you know debug
    purposes things like that um you
    certainly shouldn’t be doing that in
    production
    so don’t feel
    sorry to see the shell go when we talk
    about distro-less containers
    all right so how do you actually build
    these things um i have a link here
    there are google
    distro list containers
    uh let me i’m just gonna post that link
    in the chat so that you can pull it up
    and take a look at it
    google has uh
    several
    containers that uh have been hardened
    for production and um
    they do not contain shells they just
    contain the bare minimum of what you
    need for certain development
    environments
    or certain you know code environments
    here’s an example of one you know being
    built for java this is an example of a
    multi-stage build as well so that first
    section beginning with from is actually
    pulling a
    um pulling an image from docker hub the
    slim jdk 11 image
    and
    the important part is um
    you know that as build environment part
    that indicates that we’re going to
    reference everything that happened in
    this section later
    we go ahead and build a simple
    application create a jar file out of it
    and then the second part actually pulls
    in one of the distroless containers that
    are available from google
    and we just copy the results of our
    build into that
    and then we
    specify the jar file that we want as our
    command um the
    the entry
    point is for this particular
    distrolus images is java.jar so we just
    need to provide the command of the
    of the jar file that we want to run so
    pretty cool pretty simple
    definitely use multi-base images if
    you’re not
    if or multi-stage builds if you’re not
    doing that already
    pretty important
    all right now we are not ones to just
    believe everything we’re told so i’m
    going to pull up
    my terminal here
    in a moment
    and i want to show you
    the difference between
    a couple of the jdk images that i’ve
    pulled
    from docker hub compared to that
    distrolus image
    all right you should be able to see my
    terminal now
    just make sure okay um so i’ve i’ve
    cleaned up my system so you don’t have a
    lot to look through
    but i’m just going to pull up the images
    that i have right now that are existing
    on my system
    and ignore the ones right now that start
    with mm11 i’ll explain what those are in
    a second but what we care about are
    the images that came directly from
    docker hub so we’ve got this open jdk
    tagged 11 jdk slim and look at the size
    of this
    429 megs
    okay
    now let’s look at the one that’s not
    slim
    659 megs
    okay
    and then check out the distrolus from
    google
    206.
    so you can see there really is a
    difference in the size
    now
    that’s that’s good but i i want to know
    if that really benefits me at all i mean
    obviously not having a huge image that i
    have to ship around that’s great but
    does this benefit me in any other way
    and yeah i noticed that too um this
    created date 51 years ago i thought that
    was pretty interesting too that’s
    something i want to look at
    um
    later but uh yeah
    so um
    anyway i want to pull up um images that
    i have pushed into my personal
    jfrog platform so that i can show you
    the number of vulnerabilities that are
    in each of these images
    [Music]
    all right so um i’ve got
    the packages here
    i’ve pushed my docker packages up here
    there’s two for open jdk i’ve got the
    slim version and the 11 the the regular
    version so let’s start with that one and
    just see how many vulnerabilities we
    have i’m not going to go deep into what
    they are i just want to see the number
    just for you know a good a good metric
    okay so here we have 67 so remember
    67
    all right i’m gonna go back to the slim
    version and see how many are in there
    okay i have 36.
    that’s pretty good all right
    now let’s go into the
    distroless one
    and see what this one looks like
    i see 51
    here so did we really improve anything
    does that um built 51 years ago
    mean anything
    these are things you know like i said
    don’t just believe everything you hear
    you still need to pay attention to what
    packages are actually being included in
    your base images
    one thing that can trip people up is
    especially when you’re on an internal
    project you may not even have the
    you know because of the hierarchy of
    base images the base images that you’re
    using might have its own base image
    you’ve got to go all the way up to the
    that layer 1
    and check out and see what everything
    is being used in your your build
    all right so uh definitely check out
    those distro-less images compare them to
    other images that you are using today
    and see if they are beneficial to you
    lastly
    i did write this
    d-zone article i’m going to share that
    link with you as well
    that just gives you some more
    information about choosing a doctor
    registry
    um or any registry for that matter
    whether it supports oci or docker or
    both um it’s it’s meant to just give you
    an education on where your images are
    being stored now that you have them
    built where you put them to make sure
    that they are protected and
    that they don’t you know you don’t get
    something unexpected
    and then also if you’re interested in of
    course just playing around with the
    vulnerability checking and pushing those
    images up to
    the jfrog platform there’s a link here
    to use
    to get one set up it’s really fast
    really easy to use
    i’ll put that link here
    as well if you want to do your own
    experimentation
    all right and that is all i have for you
    um don’t forget to go fill out the
    survey to get your chance to win a shirt
    yeah thank you melissa we have a few
    questions here
    yes uh
    i’ll start with the biggest concern
    okay it’s about getting rid of the shell
    so
    um
    for developers it’s a big barrier to for
    adoption
    they feel like it will be impossible to
    investigate word issues and things like
    that
    what would you answer a developer about
    that
    well i mean depending on what you’re
    using
    like i said you should never be going
    into a production
    image
    um to
    to do troubleshooting of that sort um
    you do you can use logging you can you
    know send all the logging out into some
    other system in order to check things
    that way
    and also in your development environment
    though you have the opportunity to con
    you don’t have to use in fact you
    probably wouldn’t want to use a
    distro-less image
    with your development because you do
    need to be able to get in there and
    check things out
    um so this is i i would reserve distro
    less images for production
    you want to use you want that same image
    that ends up in production to go through
    your testing though
    uh thoroughly so um
    because you want to be able to just
    promote the same image you don’t want to
    have all of your testing done on the
    development image
    and then turn and use a
    distro-less image because there could be
    things missing there that you don’t
    realize
    so you definitely want all of your
    testing and stuff to be done on the
    distrolus image as well but yeah for
    development purposes i wouldn’t use a
    distro-less image
    i’ll check out the project pixie and
    cncf as well which is helping you to
    instrument uh quickly get all of this
    observability and
    tracing and debugging
    um quickly
    about the java 7 version which is not
    container aware
    what do you mean about that
    so what i mean by not being container
    aware is when you try to set limits like
    on cpu usage it will actually use the
    resources that are on the host
    not in the c group for that container
    so not a huge issue if you only have one
    container running on the system but if
    you have other things running on the
    system you might find yourself
    starving the container or starving other
    processes that are running on the host
    melissa can i share my screen at the
    same time while we answering the
    question we want to keep it
    survey
    thank you
    well there is
    another question from jamil noor
    it’s getting very technical so maybe
    jamil uh if you’re on slack uh maybe we
    could answer you offline
    about this one
    uh there was a quick question melissa
    about
    your usage of pre-doctor containers
    uh how and where was that
    my usage of of
    what
    the question is from
    thomas uh i might have missed it but oh
    please
    containers that you were using
    pre-docker containers
    like linux
    containers
    um
    those types so yeah um dockers made
    containers
    famous but containers existed long
    before docker
    for sure
    um docker has a bunch it’s a whole suite
    of uh capabilities right it has the
    ability to
    build to
    um
    deliver you’d uh publish to doctor hub
    and stuff like that share with other
    people it has the ability to launch and
    run containers all of that was put
    together in a single tool that
    developers can use that made it really
    easy for them
    but not to say that containers didn’t
    exist before docker did
    and there was some tooling around that
    before the occurred to put strap and
    elixir set up name spaces and everything
    yeah okay well um
    maybe we have another uh
    more technical question so maybe i’ll
    reach out to you and we’ll share that
    with the rest of the folks uh later on
    uh we’re
    way over time
    thank you melissa was really nice to
    meeting you virtually
    and we’re hoping that you’ll have a
    chance to come to canada uh once we’re
    gonna be back in person uh we’re
    probably gonna do a survey soon just to
    asking if our users are ready for in
    person and we’re going to be probably
    start looking for events and venues to
    starting to organize because we also
    missed it a lot
    all right um
    so just before we’re going to close out
    we have to uh not have to but like you
    you can do a post-event survey um and uh
    have a chance to win a prize uh thanks
    to jfrog who is uh donating today a
    raspberry pi 4 desktop kit
    and also
    titanium usb key uh that helps to secure
    supply security
    so
    i have right now 31 respondents i will
    probably make a last call uh just
    32
    just few more seconds
    uh
    okay so people are filling up still
    that’s good
    um
    julia you want to add anything uh before
    we gonna go for the draw
    um no i think i’m good i just wanted to
    give a shout out to melissa again thank
    you so much for um as i mentioned in the
    chat making this really fun um with
    bananas and monkeys uh you can’t go
    wrong so thank you for that and um for
    those of you who are interested in uh
    chatting about burnout again uh it’s
    thursday um in the cncf
    slack workspace in the burnout channel
    join us there and i think um we’re ready
    to announce the prizes i’m actually just
    gonna
    stop the recording now
    perfect bye
    so i have 36 37 okay