CNCF Switzerland Meetup – Distroless Docker Images

February 24, 2022

3 min read

Our very own Melissa McKay was able to join the very first CNCF Switzerland Meetup of 2022 with Moritz Eckert (Edgeless Systems GmbH) and Thomas Graf (Isovalent)

Session 1: “Bringing confidential computing to Kubernetes: Project MarbleRun”

Moritz will introduce the open-source project MarbleRun and discuss the challenges that arise when you deploy confidential workloads on Kubernetes. MarbleRun is the control plane for confidential computing, designed to run on Kubernetes. It is an open-source solution that extends the confidentiality, integrity, and verifiability properties of a single secure enclave to a Kubernetes cluster. MarbleRun does not replace your service mesh; it is built to run alongside your existing cloud-native tooling. He’ll demo how to build a confidential cloud-native app and run it with Kubernetes, Linkerd and MarbleRun.

Moritz Eckert leads cloud-related development at Edgeless Systems and is the primary maintainer of MarbleRun. Moritz is a seasoned engineer and has presented at top-tier conferences including Usenix Security Symposium and Conf42 in the past. Moritz is part of Shellphish, one of the highest-ranked competitive hacking groups in the world.

Session 2: “Distroless Docker Images”

Securing a Docker image does not just stop at securing the software you put into it, you also have to deal with the base image security issues. This is where a Distroless Docker image helps you, by reducing the potential attack vectors of your Docker image, you will end up with a more secure and scanner-friendly product. In this talk, Melissa will show how you can build these “Distroless” Docker images and the advantages of doing so.

Melissa McKay is a long-time developer/software engineer turned international speaker and is currently a Developer Advocate on the JFrog Developer relations team, sharing in the mission to improve the developer experience with DevOps methodologies. Her background and experience as a software engineer span a slew of languages, technologies, and tools used in the development and operation of enterprise products and services. She is a mom, Java Champion, Docker Captain, co-author of the upcoming book DevOps Tools for Java Developers, a huge fan of UNconferences, and is always on the lookout for ways to grow and learn. She has spoken at Kubecon, DockerCon, CodeOne, JFokus, Java Dev Day Mexico, the Great International Developer Summit, and is part of the JCrete and JAlba UNconference teams. Given her passion for teaching, sharing, and inspiring fellow practitioners, you are likely to cross paths with her in the conference circuit — both online and off!

Session 3: “What is Cilium & eBPF?”
Speaker: Thomas Graf, CTO and co-founder of Isovalent.

View Slides Here


Melissa McKay

    Melissa McKay

    Melissa is a long-time developer/software engineer turned international speaker and is currently a Developer Advocate on the JFrog Developer relations team, sharing in the mission to improve the developer experience with DevOps methodologies. Her background and experience as a software engineer span a slew of languages, technologies, and tools used in the development and operation of enterprise products and services. She is a mom, Java Champion, Docker Captain, co-author of the upcoming book DevOps Tools for Java Developers, a huge fan of UNconferences, and is always on the lookout for ways to grow and learn. She has spoken at Kubecon, DockerCon, CodeOne, JFokus, Java Dev Day Mexico, the Great International Developer Summit, and is part of the JCrete and JAlba UNconference teams. Given her passion for teaching, sharing, and inspiring fellow practitioners, you are likely to cross paths with her in the conference circuit — both online and off!

    Video Transcript

    there we go
    now whatever you say will finish in
    i’m kidding
    so welcome if you are joining now uh to
    this webinar more more people signing in
    this is really cool thank you so much
    for joining we are going to start in
    about two minutes
    with the first presentation
    this is switzerland so we like to always
    be very
    right at the minute
    of 4 p.m
    central europe time
    and that will be in about two seconds
    there we go
    it is 4 p.m in switzerland it is
    what 8 a.m in colorado i think
    yes there you go it’s 8am in colorado
    and welcome to the cloud native
    computing switzerland meetup uh it is
    february 24 2022 my name is adrian
    kosmacheski and i welcome you from the
    city of zurich in switzerland and this
    is yet another edition online of the
    cloud native computing switzerland
    we planned i know we plan to do this one
    in person but the situation it is what
    it is so we decided to go back to online
    and i think you don’t mind i think it
    will be in any case a fantastic event we
    have today so i’m really really thrilled
    uh to welcome our speakers and by the
    way speaking about which a little bit
    about the agenda for today uh we are
    going to start with body sacred from
    edgeless systems game behalf from
    germany uh talking about confidential
    computing with project marble run
    so this is moritz who will start today
    then we have melissa mckay from jfrog
    directly from colorado in the united
    states and this is one of the first time
    i think it’s the first time we have a
    speaker from overseas uh on this uh
    event so it’s really really cool welcome
    melissa she’s going to talk about this
    roulette docker images and that will
    happen at 1645 cet and we are going to
    end uh this event with none other than
    thomas graf cto of isovalent he’s going
    to talk about psyllium and ebpf the
    hottest new technology in town so it’s
    really really cool thank you so much for
    all of you to be here and we’re going to
    end around
    1805 which is more or less in about two
    hours so just a quick reminder this is
    being recorded as we speak
    so basically the video will be later
    available on our youtube channel on so please subscribe for
    updates we publish new content there
    very very often there’s a q a box uh
    right at the bottom of your screen so
    please use it please post your questions
    to your speakers each one of the
    speakers will basically decide whether
    to answer the questions uh live at the
    same time or
    asynchronously at the end and so on so
    basically it’s up to you uh guys to
    decide how do you want to handle this
    but of course everyone please abide by
    the conference code of conduct we never
    had an issue with this the swiss
    community is fantastic very respectful
    uh but there you go you can read it on we’ve got the code of conduct
    we are running a devops in switzerland
    survey right now until the end of march
    we are basically very curious and we’ve
    got a very very
    nice amount of answers so far about how
    companies in switzerland are adopting
    devops so please use your qr code please
    login to the surveymonkey
    survey that is on the screen right now
    why i’m saying this because there’s a
    possibility for you but you have to be
    located in switzerland sorry melissa
    sorry maurice about that but you have to
    be in switzerland because you can win uh
    one of two prizes uh worth 200 swiss
    francs each one of them being a voucher
    for a puyo cloud which is our newest
    container platform based on openshift 4
    that we just released
    a couple of months ago so please uh
    thanks for providing us with your
    information and you have the possibility
    of winning one of these two vouchers the
    other one is a voucher for digi tech
    which is one of the big
    e-commerce websites in switzerland so
    you can buy whichever device you would
    like to have for
    your birthday
    we’ve got more editions of the cnc meet
    up planned for this year we’ve got
    may august and november planned we still
    have two slots available for may so we
    already have a speaker for may
    if you are interested to speak in any of
    these events
    please contact us on
    this is a url that takes you directly to
    the website with a form the
    only thing we need to know if you would
    like to speak at our event is basically
    who you are what would you like to talk
    about and that’s it that’s basically all
    the information we need i will close
    very quickly
    please follow us on youtube because we
    have for your follow us on
    twitter on github on linkedin on
    facebook on instagram whatever you want
    please follow us it’s really really cool
    we are publishing very good information
    all the time so thank you so much enjoy
    the presentation remember to fill the
    survey if you can and if you have time
    it only takes five minutes believe me
    it’s very very short and
    go back to the agenda today we’re gonna
    start now with moritz i’m going to
    unshare my screen the screen is yours
    there we go
    thank you adrian very great introduction
    very powerful i like it
    all right
    um yeah hi uh i’m gonna give you
    an introduction into
    how we bring together confidential
    computing and kubernetes
    and especially with our project marburon
    i guess most of you will be familiar
    with the term kubernetes and all this
    cloud native stuff so i’ll take a bit of
    time and
    explain the concept of confidential
    computing so you know what i’m talking
    so the world as we see it or cloud as we
    see it um is a very dangerous place
    there are a lot of
    components people parties you have to
    there are so many potential threats and
    and malicious actors and if you transfer
    your your workloads your data to the
    cloud you
    essentially give up the protection of
    your of your on-prem physical
    and expose your data to this to this
    world and
    general idea of conventional computing
    is that you can create a safe space in
    the cloud that you can securely
    communicate with and interact with and
    thereby protect all your good data and
    workloads in the cloud
    this sounds very promising and
    of course people are very excited about
    the possibilities of the new
    types of
    applications you can essentially build
    with this and all those
    security and privacy protections
    you can you can build with this
    and there are already some examples of
    existing solutions that use confidential
    two of the probably most
    prominent ones are the the signal
    content discovery so the idea is that
    signal can tell you who of your contacts
    is already using signal without
    revealing your contacts to to signal or
    to any any other party or like
    transferring the the contacts to a cloud
    instance that where a potential
    malicious actor could could steal them
    so your your contacts stay with you and
    they can offer the service
    and on the other hand the eagercept
    which is the german e-prescription
    service that’s currently being built and
    rolled out
    there’s a talk from someone at the the
    company that builds this service at a
    conference we we did last week called
    the open confidential computing
    conference oc3
    i will have a link to that on the last
    slide so uh we’re also going to upload
    those talks to youtube if if you’re
    interested in this this kind of stuff um
    i’ll um
    point you to that it’s a very um very
    interesting um ambitious project there
    and confidential computing is not
    something that just exists in theory
    something um that the cloud providers
    are actively investing in um
    sorry to interrupt you i don’t think
    you’re sharing your slides
    i don’t share my slides i don’t think
    you’re sharing your screen
    try again
    is it back
    and now we can see your screen yes thank
    you so much
    didn’t you see the screen the whole time
    uh we only saw you yes from the
    beginning oh okay
    oh that that’s that’s not not too bad i
    guess um
    i can show you the
    the nice graphics uh i had about uh the
    exactly as as one of the secure cloud
    absolutely as one of the as one of the
    attendees is saying it was very
    interesting even without the slides so
    no worries about that yeah yeah i got
    the i got the chat set up now as well so
    i try to monitor for any funny questions
    let’s see if that works out
    yeah and some some some news articles
    about convention computing i just talked
    about and these are the two
    services signal and the
    yeah and when where i
    left off was
    we now see all those competition
    offerings popping up
    at the cloud providers
    we have a
    vastly growing
    ecosystem there of confidential
    computing technology that you can
    already use today to secure your
    workloads and cloud
    but this is just the
    very high overviews i’d like to go a
    little bit more in detail to explain
    what this actually is and and how we can
    achieve these
    isolated secure environments in the
    the most prominent example and one of
    the first concepts of confidential
    computing was from infill the the sjx
    artworks so the general idea was that
    you typically cloud stack you have this
    hardware and you have your operating
    system hypervisor all this
    and finally you deploy your application
    and without anything in place you have
    to trust all those layers and all the
    the people and the companies that have
    access to these different layers
    so now with confidential computing and
    especially with sjx enclaves the idea
    was that you
    add something to the hardware in this
    case a instruction set extension that
    would allow you to create isolated
    in on your top layer
    and those would be
    those would contain your data and your
    your data and your code
    would be
    measured when the process is created and
    would be verified that the
    that you have integrity there and then
    would be executed and the processor and
    the hardware would make
    make sure that no other layer here could
    access this code and data
    so a couple of features of
    that we see in enclaves in general and
    that were that came come with sgx is
    first of all you have isolation so as i
    said no other layer can access
    codedata that is executed inside such a
    secure enclave
    and you also have runtime memory
    encryption so at runtime even if you
    would physically have access to the main
    memory of the process you would not be
    able to to steal any code and data
    and to make this useful one of the most
    important aspects is the remote
    attestation so from inside this enclave
    you could prove to the outside world
    that you are
    running inside such a protected
    environment you can also do this from
    the outside so you can also do
    uh attestation um
    from from from that you can also from
    the outside a test at something runs
    inside an output
    and from there on then you can build
    your your applications and systems and
    you can pry to your users the other
    components that you are secure and data
    can be shared with you secure securely
    and finally they have a mean of sealing
    state so you could start and stop the
    process and store data for example to
    disk and later on unseal the stator
    only if you are running the same enclave
    with the same identity again
    so you don’t have to worry that
    your persistent data and the state of
    the application might be might be leaked
    later on
    process-based enclaves or these sgx
    enclaves have
    some some limitations they have a
    system interface
    it says no syscalls but you could
    essentially it has a limited set of
    systems you can do uh that are specific
    to to to sgx but in general they don’t
    have access to the to the system’s cisco
    interface every time you do a cisco you
    would leave the enclave process the
    syscall in the operating system and then
    jump back into the enclave
    and this has of course a lot of um
    downsides you can’t just lift and shift
    your your code and data you have to
    specifically um repackage rebuild your
    code run in those process based on
    and one development we see very recently
    is that
    to allow for real lift and shift and to
    not have to repackage rebuild things
    uh a new combination computing
    technology called the or
    yeah um
    to giving them the confidential virtual
    machines short cvms
    that not only run
    single processes inside secure onbase
    but running a whole virtual machine
    including the operating system inside
    such an isolated environment so you have
    the same idea the hardware provides
    let’s say the hypervisor which is not
    trusted to create such a secure virtual
    machine and load your image securely
    measure it and then you can execute
    whatever you want there and this gives
    you the ability to essentially do real
    lift and shift and you have the same
    type of features of isolation runtime
    memory encryption and remote adaptation
    so to compare those technologies uh we
    let’s say um
    we typically
    differentiate them
    on something we call the trusted
    computing base which is all the
    components you have to trust inside your
    your trusted environment
    if you use sjx for example to just
    protect individual individual functions
    so you split your application and only
    individual function calls enter a secure
    this is the smallest you can go and
    requires highly adjusted applications
    where you specifically develop towards
    interface the the second thing then that
    somewhere in the middle is where you
    package whole applications inside
    process-based entries um
    this means that all your code is running
    into the enclave so slightly larger
    trusting trusted computing base
    still some repackaging probably
    recognition and this kind of stuff
    needed and then the confidential virtual
    machines on the right side where you
    have to trust everything that runs
    inside this virtual machine the whole
    operating system that also has a larger
    attack surface from below of course
    because now you assume the hypervisor or
    anything that runs there can
    attack your kernel from below which is
    of course
    a lot of things that could go wrong
    there and then you have all those um
    components that run there that you have
    to trust
    and examples of these would be amd suv
    info tdx
    something that’s got a
    happening soon is called arm realms
    and the idea here is um
    that you have lift and shift
    functionality for for existing workloads
    something that’s somewhere in between
    those two worlds is the aws nitro
    running entire
    virtual machines essentially but
    their packaging mechanism and and their
    feature set is also slightly related
    more to the confidential containers if
    you will so it’s it’s hard to to really
    fit that into this picture here but just
    to mention it um
    somewhere somewhere in between
    and we build open source software to
    make these technologies accessible to so
    you can build um workloads you can
    deploy stuff to the cloud that utilizes
    these hardware technologies to protect
    and isolate um for your deployments and
    your data
    and we have similar concepts in our
    products uh we have three tools that are
    intended for these process-based
    solutions especially s into sgx
    and and we see
    the fit here if you want to create
    new types of
    solution new type of applications
    build on the concept of comprehensive
    computing that maybe
    implement a use case that’s only
    possible in such secure
    environments like multi-part computation
    ai training on on uh pi data so personal
    identifying from identifiable
    information uh data
    and these kind of things and then on the
    right side we have a tool that we just
    released last week that is
    using confidential vms to create entire
    kubernetes clusters
    inside such execution environments so
    you can think of
    running a kubernetes cluster in the
    public cloud
    while being completely isolated and
    don’t have any um don’t have to trust
    any part of the public cloud of the
    cloud provider
    that has no access to your cluster
    so if you want to lift and shift
    this this tool is intended um for this
    use case
    for this talk i’m going to focus more on
    the left side on
    how you can
    develop cloud native applications maybe
    uh repackage cloud native applications
    to run inside process place enclaves or
    you can orchestrate and scale them and
    then finally integrate them together in
    kubernetes to to build scalable
    confidential applications
    the first tool i’m just going to mention
    here briefly is called ego
    when we started we noticed there’s no
    real there is no go run time
    confidential computing
    alpha in class gx so there so what we
    did we
    basically slightly modified the gold
    compiler so it doesn’t do
    syscalls but instead goes into a
    capability library that handles
    the syscall layer so it can run inside
    secure enclaves and then we try to make
    the simplest tooling possible
    for for you to interact with and package
    and build your applications so it feels
    like just any other go development
    process without going into the troubles
    of understanding how to build and
    package into sjx enclaves which is
    or can be quite troublesome
    so we have three simple commands you you
    build as you would do to go compiler
    that builds a blue binary and then we
    sign it because every uh sdgx ipad has
    to be signed and this step also packages
    your whole ipad and then you can just
    run similar to how you would do go run
    and it will load your enclave and run
    your go code inside there
    and yeah um this is for us it feels very
    natural to our usual co-development
    workflow and and we use this for for
    go applications and most of native stuff
    is written in go so we can bring most
    cloud native stuff um
    those secure enclaves so we can already
    isolate and protect our individual
    workloads which is
    which is cool we have an example of
    where we for example you um
    take hashicobalt the
    the key values or of hashicorp that’s
    often used to distribute secrets for
    to run inside a secure enclave
    when we
    had that
    have protected our own original
    of course there exists other tooling for
    other languages there exists some
    library assets that can more or less run
    a variety of different languages and
    types of applications so we had
    we built the go part and then we had all
    the we could lift and shift all the most
    of the workloads in the cloud that we
    needed to
    into secure enclaves
    and then the second problem arises
    the question then was okay now we have
    individual workloads how can we bring
    them together
    for a modern cloud native architecture
    how can we do a microservice system
    and connect those individual services
    securely so we can distribute and share
    data between those services while
    everything is end-to-end security and
    confidential and how can we
    orchestrate that as we would usually do
    with kubernetes while everything inside
    kubernetes all the kubernetes components
    our nodes uh our control plane they’re
    all not trusted because they’re either
    completely controlled by the cloud
    provider or running inside cloud
    provider control infrastructure
    and therefore
    we came up with a project called marble
    so the idea is that we can
    use the existing orchestration tooling
    of kubernetes
    and we add
    a trusted party to that that runs
    a let’s say a trusted control plane that
    takes care of all those orchestrational
    tasks that you are have with
    distributed confidential applications
    a confidential deployment for us means
    of course
    individual protection of our workflow
    that means runtime encryption of data
    and then we need to
    somehow be able to verify the topology
    and functionality of our application we
    need to take make sure that
    any part of the infrastructure of the
    existing orchestrational
    layer is is is not trusted and can’t
    access our
    our protective workloads so all the
    the the cloud provider and and any
    anyone that has access to the call
    either intentional or through some some
    malicious act
    so the concrete questions we solve with
    mamaron are how we can attest
    the integrity and confidentiality of a
    one container or one part if you will to
    so if one service wants to share data
    with another how can it make sure that
    the other one is also running in a
    secure entry
    and then how can we
    verify this entire deployment that might
    scale up and down the whole time
    uh as one concise application from the
    so clients administrators can verify
    this before interacting with it
    and one very specifically technical
    thing was we wanted to provide some some
    form of mutual tls so you can
    essentially have something like tls
    offloading that is often done by um
    service mesh
    but we can’t use the existing solutions
    because they don’t terminate their tls
    connections inside the actual
    application inside the
    container they terminate inside the site
    car and then go through localhost
    and we can trust localhost because it’s
    called infrastructure so we need to
    terminate our connections inside the
    application inside the actual arm plate
    and then yeah some orchestrational tasks
    like how can we distribute secrets
    kubernetes is not trusted when it’s
    control plane and etcd everything is not
    trusted you can’t use the usual way of
    configuration and and secrets management
    there and um you also want to make this
    whole thing updatable scalable all this
    kind of stuff
    so in in modern run what it would look
    like from a workflow workflow
    you would install your usual
    orchestration of little things have
    various cluster maybe installer surface
    mesh like linker d
    and then you would install mile run it
    would stall the mobile and control plane
    which we typically you call the
    coordinator so if you hear me saying
    coordinator this is the control plane
    control plane piece it runs inside an sg
    example so it’s protected itself
    and it can be verified from the outside
    so a devops engineer can verify it
    and provide it with a
    configuration and this is and this
    configuration we call a manifest
    that contains how
    our confidential application
    uh should look like similar to how you
    write your yaml files for kubernetes you
    would describe
    how your your individual archways can be
    identified what their measurements are
    how they should be configured what
    secrets they should
    provided you can define some some
    role-based authentication things in
    there some
    secrets some mutual tls configurations
    this kind of stuff so you define it
    and then you would install your
    application and your application would
    be built with
    with one of the other
    enclave runtimes for your workload
    and they would come up boot up and then
    they have a small
    a small shim layer that connects to to
    marble run and is authenticated
    according to the manifest that you
    specified earlier and is then provided
    with secrets configurations
    tls credentials and these tls
    credentials can then be used to connect
    those services together
    so they are used in those mutual tls
    connections to the other service
    and they have a root of trust in a root
    certificate provided by our bomb run
    control plane
    that then can always be authentic uh
    used to authenticate other other
    services in your
    in your service mesh
    so now we have a fully
    application everything is running
    smoothly and then a user can go ahead
    and connect
    your front-end service for example
    and similarly the user can connect to
    the maverian
    obtain another station statement that
    contains a um a
    for example sgx at the station
    statement that can be verified and then
    also contains this route of trust this
    root certificate that is then be used to
    verify every connection
    or potentially verify every connection
    to the application so you set this this
    root certificate and then connect your
    front end
    and verify the the tls last connection
    by doing so
    so this is the general workflow of how
    margarine takes care of orchestrating so
    if you would now for example scale up
    another container pops up it connects to
    margarine it’s authenticated provided
    with the credentials to join this
    mesh together set of confidential
    services and then
    just works as usual and from the outside
    you can always verify the entire
    deployment using the mamaron coordinator
    so putting this all together
    with with ego
    and of course the the other enclave
    runtimes with maveron and something i
    haven’t mentioned at all today uh called
    atlas db is essentially a database that
    runs entirely inside secure enclaves
    and has some enhancement to enhancements
    to its um to its storage encryption
    so it’s more or less
    mariah db front and
    with a rockst modified slightly modified
    rockstd backend
    for a end-to-end secure database so we
    use this for for a if you need a storage
    solution that needs to be confidential
    computing graded
    and then we package together this with
    another run and you can build an entire
    end-to-end confidential native apps
    and in most cases no code change is
    required if you’re using ego
    or library as runtimes
    and one example of this is
    the open service mesh bookstore demo of
    service smash is a
    server smash from the i think it’s a
    cloud um
    part of the cncf uh now uh
    cncf project and they have a demo
    application called bookstore and we
    confidentialized it using those three
    if you want to
    take a look
    we have a blog post on how we did that
    so please
    please check it out
    i have a
    smaller demo that’s not that complex and
    can be quickly
    quickly shown prepared
    let me switch to the console
    i have an a cluster in
    in in error prepared that is able to run
    sjx enclaves we can verify this by
    running cube cover
    uh first of all we have uh
    three nodes
    so let me describe
    one of those notes
    you would see
    a lot of things happening here but you
    would see this node
    three resources regarding to influence
    jx so this this node is equipped with
    the latest sdx hardware and has the
    capabilities to run
    clips in there
    there’s not much running so far
    there are some some sg device driver
    specific things but so far it’s a
    vanilla uh
    aks edgar aka aks cluster and i can
    install mother on there doing mobber on
    that goes very quickly and installs the
    control plane
    and we can wait for this to finish
    running check
    and um
    the application we’re going to install
    here is a very simple demo application
    from the linker d service mesh
    provides a emoji voting platform that is
    three services
    one front-end services a front-end
    service with a web interface and then
    something that counts
    the votes
    so more or less a stateful service and
    one service that provides some
    application logic in this case it
    provides emojis
    so when modern is ready
    we should be able to configure
    it in the meantime there’s a question
    for you in the q and a box i don’t see
    i’m sorry
    no it’s okay
    while we wait maybe you can answer
    there’s a question
    can you see the q a box
    one sec
    that’s the chat open now i need to go
    back to
    but i can read it for i can read it to
    you if you want oh yeah please go ahead
    so uh jan bosniak is asking can you
    share some performance benchmarks i’d be
    curious where would be
    most overhead if any whether it’s
    enclave startup or per cisco slash ego
    or call
    well that’s that’s a good question um
    especially with process-based enclaves
    there there’s
    can be quite some overhead
    the enclave startup is not too bad it
    i can’t give you exact numbers on that
    to be honest but
    the the bigger performance impact is
    definitely if you have a lot of calls o
    calls means you’re right you do assist
    call and
    your encourage have to has to exit
    the secure environment and then
    return back
    um later on and handle this is called in
    the operating system and return back so
    for example if you do a lot of io i
    always handle by the system jankv can do
    this on the on its own uh even though
    there are
    people implementing switchless sockets
    where you don’t have to to to access the
    active contacts but nevertheless if you
    have a lot of those your performance
    impact is going to be bigger
    and this
    with the latest iteration of
    sgx with the isolate machines
    they were able to reduce this overhead
    by by quite a lot so you’re usually
    able to go below
    so go into single digit percentage
    it’s it’s harder to say this in a
    general manner it’s usually depending on
    your type of application but i would say
    you can make it happen to to to
    definitely go into the single digits
    and we had a
    we had a implemented in a an ai pipeline
    we were able to comparing across
    inference and training compared
    also cpu only um on intel comparing
    enclave to
    non-enclave and this was almost
    negligible the overheads so you’re close
    to two to no overhead in there
    um obviously it’s less io than some
    other application maybe
    let me see
    what’s going on here
    all right
    maybe it just took too long
    so now i can expose the
    mavaron coordinator uh api
    to localhost just uh to simplify this of
    course you can also expose it externally
    but for the sake of the demo just
    copy this command let’s just pour
    forward this service to my to my local
    host so i can quickly access it
    then i can obtain the
    mamaron root certificate so this root of
    and the the marron seal i am using here
    is automatically verifying the integrity
    of my control plane you can also do this
    by hand or verify
    the the given attestation statements by
    hand but the dci does this for me
    okay and let’s save the certificate
    somewhere here
    all right and now i can use this
    certificate to authenticate all other um
    connections to either
    um to any of the of these services in my
    in my cluster
    so this is all i need to do to to set up
    marvel run so i can now um go ahead and
    my deployment and i have a manifest
    laying around
    for the uh
    emoji photo voting
    application let’s quickly take a look
    it’s in json format it defines some
    think of it as the
    the containers of your application it
    contains the measurement of the of the
    armpits that run inside those containers
    next to packages let me scroll down a
    bit i’m gonna skip infrastructure is
    marbles and it’s called marble run for
    reason so these marbles are
    more or less mapping to the pots inside
    your kubernetes cluster and each
    marble essentially has a package think
    of it as like the container the enclave
    and then
    the parameters that this
    package should receive and this could
    anything be anything like environment
    anything like that
    and in this case we also define a tls
    tag for some
    manual mutual t-less configuration here
    but they would automatically provide a
    rhythm with an automatic mutual tlast
    tag if i wouldn’t
    define it here
    i have a
    let’s skip
    clients and and roles there are no roles
    or stuff defined i could define roles
    like somebody who’s able to update
    certain packages somebody who is able to
    recover uh
    if this whole thing
    for some reason uh needs to be turned um
    restarted or so
    but let’s keep all that there is a
    secret defined
    and the secret in this case
    a certificate that should be signed by
    the marlborough root certificate and
    this secret is then later on used in my
    tls configuration a little bit
    down below where i would say all
    incoming traffic from my web front and
    should use this certificate to to to
    show to the clients
    and then i say okay um my emoji which is
    application logic
    service here should accept incoming
    traffic on using tls on port 8080 same
    for the voting service and my web
    connects to those two servers so it has
    outgoing connections configured in here
    you will find this
    in our github repository um we have a
    the multivote app has a
    there was a github repository so if
    you’re interested take a look um
    to have a bit more time to understand
    everything that’s that’s listed in there
    but i can deploy this manifest simply
    using the c line
    and then i can on any time at any point
    in time i can verify that this exact
    manifest is deployed
    is set for this deployment so any party
    could do
    maveron verify there’s an api for that
    so you can of course use the
    use the the command line interface but
    any client can use just the api to do
    the same operation verify that the
    that is deployed remotely in my cluster
    is the one i have locally in here
    and of course i just set it so it should
    be okay
    and from that point on i’m done
    configuring my run
    i can install my actual application
    i’m using helm in this case that
    installs the emoji boto app
    and then i can wait for the whole thing
    to come up
    might take a bit
    and then if this is done i’ll be able to
    access the web front end and i’m gonna
    do a local host port forward again
    because i haven’t set up
    a public ip address for this
    aks classroom
    all right let’s see
    are there more questions in the meantime
    last one is coming up so let me prepare
    the port forwarding
    so let’s say i do
    maybe 80
    on localhost
    and then go to my
    web browser
    oh no
    this is running my inside my vm so i
    need of course need to
    need to expose it on
    all right
    and will show me a certificate error
    because my browser doesn’t know the
    mamaron root certificate
    i could of course
    configure it so it’s uh so it will has
    this as a word certificate and then my
    browser would would know that um could
    verify this
    on this application certificate and then
    i’m connected and i can use the
    application as usual
    all right
    so in conclusion um
    cc makes
    systems empty and secure and verifiable
    creates those secure environments in the
    cloud so you can use
    you can you can shift your workloads in
    there without
    the the risk of
    of leaking it to to any malicious
    party or potential
    actor in the in the cloud
    if you’re interested in those tools i
    just showed you
    go to our website or our github you find
    everything on
    every all the links in there you will
    find the tools you will find the demo
    i’ve just shown you
    if you’re interested in the topic in in
    we had a as i said a conference last
    people from the whole industry with a
    keynote from the other cto and
    everything from this conference will pop
    up on our youtube
    i think you will find the link either on
    our website or you go to youtube and
    search for oc3
    and yeah
    i also gave a talk
    similar to this one where i also
    introduced the
    constellation project product we
    released last week that had um that
    allows you for this true lift and shift
    if you don’t want to build new
    applications or have the small tcp as
    i’ve just shown you right now but just
    want to lift and shift everything into a
    private environment about or in a
    private kubernetes definitely definitely
    check out this one
    and yeah that’s it
    thank you very much
    thank you so much moritz it’s been
    really interesting um we are going to go
    directly to the next stop right now uh
    if please remain online if you would
    like to ask more questions we’re going
    to have a session right after uh the
    last talk so basically if you would like
    you can also post them on the q a box
    and i will redirect them directly to
    moritz in the meantime but uh without
    transition we jump directly to melissa
    thanks a lot moritz and melissa the
    stage is yours
    glad to be here everybody i’m so glad to
    um i know we’ve been in this pandemic
    now for the last two years but one of
    the advantages is being able to
    go to switzerland without actually going
    to switzerland
    one of these days i would love to visit
    switzerland um but i am currently
    in denver colorado
    and um enjoying
    the opportunities for virtual events
    here for now
    i’m gonna go ahead and share my screen
    okay so today i’m going to be talking to
    you a little bit about distroless images
    and securing your docker images
    i’m not getting into
    super detailed information this is
    pretty high level i think it’s very easy
    to use containers and especially with
    this particular audience this is
    something you do every day but it’s easy
    to get caught up in just using them
    thinking about what’s going on under the
    covers so in the build up to talking
    about distrolus containers i want to
    just talk about containers in general
    and what’s going on under the covers
    this is kind of a modification of a talk
    that i’ve done in the past there is a
    lot of discussion of docker in this talk
    but just understand this isn’t a a deep
    dive or a tutorial on docker or anything
    there’s lots of good documentation
    already on that so i just wanted to do
    something a little bit different and
    maybe go back and look at the history of
    containers a little bit and how we got
    all right a little bit about where i
    come from i am melissa mckay i am
    currently a developer advocate with
    jfrog but for over 20 years prior i have
    been a software developer and engineer
    in various roles in various companies
    primarily my
    my language is java i worked on java
    server side services but i rarely meet a
    java developer these days that aren’t
    involved in other types of services and
    languages especially when we started you
    know going toward microservices and
    getting smaller services out there up
    and running so i definitely have
    experience in others as well um my most
    recent experience in software
    development prior to becoming developed
    becoming a developer advocate
    kind of pushed me into some new
    territory this was while i was working
    on some teams
    practicing devops and i did get the
    privilege to learn more about how the
    apps i was working on were actually
    deployed and i got an exposure to a
    variety of tools in the devops ecosystem
    and one of those tools was docker of
    course and that’s actually how this talk
    came about originally i just started
    collecting information from my own
    experience learning getting ramped up
    and on how to containerize my apps and
    some of that was trial and error so um
    i have some funny stories about that
    um here’s my contact information you can
    always you know find me on twitter of
    um i’ll go ahead these slides will be
    made available but i’ll go ahead and put
    twitter handle in the chat so you have
    and i recently became a docker champion
    or a docker captain and a java champion
    at jfocus a couple years ago so that was
    that was pretty exciting
    okay um jfrog loves to do uh loves to
    give away stuff so i can’t skip this um
    you can go to this link i will go ahead
    and put that in the chat as well
    and jfrog will give you a chance to
    enter and win a pair of airpod pros so
    you can enter with the qr code or the
    bitly link of course and um
    um actually does it yeah well i guess
    that’s a fiddly link and then a winner
    will be selected within three business
    days and they will contact the winner
    via email and once the winner accepts
    the prize they’ll share the winner’s
    name in the meetup group with the
    so don’t miss that exciting
    okay today’s agenda i’ll go through
    container history
    i’ll go through
    a little bit about containers in real
    life um we’ll review some of the
    container gotchas a lot of you this
    might be review but
    sometimes we need to be reminded of
    these things and then then i’ll get into
    dystrolis images and what those are
    okay so let’s jump in and start learning
    about containers
    i i know that’s not the graphic you were
    expecting but
    i made a commitment to myself not to use
    shipping container pictures and any
    talks about containers ever again i
    think i’ve seen those too many times so
    this theme is bananas i hope you enjoy
    i just want to talk more about how our
    industry has adapted to dealing with
    resources over time
    and when i put these slides together i
    was reminded of a story that my
    grandfather would tell me when i was
    growing up
    he grew up in some pretty hard times and
    depression era in the u.s and um
    when he was a kid they would be gifted a
    banana at christmas
    and this was probably in the 20s or 30s
    so quite a while ago
    but bananas were such a rare treat and
    my grandmother’s family or my
    grandfather’s family they would get this
    banana they would scrape the peel with a
    fork to get every last bit they could
    off and um that might not be the most
    solid analogy but i
    think of that story when i think about
    how computing resources were in the
    1960s and 70s very limited very
    it took a lot to get anything done and
    often a computer would be dedicated for
    long periods of time for a single task
    for a single user
    so these
    these limits on time and resources
    created bottlenecks and inefficiencies
    and just being able to share with each
    other wasn’t quite enough we needed a
    method to share without getting in each
    other’s way
    or having a person inadvertently cause
    an entire system to crash for everyone
    so this required a need for better
    strategies and sharing compute resources
    and that actually started a massive
    path of innovation that we see a lot of
    benefits from today
    and there are some key points in time
    i’m going to bring up there’s a lot of
    other things that happen of course but
    these are these are some highlights
    that brought us to the state we are
    today with containers and i’m just going
    to start with tarot
    change route
    this was born in 1979 during the
    development of the seventh edition of
    unix and it was added to the berkeley
    software distribution in 1982
    this ability to change the apparent root
    directory for a process and its children
    resulted in a bit of isolation in order
    to provide an environment for testing a
    different distribution for example and
    that was the thought when this tool
    initially was entered into the source
    code that it would be used for testing
    that uh true was a great idea solved
    some specific problems but more was
    needed of course
    and the jail command was introduced by
    freebsd in 2000 which is more
    sophisticated than to root and it has
    additional features to help further
    isolate file systems users and networks
    and it had the ability to assign an ip
    address to each jail
    moving on in 2004 we came across solaris
    zones those brought us ahead even
    further by giving an application full
    user process and file system space and
    access to system hardware
    and solaris owns also
    not introduced but highlighted the idea
    of being able to snapshot a file system
    which becomes pretty important when
    you’re you’re running containers
    and then in 2006 google jumped in with
    their process containers which are later
    renamed c groups or control groups those
    are centered around isolating and
    limiting the resource usage of a process
    moving right along in 2008 c groups were
    merged into the linux kernel and that
    along with linux namespaces led to ibm’s
    development of linux containers
    then we get to 2013 this is a big year
    docker came on the scene with bringing
    their ability to package containers and
    move them from one environment to
    another that same year google open
    sourced their let me container that for
    you project which provided applications
    the ability to create and manage their
    own sub containers and from here we saw
    the use of containers and docker
    particularly explode
    in 2014 docker chose to swap out their
    use of that lxe toolset for launching
    containers with lib container in order
    to utilize a native golang solution
    when i first realized docker was written
    and go i was surprised i really didn’t
    have any idea so that was that was an
    interesting bit of trivia
    skipping over some details
    there’s a a lot of different projects
    that were started organizations and
    specs that came out but there was one
    that is really important in june 2015.
    this is important to know about because
    it gives you some insight into some of
    the activity and motivations behind
    shifts in the market as
    with regard to containers
    the open container initiative was
    and this is an organization under the
    linux foundation that includes members
    from many stakeholders including docker
    with the goal of creating open standards
    for container run times and image
    and while all of this is happening in
    the container world there’s a couple of
    other dates that are going to be
    important especially to java dubs and
    being selfish and i’m going to tell you
    some java stuff a little bit here a bit
    of trivia for some of you um
    java 7 was released
    in july of 2011 and work was started on
    java 8 which was released in march of
    2014. so when you start containerizing
    java applications specifically this a
    little bit of history is important to
    know because um you know there’s some
    issues there with this development of
    docker alongside development of java
    there were some misses and disconnects
    there that were fixed much later
    all right uh what exactly is a container
    um let’s just take a quick look at a
    real life container itself what it
    actually looks like on your system this
    is something i did first thing when i
    started learning about containers in
    i’ll just show you how it’s stored
    what’s actually happening under the
    this is probably more important for
    those who deploy maybe containers you
    know by themselves not necessarily using
    orchestration but still good to know
    there’s some there’s some things that
    can happen if you aren’t paying
    attention to how the file system is
    being used
    my first experience
    i was a developer on a new project with
    a title deadline of course that’s
    usually how it is i could argue that
    that’s how most projects begin and the
    best course of action for me was just to
    jump in and start getting something up
    and running on my machine and very
    hands-on i wanna i wanna do it and see
    what happens
    and the docker documentation actually is
    really good i have a link here on this
    slide to that
    easy to find of course so if you find
    yourself in a similar position which i’m
    sure many that’s in the past for most of
    but i definitely recommend going to
    their getting started docs which is
    listed here
    and um
    the first thing to note is just that a
    docker image is really just a tarball of
    a complete file system and when an image
    is unpacked it’s thrown into its own
    directory which becomes its root file
    the second
    is that
    processes that are involved in running
    those containers are just regular linux
    on top of that there’s just a few linux
    features that are used together in a way
    to achieve that isolation that we expect
    from containers
    are important
    this is what is used to provide virtual
    separation between these containers this
    is how the process is inside one
    container don’t interfere with the host
    or with processes inside another
    and here
    you can see the name spaces that were
    set up for a
    postgres container that i have on my
    machine running not today i turned it
    off today to clean it up for you guys
    but um i i just pulled some screenshots
    of and i’m on a mac by the way uh so
    you’ll see some some mac commands as
    um the c groups functionality also
    essential in constraining how much a
    container can use things like cpu memory
    network bandwidth etc and i can set
    these constraints by including options
    on the docker run command when launching
    an image and of course these can be set
    using orchestration as well um you’re
    not necessarily stuck with the docker
    runtime either but this is for
    development on a local box
    um here you can see i’ve i you know
    started launched a container and i
    constrained the
    memory limit of my containers running
    and then just some more details on the
    file system where containers and images
    are actually stored first off after
    you’ve installed docker running the
    command docker info will spit out a ton
    of information about your installation
    using the docker root directory and this
    is where most everything you’re going to
    care about regarding your doctor images
    and containers will be stored
    if you’re on a mac of course these are
    actually running in a tiny vm so you’re
    going to need to use some utility i used
    to use screen but it’s it’s not real
    consistent and available
    so uh it’s not working for like later
    versions um so in order to
    get into this directory to just kind of
    look around and see what’s going on and
    i have a
    command here a docker run command to
    start another container that actually
    enters um the uts namespace the network
    namespace and the ipc namespace and it
    targets that pit of one so that you can
    get some information about this um
    root directory
    it’s pretty interesting just to navigate
    around in there and see how that how it
    changes when you launch containers and
    and remove images and things like that
    later version of windows i actually have
    a windows box on order it’s been years
    since i’ve been on windows
    i swore i’d never go back but
    the latest wsl
    stuff that they have on the on
    i don’t think it’s windows 10 and up
    they have the capability of running
    linux containers natively using wsl the
    windows subsystem for linux
    that there’s a path here where i did
    find where the root directory can be can
    be found on a windows system
    so i’m excited to get that that machine
    up and running um it’ll be a while
    before it gets delivered here but
    it’ll be cool
    all right uh this slide shows just some
    information about uh where the images
    are stored on your system um
    i have
    my available images i just use the
    docker images command to see them i have
    several installed um
    i just showed a couple here
    i can inspect any image by using its
    image id and this spits out a ton of
    information but what i want to highlight
    here is the graph driver section which
    that contains a bunch of paths to
    directories where all of the layers of
    these image
    so doctor images are composed of layers
    most of you know this already those
    or not
    they correlate to instructions in the
    original docker file that was used to
    the images originally and these
    translate in direct to directories and
    these layers can be shared across the
    images in order to save space
    the lower der mergeder and upper der
    sections are important the lower
    directory contains all of the
    directories or layers that were used to
    build the original image these are read
    the upper dir directory contains all of
    the content that has been modified while
    this container is running
    and if modifications are needed for a
    read-only layer
    in lower dir
    then that layer is copied into the upper
    dir where it can be written to this is
    called a copy on write operation
    and it’s really important to remember
    always the data in the upper dir is
    ephemeral data it only lives as long as
    the container lives so if you have data
    that you want to keep intact obviously
    you need to use mounting or volume
    features and um to mount a separate
    location or a separate drive
    to access a database that kind of thing
    lastly the mergeder is kind of like a
    virtual directory it combines everything
    from lower dura and upper dir and this
    way this is how the union file system
    any edited layers that were copied into
    upper dir will overlay the layers in the
    lower dir so if you were to exec inside
    that container that’s what you would see
    this is just an example of a few
    containers that are running on a system
    i’ve got a jfrog container registry
    installation that’s pretty old by the
    way i don’t think we have that available
    anymore but it includes a container for
    artifactory and a container for a
    postgres database and the other is just
    a simple test container i was playing
    around with
    the container ids of the running
    containers match up with the container
    subdirectory names that’s something to
    if you stop a container that
    corresponding directory doesn’t
    automatically go away until that
    container is actually removed with
    docker the docker rm command
    so if you have stopped containers lying
    around especially on your local system
    and that never get cleaned up or even in
    your test systems you might see
    available space start to dwindle so take
    care that you’re cleaning up after
    yourself there is a docker system prune
    command i use that regularly when i do
    docker workshops just because i want to
    start fresh
    you can run that to help clean things up
    remove all images remove containers that
    kind of stuff or you can launch a flag
    to indicate that a
    container should be removed when it’s
    finished running
    obviously this stuff is also taken care
    of in orchestration frameworks as well
    let’s make sure to use them
    so these tool sets around building and
    running all of these images have made
    things so easy it’s also easy to shoot
    yourself in the foot in a few places it
    can be deceptively simple for especially
    people just learning and starting so i’m
    just going to go over some of the most
    common gotchas pretty quickly
    i’ll include some jvm specific gotchas
    as well that i ran into immediately when
    i first started working with containers
    the first of course is running as root
    i’ll be honest i was initially when i
    initially started i was so excited um i
    just didn’t really take this one
    seriously enough
    but now that
    you know that processes inside a running
    container are just like any other
    process on the system it’s scary now to
    run as root inside a container and doing
    that opens up the possibility of a
    process escaping the intended confines
    of the container and gaining access to
    host resources
    reducing that attack surface of your
    container by following the principle of
    least privilege
    that’s important and although containers
    are designed not to affect other running
    containers they will if someone
    gains access to your container
    and immediately has root privileges they
    can wreak havoc on your host as well
    so how do we mitigate this problem the
    best thing to do is when you’re building
    initially make sure that you are
    creating a new user uh you can use that
    user command inside the docker file when
    the container is built in order to run
    processes as that user
    and you can also specify user just when
    running the docker run command but that
    leaves open the possibility of
    forgetting to do that so it’s nice
    if it’s just built in and your
    production image is set up by default
    not to run as root many companies by now
    have set up policies and stuff to
    prevent this from happening so um but i
    i have been on teams where this was not
    paid enough attention to
    also pay attention to these official
    images that you get from docker hub you
    need to know whether or not they run as
    root or if you need to you know build
    another image on top of those
    to fix that
    even though docker provides you with the
    ability to set resource limits on your
    container it doesn’t automatically do it
    for you the default settings are free
    for all with no limits
    so make sure you understand the resource
    needs of your application
    too little and your container will die
    from starvation of course too much and
    the container could smother others on
    the system the resource usage of your
    containers that’s something that you
    want to monitor over time and adjust as
    needed and it’s good to determine if
    something is going wrong or if load on
    your system has changed for some reason
    you can use the monitoring information
    to determine if your application needs
    to be split up further with certain
    action scale differently i went into a
    whole process with that on a team of
    mine where we divided up our application
    into smaller pieces and we’re able to
    scale like our front end very
    differently than the back end services
    and then never updating of course this
    is a huge
    security issue and it’s easy to get
    complacent not pay attention to what’s
    actually being pulled when you build
    not only do you need to be aware of
    outdated versions that you specify in
    your docker file but you need to pay
    attention to what’s in the base image
    that it’s that it’s coming from um
    not updating these packages and
    libraries can you know lead to some
    pretty embarrassing results especially
    when there are tools available now um
    all over the place now to alert you when
    security issues have been discovered
    with specific artifacts and i’ll show
    you an example of that in a bit
    um even you know even if you’ve gone so
    far you’ve ensured your containers are
    running with a non-privileged user
    there’s still risk if there’s known
    vulnerabilities that exist within your
    container or even on the kernel of the
    host so from time to time exploits are
    those enable attackers to escape a
    container so keep up with those security
    this is a very java specific one
    this has to do with that
    in tandem development of java and docker
    um pretty specific to job applications
    but still important you never know when
    you need to containerize one of those
    the jvm is pretty clever at
    automatically determining settings for
    your swap and heap and your garbage
    collection behavior based on things like
    the memory and the number of cores
    available on the host
    and remember earlier in our lesson when
    i mentioned the dates for java 7 and
    java 8.
    considering the timeline of
    docker and java releases java 7 and
    versions of 8 and certainly earlier
    are not fully container aware meaning
    that your java application won’t
    necessarily obey
    those memory and cpu constraints that
    you put on your container and you could
    end up with some surprise out of memory
    killer activity which is exactly what
    happened to me
    this means
    um your
    you know your java application is not
    going to
    behave like you expect
    the reason for this is the mechanisms in
    the that the jbm used in these older
    versions to retrieve the resources
    those actually were coming from the host
    machine and not the control group limits
    uh where you would expect them to come
    there were some improvements around
    container awareness that were introduced
    in java 8 update 131
    and there’s further improvements in
    later versions but to really get all of
    the benefits of container awareness you
    really should at least get to java 11.
    that’s an lts release uh latest of
    course that’s come out is 17 so if you
    keep up with those that’d be great
    um and then image bloat this is a big
    one um
    i learned quickly about the
    in the docker ignore file so when you’re
    when you’re building
    a docker image
    um oftentimes
    you may hit you know a dot at the end of
    this command the build command which
    indicates yeah this is where my docker
    file is this is the context meaning this
    is where i am in the file system
    structure that i want to send
    to the docker daemon to build this image
    well in doing that you may be sending
    over stuff and um if you use one of
    these global copy commands in a docker
    file you could be copying things in like
    your dot git directory which probably
    isn’t you know something that you want
    to have in a production image
    and i often see like copying an entire
    source tree in one line which means one
    layer that means if anything changes in
    that particular
    set of files that layer has to be
    regenerated every time you build so
    you know organizing your code so that
    that changes the most often is in its
    own layer
    maybe consider implementations in one
    one layer and abstractions in a
    different layer
    um that’s one idea
    and then of course avoid
    you know scooting in your secrets um
    even if you go through the effort of
    like in your docker file trying to
    delete stuff
    they still exist in the previous layer
    because every layer is just a diff of
    the layer
    not good enough to just go through your
    docker file and delete things i’ve
    actually seen that
    it doesn’t work that way
    um also you know learning how to use a
    multi-stage docker build is a good idea
    that’s something relatively new um to
    the ecosystem being able to
    build in such a way where you’re not
    including um extra packaging um
    tools and things like that like maven
    for example or npm or or whatever so you
    just have what you absolutely need in
    the final image that you’ve built
    all right distro list so all that build
    up to understand why people are looking
    at distroless images
    so um
    you know pulling in a large parent or
    base image even if it’s you know
    relatively safe
    official image from docker these could
    include a bunch of stuff that you don’t
    need they it increases your attack
    surface area from a security perspective
    and shipping these large images around
    and storing them can become clumsy and
    slow and expensive since we’re all
    paying for storage these days and this
    reminds me of and i’m going to date
    myself a little bit but remember when
    i had to pay
    you know for every text i sent on my
    cell phone i’m hoping at some point
    we’ll get beyond this paying for every
    um you know byte that we’re using for
    storage but this is where we are today
    so we need to manage our resources
    um also
    distro list containers actually not only
    do they you know make a point to strip
    everything out that you don’t need
    yeah um you want just the minimum
    packages required to run your
    application this means even you know
    stripping the os down to its bare
    minimum this means no shell generally so
    there’s no exec inside that container
    once it’s running out there that’s
    actually a pretty good security measure
    and something to consider i know it
    makes troubleshooting a little more
    difficult but if you’re having to exec
    in a container on production you’ve got
    some bigger problems going on
    you can always you know in in your
    development process you could develop
    you know debugging
    type uh container or images in order to
    get beyond some of that
    this is an example of a multi-stage
    build so you can get a collection of um
    distro-less containers from this link
    these are ones that google built
    they built they’re built from bazel so i
    mean they’re not using docker to build
    these they’re just images that um
    they’ve built and had that they consider
    tried and true most of them they’re all
    based on debian
    um i’m going to go ahead and put that
    link here so you don’t have to
    type it in if you want to just go check
    out and see what they have they just
    have a few for and they’re they’re
    language specific so
    we’ll look at one for java in a second
    anyway this multi-stage build notice
    that there’s two from statements
    um the top one and you know pulls in a
    full on open jdk
    image and then the build happens
    and then the the line seven the second
    from statement that’s the dis trellis
    image so the result from the first group
    is pulled into the second group and that
    makes it very small very small image
    to end up with again there’s no shell it
    cannot exact into this image
    all right and um just one mention on
    managing your images i’m going to bring
    up jfrog artifactory just to show you
    what that looks like i’m not going to
    give you a long spiel about it but i did
    go ahead and
    pull up some images just to see what
    they look like
    just command line so we can actually see
    if these distro-less images are smaller
    i you know love to just believe these
    statements but i always want to verify
    so let’s just look at a couple really
    fast and see what they look like
    the only the last thing i want to say
    about this slide is just
    i do have
    a d-zone rest card on this for container
    registries considering where to store
    your register your images and what to
    look for when you’re looking for a
    container registry
    just note that whenever you’re working
    with docker on on your machine the
    default is docker hub you’re constantly
    going to be going out to the public
    docker hub so that’s something that
    they have recently set limitations on
    so uh it’s good to always have you know
    a cached um
    collection of images that you use in
    your system that you rely on
    okay i’ll bring the slide back up at the
    very end um in case there’s any
    questions that need to be answered but
    don’t forget to go to this link and get
    your earpods
    but first
    i just want to show you some images that
    i have pulled in
    i pulled down an open jdk image from
    docker hub and then i pulled down a java
    um distro-less image from the google
    you can see
    these top two are the same they’re both
    open jdk i just have one that’s tagged
    with my
    jfrog instance so that i was able to
    push it up there to look at some more
    details about it
    um but these are actually both the same
    image but look at the size of these
    we’re looking at 471 megabytes
    whereas the distrolus images again these
    are also identical to each other they
    just have different tags because they’re
    coming from different registries um
    that’s about half the size right 231
    megabytes so that’s pretty good i think
    you know when you when you start doing
    the math and
    have a lot of these out there
    needing to be stored or run or moved
    around in your infrastructure
    that’s a pretty big difference
    note the created date so i just you know
    recently grabbed the latest open jdk
    this date this is the epoch the reason
    it’s like that is because these distro
    list images are meant to be
    uh exactly
    reproducible builds
    so and and the spec requires a created
    date so and putting nothing in there you
    end up with epoch so that’s that’s what
    that’s about these weren’t actually
    52 years ago
    but just some something interesting to
    um note
    all right so the next thing i do i pull
    i pull these images down i tag them and
    then i push them up to my own
    container registry and i’m going to log
    in there and we’re just going to take
    a look at it
    this is a um
    a free
    version of
    jfrog platform
    and i pull these up every now and then
    when i do workshops and stuff and other
    people can sign up for these instances
    they’re they’re limited feature
    type things but they’re they’re really
    these are all of the
    images that i pushed up here and let’s
    just compare well there’s two of them
    that we want to look at this distro-less
    one and this open jdk one so just to
    check something out here
    it’s said that distress images because
    they have
    you know less
    stuff in them less content that they
    should be more secure you know less
    problems with the packages inside them
    and these have been scanned by x-ray and
    let’s just take a look at what a distro
    list looks like compared to the full on
    open jdk
    image so i’m going to navigate here to
    x-ray information um i’m not going to
    say anything about like what kind of
    security problems these are i’m just
    looking at raw numbers at the moment
    just for our benefit this is something
    for you to do for homework if you want
    to you know dig into these a little bit
    more and see what actual problems there
    are here but this is the distro-less
    images and it looks like we have three
    critical violations we have you know
    31 security problems some of these are
    low that we may not care about
    all right let’s go back to the full on
    open jdk
    drill down into that image
    check out the scanning there here we
    have three violations still but 94
    security issues
    um there’s still three there’s a lot
    more medium
    issues so it’s just interesting whenever
    you’re doing this never take for granted
    that what is being said is true in the
    past when i have pushed up a distroless
    image with like an older version of
    java and stuff in fact the last time i
    gave this talk this was absolutely
    reversed and the distro list actually
    had more problems which was an
    interesting find so today it’s looking
    much better
    all right that’s all i have for you um i
    know a lot of that might have been
    review for a lot of you but i hope that
    it was fun and engaging and you got some
    reminders of some things and maybe learn
    something new um anyone have any
    thanks a lot melissa maybe some
    questions from business we still have a
    few minutes
    for this session
    i actually have one
    that i was thinking about maybe
    you’ve heard about corkus
    yes yes yes yes
    absolutely right it’s uh yes it’s a it’s
    a for those who don’t know it’s a
    run time and framework to create
    java applications uh specifically
    targeted for containers
    and uh with very impressive results
    right because i’ve seen them like
    immediately like not even 10 megabytes
    big and you’re like how can you even fit
    an application inside of that so what
    was your experience with it
    well yeah i mean it makes sense right
    because um java applications you often
    get libraries pulled in with things that
    you don’t you aren’t actually using you
    don’t actually need and the problem with
    that is you go to these scanning things
    you know the security scanning thing and
    you find all of these problems but
    they’re things that you don’t really
    care about because you aren’t actually
    using them in your application so corkus
    does very well with that you know
    eliminating a lot of that and and making
    your attack area smaller i’m not pulling
    in libraries unless you’re actually
    using them so that would be a good
    homework assignment is to get one of
    these pull up um some images push up
    some images that are based on corkus and
    see if you know see how much of a
    difference it makes
    absolutely absolutely for what i’ve seen
    a little i’ve been playing with it you
    can even you can create a classic jar or
    a war file and you can put it inside of
    tomcat and it would run anyway so it
    really opens up
    the door to native compilation and uh
    also classic java interoperability which
    i think is a very interesting
    in any case so i was thinking about that
    but yes we had vision exciting stuff
    we try to make our container images as
    small as possible all the time all the
    so many improvements have been made with
    java because in the beginning it was
    pretty it wasn’t that good trying to
    containerize job applications scared a
    lot of people away but it’s so much
    better now exactly the same problem
    happens with net many of these run times
    or even the erlang
    beam machine basically when you have
    these big run times with managed memory
    and so on they they were created at the
    time where containers were not on the
    roadmap so clearly they had to adapt uh
    in that case workers has done its
    homework very well
    so i don’t see any questions from the
    audience but maybe there will be more at
    the end so melissa i invite you to stay
    until the end of the event maybe there’s
    somebody comes up with with a question
    by then and thank you so much it’s been
    a great great reminder of how we should
    be building our containers really cool
    and nice tips here and there so that was
    really really nice thank you so much and
    now we go directly to thomas for the
    last presentation of the day
    that’s about psyllium and ebpf thomas
    screen is yours
    thanks a lot adrian
    and hello everybody first of all
    apologies for my voice i’ve been sick
    the last couple of days so if you hear a
    slight cough that’s from that but in
    general i feel fine
    let’s jump in um let me know once you
    can see slides coming up here
    is the screen visible already yes
    perfect excellent
    all right so i will be talking about
    psyllium and ebpf
    and my name is thomas i’m one of the
    creators of the studio project i’m also
    a co-founder and
    cto i surveillance the company that was
    created or founded by the creators of
    psyllium is cloud native networking
    security and observability and
    before we go into more details on that i
    want to just briefly talk about like
    where where i am where we are as a team
    and so on so um solemn has been created
    by i surveilled we’re a venture
    capitalist backed startup with dual
    headquarters in switzerland in the us
    switzerland because i’m swiss and i love
    skiing so much so i never actually um
    moved outside of switzerland even though
    i’ve been working for american companies
    since forever i can think
    since i was 20s i’ve been working for
    red hat for a very long time as long as
    kernel developer
    and hence our team is
    basically split between the us and
    switzerland our company is is heavily
    focused on open source most of us have a
    background in linux kernel development
    or open source is a modern way um and we
    are heavily investing into open source
    we’re most well known for psyllium and
    also for co-creation and pro-maintenance
    of edpf and obviously we’ll dive into
    that some more here
    but to make it very clear what is what
    and what we’re going to talk about today
    psyllium and ebpf those are open source
    projects that’s the focus of today i
    surveilled this the company behind it if
    you are planning or if you are
    interested on sodium in an enterprise
    context in a production environment and
    you are
    you are desiring and apply support feel
    free to ping me and we can talk about
    that as well that’s enough of the sales
    pitch let’s jump into the open source
    so what is psyllium syllabus networking
    and load balancing foremost most of you
    probably heard of cilia in the context
    of cni container networking
    interface that’s what kubernetes
    requires in order for
    pods containers to be able to
    communicate with each other across the
    network because kubernetes itself does
    not really have a networking model or
    networking implementation itself but
    then student can do so much more beyond
    that kubernetes services multi-cluster
    routing they can integrate existing
    workloads running on vms or metal
    machines we can do a whole lot of
    network security from segmentation some
    of the service measures cases we heard
    earlier including mutual authentication
    but also identity based
    enforcement and transparent encryption
    and then even more important is the
    observability piece where we can
    actually tell you what’s going on
    in the form of metrics flow locks uis
    and zone we’ll dive into that
    what is very unique about sodium and why
    so many are so excited about sulim and
    why salem is currently taking the world
    by storm is because of its use of bpf
    and if you have never heard of dpf or
    ebpf before good news will cover that in
    a brief moment
    ebpf is at the core of psyllium and is
    why cylinder exists in fact our team has
    first invested about one and a half two
    years into advancing eppf further to
    then build psyllium so the whole reason
    cellium exists is because of the ukf by
    applying ebpf to the world of containers
    and kubernetes
    so what is this evpf technology it has a
    cute little logo like many other
    technologies but what is it
    actually not very complicated and you
    will get it in a nutshell because it’s
    very very similar to what javascript has
    done to the web browser
    ebpf does to the linux kernel or to the
    operating system i will you will learn
    you will understand this very very
    evpf is nothing more than the ability to
    run a program on a certain event
    for example when a system calls being
    made in this case a process is making an
    ve system called we can then run a
    program to intercept that system call
    perform some action some run some code
    and return a verdict for example block
    the system call or export some
    visibility or do some other take some
    other action we can do this for a whole
    lot of events system calls network
    packets i o operations trace points
    function calls user space function calls
    so you can even run an edpf program when
    your own application performs a certain
    or inbox or calls a certain function so
    very very very similar to how you can
    run javascript code when for example the
    visitor of your website clicks on a
    button or mouse is over somewhere you
    can run javascript code ebpf does the
    for the linux kernel
    it’s as simple as this
    let’s now dive into what this can
    actually do because why is that
    important that sounds neat but it’s not
    necessarily a completely new concept
    programmability has been around as a
    concept forever what does it actually do
    what does it what does it enable here’s
    a here’s a use case or a very concrete
    benchmark of one application of ebpm
    there are many many hundreds and
    hundreds thousands of different
    applications of ukf this is one of them
    which i think is easy to understand and
    very impressive this is a benchmark of
    http visibility
    and it measures the baseline so no
    visibility this is an hdp benchmark
    that’s the blue bar
    and then the proxy based visibility so
    this is using
    a proxy such as whether
    aha proxy nginx
    doesn’t actually matter that much
    they’re all mostly the same that’s the
    yellow one extracting http visibility so
    visibility in the form of this request
    has been made this was the latency of
    the ring of the response and so on and
    then in red what is the latency
    difference to an ebpf based in kernel
    implementation of this which does not
    require for example connection
    termination and so on and i know that
    this table will draw a lot of questions
    or we’ll we’ll top a lot of questions um
    but this is an implementation that has
    been done where we can passively provide
    http visibility with epf and you can see
    that the latency is almost the same as
    baseline whereas a proxy based
    implementation actually adds quite a bit
    of latency that’s one concrete very very
    one concrete use case here
    but if we take a step back and actually
    apply and talk about this a bit more
    broadly edpf is actually about a lot
    more impactful as i mentioned my my
    background has been linux kernel
    development for about 15 years and i
    often found myself in exactly this
    situation now somebody would come up and
    say hey i need to observe my app better
    or i need this feature or this other
    feature or i can remember
    when i was early on working on openshift
    like the original versions of openshift
    pre pre-kubernetes um openshift
    developers would come to me or come to
    our team and say hey we need this
    feature great then we will work with the
    upstream kernel community to get this
    feature in
    which by its own will probably take
    about a year or so because it’s
    complicated to do kernel changes and
    it’s hard to convince all the other
    kernel engineers but then it didn’t
    actually end there it would take years
    and years and years for that new kernel
    version to make it into the hands of
    users because most of the users in
    particular reddit users obviously uh
    consume reddit enterprise linux in
    another enterprise distribution so it
    would take up to five years for
    the beginning or the the ask for a
    change to the delivery of a kernel
    version that would actually function for
    that user and by that time
    the requirements of the user had likely
    already changed so this made the kernel
    a very difficult place to innovate
    essentially this this meant that a lot
    of innovation actually moved to user
    space why we have seen massive rise of
    user space network in user space io user
    space security
    things like g visor building entire
    abstraction platform for system calls
    and so on because the linux kernel
    itself became very challenging to
    innovate in we even saw entire forks of
    the little kernel for example for
    android because it was not possible to
    innovate quickly enough
    ebpf is now changing this all of a
    sudden because similar to how javascript
    has unlocked or has decoupled
    the browser version from innovation
    right like if some of you have been
    using internet for a long time you
    probably remember way back when you had
    to upgrade your your web browser to
    visit certain websites when was the last
    time that you had to do that
    probably not recently because javascript
    and auto programmability frameworks
    don’t require you to do that anymore and
    ebpf now unlocks the same for the
    operating system in particular for linux
    so what used to take years can now be
    done within days because we can
    dynamically load a program and change
    the behavior of the linus card
    now that sounds potentially very scary
    to some of you and some of you may even
    say well what is different to for
    example linus kernel module it was
    already possible to extend or to load
    additional kernel module code at one
    this is where the different com
    difference comes into play so ebpm this
    is the entire evpf landscape from a high
    level to a low level and this low level
    here this is the coral this is the
    operating system level this is where the
    actual edpf program will run and part of
    this you see a box called verifier and
    chip just in time compiler this layer
    actually adds a sandbox concept very
    similar to how containers are sandboxed
    as well and they ship compiler which
    will take
    generic bytecode generic edpf bytecode
    and just in time compile that into your
    cpu instruction sets very similar to how
    this is done for other interpreted
    languages like javascript or java as
    this means that yes we can dynamically
    load a program very similar to a linux
    pro module but we’re actually running in
    a sandboxed program or sandbox
    environment very similar to our
    javascript program is also sandboxed
    the just in time compiler actually
    ensures that we can run as efficiently
    as if we recompile and restart our
    kernel or our entire machine
    and then we have a kernel api which
    actually ensures that our programs are
    stable and portable which solves
    probably the lesser known problem of
    linux kernel modules which is that linux
    kernel modules are very very hard to
    make portable cross arbitrary kernel
    versions so these are the problems that
    are solved by edpf and all of a sudden
    the kernel space becomes a place of
    innovation again which is amazing
    because the kernel is a truly powerful
    place to run coding because the kernel
    can see everything the kernel can
    control everything and we’ll see
    examples on how that can be used so
    that’s the lower levels of evpf and if
    you look very clearly or very very
    closely you can actually see a windows
    logo there even though the ebpf is
    coming from
    from linux microsoft has recently ported
    evpf over to windows as well so ubf is
    now available on windows
    on top of that you see a bunch of sdks
    and libraries which will make edpf
    programmable in higher level languages
    because most of you will never want to
    write an etf program in assembly
    bytecode so you can use python go c plus
    plus c roster order languages um to
    actually write ebpf programs in and then
    you can take it to an even higher level
    which is evpf projects or the end user
    facing projects such as vcc psyllium
    falco catron dpf trace and so on an
    ever-growing list of ebpf based projects
    would actually solve use cases for end
    in the next step we’ll now focus a bit
    on cylinder i wanted to give a high
    level in showing to bpf first because
    cilium is obviously not the only vpf
    application out there
    in terms of governance uh ebpf
    has been or we together with some of the
    other industry members that have been
    involved in the ukf for a long time
    you’ve together created ebpf foundation
    it’s a linux foundation based foundation
    um you can see the founding members
    here since then the members or the list
    of members have been extended quite a
    bit and the ebpf ecosystem is growing
    very quickly
    but let’s now shift gears into syria and
    actually look at
    what type of application what type of
    services what type of values can we
    provide if in epf
    so if we go back in here you can see in
    the picture here cylinder is kind of in
    the middle of a world that shows
    kubernetes mesos darker hdl today it’s
    kubernetes of course
    but in the end sodium and ebpf is
    actually very generic and very
    applicable to everything as we will
    learn it’s not even limited to
    containerized applications
    sodium adoption here you can see across
    the board this is not even inclusive at
    all but from
    big sas providers to cloud providers for
    example gke eks anywhere are already
    using silhouette of silly by default
    alibaba in the high performance option
    digital ocean whereas other smaller qms
    distributions are
    using sodium as the default c9 to
    provide networking and security and then
    from telcos to financials
    syllabus kind of instead of an ebpf is
    very applicable to a very wide stream of
    use cases and thus adoption is bright as
    as we now start to dive a little bit
    deeper here um i’m going to offer kind
    of this stage of the journey of what
    users typically go through when they
    apply or use cilium in a kubernetes
    environment and if you want to want me
    to dive deeper on any of them in
    particular feel free to mention that in
    the q a
    window i have that open in another
    screen so i can actually see that and
    can dive deeper on that particular topic
    so first of all in stage one is what
    typically what we’re seeing when you get
    your initial kubernetes cluster up and
    running this is where you need basic
    part and service connectivity and you
    will also probably care about initial
    cluster hardening
    for example um
    making sure that kubernetes or back is
    configured correctly making sure that
    you haven’t accidentally um
    exposed your communist dashboard to the
    world and so on you might also be
    deploying something like container image
    scanning in this stage of course you can
    pretty much use any cni because all you
    need here is basic part and service
    connectivity you don’t have additional
    security or scalability requirements as
    you go into stage two which we usually
    say you enter when you run sensitive
    data workloads or you have multi-tenancy
    requirements so you have multiple teams
    um running on a on a single kubernetes
    cluster this is when you start carrying
    about zero trust network security
    transparent encryption you might want
    compliance monitoring what is encrypted
    what not what network policy violations
    are happening and so on um also you want
    to monitor your applications for example
    measure the http latency of your
    services or
    identify how many http 500 error codes
    are being returned and so on to build
    for example the typical golden single
    dashboards also you might have
    requirements around efficient load
    balancing already as you scale out and
    you very likely need to integrate with
    external workloads which are not
    containerizable or which are not part of
    kubernetes and very likely they’re still
    sitting behind a traditional firewall
    which also very likely is not cloud
    native aware so we’ll have a really
    really hard time coping with
    containerized workloads that constantly
    change ips all of the time this is where
    something like the static reverse iep
    feature can come in which actually binds
    volatile containers behind a stable ip
    which you can then allow
    through your um traditional firewalls
    stage three is typically when you
    adopt kind of a more micro services
    architecture sometimes stage three
    actually comes on stage two if you’re
    very modern and if this comes natural to
    you for more traditional enterprises
    that clearly kind of a later stage
    excuse me
    this is where things like multi-cluster
    comes into play layer seven
    observability layer seven traffic
    um git ops integration for example to do
    network policy integrations in gitlab
    github as well as advanced security
    runtime protection that we can offer
    advanced scale advanced churn and all
    the way up to multi-cloud use cases
    if you want me to focus on any of them
    in particular feel free to mention that
    in the q a channel and i will spend a
    bit of more time on that particular
    as we now dive a bit deeper into the
    onto the individual aspects so the
    scalable and secure networking most of
    you will know us already as a very
    efficient and scalable cni we can run
    on-prem we can chrome cloud we can run
    hybrid cloud so we can run opening for
    example bgp mode and overlay the
    excellence you leave whatever in cloud
    provider environments we can maybe
    integrate with the cloud wider sdns but
    then we can also actually not use them
    because some of them have for example
    limitations in scale and you will run
    into ik exhaust issues so you can go
    back and actually run an overlay
    um but then as well because we can run
    on all the cloud providers like equally
    good we’re really good for multi-cloud
    use cases because we can then for
    example run a multi-cluster scenario
    across cloud providers for a cross-cloud
    and on-time environments
    on our level left we can not only
    connect pods or services together we can
    also provide load balancing in a variety
    of ways we can replace q proxy so you
    can achieve a high performance
    implementation of q proxy that can scale
    really really well but then we can do a
    lot more we can also become what’s
    called a north south load balancer
    essentially load balancing into your
    cluster so you can turn any linux and in
    the future windows machine into a load
    balancer machine and for example replace
    entire five boxes
    in the latest version we can also
    implement kubernetes ingress so we have
    an implementation of the kubernetes
    ingress controller and can just also do
    all of the layer 7 load balancing as
    well both into the cluster and also
    services and across clusters as well
    on the security side
    first of all yes we are a cni which
    means we can do enforcement on the
    network level which is actually very
    very important than an aspect that
    should not be underestimated
    even if you’re running a high-level
    concept like a service mesh you almost
    need to be aware for example a sidecar
    proxy or a proxy based authentication
    method which yes adds additional
    security from a mutual authentication
    perspective does not prevent you from
    all the potential leaks that you might
    see on a network if a proxy does not
    support a particular network protocol it
    is obviously not able to enforce any
    rules on that so in order to have
    correct network security you also need
    always need to protect your network
    level first and then add mutual
    authentication on top which is why
    psyllium combines the two together so
    very similar to a service mesh approach
    with which is using mtls we’re taking
    the mutual authentication to another
    level and saying yes we are enforcing
    useful authentication but we’re not
    limiting ourselves to tls which means we
    have an identity based concept but we
    can enforce this mutual authentication
    with identities not on the tls level but
    actually on something more generically
    that can that can support any network
    protocol which is amazing because now
    all of a sudden you can support non-tcp
    protocols and you can ensure that
    nothing can bypass
    your authentication and authorization
    obviously we also support just standard
    kubernetes now policy then we can go
    further and look into for example dns
    based policies into api policies
    and so on
    besides the segmentation or
    authentication here we can also do
    transfer encryption we can do to do this
    using ipsec or wireguard
    both of them have pros and cons ipsec
    has the pros of being uh fips compliant
    wireguard can be slightly faster if no
    cryptographic or if no hardware offload
    for cryptographic operations is
    and then last the last pocket here is
    the expansion this is actually going
    beyond the single fibonacci cluster as i
    mentioned we can do multi-cluster in a
    completely proxy free manner which means
    you can connect multiple clusters
    together but not actually introducing
    either egress or ingress proxies which
    means you could preserve your
    natural organic network performance as
    we can then also integrate into or we
    can integrate non-kubernetes node
    container workloads as well what does
    this mean so as i mentioned cylinder is
    not really limited to containers or
    kubernetes at all you can actually run
    sodium on any machine that you want and
    integrate that machine into your mesh or
    into your kubernetes cluster so you can
    run let’s say have your existing ec2
    fleet of vms or your bare metal database
    server you can run psyllium on them and
    then have them appear have these
    machines appear as workloads very
    similar as if they were parts so they
    can be reached from parts you can apply
    network policy they can the external
    workloads can it can reach the
    kubernetes dns they can talk to
    kubernetes services and so on that gives
    you a lot of flexibility
    to mesh things together
    and to get to a zero trust networking
    concept without actually having to
    containerize everything
    switching gears a little bit and looking
    into kind of the network policy
    forensics and compliance use cases
    enforcement pawn is obviously important
    but there is more to this and it’s the
    visibility side and we’ll actually
    double click on this on this lower left
    window air bit because that’s very
    crucial so keep that in mind for a
    second while we complete the slide so
    yes we have kind of this identity aware
    network and runtime visibility later to
    look at but then we can obviously also
    do a lot of metrics we can monitor
    pretty much everything of course how
    much are we forwarding how much are we
    dropping um we have a tls parties we can
    for example answer questions such as
    what has
    what are the negotiated tls ciphers that
    are in play has anybody attempted a tls
    downgrade attack uh what percentage of
    my traffic leaving the cluster is not
    tls encrypted and so on and then we have
    more traditional network tap
    and port mirroring functionality as well
    which you for example can use to
    integrate an ids such as sorry coda
    the nice thing about this network tab
    ideas insertion part is it’s it’s
    identity where so you can actually apply
    this for a subset of the traffic only
    and then there is splunk and splunk and
    elastic search integration all of this
    visibility can obviously be shown using
    hubble ui our own ui but also data grade
    into your existing sim platform as well
    i promise that we will double click on
    this network and runtime visibility use
    case a little bit this is an example of
    a traditional network flow log a five
    tuple flow lock and you can see um
    typical network communication this ip
    talking to this ip on a particular port
    and whether it was rejected or accepted
    if we take this a bit further
    this is what we provide and
    when you look at this new view um you
    will actually first of all kind of
    question is this even still a network
    flow or is this even showing the network
    at all because it doesn’t look like
    additional network flow log at all but
    it actually is it just provides a lot of
    additional context and identity
    information but if you look closer we
    see on the lower on the left side we see
    the source identity we see a network
    namespace or kind of kubernetes
    namespace a pod name and then we see the
    entire process ancestry tree of the
    workload running so in this case it was
    a container runtime in this case docker
    d which was launching or which has
    launched a container and inside of the
    container there’s a binary crawler
    running which is a node app which is
    running node server.js which then for
    obvious reasons when you look at this
    got compromised and an attacker launched
    a reverse shell and you can see with the
    arrows you can see the individual
    so the node app actually made
    connections out to we
    can see the dns feasibility that we have
    and also reached out to the
    elasticsearch which is an internal
    and then on the netcats obviously
    we see the nc binary netcat
    which was used by the compromise code to
    reach out to the url which is
    clearly not a reverse shelter which then
    obviously received instructions and
    netcat was used to launch a bash and
    inside of that bash shell curl was used
    to now poke around and uh retrieve
    information for example from
    elasticsearch we can also see that we
    actually see the layer 7 information as
    well so we normally see the connection
    to elasticsearch we can also see that
    there was a get to slash users to slash
    search so we can even see what type of
    api call was being made
    and funnily enough this new what we see
    here is actually for the same traffic as
    here and then this network view would
    obviously never be able to tell whether
    any attack is ongoing right
    like it’s no network person would ever
    claim that you’re able to see this type
    of attack based only on the five two
    blackboard flow log and it becomes so
    obvious looking at this i’ve used an
    example of a compromised code part here
    with a reverse shell but this is
    actually also incredibly useful if we
    are if you are figuring out where is the
    network traffic coming from is this the
    actual workload is it an application
    developer running cube control exec
    running a bash and crawling around is it
    kubernetes network-based health checking
    and so on is like what what type of
    network traffic is my kubernetes control
    plane doing and so on this will work for
    literally any process on your machine
    we briefly mentioned that psyllium can
    provide alternate authorization not only
    on the network level which is here the
    layers to layer four for the networking
    folks under you but also on layer seven
    and actually for example limits that a
    front endpoint can talk to a back and
    part which can only do
    an http get to slash public but for
    example not do a post or an update or
    not access
    private or another url for example
    this same principle works for protocols
    like kafka cassandra mkhst the variety
    of auto protocols so you can for example
    also say my part can talk to a check
    cassandra database but it can only do a
    select on a particular table but you can
    never update it or you can say my pod
    can talk to a shared kafka cluster it
    can can consume on this topic but you
    cannot consume or produce on
    any any topics that other teams are
    the dns server policy is incredibly
    interesting for rendering or for
    modeling services that needs to reach
    outside of the cluster i’m using here as an example but
    another typical example would be your
    service reaching out to aws s3 cloud
    provider cloud provider bank databases
    are in particular interesting because
    they’re backed by hundreds and hundreds
    of ips which are constantly changing so
    how do you model them from a network
    policy perspective you clearly want to
    limit the egress ability of your
    services because if the code gets
    compromised you want to prevent data
    and this is what dns policies come into
    play what how do you do this if the ips
    the service needs to reach out to are
    constantly changing this is where you
    can write a dns policy like this and
    actually say i want my service to be
    able to reach out to startup my domain
    on io or it could be or some other domain
    right what sony will then do is actually
    hook into all of the dns traffic and
    proxy that which means that first of all
    it will not even allow any dns
    resolution that does not match this
    domain or this pattern which means it
    will get prevented it will get rejected
    on the dns level
    and then if it is allowed only the
    allowed set of ips will actually or only
    the returned ips will be allowed which
    means that even though aws s3 might be
    backed by a thousand ips only the four
    ips returned by the dns will actually be
    obviously this also prevents any dns
    based data exfiltration attempts as well
    because our proxy will
    kind of decapsulate all dns requests and
    not allowed to do to do data exploration
    by dns either unless you find some way
    to actually encode that into the dns
    name itself which is i guess possible
    but very very challenging to achieve any
    reasonable bandwidth and then you would
    still have to match the pattern of
    course that you are actually allowing
    all right the last piece here is the
    hubble or the observability part so as i
    mentioned this is i think probably the
    most significant part overall um hubble
    ui or hubble is the observability layer
    of psyllium and it’s using ebpf to
    transparently get observability data so
    without requiring any manipulation
    modification recompilation of your
    applications so you can deploy sony and
    hubble and you will get this
    observability data for your existing
    workloads as they are running
    you can see an overall kind of overview
    what services are running how are they
    connected you can even see all the way
    into individual layer seven calls you
    can configure this to do layer 34 only
    or layer seven as well and then in the
    lower section you can see the live view
    of what are the individual connections
    or requests that have been observed you
    can obviously filter here you can for
    example showing show me only
    the the requests that have been dropped
    or show me requests with latency bigger
    than this and so on
    all of this data
    can be viewed live or you can feed this
    data into timescape and actually build a
    a time machine so timescape is our click
    house based um mechanism to actually
    store this observability data over time
    and then you can go back and say hey
    give me this view but from two weeks ago
    in this particular time frame and so on
    all of this data like the hubble ui that
    we just looked at which is obviously all
    the way available through cli and api as
    well as all the metrics so the
    the prometheus-based metrics that we can
    look at them if we have time as well as
    our network policy editor
    which is able to visually or to
    visualize network policies and to
    graphically edit uh policies all of that
    can be either be done at a cluster level
    or cluster-wide scope are they
    multi-clusters phone or
    excuse me you can also tie this down to
    for example particular namespaces and
    say a particular team managing a
    namespace can view these diagrams and
    dashboards and all of these ui tools but
    only for the namespaces they own and you
    can tie this to single sign-on provider
    octa active directory i am whatever you
    want here to actually inherit existing
    role information you have and then
    configure all back rules very similar to
    how fibonacci does our back rules as
    the last piece here before we wrap up
    github’s network policy guardrails so a
    use case that comes up very very
    frequently now is that as teams scale up
    and use more and more kubernetes the
    desire is there for
    application teams to to inherit part of
    the responsibility for the fine-grained
    network policy
    but a security team still wanting to
    preserve some control over the network
    policies for example the network
    policies within a namespace on what pod
    should be able to talk model to what
    other part the network uh the
    application teams should be able to
    define these rules on their own but the
    application teams on their own should
    not be allowed to open up new
    applications to the outside internet or
    to expose them or to do cross-name space
    traffic this is where the guardrails
    come into play so you can essentially as
    a security team
    define a set of broadway rules for
    example app teams can define within
    namespace and everything else needs to
    be approved and then this implementation
    which is available for github and gitlab
    will automatically approve pr’s with
    network policies that are within the
    guardrails and for everything else uh a
    security motor’s a motor role needs to
    improve the pr before it can get merged
    this is available as well and allows the
    security team to partially delegate the
    creation of network policies as well
    obviously you can still even with this
    news namespace and global policies as
    well so even if you have this in place a
    security team can for example always
    write down policies at the globals
    global scale if you have opened up
    namespace level policies for your app
    scale and performance as i mentioned uh
    in the beginning is is a highly
    performant uh way of doing networking
    observability security and so on if
    you’re interested in this i will not go
    into super details here but if you are
    interested in this topic feel free to
    check out the psyllium block
    we have blogged extensively on both
    performance comparing psyllium data
    playing to other cni’s to service meshes
    as well as scalability how to achieve
    really big multi-cluster systems
    if that is topic of your interest
    and with that i will leave you with a
    couple of pointers before we
    um open it up for questions
    if you want to learn more about cilium
    if you have been if you are interested
    in this feel free to check out
    edpf is the place to go
    obviously if you want to engage with
    sodium try it out get started join the
    psyllium and ebpf slack that’s the best
    way to reach the development team to
    participate to ask questions and then if
    you want to run selenium enterprise
    concept context feel free to go to
    with that i think we have several
    minutes left for questions if there are
    any and if there are no questions i’m
    also happy to demo the
    metrics real quick
    exactly so we thanks a lot thomas is
    fantastic all of this technology is
    really really cool actually we are using
    it we are using psilium already in apuyo
    cloud which is our latest container
    service here in switzerland so we are
    it’s opening up many many different
    doors and possibilities for our service
    as well so it’s fantastic
    to learn more about it
    if there are any questions in the
    meantime if you would like to show
    something else of course please feel
    free to take this i want to make sure we
    have enough time for a question but i’m
    always happy to demo
    a bunch of fancy metrics as well
    of course
    let me see let me know once you can see
    the graphone dashboard i think should be
    coming up
    so we have a wide variety of metrics
    that we can show i’m showing kind of
    three dashboards here for example we can
    see on the network level how much is
    being forwarded here we can see that
    some network packets are being dropped
    like oh that’s probably a bad reason we
    were a bad sign so we could actually go
    in and see why was it dropped and we can
    see that we have a steady flow of
    network policy or policy related flux
    so that’s something that we would then
    for example want to look into hubble and
    see which parts are actually subject to
    we can see that we can observe the tcp
    layer so we can see that we are
    constantly missing certain tcp syn acts
    which is a clear signal that connections
    are for example timing out obviously
    this is a a demo app that is
    fully doing this
    you can though go down there and
    actually see a kind of a a lockdown of
    water parts that are subject to the
    policy deny so we can actually lock this
    down further and then go here to the dns
    dashboard which is probably the most
    favorite dashboard of everybody because
    dns is
    reason number one for any incident so we
    can see for example how many dns
    responses or how many dns fares are
    currently happening and for what reason
    and then what parts are subject to dns
    error so you can immediately see whether
    your app is subject to the less errors
    we can also see for example the top 10
    dns queries
    we have a dna an http dashboard here
    where we can see the number of http
    requests um the the latency of
    individual requests
    and so on so a ton
    that’s kind of the high level but then
    we can also go out into very detail for
    example tcp metrics and actually measure
    the smooth round trip time which is
    incredibly interesting we see here that
    from time to time
    the prometheus pawn has a very bad
    smooth round trip time which probably
    indicates that it gets
    throttle or
    rate limited you can go further and
    actually understand
    how much traffic involved parts are
    doing what is the total traffic
    pods are doing right which parts are
    responsible for for example network
    latencies going up
    and so on
    then last but not least we can go really
    low level and actually all the way look
    at for example network interface
    statistics and you can for example
    correlate that to whether certain
    virtual machines or certain nodes are
    overloaded so this is just a very small
    glimpse at kind of metrics that we can
    provide so we can go all the way of kind
    of high level http dns level that we’ll
    use from a service mesh all the way down
    into lower levels that you would only
    expect from a network device type and
    correlate all of this information
    together and this makes it makes it
    relatively simple to troubleshoot
    network issues
    very very quickly
    very impressive very very impressive so
    don’t see much activity in terms of
    questions on the chat as well
    so i think
    uh we are probably arriving at the end
    of this session
    just a quick reminder
    before we close this session down
    the next editions of our cnc meetup
    will be on may the 12th the next one
    there are still slots left so if you
    would like to present some
    technology some practice that you are
    using in your own environment please
    come and tell us we would love to learn
    from you this community is built by you
    so it’s very important that you
    then we have two more sessions this year
    planned on august 25th and november 24th
    so please don’t hesitate to let us know
    if you would like to participate
    this video will be available on
    uh starting tomorrow so
    please sign subscribe and uh share as
    well if you like if you would like to of
    course ask questions feel free to
    contact us and to contact as well the
    the speakers of today and thanks a lot
    for joining if you watch this on youtube
    thank you thanks a lot for watching
    until the end and
    first of all thanks a lot to our
    speakers to moritz melissa and thomas
    it’s been a fantastic evening a really
    really interesting session so thank you
    so much and
    see you next time at the next cnc
    computing meetup switzerland thank you
    and bye bye
    bye bye melissa bye maurice