CNCF Switzerland Meetup – Distroless Docker Images

present
there we go
now whatever you say will finish in
youtube
i’m kidding
so welcome if you are joining now uh to
this webinar more more people signing in
this is really cool thank you so much
for joining we are going to start in
about two minutes
with the first presentation
this is switzerland so we like to always
be very
right at the minute
of 4 p.m
central europe time
and that will be in about two seconds
there we go
it is 4 p.m in switzerland it is
what 8 a.m in colorado i think
yes there you go it’s 8am in colorado
and welcome to the cloud native
computing switzerland meetup uh it is
february 24 2022 my name is adrian
kosmacheski and i welcome you from the
city of zurich in switzerland and this
is yet another edition online of the
cloud native computing switzerland
meetup
we planned i know we plan to do this one
in person but the situation it is what
it is so we decided to go back to online
and i think you don’t mind i think it
will be in any case a fantastic event we
have today so i’m really really thrilled
uh to welcome our speakers and by the
way speaking about which a little bit
about the agenda for today uh we are
going to start with body sacred from
edgeless systems game behalf from
germany uh talking about confidential
computing with project marble run
so this is moritz who will start today
then we have melissa mckay from jfrog
directly from colorado in the united
states and this is one of the first time
i think it’s the first time we have a
speaker from overseas uh on this uh
event so it’s really really cool welcome
melissa she’s going to talk about this
roulette docker images and that will
happen at 1645 cet and we are going to
end uh this event with none other than
thomas graf cto of isovalent he’s going
to talk about psyllium and ebpf the
hottest new technology in town so it’s
really really cool thank you so much for
all of you to be here and we’re going to
end around
1805 which is more or less in about two
hours so just a quick reminder this is
being recorded as we speak
so basically the video will be later
available on our youtube channel on
vision.tv so please subscribe for
updates we publish new content there
very very often there’s a q a box uh
right at the bottom of your screen so
please use it please post your questions
to your speakers each one of the
speakers will basically decide whether
to answer the questions uh live at the
same time or
asynchronously at the end and so on so
basically it’s up to you uh guys to
decide how do you want to handle this
but of course everyone please abide by
the conference code of conduct we never
had an issue with this the swiss
community is fantastic very respectful
uh but there you go you can read it on
vision.ch we’ve got the code of conduct
we are running a devops in switzerland
survey right now until the end of march
we are basically very curious and we’ve
got a very very
nice amount of answers so far about how
companies in switzerland are adopting
devops so please use your qr code please
login to the surveymonkey
survey that is on the screen right now
and
why i’m saying this because there’s a
possibility for you but you have to be
located in switzerland sorry melissa
sorry maurice about that but you have to
be in switzerland because you can win uh
one of two prizes uh worth 200 swiss
francs each one of them being a voucher
for a puyo cloud which is our newest
container platform based on openshift 4
that we just released
a couple of months ago so please uh
thanks for providing us with your
information and you have the possibility
of winning one of these two vouchers the
other one is a voucher for digi tech
which is one of the big
e-commerce websites in switzerland so
you can buy whichever device you would
like to have for
your birthday
we’ve got more editions of the cnc meet
up planned for this year we’ve got
may august and november planned we still
have two slots available for may so we
already have a speaker for may
if you are interested to speak in any of
these events
please contact us on cnc-meetup.ch
this is a url that takes you directly to
the vision.ch website with a form the
only thing we need to know if you would
like to speak at our event is basically
who you are what would you like to talk
about and that’s it that’s basically all
the information we need i will close
very quickly
please follow us on youtube because we
have vision.tv for your follow us on
twitter on github on linkedin on
facebook on instagram whatever you want
please follow us it’s really really cool
we are publishing very good information
all the time so thank you so much enjoy
the presentation remember to fill the
survey if you can and if you have time
it only takes five minutes believe me
it’s very very short and
go back to the agenda today we’re gonna
start now with moritz i’m going to
unshare my screen the screen is yours
and
there we go
thank you adrian very great introduction
very powerful i like it
all right
um yeah hi uh i’m gonna give you
an introduction into
how we bring together confidential
computing and kubernetes
and especially with our project marburon
i guess most of you will be familiar
with the term kubernetes and all this
cloud native stuff so i’ll take a bit of
time and
explain the concept of confidential
computing so you know what i’m talking
about
so the world as we see it or cloud as we
see it um is a very dangerous place
there are a lot of
components people parties you have to
trust
there are so many potential threats and
and malicious actors and if you transfer
your your workloads your data to the
cloud you
essentially give up the protection of
your of your on-prem physical
separation
and expose your data to this to this
world and
the
general idea of conventional computing
is that you can create a safe space in
the cloud that you can securely
communicate with and interact with and
thereby protect all your good data and
workloads in the cloud
and
this sounds very promising and
of course people are very excited about
the possibilities of the new
types of
applications you can essentially build
with this and all those
additional
security and privacy protections
you can you can build with this
technology
and there are already some examples of
existing solutions that use confidential
computing
two of the probably most
prominent ones are the the signal
content discovery so the idea is that
signal can tell you who of your contacts
is already using signal without
revealing your contacts to to signal or
to any any other party or like
transferring the the contacts to a cloud
instance that where a potential
um
malicious actor could could steal them
so your your contacts stay with you and
they can offer the service
and on the other hand the eagercept
which is the german e-prescription
service that’s currently being built and
rolled out
there’s a talk from someone at the the
company that builds this service at a
conference we we did last week called
the open confidential computing
conference oc3
i will have a link to that on the last
slide so uh we’re also going to upload
those talks to youtube if if you’re
interested in this this kind of stuff um
i’ll um
point you to that it’s a very um very
interesting um ambitious project there
and confidential computing is not
something that just exists in theory
it’s
something um that the cloud providers
are actively investing in um
sorry to interrupt you i don’t think
you’re sharing your slides
i don’t share my slides i don’t think
you’re sharing your screen
try again
is it back
and now we can see your screen yes thank
you so much
didn’t you see the screen the whole time
uh we only saw you yes from the
beginning oh okay
um
oh that that’s that’s not not too bad i
guess um
i can show you the
the nice graphics uh i had about uh the
exactly as as one of the secure cloud
environment
absolutely as one of the as one of the
attendees is saying it was very
interesting even without the slides so
no worries about that yeah yeah i got
the i got the chat set up now as well so
i try to monitor for any funny questions
um
let’s see if that works out
yeah and some some some news articles
about convention computing i just talked
about and these are the two
services signal and the
intercept
yeah and when where i
left off was
we now see all those competition
computing
offerings popping up
at the cloud providers
we have a
very
vastly growing
ecosystem there of confidential
computing technology that you can
already use today to secure your
workloads and cloud
but this is just the
very high overviews i’d like to go a
little bit more in detail to explain
what this actually is and and how we can
achieve these
isolated secure environments in the
cloud
so
the most prominent example and one of
the first concepts of confidential
computing was from infill the the sjx
artworks so the general idea was that
you typically cloud stack you have this
hardware and you have your operating
system hypervisor all this
orchestrational
layer
and finally you deploy your application
and without anything in place you have
to trust all those layers and all the
the people and the companies that have
access to these different layers
so now with confidential computing and
especially with sjx enclaves the idea
was that you
add something to the hardware in this
case a instruction set extension that
would allow you to create isolated
processes
in on your top layer
and those would be
those would contain your data and your
your data and your code
[Music]
would be
measured when the process is created and
would be verified that the
that you have integrity there and then
would be executed and the processor and
the hardware would make
make sure that no other layer here could
access this code and data
so a couple of features of
that we see in enclaves in general and
that were that came come with sgx is
first of all you have isolation so as i
said no other layer can access
codedata that is executed inside such a
secure enclave
and you also have runtime memory
encryption so at runtime even if you
would physically have access to the main
memory of the process you would not be
able to to steal any code and data
and to make this useful one of the most
important aspects is the remote
attestation so from inside this enclave
you could prove to the outside world
that you are
running inside such a protected
environment you can also do this from
the outside so you can also do
uh attestation um
from from from that you can also from
the outside a test at something runs
inside an output
and from there on then you can build
your your applications and systems and
you can pry to your users the other
components that you are secure and data
can be shared with you secure securely
and finally they have a mean of sealing
state so you could start and stop the
process and store data for example to
disk and later on unseal the stator
only if you are running the same enclave
with the same identity again
so you don’t have to worry that
your persistent data and the state of
the application might be might be leaked
later on
this
process-based enclaves or these sgx
enclaves have
some some limitations they have a
um
system interface
so
it says no syscalls but you could
essentially it has a limited set of
systems you can do uh that are specific
to to to sgx but in general they don’t
have access to the to the system’s cisco
interface every time you do a cisco you
would leave the enclave process the
syscall in the operating system and then
jump back into the enclave
and this has of course a lot of um
downsides you can’t just lift and shift
your your code and data you have to
specifically um repackage rebuild your
code run in those process based on
things
and one development we see very recently
is that
to allow for real lift and shift and to
not have to repackage rebuild things
uh a new combination computing
technology called the or
yeah um
to giving them the confidential virtual
machines short cvms
that not only run
single processes inside secure onbase
but running a whole virtual machine
including the operating system inside
such an isolated environment so you have
the same idea the hardware provides
let’s say the hypervisor which is not
trusted to create such a secure virtual
machine and load your image securely
measure it and then you can execute
whatever you want there and this gives
you the ability to essentially do real
lift and shift and you have the same
type of features of isolation runtime
memory encryption and remote adaptation
so to compare those technologies uh we
have
let’s say um
we typically
differentiate them
on something we call the trusted
computing base which is all the
components you have to trust inside your
your trusted environment
if you use sjx for example to just
yeah
protect individual individual functions
so you split your application and only
individual function calls enter a secure
enclave
this is the smallest you can go and
requires highly adjusted applications
where you specifically develop towards
this
interface the the second thing then that
somewhere in the middle is where you
package whole applications inside
process-based entries um
and
this means that all your code is running
into the enclave so slightly larger
trusting trusted computing base
still some repackaging probably
recognition and this kind of stuff
needed and then the confidential virtual
machines on the right side where you
have to trust everything that runs
inside this virtual machine the whole
operating system that also has a larger
attack surface from below of course
because now you assume the hypervisor or
anything that runs there can
attack your kernel from below which is
of course
a lot of things that could go wrong
there and then you have all those um
components that run there that you have
to trust
and examples of these would be amd suv
info tdx
or
something that’s got a
theory
happening soon is called arm realms
and the idea here is um
that you have lift and shift
functionality for for existing workloads
something that’s somewhere in between
those two worlds is the aws nitro
enclaves
which
are
running entire
virtual machines essentially but
their packaging mechanism and and their
feature set is also slightly related
more to the confidential containers if
you will so it’s it’s hard to to really
fit that into this picture here but just
to mention it um
somewhere somewhere in between
and we build open source software to
make these technologies accessible to so
you can build um workloads you can
deploy stuff to the cloud that utilizes
these hardware technologies to protect
and isolate um for your deployments and
your data
and we have similar concepts in our
products uh we have three tools that are
intended for these process-based
solutions especially s into sgx
and and we see
the fit here if you want to create
new types of
solution new type of applications
that
build on the concept of comprehensive
computing that maybe
implement a use case that’s only
possible in such secure
environments like multi-part computation
um
ai training on on uh pi data so personal
identifying from identifiable
information uh data
and these kind of things and then on the
right side we have a tool that we just
released last week that is
using confidential vms to create entire
kubernetes clusters
inside such execution environments so
you can think of
running a kubernetes cluster in the
public cloud
while being completely isolated and
don’t have any um don’t have to trust
any part of the public cloud of the
cloud provider
that has no access to your cluster
whatsoever
so if you want to lift and shift
this this tool is intended um for this
use case
for this talk i’m going to focus more on
the left side on
how you can
develop cloud native applications maybe
uh repackage cloud native applications
to run inside process place enclaves or
you can orchestrate and scale them and
then finally integrate them together in
kubernetes to to build scalable
confidential applications
the first tool i’m just going to mention
here briefly is called ego
when we started we noticed there’s no
real there is no go run time
for
confidential computing
alpha in class gx so there so what we
did we
basically slightly modified the gold
compiler so it doesn’t do
syscalls but instead goes into a
capability library that handles
the syscall layer so it can run inside
secure enclaves and then we try to make
the simplest tooling possible
for for you to interact with and package
and build your applications so it feels
like just any other go development
process without going into the troubles
of understanding how to build and
package into sjx enclaves which is
or can be quite troublesome
so we have three simple commands you you
build as you would do to go compiler
that builds a blue binary and then we
sign it because every uh sdgx ipad has
to be signed and this step also packages
your whole ipad and then you can just
run similar to how you would do go run
and it will load your enclave and run
your go code inside there
and yeah um this is for us it feels very
natural to our usual co-development
workflow and and we use this for for
bringing
go applications and most of native stuff
is written in go so we can bring most
cloud native stuff um
to
those secure enclaves so we can already
isolate and protect our individual
workloads which is
which is cool we have an example of
where we for example you um
take hashicobalt the
the key values or of hashicorp that’s
often used to distribute secrets for
example
to run inside a secure enclave
and
when we
had that
we
have protected our own original
workloads
of course there exists other tooling for
other languages there exists some
library assets that can more or less run
a variety of different languages and
different
types of applications so we had
we built the go part and then we had all
the we could lift and shift all the most
of the workloads in the cloud that we
needed to
into secure enclaves
and then the second problem arises
and
the question then was okay now we have
individual workloads how can we bring
them together
for a modern cloud native architecture
how can we do a microservice system
and connect those individual services
securely so we can distribute and share
data between those services while
everything is end-to-end security and
confidential and how can we
orchestrate that as we would usually do
with kubernetes while everything inside
kubernetes all the kubernetes components
our nodes uh our control plane they’re
all not trusted because they’re either
completely controlled by the cloud
provider or running inside cloud
provider control infrastructure
and therefore
we came up with a project called marble
run
so the idea is that we can
use the existing orchestration tooling
of kubernetes
and we add
a trusted party to that that runs
a let’s say a trusted control plane that
takes care of all those orchestrational
tasks that you are have with
uh
distributed confidential applications
so
a confidential deployment for us means
of course
individual protection of our workflow
that means runtime encryption of data
and then we need to
somehow be able to verify the topology
and functionality of our application we
need to take make sure that
any part of the infrastructure of the
existing orchestrational
layer is is is not trusted and can’t
access our
our protective workloads so all the
administrators
the the cloud provider and and any
anyone that has access to the call
infrastructure
either intentional or through some some
malicious act
so the concrete questions we solve with
mamaron are how we can attest
the integrity and confidentiality of a
one container or one part if you will to
another
so if one service wants to share data
with another how can it make sure that
the other one is also running in a
secure entry
and then how can we
verify this entire deployment that might
scale up and down the whole time
uh as one concise application from the
outside
so clients administrators can verify
this before interacting with it
and one very specifically technical
thing was we wanted to provide some some
form of mutual tls so you can
essentially have something like tls
offloading that is often done by um
service mesh
solutions
but we can’t use the existing solutions
because they don’t terminate their tls
connections inside the actual
application inside the
container they terminate inside the site
car and then go through localhost
and we can trust localhost because it’s
called infrastructure so we need to
terminate our connections inside the
application inside the actual arm plate
and then yeah some orchestrational tasks
like how can we distribute secrets
kubernetes is not trusted when it’s
control plane and etcd everything is not
trusted you can’t use the usual way of
configuration and and secrets management
there and um you also want to make this
whole thing updatable scalable all this
kind of stuff
so in in modern run what it would look
like from a workflow workflow
perspective
you would install your usual
orchestration of little things have
various cluster maybe installer surface
mesh like linker d
and then you would install mile run it
would stall the mobile and control plane
which we typically you call the
coordinator so if you hear me saying
coordinator this is the control plane
control plane piece it runs inside an sg
example so it’s protected itself
and it can be verified from the outside
so a devops engineer can verify it
and provide it with a
configuration and this is and this
configuration we call a manifest
that contains how
our confidential application
uh should look like similar to how you
write your yaml files for kubernetes you
would describe
um
how your your individual archways can be
identified what their measurements are
how they should be configured what
secrets they should
be
provided you can define some some
role-based authentication things in
there some
secrets some mutual tls configurations
this kind of stuff so you define it
there
and then you would install your
application and your application would
be built with
ego
with one of the other
enclave runtimes for your workload
and they would come up boot up and then
they have a small
yeah
a small shim layer that connects to to
marble run and is authenticated
according to the manifest that you
specified earlier and is then provided
with secrets configurations
and
tls credentials and these tls
credentials can then be used to connect
those services together
so they are used in those mutual tls
connections to the other service
and they have a root of trust in a root
certificate provided by our bomb run
control plane
that then can always be authentic uh
used to authenticate other other
services in your
in your service mesh
so now we have a fully
connected
application everything is running
smoothly and then a user can go ahead
and connect
to
your front-end service for example
and similarly the user can connect to
the maverian
coordinator
obtain another station statement that
contains a um a
hardware-based
for example sgx at the station
statement that can be verified and then
also contains this route of trust this
root certificate that is then be used to
verify every connection
or potentially verify every connection
to the application so you set this this
root certificate and then connect your
front end
and verify the the tls last connection
by doing so
so this is the general workflow of how
margarine takes care of orchestrating so
if you would now for example scale up
another container pops up it connects to
margarine it’s authenticated provided
with the credentials to join this
mesh together set of confidential
services and then
just works as usual and from the outside
you can always verify the entire
deployment using the mamaron coordinator
so putting this all together
with with ego
and of course the the other enclave
runtimes with maveron and something i
haven’t mentioned at all today uh called
atlas db is essentially a database that
runs entirely inside secure enclaves
and has some enhancement to enhancements
to its um to its storage encryption
so it’s more or less
a
mariah db front and
coupled
with a rockst modified slightly modified
rockstd backend
for a end-to-end secure database so we
use this for for a if you need a storage
solution that needs to be confidential
computing graded
and then we package together this with
another run and you can build an entire
end-to-end confidential native apps
and in most cases no code change is
required if you’re using ego
or library as runtimes
and one example of this is
the open service mesh bookstore demo of
service smash is a
server smash from the i think it’s a
cloud um
it’s
part of the cncf uh now uh
cncf project and they have a demo
application called bookstore and we
confidentialized it using those three
components
if you want to
take a look
we have a blog post on how we did that
so please
please check it out
i have a
smaller demo that’s not that complex and
can be quickly
quickly shown prepared
let me switch to the console
so
i have an a cluster in
in in error prepared that is able to run
sjx enclaves we can verify this by
running cube cover
uh first of all we have uh
three nodes
so let me describe
one of those notes
you would see
a lot of things happening here but you
would see this node
exposing
three resources regarding to influence
jx so this this node is equipped with
the latest sdx hardware and has the
capabilities to run
sjx
clips in there
and
there’s not much running so far
there are some some sg device driver
specific things but so far it’s a
vanilla uh
aks edgar aka aks cluster and i can
install mother on there doing mobber on
install
that goes very quickly and installs the
control plane
and we can wait for this to finish
running check
and um
the application we’re going to install
here is a very simple demo application
from the linker d service mesh
that
provides a emoji voting platform that is
three services
one front-end services a front-end
service with a web interface and then
something that counts
the votes
so more or less a stateful service and
one service that provides some
application logic in this case it
provides emojis
so when modern is ready
we should be able to configure
it in the meantime there’s a question
for you in the q and a box i don’t see
i’m sorry
no it’s okay
while we wait maybe you can answer
there’s a question
can you see the q a box
[Music]
one sec
that’s the chat open now i need to go
back to
but i can read it for i can read it to
you if you want oh yeah please go ahead
so uh jan bosniak is asking can you
share some performance benchmarks i’d be
curious where would be
most overhead if any whether it’s
enclave startup or per cisco slash ego
or call
well that’s that’s a good question um
they’re
especially with process-based enclaves
there there’s
can be quite some overhead
the
the enclave startup is not too bad it
i can’t give you exact numbers on that
to be honest but
the the bigger performance impact is
definitely if you have a lot of calls o
calls means you’re right you do assist
call and
your encourage have to has to exit
the secure environment and then
return back
um later on and handle this is called in
the operating system and return back so
for example if you do a lot of io i
always handle by the system jankv can do
this on the on its own uh even though
there are
people implementing switchless sockets
where you don’t have to to to access the
active contacts but nevertheless if you
have a lot of those your performance
impact is going to be bigger
um
and this
with the latest iteration of
sgx with the isolate machines
they were able to reduce this overhead
by by quite a lot so you’re usually
able to go below
um
so go into single digit percentage
overhead
but
it’s it’s harder to say this in a
general manner it’s usually depending on
your type of application but i would say
you can make it happen to to to
definitely go into the single digits
and we had a
we had a implemented in a an ai pipeline
eden
where
we were able to comparing across
cpu
inference and training compared
also cpu only um on intel comparing
enclave to
non-enclave and this was almost
negligible the overheads so you’re close
to two to no overhead in there
um obviously it’s less io than some
other application maybe
let me see
what’s going on here
all right
maybe it just took too long
so now i can expose the
mavaron coordinator uh api
to localhost just uh to simplify this of
course you can also expose it externally
but for the sake of the demo just
copy this command let’s just pour
forward this service to my to my local
host so i can quickly access it
and
then i can obtain the
mamaron root certificate so this root of
trust
and the the marron seal i am using here
is automatically verifying the integrity
of my control plane you can also do this
by hand or verify
the
oops
the the given attestation statements by
hand but the dci does this for me
automatically
okay and let’s save the certificate
somewhere here
all right and now i can use this
certificate to authenticate all other um
connections to either
um to any of the of these services in my
in my cluster
so this is all i need to do to to set up
marvel run so i can now um go ahead and
configure
my deployment and i have a manifest
laying around
for the uh
emoji photo voting
application let’s quickly take a look
it’s in json format it defines some
packages
think of it as the
the containers of your application it
contains the measurement of the of the
armpits that run inside those containers
and
next to packages let me scroll down a
bit i’m gonna skip infrastructure is
marbles and it’s called marble run for
reason so these marbles are
more or less mapping to the pots inside
your kubernetes cluster and each
marble essentially has a package think
of it as like the container the enclave
and then
the parameters that this
package should receive and this could
anything be anything like environment
files
anything like that
and in this case we also define a tls
tag for some
manual mutual t-less configuration here
but they would automatically provide a
rhythm with an automatic mutual tlast
tag if i wouldn’t
define it here
i have a
let’s skip
clients and and roles there are no roles
or stuff defined i could define roles
like somebody who’s able to update
certain packages somebody who is able to
recover uh
if
if this whole thing
for some reason uh needs to be turned um
restarted or so
but let’s keep all that there is a
secret defined
and the secret in this case
is
a certificate that should be signed by
the marlborough root certificate and
this secret is then later on used in my
tls configuration a little bit
down below where i would say all
incoming traffic from my web front and
should use this certificate to to to
show to the clients
and then i say okay um my emoji which is
the
application logic
service here should accept incoming
traffic on using tls on port 8080 same
for the voting service and my web
service
connects to those two servers so it has
outgoing connections configured in here
you will find this
in our github repository um we have a
the multivote app has a
there was a github repository so if
you’re interested take a look um
to have a bit more time to understand
everything that’s that’s listed in there
but i can deploy this manifest simply
using the c line
and then i can on any time at any point
in time i can verify that this exact
manifest is deployed
is set for this deployment so any party
could do
maveron verify there’s an api for that
so you can of course use the
use the the command line interface but
any client can use just the api to do
the same operation verify that the
manifest
is
that is deployed remotely in my cluster
is the one i have locally in here
and of course i just set it so it should
be okay
and from that point on i’m done
configuring my run
i can install my actual application
i’m using helm in this case that
installs the emoji boto app
and then i can wait for the whole thing
to come up
might take a bit
and then if this is done i’ll be able to
access the web front end and i’m gonna
do a local host port forward again
because i haven’t set up
a public ip address for this
aks classroom
all right let’s see
are there more questions in the meantime
last one is coming up so let me prepare
the port forwarding
so let’s say i do
maybe 80
on localhost
and then go to my
web browser
oh
[Music]
oh no
this is running my inside my vm so i
need of course need to
need to expose it on
all right
and will show me a certificate error
because my browser doesn’t know the
mamaron root certificate
i could of course
um
configure it so it’s uh so it will has
this as a word certificate and then my
browser would would know that um could
verify this
on this application certificate and then
i’m connected and i can use the
application as usual
all right
so in conclusion um
cc makes
systems empty and secure and verifiable
creates those secure environments in the
cloud so you can use
you can you can shift your workloads in
there without
the the risk of
of leaking it to to any malicious
party or potential
actor in the in the cloud
if you’re interested in those tools i
just showed you
go to our website or our github you find
everything on
every all the links in there you will
find the tools you will find the demo
i’ve just shown you
if you’re interested in the topic in in
general
we had a as i said a conference last
week
with
people from the whole industry with a
keynote from the other cto and
everything from this conference will pop
up on our youtube
i think you will find the link either on
our website or you go to youtube and
search for oc3
and yeah
i also gave a talk
similar to this one where i also
introduced the
constellation project product we
released last week that had um that
allows you for this true lift and shift
if you don’t want to build new
applications or have the small tcp as
i’ve just shown you right now but just
want to lift and shift everything into a
private environment about or in a
private kubernetes definitely definitely
check out this one
and yeah that’s it
thank you very much
excellent
thank you so much moritz it’s been
really interesting um we are going to go
directly to the next stop right now uh
if please remain online if you would
like to ask more questions we’re going
to have a session right after uh the
last talk so basically if you would like
you can also post them on the q a box
and i will redirect them directly to
moritz in the meantime but uh without
transition we jump directly to melissa
thanks a lot moritz and melissa the
stage is yours
right
glad to be here everybody i’m so glad to
um i know we’ve been in this pandemic
now for the last two years but one of
the advantages is being able to
go to switzerland without actually going
to switzerland
but
one of these days i would love to visit
switzerland um but i am currently
in denver colorado
and um enjoying
the opportunities for virtual events
here for now
i’m gonna go ahead and share my screen
okay so today i’m going to be talking to
you a little bit about distroless images
and securing your docker images
i’m not getting into
super detailed information this is
pretty high level i think it’s very easy
to use containers and especially with
this particular audience this is
something you do every day but it’s easy
to get caught up in just using them
without
thinking about what’s going on under the
covers so in the build up to talking
about distrolus containers i want to
just talk about containers in general
and what’s going on under the covers
this is kind of a modification of a talk
that i’ve done in the past there is a
lot of discussion of docker in this talk
but just understand this isn’t a a deep
dive or a tutorial on docker or anything
there’s lots of good documentation
already on that so i just wanted to do
something a little bit different and
maybe go back and look at the history of
containers a little bit and how we got
here
all right a little bit about where i
come from i am melissa mckay i am
currently a developer advocate with
jfrog but for over 20 years prior i have
been a software developer and engineer
in various roles in various companies
primarily my
my language is java i worked on java
server side services but i rarely meet a
java developer these days that aren’t
involved in other types of services and
languages especially when we started you
know going toward microservices and
getting smaller services out there up
and running so i definitely have
experience in others as well um my most
recent experience in software
development prior to becoming developed
becoming a developer advocate
kind of pushed me into some new
territory this was while i was working
on some teams
practicing devops and i did get the
privilege to learn more about how the
apps i was working on were actually
deployed and i got an exposure to a
variety of tools in the devops ecosystem
and one of those tools was docker of
course and that’s actually how this talk
came about originally i just started
collecting information from my own
experience learning getting ramped up
and on how to containerize my apps and
some of that was trial and error so um
i have some funny stories about that
um here’s my contact information you can
always you know find me on twitter of
course
um i’ll go ahead these slides will be
made available but i’ll go ahead and put
my
twitter handle in the chat so you have
it
and i recently became a docker champion
or a docker captain and a java champion
at jfocus a couple years ago so that was
that was pretty exciting
[Music]
okay um jfrog loves to do uh loves to
give away stuff so i can’t skip this um
you can go to this link i will go ahead
and put that in the chat as well
and jfrog will give you a chance to
enter and win a pair of airpod pros so
you can enter with the qr code or the
bitly link of course and um
um actually does it yeah well i guess
that’s a fiddly link and then a winner
will be selected within three business
days and they will contact the winner
via email and once the winner accepts
the prize they’ll share the winner’s
name in the meetup group with the
community
so don’t miss that exciting
okay today’s agenda i’ll go through
container history
i’ll go through
a little bit about containers in real
life um we’ll review some of the
container gotchas a lot of you this
might be review but
sometimes we need to be reminded of
these things and then then i’ll get into
dystrolis images and what those are
about
okay so let’s jump in and start learning
about containers
i i know that’s not the graphic you were
expecting but
i made a commitment to myself not to use
any
any
shipping container pictures and any
talks about containers ever again i
think i’ve seen those too many times so
this theme is bananas i hope you enjoy
it
and
i just want to talk more about how our
industry has adapted to dealing with
resources over time
and when i put these slides together i
was reminded of a story that my
grandfather would tell me when i was
growing up
he grew up in some pretty hard times and
depression era in the u.s and um
when he was a kid they would be gifted a
banana at christmas
and this was probably in the 20s or 30s
so quite a while ago
but bananas were such a rare treat and
my grandmother’s family or my
grandfather’s family they would get this
banana they would scrape the peel with a
fork to get every last bit they could
off and um that might not be the most
solid analogy but i
think of that story when i think about
how computing resources were in the
1960s and 70s very limited very
expensive
it took a lot to get anything done and
often a computer would be dedicated for
long periods of time for a single task
for a single user
so these
these limits on time and resources
created bottlenecks and inefficiencies
and just being able to share with each
other wasn’t quite enough we needed a
method to share without getting in each
other’s way
or having a person inadvertently cause
an entire system to crash for everyone
so this required a need for better
strategies and sharing compute resources
and that actually started a massive
path of innovation that we see a lot of
benefits from today
and there are some key points in time
that
i’m going to bring up there’s a lot of
other things that happen of course but
these are these are some highlights
that brought us to the state we are
today with containers and i’m just going
to start with tarot
change route
this was born in 1979 during the
development of the seventh edition of
unix and it was added to the berkeley
software distribution in 1982
this ability to change the apparent root
directory for a process and its children
resulted in a bit of isolation in order
to provide an environment for testing a
different distribution for example and
that was the thought when this tool
initially was entered into the source
code that it would be used for testing
purposes
that uh true was a great idea solved
some specific problems but more was
needed of course
and the jail command was introduced by
freebsd in 2000 which is more
sophisticated than to root and it has
additional features to help further
isolate file systems users and networks
and it had the ability to assign an ip
address to each jail
moving on in 2004 we came across solaris
zones those brought us ahead even
further by giving an application full
user process and file system space and
access to system hardware
and solaris owns also
not introduced but highlighted the idea
of being able to snapshot a file system
which becomes pretty important when
you’re you’re running containers
and then in 2006 google jumped in with
their process containers which are later
renamed c groups or control groups those
are centered around isolating and
limiting the resource usage of a process
moving right along in 2008 c groups were
merged into the linux kernel and that
along with linux namespaces led to ibm’s
development of linux containers
then we get to 2013 this is a big year
docker came on the scene with bringing
their ability to package containers and
move them from one environment to
another that same year google open
sourced their let me container that for
you project which provided applications
the ability to create and manage their
own sub containers and from here we saw
the use of containers and docker
particularly explode
in 2014 docker chose to swap out their
use of that lxe toolset for launching
containers with lib container in order
to utilize a native golang solution
when i first realized docker was written
and go i was surprised i really didn’t
have any idea so that was that was an
interesting bit of trivia
skipping over some details
there’s a a lot of different projects
that were started organizations and
specs that came out but there was one
event
that is really important in june 2015.
this is important to know about because
it gives you some insight into some of
the activity and motivations behind
shifts in the market as
with regard to containers
the open container initiative was
established
and this is an organization under the
linux foundation that includes members
from many stakeholders including docker
with the goal of creating open standards
for container run times and image
specification
and while all of this is happening in
the container world there’s a couple of
other dates that are going to be
important especially to java dubs and
being selfish and i’m going to tell you
some java stuff a little bit here a bit
of trivia for some of you um
java 7 was released
in july of 2011 and work was started on
java 8 which was released in march of
2014. so when you start containerizing
java applications specifically this a
little bit of history is important to
know because um you know there’s some
issues there with this development of
docker alongside development of java
there were some misses and disconnects
there that were fixed much later
all right uh what exactly is a container
um let’s just take a quick look at a
real life container itself what it
actually looks like on your system this
is something i did first thing when i
started learning about containers in
general
i’ll just show you how it’s stored
what’s actually happening under the
covers
this is probably more important for
those who deploy maybe containers you
know by themselves not necessarily using
orchestration but still good to know
there’s some there’s some things that
can happen if you aren’t paying
attention to how the file system is
being used
[Music]
my first experience
i was a developer on a new project with
a title deadline of course that’s
usually how it is i could argue that
that’s how most projects begin and the
best course of action for me was just to
jump in and start getting something up
and running on my machine and very
hands-on i wanna i wanna do it and see
what happens
and the docker documentation actually is
really good i have a link here on this
slide to that
easy to find of course so if you find
yourself in a similar position which i’m
sure many that’s in the past for most of
you
but i definitely recommend going to
their getting started docs which is
listed here
and um
the first thing to note is just that a
docker image is really just a tarball of
a complete file system and when an image
is unpacked it’s thrown into its own
directory which becomes its root file
system
the second
is that
processes that are involved in running
those containers are just regular linux
processes
on top of that there’s just a few linux
features that are used together in a way
to achieve that isolation that we expect
from containers
namespaces
are important
this is what is used to provide virtual
separation between these containers this
is how the process is inside one
container don’t interfere with the host
or with processes inside another
container
and here
you can see the name spaces that were
set up for a
postgres container that i have on my
machine running not today i turned it
off today to clean it up for you guys
but um i i just pulled some screenshots
of and i’m on a mac by the way uh so
you’ll see some some mac commands as
well
um the c groups functionality also
essential in constraining how much a
container can use things like cpu memory
network bandwidth etc and i can set
these constraints by including options
on the docker run command when launching
an image and of course these can be set
using orchestration as well um you’re
not necessarily stuck with the docker
runtime either but this is for
development on a local box
um here you can see i’ve i you know
started launched a container and i
constrained the
memory limit of my containers running
and then just some more details on the
file system where containers and images
are actually stored first off after
you’ve installed docker running the
command docker info will spit out a ton
of information about your installation
using the docker root directory and this
is where most everything you’re going to
care about regarding your doctor images
and containers will be stored
if you’re on a mac of course these are
actually running in a tiny vm so you’re
going to need to use some utility i used
to use screen but it’s it’s not real
consistent and available
so uh it’s not working for like later
versions um so in order to
get into this directory to just kind of
look around and see what’s going on and
i have a
command here a docker run command to
start another container that actually
enters um the uts namespace the network
namespace and the ipc namespace and it
targets that pit of one so that you can
get some information about this um
root directory
it’s pretty interesting just to navigate
around in there and see how that how it
changes when you launch containers and
and remove images and things like that
later version of windows i actually have
a windows box on order it’s been years
since i’ve been on windows
i swore i’d never go back but
the latest wsl
stuff that they have on the on
i don’t think it’s windows 10 and up
they have the capability of running
linux containers natively using wsl the
windows subsystem for linux
that there’s a path here where i did
find where the root directory can be can
be found on a windows system
so i’m excited to get that that machine
up and running um it’ll be a while
before it gets delivered here but
it’ll be cool
all right uh this slide shows just some
information about uh where the images
are stored on your system um
i have
my available images i just use the
docker images command to see them i have
several installed um
i just showed a couple here
uh
i can inspect any image by using its
image id and this spits out a ton of
information but what i want to highlight
here is the graph driver section which
that contains a bunch of paths to
directories where all of the layers of
these image
live
so doctor images are composed of layers
most of you know this already those
represent
or not
they correlate to instructions in the
original docker file that was used to
build
the images originally and these
translate in direct to directories and
these layers can be shared across the
images in order to save space
the lower der mergeder and upper der
sections are important the lower
directory contains all of the
directories or layers that were used to
build the original image these are read
only
the upper dir directory contains all of
the content that has been modified while
this container is running
and if modifications are needed for a
read-only layer
in lower dir
then that layer is copied into the upper
dir where it can be written to this is
called a copy on write operation
and it’s really important to remember
always the data in the upper dir is
ephemeral data it only lives as long as
the container lives so if you have data
that you want to keep intact obviously
you need to use mounting or volume
features and um to mount a separate
location or a separate drive
or
to access a database that kind of thing
lastly the mergeder is kind of like a
virtual directory it combines everything
from lower dura and upper dir and this
way this is how the union file system
works
any edited layers that were copied into
upper dir will overlay the layers in the
lower dir so if you were to exec inside
that container that’s what you would see
[Music]
this is just an example of a few
containers that are running on a system
i’ve got a jfrog container registry
installation that’s pretty old by the
way i don’t think we have that available
anymore but it includes a container for
artifactory and a container for a
postgres database and the other is just
a simple test container i was playing
around with
the container ids of the running
containers match up with the container
subdirectory names that’s something to
remember
if you stop a container that
corresponding directory doesn’t
automatically go away until that
container is actually removed with
docker the docker rm command
so if you have stopped containers lying
around especially on your local system
and that never get cleaned up or even in
your test systems you might see
available space start to dwindle so take
care that you’re cleaning up after
yourself there is a docker system prune
command i use that regularly when i do
docker workshops just because i want to
start fresh
you can run that to help clean things up
remove all images remove containers that
kind of stuff or you can launch a flag
to indicate that a
container should be removed when it’s
finished running
obviously this stuff is also taken care
of in orchestration frameworks as well
let’s make sure to use them
so these tool sets around building and
running all of these images have made
things so easy it’s also easy to shoot
yourself in the foot in a few places it
can be deceptively simple for especially
people just learning and starting so i’m
just going to go over some of the most
common gotchas pretty quickly
i’ll include some jvm specific gotchas
as well that i ran into immediately when
i first started working with containers
the first of course is running as root
i’ll be honest i was initially when i
initially started i was so excited um i
just didn’t really take this one
seriously enough
but now that
you know that processes inside a running
container are just like any other
process on the system it’s scary now to
run as root inside a container and doing
that opens up the possibility of a
process escaping the intended confines
of the container and gaining access to
host resources
reducing that attack surface of your
container by following the principle of
least privilege
that’s important and although containers
are designed not to affect other running
containers they will if someone
gains access to your container
and immediately has root privileges they
can wreak havoc on your host as well
so how do we mitigate this problem the
best thing to do is when you’re building
images
initially make sure that you are
creating a new user uh you can use that
user command inside the docker file when
the container is built in order to run
processes as that user
and you can also specify user just when
running the docker run command but that
leaves open the possibility of
forgetting to do that so it’s nice
if it’s just built in and your
production image is set up by default
not to run as root many companies by now
have set up policies and stuff to
prevent this from happening so um but i
i have been on teams where this was not
paid enough attention to
also pay attention to these official
images that you get from docker hub you
need to know whether or not they run as
root or if you need to you know build
another image on top of those
to fix that
even though docker provides you with the
ability to set resource limits on your
container it doesn’t automatically do it
for you the default settings are free
for all with no limits
so make sure you understand the resource
needs of your application
too little and your container will die
from starvation of course too much and
the container could smother others on
the system the resource usage of your
containers that’s something that you
want to monitor over time and adjust as
needed and it’s good to determine if
something is going wrong or if load on
your system has changed for some reason
you can use the monitoring information
to determine if your application needs
to be split up further with certain
action scale differently i went into a
whole process with that on a team of
mine where we divided up our application
into smaller pieces and we’re able to
scale like our front end very
differently than the back end services
and then never updating of course this
is a huge
security issue and it’s easy to get
complacent not pay attention to what’s
actually being pulled when you build
images
not only do you need to be aware of
outdated versions that you specify in
your docker file but you need to pay
attention to what’s in the base image
that it’s that it’s coming from um
not updating these packages and
libraries can you know lead to some
pretty embarrassing results especially
when there are tools available now um
all over the place now to alert you when
security issues have been discovered
with specific artifacts and i’ll show
you an example of that in a bit
um even you know even if you’ve gone so
far you’ve ensured your containers are
running with a non-privileged user
there’s still risk if there’s known
vulnerabilities that exist within your
container or even on the kernel of the
host so from time to time exploits are
found
those enable attackers to escape a
container so keep up with those security
updates
um
this is a very java specific one
this has to do with that
in tandem development of java and docker
containers
um pretty specific to job applications
but still important you never know when
you need to containerize one of those
the jvm is pretty clever at
automatically determining settings for
your swap and heap and your garbage
collection behavior based on things like
the memory and the number of cores
available on the host
and remember earlier in our lesson when
i mentioned the dates for java 7 and
java 8.
considering the timeline of
docker and java releases java 7 and
versions of 8 and certainly earlier
are not fully container aware meaning
that your java application won’t
necessarily obey
those memory and cpu constraints that
you put on your container and you could
end up with some surprise out of memory
killer activity which is exactly what
happened to me
this means
um your
you know your java application is not
going to
behave like you expect
the reason for this is the mechanisms in
the that the jbm used in these older
versions to retrieve the resources
those actually were coming from the host
machine and not the control group limits
uh where you would expect them to come
from
there were some improvements around
container awareness that were introduced
in java 8 update 131
and there’s further improvements in
later versions but to really get all of
the benefits of container awareness you
really should at least get to java 11.
that’s an lts release uh latest of
course that’s come out is 17 so if you
keep up with those that’d be great
[Music]
um and then image bloat this is a big
one um
i learned quickly about the
um
in the docker ignore file so when you’re
when you’re building
a docker image
um oftentimes
you may hit you know a dot at the end of
this command the build command which
indicates yeah this is where my docker
file is this is the context meaning this
is where i am in the file system
structure that i want to send
to the docker daemon to build this image
well in doing that you may be sending
over stuff and um if you use one of
these global copy commands in a docker
file you could be copying things in like
your dot git directory which probably
isn’t you know something that you want
to have in a production image
and i often see like copying an entire
source tree in one line which means one
layer that means if anything changes in
that particular
set of files that layer has to be
regenerated every time you build so
you know organizing your code so that
stuff
that changes the most often is in its
own layer
maybe consider implementations in one
one layer and abstractions in a
different layer
um that’s one idea
and then of course avoid
you know scooting in your secrets um
if
even if you go through the effort of
like in your docker file trying to
delete stuff
they still exist in the previous layer
because every layer is just a diff of
the layer
before
so
not good enough to just go through your
docker file and delete things i’ve
actually seen that
it doesn’t work that way
um also you know learning how to use a
multi-stage docker build is a good idea
that’s something relatively new um to
the ecosystem being able to
build in such a way where you’re not
including um extra packaging um
tools and things like that like maven
for example or npm or or whatever so you
just have what you absolutely need in
the final image that you’ve built
[Music]
all right distro list so all that build
up to understand why people are looking
at distroless images
so um
you know pulling in a large parent or
base image even if it’s you know
relatively safe
official image from docker these could
include a bunch of stuff that you don’t
need they it increases your attack
surface area from a security perspective
and shipping these large images around
and storing them can become clumsy and
slow and expensive since we’re all
paying for storage these days and this
reminds me of and i’m going to date
myself a little bit but remember when
i had to pay
you know for every text i sent on my
cell phone i’m hoping at some point
we’ll get beyond this paying for every
um you know byte that we’re using for
storage but this is where we are today
so we need to manage our resources
carefully
um also
distro list containers actually not only
do they you know make a point to strip
everything out that you don’t need
yeah um you want just the minimum
packages required to run your
application this means even you know
stripping the os down to its bare
minimum this means no shell generally so
there’s no exec inside that container
once it’s running out there that’s
actually a pretty good security measure
and something to consider i know it
makes troubleshooting a little more
difficult but if you’re having to exec
in a container on production you’ve got
some bigger problems going on
you can always you know in in your
development process you could develop
you know debugging
type uh container or images in order to
get beyond some of that
this is an example of a multi-stage
build so you can get a collection of um
distro-less containers from this link
here
these are ones that google built
they built they’re built from bazel so i
mean they’re not using docker to build
these they’re just images that um
they’ve built and had that they consider
tried and true most of them they’re all
based on debian
um i’m going to go ahead and put that
link here so you don’t have to
type it in if you want to just go check
out and see what they have they just
have a few for and they’re they’re
language specific so
we’ll look at one for java in a second
anyway this multi-stage build notice
that there’s two from statements
um the top one and you know pulls in a
full on open jdk
um
image and then the build happens
um
and then the the line seven the second
from statement that’s the dis trellis
image so the result from the first group
is pulled into the second group and that
makes it very small very small image
to end up with again there’s no shell it
cannot exact into this image
all right and um just one mention on
managing your images i’m going to bring
up jfrog artifactory just to show you
what that looks like i’m not going to
give you a long spiel about it but i did
go ahead and
pull up some images just to see what
they look like
just command line so we can actually see
if these distro-less images are smaller
i you know love to just believe these
statements but i always want to verify
so let’s just look at a couple really
fast and see what they look like
the only the last thing i want to say
about this slide is just
i do have
a d-zone rest card on this for container
registries considering where to store
your register your images and what to
look for when you’re looking for a
container registry
just note that whenever you’re working
with docker on on your machine the
default is docker hub you’re constantly
going to be going out to the public
docker hub so that’s something that
they have recently set limitations on
so uh it’s good to always have you know
a cached um
collection of images that you use in
your system that you rely on
okay i’ll bring the slide back up at the
very end um in case there’s any
questions that need to be answered but
don’t forget to go to this link and get
your earpods
um
but first
i just want to show you some images that
i have pulled in
i pulled down an open jdk image from
docker hub and then i pulled down a java
17
um distro-less image from the google
store
and
you can see
these top two are the same they’re both
open jdk i just have one that’s tagged
with my
jfrog instance so that i was able to
push it up there to look at some more
details about it
um but these are actually both the same
image but look at the size of these
we’re looking at 471 megabytes
whereas the distrolus images again these
are also identical to each other they
just have different tags because they’re
coming from different registries um
that’s about half the size right 231
megabytes so that’s pretty good i think
you know when you when you start doing
the math and
and
have a lot of these out there
needing to be stored or run or moved
around in your infrastructure
that’s a pretty big difference
note the created date so i just you know
recently grabbed the latest open jdk
here
this date this is the epoch the reason
it’s like that is because these distro
list images are meant to be
uh exactly
reproducible builds
so and and the spec requires a created
date so and putting nothing in there you
end up with epoch so that’s that’s what
that’s about these weren’t actually
created
52 years ago
but just some something interesting to
um note
all right so the next thing i do i pull
i pull these images down i tag them and
then i push them up to my own
container registry and i’m going to log
in there and we’re just going to take
a look at it
this is a um
a free
version of
jfrog platform
and i pull these up every now and then
when i do workshops and stuff and other
people can sign up for these instances
they’re they’re limited feature
type things but they’re they’re really
good
these are all of the
images that i pushed up here and let’s
just compare well there’s two of them
that we want to look at this distro-less
one and this open jdk one so just to
check something out here
it’s said that distress images because
they have
you know less
stuff in them less content that they
should be more secure you know less
problems with the packages inside them
and these have been scanned by x-ray and
let’s just take a look at what a distro
list looks like compared to the full on
open jdk
image so i’m going to navigate here to
x-ray information um i’m not going to
say anything about like what kind of
security problems these are i’m just
looking at raw numbers at the moment
just for our benefit this is something
for you to do for homework if you want
to you know dig into these a little bit
more and see what actual problems there
are here but this is the distro-less
images and it looks like we have three
critical violations we have you know
31 security problems some of these are
low that we may not care about
all right let’s go back to the full on
open jdk
drill down into that image
check out the scanning there here we
have three violations still but 94
security issues
um there’s still three there’s a lot
more medium
issues so it’s just interesting whenever
you’re doing this never take for granted
that what is being said is true in the
past when i have pushed up a distroless
image with like an older version of
java and stuff in fact the last time i
gave this talk this was absolutely
reversed and the distro list actually
had more problems which was an
interesting find so today it’s looking
much better
all right that’s all i have for you um i
know a lot of that might have been
review for a lot of you but i hope that
it was fun and engaging and you got some
reminders of some things and maybe learn
something new um anyone have any
questions
[Music]
thanks a lot melissa maybe some
questions from business we still have a
few minutes
for this session
i actually have one
that i was thinking about maybe
you’ve heard about corkus
yes yes yes yes
absolutely right it’s uh yes it’s a it’s
a for those who don’t know it’s a
run time and framework to create
java applications uh specifically
targeted for containers
and uh with very impressive results
right because i’ve seen them like
immediately like not even 10 megabytes
big and you’re like how can you even fit
an application inside of that so what
was your experience with it
well yeah i mean it makes sense right
because um java applications you often
get libraries pulled in with things that
you don’t you aren’t actually using you
don’t actually need and the problem with
that is you go to these scanning things
you know the security scanning thing and
you find all of these problems but
they’re things that you don’t really
care about because you aren’t actually
using them in your application so corkus
does very well with that you know
eliminating a lot of that and and making
your attack area smaller i’m not pulling
in libraries unless you’re actually
using them so that would be a good
homework assignment is to get one of
these pull up um some images push up
some images that are based on corkus and
see if you know see how much of a
difference it makes
absolutely absolutely for what i’ve seen
a little i’ve been playing with it you
can even you can create a classic jar or
a war file and you can put it inside of
tomcat and it would run anyway so it
really opens up
the door to native compilation and uh
also classic java interoperability which
i think is a very interesting
proposition
in any case so i was thinking about that
but yes we had vision exciting stuff
we try to make our container images as
small as possible all the time all the
time
so many improvements have been made with
java because in the beginning it was
pretty it wasn’t that good trying to
containerize job applications scared a
lot of people away but it’s so much
better now exactly the same problem
happens with net many of these run times
or even the erlang
beam machine basically when you have
these big run times with managed memory
and so on they they were created at the
time where containers were not on the
roadmap so clearly they had to adapt uh
in that case workers has done its
homework very well
so i don’t see any questions from the
audience but maybe there will be more at
the end so melissa i invite you to stay
until the end of the event maybe there’s
somebody comes up with with a question
by then and thank you so much it’s been
a great great reminder of how we should
be building our containers really cool
and nice tips here and there so that was
really really nice thank you so much and
now we go directly to thomas for the
last presentation of the day
that’s about psyllium and ebpf thomas
the
screen is yours
thanks a lot adrian
and hello everybody first of all
apologies for my voice i’ve been sick
the last couple of days so if you hear a
slight cough that’s from that but in
general i feel fine
let’s jump in um let me know once you
can see slides coming up here
is the screen visible already yes
perfect excellent
all right so i will be talking about
psyllium and ebpf
and my name is thomas i’m one of the
creators of the studio project i’m also
a co-founder and
cto i surveillance the company that was
created or founded by the creators of
cilium
psyllium is cloud native networking
security and observability and
before we go into more details on that i
want to just briefly talk about like
where where i am where we are as a team
and so on so um solemn has been created
by i surveilled we’re a venture
capitalist backed startup with dual
headquarters in switzerland in the us
switzerland because i’m swiss and i love
skiing so much so i never actually um
moved outside of switzerland even though
i’ve been working for american companies
since forever i can think
since i was 20s i’ve been working for
red hat for a very long time as long as
kernel developer
um
and hence our team is
basically split between the us and
switzerland our company is is heavily
focused on open source most of us have a
background in linux kernel development
or open source is a modern way um and we
are heavily investing into open source
we’re most well known for psyllium and
also for co-creation and pro-maintenance
of edpf and obviously we’ll dive into
that some more here
but to make it very clear what is what
and what we’re going to talk about today
psyllium and ebpf those are open source
projects that’s the focus of today i
surveilled this the company behind it if
you are planning or if you are
interested on sodium in an enterprise
context in a production environment and
you are
you are desiring and apply support feel
free to ping me and we can talk about
that as well that’s enough of the sales
pitch let’s jump into the open source
technology
so what is psyllium syllabus networking
and load balancing foremost most of you
probably heard of cilia in the context
of cni container networking
interface that’s what kubernetes
requires in order for
pods containers to be able to
communicate with each other across the
network because kubernetes itself does
not really have a networking model or
networking implementation itself but
then student can do so much more beyond
that kubernetes services multi-cluster
routing they can integrate existing
workloads running on vms or metal
machines we can do a whole lot of
network security from segmentation some
of the service measures cases we heard
earlier including mutual authentication
but also identity based
enforcement and transparent encryption
and then even more important is the
observability piece where we can
actually tell you what’s going on
in the form of metrics flow locks uis
and zone we’ll dive into that
what is very unique about sodium and why
so many are so excited about sulim and
why salem is currently taking the world
by storm is because of its use of bpf
and if you have never heard of dpf or
ebpf before good news will cover that in
a brief moment
ebpf is at the core of psyllium and is
why cylinder exists in fact our team has
first invested about one and a half two
years into advancing eppf further to
then build psyllium so the whole reason
cellium exists is because of the ukf by
applying ebpf to the world of containers
and kubernetes
so what is this evpf technology it has a
cute little logo like many other
technologies but what is it
actually not very complicated and you
will get it in a nutshell because it’s
very very similar to what javascript has
done to the web browser
ebpf does to the linux kernel or to the
operating system i will you will learn
you will understand this very very
quickly
evpf is nothing more than the ability to
run a program on a certain event
for example when a system calls being
made in this case a process is making an
exactly
ve system called we can then run a
program to intercept that system call
perform some action some run some code
and return a verdict for example block
the system call or export some
visibility or do some other take some
other action we can do this for a whole
lot of events system calls network
packets i o operations trace points
function calls user space function calls
so you can even run an edpf program when
your own application performs a certain
or inbox or calls a certain function so
very very very similar to how you can
run javascript code when for example the
visitor of your website clicks on a
button or mouse is over somewhere you
can run javascript code ebpf does the
same
for the linux kernel
it’s as simple as this
let’s now dive into what this can
actually do because why is that
important that sounds neat but it’s not
necessarily a completely new concept
programmability has been around as a
concept forever what does it actually do
what does it what does it enable here’s
a here’s a use case or a very concrete
benchmark of one application of ebpm
there are many many hundreds and
hundreds thousands of different
applications of ukf this is one of them
which i think is easy to understand and
very impressive this is a benchmark of
http visibility
and it measures the baseline so no
visibility this is an hdp benchmark
that’s the blue bar
and then the proxy based visibility so
this is using
a proxy such as whether
aha proxy nginx
doesn’t actually matter that much
they’re all mostly the same that’s the
yellow one extracting http visibility so
visibility in the form of this request
has been made this was the latency of
the ring of the response and so on and
then in red what is the latency
difference to an ebpf based in kernel
implementation of this which does not
require for example connection
termination and so on and i know that
this table will draw a lot of questions
or we’ll we’ll top a lot of questions um
but this is an implementation that has
been done where we can passively provide
http visibility with epf and you can see
that the latency is almost the same as
baseline whereas a proxy based
implementation actually adds quite a bit
of latency that’s one concrete very very
one concrete use case here
but if we take a step back and actually
apply and talk about this a bit more
broadly edpf is actually about a lot
more impactful as i mentioned my my
background has been linux kernel
development for about 15 years and i
often found myself in exactly this
situation now somebody would come up and
say hey i need to observe my app better
or i need this feature or this other
feature or i can remember
when i was early on working on openshift
like the original versions of openshift
pre pre-kubernetes um openshift
developers would come to me or come to
our team and say hey we need this
feature great then we will work with the
upstream kernel community to get this
feature in
which by its own will probably take
about a year or so because it’s
complicated to do kernel changes and
it’s hard to convince all the other
kernel engineers but then it didn’t
actually end there it would take years
and years and years for that new kernel
version to make it into the hands of
users because most of the users in
particular reddit users obviously uh
consume reddit enterprise linux in
another enterprise distribution so it
would take up to five years for
the beginning or the the ask for a
change to the delivery of a kernel
version that would actually function for
that user and by that time
the requirements of the user had likely
already changed so this made the kernel
a very difficult place to innovate
essentially this this meant that a lot
of innovation actually moved to user
space why we have seen massive rise of
user space network in user space io user
space security
things like g visor building entire
abstraction platform for system calls
and so on because the linux kernel
itself became very challenging to
innovate in we even saw entire forks of
the little kernel for example for
android because it was not possible to
innovate quickly enough
ebpf is now changing this all of a
sudden because similar to how javascript
has unlocked or has decoupled
the browser version from innovation
right like if some of you have been
using internet for a long time you
probably remember way back when you had
to upgrade your your web browser to
visit certain websites when was the last
time that you had to do that
probably not recently because javascript
and auto programmability frameworks
don’t require you to do that anymore and
ebpf now unlocks the same for the
operating system in particular for linux
so what used to take years can now be
done within days because we can
dynamically load a program and change
the behavior of the linus card
now that sounds potentially very scary
to some of you and some of you may even
say well what is different to for
example linus kernel module it was
already possible to extend or to load
additional kernel module code at one
time
this is where the different com
difference comes into play so ebpm this
is the entire evpf landscape from a high
level to a low level and this low level
here this is the coral this is the
operating system level this is where the
actual edpf program will run and part of
this you see a box called verifier and
chip just in time compiler this layer
actually adds a sandbox concept very
similar to how containers are sandboxed
as well and they ship compiler which
will take
generic bytecode generic edpf bytecode
and just in time compile that into your
cpu instruction sets very similar to how
this is done for other interpreted
languages like javascript or java as
well
this means that yes we can dynamically
load a program very similar to a linux
pro module but we’re actually running in
a sandboxed program or sandbox
environment very similar to our
javascript program is also sandboxed
the just in time compiler actually
ensures that we can run as efficiently
as if we recompile and restart our
kernel or our entire machine
and then we have a kernel api which
actually ensures that our programs are
stable and portable which solves
probably the lesser known problem of
linux kernel modules which is that linux
kernel modules are very very hard to
make portable cross arbitrary kernel
versions so these are the problems that
are solved by edpf and all of a sudden
the kernel space becomes a place of
innovation again which is amazing
because the kernel is a truly powerful
place to run coding because the kernel
can see everything the kernel can
control everything and we’ll see
examples on how that can be used so
that’s the lower levels of evpf and if
you look very clearly or very very
closely you can actually see a windows
logo there even though the ebpf is
coming from
from linux microsoft has recently ported
evpf over to windows as well so ubf is
now available on windows
2.
on top of that you see a bunch of sdks
and libraries which will make edpf
programmable in higher level languages
because most of you will never want to
write an etf program in assembly
bytecode so you can use python go c plus
plus c roster order languages um to
actually write ebpf programs in and then
you can take it to an even higher level
which is evpf projects or the end user
facing projects such as vcc psyllium
falco catron dpf trace and so on an
ever-growing list of ebpf based projects
would actually solve use cases for end
users
in the next step we’ll now focus a bit
on cylinder i wanted to give a high
level in showing to bpf first because
cilium is obviously not the only vpf
application out there
in terms of governance uh ebpf
has been or we together with some of the
other industry members that have been
involved in the ukf for a long time
you’ve together created ebpf foundation
it’s a linux foundation based foundation
um you can see the founding members
here since then the members or the list
of members have been extended quite a
bit and the ebpf ecosystem is growing
very quickly
but let’s now shift gears into syria and
actually look at
what type of application what type of
services what type of values can we
provide if in epf
so if we go back in here you can see in
the picture here cylinder is kind of in
the middle of a world that shows
kubernetes mesos darker hdl today it’s
mostly
kubernetes of course
but in the end sodium and ebpf is
actually very generic and very
applicable to everything as we will
learn it’s not even limited to
containerized applications
only
sodium adoption here you can see across
the board this is not even inclusive at
all but from
big sas providers to cloud providers for
example gke eks anywhere are already
using silhouette of silly by default
alibaba in the high performance option
digital ocean whereas other smaller qms
distributions are
using sodium as the default c9 to
provide networking and security and then
from telcos to financials
syllabus kind of instead of an ebpf is
very applicable to a very wide stream of
use cases and thus adoption is bright as
well
as we now start to dive a little bit
deeper here um i’m going to offer kind
of this stage of the journey of what
users typically go through when they
apply or use cilium in a kubernetes
environment and if you want to want me
to dive deeper on any of them in
particular feel free to mention that in
the q a
window i have that open in another
screen so i can actually see that and
can dive deeper on that particular topic
so first of all in stage one is what
typically what we’re seeing when you get
your initial kubernetes cluster up and
running this is where you need basic
part and service connectivity and you
will also probably care about initial
cluster hardening
for example um
making sure that kubernetes or back is
configured correctly making sure that
you haven’t accidentally um
exposed your communist dashboard to the
world and so on you might also be
deploying something like container image
scanning in this stage of course you can
pretty much use any cni because all you
need here is basic part and service
connectivity you don’t have additional
security or scalability requirements as
you go into stage two which we usually
say you enter when you run sensitive
data workloads or you have multi-tenancy
requirements so you have multiple teams
um running on a on a single kubernetes
cluster this is when you start carrying
about zero trust network security
transparent encryption you might want
compliance monitoring what is encrypted
what not what network policy violations
are happening and so on um also you want
to monitor your applications for example
measure the http latency of your
services or
identify how many http 500 error codes
are being returned and so on to build
for example the typical golden single
dashboards also you might have
requirements around efficient load
balancing already as you scale out and
you very likely need to integrate with
external workloads which are not
containerizable or which are not part of
kubernetes and very likely they’re still
sitting behind a traditional firewall
which also very likely is not cloud
native aware so we’ll have a really
really hard time coping with
containerized workloads that constantly
change ips all of the time this is where
something like the static reverse iep
feature can come in which actually binds
volatile containers behind a stable ip
which you can then allow
through your um traditional firewalls
stage three is typically when you
adopt kind of a more micro services
architecture sometimes stage three
actually comes on stage two if you’re
very modern and if this comes natural to
you for more traditional enterprises
that clearly kind of a later stage
excuse me
this is where things like multi-cluster
comes into play layer seven
observability layer seven traffic
management
um git ops integration for example to do
network policy integrations in gitlab
github as well as advanced security
runtime protection that we can offer
advanced scale advanced churn and all
the way up to multi-cloud use cases
if you want me to focus on any of them
in particular feel free to mention that
in the q a channel and i will spend a
bit of more time on that particular
aspect
as we now dive a bit deeper into the
onto the individual aspects so the
scalable and secure networking most of
you will know us already as a very
efficient and scalable cni we can run
on-prem we can chrome cloud we can run
hybrid cloud so we can run opening for
example bgp mode and overlay the
excellence you leave whatever in cloud
provider environments we can maybe
integrate with the cloud wider sdns but
then we can also actually not use them
because some of them have for example
limitations in scale and you will run
into ik exhaust issues so you can go
back and actually run an overlay
um but then as well because we can run
on all the cloud providers like equally
good we’re really good for multi-cloud
use cases because we can then for
example run a multi-cluster scenario
across cloud providers for a cross-cloud
and on-time environments
on our level left we can not only
connect pods or services together we can
also provide load balancing in a variety
of ways we can replace q proxy so you
can achieve a high performance
implementation of q proxy that can scale
really really well but then we can do a
lot more we can also become what’s
called a north south load balancer
essentially load balancing into your
cluster so you can turn any linux and in
the future windows machine into a load
balancer machine and for example replace
entire five boxes
in the latest version we can also
implement kubernetes ingress so we have
an implementation of the kubernetes
ingress controller and can just also do
all of the layer 7 load balancing as
well both into the cluster and also
between
services and across clusters as well
on the security side
first of all yes we are a cni which
means we can do enforcement on the
network level which is actually very
very important than an aspect that
should not be underestimated
even if you’re running a high-level
concept like a service mesh you almost
need to be aware for example a sidecar
proxy or a proxy based authentication
method which yes adds additional
security from a mutual authentication
perspective does not prevent you from
all the potential leaks that you might
see on a network if a proxy does not
support a particular network protocol it
is obviously not able to enforce any
rules on that so in order to have
correct network security you also need
always need to protect your network
level first and then add mutual
authentication on top which is why
psyllium combines the two together so
very similar to a service mesh approach
with which is using mtls we’re taking
the mutual authentication to another
level and saying yes we are enforcing
useful authentication but we’re not
limiting ourselves to tls which means we
have an identity based concept but we
can enforce this mutual authentication
with identities not on the tls level but
actually on something more generically
that can that can support any network
protocol which is amazing because now
all of a sudden you can support non-tcp
protocols and you can ensure that
nothing can bypass
your authentication and authorization
layer
obviously we also support just standard
kubernetes now policy then we can go
further and look into for example dns
based policies into api policies
and so on
besides the segmentation or
authentication here we can also do
transfer encryption we can do to do this
using ipsec or wireguard
both of them have pros and cons ipsec
has the pros of being uh fips compliant
wireguard can be slightly faster if no
cryptographic or if no hardware offload
for cryptographic operations is
available
and then last the last pocket here is
the expansion this is actually going
beyond the single fibonacci cluster as i
mentioned we can do multi-cluster in a
completely proxy free manner which means
you can connect multiple clusters
together but not actually introducing
either egress or ingress proxies which
means you could preserve your
natural organic network performance as
well
we can then also integrate into or we
can integrate non-kubernetes node
container workloads as well what does
this mean so as i mentioned cylinder is
not really limited to containers or
kubernetes at all you can actually run
sodium on any machine that you want and
integrate that machine into your mesh or
into your kubernetes cluster so you can
run let’s say have your existing ec2
fleet of vms or your bare metal database
server you can run psyllium on them and
then have them appear have these
machines appear as workloads very
similar as if they were parts so they
can be reached from parts you can apply
network policy they can the external
workloads can it can reach the
kubernetes dns they can talk to
kubernetes services and so on that gives
you a lot of flexibility
to mesh things together
and to get to a zero trust networking
concept without actually having to
containerize everything
switching gears a little bit and looking
into kind of the network policy
forensics and compliance use cases
enforcement pawn is obviously important
but there is more to this and it’s the
visibility side and we’ll actually
double click on this on this lower left
window air bit because that’s very
crucial so keep that in mind for a
second while we complete the slide so
yes we have kind of this identity aware
network and runtime visibility later to
look at but then we can obviously also
do a lot of metrics we can monitor
pretty much everything of course how
much are we forwarding how much are we
dropping um we have a tls parties we can
for example answer questions such as
what has
what are the negotiated tls ciphers that
are in play has anybody attempted a tls
downgrade attack uh what percentage of
my traffic leaving the cluster is not
tls encrypted and so on and then we have
more traditional network tap
and port mirroring functionality as well
which you for example can use to
integrate an ids such as sorry coda
the nice thing about this network tab
ideas insertion part is it’s it’s
identity where so you can actually apply
this for a subset of the traffic only
and then there is splunk and splunk and
elastic search integration all of this
visibility can obviously be shown using
hubble ui our own ui but also data grade
into your existing sim platform as well
i promise that we will double click on
this network and runtime visibility use
case a little bit this is an example of
a traditional network flow log a five
tuple flow lock and you can see um
typical network communication this ip
talking to this ip on a particular port
and whether it was rejected or accepted
if we take this a bit further
this is what we provide and
when you look at this new view um you
will actually first of all kind of
question is this even still a network
flow or is this even showing the network
at all because it doesn’t look like
additional network flow log at all but
it actually is it just provides a lot of
additional context and identity
information but if you look closer we
see on the lower on the left side we see
the source identity we see a network
namespace or kind of kubernetes
namespace a pod name and then we see the
entire process ancestry tree of the
workload running so in this case it was
a container runtime in this case docker
d which was launching or which has
launched a container and inside of the
container there’s a binary crawler
running which is a node app which is
running node server.js which then for
obvious reasons when you look at this
got compromised and an attacker launched
a reverse shell and you can see with the
arrows you can see the individual
connections
so the node app actually made
connections out to api.twitter.com we
can see the dns feasibility that we have
and also reached out to the
elasticsearch which is an internal
service
and then on the netcats obviously
we see the nc binary netcat
which was used by the compromise code to
reach out to the url which is
clearly not a reverse shelter which then
obviously received instructions and
netcat was used to launch a bash and
inside of that bash shell curl was used
to now poke around and uh retrieve
information for example from
elasticsearch we can also see that we
actually see the layer 7 information as
well so we normally see the connection
to elasticsearch we can also see that
there was a get to slash users to slash
search so we can even see what type of
api call was being made
and funnily enough this new what we see
here is actually for the same traffic as
here and then this network view would
obviously never be able to tell whether
any attack is ongoing right
like it’s no network person would ever
claim that you’re able to see this type
of attack based only on the five two
blackboard flow log and it becomes so
obvious looking at this i’ve used an
example of a compromised code part here
with a reverse shell but this is
actually also incredibly useful if we
are if you are figuring out where is the
network traffic coming from is this the
actual workload is it an application
developer running cube control exec
running a bash and crawling around is it
kubernetes network-based health checking
and so on is like what what type of
network traffic is my kubernetes control
plane doing and so on this will work for
literally any process on your machine
we briefly mentioned that psyllium can
provide alternate authorization not only
on the network level which is here the
layers to layer four for the networking
folks under you but also on layer seven
and actually for example limits that a
front endpoint can talk to a back and
part which can only do
an http get to slash public but for
example not do a post or an update or
not access
private or another url for example
this same principle works for protocols
like kafka cassandra mkhst the variety
of auto protocols so you can for example
also say my part can talk to a check
cassandra database but it can only do a
select on a particular table but you can
never update it or you can say my pod
can talk to a shared kafka cluster it
can can consume on this topic but you
cannot consume or produce on
any any topics that other teams are
using
the dns server policy is incredibly
interesting for rendering or for
modeling services that needs to reach
outside of the cluster i’m using
mydomain.io here as an example but
another typical example would be your
service reaching out to aws s3 cloud
provider cloud provider bank databases
are in particular interesting because
they’re backed by hundreds and hundreds
of ips which are constantly changing so
how do you model them from a network
policy perspective you clearly want to
limit the egress ability of your
services because if the code gets
compromised you want to prevent data
exfiltration
and this is what dns policies come into
play what how do you do this if the ips
the service needs to reach out to are
constantly changing this is where you
can write a dns policy like this and
actually say i want my service to be
able to reach out to startup my domain
on io or it could be
mybucketss3.com or some other domain
name
right what sony will then do is actually
hook into all of the dns traffic and
proxy that which means that first of all
it will not even allow any dns
resolution that does not match this
domain or this pattern which means it
will get prevented it will get rejected
on the dns level
and then if it is allowed only the
allowed set of ips will actually or only
the returned ips will be allowed which
means that even though aws s3 might be
backed by a thousand ips only the four
ips returned by the dns will actually be
allowed
obviously this also prevents any dns
based data exfiltration attempts as well
because our proxy will
kind of decapsulate all dns requests and
not allowed to do to do data exploration
by dns either unless you find some way
to actually encode that into the dns
name itself which is i guess possible
but very very challenging to achieve any
reasonable bandwidth and then you would
still have to match the pattern of
course that you are actually allowing
all right the last piece here is the
hubble or the observability part so as i
mentioned this is i think probably the
most significant part overall um hubble
ui or hubble is the observability layer
of psyllium and it’s using ebpf to
transparently get observability data so
without requiring any manipulation
modification recompilation of your
applications so you can deploy sony and
hubble and you will get this
observability data for your existing
workloads as they are running
you can see an overall kind of overview
what services are running how are they
connected you can even see all the way
into individual layer seven calls you
can configure this to do layer 34 only
or layer seven as well and then in the
lower section you can see the live view
of what are the individual connections
or requests that have been observed you
can obviously filter here you can for
example showing show me only
the the requests that have been dropped
or show me requests with latency bigger
than this and so on
all of this data
can be viewed live or you can feed this
data into timescape and actually build a
a time machine so timescape is our click
house based um mechanism to actually
store this observability data over time
and then you can go back and say hey
give me this view but from two weeks ago
in this particular time frame and so on
all of this data like the hubble ui that
we just looked at which is obviously all
the way available through cli and api as
well as all the metrics so the
the prometheus-based metrics that we can
look at them if we have time as well as
our network policy editor
which is able to visually or to
visualize network policies and to
graphically edit uh policies all of that
can be either be done at a cluster level
or cluster-wide scope are they
multi-clusters phone or
excuse me you can also tie this down to
for example particular namespaces and
say a particular team managing a
namespace can view these diagrams and
dashboards and all of these ui tools but
only for the namespaces they own and you
can tie this to single sign-on provider
octa active directory i am whatever you
want here to actually inherit existing
role information you have and then
configure all back rules very similar to
how fibonacci does our back rules as
well
the last piece here before we wrap up
github’s network policy guardrails so a
use case that comes up very very
frequently now is that as teams scale up
and use more and more kubernetes the
desire is there for
application teams to to inherit part of
the responsibility for the fine-grained
network policy
but a security team still wanting to
preserve some control over the network
policies for example the network
policies within a namespace on what pod
should be able to talk model to what
other part the network uh the
application teams should be able to
define these rules on their own but the
application teams on their own should
not be allowed to open up new
applications to the outside internet or
to expose them or to do cross-name space
traffic this is where the guardrails
come into play so you can essentially as
a security team
define a set of broadway rules for
example app teams can define within
namespace and everything else needs to
be approved and then this implementation
which is available for github and gitlab
will automatically approve pr’s with
network policies that are within the
guardrails and for everything else uh a
security motor’s a motor role needs to
improve the pr before it can get merged
this is available as well and allows the
security team to partially delegate the
creation of network policies as well
obviously you can still even with this
news namespace and global policies as
well so even if you have this in place a
security team can for example always
write down policies at the globals
global scale if you have opened up
namespace level policies for your app
teams
scale and performance as i mentioned uh
ebpf
in the beginning is is a highly
performant uh way of doing networking
observability security and so on if
you’re interested in this i will not go
into super details here but if you are
interested in this topic feel free to
check out the psyllium block
we have blogged extensively on both
performance comparing psyllium data
playing to other cni’s to service meshes
as well as scalability how to achieve
really big multi-cluster systems
if that is topic of your interest
and with that i will leave you with a
couple of pointers before we
um open it up for questions
if you want to learn more about cilium
if you have been if you are interested
in this feel free to check out cilium.io
edpf ebpf.io is the place to go
obviously if you want to engage with
sodium try it out get started join the
psyllium and ebpf slack that’s the best
way to reach the development team to
participate to ask questions and then if
you want to run selenium enterprise
concept context feel free to go to
isurveil.com
with that i think we have several
minutes left for questions if there are
any and if there are no questions i’m
also happy to demo the
metrics real quick
exactly so we thanks a lot thomas is
fantastic all of this technology is
really really cool actually we are using
it we are using psilium already in apuyo
cloud which is our latest container
service here in switzerland so we are
it’s opening up many many different
doors and possibilities for our service
as well so it’s fantastic
to learn more about it
if there are any questions in the
meantime if you would like to show
something else of course please feel
free to take this i want to make sure we
have enough time for a question but i’m
always happy to demo
a bunch of fancy metrics as well
of course
let me see let me know once you can see
the graphone dashboard i think should be
coming up
so we have a wide variety of metrics
that we can show i’m showing kind of
three dashboards here for example we can
see on the network level how much is
being forwarded here we can see that
some network packets are being dropped
like oh that’s probably a bad reason we
were a bad sign so we could actually go
in and see why was it dropped and we can
see that we have a steady flow of
network policy or policy related flux
so that’s something that we would then
for example want to look into hubble and
see which parts are actually subject to
this
we can see that we can observe the tcp
layer so we can see that we are
constantly missing certain tcp syn acts
which is a clear signal that connections
are for example timing out obviously
this is a a demo app that is
purposefully
fully doing this
you can though go down there and
actually see a kind of a a lockdown of
water parts that are subject to the
policy deny so we can actually lock this
down further and then go here to the dns
dashboard which is probably the most
favorite dashboard of everybody because
dns is
reason number one for any incident so we
can see for example how many dns
responses or how many dns fares are
currently happening and for what reason
and then what parts are subject to dns
error so you can immediately see whether
your app is subject to the less errors
we can also see for example the top 10
dns queries
we have a dna an http dashboard here
where we can see the number of http
requests um the the latency of
individual requests
and so on so a ton
that’s kind of the high level but then
we can also go out into very detail for
example tcp metrics and actually measure
the smooth round trip time which is
incredibly interesting we see here that
from time to time
the prometheus pawn has a very bad
smooth round trip time which probably
indicates that it gets
throttle or
rate limited you can go further and
actually understand
how much traffic involved parts are
doing what is the total traffic
pods are doing right which parts are
responsible for for example network
latencies going up
and so on
then last but not least we can go really
low level and actually all the way look
at for example network interface
statistics and you can for example
correlate that to whether certain
virtual machines or certain nodes are
overloaded so this is just a very small
glimpse at kind of metrics that we can
provide so we can go all the way of kind
of high level http dns level that we’ll
use from a service mesh all the way down
into lower levels that you would only
expect from a network device type and
correlate all of this information
together and this makes it makes it
relatively simple to troubleshoot
network issues
very very quickly
excellent
very impressive very very impressive so
i
don’t see much activity in terms of
questions on the chat as well
so i think
uh we are probably arriving at the end
of this session
just a quick reminder
before we close this session down
the next editions of our cnc meetup
will be on may the 12th the next one
there are still slots left so if you
would like to present some
technology some practice that you are
using in your own environment please
come and tell us we would love to learn
from you this community is built by you
so it’s very important that you
participate
then we have two more sessions this year
planned on august 25th and november 24th
so please don’t hesitate to let us know
if you would like to participate
this video will be available on
vision.tv
uh starting tomorrow so
please sign subscribe and uh share as
well if you like if you would like to of
course ask questions feel free to
contact us and to contact as well the
the speakers of today and thanks a lot
for joining if you watch this on youtube
thank you thanks a lot for watching
until the end and
first of all thanks a lot to our
speakers to moritz melissa and thomas
it’s been a fantastic evening a really
really interesting session so thank you
so much and
see you next time at the next cnc
computing meetup switzerland thank you
and bye bye
bye bye melissa bye maurice