Container Standards Explained @ All Things Open 2022 (Raleigh)

so I’m really thrilled to be here today in person uh in Raleigh this is my first time in Raleigh so I’m glad to see all
of you how many of you are locals here ah awesome awesome see this is why we
like to do in-person conferences because we get a lot of the local um audience here
um over the past few years I’ve made a lot of progress in my career I became a Java Champion that was pretty exciting
Java is my primary background Java server-side coding but rarely do I meet a Java developer
that isn’t doing other things these days it’s a requirement now especially in Cloud native to learn all of these other
things all these other skill sets and tools I became a Docker Captain so a lot of what we’re going to talk about today
is Docker just because that’s what folks are most familiar with But please understand that when I talk about Docker
I will be explicit when I’m talking about the company versus you know like a Docker container or Docker image
otherwise I’ll explain it when I’m you know talking about like an oci image
something that isn’t specifically a Docker image um a few colleagues and I got together
um and wrote a book uh this is devops tools for Java developers a few years ago I did get
involved in a devops team I came from this world where as a developer I was you know in a corner with headphones on
doing my own thing and tossing my work over the wall I definitely had to learn
a few different skill sets to be involved in a true devops team which doesn’t necessarily mean that a
developer needs to know everything it just means you need to get really good at communication because everyone along
that pipeline along that chain needs to understand what’s going on and why
so very cool book a lot of the material that I’ll share today is included in one of the chapters on containers that I
wrote so if you’re interested in knowing more or getting more details here you go
all right today we’re going to talk about containers but more specifically the business what’s going on behind them
and why bananas because I refuse absolutely refuse to have any
presentations that are using shipping containers I’m tired of seeing those so we’re going to have a banana theme today
I hope you enjoy that and we’re fun to find pictures related to what I was talking about
um so moving on my introduction to Docker and containers was by way of a
project that was developed by a third party contractor and at the time it was agreed that this project needed to come
in-house we formed a team around it in order to do maintenance on it add more features fix bugs that kind of stuff so
in this case it wasn’t a decision really that was made by my team at the time to do this it was kind of handed to us and
we needed to figure it out so first thing that I did when I um
you know to ramp up on the code base um were to look at all of the the docker
commands that were in the readme file just so I could get these things up and running and you know see how it works I
could get my development environment on this Baseline and I’m always intrigued by learning new
things so I’m you know what is this Docker thing and um
you know this code was already written and functional so the best thing for me to do is just dive in and figure it out
get it up and running right away I had questions
here you know I didn’t know what I was dealing with I found a Docker file and started looking at that trying to
understand all of the commands everything that was happening under the covers But ultimately what I really wanted to
understand was what what was a container what was this image thing how do I run
it how do I you know launch it how do I change it all of those questions came up
first of all when you’re dealing with a new technology it’s really good to step
back and understand where it came from get a little bit of the history behind it this can help you making decisions
moving forward with the technology that you’re working with you may even decide this isn’t the correct way of using this
technology maybe we need to look at something else so if you are thinking of doing the same
you’re thinking of wrapping your application in containers and going Cloud native take a moment to consider
the specific problems that are for your project that you would like to solve how
many of you are currently using containers right now okay how many of you are using
containers in production right now that’s about the same so yeah several
years ago when I asked this question there’d be several people saying yeah I use containers but when I ask production
no one crickets because everyone was afraid of them still now we see this you
know more and more popular it’s kind of standard tool set everyone needs to understand and learn how to do
especially developers understanding how the docker files work how to build these things how to change them
so let’s go through this list this is just you know some of the issues that you might encounter some of the whys
that your team may be deciding to use containers the first one really you’re using Java
11. so yeah um you know Java 8 was released in
September of 2019 Java 17 the latest LTS release was released in September of
2021 and I’m going to pick on Java just because of my background this applies to
you know any language of course but um I do know that there are projects that are stuck on Java 8 for various reasons
legitimate reasons I understand but I also know how frustrating it is as a
developer to have my nice new fancy machine I’ve developed something in Java 11. I try to deploy to a prod machine
and it only has Java 8 installed really irritating so this is a one good reason why
containers you know make this a little bit easier on us um second my workstation is a MacBook
Pro I’m going to use MacBook in here cool how many are on Windows
nice so for the first half of my career I was on a Windows machine I started a
new job was really excited walked in the door they handed me this computer and it’s a Macbook I freaked out I had to go
home watch all the videos on what key to push where it was really irritating and
it was difficult for me this is later in my career I had my all my shortcuts you know I was used to how the Windows
machine worked now I’ve been working on a Mac I don’t know if I’ll ever go back we’ll see
um I’m pretty tempted by the WSL features in Windows machines now and at
one point I actually stole my son’s machine just to see how uh you know Docker was running on his little Ubuntu
VM that he had so uh yeah interesting interesting stuff happening in the
windows world might see a new machine in my future we’ll see um but anyway something that happened to
me during you know looking at some code once um I was doing some web application stuff
and running some tests and I don’t know if any of you have scars from the browser Wars but I do any of you
remember in JavaScript all the if you’re in this browser do this if you’re in that browser do that well I found some
unit tests that were actually if you’re on a Mac run this way
what if you’re on a Windows machine run this way we’re deploying to Linux machines so it
was just a weird situation lots of problems there but this is something that containers can help us a little bit
with so you don’t have to worry about all of the permutations and versions of os as there are out there you still
can’t run a Windows image you know Windows container on a Linux machine
that doesn’t work that’s a big that’s a different problem but for the most part you know you can not have to worry about
which Linux distribution you’re working with uh number three there’s a bug in
production as a developer the first thing that you want to do is to be able to reproduce a problem before you start
working on it well sometimes you’ll get a fire come in you’re in the middle of working on this brand new feature right
you’ve got your development environment is all up to date with the latest in the code you know in your source code so
sometimes it’s just not easy to get back to that place that you need to be especially if you’ve installed packages
and and everything on your machine that are later it just doesn’t match production anymore
so this is another reason why containers are pretty valuable in a development environment because you can exactly
reproduce the environment that your application is running in production a little bit easier to deal with finding
bugs and finding resolutions for those bugs without having to reinstall everything on your machine to get back
to where it was uh number four um it works on my machine and only on my
machine anyone who’s been a developer for I don’t know more than a year has had this
happen to them where for some reason everything just works great something is different with your
development environment than everyone else and production and unfortunately you check stuff in and things break and
you got to figure out what that is so this is very much related to number two um again you know having consistency in
software versions and libraries are going to make your life a lot less complicated and containers help you get
there uh number five we just hired three new developers all right so every single
time I get added on to a new software team or a new person comes on the very
first thing that we always make them do is update the documentation on how to get this thing installed and running on
your machine why do we do that because no one touches it after they’ve gotten their stuff
installed and working no one ever goes back to that documentation and fixes it the problem is is that as you continue
to run and as you’re doing your thing you’ve got caches that are being populated on your box
um especially with containers you’ll notice you know a lot of image caching happening and when you have someone come
in with a fresh new system there’s invariably going to be something that is different about their machine some
package some Library they don’t have that you have had for the last six months and it just have never had to
worry about so that’s always a valuable thing to do containers help you with this they get
you a development environment all up and running it forces you to kind of you know keep up with your documentation
it’s very nice all right uh six my service is super
popular so I’m moving on from you know even just developer world let’s just talk about the production World why uh
you would use containers and you know for the most part you you get the
consistency and being able to ship your application around there’s a lot of advantages that you get for scaling your
services I know I had a project once that I needed to split up with a you know back end and front end because our
front end needed to scale much differently than the back end and we also needed to be mindful of the
resources that we were using in our Cloud native environment so um that was
a really big Advantage was being able to just launch additional containers as needed as load increased on this
particular service all right all right um I think I can probably skim
through this one pretty quickly because we talked about all of the use cases here that containers make sense for
for Dev environments um yeah that uh
windows subsystem for Linux is super tempting for me to get to play with
containers on a Windows box test environments being able to make
these test environments much similar to production really helpful when you’re
getting you know your software um out there and getting it moved along
along your pipeline to release really big Advantage another thing for
test environments even for like unit testing and stuff
anyone use test containers or have any experience with that yeah very powerful
very powerful to be able to launch a particular database at a particular
version making sure that it exactly represents the SQL that you’re going to be using in your production environment
I’ve had issues with that in the past where just to get our unit test to run faster we would use like an in-memory
database which just a little bit different than the actual database being used in production this can cause you
some problems if you aren’t careful
okay so another thing about using a technology pay attention to how it’s being used in the market are other
people using it is there any you know evidence that this is becoming popular
or um one of the things that really drives me to make a decision is how much
Community involvement there is being able to ask a question on user groups
and get answers to be able to communicate with other people that are also struggling in the same area very
very important if you find a you know a piece of technology you really want to use it but there’s no one else using it
that makes it really difficult now if you’re an r d sure you can deal with that and you can start your own
Community right but when you’re in a corporate environment and you need to get stuff up and running and you need
the support you really need that Community backing behind you so let’s take a look at what happened
with containers over the past I did a little hunting and for the past several
years assistic you’re probably familiar with that software that’s a company that
produces pretty powerful monitoring and troubleshooting tools for Linux they put together this container report every
year and it’s based on an analysis of their users and part of that report includes data on container runtimes that
are currently in use and in 2017 you can see that I know a
whopping 45 000 containers were in use they didn’t even bother making a graph for this 99 of those containers were
running Docker at the time so they didn’t even you know bother splitting up the results back then 2018
uh double the size you know 90 000. I mean considering the world we live in
that’s pretty small but 83 were darker and then you start seeing some other run
times come in there um core OS rocket containers the mesa’s
containerizer lxc so it looks like other runtimes are starting to encroach on
Docker a little bit and when I say Docker in this context I mean the company the company product
moving on 2019 look at this explosion two million containers
and Doc are still holding relatively strong but note that 18 percent is container D
we’ll discuss the reason for that in a little bit cryo is still in there still got some
market share um interesting here is The Disappearance of a rocket that’s kind of a sad story core
OS was required was acquired by a red hatch in the beginning of 2018 prior to
that rocket was actually accepted to the cloud native Computing Foundation as an
incubating project and it looked like a really promising competitor to docker’s container D it dealt with security a
little bit differently anyway interesting project however since that core OS acquisition the development
of the project went dormant and in mid-2019 rocket was archived by the cncf
and in 2020 that project was ended in in February
lots of stuff ended in February 2020. anyway um
that project still exists I mean you can still go out to GitHub and you can download it and use it but there’s just
no maintainers for it anymore all right next um next year that you
know 20 20 21 they came out with their 2021 container security and usage report
hard to know if we’re comparing Apples to Apples here considering that little qualifier of a subset of customer
containers but 2 million is a pretty big number so I can see why they’ve done
that I’ve linked the report here on each of these diagrams it’s definitely worth going and taking a look at they have
very interesting demographics and data source information that make a lot of sense
and they talk about other services that customers are running as well as what they’re using for orchestration like are
they using kubernetes are they using you know swarm or something different in a nutshell what I pulled from this
was that increased usage of containerdy from 18 in the previous year to now 33
percent that Trend definitely continues again I’ll explain why in a bit
all right last one promise three million containers here this is the most previous report that I was able to find
not a whole lot of change but again we see high usage of container d 46 percent is still docker
okay now we’re looking at the market we’ve we’ve gotten ourselves to what we’re dealing
with today um now I kind of want to get into some history where did all of this stuff come from in the beginning
um clearly the market has been moving more and more services and applications are getting wrapped up into containers
and understanding how we got here and um where the tool that you might about to
be commit to for your project came from you can avoid some nasty pitfalls
and it can help you you know use the tool in the most effective way you know that
that quote uh if all you have is a hammer everything looks like a nail it’s
very applicable here containers have proven themselves to be really useful but
they’re not intended to solve every use case out there
all right starting this history lesson I’ll try to go fast but there’s a lot of info here so sit back in your seats get
comfortable in the 1960s and 70s Computing resources were a lot more limited today uh
computers were commonly dedicated for long periods of time to a single task for single user obviously pretty
inefficient so efforts efforts were started to improve that in both hardware
and software areas figuring out how to share resources without getting in each other’s way or
having a person inadvertently cause an entire system to crash for everyone for example
in 1979 during the development of the seventh edition of Unix to root was
developed and later added to the Berkeley software distribution in 1982. the system command allowed for the
ability to change the apparent root directory for a process in its children and that resulted in a limited view of a
file system which is pretty advantageous in order to provide an environment for testing a different distribution for
example without bothering other parts of the system or any other users
true was a good move and a direction that provided isolation required of us
today but several years later in 2000 FreeBSD expanded that concept and
introduced the more sophisticated jail command that utility was released in the
free BSD 4.0 version and its features were later improved in
in later versions helped to further isolate file systems users and networks
and included the ability to assign an IP address to each jail
then in 2004 Solaris containers and zones brought us ahead even further they
gave applications full file system users or full user process and file system
space and access to system Hardware then Google jumped in with their process
containers in 2006 these were later renewed to renamed to c groups which is
probably a word you’ve heard before in reference to Containers that’s centered around isolating and limiting the
resource usage of a process in 2008 c groups were merged into the
Linux kernel which along with Linux namespaces led to IBM’s development of Linux containers or lxc
okay now is when things get really interesting this is when Docker was open sourced in 2013 that same year Google
open sourced their let me contain that for you project if any of you have heard
of that project that provided applications the ability to create and manage their own sub containers and from
here we saw the use of containers explode we definitely saw that in the market Docker containers specifically
initially Docker used LXE as their default execution environment their
runtime but in 2014 Docker chose to swap that out with a library called lib container
which was a native solution written in go I thought that was an interesting bit of trivia noting that you know a lot of
this the lower level parts of Doc are all written and go soon after the let me container that
project ceased active development they had the intention of joining forces and migrating those Core Concepts to
docker’s lib container and lots more stuff happened we won’t get into all of that but something
specific for Java users which I found really interesting um since a lot of my attempts were in
containerization were in early versions of java um Java 7 was released in July of 2011.
java 8 was released in March of 2014. and you can see given the development
going on in containers at the same time it shouldn’t come as a huge surprise that older versions of java are not
fully container aware except that was a surprise for me if you
don’t know this and you wrap your Java 8 application you’re going to start to suffer from out of memory killer
activity the reason for that is because the mechanisms that the jvm uses to um
determine the resources available they actually come from the host machine and
not from the C group limits which you would expect so this is a reason why you need to get up to later versions of java
if you intend to use containers especially if you’re going to be using more than one container on a on a single
host um possible with later Java 8 update
there was some stuff you know that was brought back into the later Java 8 versions but recommended to get to Java
11 at the very least in order to benefit from most of the container features
available there all right
um moving along in June 2015 very specific event this is
when the open container initiative was established also called the oci this
event pretty important to know about it gives you some insight into the activity and motivation behind the shifts in the
market especially with Docker and on June 22nd when when they announced
everything um this they created this organization to
be under the Linux Foundation it had the goal of creating Open Standards for container runtimes and image specs
Docker is a huge contributor to this very much involved in this in their early days and in their announcement of
this new organization however they pointed out that there are 20 over 20 organizations that were involved
containerization now has evolved to such an extent that this number of
organizations they wanted to work together to come up with some common ground for the benefit of all it
shouldn’t be just a tiny isolated company deciding how all of this should
work for everyone and in order to be a successful doing this you need all of
this input from all of these different Avenues you know a lot of projects are created and evolve
around a specific engineering concern this happens a lot we see this when we
look at any of the landscapes in on like the cncf organization or the CDF
organization a lot of these are grown in isolated environments and what you really need is
to get more and more people contributing looking at it understanding the root of the problem and also inputting why it
doesn’t work for them or what they need in order to be able to use that project
super important to get everyone involved the uh this particular organization did
quite a bit for us they established some things in 2015
they you know put out their runtime spec their image spec uh their distribution
spec um Docker had a lot to do with those specs I’ll talk to you a little bit more
about those contributions in a minute July 17 is when the runtime and image
spec was actually released note how long it took to get agreement there this is
not a simple easy process May 4th 2021 they um their distribution
spec was released as well version one
month after the oci was established this is when the cloud native Computing
Foundation or the cncf was established the cncf does a ton there’s a lot of
projects that are either incubating or graduating under that umbrella but there’s a few activities that happened
that we know we need to know for our purposes using containers
uh they uh you know along with developments in the container ecosystem
orchestration was also in Rapid development orchestration like kubernetes
took on the responsibilities of that coordination configuration and management of containers in deployment
environments so those of you that are using containers in your production environments how many of you are using
containers or kubernetes to orchestrate them cool how many are using something else
can I ask what you’re using Nomad okay cool
cool all right yes
swarm okay that was curious if anyone was using swarm cool
so kubernetes version 1.0 was released at the same time the cncf was announced
I mean that was kind of the driving force kubernetes was contributed by Google it was the first project that was
brought to the cncf anyone know what the latest version of kubernetes is
it’s 1.25 .3 they have a patch version out they
try to release at monthly now if you look at their release history it was kind of scattered in the beginning but
now it’s getting getting pretty regular right on the heels of that first release
though version 1.5 came out and that included the container runtime interface
and there’s that word again run time so what exactly is that what what is the
CRI and this is important to understand the CRI is a level of extraction of
abstraction that allows kubernetes to support alternative low-level container
run times so Docker the company and also a member
of the cncf they were well on their way even back then to breaking up their huge
Tech stack known as Docker and they contributed a bunch of stuff they
contributed their CRI compatible runtime called container d
that run time was developed in order to integrate run C which is a lower level
run time into Docker version 1.11 so this has been around a long time this is
not anything new and remember when we were looking at the you know the marketing stuff the market
moves concerning run times and we saw you know container D is starting to encroach on Docker this is why this it
makes sense right this is why we see container D replacing the old full-fledged Docker stack in the runtime
world okay that’s enough history
let’s get into some Anatomy here um My Hope Is that in learning how everything came to be
um you won’t be bogged down with all of these you know project names and and spec names um
but uh there’s still some terms that people get confused about at times and
really it comes down to not understanding what the docker product as a whole actually is versus a generic
container or a generic image so let’s get into that a little bit
there are lots of things that developers need when they’re dealing with containers and images when they’re
working on them locally so before I go any further everyone understand the
difference between a container and an image sometimes I use those interchangeably
yes are we good there okay the docker stack does all of these oh we
lost our stream here
is it going to come back I hope so
we’re back all right the docker staff does all of these things defining a container building an image of a
container managing those container images on your machine Distributing and sharing those containers images with
other folks creating a container environment getting it ready to launch and actually run and managing the life
cycle of the container shutting it down when needed restarting it if necessary
and note that when you hear talk a container run times like we were discussing in the marketing material
earlier or image formats what’s being discussed there is just a tiny piece of
this whole package here so this is why
Docker kind of slept the market in the beginning because they offered they made it easy they made it easy to create and
develop these images launch these containers in a local development environment you know
it’s good when you make it easy people want to use it and that is exactly what happened
so this can talk about container run times a little bit more we need to spend a little bit more time there
um the whole Docker stock as I showed before contains a lot more than just the
run time that’s the part that’s responsible for you know launching and running the container Docker did quite a
bit of reorganizing their code base over the last few years they developed a lot of abstractions pulling discrete
functionality out and one of the results of that effort was a project called run C which was contributed to the oci
organization this was the first and for some time the only implementation of a
low-level container runtime that implemented that oci runtime spec that
was released lots of discussions about that and why that is why do we only have
one implementation um there are some other runtimes out there
and um this is a pretty active space so um there’s a list that the oci maintains
of compatible runtimes so keep an eye on that see if you see anything else pop up there but notable low low level runtime
projects include I and and I apologize for not looking
this up or asking it’s either C run or cron it is written in C so I’m tempted
to call it C run um but this is an implementation that is in
C and it’s led by Red Hat um then I found a rail car which was an
implementation in Rust I thought that was interesting that was led by Oracle but that project’s now archived so I’m
not really sure what happened there why that stopped developing a spec is uh pretty
challenging and collaboration on that oci runtime spec wasn’t any less
challenging of course it took a while to even get a release out figuring out where the boundaries are what should and
should not be included in a spec took quite a bit of time before that release of version one
but it came pretty clear that just implementing the bare minimum of the oci
runtime spec isn’t enough to drive adoption especially Among Us developers
we need additional features we need you know developers are concerned
with much more than just the bare minimum of creating and running a container we need all the other stuff we
need distribution you know we need a life cycle management we need you know all of these other things so this leads
us to higher level run times they kind of have a little bit of overlap in some cases that’s where you get runtimes like
container D and cryo which you may have heard of those these runtimes include solutions for
many of those other concerns other than just starting and running a container they include some solutions for image
management and distribution but both of these run times actually implement the
CRI which eases that path to a kubernetes deployment and both delegate their low level
activities to oci compliant low-level runtimes like run C
which is why you see that still active in our ecosystem
okay uh kubernetes deprecated the docker runtime
when this news came out there was a lot of panic you know oh no Dockers going
down a lot of misunderstanding what this actually means um this is actually a good thing that
happened and I linked to a really good blog that explains in detail what was actually going on here in the mindset
why this is happening but in short the deprecation of the docker runtime support does not in any way shape or
form mean that Docker as the company is going anywhere or losing support in
other ways not that alone anyway you can still use Docker stuff to develop and
produce Docker images and containers as you do now makes no difference on your
development environment but from version
1.24 onward you’ll have to use one of the other
supported runtimes such as containerd or cryo in a kubernetes environment in
order to run these these images
so keep that in mind the reason behind this move was just to
eliminate some very specific code that was around
running a Docker container using that runtime versus using the you know
the the runtimes that are oci compliant that everyone collaborated on
so good stuff all right um last bit of info here
um just on projects around Docker you might hear of Moby if you get involved with you know containers at all you’ll
probably run into this it’s basically an open source project that was created by Docker to help accelerate to everything
in this containerized world it provides a Lego set of toolkit components all the
bits and pieces that you might need in order to produce your own product and
this is exactly what Docker did they they collected a bunch of things wrapped it up in an in a
a product to sell an Enterprise product and that’s what you see so a lot of the
code that’s involved in Docker is actually open source it’s just packaged
very nicely to make it easy for people to use
okay any of you um and I’ll go through this pretty quickly we don’t have much time left but
any of you actually put a container or run a container on your system and actually get in down
deep and understand how it’s stored on your machine yeah just a few
yeah I mean I don’t know if I just got bored one day or whatever but sometimes
I just want to know I want to know what’s happening you know I want to know what is actually getting installed where it’s being put how it’s actually running
and it will help you understand how Docker caching works as well and how the
layers work so getting in there and really understanding that makes a big
difference first of all namespaces are important that’s what’s
used to provide virtual separation between the containers this is how processes inside a container don’t
interfere with others you can see some name spaces there that were created this this was all running
on my local Mac so there’s that caveat there interesting to see how how it looks on
other machines as well the c groups functionality that essentially constrains how much a container can use
things like CPU memory bandwidth Etc this is some file system details if you
run a Docker info on your machine it will spit out for you a Docker root directory now on a Mac this root directory is
actually running in a tiny VM so you need to be a little bit clever on how to get in there to navigate around and look
and see what’s there and I put a command here that Docker run interactive
privileged using a Debian image in order to get in there and look around just it’s pretty reliable to use that
um doing that you’re able to get into the directories
and see exactly how things are stored you’ll notice you know when you inspect
images you’ll notice that there’s an image ID that is used this image ID
shows up in various other places there are specific directories that are
used for um keeping the layers that are read only versus the layers that you can actually
write to and then those are merged together so that that’s what you see when you’re actually running a container
definitely worth looking into those uh note that the container IDs of
running containers they actually match up with container subdirectory names I
thought that was interesting something to remember here if you stop a container and you’re just manually doing
this that corresponding directory doesn’t automatically go away until that container is actually removed using the
docker RM command now lots of orchestration will take care of this automatically for you but when you’re just dealing with images on your machine
you might notice you’re running out of space pretty quick if you don’t take care of this yourself there’s a Docker system prune command
you can use to clean up stuff like that you can also you know use a flag when
you’re launching containers to make sure that you don’t forget to remove everything when you’re done
okay lastly just a few links I have links scattered all over the slides that you can go check out these are just the
basics Docker documentation has improved immensely since when I first looked at
it so definitely worth a read there’s lots more examples in there for different languages different ways of putting it all together there’s a good
community and I put the slack link there for the community and then of course the cncf
and the opencontainers.org if you get really curious and you want to dig down deep into it like for example what we
saw those layers that are being stored and how those hashes are used and how caching is used
um the specs available in opencontainers.org there’s a link to their GitHub repository those are really
interesting to look at and understand you know what is the difference between like a Docker image versus an oci image
and it really comes down to like one little flag in the in the specification
so very interesting stuff all right that’s it that is all I have for you any questions