Who broke the build? Using Kuttl to improve E2E testing and release faster | Atlassian Unleash 2023

foreign
[Music]
welcome to my talk on who broke the
build using cuddle to improve end-to-end
testing and release faster my talk is
about who broke the belt no one wants to
be responsible for breaking the build
but what can you do as a developer to
avoid being the bad guy
how can
projects leak enable their teams to
reduce the occurrence of broken bills as
you can see
manager is pretty angry with
who has broken the belt
but let’s see what developer says
developer says I don’t know who broke
the build but I will make I will make
sure that I would find and make a fix
for you
but what can a developer do to avoid
being the bad guy
about me foreign
I’m a software developer
R D J frog India
I’m passionate about open source
primarily Linux
I love playing table tennis
a quick overview of my session I would
cover a quick history of our testing
challenges and what led us to cuddle
and the benefits of our new testing
approach which is easy to configure and
minimal investment
and how do we combine cartel and CI
pipelines for more streamlined testing
and few fewer broken builds
so let’s get started
an ideal development environment is
which where which can be easily set up
using a single click primarily using
automation
an ideal development environment can
have can be leveled up developed locally
and tested locally as well
it should be very similar to production
environment as I said same as production
environment
but how can we achieve this ideal
development environment
so generally setting up a Dev
environment for a developer
may take generally day or two going
through the manual instructions in a
Wiki page so whenever a developer joins
a new team
his first setup would be to have a Dev
environment
looking at the Wiki page I generally
take a day or two to get all the manual
instructions write those instructions
see those instructions and set up the
environment but
that’s not a ideal way of doing a
setting up an environment it can be
completely automated using Automation
and which can be easy to set up
and the benefits of having an automated
setup is it actually saves time
compared to having a manual setup which
can be done probably in hours using
automation it can be done in few minutes
so when there are no manual steps there
are
no errors I would say probably error
free setup
whenever we do it we do our development
we would like our changes to be tested
very easily
so you could see when whenever you do
automation you can actually quickly
reload your changes test it locally and
just see everything works
the main advantage of having a Deva
environment which is very similar to
production environment is it saves a lot
of time in reproducing production issues
at ease so let me cite an example say
there is a production issue and you
would like to replicate that
same in your Dev environment if you
don’t have a similar environment as
production I know the the pain points
that you would go through setting up the
configuring the environment so
the main idea around having a Dev
environment exactly same of production
environment help us to
reproduce the issues and when you can
reproduce the issues you can actually
fix them and release faster
so let let us go through the problem
statement that we have so whenever as a
developer you start creating a
feature development you
let me explain the steps that you need
to go through to raise a pull request on
that feature
so whenever a developer starts with the
feature development he works on a new
feature by creating a new feature branch
then here writes the code based on the
requirement
maybe a high level document or an lld
kind of a thing then
he writes some unit test based on the
new feature that is being developed then
he adds some unit tests then testers
tests locally and then commits those
changes locally if he uses a git
repository then he pushes to a remote
git repository okay
then
usually when I say end to end test e to
a test refers to end to end test here
and these tests are actually performed
on a remote CI CD server
remote CI server
so whenever a developer commits his code
and pushes it to a git repository we
have a remote CI server which triggers
these end-to-end tests say for example
if these tests fail due to a bad commit
then developer comes back to his code
fixes that and commits that and then
pushes it again
say that the end-to-end test fails again
then the
then he needs to fix and commit again
so this round trip continues
I’m wasting the developer time in
releasing a product or a feature faster
so
only when these end-to-end tests are
successful Ein and
mode request or a pull request is
successfully approved then only it gets
merged to a feature Branch gets most to
a Master Branch or a main branch
so let us briefly get into the
end-to-end architecture
you can see over here when a developer
starts a feature development this is a
graphical representation of how
end-to-end tests are run
so when the developer starts a feature
development he writes
a specific code then unit test if
something fails he would fix that unit
test and
he would make sure all the unit tests
are successfully run in his local
environment
and then he commits and pushes the code
to a remote git repository that’s where
that’s when a pull request is being
raised when a pull request is being
raised
you can see the remote CI CD server runs
these tests okay
and if these tests fail you need to go
back
the developer needs to fix those
tests
and commit and push it again so this is
a round trip of fixing one issue so how
how can developer avoid these end-to-end
tests which which actually run on a
remote server
to be easily run is there any way of
actually testing having a setup very
similar to end-to-end test locally let
us see how can we actually Leverage
end-to-end -to-end test that can run
locally so that developer can test these
end-to-end tests locally and able to
successfully test his final code changes
before actually pushing the code
we have a problem statement where I have
actually briefed that
since we run
end-to-end test on a remote
remote server the round trip of fixing
that issue is huge I mean the time that
gets wasted of in fixing the issue is
used instead of that I’m proposing a
solution where why can’t we run these
end-to-end tests locally instead of
running these end-to-end test remotely
use local end-to-end test
which would mean you can actually
commit the changes locally to a feature
branch and then do end-to-end testing
so let’s
take an example over here so instead of
actually actually committing and pushing
it to a git remote server
you can actually
use
git commitment to those changes where
this has actually fair so let’s take an
example over here
so whenever a feature branch is created
he writes developer writes the unit test
as well as writes add some end-to-end
test based on that feature
have a local end-to-end test environment
he would be able to test those
things locally if test fails he can
actually
fix those using git commitment
and this way instead of actually pushing
the code and running it in a remote
server you can actually reduce the round
trip of fixing these issues which would
eventually save time developer time
developer productivity you can actually
focus on new features
and it actually improves the ability
team to release faster when you have
fixes few broken builds you can actually
release faster so let us go
how can you actually leverage these
end-to-end tests locally
in the previous diagram that I have
shown where remote end-to-end tests are
run you could see these end-to-end tests
are actually run in a remote CI server
in this diagram you can see instead of
having a remote you can have remote
tests as well in in addition to that you
can actually have these end-to-end tests
locally you can see in my diagram
we have both end-to-end tests that run
on a remote server as well as in the
local server as well
so while evaluating different tools uh
and more specifically on kubernetes we
were evaluating uh how can we leverage
end-to-end test Frameworks in talking to
our teams we discovered that many
developers weren’t running sufficient
integration or end-to-end tests in their
local
environments because it’s very because
it’s too difficult to set up and admin
administer test environments in an
efficient way
that’s why we decided to rethink our
entire local testing process in hopes of
cutting down
the headaches and valuable time wasted
we discovered cattle which is a tool
primarily for kubernetes testing
operators
we have connected cattle to our CI build
that has empowered our developers to
easily configure it ever environment
locally that accurately matches the
final test environment
so let us dig deep into cattle what is
this cutting what is this framework all
about
cattle is a kubernetes test tool
framework
it’s a toolkit for writing test mainly
designed designed for testing operators
and controllers however it can be
declaratively test any kubernetes
objects
as you know testing kubernetes operators
is not easy the motivation here is to
leverage the existing kubernetes
ecosystem for resource management yamas
here
in a way to set up test as well as to
way to assert test within the cluster so
this Kettle is a yaml based tool where
you can instead of writing a gold code
or a Java code you can just write yaml
based
yaml based structure and
we would see in the later part where I
would demo how this Kettle would be used
to
test a kubernetes objects
it has few more benefits like it
accelerates the ability to create
end-to-end testing in a kubernetes
environment
so let’s get started
how can you actually install cuddle so
if you are if you are a Mac User you can
just you
use Brew Pro install cuttlecli it would
install that I would show you at the
demo part
little later
uh if you want to use a
native way of installing curtain you can
use crew crew is a cube CTL packet
manager
you can use cubectl crew install cattle
and if you are a go developer and if you
want to integrate cartel to your
end-to-end testing framework you can
still use that using go you you can use
go get github.com
[Music]
Builder Kettle that way you can add
cattle as a library into your code and
start writing your test in my
in this presentation I would more focus
on cattle as a CLI
cattle can be both used as a CLI tool as
well as an API for testing
end to end or integration tests
cattle is for those if you are an
application admin who wants to automate
the creation of a new kubernetes
environment or if you want to test
kubernetes application on multiple
versions of kubernetes
or if you are a developer who wants to
easily test operators without writing go
code
not only operators but also any
kubernetes objects
using cartel we can actually write
portable end-to-end integration text
integration test
for kubernetes without needing to write
any code implies which is very native to
kubernetes most of these kubernetes
object or resources are yaml based
so let us dig deep into how cattle can
be used
so cattle has primarily three uh three
things one is a test suit
is a collection of tests
this is a test configuration file that
you are seeing on the screen
you can see it has an API version it has
a kind which is a test suit
uh so when you want to run this test
suit you can actually use an existing
cluster or you can use a kind cluster
for example see here I have set kind
custom kind start kind as false so which
means when I run these end-to-end tests
on a kubernetes cluster which would make
use of an existing cluster not use kind
for testing
or if you want to use a kind cluster for
testing you can set start kind is equals
to true
that way it uses kind for testing
name is just an end-to-end test
and you can see I have a test directory
called e to E
where I’ve specified the test directory
path
test suit also supports native commands
something like you can use any bash
commands for example I’ve used a Helm
repo ad next is as I’ve said cattle has
three primary things one is a test suit
one is a test step
then there is a test assert let’s talk
about test step
so when whenever you want to write a
specific end-to-end test you can have n
number of test steps
so this is an example where
where the kind is tested you can have
the commands that you can run to
actually install for example I’m running
an Helm install of an artifactory over
here
once you do an install you want to
assert those changes so cuttle provides
us a
test step assert where you can actually
uh
asserts specific condition say for
example I install s
resource and I want to test that
using kettle you can use this assert as
part of the files that you have and this
assert would actually
test the current status so for example
the status that I’ve set has current
replicas as one and ready replicas as
one so if I go back I’m trying to
install artifactory here with one
replica defaultly it installs one
replica then I would do an assert
artifactory is a stateful set
application where you can see I am
trying to assert the replicas here when
artifactory installs
and I would use this asset to check
whether the ready replicas are one or
not
if we specify wrongly cattle will
actually throw an status as failed
I would Deep dive into how cattle works
with the demo at a later part
so let us see the complete test suit
structure of cattle I have a demo
application I have some tests that I’ve
added end-to-end test you can see the
folder structure here
it has two folders one is installed
other is the scale
so
say that you are trying to install an
application on kubernetes and if you
want to do an assertion and Ascension of
the basic install and then scale that
kubernetes application
using curtain and then
assert that assertions
so let us quickly Deep dive into a demo
so let me as I’ve told you there is a
demo app that I’ve created let me show
you the structure
I have an end-to-end test that I’ve
added it has two tests one is install
other one is scale test
uh let me add the
total test configuration file
you can see here I’m running a test suit
with a kind set to true so this
end-to-end test would use kind cluster
and you can see a kind context this is
the name of the kind cluster that you
want to install
and there is a name end-to-end test and
the test directory that I am using is
end to end directory
this is the same configuration file that
I’ve showed you in the previous slides
let us quickly run this end-to-end test
using cattle so
Cube CTL Kettle test would actually run
the test say that I would like to only
run a specific test rather than entire
suit how can I do that
so
so if you want to run a specific test
with n number of test suits
n number of tests you can actually use
hyphen iPhone test installed so what it
does is
cattle would run this end-to-end test
only this end-to-end test not the scale
test
let us quickly run this and see how it
works
so you could see it is starting a kind
cluster let us get into that kind
cluster to see what’s happening so I’m
just exporting a cube config file which
is created by cattle
I use a tool called K9s to view the
kubernetes resources
you can see here it is trying to install
it is trying to run the
test suit
and with this creating a specific
namespace so it has launched a kind
cluster and then created
a namespace you could see it is only
running a single install test it’s it
says to test but it would only run a
single test because since we have
used
hyphen iPhone test install
so which would mean it would only run
single test
so it is trying to install artifactory
it would take some time but let us
quickly
quickly go and check what is happening
into the install part
you can actually see the logs over here
but you can actually if you want to get
into the cluster and namespace and to
see
uh what is actually happening in the
install you can actually use this
canines tool get into that specific name
piece and see the logs
and would get into the logs
you could also see what all the
containers that we have
we have
one container and other init containers
you can see init container as false and
other rest of the containers as true
this is more like a graphical view of
uh
seeing the kubernetes resources
katil would also have a
timeout of 300 seconds
foreign
let’s see whether artifactory is running
you could see still it’s
the status is init containers
so let us go back
and see what configuration that we have
so this end to end test uses help in
upgrade install artifactory
on a specific namespace so by default
the assertion that we are doing is a
single replica so when artifactory
installs
we would try to see the replicas as one
say if if the replicas are one and ready
replicas are one and the assertion
becomes true then the test case is
actually successful
you can see artifactory is starting up
since this is a kind cluster the images
that needs to be downloaded
onto this kind cluster so it would it is
taking good amount of time here
otherwise if you use an external uh any
cluster it would uh
it would download very fast
pair with me for a few seconds
so artifactory is starting up
once artifactory installs an assertion
would happen which would verify the
ready replicas R1 or not
okay
you can see here
uh
it has run this test
it has successfully uh
it has successfully asserted as well and
it would delete the namespace as well
you could see here it has the test test
is passed
so let’s quickly see a failure test case
where I would set this
read replica search too and which would
fail then
then kutel would say it has failed so
say that if you want to uh
run the entire test suit
since I have only ran the install part
if I want to run the scale part as well
I can
so these tests are critical Frameworks
allows parallel running of tests so
since we have two tests let me quickly
change back
to the original code and run the entire
suit rather than running a single test
okay
which would mean it would run two tests
parallely since we have two tests that
have configured
let us quickly see
get into that
kind cluster
you can see here two rest two tests
would run parallely in two different
name spaces
so you can see artifactory
one instance is running
with the install step is running in one
namespace and the scale up is running on
another namespace let us go into the
scale of things see what we have done
so instead of installing the default RT
Factory I have installed artifactory
with two replicas
and I’m doing an assertion based on the
two replicas
this would
run both the test if both the tests are
actually successful then
it would pass the test
using curtain
which has actually provided as
an option of running the entire test
suit locally which is exact same that we
run in a remote CI CD server which is
actually saved as lot of time and
developed productivity and we were able
to release much faster you can see the
references of curtain
the documentation is referred over here
cuttle.dev slash docs kettles GitHub
page
github.com codobuilder.catel
canines
which is a very useful tool for us at
least to see the resources in kubernetes
hope this is useful to you
if you have any further questions you
can probably reach out to me on LinkedIn
or Twitter thanks for watching