Who Broke the Build: Using Kuttl to improve E2E testing and release faster | Berlin Buzzwords 2023

good day everyone welcome to my talk on who broke the build
nobody wants to be responsible for breaking a bill but what can you as a developer do to
develop it to avoid being a bad guy how can project leads especially
managers want the teams to reduce the breakage of bills
how many of you are managers here
have you ever seen this
okay so the manager would ask who has actually broken the build
then the developer would say I would check because I don’t know who has break on the bill in my team
probably someone maybe me as well so let me check and come back okay
okay quick introduction about me myself Ram I’m a senior software engineer at r d
Bangalore India I’m passionate about open source I love playing table tennis
and just a quick introduction of what our company does so that you would also get to know about us so jfrog
headquartered in Netanya Israel founded in 2008 public NASDAQ listed
around 1100 employees we have around nine location and expanding we have around 7 000 probably
last quarter six products six and expanding all of
them are oss2 hybrid model and we have we have a universal jfrog platform
devops platform and most importantly we contribute to the open source
okay as you know software is everywhere and it runs on everywhere
Healthcare probably social media Transportation
energy wherever you
okay does software upgrades matter really yes say you have a Tesla Model
and it says
during the upgrade process you will not be able to drive the vehicle so it means upgrades are necessary and
you go you won’t be able to drive your Tesla
so jfrog vision is to power all the software upgrades in the world
I mean to be simple I mean the word might take simple but it’s
huge I mean wherever you have a software it’s related to an iot devices it may be
related to software mobile devices or cars you mean
so it’s been trusted by Enterprises most of the companies you see use jfrog
for their software upgrades
and we have a developer ecosystem around us we work with AWS Azure Google and all
the other partners to collaborate with them
so let’s get let’s get started with the with my talk around how software development has actually
evolved so as you know most of you might have followed a waterfall model and then
moved into an agile model where you do a delivery probably in a weeks or a month’s time
and then we have a devops model where it’s not to do with the collaboration between different teams
and the major difference that I see is instead of doing a monthly or quarterly
release we do releases every day that’s the major difference when you move on to a devops model
so let’s get a quick agenda of my talk how development environment is actually
necessary and how developers can actually leverage some of the tools to
do end-to-end testing how many of here does end-to-end testing locally or somewhere right into enters
and how do you test them when you mean docker
[Music] okay
okay okay and we would talk about a tool called cuttle kubernetes test tool which is a
cloud native cncf project and a quick demo and a summary of the
things that we discuss okay so an ideal development environment
say I being a new developer into your team when I join a new team
the major task for me is to set up an environment but most of the time you would look up
for a Wiki page to do that when I mean a Wiki page look up a documentation copy
this stuff copy some commands run those commands set up the environment which might take
probably two days at least two or one or two days to actually set up the environment
but we were looking at how can we actually be more productive in setting
up an ideal development environment something like a single click setup and you should be able to develop and
test locally and also it should be as same as production environment
okay so dude for having a ideal Dev setup we need some automation around
that so which would mean instead of actually
copying and pasting the instructions on a Wiki page you can do some Automation
and write scrum scripts so that it saves time and also a few hours as
well and when you have no manual steps which would mean error free
and also quick reload say I develop a feature I want to test it so that I
don’t want to deploy it on a remote server and test it out so I want to quickly reload my changes on my local
thing and then push the changes
as I have talked about the Java environment should be as same as the production environment
the main idea around this is reproduction of production issues locally say we have a bug or an issue in
the production and you would like to replicate the same thing into device environment most of the times that is
not possible okay developers would run into a different system they develop it on Mac and they deploy it on Unix or
Linux on some operating systems they don’t have a debugging environment
okay so let’s understand the problem currently as part of the talk
so when you do a feature development I create a local branch
write some code around that around that feature and write some unit tests see if it
everything works then push or commit the changes I but most developers don’t write end to
end test I don’t do that previously and then commit the changes and put to
push us to the git Repository so the problem statement here is when a
developer commits some changes pushes the code there might be a chance that end-to-end test runs on a remote server
and the these tests take probably couple of hours maybe a day
to see all the scenarios that are being tested say that a test failed due to the commit
there might be due to a bad environment as well but let’s say
the end-to-end test failed due to our bad commit then the return time the round trip to
come back to the developer is huge when you have these end-to-end test remotely
so how can we avoid this is there any way
okay So currently the remote end-to-end test that we talk about once the developer writes the code write
some unit tests say something fails he fixes them again then
pushes the code commit set then on the CI remote CI CD server he runs those tests says
something breaks there the round trip here is huge to get back
so is there any solution for this so at jfrog we were evaluating is there
any tool to actually do those end-to-end tests locally Weaver Cloud native
company and we were evaluating some tools open source okay before I introduce the tool let me
talk about how those remote end-to-end tests can be delivered locally as well
using local end-to-end test so with that whenever I see some runs
some end-to-end test locally if something fails locally I would be able to commit
amend those changes and push it so whenever you do some changes locally
you push it you commit it and then run the test so if the tests fail again just amend
those changes fix and commit it this way reduces the round trip so that end to
end test run completely on your local laptop you don’t need to worry about whether an end-to-end test environment
is failing due to some environment issues or not okay so my suggestion over here is bring all
those remote into remote tasks locally okay so the change here is instead of
doing a push to a commit and amend those changes
okay so let’s talk about so we were evaluating some open source tools we
found a tool called cattle cattle is kubernetes test tool
and let’s
and what we discovered when we when we talked to our developers locally in our
in our company they are not running sufficient and integration or end-to-end test
the reason is simple they don’t understand the infrastructure around how to run the end-to-end test and they
don’t know how to reproduce the same production like use cases in there locally
so so when we started discovering this tool
we thought how can we leverage developers to write end-to-end tests on their test code itself rather than
depending on some external environment to do that
so kubernetes cattle is a kubernetes test tone this is primarily a test
framework to write declarative test mainly for testing kubernetes operators
or anything to do with kubernetes okay it’s a yaml based declarative
tool and by using yaml base said I’m a go
developer or a Java developer or a python developer you don’t need to worry about the underlying
underlying language that you use you can still use curtain to test your
infrastructure or test your changes code changes and this way it creates an ability to
set up an end-to-end cluster very easily okay okay so to get started how do I install
kettle if you are a Mac User you can do a brew tab Kudo Builder tab
and do a brew install control CLI if you are a Linux user you can use a
cube CTL crew install kettle and if you are a go developer and if you
want to integrate this into your code directly you can still use a API
integration directly go get GitHub codo Builder cutting so the three ways to do that
you can choose whichever you need okay and crew is a packet manager for uh Cube
CTL Cube CTL packet manager so cattle is both a CLI tool as well as
an API testing tool okay so
where would cuttle be useful okay so if you are an application admin
who wants to automate the creation of new kubernetes environment say for example I’m a go developer I want to set
up an kubernetes environment spin up an environment create a namespace run my test and destroy those things which is a
painful process if you want to write a go code okay create a namespace go
testing is also a module that would provide that but writing those go code compiling it and running those tests is
a painful process okay so you can use a curtain which would
actually provide creation of namespaces creation of an environment running those tests parallely as well okay and if you
want to test your application on multiple versions of kubernetes for
example if I want to test my on test my application on 1.19 of kubernetes 1.27
of GA version of kubernetes which was released very recently you can still do
that you can just provide the kubernetes version if you are using a kind kind as
a cluster you can just specify the version of kind that you are trying to install okay
is also for developers who want to easily test kubernetes operators so
operators are more to do with custom resource definitions and CR CRS and crds
okay so let me quickly get into the how cattle would look like that the cattle
testing suit would look like so we have three things one is a cuddle suit we
have a test step and a test assertion three main things around it this suit is
a configuration where you have you define the entire project structure how
how my suit would look like so so you can see here the API version is cuttle
Dev V1 beta1 so it’s more like a crd custom resource definitions and a kind
is a test suit you can see the kind start kind so as part of curtail
kind is by default true if you want to disable and use an external kubernetes
cluster to test your changes you can still do that by setting kind start kind as false okay if you don’t
Define anything over there it uses the local Docker desktop on a Mac and
install using kind as a cluster default cluster in the namespace that I’ve
specified is just a name sorry name is end to end and the test directory I
would show you at the end with the demo and what I’m trying to do is add some
commands so that I set up my environment as a prerequisite and then run some test
on top of it so this is a test suit configuration test suit is more like a collection of
all the tests that you combine in next is the test step so the test step
is you can see API version is cattle Dev V1 beta1 this is also a part of the crd
custom resource definition and the kind is test step instead of a suit this is a
step okay and you can see you can run any commands that you like okay you can write your
own shell scripts that you would invoke so if you have already have a shell script this is more like a skeleton that
you are including it and just running your test suit
next is the test assertion step say for example I have written a test I would like to verify that it is behaving as
expected so what I would do is I would do an assertion on based on say for example I wanted to
install an application and then do an assertion whether if it is a stateful set application you can do a stateful
set and set replicas and see the ready replicas are one okay
okay so let me quickly Give an example and see how the test structure would
look like so as part of the demo I would have a folder called test under which we call
it as end-to-end test and then have different folders have you ever worked ever
worked with something called Flyway yes yeah flywheeler has a similar
structure where you define the database migration scripts in a sequential order okay which is very similar here where
you would Define the steps or the test in a sequential manner see here 0 0 0 1
you can add that I would show it in an example and an install and scale so have two
tests first is the install another is a scale step okay and cuttle test.yaml
file is the suit file that we configured
so let me quickly
switch back and give it demo on what we have
you can see that my screen right so I have some end to end test
and I have four tests as part of the example that I have one is the install
and there is an install with a extra configuration that I would like to do another is a scale test and there is an
upgrade test so four tests that I would like to run parallely
so cuttle supports running of these test parallely by default it supports eight parallel tests okay that you can
configure if you don’t Define anything it would expect to be a parallel one
okay and you can also configure the reports as part of the running to running those
tests so if you want to if you want to run these end-to-end tests and see a report in an XML you can
you can probably pass a flag passing it as hyphen hyphen XML or hyphen hyphen
Json so let me quickly run a test and then show you
so Cube CTL curtain test so what it would do is
let me show you the configuration file that we have so this is a configuration file
suit file where I have set the kind start kind as true even if it is set to
even if we don’t Define that it would be defined as true by default it’s true and
the context I have set it as buzzwords it’s more like kind context you can ignore that and you can see line number
12 there is a timeout so 900 seconds is the timeout that we have
if the test doesn’t complete in 900 second it would fail okay so let’s see what it does
before we get into that let’s show show what tests that we have
so I have installed install as a split scale and an upgrade
and an install would do just an install of an artifactory and then do an assertion to see if
replicas are one what I would do as a split is there is a
concept of split containers services to containers in artifactory I would show you
where I would set this flag to true and then do an assertion and the scale thing is
I would do an install and set the replica account as to
and do an assertion whether two replicas would come up Okay the third test the
fourth test is an upgrade test where would I would first install and then do an upgrade okay with a
different replica account okay so let’s get to the console and see
Where We Are so you can see here I have run Cube CTL
cartel test it has created a kind cluster and running those tests so let me export this kind cluster
configuration and see this running test in a canines before I do
that let me export this qconfig file
okay K9 is a tool to view kubernetes clusters it’s an open source you can do
a brew install K9s that would install K9s it’s a graphical view of viewing
kubernetes clusters so what we have done is we have ran four tests so each of these
four tests would run in a different name space and give us the result so you can see four namespaces are being created
cuddle tests are harmless thing this one is another namespace
let me quickly show you all the namespaces that we have okay
so if we have four namespaces four tests running parallelly
it would roughly take around six minutes that are previously done so what it does let let us go into the
one namespace and see how this is getting installed so you can see it is trying to install
artifactory on the latest version is 759.11
and this in the next namespace where I’ve Set uh service split to
container services to True where we have all these micro services in artifactory
running with true enabled so this is not the default configuration so I would like to
test this configuration as well the next thing is the
I’m doing a install scaling up so first do an
install and once this is successful the second replica account the second
replica would also come up okay and the fourth namespace we are doing a
upgrade test so if we would first install artifactory and then change the replica account to two Okay so these are
four tests that would run parallely and we could see the logs over here as well
so cartil provides say logs in such a fashion where you can see
which test is actually running and what locks you can see you can see cutting says harness install this is the install
test logs okay and and you can see upgrade zero one
test so let let me go back and show you I’ve referred to Flyway here previously
so we have these versioning similarly to Flyway where
you can see there are two internal tests that would run in a upgrade scenario
okay these are steps I would call it and each step 0 0 install would run first
then zero one install would start first flywave follows the similar uploads of
doing your database migrations cartil also follows a similar approach so if you have 10 steps as part of a task you
can actually version them in such a fashion 0 0 1 0 1 0 2 0 3 0 2. okay
so let us go back and see where we are on the test path okay so some of the tests are running and some of the
uh are done in the meantime I would like to show you the while the test run and give
us a report I would like to show you the documentation around cattle okay
so we have a detailed documentation on Cuttle and the prerequisites you can have a
kubernetes cluster minimum of 1.13 and a cube CTL version of 1.39 above
how can I install curtain you can use cuttle using Brew install or or whatever
that I’ve referred in the previous slides and there is a CLI usage how the tests
would look like writing your first test crew install cattle is more like Linux
way of doing this and there is an API integration as well
where you can integrate this into your go code as a library and then start
writing cuddle test inside your go code as well
and the steps so this is a detailed documentation where
this is sponsored by cncf and
which is under a name called Kudo Builder Kudo was primarily focused on
writing a declarative way of writing operators kubernetes operators this tool is part of their testing effort in
testing those operators okay that was the primary reason how cattle cattle was developed okay and which is very active
this this release Cycles happens every probably two months or three months and we have
Kane who is an active contributor and we work with him on any improvements on
that so jfrog doesn’t actually work on this Kettle we only use this tool
internally to test our kubernetes operators as well as kubernetes resources
so let’s go back to the C and see where the tests are so you can see all the
namespaces got created and also being destroyed in the sense tests have ran we
would see the results how they are now okay let’s go back to the logs and see you can see
you can see here cartel has run four tests
and first is an install next is an install underscore split third one is the scale fourth one is an upgrade and
the four tests are passed so with this I would be very confident that whatever
the changes that I’ve done in my code along with these end-to-end tests would be committed to a git repository would
work on that environment as well okay that’s the confidence that a developer needs when when he writes such code and
this is a tool or a framework that provides developers with such an opportunity to
enhance their time and as well as their productivity
okay I would quickly go back to my presentation
okay so the references cattle has a I’ve
shown you in the git GitHub page as well as the documentation and there is a dedicated slack channel on kubernetes
which is called Kudo and you can directly directly refer or register as a free
registration and there is a tool that I have referred as canines that you would install as
you can refer to as GitHub link as well so let’s take the summary of this talk
what we have learned cattle is an open source tool cncf approved and
there are so many contributors actively contributing to it it can be used for local end-to-end
testing when you have end-to-end test run tests
running locally there which would mean developers are quite confident and there
would be very few broken bills okay there might be bills that are broken on an end-to-end environment not due to the
changes that a developer has done maybe due to some environment issues but at least developer would be confident in
pushing his score to a git repository and getting reviewed and merged to a Master okay
which would mean release faster and which would also mean happy developers okay
and happy customers as well
any questions sure
do you want to take Mike
thank you it was very good presentation I want to clarify it is accessible only
in kubernetes right yeah primarily kubernetes but this can be used right
most of the companies are actually moving towards kubernetes and Cloud native so the focus is more on that
front Okay can you please repeat your question are you referring to something like can this be used somewhere else
yeah yeah for example in ECS uh in AWS for example is there some Integrations
yeah you can use this tool as well there because we are actually not
uh let me quickly show back to the presentation and show you okay see here
see here uh let me go back see here what we are doing is we are
adding the test suit steps here if you want to add anything specific to non-kubernetes you can add this as a
shell script or run those commands and can be used so this can be actually used for any end-to-end testing Frameworks
even without primarily written for kubernetes but this can be used for any things
okay thank you
any questions
[Applause] we have been using this tool for quite a long and we were actively working with
the main contributor as well in in understanding and how testing can be
improved and productivity can be improved as a developer
okay thanks everyone [Applause] [Music]
thank you