GoLang Unit Testing @ GoSG (Singapore) Meetup

hey stanley
hey johnson how are you i’m good yeah
it’s a midweek uh yeah weekend
is two days away so really looking
forward
yeah it’s been a long day yeah
yeah so hi everyone uh thank you for
joining us today on
a wednesday we hope your week has been
going great
and i’m jason from the google developer
relations team
uh before we begin today’s session
organized by go singapore uh let me just
share a little about google develop
developer space for those of you who may
not be familiar
so um it is a platform for developers
and startups from around the region
to learn and connect with one another we
have a physical space in the google
singapore office
but due to covet we moved everything
online and so that’s how we started
our youtube channel so which probably
many of you are watching from right now
and as part of the google developer
relations team we aim to
empower and connect the community to our
people
programs network and technologies and we
are really excited for the awesome
speakers and sessions we have lined up
for you today
um so stanley uh yeah i think many of
you will be familiar with tech chun from
go singapore
today we have a new face stanley he’s
yeah he’s super active and he has been
helping out a lot behind the scenes
so uh yeah over to you stanley to
introduce more about yourself
and go to singapore thanks jesse hey go
first
welcome to our april meetup so uh
yeah i think it’s been almost a month
since our last one
uh shout out to our google developer
space folks who have been
working very hard with our co-organizer
me uh able and uh dick chun who have
been working hard as well
to bring you this event uh so before we
start our event i would like to uh
bring in ari who will be giving out some
awesome j frog
swags who doesn’t want swag right
uh thank you so thank you so much can
everyone see my screen and hear me okay
yes okay great hi everybody i’m ari
waller and i’m the meetup event manager
for jfrog
and we’re really honored to be here once
again at go singapore
i’m not sure if you all realize this but
this is actually the fourth jfrog
speaker in the last two years
so it really means a lot to be part of
your community um especially uh during
these trying times that we face so thank
you again for that
i’ll share just a little bit about who
we are jfrog is a devops software
company we were founded in 2008.
many people in the devops community know
us best buy our flagship product
artifactory
which is considered by many to be the
gold standard for managing your
artifacts and dependencies
so what does this mean for go developers
over three million
um software developers and devops
engineers use our software on a daily
basis
um here’s something that you may not
know yet jfrog actually has a free cloud
version of our software especially
available for the go community because
it ties in seamlessly with the go
package manager
and if you’re working with docker as i’m
sure many of you are
jfrog’s free tier on the cloud is
capable of functioning as a pull through
cache on docker hub
and here’s the best part since jfrog has
partnered with docker you’ll also be
exempt from any rate limits
on polls from free or anonymous accounts
but now on to the good stuff
as you can see on the slide um you can
scan the qr code
and i’ll drop it in the chat shortly as
well um but i think the
the bitly link is pretty easy for you
all to remember also
um and um we are going to
offer one one of five
limited actually i’m going to increase
it to 10 um instead of five
one of ten winners of limited edition
um go t-shirts with jfrog um and i think
uh
who has i think was it stanley was he
did you have one on who was who
i do yeah all right he’s about he’s a
much better model than i
uh thanks uh he’s um
and we’re going to be able to offer you
also one of our liquid software
books it’s a book written by several
members of the jfrog team and authored
so we’ll send that out to one of 10
winners and you’ll be contacted by email
if you join so we’re joined the raffle
so we’re excited to give those out to
you
winners are going to be selected in two
business days and you will be contacted
via email
and um and then we may even congratulate
some of you on social media as well
last but not least i just wanted to
share a little bit about our free
annual devops global conference that’s
going to be coming up
and that is um on it’s going to be on
june 1st through the 3rd
in in in asia and jfrog wants you to get
in for free
basic besides a sample of some amazing
speakers that we’re going to have
we’re also going to have an amazing um
ceo panel
uh from hashicorp pagerduty datadog
elastic
and of course we can’t forget jfrog uh
where
there’s just going to be some really
awesome talks uh throughout the day so
we want everyone and go singapore to
come in for free
and um there is the qr code but
as usual i will go ahead and drop that
in
the chat so that you will have that as
well but again
i’m sorry if i took a few extra seconds
but i just want to thank you again we’re
just really honored to be
a part of your community and you all do
such a great job doing that especially
in
the virtual times that we’re in so thank
you
awesome thank you so much ari i’m
totally in love with
j fox right i love the design uh so
let’s let’s let’s start our event so
today we’ll be welcoming two speakers
atkash from jfrog as well
and iman from send it so
yeah let me so
first we will welcome utkash onto the
stage
to talk about unit testing in go
thank you uh hello everyone and uh
yep i’m glad to be here today and uh
i hope you guys are doing uh very well
uh in this
middle of this pandemic so let’s get
started
so uh this there is a very famous quote
from ellen perlis and this is from the
very beginning of
the software development around the 60s
so this quote is from his talk uh he
says that
a software system can best be designed
if the testing is interlaced
with the design instead of being used
after design
so it is not a new idea it was there
from the very beginning
and i really advocate that writing unit
test
as part of your development process
and it really helps okay let’s
understand why it is
important so who am i
uh i am a go developer and uh i have
been programming in go from past four
years
in fact i started my career with the uh
with the project which is written in go
and uh
i have been part of the community
software projects in
jfrog lately but recently i joined x-ray
team
and i am contributing there apart from
that
i am an open source contributor i have
made contributions to many open source
projects like humanity csi uh longhorn
chef etc and
i’m a part-time writer i share my
learnings in a blog on medium and
i am an early morning person okay
let’s get started so
this is the same slide that ari has
shared
you could probably scan this qr code
and uh win swags
all right so these are
some of the top topics uh that we are
going to cover
uh unit test what when and why
unit testing workflows like table driven
testing behavior
and testing unit testing techniques
and there will be a short demo and a
full
after that there will be a summary what
we have learned so far and then there
will be a q and ssn
okay so uh when i started my career back
in my previous company
uh uh so
everything was tightly coupled with the
kubernetes ecosystem and containers
so you my you guys probably heard of
project open abs which provides
containerized storage for the containers
so one of the main difficulty was
whenever we are making any changes
logic which is logical we need to deploy
a communities cluster and
build a image and post it to the docker
and then deploy our changes to the
kubernetes cluster
and then test our changes so this was
this process is really
very cumbersome and which was making our
development uh cycle very slow so
uh what we have decided that we will
make our code very abstract
and we will write unit test and improve
the test coverage so that we can
test our changes quickly uh
whether it is logical uh or it is
anything so that
we don’t have to deploy anytime and go
to the kubernetes cluster and test
everything
and we can logically we can test our
changes
immediately by writing unit test cases
and modifying
so during that day i realized how
important unit test
unit testing is for any project and i
really advocate
everyone to start unit testing uh as
soon as possible because it really
reduces the chance of making mistakes in
your project
okay so what is the
unit if you ask this question to people
they will
probably say a unit could be a method
a class okay a package
anything they will name as unit
but in go you design your test case
keeping
package in a mind as a unit so in a go
a package is known as a unit so you have
to design your test cases accordingly
so this is this is just a basic
definition of unit testing
it is a way of testing where you mainly
focus on the behavior of your unit
rather than you go deep and
spawn some of the environments and then
test your changes
so it it it basically means that you
have to test the behavior
you will provide some input and then you
will expect
some output that is matching your
behavior or not that you had expected
okay what unit tests are not so this is
uh tweet by dave channy
so he writes that it takes if it talks
to the database if you are communicating
to a database
in your unit then it is a not a unit
test case
okay if it is communicating across the
network you are it is
it is calling some rest apis then it is
not on unit test
if it is creating reading from file
system that is also an auto unit test
it cannot means at the same time you
cannot run multiple test cases
then it is also not a unit test case so
what we have to do we have to remove all
the third party dependencies or
any kind of dependencies that is
required to run your unit test you
should only focus on your business logic
not on the other logics okay you should
only focus on your business logic not on
the environment
so we will learn about this how we can
do
okay five unit tests in go
it really helps in verifying small
changes
especially refactoring and debugging
suppose uh
you are uh just now make any changes
recently and you want to validate it so
what the user processes you will uh
build your binary push it to the your
pipeline and then you will
see the logs and verify whether you have
written correct or not
but it it will take some time
you will make some changes deployed to
pipeline pipeline will take its own time
but to verify the same changes
immediately it will be better
to write a unit test and then make some
changes
do refactoring and then validate whether
it is working or
not let’s uh quickly go through an
example and we’ll come back to this
slide again
so there is this function small change
it is taking two parameters small and
change
which is string and then it is concat
basically adding those
those two strings and returning it so if
you see this
uh see this function it’s not a very big
you can easily miss there is a very uh
probability is very less that you can
make any kind of mistakes here
but what about this case
this function is taking 25 basically
here in this
constant parameter that you see insert
pull request statement here it is taking
25
arguments so chances of making mistakes
is high here
you could replace means there could be
some string data type there could be
some integers you could change its place
there is a possibility that you might do
that because there are 25 parameters and
you might make some mistakes so unit
testing helps
in such scenarios and
you you can verify your changes quickly
that you have not made any logical
errors
in in such case you can rectify it and
immediately
fix that okay let’s uh go back to the
previous slide
measures the quality of the code so how
do you
measure the quality of the code there
are various parameters
like uh you cyclomatic complexity
okay and uh whether your test is usable
or not
whether you can abstract your code or
not
so that’s how you measure the quality of
the code so
if you if you think that your code
quality is not good
only because you have not written any
test cases and unit test
case writing unit test case really
eliminates those
uh it really eliminates
whatever uh be the issues with their
code like if it is not uh
if it is having a lot of psychometric
complexity or it is depending depending
on some of the third-party dependencies
you can eliminate that
using uh while writing your unit test
cases
okay it helps in understanding the
complex logics
so recently i have been to this
part of x-ray team and
it has around 400 k lines of code
so if i would start from the main
function and
then it will take me almost six months
minimum
to understand entire project or maybe
more than that i cannot cover everything
so i cannot really start with the main
and then
learn about a project and then i will
start contributing
to it so if there would have been some
unit test and it is there so i can
quickly pick up some of the
part of the code and read it and make
some modification
learn the complex logics inside the unit
and learn about a whole project and
immediately start working on some bug
fixes or features
so it really helps there in
understanding the complex logic
helps in understanding cause of failure
quickly so as
i said earlier if you will uh write your
if you will do some kind of bug fixes
or you will refactor your code and there
is
there is chance of failure you can only
get it
while running your binary in a pipeline
or locally
you you cannot uh
you cannot know the cause of failure
immediately if if you write unit test
cases you will get to know it
very early helps viewer to understand
the
fix fixes and changes so reviewer can
also go through your
test cases and if there are any kind of
bug fixes that you have done
he can quickly go through the test cases
and find out okay you have covered
all the edge cases and he can understand
your code as well
by pulling your changes and you can
modify and understand that
it is a great way to learn about the
language as well as project so if you
are new to a language
it will really help you because you will
get to learn a lot of new stuff inside a
unit that has been used
and as a project as i told earlier
okay so when do you unit test
i will cover this at the end of the talk
okay moving on
so there are various testing workflows
like table based testing tables table
driven testing
and behavior different testing so both
of these methodologies
are uh widely used in the industry
and in various projects so far what i
have seen that
people usually use table driven testing
for for writing unit test cases
and behavior and test cases for uh
basically for functional tests
or integration tests they use behavior
even testing they use some tools
like gingo which is written go
or groovy so
we at jfrog also use the table based
testing
for unit test and we use the behavior
and testing
approach in in functional testing and
integration testing
so if you follow kubernetes project you
can also see that they are also using
the same
pattern they say end-to-end testing and
it’s
written in gengo they are using gingo
framework
so in let’s uh discuss about
ah sorry sorry for the interruption so
just to give me a moment
okay so in table based test there is a
map and slice used uh to build the
input and expected outputs okay
and basically what you do you create a
map
of inputs and then expected outputs and
then you run
then you run your unit test cases
based on those inputs and expect outputs
it is easy to implement
and easy to extend further as well
because what you have to do if you found
find some edge cases
you can quickly add you add a case and
then
extend it further and verify your
changes it
gives a clear visualization of what’s
covered and what not
so let’s say you are you are covering
you are writing some of the unit tests
and you are adding some other unit test
cases
and you want to see whether you are
covering or
covering the part of your logic or not
so
using unit test there is a flag called
hyphen curve hyphen cover so while
running your own test you can just
add that flat and it will show you what
has been covered
and what not
behavior during testing so it is a
like it’s usually the main purpose of
behavior and testing to
uh make it
in any layman can understand by reading
the description of the code
uh he don’t need to understand the
syntax of the code he can it’s written
in such a way that
anybody can go through the source code
and he can see that it’s
very much readable and it’s helpful
in let’s say there are many stakeholders
in the company and they want to
see whether what has been covered and
what not so in that case also
be heavily doing testing works they can
easily read the
description and understand okay this
functionality has been
covered it requires planning and lot of
effort because you you need to plan out
what all features that has been given
from the stakeholders
and then you will have to learn the
syntax as well
because every behavior testing framework
has its own advantages and disadvantage
and you will have to learn the syntax so
it’s also helpful in understanding the
functionality of a project
because once you have covered all the
functional tests anybody
can go through it and understand what
all the features that is this
uh covering especially to the
stakeholders
learning curve is little high because
you have to learn the syntax
okay and gingo and gomega framework is
preferred by various projects
that is very widely used and people
gingo is a
basically bdd library and omega is a
matcher library that is used by people
so let’s guide you
let me show you some example how it
looks like so there is this
function called sum which is taking two
inputs
and this is the main function from where
i’m calling
and let me share that how would bdd
testing will look like
so this is the main function uh where
i’m calling this sum
and here we have main test dot go
so describe so this is a sum function
that i am testing i am describing about
it
and there is a before each where you
initialize your variable
and then context where uh you add
some uh some specification about the
test
like what you are going to do and then
expected output
so you can clearly read it like by
providing two digits at input it should
return some other two digits
and then there is some expected input
and outputs
so let’s run it quickly
here you can easily understand that sum
by providing two digits and input it
should return the sum of the two digits
so it’s clearly uh any layman can
understand this
what this tests are doing but uh
if you run only go test
is blank right it is blank it is not
giving you
idea what what it is doing means unless
and uh otherwise
uh there is some programmer who uh who
has written this test case or
uh who knows go they can only uh
understand this otherwise this no layman
can understand of what is written in the
tests if it is written in plain unit
testing and
go but using the ability framework it is
very
easy to explain it to others
okay now coming back to the slide
so this is the same that i have showed
now
okay now coming to unit testing
techniques so
there are various unit testing
techniques like sub tests
and parallel tests so in in subtest
basically there is a one test then there
are multiple tests
on under that particular test then there
is a parallel test
where you can run test parallelly each
of these
tests parallely and then
mocking will learn about mocking as well
and uh there is a fuzzing so fuzzing
uh i will explain it later but i will
not cover fuzzing since uh
due to the time constraint so let’s
quickly jump to the sub test
and parallel test so if you see this
code
test sequences so there are three sub
tests some tests
some sorry there are three test cases
some test some other test and some any
other test
and just for representation purpose i am
not calling any function i am just
uh doing time dot sleep so if you run
this
test sequential test it will run
sequentially and it will take
three seconds to finish deodorant first
some test will run then some other test
then some any other test
whereas if you see the parallel example
test.go
i am calling t dot parallel so what it
does
it spawns each of these run closures in
a separate go routine
and runs this parallely so it will not
take three seconds rather it will take
one second based on the availability of
the cpu
okay mocking technique so
mocking technique is very debatable uh
because people usually sometimes
recommend sometimes don’t recommend
because
it adds a maintenance cost to the
to the project that’s why sometimes
people
avoid mocking and they they avoid unit
testing as well due to the same reason
and if there is a lot of uh if the
project has not been planned
uh very well from the beginning then it
is very difficult for people
to uh start writing unit test cases if
projects uh is increasing day by day
so let’s uh learn about this
market there are various mocking
techniques uh that i use
and people use it so the the main
purpose of
uh mocking is to basically isolate uh
uh isolate all the extra dependencies
that
third-party dependency that you are
having and mainly focusing
on what is the behavior of your code or
what is your behavior of your business
logic
okay it is it will not depend on any
kind of uh
third party libraries or let’s say any
kind of database or system calls
whatever you are doing in your code so
you have to mock those
and test your business logic so it is a
one-time effort but uh
really helps in verifying your logic
quickly so there are various ways to
mock uh there is a tool as well margin
which uh which uh generate mocks for
your code
so i will show you shortly
okay so one is the this this there is a
this quote uh in golang this is very
famous
except interfaces and return strip
so let’s let me show you the example
all right so this is the sample app
where i am spawning a database
there is a database database of setup
and
this is calling new user and then
inserting it
so if you see this insert user function
that i am calling this this
we cannot unit test this if means for
unit testing this one i need to
start a database right database instance
but there is a technique
the technique is i am using i am
creating an interface
and there is a user uh structure that
is implementing that interface okay
insert function
so what i will do while unit testing i
have to test
my logic let’s say here i am not
written many logic but let us something
like if there is some logic
if id equal to something
okay if i’d equal to one so
my focus when i’m writing unit test my
focus will be
testing only if i d equal to one
or any uh or if there is any other kind
of business oil that i have written
my focus will be to cover that case not
calling database and testing that
so what i am doing here i am calling
new user okay
so here i am returning uh
structure and accepting
users users is a interface
so the what what will the benefit of
this one so benefit
of this one will be whenever i am
testing insert user i will pass
the mock object of the user instance
and whenever uh mock
uh mock instance object the users and it
this will this insert will be called
this will not be calling the database
insert
rather it will be calling the mox insert
mock objects insert
method so let’s quickly go through the
test
so i have given actually two examples
there is
one that i have generated a mock uh and
here i have implemented the mock uh
there is a mob struct that is
implementing user interface uh
there was only one insert function so
i’m using that so there if you see there
are two test cases one is test user and
test user
insert user two so here i am using a
library called
gomo.new controller so i will show you
shortly
so if you see this mock database there
is a user
mock.go so this file has been generated
uh from a tool called mobile
okay so there is a tool called margin
and
here uh i can call this from previous
package yeah so this is my file
and there is a user interface and let’s
say i want to
generate a mock for user so there is a
margin tool
margin and dot means current directory
and then users
interface so what it will do it will
generate
a mock for me so if you see
this is a mock database package
and it has generated this mock user
structure and
uh this insert function uh that is the
implementation
of uh and that is the implementation of
the insert function
which is a mock one and i will be
calling the same
uh insert while testing that
so let’s test this
okay so let’s go to the test again and
explain you uh what is happening so in
test
insert user i am calling uh
i’m passing my mock user okay that
the mock implementation of user that i
have created
and in insert two there is a go mark dot
new controller that i am calling this is
the library
uh that you can use and here
uh when it will call the insert function
it will basically call mock use uh
this mock database uh mock database
insert implementation and here is the
return parameter that drawn
that i want in the return like 0 and
uh in case in this case i want error
because in the previous case i had test
the positive test case and here i want
to do the negative testing
and here i am providing times one means
it will run only one
if you again call this insert user it
will return an error
so this is one uh way of
unit testing means implementing marks
now let’s go to the another techniques
so there is this uh complex function
and it is it is uh basically accepting a
struct
and is calling get from db publish
okay and then at the end it is calling
some notify service
so uh this code is just for
representation purpose i
just uh my purpose is to explain you
what is the problem with this code so if
you see
there’s dependency on get from db which
is
external dependency on database and then
this is uh let’s say it’s using some
kind of
messaging queue like pubsub so it is
calling publish function
which will again call to the google
cloud
basically pops up instance of gcp
and then it will verify the
uh this logic so my
primary purpose is to test this logical
uh this logic
ss dot field equal to one not this one
so
i cannot unit this complex right
so what we can do here to simplify this
so what i have done i have created three
functions get db func
node notify funcs and public func and i
am passing this to
simplify okay get
notify and pub so what i will do
whenever i will write unit test case i
will provide the implementation of
get basically mock implementation of get
notify
and publish okay and
what it will do it will give the
expected output that that
i’m expecting i will return from the
from the functions in the implementation
and then i will verify
these logics okay so this way i can test
it but
still there is some problem with this
one
the problem is let’s say this library or
this function that you have used
is used by some other packages or by
some other people in their different
projects
so if you modify this function this will
uh
this will be a breaking change people
cannot use your function and
update their libraries that latest
change that you have done
okay so this is the problem with this
approach because you have to change the
function signature so is there a better
way yes
so what we can do we can so this is the
test case
uh that uh we are
so in the simplify uh that i explained
now
this is the test case for this one i
created the
function get publish and notify
and then i’m passing to simplify while
testing it
so this is how i explained a few minutes
earlier
and there is a better way to do the same
thing
that is just do a implementation of
simplify in a just wrap in a structure
and
put this notify publish and publish to
as a
uh as a part of the structure
and while testing this okay while
testing this
you can do something like this you can
create instance of some struct
you can create uh function disclosure
get pub notifies it’s the same stuff
that we have done earlier
but now this time i am putting this in a
structure
and then we are calling better.simplify
which is
testing this function this way we can
avoid uh
we cannot basically we can avoid any
kind of signature changes function
signature changes and it will not break
the library
so you can still have the same changes
but it will not break it
okay now coming to fudging techniques so
it is a testing technique where you
provide some kind of invalid
or unexpected any kind of data to the to
your
functions and see how it is behaving
right so in in some cases what you will
see that it is crashing
it is uh it is creating some kind of
noise in your
in in your setup so this really helps
in uh in testing and determining if
there is any kind of memory leaks
or presses so you can get to know about
your code and do
any kind of chaos testing using fuzzing
techniques so it really helps
in in such scenarios because if it will
go into the user’s environment and they
will
they will do their testing and they find
out that they will complain
right so it’s better to do fudging
techniques by your
uh fuzzing techniques uh by your own
and test the behavior
uh i think
okay so why the fudge so recently uh i
was working on a project and it was uh
downloading some kind of file
and then parsing it and then it is
calling some function
function that is uh basically uh
parsing that file and it is storing in a
map of interface of interface
so the problem was uh later i made
another implementation instead of
downloading a
library i downloading that file i was
passing the exact
uh exact structure that i was i
i needed so what was what was happening
there was no check
like in this case uh i’m creating a
variable of
interface and this is a string basically
so if i will do i dot string it will
pass
but uh if i will do i dot float
it will fail because it is not flow type
okay
so this is this is how they have ignored
in the in the function they were
directly
passing into the string because earlier
it was being downloaded so they know
that
the type will be string but later when
this is coming as a
this was coming as a rest api in the
later implementation
this this change was broken because it
was not coming as a string
and this check was not there it was
coming as null
so that it started panicking and the
functionality was not working so
this is why we need first testing so
that we can provide invalid
and uh invalid inputs so that it will
break and
you will know the change before pushing
it to the customer
so there are libraries that you can use
sorry i think it’s uh
my slide is not moving
okay
so there are uh actually some libraries
like
google gofus which is a testing first
testing framework for go
so it is helpful because you might miss
some
complex edge cases that i explained what
happened
with me uh and if you do
if you add first testing to your code it
really improves the quality of your
software
i think there is a problem it is not
moving to the next slide
so i will share from this screen only
okay so now coming to uh
when we should unit test our code
so we should always write your
unit test for our code sometimes time
doesn’t permits or
your manager won’t allow writing because
of the
clear timeline that the strict timeline
and you cannot uh write your test case
you are mainly focusing on deliveries
right but you should always try to write
the united test case uh for your tools
it really
improves the quality of the software
there will be less chances that you are
going to make any mistakes
and if you think it’s really easing your
job for me
in my previous company it really in in
this company as well
uh i’m i’m finding it very uh helpful
if there are any unit tests written i’m
able to easily understand
and make any kind of changes or fixes
right away otherwise i would have to go
through the entire
test code i have to start debugger
understand the
code flow then i would be able to know
and start working on a project for any
kind of
changes or fixes also if you do not have
because when you start project you do
not have
integration test coverage right so uh
writing unit test case will give you
some kind of confidence
while uh file
it will give you some kind of confidence
so that you can release your project and
release your product right when uh your
testing will be
in place it will uh help you in
verifying whether
everything is logically working or not
also if uh business logic depends on the
third party components
uh writing mock is one time effort
but it is very helpful also earlier it
was very difficult
in go to basically you will have to
write
it manually but recently just couple of
years back
there was a tool mocks in that has been
in the community from a file and people
use it it really helps
in generating the mocks and
making your job a little easier
okay so what we have learned so far
testing behavior not the implementation
we have to test
just our business logic not the
we do not have to test how database is
working we have to test how
our logic is working make everything
configurable by
inheriting them in instruct avoid global
variables
because if there will if your code is
written for
any concurrent use then there is there
might be possibility
that same uh go routine is accessing
global variables right so there is a
chance that you
may miss and there is a you you
accidentally
add races to your code use interfaces if
there are
third-party dependencies okay
because it will be helpful in mocking
okay closers helps you a lot in
extending your code further
so as i shown you earlier so
there was a get db call and then publish
uh
a call which was calling to pub sub api
then there is a call to notify
so these are all third party
dependencies so if
if if you really have to so the the
benefit of this one
today you are depending on a pub sub
later if the
uh need arises that you want to switch
to uh rabbit mq or apache kafka it will
be very
easy for you because you will have to
just give the different
implementation of the same function uh
right earlier you are using pops up now
you in some other use case you are using
apache kafka or rabbit mq you just have
to change the definition your business
logic will remain
at the same place it will not going to
change
accept interfaces and returns as i told
you it is
it really helps you in mocking avoid
mocking networking instead of make
helper functions to test the
functionality let’s say you are
spawning up spawning up a server to test
your some of the changes like rest api
and all instead of that
you just create some kind of mock
functions that will
do the job for you it will create some
fake response
or fake request for you instead of
spawning a
whole server right it will it is not
only
resource intensive also it is little bit
hard
okay in doing testing although
go itself provides some test server
which is uh really helpful in such cases
but i would say mocking is a good idea
then
creating your own server or starting a
server
all right so these are some references
that
i took help
and there are some more links this is uh
the link
to my blog that i have written if you
want to learn about gingo
and getting started with it you can
follow these laws there are three parts
and then there are goods testing is hard
only if your code is not organized well
and
writing unit test really helps in
organizing your code it
improves the quality of your code and
these are guidelines
not rules any questions
awesome thank you so much akash that’s
why that was a great talk
uh so we have some questions coming in
from
our youtube let me just pull them up so
first question from
syrian uh what are your two
what are your techniques to avoid
fragile tests
uh an example is a change causes a large
number
of tests to fail
oh can you hear me
yeah yeah yeah
okay oh i did
could you hear the question jason
sorry oh gosh can you hear me yeah i can
hear
oh yeah so what are your techniques to
avoid fragile tests
uh sorry oh yeah so uh there’s a
question
that came in from youtube uh what are
your techniques to avoid
fragile tests like uh
a change that could cause a large number
of things to fail here
yes so actually there is no such
techniques to avoid the
fragile test it it it really uh
depends how you have written your
business logic or especially it depends
on your implementation of your
methods so if you are changing your
behavior
rapidly right just follow the approach
that i have shown you if if
just use closures approach so that you
can change the implementation on the fly
when you are going to test and if there
is some behavior that is
changing rapidly like i gave you example
like if you are using pops up
today and tomorrow you want to make use
of rapid rabbit mq so what you have to
do you just have to change the
uh change the definition right change
this
implementation of that function so it
will it will not break
later you can anyhow switch that today
you are using pops up tomorrow you can
easily switch to
that because your business logic will
always remain same it will not going to
change
infra will change but your business
logic won’t change
right so that’s how you can avoid the
fragile test cases or writing fissile
tests
so it’s like modularizing your code so
that
you have clear boundaries right yep
cool awesome ah we have another question
from syrian
uh at what point is it worth investing
learning time to learn tools like mock
gen
okay so it’s it’s very easy actually in
fact
i i i thought it is very hard but today
morning i just did some experiment
experiments and it’s it’s
straightforward there is just simple
command
using ghost generate uh means maybe
probably in
in in few weeks or
in next week or by this weekend i will
probably write a blog about it
and uh post it there it’s very easy just
you can go through the documentation of
morgan and you will find it is it’s very
easy
awesome it really reduces your effort of
implementing your
mocks by your own right
yeah please drop the blog link into our
event comments if yeah
yeah if you wanna write it cool uh
another question from mass massage
uh how to test if there are any
dependencies
on the function inside my function
yeah so uh this is from boiling down to
the same
uh same uh approach so
uh there is this slide uh
are you still uh sharing my slide uh
yeah yeah
yeah so here if you see this is a very
uh you can say it is tightly depending
on the
database get from db and then publish
right
it is very difficult in this case to
test this function you cannot test this
complex function without
uh somehow eliminating the dependency
from database or publish
right so the technique is to use same
close either closes approach
or using uh interfaces approach just
accept the interfaces and return structs
that will help in modernizing your code
and making it testable
awesome couldn’t agree more with that
cool so uh oh yeah i just said couldn’t
agree more with that
okay yes so uh i don’t think there’s any
more question in youtube but
if you have any question uh
even after session has ended you can
still stand in youtube
and he will reply you in the same track
uh so thank you so much rukash for your
talk that was really
insightful so next up
we’ll be welcoming iman
from zendid to share about
gold performance profiling
it might please take it away
sorry man i think you’re okay yeah sorry
hi everyone uh i’m iman uh
thanks go sb for having me in this
session
uh today i will talk about optimizing
and profiling in volatile cpi
uh yeah sorry
before that uh let me introduce myself i
mean
tomorrow i’m an engineer in senate uh
working in that platform
and front team my day-to-day is only
like maintaining our
core library like internal library and
in front and helping the team to
build the front detection system
so yeah what we’re talking about today
is uh
what is software optimization and why we
need it what is system profiling
and profiling in golang and step how to
do the profiling in golang api
and the final result is how to optimize
our golan api so
software is software automation so
optimization optimization is like
the action of making the best or the
most
effective use of situational resources
that we have
let’s say to give some example let’s say
i have a server here
a big server with 100 gigabyte memory
and then 64 core cpu
but then the problem is it’s only
surfing one user
so for this case the resources is to
overkill only to something one with user
right and then another case is like i
have a small server
but it’s only one gigabyte memory and
one core cpu
but it’s not only surfing one but it’s
one itself being like 100 or 1000 or
even 1 million
users for this case obviously our server
will overload and the reason is our
performance will
be graded and the customer’s time will
be slower
and maybe the software will be down
so what we learned from this two cases
is uh
we need to optimize our server to get
the optimal resources so
we from the optimist why we need to
optimize because we want to know what
our system needs
and we want to reduce or maybe improve
the resources
that our system want and also to
increase our
api performance
this is the steps that i usually do when
to optimize my own
test api first is like
obviously we need to have a one real
application
that we want we want to decline to
productions and the second step is to
do profiling to our application get the
data from the profiling
and do position whether it’s safe to
deploy to production or
we want to improve a bit more and next
is
repeat the step until we really
confident
and the final step is to deploy
so in this session i will
talk about uh i will use a simple
study case uh first first step is like
to have to have the application that you
need to deploy to productions
i already create a simple application
can be found in my
github repo
and the second step is like to do
profiling
but before that i will talk a bit about
profiling
copy this from web so profiling
is like a dynamic program on us to
measure like how many resource
our application won how many how long or
the execution time and the goal is like
to to optimize our applications
kind of uh why we need profiling yeah
first also you know we want to know
what this our application needs from
memory and from the cpu or maybe from
dix and know how long is the processing
time or response time of our api
we want to know like exactly which code
for which function result will be
to improve and lastly like
we want to more confident before
releasing into production
so to force go long apix
case uh the steps to do profiling is
first we need to import the library for
proof is uh
building liberty that go long profile in
england the second step is
we need to run the application and then
we need to do the load testing
for the load testing tools uh you can
use apache bin smart or ab
or hey i usually use hey
and or maybe you can use any
protein testing tools maybe like gatling
or b method
and while we do the load testing
we need to run the proproof to get the
data to get the insight about our
applications
so if you guys can see i will send this
letter but
we you to to use the profile resource
really simple
you only like to import the proof and
then
create a profiling software
so in my application i create to report
i open to port one is for the
application port and another one is for
profiling port
so to keep for example i will do a
short demo how to use the
proof so
here i have the the application
here what i what i’ve done is like to
import the
proof like only one line here
in my application and then
the next one is to open the port for the
profiling i run it on port 8080
and next step is we need to run the
application
so the first step is to run and then do
the load testing right
so here i will
try to call this api
it’s give us the result here
and i will do the law testing
so hey if you guys are not familiar with
hey
so hey it’s like a lot testing like
apache apache
benchmark uh min c means the
number of concurrent requests so i made
the 50
concurrent request yet here meaning the
duration how long we do the load testing
so within the one minute we will
the hair will do the load testing with
50
concurrent requests to our local
software
so while doing the load testing i will
while doing the testing i’ll run the
first proof tool
the command is good tool to proof and
port yeah you can you guys can read this
letter in
in google docs
so now we are waiting for
proof to fetch the
profile results so to our application
usually take one minute or
yeah maybe 40 seconds
yeah give us the result now
so here
you can see the proof
interface also the profiling or
the blood testing only done you can see
the rps
is 27
it’s quite slow and then
you can see the distribution of the
response time here
and total status code distribution
is 200 with 16 600
responses and then here in blueproof we
can
check the cpu profile
the comment is like you can just top
10 to get the top 10.
you will see the result here
i usually do i actually do the when want
to check
if my code is slow or not if i my course
really want to
improve is i do
the sorting top 10 like based on
cumulative
the chrome here i will spend later but
basically basically this one only to
sort your
to sort the base on the response and the
execution time to
each function under under our
applications so
here we can see this is the function
that’s articles it takes one second or
maybe almost
two seconds
and you can see the
pad here is in the handler
yeah so this this function is take two
seconds
to execute for each
for each it’s uh duration
it’s rotation so
after that that one is this one is just
a simple
uh demo how to use how to import the
proof in your project and how to do the
to do the proof of
to get the result from the proof
the next step is to keep the data
so yeah as us demo previously you can
detect the slow
code in the proof just like what i’ve
done here
we get the slow code here is the fast
articles
because that’s our our article right
and i mean that’s our function like uh
and this one is a
internal function so we don’t have
control on it but this one
i believe we can still have control
and to get data another data is we can
use uh
to get a graphical data you also can use
reproof and also
using third party ad concase
i use speedscope.add so
it will use a like a graphical result
like a
like the result will be something like
that if
we use a sweet scope
the next is so we that’s based on our
profiling our first profiling
we know that the pets article is low and
we know we get the result about our rps
is around
around 27 rps
and that’s quite slow if i say so
we want we want to maybe to look up if
we want if we can to improve
maybe we can’t we want to improve the
rps so it’s more faster
oh yeah so i i haven’t talked about it
before a bit so
food proof is uh invented by google for
perfectly
the list tool comment you guys can try
in your locale
using dot that’s help
usually for my case when i
do profiling to my application i use
this two comment
pop-n and the next parameters sort by
cumulative or flank and then
the next one is i usually use list
comment
so this comment is to
check your function in detail let’s say
let’s say i have this i already know
that this function is low
but i want to more deeper why it’s slow
which function which
could make it slow i can just do like
something like this
please fetch particles
it will show us uh for each line which
which got the slowest one
so this code it takes 90 milliseconds
but this code it takes one second
only this code so we know the root cause
okay maybe the root cause is the gate
order but
the but then is it the daughter or our
function
why is slow we can check again to our
functions
like
it’s slow maybe because we do a
synchronous loop here you guys can see
like
this is really a synchronous look for
every
data every item in the data we do
get outdoor detail and assign it to the
array
so maybe
we can improve this maybe like using
guru routine or maybe using that
request to get based on the data
uh and then uh this is like the
parameter that i’ve used in the proof
tool
like i said like i do preferably like
top
top ten based on cumulative
so coming here is remaining cumulative
it’s total time
needed to execute the whole functions
and we also have like a flat here
also we have a sum flat is total time
needed
to execute the function excluding the
external function
so it’s the core logic of the function
sum is just like a
representation of the line
so to give a short example we have this
sort
function called call me and
another function called booty and call
john and don’t call me
it call it calls call booty and call
john
so from this uh example
we know that uh the cumulative for the
this the calling function is five second
is because when it’s the sum of the its
total second of the function executed
and flat is the the core logic of the
call me function
here without without callbuddy and
without call john
for me only need two seconds so the flat
is
two seconds this will be useful when you
want to know
your function is it is it your function
slow or your external function is low so
you will see
you can get this you can use this in in
the proof as well like
from here you know that the cumulative
for the function is like two seconds so
also the next thing is uh using flat
grab rendering
aside using the prefront you also can
use flame grab
tools for this
session i will use a speed scope pad
maybe in production you guys are so
familiar with it uh
like in data dope there are chloe has
this feature
but for this presentation i will use a
speed scope
this is the example of the speed code
result so
as you guys can see this is in time
frame manner like
in six sec in seven seconds uh
we have a receptacle like execute for
four seconds and then for how to read
this like uh
this is the core function of the first
article that we
have so
when when user called this endpoint it
will goes to
fetch article function so from there we
know that uh
this function takes so long to execute
it takes like four seconds based on the
profiling result
and from the four second they get out
already
takes like three seconds
350 seconds 3.50 seconds that’s
quite a lot so we know that uh
the what function we need to improve is
this
to like how we can actually
get outer body sorted
and maybe cut a half or maybe
like more more faster maybe like vm to
two second or
one second
the next step is like after we get the
the profiling result
we we made the decisions like we already
have the data from the testing that the
fps is 27
and then we have the profiling result so
display that uh
first article and get auto value and
make the
make the function slow
and then we asked our self is it okay is
it actually acceptable
or do we want to improve if you guess
okay then
no need to improve but if you guys think
that we still need to improve it to
make it better so we need to
find the solutions
so for my case i i propose two solutions
the first lesson may be what we
usually people think i
like maybe we can do a parallel or
trading using routine so
instead of synchronous threats for the
outer details what if we we do like a
we add we pass the get outer id to
gurudin
and maybe it is faster maybe it
it will solve our problem maybe we get
the
more better result but then after after
i
tried
after i tried the result still quite
similar
like only improved a bit by
from from 27 to
41. that’s a good thing i mean
it’s already improved and then the
maybe the next time it’s like it’s up to
back again to your to
to the to this part is it okay for you
like from 27 to 40 is it okay for you
or you want to improve to a better
result
maybe maybe
some of your friends propose a solution
like maybe what if like we do
bets query to database to get the
outdoor details
that’s what i have done here instead of
doing it one by one using guru team
i try to matching to the
query to get the authors based on the
array that we have
so here if you guys see like
this function is fetching the articles
from database assuming it’s already
like this one is like selecting the
title body although from the table of
articles
and it’s written to array and then and
then
what i have done is like uh instead of
uh passing to routine one by one
i i made an array of the authorities
something like here and then
i do create a new function get auto by
ids
and passing the array of the author
and after that i assign its author to
its item
to its article uh the this function will
be returning like a
map of outdoor based on the auto id
so if you guys to see the result code
here for example the the code what i’ve
done is like
get author by ideas i pass the outdoor
ideas array
and fill returning a map of
author which is the key is the author id
and then the function
will be called here
and then i will make the
the author to its article
interesting part is that this this
dissolution is quite
faster and maybe even faster than the
previous result
like
i’ll leave it like maybe i’ll show the
result they have done
so what i’ve done previously using the
new new solution it’s really faster
like it’s 10 times from the guru team
part
it’s 500 fps
and it handle 107
the request yeah response
and that’s really a fast
solution compared to previous one
let’s wait here for the real reason
oh i think this was the resource uh okay
but it’s faster i mean like derpy is
still higher
compared to the previous one
this this result i’ve done it in when i
don’t have any
i don’t have any application running so
it’s
quite different so what we what we
we can conclude from this research is
like
usb proof to know which code you should
to improve
uh and then do a lot of testing before
and after
the mess to know if you really make an
improvement or not
and then optimize after profiling i mean
like avoid
promoter optimizing i’ve been seeing
this in my
friend like when they talk about a slow
slow response time
well you what the resolution that comes
first is like directly
proposing case i mean case is not wrong
but
before moving to case let’s try to
profile our application to know if
there’s any slow functions
and then looks for the problem and
choose the right solution
for i mean here is like for study case
that i’ve done previously
using corrupting like me might be not
the best i mean it’s
still the best rather than synchronous
call but it’s not the best maybe for the
style case that i’ve done previously
like the best the batch solution is
still better
compared to goal team
and then for the better experience uh do
a lot of testing
and profiling video server i mean here
like what i’ve done here is like
do all the testing and profiling in my
local sometimes we have a lot of
applications running in
local and consume our resources
so to get the better experience maybe
you
create a copy of your production server
and then you spam a
staging or idle server and
like a client and then do the profiling
from there like
between two servers that will give you a
better result
compared to do in local
actually i think that’s all from from my
side
i give back to stanley
thank you so much iman that was a very
awesome talk
that’s a lot a lot of information to
take in
so we have a question here from youtube
uh from syrian so
when do you decide to spend time on
profiling and optimization uh should the
time
be better spent refactoring code to
clean up the codebase first
uh so i i think i’ve said this in
previous
right when should i do profiling is
before
do before i deploy my application
so so when you have a feature
and maybe like a new feature you need to
do profiling first and then
to ensure the profiling is like to know
what your
server needs and maybe to know if
there’s any
slow code yet that we can improve if
even it’s like a bit
it’s still helpful yeah when when is
decide to spend time profiling yeah
before you
deploy your application to production
it’s maybe between after
ta or deployment yeah something like
that
after we do pa or testing in staging
and we want to release to production
that is time you need to
uh profiling uh how about
after the app has been running in
production
so like i i think i think there’s
another solution
uh i think i think
steven want to ask like for those
production apps like
when do you decide between whether you
want to optimize for performance
or you want to like spend time
refactoring the code
yeah based on my experience uh usually i
turn it
after like uh
we want to improve like maybe we want
like some there’s a case like we want to
just like case maybe the customer
already complaining about the response
time
that’s the time you need to spend time
maybe
uh one sprint or one month to do
profiling for your entire application
the second time maybe when when you when
your company want to
uh reduce the cost so you need to like
to know how how many servers that
really your application needs that’s the
time you need to do profiling and for
optimization
but the best solution maybe every time
you release a new feature or release a
new
fix you need to do profiling to ensure
uh the rps is not the degraded or
to ensure your app is still still the
same with the previous
version yeah so it should be
like part of the test suite you would
say
cool yes let me see if there are any
more questions coming in
oh there’s a new question from maasaji
uh is people can pre-proof be used
for an app that process data from stream
or maybe message broker
yeah preproof can be used to to
people with internally will collect all
your
resources other application resources i
mean like
how many uh memory you use in your
application how long you take for each
function
as long as is under the golem
application
you can still use proof even it’s like a
working or a
key key worker or key runner yeah
cool
all right let’s give another couple
minutes
to see if folks have any further
questions
yeah i guess that’s all the question
from our audience
uh yep if you have any further questions
oh that’s another question uh there’s a
new question from
luis ma uh is there any disadvantages
of this profiling method that you
mentioned before
um maybe the disadvantage is like
you how to be smart automatically
i know data documentary have like
kind of flame grab rendering you guys
can use
uh the dog for that i’m not sure for
other monitoring provider
yeah the only d7 test is like you need
to do this manually maybe you need to do
is to do this
in ci city and configure your
ccd to get the same result yeah
nice
cool
okay i’m gonna let’s just wait for
another couple of minutes and see if
there are any other questions
i know that and you haven’t gotten
dinner so yeah
we don’t want to keep you for too long
but
yep
i guess there’s no further question from
the audience
so yep thank you so much iman for the
talk today
and that was really awesome uh
yeah so i guess uh that concludes our
events today
uh please don’t forget to follow us on
twitter at
golang at roland sg uh
and thank you for tuning in today
have a great evening see y’all