Go Module Versioning: Evolution Through Today @ GoSG Meetup – 2021

today i’m going to talk about go module
versioning uh evolution through today
uh i’m going to talk about how go module
has
uh uh you know the versioning system has
a value evolved from go 111 to 116 which
is released in um
last february uh and so a little bit
about myself as i already told i’m a
software engineer at jfrog
and now i’m a gopher uh you can always
reach me out at
my twitter handle engineer metalliby or
you can connect me with linkedin
if you have any question about today’s
talk or anything about go or anything
about anything else
uh we can always connect uh
so before going further i’m going to um
you know whether it’s virtual or it’s uh
you know
real meetup we um always you know
get excited about the swag so um as part
of jfrog we are going to give one of the
go t-shirt
so you guys can scan the code here and
you will be in the raffle and one of the
person will
get selected and will mail the t-shirt
to you uh you will see this
barcode at the end of my slide too so if
you don’t catch it here you can catch at
that time
so let’s get started uh before going
further in my talk i want to
um this is a famous i mean i really like
this uh
saying before from uh simon sinek which
says that
we say what we do as a software engineer
or you know any person we say what we
want to do
we get the requirements and everything
and then we come to the part
we sometimes say how we do it you know
how we want to implement it or we
already know how to do it
but we really say you know why we do it
why which is actually the core part uh
we actually thought
we we really don’t uh you know go much
deep into that or think about why we did
uh what part uh uh you know is actually
being done
and how it is done so in this whole uh
presentation which
in this whole talk with each versions
i’m going to talk about
what how and why part like why all these
things have been placed and
emphasizing more on the white part
so just to give an overview we will be
talking about go 111 module
versioning system and then we will talk
about what has been changed for pseudo
versions in government
and then in go 114 what is change
related to the rendering so if you are
still using rendering
uh you know what has been changed in 114
and finally in 116 what is
there related to the module so
i’m talking a lot about you know
versioning and all those things
why we even need versioning so uh if we
if you think about this question you
know what what comes to
our mind first of all dependency
management
yeah of course um you know uh i have a
code i’m building a code and i have
uh i there are a lot of libraries
especially for go as an open source
language there are a lot of libraries
you know people are building along
around the world
and we can always use them and even for
for the um you know whether it’s
making connection to the database or you
know any utilities or anything so i need
to depend
uh i need to manage my dependencies i
need to see that whatever
particular version of the code that is
there
you know if i if i use uh that for over
after two months or three months it
should uh give me the same result and
you know i’m able to reproduce my builds
second is better software engineering
software engineering happens
when um you know software developers
collaborate with each other
that means they work on the same piece
of code which is not
static right code is evolving over the
period of time
and so one software engineer who is
working on one piece of code how the
other person knows that
okay he or she is updated with the code
right so in order to collaborate
more more and also for the reproducible
builds obviously the
person who is sitting here here in
california should be able to
have the same piece of code built
successfully than the person who is
sitting and any other around the any
other
corner in the world so you know it
should be reproducible
uh everywhere so these are the main core
components that we actually wanted in
the versioning system
uh in any of the languages and so here
that’s why go has put some of the
efforts to put it in
to put it in place so i
uh i’m not sure how many of you have
actually
were on uh go version 1.11 and before
but there was not any versioning system
that means
go to go to link was not aware about how
to get a particular git tag
so whenever we do do go get it actually
gets from the master it doesn’t get uh
if we do go get we cannot actually give
a particular git tag or version that we
want to get that particular version
and obviously though there is no native
dependency management system there was
not any formal dependency management
there were tools like glide gold app
which were there
actually to manage your dependencies but
they were not uh
you know uh combined with the go to lane
so to look all these uh problems uh
uh to solve all these problems with
gohan 11
go has introduced the concept of module
and with the environment variable go
one one one module uh it has three uh
um things like you can put it on you can
put it auto or
you can put it off so if you are if you
are putting it on that means you want to
run it in a
module away module system if it is auto
that means
uh the your project is module aware so
if you have a go dot mod file and you
are out of
co-path then it will work uh then it
will work in modular
mode and obviously off that means you
don’t want to work in module of your uh
mode
so i’m talking about a lot about uh
modules so let’s go into the
uh what uh how and why part
of this first of all uh what is a mod
what is a go module go module is
actually uh you know
collection of packages uh which are in a
file tree and it has a go.mod file at
its root
now how does the go dot mod file looks
like uh this isn’t a sample example
uh go.mod file has a modules path
at the first part that here is module
github.com containers
common this is the full module path and
then it will have a go version with
which this particular module has been
built
and then it will have a required
directive
required directly will have all your
dependencies that means module path
of your of your dependencies so
uh as we know uh you know go uses the
whole module path
for for the dependencies for mentioning
the dependencies
so uh here there is a module path for
uh and along with that there is a
version
now go prefer to have a semantic version
as the versioning system that means what
is semantic version
we know that it should be in the form of
major.minor.patch
that means whenever there is an uh
incompatible changes we should pump up
the major version if there is any
compatible changes we should bump up the
minor version
and if there is any bug fix for for the
uh
minor version then we should bump up the
patch version so
that’s how uh go and these are the
release version and these release
version are being introduced in go by
you know
tagging any version in in your version
control system
so that’s how release version looks like
but that doesn’t mean that you cannot
have a pre-release version in go you can
always define a pre-release version
like this using a hyphen at the end and
you can
specify some string so that’s how whole
go module
uh uh dot mod file looks like and that’s
how versioning are being defined for
your dependencies
now let’s go to the why part why this
versioning has been put
uh this way and how they are imported
so principle of module versioning for go
let’s let’s take an example suppose uh
i’m working on a
module a and i have an dependency on
module
b which is work which has a function f1
of f1 which has a dependency on module d
version 1.0.2
and i’m using that i1 function similarly
i’m using another uh i have a dependency
on another module c which is using the
same f1 function but of d but off
version 2.0.2
now the question comes which version of
d
should a refer or should a import
whether it should 1.0.2
or 2.0.1 so in order to
uh solve this problem uh go has defined
the principle which is called as
semantic import versioning
which actually says the two things that
all the
uh modules should define semantic
semantic version that means x dot y dot
c
and also for major version instead of
whenever you are bumping your major
version uh you should include in your
module path which is so this this
this thing is called as import
compatibility rule so that means
whenever you are
bumping up your version from two or
higher
you should include that in your path for
uh zero and one
this rule doesn’t follow uh as zero and
one one is basically for the you know
test and uh
basic release so that means coming back
to my example
now d d will have a module path for one
point zero point two
as d but for two point zero point one d
will have d v two now a can import both
d
and both d slash v two and uh use both
of the functions and will not have any
problem
so here that’s how the major version has
been uh you know put it in into the
place
now going back to my mod mod file if you
see
uh this particular kubernetes dot io key
log
it is using a major version 2.4.0 and if
you see version 2
is there in the module path now with
this
there are some of the problems that has
come
you know for the developer is that
whenever you want to bump any of your
dependency version from uh you know
version two and higher you have to
change it in your go dot mod file
manually and also you have to change
all the import parts but in all of your
projects so there is a little bit pain
but for that there are some of the tools
which are available like
fixed importance or and all other things
that can be used to
uh fix all those import path quickly
and also whenever you do go list
there is a little bit problem of
discoverability of all the major
versions that means go list of go tool
will not list all the major version it
will only list all the major
minor version under a particular major
version
so the only way we can discover our
major version
is actually going to the source location
and do couple of call commands to see
what all my
major version are there for that
particular source location
because the key for all the major
version is that their source location
will be the same
now coming now the major versions are
being fixed but what about minor version
and patch version
are are they still fine to import so
uh let’s take an example i have a module
a and i have
a dependency on module b which in turns
and have dependency on module d
of version 1.3.2 similarly i have
another module
uh dependency of c which in turns had
dependency of d
of another version 1.4.8 and i also have
direct dependency
on b of 1.0.2
now these latest version is 1.5.0
now according to the most of the
language whenever this kind of situation
will come
uh latest will will get imported that
means 1.5.0
but that is not true with the go
languages with
in go uh there is a principle called
minimal
version selection that has been
introduced that says that
the module should import only only
the latest but not the greatest that
means
the minimum version that is that is
being uh
that is needed to actually uh you know
build a particular module that will get
imported rather than the latest one
so here if i see i will not have 1.0.2
because that will be
too uh old for a and
one point five point zero will be two
new for a so
a will import uh one point four point
eight of the version
so that’s how the uh minor version
of go has been fixed and has been put
into the place
and it it really helps to have all the
uh versioning system in place properly
now uh let’s go further now uh
go 113 has come and there and it was
found that there was
still some discrepancies floating around
because of the
uh pseudo version so there are some
things called
pseudo version that people use and they
were modifying
uh you know through hand rather than go
itself to create it
and because of that there was some
problem in getting the reproducible
build
so so in order to put to correct that
in go 113 go has introduced some
validation for sudo versions
so now let’s go to uh what how and why
part of this
first of all what is sudo version sudo
versions is actually the versions
uh you know that are that are actually
uh put into place whenever
we want to develop against against a
particular project which doesn’t have
any released version
uh or you know whenever we have enough
we have dependency on
on any project which has not been
released so in that particular
uh cases you know uh uh developer uses
pseudo versions
how does that pseudo version looks like
so they looks like this big
string if you see it has three component
first component says the most recent
tagged version in the commit graph
second component is the commit time in
utc
and third but third is the prefix of
commit hash
generated by the underlying control
version system
now sudo version uh practically should
never be typed by hand
it all go tool is aware of that and
whenever we
attack any version it automatically
generate the sudo version
uh for you uh with the timestamp without
with that particular timestamp but there
are some
tools or there are some outside tools
and people actually you know
type them by hand which which was
creating some discrepancy
so for example uh this most recent uh
tagged version in the sudo version
actually takes part in the minimal
version selection which i explained
before
so if i modify this uh this first part
uh to any other then it will actually uh
so it will actually uh mess up the whole
ordering of the system
and actually pin this particular sudo
version to your module
which might not be the latest and also
the second part which is the commit time
that also uh take part in ordering of
the
sudo versions so which particular sudo
version is the latest one
is being tell by is it can only be uh
tell by the
uh timestamp so if we change the this
time
timestamp it again changes the whole
ordering of the system
so uh because of all these problem in go
113 there were some
restrictions that have been put so
now what are those restrictions
first of all the most recent commit
graph the whatever is the tag it should
whatever is
there in the underlying version control
system for the
major and second is the commit time
it should be commit time that is
generated by the version control system
and third is uh the commit hash is so
it should be the same commit hash that
is being generated by the go
tool there was also other restriction
uh like a pseudo version can have a plus
in compatible
suffix only if it is needed by the way
by the corresponding major version and
if it doesn’t have a go dot mod file
and this is true not only for sudo
version any other versions
and also last but not the least is the
checksum server validation which was
enabled in go one
from 113 so if you see along with
your go dot mod file you would have seen
a file called go.some
which actually has a cryptographic
hashes or cryptographic
uh checksums of all your uh you know
module
dependent module versions and the same
hashes have been stored in a checksum
server
so now with go 113 whenever we do any
goget command
whatever in there in your go dot some
file the checksums
jackson is being validated against
whatever is there in the checksum
uh server and if it doesn’t matches it
will not allow you to get that
particular version
so let’s see how these changes have come
into effect
so before 113 if you do a go get
on any particular pseudo version which
is actually
not the correct one and that means if
you see in this example
i have actually changed the timestamp so
what it used to do
it used to get the latest for that
particular uh
major version and pin that particular
module
pin that particular pseudo version in
your go dot mod file which is actually
the wrong pseudo version
but after go 113 if we try to do
with our wrong pseudo version then it
will give
us an error saying that uh you know
the maybe the timestamp doesn’t matches
or the
uh checksum uh is invalid and
these kind of problems uh these kind of
errors we can start seeing them
now how to uh solve this if you have any
you know
uh invalid and uh not good pseudo
versions
so we have to fix our pseudo versions in
our directive direct dependency so if
you have any in your direct dependencies
and also any in your transitive
dependencies
so for direct dependencies we can use
require directive
so instead of posting the whole sudo
version uh
we can only put the last commit hash
part
and when we run go more tidy or go get
go tool uh go tool will automatically
get the correct sudo version and pune
pin
it to your module and similarly for
transitive dependencies
we can use the replace directive instead
of required so if any of my dependencies
has uh uh you know incorrect pseudo
version
we can use we can do the same thing use
the replace directive and use only the
commit hash part
and then we can successfully build all
our
incorrect sudo version in go 113 and
above
now let’s move further now comes go 114.
so in goan14
uh it was uh rendering was given perform
preference over the module cache before
that there was a module cache which was
which was being
maintained and whatever uh whenever we
do go get or anything
uh it used to refer to the module cache
but when i
but from go and go 114 if you define
uh go go 1.14 in your mod
file and if you enable uh and if if you
have a vendor
uh directory then it will always look in
your
vendor directory for your dependencies
rather than in your module cache
now let’s go to what uh how and why part
of funding first of all what is
rendering
uh most of the people uh you know what
they do they don’t uh
want to uh there are many reasons
actually to maintain
a copy of your dependency inside project
that means whatever libraries you are
using you are having
a full copy of all the code in your
local
and then you build it rather than having
in any central repository
uh this was b this was being heavily
used before
before uh go module was introduced but
after go module it has
subsidized it now let’s see
why we need uh rendering we need uh you
know rendering maybe
uh we have a fear that you know uh
no first of all uh i have some of the
dependency
and i want to change any code in my
dependency to debug something
so uh the only way is that to take that
particular code in my local and debug
that
so for that we can always use replace
directive we can use replace directive
in our go go.mod file and have that
and you know download that just for time
being and then
debug it rather than putting it in our
in our git repository the whole code of
our dependency
second is it might be possible that vcs
is down
you know there is networking issue or
dependency might appear
especially for this uh open source
uh language you know there there there
might be
author who can just pull out the module
and it will not be there
or we can or if the dependency is using
vcs other than git then what are the
options we have
so for that we can always use go proxy
there is a go proxy
package uh there’s a uh you know public
go uh proxy
uh there we can always define in there
and we can have it cached there
once it is cached there it will never
disappear also
uh there might be a cases when we are
thinking about
about mutability you know it can change
that there is some piece of code of the
same version uh which was there and now
it is not there
it has changed or if we have to build
without a network there are some of the
uh companies that doesn’t uh allow to
uh go outside the network so in that
case we can use
uh you know private go proxy uh rather
than the
public one uh like artifactory and in
artifactory we can have both public and
private modules
now what is the uses uh also
when we don’t have all these
dependencies in our uh
you know uh local then obviously it uh
saves the space and also we will have
cleaner commit locks because we will not
have commit locks of all those
dependencies obviously our whole
uh code will shrink it down to the
smaller one
so now uh we we should think that
whether we
we should still use the rendering or not
so it’s up to you
up to us but module has uh since module
has come
it has subsidized all these problems uh
to use it in a better
better way now last but not
least recently in february go 16 has
introduced before that go
one uh go one one module was always made
to auto
that mean that means whenever you are
working in a
language and when whenever you’re
working in a module it is a if you have
a go dot mod file it will always work in
uh module aware mode but uh with go 116
it will be
by default be on that means that means
it will always be on and there are some
good features that has been coming to go
16
uh so see uh uh since one goal 116
go install will be using uh versions
along with the module path before 1 16
go install
will was using always the module path
that means you can always get the
latest but not a particular version but
with go install we can now
get a particular git uh tag so since go
install has
been introduced with the version go get
is going to get deprecated for build and
install
that means now we should all we should
be using goget only
uh only to uh download or build our
dependencies
not to actually install them and if if
you are doing that
maybe you should start moving away from
it if you are using the module
and start using go install for that also
in go 116
go mod vendor and go more tidy will have
a minus e
flag uh enabled
uh and there are some of the good
directives
uh that has been uh introduced first is
redirect directive
retract directive is nothing but uh you
know whenever
sometimes uh uh an author uh release
some uh
particular version that that he or she
thinks that it that is not good or it
has some vulnerability issue
or it has some kind of or its smart file
is not good
and they don’t want other people to use
that particular version so they can
always use this retract
directive and put that particular
version with some comment like
broken go dot mod file so other other
people will be able to
uh see that and will not be using it and
go tool is also aware of that
also uh this new feature go embed has
come
finally in go we can use the static
files
right to embed like our html file using
this go embed directive
and last but not the least is exclude
directive
uh so exclude directive was all always
there before
but what it used to do it uh it never
whatever
uh particular version that we have
defined
in our exclude directive what it used to
do it used to get the next latest
version for that
and that used to break the builds but
now
from go 116 uh forward it will just
ignore that particular version
and it will not use any other version in
replace of that
so that’s how we see how go uh has
evolved from 111 to 116 a lot
and uh there’s a latest survey that says
that most of the people from go packages
has moved to go module
like 97 people so if you have not moved
you should move it now
because from 116 module is on
uh for for the auto
um so that’s it from my site if there
are any questions
um feel free um to ask me
uh and again my twitter handle is there
my linkedin
and there’s again a quote for a cool go
for t-shirt
thank you all