Make Go Builds Fast with Versioned Go Modules, JFrog, and Codefresh

so welcome everybody glad you could join
us I’m pretty excited about this because
I go is a relatively new language for me
I’ve just really started experimenting
with it and I know that J fragra has
been doing a lot of cool stuff with go
lately so I wanted to bring Craig in to
put together something interesting and
basically I think it really boils down
to two things one we wanted to figure
out how we could speed things up but on
another plane we wanted to figure out
how we could also make things more
secure so we put together a little bit
of content around that and a demo and by
we I mean Craig Craig did all the work
but then he showed up to share the work
and that well that was the critical
thing so two teachers pray and if you
want to share your screen we can
introduce ourselves but then I’m the
chief evangelist for cook fresh and
mostly focused on containerization
continuous integration continuous
delivery really suggested pipelining
kind of stuff and then Craig is actually
the director of product at J frog so
anything else you’d say to introduce
yourself sure so my background is
completely different than yours I’m not
exactly developer I’m going to go to my
background was actually in statistical
programming and then the last 15 years
I’ve been working on solving developer
problems for distributed systems so this
is actually a great thing for me because
it you know we decided to do this myself
oh we took them ourselves off the screen
maybe you can see the little t-shirt I’m
wearing is actually J part one if you do
the go challenge you can’t win this
t-shirt I have the URL at the very end
so you can turn that out the Frog and
the Gopher together perfect I want to
myself okay so let’s see I guess we
should just jump in let’s do that thank
you so first I wanted to talk a little
bit about J frog and what role we play
you have a developer ecosystem to kick
things off and position it so the reason
I chose this slide is that it kind of
to the point when working with fresh and
I think a lot of your audience are
probably coming to go French for those
sort of cloud native applications
perspective what we we talked about is
how having a container registry is a
necessary but not sufficient condition
for having very operable applications in
kubernetes so you need automation come
fresh provides killer automation you
need a doctor for industry in home repo
and you need good Bernays clusters and
cook fresh makes all of that stuff super
manageable now what you put into that
docker registry is really where
artifactory comes into play so this what
this picture illustrates is the you know
kind of on the right hand side you’ve
got your helm definition of your
application and you’ve got your level
charts and all the docker images
orchestrated in the pods the thing is
when you deconstruct what’s actually in
the docker image it turns out it’s
composed of a bunch of sort of upstream
packages sometimes there are operating
system components that your app depends
on often their packages from your
application development framework then
you package that up into your docker
image publish that to your app and so
today we’re going to focus very much on
the left-hand side here we’re gonna look
at go modules and how artifactory works
together with code modules and confer
can automate all of this process for the
demo today we’re gonna give an end
Africa watching packaging thing going
forward we’re going to build that out on
this to show you the best practices for
how you do that not very much it isn’t
deploy those here computing clusters so
artifactory is this repository
management platform that allows you to
have a consistent way of managing all of
these different kinds of packages that
you used the logic of cognitive
application and it’s this platform
that’s available both in the cloud as a
service and on Prem so that enterprises
who want to have close control over
their packages and do that
it’s his a are we using the cloud
version today what we’re using as a
cloud
yes in fact it happens to be deployed in
Hazzard and so it’s accessible the rest
API is CLI IDE plugins the UI which
we’re going to look at mostly to this
well the UI and the CLI over to the
workout today
so that’s enough about that so let’s
talk a little bit about go why are we
here right so like I said I come from a
statistical programming background I
used to work a lot hated our sass Python
SPSS those kinds of things and then I
learned C++ all these decorations and
crazy ways of expressing things and lots
of ways to shoot yourself in the foot
and very sophisticated and very
difficult to use ways to do concurrency
multi-threading well we’ve gotten to the
point where multi-threading is it should
be just an innate thing you don’t you
shouldn’t have to have all these
declarations well go with this new
language that is natively concurring and
takes advantage of all of the hyper
threading capabilities of modern
processors so that’s one of the things
[Music]
spending some time on it they had next
thing is the networking is just assumed
right used to do all kinds of stuff to
do that now it’s just straightforward
libraries that are built in to do all
very high performance capabilities with
basic stuff you can obviously import
code and dependent on other people’s go
to do additional things and that’s
that’s great
the next thing is that for performance
reasons it’s not interpreted it’s
compiled from source it’s an executable
that’s specific to the hardware
architecture yeah that’s one of the nice
things about that is that I’ve noticed
and and I think it’s one of the reasons
that go is so popular with containerized
applications is that when you do compile
from source it really uses these
streamlines sort of libraries
the images you get are incredibly small
whereas I like if you want to say hello
world in Java like you’re in gig into
that thing right like you’re here gig
into that image just to say hello
because it’s gonna pull it all the stuff
and it’s a whole whole stack of stuff
goes so streamlined and it’s the builds
can take a little while but once you
have the binary and the docker image
it’s so small and it runs incredibly
fast so it’s very very performant in
that way yeah so somehow it’s managed to
balance that need to be slowing the
performance with so it’s super easy to
build your go binaries for multiple
platforms so you get a lot of the
benefits
thanks to containerization we can move
those binaries anywhere life is
beautiful one of the things I’m just
loving about it is the syntax is just
incredibly simple you read it what it
says is what it does interpretation
makes it much more maintainable
so that’s that’s the last thing is that
the community yeah the the golang
Twitter game is pretty strong and we of
course gopher Khan was just a few weeks
ago and my friend came back from it and
I asked him how it was he said it was
really boring and that’s actually the
best thing you can hear of how about a
language conference is we got through
all the exciting problems and now we
just have some boring problems and that
that’s the moment when you know that
it’s come of age and it’s ready for
primetime is the conference was kind of
boring they didn’t have cool stuff by
the way but but anyway I didn’t get to
do part was there so with all these
great benefits these come comes a couple
of challenges so when you’re depending
on somebody else’s code in its imports
from
a reference to some to a namespace and
the namespace gets resolved at Build
time and so the namespace isn’t under
your control
thank you as a as an enterprise
developer is an enterprise company be
uncomfortable if those those external
resolutions are inside so you don’t you
don’t know that somebody’s not out there
changing the code right maybe your code
your break because somebody else makes a
change to the code and it’s not
expressed you know version so
fundamentally the question is I got this
thing working today I tested today but
when I need to go to make a change
tomorrow well somebody else has made a
change that breaks my code as the
community of code just explodes the
number of awesome things that you can
and should use has just exploded a long
time right so there’s tons of extensions
to these are similar there challenges
with haven’t like NPM right precisely
the the challenges that go though
natively didn’t have a notion of
packaging beyond the binary by by the
philosophy of itself it’s about build to
binary right and so the notion of import
was explicitly about the code itself and
so in order to address that over the
last few years in the NGO community that
the various efforts to try to resolve
this challenge because explodes and code
reuse becomes a really big thing lots of
alternative approaches were proposed
there are at least four that I know of
and Jade frog you know we’ve been
involved in helping resolve the
controversy around the different
currents and in fact we’re involved in
the in tribution wasn’t the introduction
of some of the capo modules it used to
be called version to go so the proposal
was made several years ago for version
go if you if you do search on
funny bunch of great proposals blogs
discussions about version code so we put
our weight behind that and it landed
even go in version 1.1 just weeks ago
few months ago and and actually an
artifact reintroduced private registry
for it before was actually natively
supported and go back in May and so
we’ve actually built robust support for
it and it turns out and it’s actually
really easy to take an existing project
and they could take advantage of modules
so what is a module module is a
collection of packages so these are
these external dependencies that you
depend on and they they’re they’re
downloaded so you have a local copy
basically turn it up in their version
and so what this means is that you no
longer have to go to some unknown source
like a lot of modules but a lot of them
work people publish them anywhere they
want right and so dependency on github
is not a terrible we also know that it’s
always as fast as we want it to me it’s
also right we want to validate it and
put it into sort of a safe zone where
we’ve maybe we’ve done some security
scanning or something to validate these
modules and put them to a place where we
feel comfortable that our team can pull
them in now there’s a lot of there’s a
lot of use cases around needing to know
that you’re getting exactly the same
code that you got before reproducibility
is one another security ensuring that we
were using the same code that we did the
security test on the form there’s also
or maybe traceability is one of you
think you’re like autumn abilities okay
no it was there I know was there at this
time I know it was there at the next
build yeah exactly
and so to get started it’s actually
super easy once you have go version 111
you go you get into your project you
want to go on it in it and that will
actually create Traverse your code and
look for all of the all there so it
builds this follow-up and then
artifactory uses that file as an index
to then go and understand how to go get
those code and then create the modules
itself and spoil knowing in our Factory
so let’s let’s take a look at what that
kind of workflow looks like so we’ll get
right to the point in get into the demo
so on day one you know you start your
project you you’ve done your go tidy
come on tidy and you’re working with
iron factory then what happens is you’re
trying to do your bill the bill goes
start a factory to trying to find the
modules it doesn’t have them and it kind
of complains a little bit I don’t have a
local copy so what I do so in step two
what we do is we do it the old way I go
directly to packages are I pull this
packages back and then they package them
up and I push them like publish the
artifactory and and then what happens is
everything that’s sort of all the
dependencies they get stored in our
factory I understand the traceability
between one bill and all of its
transitive dependencies and then another
let’s just say another development this
is another use case for why you
another developer wants to make sure
they can do the same thing at exactly
the same packages well it turns out if
they’re all locally resolved this is
very important for use the language like
a lot of other benefits all of that into
fresh and show how we could do fast safe
delivery of these packages yeah that’s
great so just a trees code fresh as you
as you hear for those that aren’t
familiar with it
code fresh is a CI screen platform
designed specifically for the world of
containers so we have built-in steps for
doing things like building docker images
for doing advanced deployments
blue-green Canary plants in the
kubernetes and then as Craig mentioned
we have some really great integrations
with Jay frog where you can use
artifactory is your docker registry your
homework your home repository and that
kind of goes to our philosophy is that
we really want to be an end-to-end
we got your commits over here and we got
your deployment environment over here
and we want to glue all this stuff in
between or well I don’t know a better
metaphor so here we’re looking at our
repositories repositories and and we’ve
got we’ve got that going all right yeah
so I’ll just step through sort of what I
did the first the first thing I did is I
created a project where where what I do
is I’m building a docker image to reuse
to be blue between code fresh and
artifactory
so what this glue is is actually uses
the G frog CLI
and what I’ve done is I’ve baked it into
a docker image that’s just stored here
locally so that I’ve got my local image
to use over and over again so that my
builds go super fast and then what I
have is I have this project to get a
project actually and I just formed one
of the dance projects that he uses to
demo fresh that’s why I used a demo
canary canary and especially with this
but what I did was I just did exactly
tiny and it turns out that the so that
wasn’t very interesting so I hacked into
it a couple of dependencies and what
those dependencies do are just put a
quote into his little boxes it uses a
library here called our CIO quote so
here’s an example of package repo and
when it did the build it put it into
artifactory so the most recent build is
here so what I can see here is that in
the modules that got published by by
cook fresh it creates the mod file
obviously it’s version the semantic
versioning and it’s a there’s a zip file
that contains all of the contents of
that and then I can see here are the
here’s the direct dependency I had which
was our SC quote version 1.5 – and it
has both as if Anna MAHT so we’ve we’ve
taken about transitive dependency we’ve
packaged it up and we shoved it in the
artifactory and i can actually go and
find that module in artifactory and see
Factory it understands the structure of
these things but this is know in my
local NGO repository so what I’m going
to do is I’m going to kick off I’m
making a small change in the code I’m
going to do very bad style here
pre-staged this new dependency there –
just do some trivial use of that package
bill there so let’s commit those changes
now what happens magically of course
it’s magic because code fresh
understands the notion of and now I see
my I see my new commit here and it’s
kicking off a build process
so I’m come fresh it’s got this really
nice new UI but they just add in where I
can visually see what are all the steps
so I’m gonna dig down just a little bit
into the steps and the code themselves
so what’s gonna happen is just like I
showed in the slides it’s trying to do a
bill resolving that new dependency and
if we go in here we’re actually gonna
see that there’s an expected error
remember I went to try to go to our
factory it couldn’t resolve this new
dependency because it didn’t exist in
our factory so couldn’t find it so what
it’s doing now it’s artifactory is
actually going in and creating the new
bill to pull down all the packages
rebuilt them and publish them into our
Factory and it published the
dependencies in our factory and now
we’re going to go and publish the build
and then finally in my steps just spin
it for nicety I create the binary
and I published them in our factory
using this great tool called cohesive so
what it’s doing is it’s creating this
new build so if I go here and look at
the builds I’ll now see this new build
that now has new dependencies so here
you can see it’s kind of additional
dependencies on this DDO library so now
I’ve got a local copy somebody else
codes and uses that same library I know
that they’re gonna be able all exactly
the same
so what does that look like repository
so here are conceding my local repo it’s
now created this this new package in a
namespace just like mine is my new go
build here so artifactory know how’s
that set up and that was actually really
easy to do with code fresh let’s take a
look at so what I did is I did it in
cook formal because it’s actually a very
straightforward way in which to express
this logic so what I did is I just said
three stages because we have to go
through all the steps so the first stage
tries to do the naive bill if there are
no dependencies or all the dependencies
are resolved from artifactory then it
sets this variable yeah all the modules
existed that’s great what happens is it
starts it checks this next step and says
well if the mods all exists are true or
not true that’s not true that’s then you
have to do this step which means you
have to download all the packages
otherwise let’s do all the build let’s
collect all the environment variables
let’s then publish all of these modules
to artifactory and all the dependency
information and now one of the things
that makes this
unique and I mean for people that are
new to cook fresh animal each of these
steps is its own container and so the
way that we basically built this sort of
immigration is we just built a docker
image to have the J frog CLI and the
goat ruling built into it and then here
you can see they were actually
referencing it says go binary build info
publish and then it specifies the image
it says this is the image I want to run
in this container to execute these
commands and then the commands are laid
out here and so very easy to build that
and if we weren’t going to make this
like a generic step that other people
and our teams could use we could
actually you know basically script this
and put it into the image itself and
send that set that script as the entry
point at which point people would just
maybe pass in you know they’re like a
variable that they needed and they
wouldn’t have all the commands they
would just have this is the image that
does this step and it’s reusable anybody
can grab it and it sits on the rails and
sent ready to go and so I think that’s
what we need to do next time yeah I’m
gonna be the next thing thank you yeah
absolutely
in this case I’m simply publishing all
the modules in the artifactory and the
last step I’m doing is I’m actually
uploading the binaries into the binary
release or ICS here’s my binary and
simply automated the process of making a
change in git repo and automatically
resolving all of the dependencies all
the dependencies that are already local
and then uploading them into our factory
and we could we could have tasks that
would happen before it’s before it goes
up such as
tests or security scans we gave a new
performance tests on these things and we
can basically set those all as
gatekeepers into artifactory to say you
know it can’t go to our factory unless
it passes these checks absolute move
that would be set here so I think what
we’ve done that we’ve kind of
illustrated the principle which is that
you can make your builds reproducible as
well as the artifacts that are versioned
in a semantic way using this new now
this gives us the sort of
reproducibility it does give us that
traceability and kind of auditability
stuff for performance the depend on your
setup you’ll see different performance
impacts and in most cases you would see
a slice speed up now that we’ve done
this to take it further we could
actually add in some caching so if we
want to go back to the slides
sure have weren’t you on this after the
demo and then the QA so we can actually
speed this up further using some kind of
fresh build optimization so this is sort
of automatic but when it comes to go
there’s maybe an additional step that
you would take so to do that first just
to understand why this is going to work
you have to understand it every step is
its own container now what we do at code
fresh is we automatically attach a bill
volume to your pipeline every time and
for those that are kubernetes people and
are interested how this works out of the
covers basically we create a persistent
volume claim for every pipeline so when
they rerun the pipeline we remount that
volume into those each container so that
so that the cache is actually available
so in order to now this is useful for a
couple of reasons in the pipeline that
you just did you had a step that checks
dependencies and it
variable where did that Maria will go it
went to the build bottom and then in the
next step it can look at that variable
and say hey are the modules there or not
the variable tells it no they’re not
here I was wondering about the
persistence between the steps yeah yeah
you you said it in the code fresh volume
path right where’s the variable for that
guy
so the next thing you can do to actually
speed this up further is we can use the
CF volume path as our go path now what
that will do is basically it’ll download
all of the modules onto that volume that
is persistent and next time it runs the
first thing we’ll do is it will say
it’ll basically looking at this this
first it will look at the build volume
and say hey are there modules here and
it will say you know if they’re not
there and I’ll say oh they’re not here
locally in the volume we need to pull
them then it will look at artifactory
and stay here the modules in my might go
what do you call it a go module repo my
go module repo and then if they’re not
there then it will go further into the
source and actually pull and then build
from scratch but once it’s done that
it’s now sitting in the build volume and
it’s now sitting in artifactory
so this gives just this will give a
pretty big performance boost
I think our build time on that was like
a minute 19 so really was very very
quick we could probably shave you know
30 seconds off of that by setting the
building ball you here and depending on
your your particular build this app is
very very simple
yeah it’s not a complex app you’ll get
much much bigger say oh yeah let’s let’s
hack that out after the the webinar and
we’ll live tweet that or something so so
that’s that’s what we’d do to go another
step further and actually speed these up
further and it basically is is like
taking it’s not it’s not like a
precompiled binary but you you and you
eliminate all of the network
overhead that you would have in this you
actually get all the beauty of what is
go because one of the module module is
actually prickly it’s it’s hard easy
semantic versioning of the source code
doesn’t anything about the works Wow it
takes advantage of all of this
intelligence yeah
great so now we can move into the Q&A
portion so someone asks what is the
advantage of getting a go mod from Jade
frog compared to fetch it from github
other users can also just browse github
and find other modules right also within
a big organization that’s absolutely
true the challenges that you have in a
big organization is one of control right
so if you proudly github for packages
the owner of the repository where those
packages are is the limit control of
those modules now if that’s your own
private repository then and you’re using
so you can have enterprise then
obviously you can optimize things there
but it doesn’t take advantage of the
semantic versioning the caching of those
in our factory and yeah yeah you you
even if you’re browsing github and like
the stuff that you control vs. and stuff
that other people can control that makes
a big difference right because if
someone else is controlling that go
module and you’re just always pulling
from that external source may be
embedded it may be you you know maybe
there’s some sort of process you might
wanna go so it really depends on your
organization and so in companies where
they need to put in place vetting
process over external dependencies this
is an absolutely critical step in
organizations that don’t do that kind of
vetting then
it’s somewhat less interesting yeah it’s
still valuable from perspective because
you’ve essentially got the tar.gz of the
code so some change happens out of fear
control on that source whether it’s or
somewhere else then you know you can
still reproduce it yeah well and I mean
the the final step of this would
generally mean that we actually build
the docker image which we didn’t do here
but the docker image would also be
persistent here both within code fresh
and artifactory so we could actually
pull that docker image to get that as
well
emmanuel asks in which artifactory
version is the Godot mod supported I’m
using six one oh the latest version is
six three three
yes I believe this so we have introduced
it in May that would have been this is
limited to go 1.11 and go one nine
eleven is new enough that it actually
hasn’t even made it into like after yeah
it’s not it’s not in the normal package
manager switch that dude when we know
the dark bridge which it had to install
from source we could use the package
manager to do it so it will I mean it’s
sort of cutting-edge stuff right it’s in
the latest release so if you’re not
using 1.11 go anyway you won’t have the
option there was another version go
prior to that but it was it’s a huge
advantage the version of artifactory in
which the go primary flows was
introduced was oh so many of 28 so
emmanuel should have this correct
available so permission
okay so Emmanuel feel free to reach out
about through the chocolate we’ll see if
we know absolutely Tom asked when I pull
a third party package down from github
and load it into artifactory does the
version number match a version the
original offer supply or is it a version
number only in my local artifactory so
the version number is if if the package
uses modules the publisher of that
module will have provided a semantic
version for it in which case it is bound
to the version that the publisher of the
package so it so it doesn’t matter that
you’ve pulled into artifactory this
romantic one of the things that we did
here which i think is is a best practice
is that we we report from code fresh the
bill by d2j frog and instead of actually
reporting a build we report the commit
ID so this is kind of a choice you can
do either I like to always use my commit
IDs because then I have just really good
parity from my source code through all
of my tooling where I can look at I
actually use it for my docker image tags
I always put my my my commission on it
so that if I look at a docker image I
mean we think code fresh I can actually
see what’s running I can click on the
image it’ll take me I can see what code
was made but barring that even if I’m
just looking at in terminal or something
I can see okay well here’s the
commission off and now I can look it up
so good that that concludes our
questions so next steps we’re gonna be
publishing this and you can share your
screen again to get started trying this
with Cote fresh we do offer a 120 builds
a month for free so you can go create a
free
fresh account and you can do this whole
demo on a freak out but I did you did it
on a freak out
yeah Craig was like can you give me like
the special account I said no just go
and forge a frog I believe it’s a free
30 day crap so we have a 30 day trial
and I hear the URL for the challenge to
go from this t-shirt basically walks you
through the manual steps to do the same
thing we’ll we’ll get this content yeah
and so look look for the blog post
probably today with just links to the
slides and the video so if you want to
send it on to colleagues so they can
watch it and then the code is already on
github I believe oh no the yeah Mille
isn’t so we’ll put the amylin today
perfect okay sweet well with that thank
you very much and we’ll see you around
[Music]