Building a Kubernetes Powered Central Go Modules Repository @ Munich Kubernetes Meetup – 2019

so welcome thank you home tomorrow
thank you for coming and we want vision
that we enjoy with kubernetes and we
actually like so it’s pretty fresh shiny
and I will show you what we did to get
your feedback on this kind of this is
that hello word stop a let’s see so they
understand what it is about what the
application itself does and understand
that and do a quick history so don’t
forget the majority view and for the
rest of you just quick recap so released
in the being in development to since
2007 12 years now first person released
in a well 1.5 is an interesting version
because they moved most of the sea star
and the edit support for vendor Inc and
that’s kind of important for when
learning to talk about dependences now
when I was a few plugins were supposed
to be kind of another
in modularization of Rhonda etc that was
last year
and there is one thing missing and the
mission thing is managing managing was
starting because you are and they came
from obviously and because as long as
possible which is fine it’s brandy you
was released for something less than
four months
so now
are in the default which is so instead
[Music]
instead our
[Music]
what’s always been there for you know
van Buuren basically means I now the
dependent on someone else’s book so I
will take a snapshot of their code and
I’m going to commit it together with my
code to my birth control this is the
expression on your face is because
bender is horrible it’s horrible in any
possible way because what you actually
do you do a very very bad type of
forking you take some houses code you
disconnected from their history branches
tagging with all the meta data you can
meet it with your code so now you cannot
get updates you actually don’t even know
in what point of time you took the code
from the other person and eventually you
will self it’s implants in this code in
your version control that velji
completely from the original so the link
is a horrible idea
we didn’t have much options but now we
actually basically is that when you
declare a dependency what is going to
happen is go will go to the original of
this dependency and will build you an
artifact version artifact from the
latest style or the latest commit of
this Bible and it will be very soon it
will be specified through your metadata
and the code will actually continue to
leave in the repository you can always
find it there but there are some flows
in doing this process of I’m going to
get the source of my dependency and
create water on the client the
mutability is not guaranteed in this
scenario because in if in six months
from now I want to download the same
dependency that they used now I kind of
hope that the source code is to them
most of the time it is right people
don’t go into ether in our projects
right left got sometimes we do and when
we do it actually compromises their the
way that I can record it the other
problem is that drugs in github
are mutable you can actually force
commits into an existing table and that
means that even if the code from half a
year ago is still there and it’s still
tacked it might actually be other code
that the one if you build healthy order
and that’s obviously problem is one
right so it’s not a beautiful people can
go ahead delete their project
dependencies and also obviously trust
cannot comes from the mobility issue as
well right because someone heated a
motor home a new traffic dependency so
yeah this is me my name is barb and
developer advocate with zero also choose
figure officer
t-shirts and stickers there is like the
entire table of our stickers over there
if you didn’t take some go ahead and
take some after Mecca and wither in case
you I didn’t always begin before I
introduced myself to get in how it’s
important to follow me on Twitter and if
you’re still not convinced is actually
another slide
the most important slide of my talk is
this
so in the show notes is a special page
when you have a landing page they came
for the next up it has the slides
already there it will have the video if
I press all the correct buttons by
tonight you will get stickers all the
links that I will mention are already
there and you can comment there and wait
you stock and just a small raffle for
thanking you for being here is also
there right so basically this is the
only URL you need to remember
conveniently it is on every slide so you
don’t even need to remember that so
let’s now step into Co Center really and
one of these and why are we talking
about
so I know Center is a publicly available
source for go models right basically now
instead of working directly with Gina
and trying to review the sentimental
you can get a packed module from from
the center and it is for community it’s
absolutely free there is no like it’s
it’s not protocol in any way you just go
ahead and use it you don’t sign anything
we search in which you can check if the
module that you want to use is already
there and if not you can easily edit and
when you add to go Center we will go
ahead and capture the entire dependency
tree of the module that you are
interested in and make sure this
dependency tree is now immutable
persistent and you will repeatedly get
the same results when you try to resolve
is dependent as the male those imagery
deliveries for pretty obvious reasons
its research in general does a
dependency management and we have heard
about and jeffrey vachs rates etc
available for whatever your CI
environment is running on we actually
have in fact when x-ray installers for
every possible not every but all the
popular orchestration buttons so we have
we have one factory that ransoms work
which runs on mezzos DCOs and ransom
grantees so we kinda have experience
resolve them and we will really
solid choice of three and we can
leverage out the box features of station
obviously and then brings us to an oil
actually one very nice or appearance
over from the platform just because
again that’s been mostly true going to
be is offering today there is very easy
pleasure and we actually can benefit
from existing features like bigquery and
step rather that you can get similar
stuff by ourselves
but why bothering when we can have a
comfortable now how to build a more
there microservices application who
knows what’s this
yes manifesto open water that micro
services a micro services application
and throughout the talk I will try to
convey some – we believe that those that
are obviously answers all the factors
and all the requirements of the twelve
are two up and through we’ll talk about
some features and I would like to remark
them with both of those requirements a
certain feature actually observes and
first example will be the first example
we’ll be talking about feature that we
use and then serves a very dependent and
we didn’t have to build a
so for example you mentioned if we use
the popular commercial subscribe model
for our Q’s instead of going and trying
to find some solution for that we
actually used a top service the same
with database so obviously we have a
database of the modules that we it is
amazing a dependency manager for
kubernetes and we actually allows you to
they have verse which are there’s a
group of my services and there are we
separate values from the environment and
actually one of the founders of the
Reapers works for Jethro so it actually
was some very simple and basic best
practices I want to ask who is not doing
that just not to embarrass you and all
of you are doing anyway right so it’s
very important to provide image to your
nodes and pawns and sequence management
it to another obviously important stuff
use secret management provided fibre
where enemies do not commit passwords
into it yeah now the lip and also don’t
even use environment variables for that
because if you can log into your data
you
actually examining a file that contains
all environment variables and buzzwords
so that this has a pretty advanced
management namespaces and fosters a
allow you to use smart pipe lines on
similar and they to feature branching
won’t see there so let’s talk about
infrastructure for a second so I already
mentioned our factory artifacts
promotion pipelines we can have begun
tells for many many years and what we
preach is that after your after you
start using your software you should
actually promote it across different
environments in your repository a once
the quality gates requirements are
satisfied and from each of one
repository
you should deploy to the magic
environments random environments so we
can what you see here we have a factory
and then we have a bunch of docker
images and num charts and after each and
every requirement satisfied you’re going
to deploy to the right balance posture
and we try to put our money our are not
for Iraqis and we went and created five
different factors for for four different
environments so we have the CIA death
with the test will take station where
the prod and we have pipelines and
promote from one through
examples of what are the decisions and
upon which we actually promote and they
are absolutely similar the differences
are in the limits because obviously for
smaller environments we don’t need the
limits of production and it also allows
us creating the in one of them feature
branches without interfering with the
others so obviously in death and the CI
we can create future branches using
labels so they will be separated from
other containers and will run different
set of features so being tested and
verified because in terms of
architecture so we have six micro
services Ronnie the first is discovery
and the goal of discovery is getting
from somewhere the list of modules that
list of URLs that we need to create
modules for and this somewhere is
abstracted obviously so for now for
example if you add a new module in the
invite the discovery will kind of get it
and will be to build a dependency tree
and know what needs to be done and
another source for discovery to discover
what is the b2b modularized is the for
for view so for example and people try
to use
and they’d be finding more truths that
they want but we did receive their 404
request so we can start gathering them
and figure out popular libraries that
people are requests and make sure that
we go ahead and what are those another
option might be a finding popular
libraries by the number of stars and get
up so it’s kind of returning go and have
like 5 or 10 or more stars probably we
need to prepare a little for it all
those different kinds of discovering
what waters need to be notified not
whether it’s our the beginning of our
chat bots or slack but for now it takes
information from the Prometheus Edgar
fauna and reports in the slot events
like you know problems in production but
also Beauty nuts so for example when
deployment starts and it takes down some
of the courts because it’s going to
update them and we see that the polls
are down actually we can clearly see
their cabins in the context of the build
so we know that it’s not a production
issue is just a redeployment of the new
processor that’s that’s the guy that
actually got the job and we trance go to
create motors from the model itself that
our discovery service is covered and all
the dependencies and
runs parallel go clients in a salute
environment to create a previous Motors
and once this is done we have the
validator
which actually tries to build now
against those unique Reed modules and it
verifies that the resolution is correct
that the modules were actually created
now resolve from the center and whatever
it creates in the local cache actually
mentions to what we expected that’s our
current integration test and we divide
by N and acquire itself for the wide
statistics they’re ending its ever the
search so the only reason we should not
we can go is there if they’re not just
from that everything else is even go and
making the relationship this is much
worse
so seen here that if you are a user and
you use the UI go back and then interact
with the SQL Server give you the results
of research and if you submit a
inclusion request we will actually add
an entry empower subscriber for the
discovery service to discover it the
distribution is one of the theories in
you request a verifiable actually good
and
and process this request the processor
will actually go get not downloaded the
sources and do build modules of course
if appropriate are free and eventually
will deploy their artifacts that
produced into our factory into our
development repository from there it
will be promoted to staging if
requirements such as open source license
present exists because obviously we
cannot hold it like and then it will
also submit a request for a
valedictorian we’ll take the newly
created modules from staging and in
everything works will actually promote
okay so in terms of loops so there’s
about five so all this is actually
discussed and Factory is in a separate
grant sponsor and we are going to talk
about wine and there are Google services
like a passive investor basically I have
it here and you can see but that’s yeah
and what you see here is actually
started to see all the other kubernetes
services are here so near the validator
I’m staging right so we don’t expect a
water stop is going on and event
exclusion operator see ya so this is
page students by
[Music]
[Music]
[Music]
[Music]
you can see that so we were surprised
how well the scaling work for us to do
is capture we just had a short script
that run pupil commands and you know
and we are looking at two corner and
here powerful fill and I have the link
in the show notes which looks like very
simple straightforward a tool to run a
house a comes test himself the other
thing that I wanted to speak with you
balance on this scheme that we did so
obviously we have fools for programs and
one may think that notes can now go
ahead and relax its women the truth but
it’s the other way around so deploying
the user deployments to the realm
actually did most of the work for us and
this is kind of the compiled help chart
results which are also already
incorporated resources but they all came
from a from them you can see their staff
that our limits so we leave in this one
to eighty percent you and memory and
then we are going to ask for my will ask
for more notes and working now and
hopefully they are found and affinities
are also very very important aspect so
we use both affinity and affinity
affinity we used for assigning notes to
can be assigned to opinion stands
some members
it’s kind of an America that created
[Music]
been created by request and Kymco down
anytime we can all of that for yourself
and return our Father if you get all of
it if your obligation is stateless and
can be easily skill levels of mine I
will shut off with ya
and they are wait so what I wanted to
leave a non-market for God is the story
of how a company say I think up to 30%
under a cost by using those those notes
obviously not four of them so for
example
you have an R factor which is stateful
because it has like storage and
everything in state so we cannot use
that for that but stuff like most of the
most of the services promotional can
actually use those thoughts and the
affinity is a way to declare that some
points cannot coexist with others right
so this example say okay whatever you do
do not put this this point
together with December’s and then
because you can optimize the different
things if you have nodes which are CPU
intensive such as the processors if you
can bother them with nodes which are
let’s say memory or never bound and then
find but not with other nodes which are
fauna that’s kind of normal stack that
you used for a logging monitoring and
the mobility again yeah so I hear
so that’s the screenshot that I do on
testing is from here so basically you
can see here as I mentioned we don’t
really need to watch this all the time
because we have those notifications q
slack and then we’ll only get
information when there is program
sidecars you
you know the drill
right if it’s normal to have one
container of paranor code but their
Arsenal’s
are for our a web the web through my
notes so reports in which we bundle them
with the nginx and that makes very easy
for us to configure access and that’s a
useful actor that’s exactly the example
that I will give you so we have an
efficient on Jenkins with beautiful
started and then we can see here
somewhere and bounce okay that’s
configures and ready and now it’s ready
this container is already and now it’s
ready and because it happens after the
job started and before the job finished
we can actually make sure that that’s
okay is just a new deployment and we
don’t need to check we obviously can do
that we’re here we can actually
eliminate those altogether or remove
them to another channel so we won’t even
bother to you know to check if they are
that we actually plan much more scary
because we kind of have them all which
is so first of all scaling of notes
obviously when we have too many ports in
one node we need you know
compute our the service that does the
computation the build itself the modules
is a CPU intensive on the runs go
clients that you know calculate depends
3-0 it’s not so scale based on in the
excuse so obviously it’s very easy to
scale based on number of messages in
queue if we see that we have a lot of
messaging in queue for asserting
operation that means that we need more
ports of whatever operation it goes
right you see that we have too many
requests for discovery because the only
thing that we miss here is automatically
scaling average battery instances and
that’s because in kubernetes as of today
there is no way to scale based on
network based bandwidth bandwidth limits
and that really is never bound so it
means that if it will utilize your
network a interface because before it
utilizes your CPU or your storage
[Music]
to see some day we can say if we see our
network goes above 50% that what
actually delegate is pretend to spin New
York factory notes and our Factory on
kubernetes is completely elastic ready
to spin as many notes as you want you
have a question – surfaced in your
license bucket but for us
fortunately we know someone who knows
the real issue is that we really don’t
know how we need them when can we so we
have a road map mostly on the product
side how we’re features to go centre I
think one of the most brilliant ideas
for us will be providing some insight
the quality of the dependencies that we
are serving and therefore obviously get
more viable Center but also some a
improvements on the technical on the
operating side well so as I mentioned we
really want instead of just slack
notification going to to chat ops so we
can for example stop builds using a
using slack or a spin up and down
services using slack so as I mentioned
between unfortunately need to sit by me
by your own eyes that some hard
Network is choking we can actually speed
up one factor services by just sending
your messages into our web channel the
same in Gloucester Factory
once we figure out how to scale
automatically we can actually
we are and this is a great question
thank you very much those actually have
Q&A section in which I also this is a
question
so first of all the beauty of those
Center as opposite to other several
depositories from other languages in
communities likely to be registry or
growth center is that there is really no
dependency or no single point of failure
in the center because it’s just a matter
of one environment variable that
distinguish between you you work with
job center or you go directly you know
and it sounds in heaven though some who
is down we’re out of business
what hopefully your dependencies are
still an anomaly world right now from
the perspective of what are the chances
that one will happen compared to another
we’ve been in business for 10 years and
we are a billion dollar evaluation
company comparing to some guys that but
the real the real advantage here is that
as another point of redundancy without
adding another point of failure so it’s
just a matter of you more so Google and
and Russ Cox announced just couple
months ago in December just before
holidays that he sees this vision of
figuration of several reported words in
which Jeff from has one Google has one
maybe minus one has one and we all share
those waters to provide more redundancy
and it’s really up to you to which one
you use you will get
[Music]
[Music]
because if we already captured your
source code and building module out of
it
this module will be in the factory
servers of go center and whenever you go
your module will get the same for real
what the regional was now it is really
confusing because we claimed that during
that pollution also the utility version
one means one thing but it will look at
the sources of polish for some activity
we see other thing and then goes back to
who’s right here what happened when we
have other content and what we actually
think about is providing the rating for
dependencies and actually working people
from using the dependencies that show
one thing in the source but actually
have other thing as
as a builder but there is no way around
its conflict and if someone already used
the previous version of 1-0 it should be
consistent hey Perez identify this is
more for your electron hi farm and
pushing it together and we’re only like
the numbers of actual rates it was a
convention down there
there will be some separation there that
will be mostly oh yeah yeah absolutely
absolutely same definitely would help
and we’re going to use Google’s inner to
catch the motors closer to the noodles
and we are also the first testing that
we need is nothing to there very early
adoption that we see now so we loaded a
forty five thousand more troops in just
couple of days to preheat the course and
also we’ll have enough a popular models
of all we can build and run and
architecture is the last enough to stand
a stand alone that we can imagine
telling live down spits a at the moment
differ from place to place just because
we didn’t plug the city and in yet
saying that even with those pregnancy
issues that you actually have to go to
the Google servers to get the artifacts
from
and comparing the Eternity and that’s
chronic sources use the clock protocol
from Gina and then building those
archives on your machine every time for
each and every developer the performance
gains are already enormous even before
we actually started playing in decision

3:35:03
NOW PLAYING
Operating System Full Course | Operating System Tutorials for Beginners
Academic Lesson
280K