K8s Controller Reconciler Logic in Go @ Kubernetes Community Days Sri Lanka [Live Series]

welcome back everyone we have reached the second session for the day of casey srilanka and let me quickly introduce
you to the second speaker for the day say hi to om khali who is a software engineer working at jfrog and who is
working on the installer team to build the kubernetes operator for deployment management application but if you leave his tech part aside he is also a sporty
guy and an interactive religious player so say hi and let’s welcome him and he
can stay and take over the street hey everyone hey everyone this is om here i’m so
happy to speak at the kcd sri lanka 2022 uh today i’ll be talking about the custom controllers and reconciliation
power of controllers and how to implement it and what are the advantages on top of it so let’s discuss
on that uh thanks for already giving us our intro i’m just skipping this slide
so i have a question uh first question for everyone uh what is controllers like uh we have been heard like hearing from
a lot of about the controllers and all so what exactly it is so
just think so basically controller is the way where uh
it’s uh we wanted to achieve let’s take example now uh
let’s take a live example like we have lot of resources existing right kubernetes resources like deployment
state full set secret and n number of resources right so if we observe that uh if we have a deployment
uh with the replica tool uh even if you delete two of replicas new replicas comes up right whatever the
configuration we give in uh deployment.ml5 that resource file so that configuration
comes into the uh picture in form of like resource come with the pod where it’s all configuration
so how this is happening like how who is it or who is controlling that pods or how the
this is getting configured and all something is there right something is there that is taking care of all parts of the
resources and which is actually making it run and keeping it healthy and everything uh so something is there
right so if you observe so that exactly where the controllers play role so that
is the controller who is responsible for taking care of pods like replica sets whose responsibilities to keep their
replicas up and what are the replica counties there so this is a where the deployment is our controller
stateful set how the controller every every resource how the controller right so what that controllers do basically
that that watches for the event uh what are the resources we provide and
its responsibility is to take desired state to actual state so that’s where the controller comes in let’s discuss
more about that like we just talked about like controller watches for the changes to
specific resources in kubernetes api and it reacts by creating updating or deleting the resource
uh we will uh go through writing a controller in go in today’s talk uh also the controller
is a reconciliation loop that reads the desired state of the resources from kubernetes api and task
uh text section to bring the cluster’s actual state closer to the dji state so
in the controller uh in order to write controller in today’s talk uh we will use the operator sdk
so it provides the controller it provides our complete uh framework on the bootstrap bootstrap
controller where the already main controller logic is already there where we have to taken care of the reconciliation loop so we
don’t have to worry about lot of each and everything about the controller internally it it provides a bootstrap
way operator sdk so we will use that it will give a framework uh where we it will provide us
a place where we have to just write our reconciliation logic so for the operator sdk installation uh
there is a documentation of setup that is the prerequisite and just we have to run a couple of
steps like this operator in it to enable the operator sdk thing by providing the domain name
example.com our repo whatever the repo we are we are going to have to set that repo inside our operator sdk setup
and the second thing is that we have to create the api for our controller it’s a control api we can provide the
kind whatever we want in today’s demo uh i have taken a portrait to her kind so here is the thing to understand like
we can give the pre existing resources kind as well like for uh deployment any kind we can provide or
we can provide our own kind also where we are going to create our own different custom resource only
so if you provide the if you use the default resource like uh existing kubernetes resource then you
have to give the kind and the source is equals to false that that thing have taken care and once you get uh
once we run these commands we will get one complete uh setup from operator sdk
where let’s have a look against that now
so i already ran those commands and i have already
complete setup and installation reconciliation logic and everything is written here to understand the stuff
and so to save the time so this is the complete folder directory structure we get from running
these two commands operator init and operator create like api so once we run these commands uh
we get this type of file directory where this is the api folder where we
will write our own like we can write our own custom resource kind definition
and this is the config where all uh config manifest will get generated like
operators deployment manifest like our back rules default uh crd which is the custom resource
definition for our custom resource and couple of other files like customize
files as well which i have removed unwanted files so this is in config we get that uh next
thing is like the main uh gateway of the operator here is for controller is like
for the main.gov where we set up our some a lot of things like for the matrix for prometheus uh matrix powerport uh
logging say logger stuff and we also set up our controller also so in uh here i have set up the
controller stuff which is like this is the
api definition which i have taken and in that in that definition we have
this definition mentioned type which is the our kind putter to head the definition mentioned we have registered
that definition with our controller we have set up with the manager and this is the gateway where we register our
staff controller where we the api thing where we set up with our manager and a
couple of other stuff also we get so inside this now let’s go to
this definition setup so in setup with manager stuff uh
here we have two things now fours and ones so basically this this function is
responsible for watching the events again against our resources
so so what uh i’m trying to say is here currently in today’s demo uh i have just
generated operator sdk generated one api which is supported to head
which means that we have created our own different resource custom resource for that we got
apis which is holding definition for that resource now we are using that definition
inside our reconciliation loop for controller and we will have a watch against that
and to register to how this is getting watched so we have to we have to set that to whom we have to
watch so the primary is like the uh here is the portal which we are like going to
talk more on so this is the new kind we have created api for new kind like currently
example like many like uh like service for
we have been talking and taking example like so similarly we will have one more resource now ported to here like similarly like if we
like we we do like qc tailgate parts now we can do the cubes it’ll get ported to here so it will give all objects of that
portrait to header resource so in four we have to set the
for thing for our controller to primarily to whom
it has to the watch the events so whatever the change happened in this portal research like in create delete
upgrade so any event occurred any new stuff happen this will watch and once it gets watched
it will reconcile what it will reconcile this function like reconcile which is the bootstrap way we which we got from
the operator sdk which is the controller pleasure one main
method where we have to focus on now and first thing is for second thing is once so once is the like basically
setting up the owner references uh what owns to a what wherever uh once we
have set up suppose the spotted to head resource has created one deployment
and one service so anything occurred in deployment also any event occurred
against deployment then also it will reconcile so there comes the owns comes into the picture
so we have uh we have created our operator sdk setup
we have created api portal uh we have discussed about the
directory structure and the main file inside main we set up with manager our resource so inside that main we have
force uh we can set the force and once for is the for whom like primary
resource where we have to watch and reconcile on the an event and we can
also set owns and this controller manager this uh there are a couple of others like many
things get supported we can add our some specific resource and resource name also
to have watch against and only that resource suppose deployment is having and deployment some
name is there so we can have events against watch against that specific deployment only so
that can also be possible so now this resource we have said force and
owns uh now we comes to the reconcile let’s talk about the more on reconciliation
so as we discuss now controller is the where uh controller’s main responsibility is to
take uh desired state to actual state how then internally having sync uh it watches the
resource if any event occur it reconciles so let’s talk about the reconciling
so in reconciliation uh we have now resource right custom resource
now we have to support we have to give some uh
responsibilities to that resource right like we discussed earlier uh replica set what is do its responsibility is to keep
your ports up some parts goes down it again stack up whatever the replica account it has given to replica set
through deployment it will be responsible for managing those replica account right so now we have our own
resource which is the portal to head the source kind name we have given
so what is the responsibility of the potato here uh here let’s uh take example i have
taken this portal as example here it can be anything like uh for example kcd sri lanka 2022 we can
have kind name kind api so what is the responsibility of photo a
responsibility of the porter to head to create all the deployments and all the services
all resources which is required for potter to help application right so let me take you to the cr now
so cr manifested this so
we have a kind porter to here uh we have metadata we have specs like this is the
structure right every kubernetes resource inside the structure if you go for power then we will get the power specs
and other resource packs we have that specific uh specifications now we since we have writing our own resource now
for for the controller uh to watch so in this specs we have
defined our own definition so this is the definition we have written now the controllers
thing is that two uh to do the whatever the responsibility
wanted to give uh that we have to write here suppose that is the cr right so what this crap cr having some uh
some specs written right main heads left leg left arm so this is the file where we
have our configuration specification now what is the controller’s responsibility controllers responsibility is to
get that file to get that resource file get all specification and create our
resources what resources uh deployment services suppose secret
persistent volume uh any number of things we can create here so reconciliation is having like a it’s a
it’s a lot of things which is handling by reconciliation only so where we don’t have to much worried about internal
working where we just what can do now we have to get the our reported to her cr so
uh we’ll talk more about like we’ll take on demo also by installing the commands like applying the manifest and all let’s
go through the code base first so what is the main work here is like we are getting
that resource whatever the resource we have applied suppose this manifest we have applied see alright so
we have appliance means that it has uh created that object in that api portal
right so it means that some event occurred event occurred means that this reconcile will get called once this
reconcile gets called these functionalities we have now what this functionalities do
it gets it is having gate call to kubernetes api server uh to this end plug to port to head
resource and we are getting that object in this uh
potato head uh environment but so once we have this now we have if this environment is having all
cr of this right so once we have the cr we can we have the specs our details so
in this demo we are talking about the create resources and manage
resources so what is the written here in create resources uh create is basically
so this object is having all details of our cr custom resource
so what is the create resource we’ll do now create resource will have that
object we have passed that object and
in this object also what we are creating what all the resources we are creating
we are creating one uh deployment and one service
so in deployment we have passed that object which is holding all specs let me take it again so this is the uh you know
this is the env which is holding this thing how through the api now at this point
we have this all cr specs everything for the complete object now
we have passed that we are trying to pass that object through create deployment and create a service where
create deployments uh is responsible for creating a deployment
for like easy uh understanding and very like to get things through one blink of five
i have just uh taken it outside by one by one kept r so we have just initialized one empty object of the
deployment and we have a gate client against a gate function against the
deployment uh that whatever the deployment we have so basically what we are trying to do here is
uh we are trying to get the api thing to deployment if since
if we fail to get that object in that deployment if the error is not equals to nil and if
that object is actually not found on that api so then we have to create so basically
what is happening it is just checking to the a deployment api server this object
whatever the empty object whatever the object name and name space very fast it is there or
not if it is not there then only we have to create right so if it is not found then only we can
have a create api call which where we are passing the name from that uh getting that by dot thing
potato dot specs dot main dot name which is the name of like uh image which is the environments we
are passing to this function and what this function will return this
function will return the object of deployment with the provided specs and it will create that
we will go into the deployment uh one thing let’s talk before that so why we are doing this is not found
right so we are getting the avr having the apa call to get the deployment now
there are let’s talk there are a couple of ways we can get this error one is like actually
deployment is not found and let’s take example one is like we don’t have permission itself to call the
deployment right uh we haven’t provided the our back permissions to get the deployment resource
uh or it can be have any kubernetes error or kubernetes also
so here we clarify that if error is not equal error is having not equal thing it means the object itself are not found
that is the only thing not the other things are there not other error is there then we can create if
there is other error we don’t have permission then it’s not worthy and to get create also so we are not clear
about it resources there or not permission only we don’t know so in that case like we have to handle different scenario so
here we have if the object is not found then we will create it and this rs.deployment is returning a deployment
object now how it is returning let’s discuss now so
here i just set up this image is not uh if it images now we are setting up the default image if it is passed from that
cr only this all enemies and all arguments is taken from that cr only if you see here
ah so in this case we have this name uh in this deployment we are passing couple of ways
uh name of that image namespace env
and in this deployment we are taking these things from that this everything is coming from cr spec only
custom resource pack and we are just taking that thing and we are creating a deployment object
and updating that thing like name as a deployment name adding couple of labels match labels also to whom we chose to
match and we are giving the labels to pod
and inside the port spec we are updating that name image i mentioned policy
so this env and all whatever we have this is taken from crv cr this our kind
what are the cr we have from this we have taken and we have passed these two deployments
and we have the deployment object now whatever the deployment we have in helm
uh we get templates right what are the values dot ml we have we have the helpers and we
pass that values to templates so similarly in uh operator controller we
take things from cr internally using the golang we pass it to our objects
and we we also set our own reference to that object so who is the owner reference of this thing is the
portal to head so that way we also set the pointer reference and
we return one object what this object is now this object is here
at this point we will give a deployment object now this will create api call that deployment object
and it creates actually deployment so similar way uh similarly we try to get
that deployment if it is not found then we go we take one deployment object and we create that deployment object
so similarly there are a couple of things like left arm imported to it a couple of services resources we want that all has
been created in the similar way one by one and also in
service also to expose that to outside world also from container we
have creating a services also and in service also similar thing we are getting that service if it is not found
then we are creating that service by getting the object service schema object
like this we are setting up the step for the service we are setting up the owner
reference of for that service and we are getting that service here
uh like so at this point we have taken cr from controller this is the cr and we have passed that cr object
to our create resources now this both functions are getting called and this will create
if it is already created we are simply written in null so if it is like if the error is
nil then we are nothing we are just returning at one point
and just second sorry
sorry for the distance so
feel free to ask any questions and all so
controller is basically it is what it is doing uh it is having the watch against the
event whatever the api kind we are given and if any event occurring uh through
that to that kind or resource it is reconciling
controller and after reconciling we have given our own responsibilities to that
reconcile loop controller is reconciling in this uh demo what we have what
responsibilities we have given we are taking up the resource
and whatever the kind we have provided we are on the change on the event soccer that
it will reconcile and in this reconcile we are getting that resource and once we get that resource
we are performing our operations like create operations where we are creating n number of deployments
and we are also creating a resource services where for the deployments
and we we have created things right so now let’s take example that uh this is the
file uh we have applied right with this all spec now i wanted to update the version
to 0.1.2 or 0.1.0 some downgrade or some name we have to update here or
namespace anything wanted to update then we will come here we will update similar like our
stateful set only like stateful set deployment what we do we just uh deploy the application we wanted to update the
image stack we come we update the image type in deployment specs and we again apply that deployment email right so
similar way we wanted to uh come here and we wanted to update we will update this and
once we updated some tag name or something again we have applied this some with some change with replica container
again this event will happen right again some change happen to that resource event will happen this will reconcile
after watch so in that case uh we have already created resources right but in this case
our resource has been created but some of uh some modifications happen in this uh specs
or some cr in that custom resource if something is changed right now we have to upgrade that application internally
so once we update the deployment image a new old port goes down new best new comes up with the new image right so
similarly we have to logic for that also which one is the manage so in the manage uh we have given support for image right
now so any event of updating happens or it will reconcile it will update the image
so how to do that again the same thing uh we have an empty deployment now
uh this code one can say no i want i can write one this code in more uh
modified way or not yeah i’m happy i this this flow is written just to get simply understand
so we have a get client get api call to deployment uh if deployment is having error is not found or not in this case
we we are talking about the manage resource right if api is not found we are we don’t have to worry about we have to
simply return it because we have to manage if ap if this object is found then only we have to update
right so in this function we are simply checking the whatever things we wanted to check
we are checking the replica account if replicas count is not equal then we are updating uh
that deployment with the newer replica account okay so image or replica account
something change we are uh updating that and we are having the update call now to
whom to the deployment so
so suppose i have updated this version tag and update apply this manifest so in
once it gets applied it will reconcile so it will not go in it will go through the create functions uh
but uh it will return error it will return null value since it is already there all resources are already there it
will return just nil until and unless some kubernetes error or something is not messed up so
once we get that we will go to next thing which is the manage thing in that manage what we are doing we are just comparing the actual state versus our
desired state how we are comparing uh we are getting one object that gate object one empty uh
environment we are initializing and we are getting that deployment inside we are getting the actual deployment
inside this uh variable through this api call now this deployment is having actual state and our this cr is having a
desired state so we are simply uh passing that to some function call and we are checking the
actual state and design state if the actual state is a desired state is not equal to actual
state like there is suppose size or replica account then we are simply updating the uh
actual desired state with actual state now the new actual state is our desired state
so this way what we have done now we have written logic for creating the resources and once we have created that we are
managing the resources by matching that actual state versus the desired state
so this is where the controller’s responsibility and controllers play role so now here we can give uh we can have
end our use cases like which we can see like helm don’t support or help don’t solve like upgrade cases
if something breaking change is there we can simply write our own reconciliation loop
and we can uh set our kind either we can write our own custom resource or we can use the
existing uh resource kubernetes or we can write a reconciliation controller loop for that where we will write what
are the resp what are the logic complex logic which we wanted to solve using controller we can’t write their logic
here suppose when someone wants to update the some uh
for specific things uh once the parts comes up they wanted to add the dynamic level they can come and write their own
logic uh someone wants to play with the service thing so what are the response
what are the uh challenge we are facing in current stuff we can come here and we can write actual
codelang logic which is our much more power and we can directly write here whatever the
logic we want and we can test now so once we have done that we have to build the docker image and push it some
registry and now what we have currently uh we are basically
operating so i already built one image uh potato head demo operator with the demo tag and
we have simply this so this is the deployment basically which we are using and this is resources on we
are given this is the image tag and this is the binary uh we are just
while building that we have passed the we have images the binary thing and we will just run the
controller stuff so how let’s paste it local now
we already have commands here so this is the github where i have kept all the stuff so we can simply use this
stuff here uh kubernetes uh cluster i’m using the docker desktop
cluster and let me apply this thing crd has been created this is the
definition file before we apply if we are using some we are using going with the custom resource flow
in that uh manifest if we give the our own kind a different custom resource kind then we have to give the definition
for that to create an api so we have applied that now let’s uh
install the permissions and all uh deployment file and all are back and
that’s created in operator namespace and we can see now actually
uh operator pod is coming up let’s have a watch against this it’s what is this operator is doing the
same thing what we have discussed now reconciliation thing it will keep on reconciling on every event and
that all functions will get called so and that functions we are giving our permissions through this r back
let me take over through through this now so
for the demo we have a cluster role and we have given lot of permissions here one has to reduce this in actual
implementation so this cluster scope is having uh this
internal stuff that porter to help status also we can create update delete and all uh finalizers also potential
heads resource and this is the our resources now right straightforward set uh it’s the default
resources we’ll be on service account in this like i just for demo purpose i have kept all permission
said what like we can do like this and like for now we wanted the deployment and services we have to like
just like we can update we are doing the update thing we are deleting uh we are creating
we have a watch and we are get strings you can have a list also getting all
deployments so like this we have to give permission so basically service account is used to authorize the
uh internal processes like processes which are internal like inside the containers to authorize to the api
server with these permissions like so they can now create deployments delete update what are the works we have used that things
with that controllers process can do now so so let’s talk uh so this is
deployment uh for the operator and are back on all we are given
so now let’s talk one more point here is the scope thing uh in controller operator we can scope the
controller working uh there is a one more environment only up one more thing we can pass here like
the name space so operator so whatever the name space we do here
so it will reconcile to only that frame space resources only
and if they we are going with the name space we can actually have the our back also
on roll not actually the cluster rule but for demo uh i have been i’m using the oculus cell scope only
uh on the cluster scope we will have some security concerns like having this permissions for the secrets or not
it’s better to have a namespace scope operator or controller
so now we have discussed our stuff and now let’s apply the so we can see
now our operator is running now and let’s apply our cr
so inside operator we have logic for that like once we apply cr it will reconcile it will create powers and all
all things so let’s have in different ace question
i’m using k9s to have the watch okay so operator is running now let’s
apply this thing
we can see now we have just applied our cr okay what this is exactly uh this portrait to hammer like config cr
manifest portal to ml this is this file config cr manifest ported to head.tml
so we have already set up the for for this resource and it is actually watching events again this resource so
we have applied this right so once we applied event occur once the event occurred
it went through the our reconciliation loop like create and manage so in the create we have actually taken
this portrait to head object resource object inside variable and we have passed that variable throughout our
methods and using that we have just checked we have created all resources if
resource not found effort is already there it will manage now we have seen the create stuff now uh
since we have once we applied that it has created all the deployments and services i just i showed
you right some deployment functions and service functions uh let me take you to again here
inside the create we were talking about like creating uh deploy and service services like the
both things has been created now so these has the ports are
hvc we can see services all the services has been created
uh through that functionality methods like this uh
when down all services has been created similarly we also got the deploys deployments all deployments has been
created so this is how the controller is uh works
basically we have logic there we are using the bootstrap way of
controller through using the operator sdk we are we housing writing logics and we
have actually seen the installing apply now let’s talk about more um
upgrade case right we have written one more method is like one is the managed thing also what this manage thing do
it’s things the replica count and also it uh sees the image tax and all okay so what
version we are using now let’s change some version to 0.1.0
okay and replace that now this cr is a change like something without
changing this now let’s apply the new things here okay
okay so let’s we have change the operator image tags for all deployments and we are simply applying
it right so once we applied this again some event occurred to that custom resource uh
and once the event occurred reconcile reconciled loop has been called and now this loop since it was already created
we have written nil nil everywhere so objects were already there so these things will not uh
come into the picture more here now this field will come into the picture where actually we can see the actual state and
desire state were different image stacks were different so in this that methods goes it actually
updates our resources actually updates our actual state with desired state and we can see our new tag is the 0.1.0
which is the so this is how the controller basically
works we have our new parts and new resources come came up
and we can actually play around we can actually write all n number of functionalities here so
for demo purpose i have added only create and manage one says like okay i want also have some
check against that envs also if some envs negative or some user has passed some
wrong values i wanted to have check against that against this cr also so they can simply
write one more method here for this object check also uh someone is interested in saying that okay i wanted
to also have some methods like more functionalities this is not solving my problem i wanted to have on top of this
one more method which i want they can simply come here and write one more method there and they
can have complete control over their resources either it’s a default kubernetes
resource kind uh it’s or their own custom resource can they can like we can write simply our
logic and every reconcile will like basically takes care of it uh also we have set the
owns also right we let’s talk more about now so owens and what this means you want this uh
what are the resources we have created like deployment it’s also having watching the watches even against the
watch on the diplomats events also how now let’s take example i am deleting one
deployment left leg okay i have deleted
so so left leg has been like uh
so
okay yeah see so we we recently deleted the left arm right
uh this is so we can see now just some time back it has like again created a new one so it it will reconcile on
changing our resources also like what are the resources we have for those resources also so newport came up right
so if we delete this for if we delete this service again new service will come since we have that if not found
condition uh we have given owner reference set so if we pass
this it will reconcile on change in any deployment uh event also if you delete the deployment then also it will
reconcile and create again new so uh any event
against our owns also it will reconcile basically so this is the application now we can
put forward and open uh if it is if it is clustered if it is load balancer or like we can
use open that and we can check it actually
so it’s much seamless and straightforward uh writing controllers and since uh we
already have a lot of like bootstrap frameworks also like operator sdk to get only to the focus on reconciliation look
rather than worried about internal lot of things sinker watcher and
so we don’t have to worry about that we can write our own controller logic and we can write our
own responsibility what we wanted to do and we wanted to solve our complex uh issues
so sounds good so i’m looking forward for more questions and all
so one can come and like one can try this repo uh it’s own colleague a kts operator and they can play they can use
this code also it’s open source and write their own droplet on top of this only
so using like some of the examples just uh is there any question
okay so some of like examples just i have taken like we can take like deployment
management we have good flexibility and improved consistency we can automate our needs to easy achieve competitive
compatibility uh auto updates for the application uh taking and restoring the backups of the
application straight forward it’s very excellent for having the state management also so we can have our
own api we can stimulate the failures in all parts of color cluster we can also focus on more on the resiliency also
through the controller as like this reconcile basically happens
on if any error occurred then also it will reconcile what are the changes we are happening in desired state like force and owns whatever there also it
will reconcile and also we can set our sync time and all also so after that specific time it
will reconcile if we said that so it has done already so this is like
operators basically flow where it uh whatever the user has been saying like we have applied that
manifest right cr manifest so custom resource we have applied or we can have watch against our own
like some kubernetes resource we have given if the custom resource flag is found then it will like just observe the
existing source in this like operator we like custom resource it will basically watch the changes in even once it was it
will reconcile it will take to the current state like actual state what are the result status
uh we have gone through the demo and i’m so happy and thank you i’m looking
forward to connect with you all on the twitter linkedin or like twitter so feel free to reach out to me for any
questions or anything thank you thank you so much thank you
thank you for that wonderful
okay thank you all of us thank you all of you all for joining and uh let’s meet in the
next