Kubernetes Operator For Deployment And Management Of App @ Open Source Summit Latin America

hey everyone this is om and happy to

share that I’m speaking at open source

Summit light in America 2022 about the

kubernetes operators and how it is used

for the deployment and management of the

application in container con

uh let’s have some small intro about me

I am home from India I’m a software

engineer at J frog from RND team I’m

also a certified kubernetes

administrator

and I I am core contributor and member

of cncf incubating project lit mask

yours

and a sporty guy I’m International

related chess player so feel free to

ping me for chase after this talk

um

okay so when I don’t contribute in open

source I work for the jfrog and I am

happy to share that uh one of 10 from

attendees

have a chance to grab the jfrog T-shirt

so scan this QR code and

register yourselves and winners will be

contacted through email

post event

so let’s come back to our top

let’s think about the

era how AI is changing how we used to

install our applications

uh how we used to deploy our

applications in earlier era when

physical servers was there and now how

we are deploying our applications in

today’s era in continualized form so

we have changed a lot right so earlier

days like physical servers then we came

to VMS now we have containerized

so let’s talk more about this yeah are

you ready let’s fasten your seat belts

and let’s come

so what are the containers basically

container is a unit where we package our

code

and all dependencies we uh for our code

and we keep it or we push it to some

registry so it’s more on the unit where

we keep all our required files to run

our application

and

there is there are platforms also like

Docker which is used for the shipping

and running our applications

so we have containers now we have

platforms to run it to ship it

now let’s talk about more like to next

thing

which is the orchestration what is

orchestration

or container orchestration is automation

of much of the operational efforts

required to run our controlized

workloads and services

it’s a more on the automation

configuration on top of the containers

so we we can see there are n number of

tools we have now like kubernetes

yeah which is a core CSF graduate

project we have also the Dockers from

mesos and many others

so kubernetes everyone is her aware of

the kubernetes and it’s one of the

booming topic like it’s one of the

booming technology in today’s era

so let’s talk more about it and like

next thing

so let’s see this uh picture what do you

think from this like yeah there are a

few hands there is a shape there is a

crane and small boxes like we can call

it as containers right

yeah

so they are arranging that containers uh

they’re putting it in some cream and

cream is shipping it to

uh big ship and ship is uh

moving around so let’s think like in

kubernetes Fair now kubernet this boxes

take it as a containers that crane is

like schedule a ship can be which can

think of the nodes now

and it’s more on like a application

installation right though the way that

the containers are orchestrated like the

configured and how the flow is going on

is it sounds good right

so

taking one step next that how kubernetes

create or modify our instances of the

ports that holds the controlized

application

it are holding and

taking it to next thing so we there are

many ways to do this

to make it installed to make it modify

so for that we can have

some manifest where we can pass this

actual configuration metadata specs

for this

application

so there are couple of ways through

which pass or we can install to make it

installed

so earlier days let’s think about that

yeah earlier we used to have some yaml

file light even if we we can see them we

have now also we have the yaml files

which we we add some specs

some configurations in that

like Port spec or deployment spec or

stateful state spec we add that and we

just apply it or we created that and

once it get applied it creates some

resources like pods or deployments that

full set right

so this is one way to install this

if some if someone says that I have a

simple application with one power one

service or some

with some volumes and all so yeah it’s

very easy and very easy it’s

straightforward to use the yaml files to

install that application now someone

says that I I need some customization on

that I need some manifest on the top of

that so that I can patch that

configuration into single MLA like it

can be yet installed it more

a bit more complex from yaml files but

yeah it’s requirement basis like yeah

they can use the customize thing now

someone wants to have their own

configurations in that we they don’t

want to directly take things and apply

to the magnifice like uh so they want

some configurations on the top of that

so that that can’t

and once application get installed they

wanted to do some upgrade and all with

with that configuration they can go like

you you may say like yeah Helm is the

good thing realm is the best option to

use that yes

but still we have many issues

it’s more on like the requirements

perspective

it’s more on the requirement perspective

but still uh if some

upgrade case came up where the DB are

schema changes let’s take example that

DB schema changes and we need to upgrade

the version of that and in the new

version there is some breaking change we

cannot go for the

customize or the helm thing there we

need some uh operations should we get

done with the configuration

so there were the

operators comes in the picture where it

solves actually the complex problems

might be there is a migration out

upgrading some breaking changes there

operator will visually handle that

because operator has the

reconciliation Loop where we can write

our own logic whatever the logic what we

wanted to write to get the upgrade

things so using that we can have the

more control over our installation and

management of the application

so in this topic like let’s move

so in this topic like we will morally

talking about the controllers operators

kubernetes operators how we deploy it

and then deep down into the capabilities

of the kubernetes controllers

so are you ready

yeah

kubernetes operator which is a

controller application specific

controller that can help us to package

deploy and manage a kubernetes

application

which is extending the kubernetes apis

and enabling it to configure our

applications application resources

using some of the structured and proper

way

uh

let’s let’s take example that

we have a stateful set right what’s the

use of excel actual State full set

response what is the responsibility of

the stateful set resource to run the

stateful application right so same that

like the same way

using the operator uh we are going to

create our own resource

so if we do the cube CTL get stateful

set we can get the stateful set objects

so now we have like own uh own resource

where we can Define our own kind on the

specific that under place of stateful

stateful set example so we can do now

cubesatel get our own kind name so there

it will rest out our own resources

it will have we will have our own custom

resource definition as well where which

holds all definition and once which

creates the

API for our own resource so let’s talk

more about this now

so

some of the use cases of our operator

is like we can deploy and manage our

applications with very seamlessly it’s

very good uh having flexibility and it

actually improves the consistency

we can automate uh as per our need

as as we discussed like that with it it

actually helps the upgrade cases and

much ads cases where actually the helm

and every other installation waves fails

operate against all that and it has

having many capabilities and many

features which can we can actually use

for The Operators and to build our

operator for our installation

uh it’s it’s it’s been always a

sound that it operator sounds a bit

complex there is a noise but uh early

what used to have happened everything

was like it’s more on like

to write writing everything from sketch

is a tight different thing but now we

are so lucky and thanks for the mini

Frameworks which actually abstract the

internal working of the controllers and

which provides us some functions where

we have to just write our own

functionalities inside that so that is

where we call it as a

reconceller which uh Frameworks provides

us where we have to Define like we

Define our own functionalities on if the

application is not installed it’s a

fresh installation we can write their

logic of our fresh installation logic if

already application is installed we can

write our own upgrade Logic for that so

it’s more on like whatever we want we

can we have the complete uh flow control

over there in demo we will see we will

see the code exactly where the

reconciler comes and where the crd CRS

and we’ll talk more about that

so this is more on that so there are

Frameworks

like uh there are some Frameworks which

actually provides us the SDK for the

operator

like operator framework um which is also

in CNC incubating project we have the

cube Builder kubernetes operator

framework

uh one more on one more thing that Cube

Builder also getting integrated like

integrated with the Opera operator

framework there are a couple of PR spot

merchs there uh we will see what Builder

is adding the point

let’s talk more about the flow of the

operator now so

one side let’s take the desired State

one side let’s assume that the user is

uh

seating and other size the let’s think

about the in cluster we have some

application installed right the current

state is there

so let’s see let’s take example of the

stateful set instead full set what we do

we add some configurations in the yaml

file and we apply the terminal file some

pores come up right so same way some

internal configuration is that which

takes the configuration for the yaml and

updating actually and creating some

ports with right whatever the uh

stateful set features have like suppose

persist and volume claim template has

provided on the top it will take that

and it will create volume hold amount

like and will actually use that volume

and other features also other specs also

whatever it is there so same here in uh

when in operator also we write our own

logic own reconcile logic we have this

uh operator reconcile comes where user

applies their changes in the Manifest

custom resource just think like example

as a state full set resource

where we update and we apply it create

support even if we next time we update

some image tag or if some

some if we update that stateful set with

the new configuration and we apply that

it actually takes the New Ports comes up

with the new configuration right so same

way in operator also in our resource

also we have a custom resource where we

keep our we have a kind their own kind

we have with their own specifications

which we Define in our operator

so these specifications we have to pass

and we have to apply that that once we

apply that our reconcile Loop will run

and it will update our current state if

it is a fresh install it will create if

it is already installed it will update

that

so it’s similar to other kubernetes

resources we are creating one more

resource in the kubernetes

with our own requirement with our own

feature with our own kind

so

when uh let’s talk about the recursor

loop how it is called and when it is

called so reconciliation uh basically in

operator internally what it is doing so

once the user applies our custom

resource that custom resource uh get

specs get updated into the adcd database

and we have the

Watchers you know on the event if some

event occur in the etcd database or

Watchers of those and it reconciles the

loop

there are multiple ways where

reconciliation happens if there is some

error throughout the reconcile Loop

occur it reconciles again

if some desired state or change

or some also the existing resource where

we set our owner reference which is owns

if the if even if there is some change

some failure occur in the current

existing application it reconciles or if

something uh changing our resource like

our kind resource then also it

reconciles on changing the

desired State and also there are some

default values also after that it

reconciles its default is 10 over

in some case it might be different if

some options are set so this is the

rinconcile loop where we get that

there are

operator SDK we are morally talking

about the operator SDK theme

we have taken that as example while demo

also

so operator SDK has the like levels

different levels where the supports for

the helm and symbol and golang

and suppose the basic installation Helm

operator suppose the basic installation

and the seamless upgrade where ansible

and go actual supports the all five

stages of the operator SDK of the

wherein first we we have just a basic

installation

in second we have the seamless upgrades

in third we have the life cycle of our

application even if the backup failure

recovery and these features comes in the

picture in fourth level we have the Deep

insights on the basis of the Matrix with

text and which is some uh some resource

example as a resource got exhausted we

for some power we scale it as a

horizontal or we perform we do some

operations or

it can be any type of operation and

fifth one is the autopilot thing where

like it’s a horizontal vertical uh Auto

configuration Auto tuning of normal

detection schedule and tuning and many

more comes in the picture

foreign

there is one question around like uh we

have customers and we is it possible to

is it running on every uh distribution

because someone uses the Azure

kubernetes engine someone uses the gke

eks and many other way so is it running

out yeah it’s operated SDK said wherever

the kubernetes Run operator will run

there

let’s have the demo time now are you

excited

um we just have the GitHub link here of

we have created like a work on the

potato head operator Potato Head is the

CNC of project to understand the micro

services so I have written their

operator on the top of that

so this is the crd

which is the

actually holding a custom resource

definition actually holding the

definition of our CR which we mentioned

uh which we Define that definition in V1

Alpha One this is the default generator

templates like default generator files

directory and all where we have to

update and some directories are I have

created

so V1 Alpha One API

so here uh we have to Define our type

what all things we have to take in our

uh kind like our resource so we have

like suspects status

in status we have currently I have

mentioned this many things

so this is the place where we actually

write our specs

type and we we run some commands like

make generate uh which actually creates

the Deep copy update that which which

helps during the runtime

which which Jeff validates like uh

the types what we have written

and there are where we after that we run

the make manifest command which actually

updates our crd which is under this

config crd basis

so once we run the make manifest command

this manifests get updated suppose some

tag we have added in the types dot go

with some flag with the Stream So that

definition get adds here and it keeps a

list of all our definitions

which is which we have defined in our

types what are the commands are there it

will be reflected at and that gets done

with the help of there is a control

agent binary

so

and this is the actual the controller

folder uh this let’s go to the main file

where uh where we Define our controller

the manager and leader gets initialized

and this is the controller where we set

to the manager

uh where

so here uh the just sometime back we

were discussing about the two things one

is the force and one is the ones four is

the

resource to whom the controller will be

primarily watching and the own sees the

resources uh which during this reconcile

during these things we create and we set

our own reference to that resources same

like if you see the replica sets like

replica sets own a reference to their

ports similar way now the operator SDK

sets the owner reference to

their created resources earlier we

earlier it used to support some

functions on the like on the basis of

suffix or prefix we used to

get a

set our

reference now we have owns

and this is the reconcile Loop which

keeps on calling on every change every

desired State change or some error occur

or some change in our force and owns on

the basis of force and on change and

also there is a default time there we it

get calls and it get rerun

so here we write our own logic create

resources and manage resources currently

in

the more we have

so

the installation will be like

first is crd definition installation

second thing it comes to since we we are

building this operator and we are

building a Docker file on top of

and we have to install this so in this

controller we are playing with the

kubernetes resources like

suppose we wanted to create one state

full set and one uh

config map so we have to give this

permissions we have to authorize our

container in the authorize our processes

inside the containers to create or

delete that resource so that all the

authorization has been done using the

service account

so that our back permits rollback

success control we create where we have

the service account rules and role

burning if it is on cluster scope we

have the cluster role and cluster

binding

so let’s apply this thing now

foreign

it was already there it just configured

that now let’s create the

I’m using the operator name phase

so it has created

now we can see the

operator content is getting created

so the this file uh we have a Docker

file here

yeah it is running now we can see it is

running

now let’s apply our CR

which is the custom resource let me show

you this

so we have a kind our own we have

defined our own kind we have defined our

API version group version and

version

this is a metadata uh this is spec we

can see like for other things suppose

example stateful set we have been taking

that so in stateful setup we also see

the similar kind of structure where you

see API version kind metadata aspect so

here also we have defined that in that

way

and we have defined our own specs now

here whatever the specs what we want

so once we apply this thing

we can see that once we applied this CR

um

that reconcile Loop call get a call in

and over their functionalities were

there to create these resources

so

it is called and see if we can see that

like all uh

ports are up and running so let’s have

some test like check against the upgrade

case how it will be having upgrade

so we have that csmcr here also

so here the version is

0.1.0 let’s let’s change it to

0.1.1 I have changed it everywhere and

let me apply this

okay let me confirm that

foreign

yeah it is already a 1.1 so it is

because of that it didn’t get updated

let’s let’s move it to 0.1.0 and apply

this now

yeah it has configure now let’s see the

ports

so older ports are going and like New

Ports has become

New Ports has been coming

let’s

Now new ports are up

uh let’s get our old description

describe our old

describe our CR custom resource

so we can see our version is 0.1.0

and our ports are also up with new

version

so it’s more like seamless and it’s

similar like we can write our own logic

uh as many as many as the features what

we want we can write that something is

breaking change we have added the logic

for that change

and if let’s take example like uh if the

application is running and

something failure happen we have we have

updated the Manifest and we have applied

this in upgrade case and something is

fell in that

image only and it’s not coming up and

running so what is in that case so we

can write a logic in that case so that

if something fails there it will

it will again take the older version and

again install that or it continues with

the older board

uh New Ports if there is some failure

rocker it will come go down and order

will stay

so that way our application will be uh

application will don’t uh will not have

a downtime

so this sub more on the operator so I’m

looking forward for more questions and

answers

and yeah someone wants to test locally

they can clone this repo and test

locally and

if there is something they can raise the

pr as well

and

thank you so much they can scan one more

time

and feel free to contact me on Twitter

Link in GitHub dave.2

looking forward to connect with you

thank you