Python at JFrog @The Chicago Python User Group (ChiPy)

hello everyone uh welcome to this
evening’s edition of uh the chippy main
meeting we have a very uh special
edition this time for you guys it’s the
annual python power business event uh we
have some really great speakers today
i’ll be getting to
those individuals shortly uh but first
is to uh do a little bookkeeping here
we’re gonna go through
our our standard kind of chippy layout
um
just to
introduce chippy and make sure everybody
knows who we are what we do
so again welcome to this evening’s
edition of the main meeting which is a
special edition of python powered
business
i’d like to give a quick thanks to our
sponsors jfrog break tree and zorro uh
they provide a lot of funding for us to
make these type of things happen and
keep our website running
and when we do go back to live events uh
getting everybody a slice of pizza at at
these events
now jfrog is uh one of our good sponsors
again they are
a sponsor a raffle for us every
month uh this month we have three
adafruit clues that
are being raffled off they are circuit
python compatible you can
scan the qr code to enter or you can
click on
the link that will be dropped in the
live chat
the winner will be selected in two
business days from this event
and then the winner will be shared on
our meetup page and congratulated on
social media i will post this a couple
times throughout the meeting just so
everybody is aware
chippy does have a code of conduct if
you’re interested in
and what that is specifically please
click on
this link the chippy.org pages slash
conduct
basically just we ask everybody to pro
to behave in a professional manner
what is chippy chip is a python focused
community that was started in 2003.
our board members are myself aaron
elfquest is the secretary chris ludite
heather white joe
we have several initiatives with chippy
one of those being the project night and
wedding field which will be held next
week uh buddy mentorship program at
chippymentor.org
we have a website set up where
if you want to mentor people
in python or be get some mentorship
we’ll try to
match people up on that site by their
skill levels and
what you’re looking to learn
there’s the python lunch break there’s
an elbow sig and then we do have a job
board on chippy.org
there’s several postings on there right
now if you’re interested and looking for
a job if you want to uh
to post on chippy.org please do
pretty straightforward
upcoming events again virtual project
night will be thursday november 18th uh
elbow swig is happening on december 2nd
that’s going to be uh advent of code
edition
and then the next main meeting will be
on thursday december 9th
if you need to reach out to chippy
you can find us on slack twitter or
chippy.org on the contact page
tonight’s schedule we have
five really great speakers lined up for
me to talk about their businesses and
how they use python to
to power their business
daniel keeler from jfrog joe nylon from
zorro
patrick fuller from new met industries
santiago santana from narrative science
and jordan bettis from nielsen
at the end of each each
talk the speakers have graciously agreed
to do a q a
please just type your questions into the
youtube live stream uh please be aware
between
what we’re posting and what you’re
seeing there is about a 10 second delay
in that we’ll try to eliminate the dead
air as much as possible
and then after
all the speeches are concluded we will
have an after party at chippy.town if
you’re interested in walking around in
virtual space and maybe meeting some of
the speakers and
your fellow chippy community members
and with that
i think we’re going to go into our first
speech
our first topic with daniel from jfrog
hi daniel welcome hi thank you thank you
aaron thank you very much
how are you guys um
so i’m daniel uh from jfrog automation
lead um i’m using uh python for the last
um
nine years um
here’s my email my github account feel
free to reach reach me out for questions
or anything
so um
let’s start by
why do we use python i mean when when we
need to choose a programming language uh
that will keep us as productive as
possible so definitely python is the
language that uh that comes to our mind
and uh there are
probably all the other reasons uh most
of you guys
know that
batteries included okay so the variety
of libraries
uh is definitely a big factor for using
python uh where we can focus more on
solving problems and writing even less
code but it’s structured well structured
and with an easy syntax
and
pretty much readable code
the learning curve so
probably anyone coming from any other
language or even learning the first
programming language
so python is a very welcoming
programming language definitely the
portability is definitely a very
important factor here
where you can run the code
moving from your laptop to containers to
vms to the cloud is
super easy and the community definitely
the community is one of the biggest
advantages for python because
you know when you have problems uh and
you you you have questions you need
answers as fast as possible and with the
python community uh you have it
okay so i want to give you guys a few
examples in jfrog where we do use python
uh
you know in our uh software uh
development life cycle okay
so let’s start by um
when we
need to provision and configure machines
okay
so we use fabric fabric is a library
that actually
allows us to run shell commands okay
everything using the ssh protocol and
but everything that you get back
from from every comment that we run
is a python object so it’s very very
easy to manipulate any any response and
do whatever you want
with with
all the objects that you get back okay
you don’t need to install any agent in
order to use this library
you just need really just need ssh
access to the machine and everything
happens um
and these pure pythons so probably you
guys uh you are aware of many different
tools that can do the same
the same thing
but i i think that
when you need something less opinionated
and go to for pure python i i i feel
that fabric uh
could be a good choice
okay so we use it mainly for uh creating
uh in
big uh setups uh it can be uh
normal absorb even high availability
setups with a very very big topology so
it handles it uh very well i i could say
um
another example uh where we use python
is for uh in our pipelines okay
so
python libraries uh they are very
helpful and
they’re very helpful why why while we
are building uh our software and also
deploying uh the software okay so all
over the the pipeline uh we can use
python of course uh for example using uh
cli like two like
arc parse is the the simplest one i
would say uh maybe because it’s already
built in but
um
creating just a cli tool like like you
see down here uh that you can get
parameters uh in the cli and handle any
logic from there it’s super easy uh it
can it makes also very easy to move from
one pipeline and from one
uh ci cd server to another so you keep
all your logic inside of
of python
of the code and then moving from one ci
cd to another is it should be very easy
because all the logic is already handled
there
uh
think about it you need to handle many
different uh
api calls
inside of your pipelines talking to
different services getting different
statuses
so yep
nothing better than i think nothing
better than
than python to do it okay using for
example requests or you just url leave
so we can handle this very easily there
if you still need to use shell commons
you can go always to os system or
subprocess p open uh
i personally like
more the the subprocess p open approach
where you have more um i would say
granularity
and on what you send
and what you get back okay to handle
everything it’s
for me at least it’s easier with the p
open approach
okay and uh think about it also you want
inside of your pipelines you you
executed a lot of different tasks at the
end you want to just to get a report uh
or different reports for each stage so
python is there also it’s already there
so
it’s super easy you’re using for example
uh jinja two uh
template engine you can you can um
create reports say very rich reports
dynamic reports for
everything that is happening inside of
the pipeline okay
another example i want to go through is
for
end-to-end tests and benchmarking
most common one uh
pi test for functional and api testing
so it’s being using all over the place
uh
we have we have been using also for the
last couple of years we have been using
locust okay where
we can it’s a framework load test
framework where we can simulate um
different users uh
going through through the endpoints that
um
you define the endpoints users go there
they
they go through all the endpoints we
have
all the information about the response
response
time and response codes failures we have
all these statistics already built
inside of the framework so it’s a good
way also to understand how the
application
behaves with many users
okay we create uh our own uh
python client a client as modules okay
mainly api wrappers
where
we can share between different teams uh
you know abstractions
where is it should be easier to to to
just
go there and and
use any part of the the public or
internal rest api
for any part of the platform
context managers is a goodie
for python and it’s it’s a
elegant an elegant way to to just create
objects and use them and and even
handle all the object creation and
teardown of the same objects
you can see down here i i added an
example where for example we create a
client okay
and you just create it once
as an object small object and then you
go there inside you give all the logic
that you need
again as modules that can be shared
between the whole company uh people from
different teams sharing different
clients
um
satellite services uh that’s how
i call it
so satellite services is you just in a
few minutes you want to bring up a new
service okay and you want to have a few
endpoints where you can populate that uh
and get data back
uh from this satellite services and
you know with the flask for example is a
super easy way
we use it a lot
okay
one more example is way for machine
learning and data filtering in jfrog
we deal with uh
a lot of data okay especially data
related to
uh components packages open source
packages and security vulnerabilities
okay so dealing with all this data
uh
when we deal with it so it’s very
important to
to have the right the right way to deal
with it and here we use actually uh
psychic learn
the machine learning uh
approach also uh modeling uh with spacey
okay
where we can classify candidates uh
for components and and finding the right
uh security vulnerabilities related to
the same components and bringing more uh
insights about
cves and descriptions of the same cvs
and bringing a lot of information that
is related to security and licenses for
the components
okay
um
yeah where do we store our python
modules so this is an important part of
the process because as i said we share
between different teams uh these modules
uh some of them come from outside
some of them
they are
created in-house and just shared between
uh the different teams in the company
so for external modules for example we
cache them in artifactory uh so it’s
easy and fast we have easy and fast
access
meaning that even if pi pi or
fire host python hosted
they are down you still have access to
these modules they are cached in-house
okay
and then we have also the internal
modules that we create and
we don’t need to put them out we just
keep them inside of the organization
uh it’s very easy to share between each
other
totally integrated with the cicd
pipelines of course
okay
and
one important aspect of dealing with
these modules and
internal modules is to scan
for uh security vulnerabilities if they
have some
uh and
what is the level of the the
vulnerability and the severity of the
vulnerability and also for compliance so
licenses which licenses
are actually part of the same packages
this way we we according to legal
aspects and we can understand if we
can use or not the same
uh modules
so this uh this part is it’s very
important aspect of of the the
life cycle okay the development life
cycle
and
yes i think uh
i would like to maybe get some questions
for you guys from you guys
i’m open for questions
yeah i i have one to for you daniel
you seem to use python pretty much end
to end in a lot of your dev ops
processes and for spinning up api
services has there any been any cases
within jfrog where you’ve found it to be
maybe not a good fit where you migrated
to another language
for specific use cases
or use python pretty heavily
so in jfrog actually
we we use uh different uh also different
programming languages we use uh
java very heavily we use golan also very
heavily uh but python as you see fits
for many use cases
we have also a new
new services being added to the to our
platform today that they are actually uh
written in python
but we have we have different actually
today we have different technologies and
programming languages for different
services running under the same uh
the same umbrella i mean the same
platform okay
uh
so yeah python it’s it’s it’s
impact with python we want to make sure
that the
at least in my team we want to make sure
that uh the r d teams they are
productive i mean
just solve problems you know but we also
use it uh as
part of our services inside of the
platform
okay
thank you that’s i mean it sounds really
like
just from end to end is
so nice to see that the language being
used so heavily i’m sure our community
really appreciates that that those
opportunities
exist
uh from starting from fabric and using
the pipelines command line tools you
know
rich rest apis you know
system os that you mentioned jinja for
templating for reports and
locus for load testing which is a new
one for me
um but
yeah a lot of a lot of stuff going on
there do you have a favorite python
package that
that you just you know really like be it
built in or third party that you know
stands out to you and either it’s like
execution or
how it abstracts away
work for you
uh i think uh fabric the first example
that i gave
fabric is we’re using it heavily
in our day by day especially for uh
provisioning and configuring machines
yeah this as a third-party third-party
module i would
i would say fabric but uh yeah we are
also we have also our clients that we
develop in-house
uh and they are very very useful to
actually
that’s great yeah i i got introduced to
fabric several years ago by a mentor
when i was going through the chippy
mentorship project and
uh yeah this is a great little package
for
controlling one or several
uh servers instances that are remote
really kind of helped me clean up my
deploy process when i was
a newbie and just making things kind of
repeatable from my laptop to aws
instance
uh we do have a couple comments here uh
we have spark c saying it’s really neat
to see how python is being used as so
many processes and different ways
throughout the process
um we didn’t get too many questions but
it’s nice to see that that comment uh
from the community
so
given that we don’t have any
other questions i think we’ll we’ll call
it on this talk and i’d like to thank
you again daniel for your time tonight
for excellent presentation and showing
really how python is used in your
business really from end to end through
devops and services and everything it’s
great talk thank you very much thank you
very much aaron thank you very much guys
have a good evening hope to see you at
the at the live stream but i at the
after party but i know you’re on quite a
different time zone and probably looking
to get some sleep
yeah yeah i’m in israel in here yeah we
i am in on 2 a.m
well thank you for again so much for
staying up and
and staying here and giving that talk
we’ve got a lot of things coming in on
on
the feed as well
that’s great aaron thank you thank you
guys thanks
and then
at this point in time i would like to
introduce joe from uh
zorro and
hi joe how’s it going i’d like to give a
great i will let you uh handle your
introduction and going to the talk thank
you very much great thanks aaron yeah
thanks uh thanks chippy for for having
me this is i’m i’m really excited i used
to go to chibi years ago all the time
and i never presented so it’s a great
opportunity for me um so i’m i’m joe
nieland i’m uh
a senior developer at uh zorro focusing
on python um i’ve been doing python my
so personally about eight years um and
then total development about
14 or so years
um
so um
at uh
daniel that was a great presentation
i’ve got to say uh there was a couple of
things that you guys used that i forgot
to add to my presentation so i might try
to shoehorn them in
um
but uh so let me just um
let me just get started so
um
so who is oral um zoro is a an online
distributor of products uh for mostly
business to business customers but uh we
do also sell to individuals uh we focus
on uh small businesses uh to find what
they need and grow to maintain their
businesses
uh today we got over eight million
products available i was like personally
when i started zorro three three years
ago we had about three
less than three million products so
we’ve we’ve seen some some outstanding
growth there
and we work with third party suppliers
uh to provide uh products to fulfill
orders for our customers
so um
because we’re growing so so so quickly
uh we need a technology stack that that
scales at an equal pace
uh so why python so
zora has been using python um longer
than i’ve been there since since its
inception about ten years ago
um
but python for for us it’s um
it really allows us our developers to uh
create new features uh quickly and uh
efficiently and i think that that’s
python’s greatest uh strength is that
uh you know it’s easy to read is easy to
write
um so that gives us a
competitive edge over other uh
e-commerce platforms
um
we have seen some some
great growth in our products but also in
our development team so now coordinating
50 developers
in a singularization
um
it requires a language that’s easy to
communicate in and so python is you know
even if you don’t know python you can
pretty much read python so it’s easy to
communicate among developers as well so
our front-end people can
review python code
so uh and most importantly last and most
important is that the python community
is robust and thriving and so that is
has been helping our our technology
group grow uh because um
you know there’s no there’s no shortage
of great
just excellent python devs in the
chicagoland area
zoro itself is a full remote at this
time i mean you know
but um we do have offices in chicago and
buffalo grove and so sometimes we go in
there
if you’re in the area but we’re hiring
for
remote positions anywhere um
so
um i’d like to talk about uh where pi
where where zoro was and where we are
where we are moving to so this is
what we call the monolith uh and this is
a a situation a lot of companies that
are seeing growth um
uh
get into um and it’s common because you
know you want to start you usually want
to start with a monolith when you’re
when you’re small and then you grow into
a more um
distributed system
so um
we have um and this is my presentation
is going to be mostly web focused but we
do use we do use python for our etl uh
systems as well
um so our main monolith is django um
django does all of the routing and and
it does all the the processing and then
it hands over uh context data to our
vue.js ssr and then view is and then
django sends it back to the customers
um so on that server we also have our
cms and then on separate servers we have
you know product and category data um
and then we have a uh third-party
providers for things like search
recommendations uh texts
uh shipping and many many others uh we
also have uh the the erp which is
separate so django is kind of like
packing this all in there
um so because it was a monolith uh we
could only deploy um so often um and we
were deploying once per sprint so every
two weeks uh we would deploy and it
would just be like this massive you know
amount of code um and that can be
difficult to hear
um
if there was an issue with a with with a
single feature it could hold up the
deployment for unrelated features
um so uh django itself was doing a lot
of the heavy lifting um and uh most of
our business processing was uh
like actually synchronous as part of the
the customer request
uh so what did we do we went to
microservices right everybody’s doing
that so um
over the last two years we’re trying to
split up the monolith and um
all of our devs most of our our back end
devs are
python focused so
uh we wanted to stay with python but
moving into microservices so we chose
flask as our um
as our our main microservice um
framework
so uh microservices uh they either uh
communicate through like rest uh https
calls or
um and message queueing systems so you
know there’s
some asynchronous communications some
synchronous communication
um we
uh we don’t use the django rest
framework but we do uh what we use is
some called uh connexion which takes a
swagger yaml
like open api
contract yaml and converts that into the
routes
um and so that that lets us uh do write
the actual microservice code business
logic code in pure python
and then we also have the freedom to
move to a different framework if we need
in the future which also adds some
agility we’ve got like a kind of a flask
wrapper around pure python
business code
um so this is just kind of a uh expanded
diagram uh this is extremely simplified
you know obviously for for this but i
just want to go over uh some of the
things that that have uh changed and uh
this is you know some of the things that
we have already so we still have django
we still have view
um
but it’s just doing a lot less of the
load um django uh connects to our uh uh
cms uh which is is now separate we’re
not using
uh cmsdb right there on the same server
so we we have a little abstraction there
and then we actually have our view ssr
um
that is doing all of the connection to
our flask microservices so now we’ve got
a catalog service for products and
categories we’ve got a search service
for a search provider we’ve got a
recommendation service for that
we’ve got a
cart service which then connects to our
erp as well as tax and shipping provider
which are also behind microservices
so
you can see that this is this has just
been uh become a much more uh
distributed system in it and it makes
our um
all of our lives a lot easier
so um
we build all those
microservices on uh docker containers
which flow through jenkins
and we use kubernetes in the cloud so
all of the scaling then is handled
through
kubernetes and we can focus more
on um
more on
the actual business code
rather than having to deal with you know
um
uh performance on our um
servers
so um
the the main benefit that we get out of
this is that we can deploy those
microservices ad hoc
so as the feature is developed it can go
right to production
um the model that still exists uh but
it’s you know significantly reduced and
we can deploy that now once a day so
going down from a deployment to every
two weeks to once a day and including um
being able to to do more focused
deployments um
uh
you know two three times a day uh per
service um
has definitely helped us there
uh so i just want to show you a little
bit of our code um
this this is an example on on the right
here of a um
what the open api or swagger yaml looks
like see we have um
we have our paths um i wish i could show
you more of this but we have our paths
um
defined there and uh you know whether
it’s post or get and then uh the actual
uh link to the the
python function that runs when you make
that call
and then we can also define um our
request and response bodies
um so that
we don’t have to we don’t have to
actually write any code to to validate
request responses it’s just that they’re
just all right here in the ammo um this
is um
basically
i clicked that this is on the on the
left here is is basically the extent of
of how we use flask
we just build an app and we import the
yml and that does all of the
um
all the routing
for us
we also have um flask handlers uh which
i don’t have included here but that’s uh
things like uh you know 500 error
handlers uh before request handlers have
to request handlers
um so what are some of the obstacles
that we ran into
when we move to the system
so
growing at the rate that we’re growing
um
both in like amount of data and amount
of developers