DevOps Tools for Java Developers – Devnexus 2021

[Applause]
[Music]
hello and welcome to
devops tools for java developers my name
is stephen chin
i run the developer relations team at
jfrog
long-time member of the java community
and very happy to be talking about a
topic which is near and dear to my heart
now i think
most of you last year when this pandemic
first broke out
first of all you probably wouldn’t think
that you’d be attending devnexus
virtually this year last year it was
awesome to be able to meet everyone in
person
in atlanta but also i don’t know how you
guys felt about it but i
i somehow pictured that it would be a
lot
cooler than it actually turned out so
this is how i figured that i would end
up during the pandemic
maybe um walking down the highway
with my trusty dog fighting off bandits
searching for food
but this isn’t actually how the pandemic
turned out for most of us and
what actually happened was all of us
technology geeks
fortunately were able to continue
working from home
kept adding more monitors fancier chairs
and keyboards and mice
in the case of me a lot more microphones
and
cameras and whatnot and um we’re
fortunate that we’re able to continue to
to work from home be productive
and kind of weather the storm and
i think that’s it’s great that
technology professional is able to do
this and
with that also comes some some um
horrible decisions as well
and given the extra time in bandwidth
um i went ahead and started to to write
a book again so one of the books which
i’m working on currently for riley is
devops tools for java developers it’s
kind of a
end-to-end view of everything which
you need to know as a java developer to
be a expert in
doing your build automation doing
continuous delivery turning into a
hyper-performing devops team
um this presentation you can find
online at the the link you see there
also
there’s a landing page there to enter a
raffle so
my employer jfrog is generously giving
out probably the coolest lego set in
quite a while
a baby baby yoda lego set
um so just scan the code there to go to
the website or type in the url below and
that also has the slides and all the
content for this talk so no need to take
notes
while we’re going through the content
and the topics we’re going to cover
today are
how to handle source code so how to
structure your
how to get a single source of truth
continuous integration
building and testing on on every
check-in
package management so that you can
manage your binaries have a full build
materials
and history for traceability doing
security scanning
of dependencies and containers being
pushed to production
and finally continuous delivery for
distributed automated deployment to the
edge
and beyond and to start out let’s let’s
dig in
on on i think everybody a topic which
every developer
kind of begins with which is how do you
how do you get the right source control
system
and team collaboration to support your
project needs
and i think if you look at the the
evolution of version control systems
we really started with some very
primitive tools
which went all the way back to
simple things like emailing files around
or even before email sharing files on
shared file control systems
um we we eventually came up with with
proper version control systems the first
generation of which
we’re locking version control systems so
if you’re if you’re old enough to
remember this
don’t raise your hand too high or
everyone will know exactly how old you
are but
systems like sccs and rcs
allowed you to do this on unix systems
and this was kind of the first
attempt at doing collaboration without
stepping in each other now if you if
you’re only locking at a file level
the the challenge with that especially
when you have a large project is
dependencies
aren’t limited to just files so you
might be changing an api and you have to
change all the code which is related to
that
and therefore what you’d like to do is
you’d like to be able to
make changes across the system without
locking
every single file in your project and
this is where the second generation of
systems came in
with open source projects like cvs and
subversion
which allows you to go ahead make
changes to a whole bunch of files
not worry about actually locking the
files or
preventing other people from making
changes to the same files
and dealing with the conflicts later as
merge
merge resolutions so this this is
perhaps the the greatest
advance in version control systems but
also the greatest bane to all of us
because now we we spend an enormous
amount of time
um either resolving merge conflicts or
um
figuring out how to avoid producing
emerge conflicts with
a better team organization and the kind
of the the latest generation of virgin
control systems
are distributed version control systems
and the concept here is
why you need a centralized version
control system now everybody can have a
full copy of the version control system
locally
you can work entirely offline you can
even
synchronize changelists directly with
people without going through a central
server
although in practice most folks have a
uh
kind of a single source of truth for
their version control which is stored in
a
in a central um getter mercurial repo
and the two most popular version control
systems for for this
are our git and mercurial although
um over time git has
has clearly won out and is the verge
across system of choice so if you are
still
using cvs
my apology is time to time to upgrade
and stop having your repository
corrupted every
every few days if you made the move to
subversion that was a huge upgrade over
cvs but but again
i think this isn’t really the path
forward and
um in the past um
10 years or so everybody has basically
migrated over to
git and um distributed version control
systems
and if you look at the way distributed
version control systems work basically
the the standard
um practice is you have a local
repository which is a full repository so
it has everything
um locally it also has a a working copy
of the code
and the working copy of the code is what
you’re modifying you then check into a
local copy so
um in the concurrent or the centralized
version crawl system
that repository will be stored in a
remote server and every time you
committed or merged you’d have to
pull back changes across the network
this allows you to do everything locally
until you’re ready to push or pull code
from a remote repository and the the
standard way of doing this is having a
server which holds the remote repository
and then
um pushing and pulling changes it to or
making poll requests and having somebody
somebody manage and integrate all those
changes so this is this is really the
way forward and what everyone’s doing
um one side note on this when we when we
get to binaries and package management
is that this also drove another
um revolution and vcs which was don’t
store your binaries don’t store big
blobs of data
dependencies and version control because
then you end up with unwieldy large
local repositories so when we get to
package management this is one of the
things which
which drove that revolution as well
and you can see from the um stack
overflow survey this year that
um there’s a bunch of different tools
that you can use to manage
your git repos remotely the most popular
of which
is github so this is used for most open
source projects as a collaboration tool
as a way of
managing poll requests as a way of you
know getting more folks involved in your
project
and git lab is another commercial vcs
system
um you also have um microsoft teams for
folks who are
on net platforms
and a whole bunch of of different
solutions for team collaboration like
slack and jira
and confluence which are all things
which we use on a daily basis but
you can see that version control systems
and tools have definitely pushed the top
of
developer tools which we all care about
and a common pattern for how you manage
all of your um branches and your merge
strategy when you’re using git
is called git flow so this is kind of
the great granddaddy of all the
different
um strategies for how you can organize
your repositories
it is both the most complete but also
probably unnecessary overkill for most
projects
yet there’s a lot of branches to manage
with a develop branch a release branch a
hotfix branch so you have
you have way too many branches which you
have to matter manage simultaneously
and it’s it’s really most suitable for
software which is released with
versions where you have to do patches
against previous versions and you have
to maintain
patches against multiple versions of a
code base simultaneously
if you’re doing any sort of web-based or
server-side software
you really don’t need to worry about
this because there’s always kind of one
latest version
which you’re updating you’re maintaining
and you’re patching and you don’t quote
have a
an old release which you have to patch
separately so it makes the
management of branches and patches much
easier to do but the
the basics of git flow are kind of
underlie the rest of the
um change management strategies which
i’m going to talk about
because this is the concept of folks
making changes in feature branches
then pushing them back into a develop
branch or making a pull request and have
somebody pulling into a development
branch and then using that development
branch as kind of your your main
code line which is supposed to always
build
and be a stable code line for folks to
pull from and to continue to make
changes and this feature branch style
development
really works well for teams and it
avoids a lot of the merge conflicts and
issues which
folks have had with um with managing
version control systems
so the key properties of git flow are
you have a development branch
a branch per feature you’re not using
fancy features like fast forward merges
or rebasing
and you typically keep a separate
release branch
um github came up with a much simpler
model which
is calling github flow which gets rid of
the need of having release branches
so this makes it much easier since
you’re managing fewer branches
it still doesn’t use fast forward merges
and rebasing so it it
ends up with a history which is a little
bit more complicated to look back at if
you want to
look at changes but it also avoids the
the possibility of rewriting history
causing issues with conflicts on the
server
git lab flow is an evolution of github
flow
it’s similar and then it doesn’t use
release branches has
one branch per feature but embraces
things like rebasing history so if you
rebase your history
before you make a check-in what you can
do is you can keep your history and get
to be kind of a single linear flow which
is sometimes easier to read
and also the um the merge strategy
allows for things like fast forward
merges which
um again reduces the complexity of
branching now
that said you have to be careful when
you’re doing re-bracing and merging
because if you
push any changes before you actually
make
we try to rebase then you can cause
issues with folks who are relying upon
history which you’ve then rewritten and
that that causes all sorts of havoc in
the backend git system
so i i hope that you know by now you
guys know enough are already using git
have choosen a branching strategy have
your team workflow worked out
and kind of the next big step for teams
is figuring out how to do continuous
integration running all of your tests
and making sure that you
are having builds and test and
qa in multiple stages and typically the
way this would look at is
you check your code to source control
and every check in you have some sort of
hook setup it
there’s a ci server set up which will
when it sees that there’s a change we’ll
pull a version of the code um we’ll run
it and we’ll do
and we’ll push into an integration
system where you might run integration
tests
if the quality controls and all the
tests pass then you might move it over
to a system testing
environments again you’d have some
quality thresholds acceptance tests or
different things and then it would be
pushed into a staging environment
and then finally if you’re able to
verify the features meet the business
requirements and
have folks test and make sure that it’s
exactly what you want to push into
production
then push it into a production
environment and this this kind of
incremental set of quality gates is
extremely important because what it
prevents
is things like what happens
to our friends at night capital so night
capital is what kind of one of the
classic fails
in the software industry where
they were they’re a company which
specialized in building systems for
trading applications and um
when you’re in the financial industry
it’s extremely important that your
systems
um are stable and reliable and you’re
not introducing any regressions
into production on the deployment of a
new system when they were introducing a
new
a new trading option knight capital ran
into an issue where the system was
executing trades
by itself erroneously causing
10 million trades per minute
to to be executed and at the speed which
financial transactions happen and the
the cost of financial transactions
over the course of about 30 minutes
which it took them to to diagnose and
figure out what happened
this was 440 million dollars lost
and there were 154 different stocks
involved which were
which were traded and kind of caused
complete havoc in the market as a result
and when you get down to the root cause
of what actually happened with their
system
um this turned out to be an issue with
how they deployed code
and they deployed the code um to
create a cluster of eight servers um
when they went from staging to
production
they didn’t have an automated system for
this it was a manual system for pushing
changes to production
and seven of the eight systems were
updated successfully
one of the eight systems wasn’t updated
successfully
and there was a api change in the system
which wasn’t very obvious it was an old
api that was being reused which again is
another anti-pattern
and what happened is when any of the
exchange at the trades hit the one
server which was not updated
it would use some old apis and that
would execute erroneous trades
their first attempt to fix this was to
roll back the changes
and since all the clients weren’t
weren’t rolled back it was just the
servers which were rolled back
what happens then is that all eight
servers were
executing erroneous trades and then
they lost even more money so um
eventually they figured it out and they
staved this off but it basically caused
the company
a huge financial loss and eventually
they were acquired into another company
when
all the dust settled and all the damages
were collected
so um going to the next topic we’re
going to skip over package management
we’re going to come back to that
but i think security scanning is an area
which is extremely important
and this is again another thing which
can mean the difference between a
successful business
and one which has extreme
challenges and problems and in the
software world if you
wanted to do a parallel of this security
vulnerabilities have basically become
the new oil spills which can cause
dramatic and catastrophic damage
if you’re not careful about mitigating
them
so the the analogy here would be this is
this is a classic oil spill problem so
taylor energy back in
2004 was hit by hurricane ivan and is
um was leaking 300 to 700 hours per day
and due to the damage of the hurricane
it cost 435 million in cleanup
um and they they finally figured out a
solution to this just recently
and if you look at open source software
we all
run open source software open source
software’s in 99
of enterprise projects and 75
of those projects contain at least one
vulnerability
with about 82 vulnerabilities per code
base average
so if you think you’re
excluded from having potential security
vulnerabilities in production
as long as you’re using open source
software you’re definitely not
and even the software you write yourself
is um susceptible to
security vulnerabilities which can cause
damage almost equal to a
um to an oil spill for example what
happened to equifax
and their data breach which is kind of
the the classic fail in the software
industry
where there was a vulnerability in
apache struts
cv 2017-5638
and this allowed attackers to get
in to equifax’s databases via an
unsecured server
once they got into the internal systems
they were able to tunnel into
other systems and actually get access to
consumer data
they had access to systems from march
through july of 2017
and this cost equifax 1.4 billion
in cleanup costs and 1.38 billion in
consumer claims affecting 143 million
consumers so um this is what happens if
you don’t pay attention to security
issues
and um eventually this was attributed to
the chinese backed military so february
of this of last year
they finally indicted the chinese
military for for being the ones behind
the attack
and actually this is probably a good
thing because so far none of the
consumer data
actually was exploited from equifax it
was just consumer claims which are
worried about
data being leaked so getting back to our
oil spill problem if you’re not paying
attention
to security vulnerabilities you could be
the next equifax or the next oil spill
unfortunately but in both cases we we
now have technologies which can help us
to solve these problems
um for the echo for the taylor energy
oil spill they built an oil containment
system
which they retrofit which collects the
oil and pumps it up to ships that now
collect it and resell the oil coming
from that so they brought the um
the flow rate down to about um
a thousand gallons per day or and um
they use a 200 ton steel box on top of
this system to contain the oil
um three to five gallons per day once
they’ve contained the oil
and for security vulnerabilities you can
now use
security scanning tools like jfrog x-ray
which allows you to scan
binaries you can use other systems
upstream
to scan your code base and there’s new
systems coming out which do run time
vulnerability scanning
so the next phase in your contain in
your
devops pipeline is continuous delivery
doing distributed automated deployment
to the edge
and beyond and the reason why this
matters so much is because we’re all
using devices
like smart cars smart tvs
cell phones and even smart fridges which
need a continuous supply
of software updates to allow you to use
your devices and
enjoy the latest features so do you need
this both for new feature delivery and
also for patching vulnerabilities and
usually as consumer when you’re looking
at do i want to update
to the latest version of software the
thought process is
you know here’s a new update do you want
the update
so hopefully the answer is yes there’s
actually some cool features and you that
you wanted it
is it risky um well
if it’s not risky you’d probably update
but it might be risky and you might not
trust it
so then you’d have to verify it and if
you have to verify the software release
this often is a costly verification
before you make the update which can
cause consumers or software developers
to not update
to new libraries and a classic example
of this
is um the new jdk release model
starting in java 9 with modularity
which was announced at devoxx belgium
2017. so
this is kind of what the new reality for
all java developers where we get six
month release cycles
we have a continual stream of releases
coming out with lts releases
and um incremental releases
and there was also a big shift with java
9
where modularity was not entirely
backwards compatible and broke a bunch
of things
in systems and libraries which we all
depend upon
subsequent releases have been a lot less
dramatic in terms of the change but
also the system for deprecating features
has also been changed where now
deprecated features can be released in
any of these releases as well
so if you if you look at this from an
adoption standpoint
um for for folks in the chat um if you
don’t mind
let us know what java release your
current running
um and i think you know basically for
for most software developers this kind
of falls into
three categories either you’re you’re
still stuck on java 8 for some reason
because you can’t upgrade or or you
choose not to upgrade
um you made it past the the java 9
threshold then you’re on a nice lts
release
like java 11 or
maybe you’re on the latest release of
java so you’re running java 14.
so earlier this year
there was a great study done by sneak on
the java community and what they did is
they pulled also
um what all the software developers were
using
and their data showed that 64 of
um folks we’re still on java 8.
11 of folks made it or 25 of folks
made it to java 11 and 4
of folks were on the latest release of
java which at the time was java 12 last
year when the study ran
and if you if you kind of look at it but
go back to our flow of
why this happens so as a
as a java developer this is kind of the
um the flow which you probably ended up
going through
so the first question is a java update’s
available do you want it
hopefully you want it some folks
actually don’t want modularity
but assuming you want it the next
question is it risky
so um in the case of modularity we know
there’s backwards compatibility issues
so yes
it is risky do we trust it
probably not so then you have to verify
it and if you’re going ahead and trying
to verify something
like a major jdk upgrade we all know
that this is quite a large undertaking
and i i think this also is the reason
why a lot of folks have chosen to skip
or defer updates because either they
they don’t really need features which
came out in java 9 and beyond yet or
they haven’t been pushed to
to adopt and um the verification process
to to update to java 9 plus
also is quite um uh quite a lot of work
and um when you when you don’t stay
um don’t have a good process for doing
continuous delivery
what can happen to you is what happened
to another company which we all rely
upon for updates
which is cloudflare so you want to be
releasing continuously
both for features and also because you
want to avoid security risks
and having an automated system and
having a way to do this which is
repeatable is highly important
so in cloudflare’s case they had a huge
outage which was caused by an update to
a fairly innocuous regular expression
and for
all the the experts in the audience who
are pros at regular expressions
what do you see that’s potentially wrong
in this regular expression if you’re
trying to run it on a large set of data
okay so this this like like anything
else this is kind of one of those
those tricky problems and the
interesting part of the regular
expression is highlighted here in red
um so the dot star uh question called
dot star equals
dot star um as you guys know there’s
capture group
the question the parentheses question
called as a capture group so you can
safely ignore that and what it
simplifies down to is a dot star dot
star equals
dot star and what you’re doing is you’re
doing a double wild card
capture and what happens in a lot of
regular expression
engines is when you have a double wild
card on a very very long search string
which happens
when you have large data sets going
through then
this essentially becomes an
exponentially
costly query to run and this caused the
entire cloudflare system to um
to go down as a result of that so if you
caught and figured this out
give yourself a big pat on the back for
being a
regular expression genius for the rest
of us who didn’t notice it
you’re just a mere mortal and you would
make the same fix if you didn’t have the
right automation
and processes to do continuous delivery
so the last topic i’m going to cover is
package management and
i think if you look at this this
spectrum what ties everything together
source control systems continuous
integration security scanning contiguous
delivery is you
you on all the systems you’re either
building or consuming
um a binary whether it’s a docker image
or
whether it’s a a module um
like a java module a node module or
something else
and if you want to be successful with
having a centralized repository like
this it really has to be
support all the different formats and
possibilities
kind of be the the switzerland of
different continuous integration systems
different
binary formats um and this is exactly
what um what we do at jfrog with
artifactory which
supports just about every tool in the
industry for build tools dependency
management
repo repository ci servers
provisioning tools we we integrate with
just about everything
and serve as a central repository and
understand
natively how to pull apart jars how to
pull up our docker images how to serve
as a container registry
so um this is kind of the the crux of of
your overall system
so this was a quick lightning talk
through the world of devops from the the
eyes of a java developer
i hope you learned a little bit more and
um the the book which we’re working on
devops tools for java developers will be
released later this year but we already
have a preview available if you’re
interested in that
again you can enter to win a baby baby
yoda lego set which is awesome
if you follow the link and i’m more than
happy to answer questions for folks
who have them so thanks very much for
coming to my talk
at devnexus and i hope you enjoy the
rest of the conference and then i’ll get
be able to see you in person
next year when we’re all back to um
normality so thank you
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
you

Try Redis Free
Ad
redis.com

3:18:01
NOW PLAYING