What is “Supply Chain” attack? and How to Prevent it in DevOps Pipeline @ DevOps Stage 2022

hello everyone and welcome to my session today we will talk about a fly chain
attack and how to prevent it on the hcicd process we’ll talk about different types of supply and chain attacks but
before we uh going to dive deeper to the technical Parts let me just introduce
myself my name is Vitaly Davidoff I’m an optic lead at trade fork used to be
developer for many years moved to security field about seven years for seven eight
years and now I’m responsible for the old cross products it’s a trade for
platform so what is going to be our agenda today uh as I said we will talk about Supply
and chain Security Programs so uh what type of attacks we have uh we will see
how malicious packages might be infected by uh attackers or by malicious users
and how to detect those packages in fact packages and how can we use best
practices to avoid supply chain attacks and we also have time for questions and
so let’s get started and first of all supply chain so what is the idea behind
supply chain first of all as you know uh we are using
open software packages everywhere uh he’s going to be the most part of our
software or of our applications as you can see here has actually once I have
password and build my software I have more than 75 maybe 80 percent of the
runtime code appears from the first third-party open source packages and not
from my first code and that’s something that good because
it’s it’s actually a very good Trend you can see here it’s not so uh up-to-date
craft but you can see the trend that uh we are using a lot of Open Source uh new
packages every year on the different programming languages difficult different
um uh build managers uh like on the Java maybe and Pam
et cetera different languages um also the trend of the download for open
source libraries as you can see here again it’s from 2019 but the trend is
the same at this year and it’s going to be the same for the next year because we
are downloading open source we are using open source packages uh components uh
like the as you can see the examples of the npm and Java and the billions of billions of downloads as you can see it
started around 2015-16 because as of the at that time we introduced the ICT process and once
we have automation for our builds for our continuous integration
we suddenly started to download a software open open source components in
branches and millions um so what the uh so why it’s good actually
the first of all it’s uh very simple way to do something from the programming
point of view from the developers okay uh just a example you can see very very
short 10 lines source code and just to open listener for request
response and and that’s it okay on the specific word but just think about it
that I need actually to program the code all the functionality for open ports for
uh uh touch and actually you deal these
requests and responses on the low level but I don’t need I what I need from the
uh from the my side as a developer I just need to include okay our part of
the required Express package express is a very very popular open source
package um and that’s it okay so I can 10 lines
of code and I have a HTTP listener uh for requests and response on the
specific part so it’s very clear it’s very short it’s take me uh maybe can take me five
minutes okay 10 minutes to do so just to understand what is expressed and the
review documentation and see what options I have and that’s it okay it’s very clear it’s a short code
great okay so uh I don’t need a create a
network manipulation quality gaining again or different applications so I can just use express and that’s it so create
is the best practice okay but what we can see on the runtime bottom line when we’re running okay or
we have uh our software released we will see the huge number of code lines okay
so you can see here so actually why because Express itself uses
third-party components and those components in other companies so it looks like that
so we once we include uh or uh bring our
Express open source as the my first first dependency Express will bring
other dependencies it does dependence or other dependencies its own so they
actually as you can see um on the on the runtime I will have a very
very big the huge line a number of Code online Productions okay so uh first of
all just think about that I don’t need to code all of it okay so I have it uh
for free and I just need to use one line of code in order to bring all of these
uh these graph in my code so there’s the actually uh the idea behind the open
source components and the usage of Open Source company so the supply engine is actually how can I bring uh Express for
example to my code and then how can I Supply my my software to someone to to
our to my customers okay and this process we call it supply chain
okay so as I said at some points uh regarding supply chain in general okay
so as I said modern software development and application integrated party code
um it’s very clean it’s a good it’s a good practice and I I saw my delivery
some time to Market it’s very short because I’m using open source third party packages I don’t need to pay and I
just need to to verify the license so that I can actually produce or include
the host third parties packages to be part of my applications if any
a so but we have some points uh I’m just
wanted to uh to emphasize so applications trust third parties to supply secure code
okay so this point is um
might provide me the the different feelings because
application shouldn’t stress the third party to supply secure code it’s something that I cannot agree
uh it’s it’s true for our for our days but just you know think about it uh
everyone can add any type of code any uh any anything
okay to npm for example registry okay so we trust those contributors but
how can we should contributor uh will be the uh not will be the malicious one
okay or I don’t want to open some back doors or do something uh something wrong
um okay so it’s very very uh problematic Point trust third party supply secure
code because it might contains vulnerabilities or it might also contain
malicious code so uh and why attackers want to do so or
malicious user attackers okay so I will use attackers so first of all it’s a high spread
attack why because I don’t need to deal these application and all the layers of
defense for specific application specific environment
um what I need to do I need somehow okay we will talk about it you need somehow
to infect our supply and chain process and provide malicious package and once
someone or a user users developers devops uh Engineers will
search and bring my malicious package this package will be in uh in a lot of
on a lot of customers okay and as you can maybe you remember uh sorry
uh maybe I remember the uh solar wind attack where we actually won uh one
malicious component and lead to a lot of bridges uh in the different in the
thousands of customers just because one package so I have work once as an attacker to
create my malicious package and then I will try to spread it to uh as much as possible
a low effort yes again so I don’t need to bridge all the security uh security
controls on the systems and take control other services
Etc okay I’m just sitting and wait until my
code will be on the runtime and once it’s around time for example I will receive certification
or I will receive the request for open uh open repairs proxy and that’s it
sorry I’m just need to wait actually a lot of technical skills required is
the um it’s correct because I actually need only to be uh to have the coding skills
and understand how can what can I do so what these options how to leak data or
how to work with files but I don’t need to understand how to bypass Network
protections how to play Fast browser protection and web server protection API
protection so I don’t need it because once my code is already inside so I
don’t need to to bridge all the uh all the shoots yeah and the trust in the relationships a
relationship between parties can be abused so it’s as I said so we trust
those contributors and that’s how uh I am actually can
um go to the code okay go to the packages
or for example I can do something in order to upload my new malicious version
to existing security project okay so and once we have trust between parties uh
customers or users developers Engineers or devops Engineers who will use it
uh just the short overview of the from the zero tune um
it’s how customers have companies pay for uh different Bridges related to the
supply chain attack like uh it might be a remote code execution by some other
attacks on the npm on the uh some others and Justice so you can see
it’s a huge number uh just now just to be uh to be sure
that uh pipeline code or pipeline attack
is uh is not there I will pay a lot of money okay just just a cloud
um okay so damage okay so what damage uh I think uh it might be everything okay
from stealing the access key and just completely destroy database and open
back doors like uh reverse shells so attacker can access my uh access victims
um machine or clusters it also might be something
less less damaged but it’s still the problem where the attacker will use resources
victim resources for crypto Miner uh so as I said it might be different
different options for different attacks and all of them part of the supply chain
problems supply chain attacks so now uh let’s talk about threats or uh
what can we uh what kind of the attacks we have on our supply engine process uh
I will divide it by the two parts one is the known and one unknown part so known
Parts is then we have we used we’re using packages these
non-vulnerabilities what doesn’t mean is that uh research teams from security security different
security tools software compositional analysis tools they will investigate uh
problems or vulnerabilities on the existing packages and they will
add knowledge of this vulnerability to the public
database public vulnerability database this issue will receive something like
CV ID so the common vulnerability enumeration ID here’s just an example uh how it looks
like so it’s a cvid is the year the number of the issue the package name or
a product name versions description of the product and the severity security so for example
if we are talking about problem which can lead to remote code execution or
account takeover or some other huge damage on the system it will be higher
critical severity but in any case I have um I have CV number of cvid and I have
the all the description all the parameters so it’s starting to ignore and once I
can scan my third party uh component and see the list of cve so what I’m doing uh
I’m scared we will talk about the tools or vendors and we can use a software
composition analysis tool we will scan our package and we will see the list of
the uh of the vulnerabilities so we’ll request a nvd database and receive CV
CVS list for this specific package for this specific version and then I can take an action like okay you can stop my
build or I can think about maybe to change a package version
Etc okay so at least something no okay I call it TV but we also have the second
part the second part is the malicious component results Series so it’s something the zero days uh we call it
zero day so it’s unknown you don’t have cvid on the database and and maybe uh
just introduce the malicious code as part of the existing package without any CVS it’s something new
and we will talk about it
and that’s the real examples uh I think I don’t want uh I want to spend a lot of
time here as you’re aware of the Spring Trail it was the remote code execution on very very popular uh Java Frameworks
spring here sdcv so it was researched and called the CV and it was the
critical I think and I remember well um so it’s again so once I’m using
spring and vulnerable version of the spring once I’m scanning my spring
packages this software composition analysis too I will receive this CV from
the database and can say ah okay so maybe I’m wrong I need to change the spring version to the um to the not not
affected Etc solar winds again as we discussed so the solarwinds is what the huge impact
once the um and once the attacker used to publish
malicious package and was infected a lot of customers it was the very very big uh
big noise from this uh from this issue you can read it about it and actually
something are not on malicious component component introduced on the npm uh on
Azure against the large scale it’s going to be always the large scale because as I said I need only upload malicious
codes malicious package once and and all the users will will touch the whole
thing okay so let’s talk about non-malicious
packages a once we have CV IDs once we have
databases no CVS we need to scale our software or third-party packages in
order to see if we have those CVS as part of this this version we are using
so where we are scanning and it might be part of the ID integration for example
for IntelliJ for Microsoft Microsoft tools Etc so I’m scanning it during my
development phase and I see all ecbs I have I just bring from the
remote repository and see if we have vulnerabilities so I can change it immediately it’s a
it’s the best way to to catch uh known CVS in the third party code
during the development process we also call it pre-commit scan uh the
next uh is the build phase in most cases we want to scan our code and our
artifacts we are using what we created as part of the build before we merge it
to master and I I would say that is the first game
first game because in this point I can stop my CI process it say okay so I have
vulnerabilities I have critical CV on the Apache common package for example it’s very popular for the 12 developers
and once they have this critical issue uh I want to stop my process and fix it
before it’s going to the master is the first key it’s the first and very
very important gate and the last one is the deploy phase at least yes at least
we need scan it before the deployment part because we might introduce more
packages as the final release or final build and again we’re going to scan up
the software composition analysis tool and see if we have new CPS or new introduced CPS for uh for our packages
for our containers and again we can take an action here and say okay so we will stop these process
at this moment because we have critical hybrid vulnerability CVS okay it’s based
on the company policies so um if you have policy uh which uh restricts access
or restrict usage components this critical and high issues on the production you can just stop this
process in this point so it’s the uh where we are scanning in general okay behind it
so what is software composition analysis it’s a solution okay or it might be in
the uh [Music] Enterprise solution it might be free solution it doesn’t matter for now but
it’s actually scans open source components for non-released on all CVS
it’s also a scans for the license issues and as we saw we should integrate it in
different place in our cicd process once
we do pre-build or pre-compile process and the build process when they deploy
point um so it should be integrated as part of
our CI CD and vendors you can choose your best or
your your favorite and it’s a lot of uh it’s actually the open source it’s all
about speakers Checker and it’s a free uh for you to to scale your software
your uh your third-party components it’s also the white Source the jforg X-ray
the sneak and others okay so you can just see based on your your needs
um and you are you know your Integrations for example if you have artifactory it’s
obviously that you will use uh x-ray but if not so you can maybe maybe use white
source and such so I will choose any anything you want
uh here’s just an example and so actually uh we have the package and we
have the version and we have a severity critical on it we have description of
the issue uh and we also we can see it but we also have the mitigation plan so
just to now uh fixed version for this package for Echo Power package
yeah so what can we do it’s a mitigation plan so what we just need to upgrade to the detergent uh to the next major or
maybe fix it doesn’t exist yet okay so maybe vendor uh didn’t provide the effects
so all the information will be on the uh on the vendors report for the known CV
so now let’s talk about unknown part and unknown part is the infected packages uh
where the attackers can add some malicious code to the existing or to to
generate the uh the new packages and try
to trick our devops Engineers to take so different types or different packages
uh issues like type of squirting masquerading
trial we will skip it today dependency confusion we’ll talk about defensive
confusion or name squatting and hijacking so let’s get started type of squaring as
from the name as a malicious user I will do something very very uh you know
classical like okay so I will create the new uh registry with new domain but this
name will be very very similar to the existing one the good one okay like Google so the classical classic example
is the goggle.com instead of google.com uh okay and then maybe
it will be the typo on the uh on the build manager or package manager site
and it will use my my register instead of the original one also I will push it uh as part of the uh
Twitter uh Twitter messages and maybe uh some other
other part of the on on the support side uh where I can
answer the question and uh add my link to my project instead of
the Google okay so I will push it during the community okay and I will see if
someone will will use it um
so it’s uh I think it’s very very uh straightforward uh just an example of
the using the uh instead of very very popular MP lot lab okay uh we saw that
empty lotlib was used okay and used actually we saw usages
uh okay so this package is not a remote code execution or a backdoors but it it
used for a crypto mining it’s also the problem it will be a huge problem a
problem and what can we do we can just need to be sure that we are using the correct
name correct project name uh we also can a lot like like many many
personal companies they just can buy the all uh type or
um type on names okay of their primary
domain like Google and now for the all the Google and googly and some error
some other domain and one someone try to access the
um by mistake and the google.com it will automatically redirect you to the
original to the to the right one so there’s the option how can we provide the protection
against the type of scoring but in this case as I said a packer will create
their own project okay this malicious and and provide the malicious package
so uh the next type is the masquerading so the masquerading is very similar uh
to typo but also the um the most the more dangerous why because I am to
create this similar project again I I should change the name for
sure because I can’t create the same the same name but I will copy all the metadata and all the
um uh all the links like a link to the projectory ETC
and it’s very hard to adjust to to look at these projects and
say it’s original or not uh like see it in example okay here
um I I will not ask you to to find to find the differences but I I will point
of those differences but in general it’s very very similar yes so as you can see the same the same metadata the same
method are the same um a repository a repository path okay and
the difference here is that the name okay the name of the name of different
it’s it’s not the typo it’s some kind of the prefix or postix added to the
original names in general like Market GS instead of market and if Market is a
very very popular package and you can see the number of downloads here
but let’s see number of downloads on the malicious one then okay again it’s the it’s still
someone downloaded someone used it um so that’s the also also the problem
there are just copy paste uh or just Fork the existing one and create the
money and add the malicious coordinate in general this uh this malicious
package will do the same as the original one so if I will run my code it it will
be okay okay it will be runable everything okay but uh the the attacker can add just in malicious part inside
the original code and do something in addition to the uh to the original code for example open reversion
okay so there’s very it’s not so easy to recognize uh masquerading because it
might be um our usage of Marg JS or marked not
you know all the devops engineers and developers can actually understand what
is the correct one uh so we need some help some we need to be
distinguished distinct between two okay and say Okay so this one is the correct one okay and not this one it’s not so
easy uh hijacking uh it’s a very very hard to
um to abuse or to create Contracting part but it’s very very dangerous so
what the problem here the problem here that uh I’m as an attacker the malicious
user I can’t uh I can take other existing projects okay uh I I will take
the token or somehow to bypass the username password uh whatever but I will
take control uh on the existing project as the original user as an administrator
for example and then I can change or add the new version of the existing comment
just think about it that if we are using Apache common for example we have Apache common and I somehow can log in as
administrator the attacking user can bypass authentication method and login is an administrator to the Apache
project and add new version of the Apache common package game over okay it’s very very hard to identify
this attack and so it’s um it’s the big problem
it’s it’s the it’s the hard to do for attacker but it’s uh no it’s very
it’s a huge import impact and unfortunately we saw
attacks like hijack with the account take solver
um on the npm for example and most popular way actually to take the
account is the leverage expired in maintainers on the project so by just to
recreate the emails so the time I will use expired emails as an attacker I will
try to register the same email for uh as
used by the old expired maintainers and I will try to recover password so for
this for this email address and then I can actually taken to take an account
and add something malicious so please be
aware and just remove expired maintainers or expired emails from your
maintainers list it’s a best practice so now we get to depends the confusion
it’s the most popular attack or related
to supply chain so the idea here is that
um build of package managers will try to find the highest version on the remote
report for this for the specific packages on the remote Repository
um and if as an attacker I can now contribute the the package is the
highest version existing package with higher version um build manager will take it
automatically and bring it to be part of the software so what I I need to do I I
need to contribute so I need somehow to upload the new version
of the of the package like you see here’s an example and I I should set the
highest version here the High version of the existing one in order to be sure that I’m the last one I would say that
the highest one as you can see here as an example I can set the version ID
something like that okay in the If the previous version uh was 2.5 so now it’s
the six this is a huge number for sure it will be the the highest version for
this package at least for in our next day or two so um
and it’s it’s not so easy to recognize this type of attacks and fancy confusion
because uh as you know the uh it’s housed the uh the package manager build
manager uh reports story manager works or they they will look they will search
for the new version of this component on the repositories and the one I just need somehow to
upload the this highest version there so it’s very very popular
um is just the uh before we continue I
wanted to provide some overview of the best practices or some tools you can use
as part of your process you say is the ICD process so in general what we need
to do we need uh several parts so we we first need to be
sure that we don’t have typo type of supporting okay so it’s a different packages for different uh different
package managers or build managers exist exist in order to to verify that we are
not using typo um we also have the uh expensive confusion
Checker I will explain it later so what what can we do I guess the defensive confusion how can we verify that we are
not using the wrong package version okay so if we see something
abnormal like 9999 on the versions and something the uh the jump came from the
version 2 to version 10 for example so it’s not it’s not normal and a normal
version versioning for this package we can say so that something wrong here
let’s stop it okay let’s at least alert on a domain check it’s again it’s uh
against the typo and the masquerading
um and also the uh unusual activities uh on the
um this statistical uh your unusual activities on the domains okay it’s like
repositories against it’s like a number of downloads um names okay then the new new domain
this very similar name Etc so it’s a lot of tools exist that you can use in order
to protect yourself against the basics at least okay so now let’s um
let’s back to the dependency confusion and as I said the big problem here is
that the um artifact managers or repository managers
uh will try to find the latest version on the remote repositories uh
in general when we are using repository managers tools like like gitlab
artifactory Etc we call it a virtual repositories okay
because it’s also will provide us the proxy for local repositories and remote
repositories and as I devops engineer or developers it’s transparent for me that
and that I will just want to bring some components I don’t really know if it
will be uh taken from the my local repository or from the remote repository
it’s based on the version Etc and here uh we can control this
process and we can say okay I only want first to find these this package and
this version of my local repositories and if not if it’s not there only then go to the remote repositories in the
finder and it in this case uh we call it a priority resolution so I want to set
priority for search search priority for the repository managers and say okay the
first one first of all you will find it on your local repository always okay so
it’s your first priority if not there only then maybe maybe you will go to to
find it on the remote much repositories so while I said maybe
because for some cases I want to prevent a
repository managers to find on the remote repositories why because we are
talking about internal um internal package so this package no
way it will be on the remote repository so I don’t want my repository manager
will you know try to find it on the remote on remote domain so it’s it’s all
it should be on the local always so I will prevent uh file it on the remote
repository and also the flag that vendors provides and say okay so um we
will talk about it on the next slide we want about the sculpt
you know lets me let me go there so we are talking about
um sculpt packages so I can Define the uh name okay or the scope of my package and
then I can exclude this name or this scope from the search uh from the
request for the uh remote repositories on the on my on my uh repository manager
system so I can say Okay so this path for example this passes only belong to uh internal repository it’s my internal
packages in my internal packages so I I also want to
um to use it internally only and please do not try to search it on the remote
remote repositors that’s debate is the main idea here okay uh what else I also can
um use the flag and map my latest version to the specific version on the
local on the local repository so why again it’s it’s a it’s a very very good
practice against the dependency confusion because I can say if if I’m
not using a specific tag for the version for example it’s not best practice but
yes or maybe I’m using letters tag or I’m not providing the tag and it’s going
to be automatically latest result will go into resolution for the latest version in this case I want to
say okay even we have the highest version on the remote repository and I will my latest
version is this one specific on the local repository and then my repository
manager will try first of all to see if I have this flag on the nylon and then I
will say okay so you wanted letters and the letters means this specific version on the local Repository
uh okay so it’s it’s a good uh good practice but just to be aware that it
might be uh might provide a lot of noise because developers will see the same same version all the time uh they
they’re using letters but they will see the same same version uh so it’s a big
question how they will act in this situation uh okay so as I said uh something uh one
more uh tip for the build manager layer so as I said so we can we can skip a
version tag okay we can use latest or we can skip it like you can see here so we
just we just ask for LTS Alpine and by default it will be uh the latest one
okay the latest version uh so it’s not the best practice as I
said it uh produced unpredictable image in this case to be part of my
production so I want to avoid it and else I want to avoid dependency confusion attack again uh someone can
actually upload the highest version of of Alpine of LTS Alpine this malicious
con so I want to use exact specific uh specific version of the LTS Alpine so
how can I do that so the best practice is to provide the hashtag the hashtag is a looks like this or the shot too and
they are hash okay in this case uh my uh build manager or and my repository
manager will bring the specific package in the specific version to my uh to my
project it’s a no way that someone will change it somehow because I’m using the hash uh or it will bring the different
version okay so I’m asking for specific package okay so it’s very good
uh well it’s a best practice actually okay so we discussed the uh sculpt
packages and now the last touch um as I said we are linked on the trust
between us between uh between the customers and the contributors
open source packages contributors so but we want to build
um some trusted networking some maybe peer-to-peer networking and then
um as the contributor again place or I can upload my package to this network uh
this package will be reviewed and scanned by different tools by uh the big
Some solid authorities and I don’t know who what company will be will take the
part of this authorities or maybe Oracle maybe Reddit whatever and so they will be responsible to scan and verify and
sign this new package you upload the package and then as I can as a user or
as a devops engineer I can just need to configure that I’m going to use this networking as my remote repository or
local repository or virtual repository and then I will be a very sure that it
will be the the good package so it’s no way that someone will masculating a type of squirting there and
it’s it’s a good it’s very very uh protected networking on this very very
big trust okay so um and one of the projects in this in
this era is the Persia I also suggest you not to read about it or maybe uh to
start use it it’s uh it’s just started okay but it’s very interesting so if
it’s uh it will be uh with pushing it this project uh and the idea is the it’s
a very good idea okay and a very big Potential from the security point of viewers from the from the performance
point of view so it’s uh I think it’s a good idea so uh um I’m suggest you to um to read about
it um and that’s it thank you very much uh
I think and I hope it was uh useful um
you can contact me by LinkedIn and send me messages or query your questions and
have a good day thank you