DevSecOps for Kubernetes-Based Applications@ SKILup Day DevSecOps

September 17, 2020

< 1 min read

DevSecOps is so important as Security is everyone’s responsibility as demonstrated in the rising trend in enterprise interest in DevSecOps. During this SKILup Day event with the DevOps Institute, Sven Ruppert, explores DevSecOps for Kubernetes-Based Applications from a technical, process, and cultural point of view.

View Slides Here


Sven Ruppert

    Sven Ruppert

    Developer Advocate

    Sven is working as Developer Advocate for Jfrog and coding Java since 1996 in industrial projects. He was working over 15 years as a consultant worldwide in industries like Automotive, Space, Insurance, Banking, UN and WorldBank. He is regularly speaking at Conferences and Meetups worldwide and contributes to IT periodicals, as well as tech portals. Additional to his main topic DevSecOps he is working on Mutation Testing of Web apps and Distributed UnitTesting besides his evergreen topics Core Java and Kotlin.

    Video Transcript

    hello and welcome to devsecops for
    kubernetes based applications
    my name is van rupert and i’m developer
    advocate for jfrog
    specialized for devsecops
    what do we want to do today we want to
    talk a little bit about what are the key
    features or the key points for
    cloud native development or cloud native
    in general
    because this will lead us to some
    special things
    called devops or devsecops here we want
    see why devserkops is good for business
    and what are the key points here
    and this will lead us to the developer
    we want to have if you are implementing
    from scratch so the main focus will be
    how will be the view for you as a
    if you have to deal with dev sag ops and
    if you’re developing for kubernetes
    based applications
    one point in the end will be what is a
    good thing or what you can do
    to speed up your production line or your
    ci environment
    if you are dealing with long running or
    with complex
    um ci pipelines what is true
    immutability and why you should go for
    one thing for everybody is if you’re
    dealing with new toolings or new
    and you have already an existing
    infrastructure how to integrate
    jfrog tooling in your existing
    environment yeah that’s it so
    let’s start better view
    cloud native if you’re looking for the
    words cloud native or what it’s
    mean in the internet you will find that
    it’s a
    broad topic it will go from service or
    oriented architecture of api um oriented
    through how to use container and all
    this stuff and everything is managed by
    so the meaning of this is not only some
    single topic about how to do a or b
    it’s a broad topic about software
    development in everything
    around reading about it will show you
    that you have a lot of different
    meanings opinions and all this stuff but
    in general you have
    via cloud native foundation the
    possibility to
    get the condensed and the official
    meaning of this
    finally so it took some years but it’s
    available so
    cloud native what does it mean it means
    that you have solar service oriented
    service oriented architecture means that
    you’re breaking up your
    big big application into smaller pieces
    microservice is one word or the other
    thing is that you’re just going to
    serverless functions on
    which means that at this point you will
    very tiny part projects or from the big
    project you have
    tiny sub-projects and the sub-projects
    can freely decide
    what technology they’re using and
    how they want to deliver that inside the
    container that
    could be managed by a docker or via
    biocommunities in the end
    so this is a container-based
    infrastructure that
    everything is running inside docker
    managed by kubernetes
    the service oriented architecture is a
    way how to split all this stuff
    and then there is is api oriented
    api oriented communication is just
    to define how the parts are
    sharing information and
    all together is the possibility to
    choose freely the technology
    it’s good and bad at the same time so
    it’s good because you can
    freely choose it’s good because you can
    get rid of legacy stuff
    but it’s bad because you’re dealing
    always with all these tiny details of a
    new technology
    if you’re a senior in one technology
    doesn’t mean that you’re a senior in the
    technology as well your ramp up time
    will be maybe faster compared to a
    youngster but in the end if you’re
    senior here you’re a youngster here
    and this means all this stuff like
    security issues
    best practices and uh what are the good
    and stable subcomponents or dependencies
    you have to learn again and this is a
    bad thing
    because with new technologies you have
    new attack vectors that can be used
    new ways of breaking into your system
    and all this stuff you have to learn so
    you need tools to help you
    to identify what kind of security
    breaches are there
    and then the other thing is how to deal
    with license
    comparing java with javascript for
    the java ecosystem is a way more stable
    and the javascript one is very
    active it’s it’s developing a lot of
    stuff in short time
    but the dependency tree between
    components and how this is versioned
    and the security issue there and the
    compliance issues
    this is a completely different thing
    okay this is the technology how to run
    it uh
    on a technology how to uh split it up
    how to communicate between all the
    and the whole thing that is managing
    this one is called
    devsecops so the cloud native foundation
    is explicitly talking about
    cloud dev segues not about devops
    so the main thing is what is
    the difference between devops
    and devsecops this is one thing but
    i want to have a few more um ideas what
    does it mean
    or what what part of this cloud native
    full stick we want to
    look at i want to have a detailed look
    how to implement stuff so if you’re
    looking at this part from
    coding software until let it run in
    we have through the application this
    piece of code you’re writing as a
    developer and this is the first thing
    you’re providing this one must be
    wrapped inside linux or must run
    inside linux and this one will be
    packaged in docker images
    that will be delivered
    via registry uh to your kubernetes
    so these are the four steps i want to
    have a look at
    and if you’re checking here so if you
    have a security breach
    inside your application during the time
    you’re coding it will be available in
    all other stacks
    maybe it can be masked or hidden by some
    stuff like a firewall or reverse proxy
    but in general their security hole
    is available the same thing here with
    license so you can have license
    or transitive dependencies here during
    the time you’re coding the application
    you have the same thing
    in the linux distribution and for sure
    in docker and kubernetes as well so
    everything will be managed or the whole
    life cycle of this will be managed by
    term called devsecops okay this is
    so have in mind if you’re
    pushing security or if you have security
    issues during the
    development time then they are available
    until the end and this is where this
    term shift
    left is coming so if you’re rotating
    this one
    at 90 degree or 490 degree then you see
    that you start coding application linux
    docker kubernetes and
    shift left means that you’re going in
    this direction that even during the
    coding times the earliest possible time
    you are starting with the term security
    so and inside the application
    development again
    coding part and then testing and so on
    shift left is the same orientation
    speaking about dependencies in the java
    world you have a bunch of dependencies
    because you don’t want to reinvent the
    wheel that makes sense
    because if you want to sort something or
    want to create a pdf library just use a
    dependency for it
    but you have no control about it you
    have to trust this guy not only that
    he’s creating good quality that he’s
    maintaining this stuff
    but as well you have to check for
    example for compliance issues
    so is all every dependency checked
    that he is using so not only you are
    checking the
    compliance issues for this one for this
    dependency but you have to check it for
    the next
    level again so all transitive
    dependencies must be checked for
    and as well as for compliance issues so
    it’s easy to analyze and so stuff but a
    lot of stuff is based on trust and you
    have to check
    security and compliance issues this is
    one thing
    so um yeah to achieve this in depth
    cycle ops
    well it’s the same like devops you have
    to optimize everything you have to speed
    up your production it means
    get rid of boring work from the
    developer let the ci
    environment do all this things say
    security to integrate is just
    extending a little bit the tooling but
    you’re going through the same stuff
    you’re using cr environment you’re
    adding some checks
    you um train your people that they have
    security that they are security-minded
    or securities skilled
    the same thing you have done earlier
    with operations
    so here one thing that’s very important
    is be reproducible
    everywhere so every tiny step must be
    somehow reproducible otherwise you can’t
    analyze later if you have a security
    so in general devsecours means on the
    box as early as possible and make sure
    that there is no security and no
    compliance issue
    and make sure that this is as soon as
    eliminated inside your production line
    the next step will be security from
    scratch and how this world looks like
    for you as a developer
    security from scratch
    now we saw what what are the basic ideas
    about it
    and we’re going back to one picture i
    showed in the beginning it was this
    application linux docker coordinated
    universe stack
    that is more or less basic for for cloud
    he i explicit exclude
    the concept phase it’s true if if you’re
    talking about security
    even during the concept phase you should
    have an eye on security
    you can’t have security per definition
    or not per definition per
    architecture so this will have some
    um influence on it so
    even there you have this security id so
    this concept this way of isolation this
    way of reporting
    it will include changes in processes how
    to deal with all this stuff
    but it’s not part of this talk here i’m
    just focusing from the application
    down to the kubernetes deck and just as
    a reminder
    if we have some security hole in the
    during the time we’re coding it will be
    in the linux
    in the docker and in the kubernetes
    layer as well so if you’re doing all
    this stuff
    as early as possible then it will help
    us to minimize the risk
    but talking about the different layers
    if you have different application or
    microservices and the freedom to choose
    what is the right technology you want to
    immediately you will have corresponding
    infrastructure things
    for example if i’m talking about java
    then i have
    for my application a maven repository
    the main repository is a single source
    of trees
    if i have to check what is used in my
    and based on this information i have the
    possibility to analyze a whole graph
    but then talking about linux i have the
    linux if i’m using debian i have a
    debian repository
    in the background and this is source
    where i’m grabbing all my stuff
    and in this repository i have my
    binaries the license information and all
    this stuff
    after i’ve done this one going to the
    docker image layer
    i have my docker registry and again
    this is a repository where all this
    stuff is stored
    and if i have access to this one i have
    access to the whole binary stack
    and can analyze every single layer
    inside my docker container and the
    docker files and so on
    and after this one how it’s mounted
    then i have my kubernetes layer and
    then we can talk for example or about
    hem repositories what we see here is
    that we have more or less a common part
    of this deck for example this
    uh docker helm part and then based on
    the linux distribution a debian
    repository or an alpine repository
    but then the biggest fluctuation will be
    in this application layer if you have
    different languages different
    technologies we have npm or maven
    or paybob nougat or whatever
    so we have different levels here
    different repositories
    the good thing with artifactory is that
    artifactory can handle all these types
    last time i checked it was i don’t know
    24 26 different kind of packet managers
    so you can have all these repositories
    inside artifactory so everything that’s
    coming out from the internet or in from
    the internet
    will be stored in artifactory and hold
    the good thing is you’re more or less
    independent if you’re grabbing it once
    and storing it
    you can say okay now even if the
    internet connection is not available we
    can produce
    internally caching for sure but
    talking about security x-ray
    is a component that will scan everything
    that is in artifactory how to do this
    and how to use it i will show it a
    little bit later but
    think about artifactory and x-ray as a
    that x-ray will have a detailed view on
    every tiny binary that’s going in
    that information you can consume via web
    or vian rest api so a machine can deal
    with a combination artifactory and x-ray
    or the human can directly interact with
    with it via a web ui i will show you
    things how it looks like but the main
    thing is you have
    one single point all binaries
    in all configurations are in and even if
    you have to
    for example think about the security
    payload injection even if you want to
    manage this one this could be part of
    artifactory inside the generic artifact
    repository so all these layers
    will have different behavior but you
    have different update cycles you
    have to know the knowledge of this
    repository so to deal with
    transitive dependencies and get the
    license information
    out of it
    how to define this stuff artifactory
    will give you the repositories x-rays
    connecting to it and now i have to
    how to search for security and
    compliance issues we have three levels
    for this definition inside x-ray one
    thing is a rule
    a rule is a stateless definition what
    should happen
    so you can say if you find something
    within c
    v s score from a to b please
    write an email too or please start a web
    hook or
    whatever break a build so this one
    is stateless and independent from the
    repository itself and this is an atomic
    and then you can combine this to a more
    domain specific policy a policy is a
    logical name
    and a combination or a
    aggregation or a composite of rules so
    the rule itself will describe what
    should happen
    a policy will have a logical name and a
    bunch of rules so here you’re more
    domain specific so for example the
    policy for web apps a policy for
    whatever and then to connect this one to
    the repositories you need
    watches a watch will connect build
    information or a repository
    with policies and then you can see the
    so the maintenance here is quite easy
    you have a very fine-grained way to
    describe what’s important for you and
    how to react to this one in a generic
    way and then you can just combine it to
    different repositories the good thing
    here is that you can just
    create a watch to for this combination
    i will explain a little bit later if
    you’re talking about the repository
    structure and what you can do here to
    kill rebuilds and to be really mutable
    so have this your mind rules are
    aggregated to policies and policies
    via watches are combined with the
    resource you want to
    you as a developer you have now the task
    to implement a use case or proof of
    concept or whatever
    so what you’re doing you’re starting
    your ide you have a tiny fresh
    side project you just start from scratch
    assuming that you’re including java and
    using maven
    but you have different other yeah
    package managers as well so
    it’s not it’s only an example because
    i’m a java developer
    so you start writing your pom xml file
    and you add the first dependency because
    you don’t want to reinvent the wheel you
    don’t want to implement this sorting
    or whatever and then immediately you
    will see information
    inside your ide so we are providing for
    intellij for eclipse vs code and all
    on an open source base plugins for the
    so if you have for example netbeans yes
    so far i
    don’t know it no plugin for this one you
    could create for example for netbeans
    here and plugin based on the
    we are providing on github but the main
    thing is how to use this one
    so you are adding this one and then you
    have this ide plug-in
    and you will see immediately this
    will have this security information it’s
    green and then you’re checking all
    transitive dependencies
    and you can see the license information
    as well
    how this looks like and how to how this
    handling is i will show you for
    intellij and this is the next thing
    okay now we have a short view to the
    x-ray ide plug-in
    i just added in my palm dependency so
    this is about inversion it’s a little
    bit older one so that we
    can guarantee see some compliance issues
    what you have you will have in the ide
    plugin here
    some some kind of window i’m using
    intellij so it
    could look a little bit different on
    eclipse or whatever
    then you will see the dependency we
    added here already here this uh 4120
    4120 and then you have the whole
    dependency tree and you see
    there is a security issue with jackson
    data binding
    you what you get is a ranking if there’s
    a high medium
    or low security risk you will see the
    license information or type version and
    so on
    the good thing is you can see here all
    detailed information for example
    if there is already a fixed version
    and what time is summary of the issue
    one cool thing is if you are somewhere
    then you can just go here and say
    sharing project descriptor
    then you’re jumping here back to this
    one and the good thing is if you decide
    to exclude you can just go here and say
    exclude dependency what you will get is
    the dependency exclusion that’s it the
    only thing i want to show is how to
    install it so in intellij is quite easy
    again just plug in you’re searching for
    jfrog plugin
    updating it or just installing it in
    intellij you have to do a restart
    in eclipse could be different and then
    the only thing you have to do
    is you’re going to the settings of this
    x-ray plug-in
    adding your coordinates and then you can
    connect your intellij to the x-ray
    that’s it have fun you saw now how this
    is realized in this ide plugin and
    this was just an example with intellij
    but it’s available for other ids as well
    so this is a way a developer would see
    it immediately
    and now i want to just
    show you how how it’s available on on
    the web ui and for this i’m taking an
    for example uh the next step so we coded
    the application now we want to wrap it
    uh on linux docker so that we can
    provide it later on
    um kubernetes so what what you’re doing
    if you want to wrap your application
    your fed jar for example that’s running
    uh inside the dockerfile
    you you start with the dockerfile then
    the first line
    from and bang here we have it
    we are using a base image and this base
    you have to analyze so this docker image
    will be based on debian ubuntu
    alpine whatever and immediately you
    should have view
    on what is going on on this operation
    level operating system level
    if you have a library somewhere in your
    linux distribution
    that is critical and then you can decide
    if you want to have this package or
    you’re explicit de-installing it or
    or or so the tip will show you
    how to get this information about doc
    images and again over the
    java site but this time with the web ui
    what we see now is how to create a rule
    how to combine
    different rules of policies how to
    create a watch and how this report will
    be accessible inside your web ui
    and then you will see that from your
    your tiny jar file up to the operation
    system and the docker image you will see
    everywhere in this tree
    what’s going on because x-ray know
    the whole dependency tree that means
    after we combined
    the different technologies x-ray knows
    okay this jar is inside this layer
    inside this docker image and this docker
    image is based on this linux and this
    linux will have this
    glibc whatever version
    inside every time we are
    scanning or we are updating the security
    database immediately if we are pushing
    this one to our installations
    if you are not connected to the internet
    because you are on prem you can just
    download on a regular basis security
    database and provide it internally
    immediately if you have this one the
    whole graph will be rescanned
    and it means even if you provided it
    yesterday you binary your docker image
    and it’s
    running on production and if you know
    something new it will show you
    that this image is infected how this
    looks like how this
    report looks like i will show you on the
    web ui
    now how to create the report
    vulnerability report for this first of
    all you have to
    log in into your platform
    by the way this is a free tier it means
    if you want to activate a free tier for
    you to try all this stuff out
    follow my how to about 3t activation
    or just go to start minus
    so after login what you need
    to create reports is
    you need re repositories where something
    is already in
    so i have here a docker virtual
    repository that is combining a local one
    and a remit one and
    inside this docker repository i have
    the build pack depth docker image
    but you can do it with everything else
    so it could be a maven repository
    whatever but you need something to
    with content so to create the report
    go to security and compliance go to
    then you can create a report you should
    choose a
    name in the free tier you have only the
    possibility to scan for vulnerabilities
    if you have a paid plan then you can
    scan for compliance
    as well so license checking the next
    thing is you have to choose scope
    i’m just scanning repositories right now
    but you can do the same stuff with
    builds and release bundles
    so it’s just the same i’m focusing here
    on repositories
    the next step is just the selection of
    the repository
    i will just scan here for my docker
    a and b selecting then
    if you see it here this is what will be
    save you can add some more detailed
    informations what you want to have
    inside this report or not so for example
    you can
    choose if you want just a range of
    zvss scores i’m just collecting
    everything you can
    select the date and scan data i’m just
    scanning for everything
    with every cvs escort on all components
    after this generate report
    as you can see it will take some time to
    do the job in the background
    it depends on the amount of repositories
    and the amount of content inside of
    if this is created once you can
    go here and say i want to have a view on
    it i want to see detailed or i want to
    export or delete it
    we want to have a view this is a report
    what you will get is a cve number
    a short summary the severity
    and then detailed information about the
    cvss score
    and why and the component
    you can see it’s uh the lip that is
    the artifact let’s affect it infected
    and then the path where you can find
    this stuff
    and very important if you have a fixed
    version for this
    already if you want to export it go here
    to export
    then you can choose between different
    formats i’m just using pdf
    for example will take some time and
    you will get this download i will open
    this one right now
    and i will show this pdf
    here is a pdf with all the detailed
    so with this you are able to create on
    the fly reports
    about the content that is inside your
    so we have one thing that is not
    easy if we are now leaving the
    application docker linux layer
    because we want to talk about kubernetes
    and how to do the last steps the last
    step is
    if you have docker images you must make
    sure that your kubernetes stick is
    explicit using
    your repository to grab this
    dock image even if you’re selecting
    official ones
    so how to make sure how to deal with
    this for this
    we have a tiny open source project you
    can see it’s actively maintained so
    uh grab it it’s called qnap and that’s
    available on the jfrog
    qnap on github so it’s open source check
    and it’s a kubernetes dynamic administ
    at mission webwork complicated work for
    so what it what it is doing is
    it will make sure that first of all all
    images that are requested are pulled
    from some
    special repository that is under your
    control and scanned and only what say
    inside and free it could be instantiated
    and the next thing is if you have a new
    version of this docker image
    you must make sure that as soon as
    active instance are killed and
    redeployed within your version so that
    you have more control about this one and
    how to manage
    where is the source where grabbing all
    the stuff so that kubernetes is just
    um green images this is for example
    done with qnap so have a look at this
    one and
    now we have from application development
    over how to update the linux stack how
    to deal with docker images and make sure
    that you have clean layers
    um up to how to manage this docker
    images in
    kubernetes and it’s under control how
    long it will
    live and how fast it will be updated in
    which way will
    it will be updated and what docker
    images are used
    if you want to try all this stuff by
    yourself it is easy because you are just
    going to this platform
    well it will take approximately 10 or 15
    minutes and then you have the whole
    ecosystem ramped up for you in your
    of your choice amazon google and i don’t
    i will say thank you at this point if
    you want to reach me because you have
    more questions or you want to
    just stay in touch the best way to reach
    me is twitter
    and i’m more than happy to have feedback
    and start a discussion with you about
    the topics devsecops java cortland
    mutation testing tdd whatever
    just call me and so far i would say
    thank you very much for attending and