DevSecOps: Low Hanging Fruit @ DevOps Sydney Meetup

September 16, 2021

< 1 min read

DevSecOps: Low Hanging Fruit 
These days Software Development is reliant on multiple dependencies (ie… Kubernetes, Operating System Layer, Java App etc..) Building one logical point for all dependency vulnerability scanning is something you can easily institute.

In this lightning talk we will:

• Show you a tool you can freely access to create a single point for dependency scanning before you build your proof of concept.
• show you how to access free vulnerability scanning access using X-Ray for fast, easy, and accurate results

View Slides Here


Sven Ruppert

    Sven Ruppert

    Developer Advocate

    Sven is working as Developer Advocate for Jfrog and coding Java since 1996 in industrial projects. He was working over 15 years as a consultant worldwide in industries like Automotive, Space, Insurance, Banking, UN and WorldBank. He is regularly speaking at Conferences and Meetups worldwide and contributes to IT periodicals, as well as tech portals. Additional to his main topic DevSecOps he is working on Mutation Testing of Web apps and Distributed UnitTesting besides his evergreen topics Core Java and Kotlin.

    Video Transcript

    all right fantastic
    um so welcome to the sydney devops
    meetup uh i am lindsay and my
    co-organizer michael is working
    somewhere in the background
    um before we start i would like to do a
    quick acknowledgement of contrary so i’d
    like to acknowledge the traditional
    owners and custodians of the country
    throughout australia which in my case is
    the dark and young people and we
    recognize their continuing connection to
    land waters and culture with our
    respects to their elders past and
    present sovereignty has never been
    seeded and treaties has never been
    signed it always was and always will be
    aboriginal land
    lovely to have you all here in the
    september meet up
    uh quick refresh on the code of conduct
    before we get stuck into it so we don’t
    tolerate harassment and meet our
    participants in any form or
    communication should be appropriate for
    a professional audience please remember
    that harassment and sexist racist or
    exclusionary jokes are not appropriate
    here great thing about doing this on
    zoom is that it’s really easy for me to
    boot you haven’t needed to do that yet
    but the option is always there
    let’s get stuck into it for tonight uh
    this is the agenda um so we’ve got the
    intro which is literally the thing that
    you’re listening to right now
    then i’m gonna go
    introduce our first speaker in talk
    number one uh we’re going to do a quick
    events and job section in the middle and
    we’re going to roll into talk number two
    so hopefully have this all wrapped up in
    about an hour and you can get back to
    um maybe lock down life maybe not lock
    down life it sort of depends on where in
    the world you are
    um just a quick heads up on the job
    section that we’re going to do a little
    bit later so if you are looking to fill
    a position either you want to fill the
    position or you want the position to be
    filled um there’ll be a 30 second slot
    where you’ll be able to talk about any
    positions that you’re hiring for or talk
    about yourself if you’re looking for a
    new position um and so we’ll just get
    people to drop in the chat uh if they’re
    keen on doing that and feel free to do
    that anytime from now
    let me let the next person in
    um so for tonight the talks are we’ve
    got sven rupert uh calling in from uh
    sunny germany i believe um talking about
    devsecops quick wins a low-hanging fruit
    and we also have kieran sweet uh talking
    in talk number two slot about cloud
    standards with architecture decision
    records so really great content tonight
    i’m really happy that you can all be
    here with us and with the fantastic
    speakers dialing in from quite far away
    so quick show of hands and feel free to
    use the reactions in
    zoom or uh or you know drop something in
    the chat or even like enable your video
    and let us see your beautiful faces
    um who is the first timer tonight
    awesome lovely to have you
    anybody else
    oh yeah there’s another person fantastic
    lovely to have you here
    as well
    oh yeah that’s true
    but uh new folks all over the place uh
    well it’s lovely to have you all here i
    guess the other thing i’m sort of
    curious on like the doing doing uh
    these online meetups and during covert
    uh sort of the great level when it comes
    to how far away people can be could be
    dialing in from i’m curious to hear is
    there uh oh can i can i get a shout out
    from all the folks that are uh that are
    based in sydney
    i assume there are some folks hey there
    we go we’ve got a couple of fun thumbs
    up from richard and stephen and tarly
    and philip fantastic lovely to have you
    all here and uh alareza as well anybody
    coming from further afield but perhaps
    still in australia
    adelaide adelaide awesome lovely to have
    you here
    anywhere further afield in australia
    than adelaide got anybody calling in
    from wa today or tazzy
    not that i can see all right let’s go a
    little bit further anybody from outside
    of australia
    sven you don’t get to answer no okay you
    just did it did answer very good
    anybody other than sven dialing in from
    i’m not today
    sven you might be winning the prize uh
    which is non-existent but you get the
    honorary title for uh for this meet up
    lovely to have you here oh we’ve got a
    few people in the chat as well we’ve got
    oh jim coming in from shenzhen in china
    spectacular wow
    well that’s lovely to have you all here
    uh it’s one of the
    few silver linings to the pandemic uh
    means that we can sort of lower the uh
    lower the barrier of entry to get into
    events like this oh we’ve got uh david
    coming in from kuala lumpur as well
    lovely to have you joining us from kl
    all right well the good news for all of
    you coming along to uh this event is
    that the online meetups are going to
    continue until 2022 uh although we we
    did hit that magic
    uh 80
    uh first vax in new south wales
    milestone in the last 24 hours um and we
    are going to be in having some form of
    opening up in the next couple months um
    we are going to continue with the online
    meetups until at least early uh 2022
    i’ll be honest i am not planning to go
    back to in-person meetups um actively uh
    but you know hopefully sometime in uh
    in you know the first quarter or maybe
    even second quarter depending on how
    we’re going with lockdowns and whatnot
    we’ll get back to some in-person meetups
    so for the folks that have uh that are
    sort of missing out at the moment um
    help is help is on the way we will have
    some in-person meet up soon um it all it
    almost feels like a bit of a a loss to
    get to this point of having like well
    running and functioning online meetups
    and then to this digital to go back to
    in person so i need to need to find a
    way to sort of keep that spirit alive a
    little bit if you’ve got any ideas
    please let me know because i don’t
    all right let’s dive straight into the
    first 12th of the night we’ve got sven
    rupert he’s talking about devsecops
    quick wins and the low-hanging fruit i’m
    going to stop sharing my screen and sven
    i’m going to hand it over to you
    okay so
    thank you very much so that i can
    be at your meet up
    i was in sydney before the pandemic
    quite quite regularly so at least every
    half year so if the pandemic is done at
    some day i will be there
    physically definitely so uh because
    every chance i got to go to australia i
    just took but mostly then i spent a
    little bit more time so traveling along
    the coast and all this stuff so i’m
    missing it really definitely i’m missing
    it and even my kids are asking
    uh damn but so far
    we are now online and we are talking
    the low hanging fruits of death tangos
    just a quick question um who’s working
    with security stuff already in in
    this devops environment pipeline
    whatever you want to say to it anyone
    okay one shaking hand
    one two
    not so many good for me
    so i’m not repeating for too many people
    and well let’s see so hopefully the
    connection is stable uh we’re just
    crossing the half of the world and my
    the good thing is uh for me it’s just 10
    a.m so kids are in school they’re not
    destroying my internet bandwidth so this
    is good for you i think i hope the same
    is on your side so that we have a nice
    connection okay let’s talk about the
    difficult part i have to start sharing
    my screen
    i got it done once today already let’s
    see if i’m able to do it again
    i hope you are able to see
    my screen now
    by the way if any one of you knows
    how i can get back
    the chat window so if i’m activating the
    chat window and then i’m start sharing
    the screen the chat winner is
    disappearing anybody of you knows how to
    get rid of this behavior and zoom
    please if someone has some idea please
    let me know
    okay i’m asking this question i think
    uh since the beginning of the pandemic i
    have no answer so far
    and uh well no future requests okay so
    my name is flynn and
    what we are talking about today i want
    to talk a little bit about the
    low-heating fruits the low-heating
    fruits or the quick wins or
    how to start with staff circles mostly
    always the same so i’m attending
    different workshops and and
    presentations all this stuff and people
    are coming through me and asking me okay
    so i have nothing what’s the best way to
    start uh what what is the key point to
    start how to sell it to my manager
    will it be expensive
    do we have to change our processes and
    what else are so
    many questions
    well it’s easier as as you can expect
    but we will see or we can start with
    integrating security in this devops
    and i want to see
    the difference here between different
    parts of security so most people if
    they’re talking about security they are
    just talking about vulnerabilities and
    cyber attacks
    but business continuity or saving the
    business for
    or against a lawyer this may be some
    part of security as well and sometimes
    more important
    oh i’m getting a bunch of
    yeah there’s i can read them out to you
    so that maybe you can see them or at
    least you won’t be able to see that you
    can hear them from me uh oh we’ve got
    one suggestion from richard borges
    saying try alt h to open the chat window
    perfect perfect i
    so i’ve done it already i just want to
    get rid of this behavior and start
    sharing the screen and the chat video
    that’s already there is disappearing
    i don’t know well okay so off topic um
    by the way my name is ben i’m from john
    yes already mentioned and if you can see
    i’m mostly in the world so every three
    minutes i can
    grab with my kids or by myself i’m just
    going to the woods spending their the
    day the night and so on and it
    was covered i i got the idea that why
    not doing online meetups from time to
    time if they’re recorded
    in the woods so if you want to see one
    of my recorded talks then maybe the
    youtube channel is something for you i
    have them in german i have them in
    and then you can see
    i’m talking about different topics in
    the woods in the german woods
    not so big as australian ones that’s
    still some nice places there
    okay and by the way if you want to have
    my shirt
    you can grab one of these twenties
    go to
    you can do me a favor if you like this
    talk you enjoy it please give me a big
    thumbs up then this rating and then my
    my manager will be happy with me and
    give me more opportunities to go
    somewhere on this virtual meetups and on
    the other side you can grab one of these
    shirts it’s not this one this is from a
    conference but we have this cool frog
    shirt so ah you will see
    some some stuff so okay enough of this
    while we are talking about death
    circuits step cycles is more or less
    okay other way around what what’s going
    on in in the industry in the industry we
    trying to
    split up more more our monoliths where
    we try to go to this micro service world
    maybe serverless world whatever what i’m
    talking about
    you can do it with monitors you can do
    it with serverless with functions you
    can do with microsoft
    the cloud native is more or less a
    little bit excuse the excuse for what i
    try to explain and what i tried to
    highlight here at this moment
    if you’re going to these different
    stacks and different levels of
    abstraction we have something for
    example service oriented architecture
    long long time ago it was with soap rmi
    and all this stuff now it’s with
    different techniques so but the main
    idea is that you have different
    lit up parts and they are start
    if you are talking about technology can
    can the technology help you to identify
    this bunch of use cases is good for one
    microservice and this bunch of useless
    is good for the other microservice
    mostly not so this is something a human
    has to decide so if you have security
    aspects on the architectural side so
    these use cases must be
    fit together and then it must be
    isolated or whatever this is something
    the human must
    must do so the machine can’t make this
    decision for you if you’re talking about
    the next layer so how these pieces are
    communicating about the api oriented
    the technology
    can help a little bit more if you are
    talking about how to wrap the data it’s
    not able to to say what is good to send
    over the wire but it’s good to say how
    to send off the buy and if you’re
    talking about security the machine is a
    little bit strong and helping you how to
    make this secure
    so encrypted channels or whatever
    but if you’re talking about the
    infrastructure of every piece could be a
    container-based infrastructure operating
    system whatever
    we’re talking more and more about stuff
    where the machine can help you more to
    protect it with firewalls with analytics
    with observability and all this stuff
    because we we are going more and more to
    a technical layer that is not so
    dominant use or it’s it’s not a use case
    abstract thing that we want to solve but
    it’s more technical thing where the
    technology can help and we are talking
    about the devops layer or the in the dev
    segments layer so how to code how to
    build binary and all this stuff this
    really is implementing stuff the machine
    is really really good in helping you
    against vulnerabilities and compliance
    so based on this layers you have to
    decide where you want to start and i’m
    focusing on this devops layer a little
    bit about this container operating stuff
    but if you’re talking about security
    security by concept is one of the
    important things
    but tool steps are not helping you so
    good so if you want to start with
    security and you want to have the
    loading you have to focus on the
    technical part because the machines are
    quite good so we have different
    challenge why we have to change the way
    how to think about security for example
    with these shorter life cycles with
    these smaller parts we want to create we
    good sides and bad sides the good side
    is you’re getting rid of old stuff soon
    there’s just a ton of microservice you
    can throw it away you can rewrite it
    fast good on the other side
    with this shorter life cycles instead of
    maintaining things instead of improving
    things we are
    throwing it away writing a new and with
    this we have all the challenges of the
    new system with pardon a new system
    again okay
    it will be even more challenging if we
    start using polyglot environments
    every microservice in a different
    if i’m a senior in java i’m definitely
    not a senior go i will start writing go
    i will be fast in learning go for sure
    but i have to run the whole ecosystem
    the whole tool stick again
    with all goods and bad and i have to
    think about security in a complete
    different way because
    the environment is a different one i
    will learn a lot but we’ll do all the
    mistakes again
    so polyglot environments even in one
    application is even worse so if you’re
    talking for example about advantage it’s
    an open source framework to write
    web-oriented web pages server-side web
    pages you can write them in java but
    they will generate all this stuff by npm
    and the communication and the
    server-side client-side communication
    and then you have two different
    artifacts so you have a polyglot
    environment even inside your own
    microservices single microsoft and this
    is really a beast you see one tech layer
    but you have a bunch of other
    technologies in the background with
    different lifecycles so vulnerabilities
    a lot of stuff we have to focus on okay
    now i’m cutting away all these different
    pieces around so all these different
    nodes and kubernetes tags and whatever
    i’m just looking at one micro service
    now one thing i think what you start
    coding what what you’re working on so
    you have an application where you’re
    writing your code then it’s run
    running on an operating system that’s
    the next layer then this operating
    system is wrapped in some virtualizer or
    maybe darker full or whatever
    virtualizing you are taking using and
    then this abstract unit is distributed
    in this kubernetes universe so someone
    is managing resilience failover stuff
    heart beating and whatever a huge system
    the whole thing is that with every layer
    we’ll get a new way of vulnerabilities
    different attack vectors per layer and
    you can imagine it’s like like an onion
    if if you have a vulnerability inside
    your application where you’re coding
    where you’re adding dependencies then
    this vulnerability is existing
    all the time
    in all layers directly or indirectly but
    the most people are forgetting is the
    devops layer itself they’re focusing on
    this single thing i’m writing now this
    piece of code but it’s a whole tool
    chain around it
    it’s a disaster the whole devops
    pipeline is a disaster just mentioning
    the solar insect where this company
    was attacked and
    the person that have done it
    the group then modified the build
    information or the build pipeline so you
    had an attack against the supply chain
    just a question whoever checked
    their own ci environment against
    vulnerabilities and start hardening it
    whoever have done it mostly people are
    not thinking about it they’re hardening
    their production system they’re having
    the application they want to deploy
    but the whole tool said
    the compiler the ci environment the test
    environment the penetration test
    environment whatever they are not
    happening with this stuff or they are
    not maintaining it in the same way they
    are doing it with a
    with a production system and this is a
    mistake mostly
    so um
    i see
    it lasts here and anyone wants to check
    it later ah there are slides from
    someone okay
    in this single microservice node we have
    technical requirements and we have
    domain specific requirements and again
    the machine is good in technical
    working against vulnerabilities here but
    they are not so good in domain specific
    related vulnerabilities can this use
    case misused to do something against
    text or text or whatever
    but it could be
    how i can increase the right to be an
    so you have to make sure that you are
    not forgetting one of the sites but we
    are focusing now on the technical side
    again and we are looking at two things
    vulnerabilities compliance issues on the
    technical side because there the machine
    is good and you can start quite easily
    with this
    there’s a turn shift left i could
    remember this one if i’m just rotating
    the whole stack shift left means start
    as early as possible with removing
    vulnerabilities and compliance issues
    just to make sure that they are not
    existing in all other layers and on the
    other side if you are removing them
    early it’s mostly way cheaper than
    removing them later in production
    there’s a worst case it’s in production
    okay then you have to stop you kill the
    environment whatever you have to roll
    back and so on so it’s way easier if
    you’re doing it as early as possible so
    what is as early as possible but before
    we are going to this one
    for every layer you are using
    you always
    have a corresponding package manager or
    binary manager or a dependency manager
    that’s correlated to exactly this layer
    for example the application if it’s
    written in maven you have maven
    repositories linux maybe debian
    repositories docker has docker
    repositories kubernetes has helm charts
    and the corresponding repositories and
    so on and the most people forgetting
    it’s a tool step so if you need more or
    less generic repositories where you have
    all these binaries in an immutable way
    so that they can’t be compromised or you
    can replace them from the distribution
    of the whole toolset so for example your
    whatever you’re using their own scripts
    and so on
    so you have for all layers corresponding
    managers package managers or
    repositories we should store the binary
    so that you’re in a pin from outside
    that you can verify these binaries and
    then you can make sure that this
    binaries are not compromised later so
    with regular checks fingerprints and all
    that stuff
    for everything you need not only the
    binary but you need the metadata around
    the spine
    talking about maven dependencies for
    example you have a dependency that you
    are adding to your application and then
    this depends you will have different
    dependencies so
    it’s not enough just to check the single
    binary you need to understand the metal
    universe around this binary to identify
    what is correlated to this binary what
    is directly or indirectly used if you’re
    using this binary because in production
    it could be actively used then
    so it makes sense to hold this binaries
    and it makes sense to understand this
    meter information
    and at jfrog we have this art factory
    they we have all these package managers
    we are understanding the metadata of
    this package what we are managing
    and x-ray is a vulnerability and
    compliance canal that can work not only
    with binary overall takeaways it can
    work with the metadata itself and
    additionally if you’re building a binary
    we can use all this context information
    and putting a background factory in this
    build info part this is important if you
    coming to this topic of this asp on the
    software builds of material based on
    this executive order from mr biden
    so if you’re working directly or
    indirectly on the product that are used
    for the u.s government you will be soon
    pushed to the requirement that you have
    to build a full list of ingredients with
    fingerprints and immutability of all
    parts if you’re working um
    for some product that’s directly in
    directly used by the us government okay
    so there’s a question in this aws x-ray
    now x-ray is a product from jfrog
    x-ray is more or less one in compliance
    cannot i
    will explain it a little bit in detail
    we have two different things if you
    start with this we have one time to
    market and make or buy
    this time to market mostly in a company
    it’s oh yeah with this use case as fast
    as possible it must be into production
    what happens you’re describing the use
    case then everything is done so you have
    this requirement
    and after this requirement is
    defined the whole production line is
    more or less important so you can
    produce it you can test it you can
    validate it and you can push it to
    production no challenge
    why not with vulnerabilities you have
    the requirement there is a vulnerability
    why it’s not impossible to immediately
    change something because the requirement
    is changing against the vulnerability
    and then pushing it to production
    the companies
    will have inside this security pipeline
    interaction with humans in a way that
    they have to validate re-authenticate or
    whatever and it’s just slowing down so
    if you’re working against
    vulnerabilities you have the same
    optimization goal with time to market
    for use cases and explain it to the
    management in exactly the same way it
    must be done as fast as possible there
    is a requirement
    all processes must be optimized in a way
    that you can do it as fast as possible
    the most stuff must be done by machines
    not by humans and all human interactions
    with verification or yes and no gates or
    whatever must be eliminated so that you
    can just work as fast as possible
    against this
    then make or buy mostly in this projects
    you have the decision should i write my
    own pdf printing library or should i
    add a dependency a developer has to do
    it every day it’s the same like should
    we learn it
    by ourselves or should we hire an
    external consultant
    by the way mostly if the external
    consultant is right because he’s from
    external and hired for exactly this
    challenge maybe we can think in the same
    way about the dependency maybe maybe not
    but the main thing here is that as a
    developer you have to do exactly the
    same that make or buy a decision should
    i do something by myself or should i use
    an existing thing
    if i’m deciding mostly for buy because
    why i should do everything by myself why
    i should reinvent the wheel i’m buying
    i’m adding a defensive i’m adding lines
    of code from outside i have to trust
    if if you’re talking about this one and
    just comparing how much i’ve done by
    and how much is a dependency somehow i
    would see that over all layers the
    amount of dependencies mostly the most
    dominant thing
    so in an application i’m writing a
    million lines of code but mostly i’m
    adding a few million lines of code with
    dependencies so dependencies are the
    bigger part than stuff i’m writing by
    myself if i now have to focus on the
    low-hanging fruits a quick win of
    security i must focus on binaries
    because if i start scanning my open
    source code i’m writing i have to deal
    with with
    machine learning pattern matching all
    these kinds that try to identify the
    context i try to write down in my code
    and this technique is not so
    good so far they are working on it and
    it will be a really good thing but it’s
    perfect right now okay but on the other
    side if i’m focusing on binaries i’m
    catching the biggest part of my
    application already with binaries anyway
    and scanning binaries is a very
    very straightforward thing i’m scanning
    binaries i now have knowledge about the
    binaries and that’s it and then i have
    to replace or not to replace it okay
    and this will be even worse with every
    layer on the operating system i’m just
    adding a few configurations the rest are
    binaries docker it even starts with a
    from statement kubernetes home charts is
    the same i it’s just a composition of a
    huge text step
    and again most people are forgetting the
    whole defaults towards the ci
    environment tooling compilers and all
    other stuff even this are just
    dependencies so if i’m looking at make
    or buy whatever i’m deciding in my
    application the bypass is by far the
    most dominant thing in the whole text
    what should i do first focus on
    dependencies because with this i’m
    more than 80 90 percent of my tech stack
    is a dependency somehow
    so i have to focus on dependencies
    and i have two things i can get out of
    defenses so in the security world i have
    different techniques i have this static
    application security testing that’s
    focusing on scanning binaries and all
    the assets i have then i have this
    dynamic application security testing
    this try to identify how i can attack
    the system so the system is running i
    try to break in this is mostly based on
    the most common vulnerabilities and this
    is done by machine learning and try to
    attack but for this the application was
    run already so it’s very late in the
    pipeline then i have this self-healing
    process since um that they’ve done this
    rasp um so that that i try to identify
    what part is just attacked and i’m
    deactivating it so that i can block an
    active cyber attack but this is already
    done in production i have this
    interactive security testing there’s a
    mixture of dust and dust but the easiest
    part and the fastest and the earliest
    part whatever you do so all these
    techniques are good
    no question about this but the basic if
    you start with just static application
    security testing because this is the
    earliest in the pipeline you can do
    and it’s focusing on the biggest
    part the binaries inside your system and
    you can do two things you can scan
    against compliance issues means that you
    have to scan what
    license is valid yes or no which is
    allowed at this stage inside your
    because the right license at the wrong
    place in your production i can cost a
    lot of money
    on the other side you have one-time
    effort in the beginning you need someone
    who is able to decide what is a white
    license for this part of the pipeline
    involves a
    black license so what’s allowed what’s
    not allowed for this you need someone
    who’s able to
    uh think about all this lawyer stuff is
    able to decide it or must decide it then
    you have the list of valid license and
    if you have done this one once you just
    feed the machine with these are the good
    license these are the bed license for my
    case and then the machine is doing the
    that’s it with vulnerability is a little
    bit different so you don’t need a lawyer
    in the beginning you can immediately
    start scanning with a machine against
    vulnerabilities and then
    of the time you will get vulnerabilities
    vulnerabilities vulnerabilities and the
    human has to decide what to do so how to
    get rid of vulnerabilities these are the
    two sides of preparation recurrent
    effort and initial effort to start with
    compliance issues and vulnerabilities
    but if we’re talking about this one then
    we have difference in behavior
    compliance issues are single points in
    your whole texting
    and the beast here is that if you want
    to find okay this license is good mostly
    the project is not changing the license
    anymore but sometimes it’s changing the
    for example this
    java ee jakarta
    eclipse ee stuff in the java ecosystem
    so we had a change of license so we have
    to get this one if you have a dependency
    and the transitive pencil it’s a
    indirect use dependencies uh switching
    license you need to know
    so this is good that the machine is
    scanning it even if this
    filtration is not so high but on the
    other side if you are
    able to find something that you have to
    um eliminate in compliance issue then
    mostly it’s a replacement
    it’s not a different version it’s a
    replacement of module and this is a base
    you need a semantic evil implementation
    that is able to fulfill the same job
    so this is a bad thing finding
    compliance issues
    with vulnerabilities it’s a different
    base vulnerabilities are in one layer
    but the combination of different
    vulnerabilities are the amount of attack
    vectors you have so you have the
    vulnerability itself and then you have
    all the other vulnerabilities and the
    vulnerabilities can have
    lower and cbs as courses come
    vulnerability scoring system by the way
    if you want to know more about this on
    my youtube channel i have a small info
    about this one you have different
    components how to rate this and how this
    fits to your environment
    but even if you have lower risk
    vulnerabilities in different layers it
    can be combined to attack vectors that
    are critical for you
    so here it’s really necessary not only
    to scan the whole graph once it’s
    necessary to get the full impact graph
    of vulnerabilities so this vulnerability
    is used is in this job this jar is in
    this web archive this web archives is
    using this docker file this dockerfile
    is used in this helm chart running in
    this environment in production okay so
    you need to know the full impact graph
    to identify all the tech vectors of your
    there’s one thing that is important to
    note so if you’re talking about
    vulnerabilities the either thing to
    identify if they are known
    what is the lifeline of a vulnerability
    and what is a part where i can jump in
    to to work against vulnerabilities so we
    have accidentally on purpose creative
    vulnerability that’s one thing
    if you have a bad guy then you try to
    hack something compromised could make
    him bad commit whatever um are we able
    to detect this one
    well not not really so if i’m a security
    researcher i’m able to identify it okay
    the regular developer of those of us
    they are not able to to work at this
    line against vulnerabilities you can do
    the best practice with your code that
    that you’re not producing code that is
    vulnerable but
    the really analyzes about
    vulnerabilities in their code this is a
    different b so we have this researchers
    and this is a bad thing because who is
    searching for it and why they are
    searching for it they are searching for
    it because they want to earn money the
    most of them
    some of them just want to make it safe
    and give it for free everybody but the
    most of them want to earn money there’s
    a huge market in terms of
    vulnerabilities and this is leading the
    whole structure how this is handled and
    what’s the impact for us as a regular
    developer so there is a vulnerability
    somewhere and someone is
    able to find it if it’s a bad one he
    will just sell it in the darknet and
    we’ll make money out of it can we do
    something no it’s hidden for us if it is
    a war taker he will go to one of those
    vulnerability database providers and
    will say hey i have here something or
    going to the product or to the company
    that’s building this or to the team
    that’s building this open software
    whatever so we have some kind of contact
    and he tried to offer this information
    so and then it
    who’s able or who will get that
    information it depends on the behavior
    of this person and there is no structure
    you can’t say okay this is yeah like
    like here here you will get money here
    or money there and who will be the
    if it is someone who just wants to push
    this information out for free
    good he will provide it everywhere for
    free nothing but the most
    the most stuff is
    people want to earn money then they are
    going to different vendors
    and saying okay i fear something wasn’t
    value off by the way the cbss is very
    it’s not so straightforward because the
    hacker want to have a high serious ass
    value the company of the product want to
    have no cbs s number so this behavior is
    not so trivial
    so but he’s going from one vendor to
    next and say okay how much money i have
    something how much money and who’s the
    well we don’t know with the most money
    whispers experience whatever
    so by the way
    we have no control who will be the owner
    the first owner of this information
    about this vulnerability
    that means
    we have no impact about this
    through this one but at some point this
    information will be public available
    means it will be part of some of these
    available vulnerability databases what
    does it mean for us as a developer
    whatever vulnerability database i choose
    it is the wrong one definitely so the
    only thing to work against this one is
    working with aggregators so we have done
    the same experience and with strafe rock
    we start aggregating different
    vulnerability databases
    even with a new acquisition of this
    company we are searching actively for
    new vulnerabilities so this is the only
    way so don’t focus on one provider use
    aggregator don’t go to the plane
    vulnerability databases use aggregator
    whatever aggregator you’re using
    building a superset it’s way better than
    focusing on one
    the next thing is if this is available
    doesn’t mean that’s consumable by me
    if it is in some of these commercial
    databases i can’t consume it if i’m not
    paying money and this is a bad thing
    security we’re talking about time
    we must react fast and if you like it or
    don’t like it doesn’t matter but mostly
    the commercial providers are way faster
    than the free providers
    if it is good or bad
    no no comment on this one okay
    but if it is even worse if the security
    information is going straight to the
    company that creates a product because
    sometimes you see just
    security fits no detailed information no
    information about how critical it was
    and so on it’s just really
    some security fix okay this is even
    worse in my opinion
    at this point where you can consume this
    information if you’re paying or using a
    free one whatever
    the first time you can you have access
    to some kind of resource to use
    vulnerability information it’s the first
    time you can implement or you can work
    faster with this information so you can
    speed up with paying a little bit so
    that you have this information earlier
    but whatever it means
    in this moment the information is
    consumable by you so that your scanner
    can work with this so that you have the
    identification this binary is that one
    the time is running and this is where
    you should focus on not on selecting the
    right provider and all this stuff first
    make sure that from the knowledge is
    consumable until it’s fixed in
    production is when you have to work as
    much as possible to automate everything
    because this is the only place where you
    really can influence everything
    and on the other side for the most
    people this is the slowest part and this
    set yeah this is really bad because you
    have everything in your hands and then
    you’re standing in front of yourself and
    you’re blocking yourself and then the
    is public available in some database you
    have it quite far because you have this
    uh knowledge inside your database and
    it takes weeks to fix it and this is a
    disaster because the most critical part
    is if the information is available and
    more critical is if there is a patch
    available because it patches a very
    information how to use this attack
    vector against systems so the worst case
    is information is public available that
    patch is available and you haven’t fixed
    it because then this is more or less
    super cool to use for someone who knows
    how to use patches against you
    okay so focus on
    your own things okay
    optimize it
    what is the safety build for you as a
    developer for you as a developer the
    safety build is a perfect test coverage
    because with the test coverage you can
    check if you have a semantic equal
    implementation against compliance issues
    if they are working yes or no
    and you can immediately change versions
    if you have the knowledge about
    vulnerabilities because fighting against
    vulnerabilities is mostly
    creating a new composition of the same
    components in different versions
    and this must be done fast so you need a
    very strong test coverage i personally
    really like rotation testing if you want
    to know what’s mutation testing i can
    give a talk about this one but rotation
    testing is way way way stronger coverage
    compared to line coverage if this is
    integrated in the ci pipeline mostly the
    change between versions is way easier
    way small smooths and faster and this is
    exactly what you need a change in your
    versions and then it must be completely
    automated checked and distributed to
    production this is the best what you can
    do against vulnerabilities so an
    efficient dependency management
    has the highest impact against all the
    known vulnerabilities okay so if you
    want to start with staff circuits focus
    on test driven development and the hard
    test coverage so that you can react on
    the use case you have to change because
    there is the requirement
    here’s a vulnerability okay
    this over all layers means
    you need the knowledge and the access to
    all binaries in all layers because we
    are dealing with management dependency
    management and you need a central place
    where all binaries are coming through
    and not only for and this is a mistake i
    see so often they have this single point
    of grabbing all these binaries for the
    application but why not using this one
    for example the debian repositories
    this management in in your local system
    what you can scan to feed your router
    your infrastructure and all this they’re
    using this dependency manager or this is
    this binary manager in the middle and
    this caching mechanism and scanning just
    for the application but use it for the
    whole infrastructure as well you have it
    so use it if you’re able to to replicate
    deep in repositories so that you’re not
    grabbing it from outside from some
    mirror that could be compromised
    go over your own repository manner scan
    it block vulnerable
    binaries with vulnerabilities so that
    they are not actively uh used again in
    your system
    this means
    having the binaries in one central place
    grabbing all this inside so that virtual
    classes not one instance could be highly
    available or clustered or whatever but i
    mean this logical place where all
    binaries are coming through over all
    tech layers you’re using and then you
    have the full impact graph of all
    oh there is something in debian but this
    debian is in this router use
    this will give you a good view focusing
    on the binaries and then if you want to
    try it out we have this free tier where
    you can use artifactory as an aggregator
    for all the binaries and then e3 to scan
    against vulnerabilities i don’t know if
    your register take three or five minutes
    and then you can play around with this
    okay so far what i can offer is if
    you’re interested we can have a workshop
    and do this stuff practically
    so hardness system for example if you
    want to know more about techniques i
    already mentioned like this what is
    mutation test and why we should use it
    or how this is integrated or this built
    in for what it says just let me know we
    can have
    definitely an a different um talk or
    workshop about it or we are offering a j
    for this regular dev setups workshops
    where if you have a client that wants to
    have this on so otherwise
    it would be now the perfect time for
    anyone there with questions
    and uh if you’re a little bit shy and
    don’t want to unmute feel free to post
    them in the chat as well and i’m happy
    to read them out
    so if you just start
    i think scan binaries let’s machine do
    the job optimize your production
    pipeline that’s the most thing and shift
    left shift left that’s most focusing on
    the ci pipeline the last comment here is
    you can shift left even inside the id i
    mentioned we didn’t mention it here but
    we have this integration of the
    securities kind of movement inside the
    ide so that if a developer adding a
    dependency in the definition you already
    will get the information what kind of
    vulnerabilities are here or in
    transitive views on dependencies that
    are used indirectly so shift left means
    really go to the earliest point
    what i’m not covering here is
    security by concept this is something
    that is not covered by
    like two words so far
    awesome well it doesn’t like we’ve got
    any uh
    questions in the chat at all but i just
    wanted to say thank you sven that was a
    that was a lovely talk and i really
    appreciate you uh dialing in from the
    other side of the planet to give it
    yeah thank you very much and
    well enjoy the rest of the
    night for you right it’s what’s time
    your time uh it’s quarter to seven here
    oh early early early that’s right we’ve
    got the whole night ahead of us
    yeah perfect enough time to check a few
    youtube videos later
    so five from my side and
    the next one
    all right let me share my screen again
    and we’ll get rolling into the next
    uh so actually quick thing as well uh
    for any of that feedback that you want
    to give to sven i’m just posting a link
    in the chat there you can go drop your
    feedback there i know that he would
    really appreciate it
    so events we’ll skip through this pretty
    quickly um the the main event that i
    want to talk about tonight was uh devops
    days which was scheduled to happen on
    october 28 and 29 in melbourne clearly
    events have uh unfolded and
    that event is not happening at that time
    um even if it were to happen i think
    they were able to get maybe 50 people
    into a room so not not exactly you know
    great sort of devil today’s event so uh
    i believe that the team is going to be
    announcing a reschedule it’s going to be
    in the first quarter of 2022 uh but
    still in melbourne um still same same
    location just a different time so um
    keep an eye out for that one and as
    there are updates on when that’s going
    to be i will make sure that i talk about
    that in these meetups as well if you
    haven’t been to a devops days event uh
    they are definitely the highlight of the
    devops space uh you know just globally
    but also here in australia so half day
    of structured talks uh the other half of
    the day is uh open spaces so just
    networking with other folks being able
    to talk through different issues and and
    you know get get feedback from your
    peers and ask questions and share the
    knowledge that you have with other folks
    as well so definitely worthwhile coming
    along to one of those
    otherwise there are a couple of other
    conferences that are coming up on us
    pretty quickly there’s p99 conf which is
    happening on october 6 and 7. that’s
    going to be online
    we’ve got sneak con which is happening
    on october 5 to 7 which is also online
    uh and there’s also hashiconf global
    which is happening on october 2019 and
    20 which is online as well there are a
    couple of web directions uh conferences
    that are coming up in particular they
    have a security one called web direction
    safe which i just realized i don’t have
    a slide for uh but that’s going to be in
    december so if you’re interested in that
    uh if you head to the web directions
    website and yes um
    great call out there tris
    uh lca i believe i don’t know
    if they extended their call for sessions
    but they’re definitely having a
    conference uh in mid-january if you
    haven’t been to lca it’s definitely one
    of the premier open source conferences
    in the world it’s definitely the largest
    one in the southern hemisphere
    so if you’re interested in talking at
    that if those call for sessions are
    still open you should definitely submit
    but it’s a definitely a great conference
    to head along to if you like really deep
    technical content
    all right so i alluded to this a little
    bit earlier but we’ve got a job session
    so the way that works is that if you are
    looking to hire people or if you are
    looking to or looking for a new position
    so you’re either looking to fill a
    position or looking to fill a position
    you’ve got 30 seconds to either talk
    about yourself or talk about the job
    that you’re hiring for um so i believe
    kieran popped up in the chat a little
    bit earlier about some positions at
    source group that he wants to talk about
    so i’m going to let him go first yeah
    cool um this is a bit weird because i’m
    like introducing myself before i
    introduce myself um
    yes my name’s kieran i work for a source
    group we’re a consultancy we focus on
    cloud technologies uh aws gcp a bit of
    we’ve got open roles for
    technical consultants in sydney
    melbourne and auckland and wellington
    in my realm we do also have other open
    in asia singapore malaysia
    and also north america specifically
    canada toronto which is out of my
    wheelhouse and one day when people can
    fly they might be able to go and apply
    for those ones and move there as well
    but if you are in those regions you do
    know people who are looking for work um is our website we have a
    link to recruiting and talent uh check
    it out or you can hit me up on linkedin
    just google me there’s not many other
    care and suites around i’m pretty easy
    to find
    that’s it that’s my pitch awesome thanks
    all right have we got anybody else that
    is looking to spruce a job or spread
    feel free to unmute and go for it
    or it might just be kieran for tonight
    counting to eight in my head waiting for
    people to
    no not happening great oh no i see
    another another on i’m muted there we’ve
    got a mohammed are you
    looking to talk mohammed hi my name is
    i’m recently a master graduate from
    in sydney
    uh so i have a
    four months internship in the west role
    uh so i’m just looking for a junior
    diverse role or junior cloud engineering
    in the field
    so if you have any role or any
    kindly refer me or anything
    which helped me
    awesome thank you for uh thank you for
    speaking out muhammad it’s really great
    to have new folks in the industry come
    and talk about themselves um if you want
    to contact muhammad feel free to hit him
    up in the chat on the side there or
    muhammad if you feel comfortable feel
    free to post your contact details on the
    uh on the meet up event as well so
    people can find you after the fact
    lovely all right anybody else before we
    hand it over to kieran for the last
    part of the night
    all right i have counted to eight in my
    head it doesn’t look like anybody else
    is uh going to give it a shot tonight
    but that’s great thank you both kieran
    and muhammad i’m just going to hand it
    straight to kieran right now he’s going
    to be talking about cloud standards with
    architecture decision records kieran
    take it away all right
    i’m gonna uh just i’m gonna minimize all
    my messages and stuff because uh
    i know i’ll take any questions at the
    end um
    just getting ready
    all right
    almost done
    all right moving things around uh can
    you see that okay
    yeah it still says that it’s loading i
    think we might be having a bit of the
    the thing that we were seeing before
    when we were testing it out let’s see
    how we go once it turns up i’ll flick to
    the first slide um yes there we go my
    video off doing some weirdness for my
    internet but uh
    all right um look thanks for having me
    um so i’m just flicking straight over to
    my uh
    my introduction slides so uh yeah today
    i’ll be talking about uh architecture
    decision records especially specifically
    in the context of uh cloud resource
    development uh but also wider ecosystem
    you know development uh when you’re
    building out sort of cloud platforms
    products uh and other things um so i
    guess first up just a little bit about
    me um this is actually a lightning talk
    so i’m actually just going to kick off
    my timer so i keep on uh
    keep on time my goal is 17 minutes
    maximum so um yeah just a little bit
    about me my name is karen sweets i’m a
    principal consultant source group so we
    are a consultancy we have
    some global reach we were started here
    in sydney
    australia my personal background is
    i started
    my career in doing linux and windows
    system administration i got pretty heavy
    into uh configuration management uh with
    puppet and other other products that was
    sort of ten years ago where i started
    learning all about infrastructure as
    code when it was in its infancy and
    probably for the last eight years since
    i’ve been at sourced uh i’ve been
    working primarily with aws
    um i do a lot with it uh currently
    and i’m pretty dangerous with the others
    i’ve done a bunch of azure
    and i’ve tinkered with gcp
    and those kind of tools
    i like automating things i’ve always
    liked them i’m a pretty average coder
    overall but you know i might do and made
    a career out of it
    but i also like helping teams be more
    effective uh when it comes to to scaling
    out teams and
    and helping them automate and be more
    effective my background is more
    traditional infrastructure but i have
    sort of been trending towards a
    development sort of view on uh on
    technology for probably the last few
    years um i do sort of write things up
    and throw them online at that url notes
    dot i do have a blog post
    that covers the content in this
    and i’ve got a bunch of other stuff that
    i’ve been uh sort of writing over the
    years i dropped a white paper on sort of
    cloud patterns earlier in the year uh i
    like on twitter you can get me there or
    you can find me on linkedin uh i’ll drop
    a live
    thing out after so that’s a little bit
    about uh about me
    all right
    um so yeah enough about me i kind of
    want to talk about a problem space or
    one one of a few problem spaces that
    i’ve had to deal with this year
    um about 18 months ago i
    i was engaged by a large financial
    organization to build out a greenfield
    aws platform now this was starting
    effectively from scratch new technology
    choices new reasons to do so
    um and it was their
    strategic landing zone for all of their
    future workloads
    and you know i got engaged into the
    organization and you know they’d already
    picked some of the tooling choices and
    things like that but to build this
    platform for their applications uh it
    was an aws platform first and foremost
    uh they wanted to use uh hashicorp
    products to build out the full
    capability the landing zones and things
    like that
    so you know terraform was first and
    foremost the tool of choice uh they were
    using terraform uh enterprise as well
    for some of this uh and then things like
    paco and other parts of that ecosystem
    were part of it
    now this was landing applications across
    the spectrum of aws as landscapes from
    windows to linux apps um so there was a
    wide range of um you know technologies
    in play this also covered you know a lot
    of other you know serverless
    technologies lambda
    and and such uh on there as well plus a
    container platform was being built out
    there and all that stuff so when you
    look at the technology landscape for
    this project um we had all of the
    infrastructure as code stuff plus we
    also had bash we had python red golang
    we had node.js we had puppet
    the list actually just kept growing um
    it’s all in there at the moment so it’s
    you know it’s a pretty big initiative
    so yeah it’s been quite interesting
    when the project kicked off it was a
    very small team and uh you know i was
    probably actually one of three people
    who started working on it but the
    initial team composition
    started by
    you know about 10 people they’re all
    co-located uh you know actually in
    melbourne i moved to melbourne for this
    project and um you know we’re all in the
    same offices
    and the code base sort of started as a
    couple of repos um you know with the the
    core aws platform we split out things
    for you know network repositories um and
    you know the core landing zones we had
    stuff for repos for you know the the
    actual structure of aws’s um you know
    organizational structure and things like
    and then we added different repos as we
    needed to for additional functionality
    that needed to be out of the the core
    code bases as as features
    as the project matured um
    the team began to go
    so what happened was was that you know
    the shared services team came on board
    they needed to build out zoe’s and linux
    amy’s and automation
    we introduced an entire new region of
    developers um we brought people in from
    india we got another 10 developers in
    new zealand initially
    then they decided they wanted to rework
    the network so they brought in a
    networks team and said you know we want
    to move away from the squids for egress
    we’re going to go and have a look at all
    these new aws network firewall stuff so
    they came on board started working on
    the code base then they decided we’re
    going to bring the terraform modules
    that the platform team
    is working on out they’re going to give
    them a whole bunch of devs another 10
    people give or take then we started on
    boarding customers so another 20 people
    there who wanted to help out you know
    across the platform uh various skill
    levels um
    and then you know we also had an
    operations team we needed to kind of
    support the front door you know and that
    kind of stuff
    so then you know things were ticking
    along and then the project got bigger
    so what we saw was that the customer
    wanted more features they wanted things
    to move faster
    very common uh evened out throw more
    people on it so um
    so you know the cicd team came in five
    people i can’t these are kind of finger
    in the air um
    numbers but it does represent a holistic
    view of the kind of uh numbers we’re
    seeing on the platform uh they wanted
    automatic account vending to happen so
    they wanted people to get more amazon
    accounts faster so they got an
    enhancements team to come on and start
    working on the same code base introduced
    new features new repos
    vmware and aws came along and said hey
    like we want to hold another landing
    zone uh more people came along
    um then they decided to do it out of
    container platform and
    uh what else did they put in there then
    and then
    you know they were like actually we want
    innovation accounts so we want to give
    people like free reign over here so more
    features and the list actually just
    kept going on um
    what ended up happening is we probably
    have about over 100 developers one way
    or another on this platform now uh
    building features bringing things up to
    speed uh and then covert happened
    and so
    this infrastructure project on aws
    went from you know
    a couple of people in melbourne
    to a fully blown remote development pro
    program of work
    a fully blown software project really
    all hell broke loose really to be to be
    this is a good news story by the way uh
    things are much better today
    but the cracks really started to be
    you know started to to appear uh and so
    what we found was is that the more
    developers that we had
    the less output that we
    were getting from them
    the main observations were that many of
    the teams were falling short of their uh
    their sprint objectives uh we we found
    that there was a massive drop in
    developer output
    and then
    the um the covered remote delivery model
    or time zones and you know
    working remotely and not having that
    personal interaction um really
    made the team struggle
    so um
    what happened was is that we wouldn’t
    talk to the teams you know and we said
    you know what what’s going on what are
    your biggest problems
    a big part of it was that code reviews
    and releases were coming different are
    difficult there was a lot of tribal
    knowledge going on between the teams um
    we found that
    when pull requests and things were going
    in um you know everything was being
    named slightly differently there were
    naming standards that were not really
    defined well uh
    um were spread out through multiple
    knowledge bases like there would be
    stuff in gear there would be stuff in
    uh confluence there would be stuff um
    you know on the back of people’s
    notepads there was tribal knowledge you
    could go and call this person or hit
    them up on slack or find this pull
    request and copy what they did kind of
    i think one of the most frustrating
    things for any technologist is to be
    referred to about standards that aren’t
    ever actually written down
    then other things started happening like
    the language selections weren’t clear
    you know we were mainly writing in
    python until we were writing a node and
    um you know it’s really hard to to be
    when everybody’s kind of doing the same
    thing different ways
    it was really hard to introduce further
    automation into the platform um you know
    i’ve always felt that standardization is
    the enabler to further automation
    and it was also really hard to bring on
    new developers somebody who would come
    in with like really good skills and it
    was impossible to get them working
    uh and as a result the the relationships
    were becoming strained uh what we found
    in the pr’s in regards this picture was
    that like you know for four pr’s i would
    all come into a certain repo and then
    like nobody wanted to rebase because
    they were all conflicting and stuff all
    of the time
    what we really needed was
    a clear way to capture socialize and
    collaborate on our aws and related
    ecosystems development standards
    it was around this time that there was a
    guy in my team
    i was actually
    seconded into the new zealand part of
    this program of work and there was a lot
    of experienced developers there um and
    one of them sort of said to me
    you know we really need adrs and you
    kind of put me on to to this concept and
    now an adr is is an architecture
    decision record now what they really are
    are just small text files that capture
    key decisions with your software project
    what they really focus on is recording
    key decisions
    and really focusing on the standards and
    what you’re trying to achieve
    with that particular standard and
    recording it accordingly now
    what what they’re focused on doing is
    not thinking about a full architecture
    solution but a key decision you’re
    making and
    recording it and placing it somewhere
    where everybody can review it
    now they’re really lightweight in nature
    generally what we find is that we we
    have no more than a title and an id a
    status is it you know accepted and
    applying to the environment or is it you
    know sunset because it’s not applicable
    anymore if it’s been replaced by
    something else
    the context of this particular decision
    uh and the consequences of not
    putting that standard in place
    now you encode the the adrs themselves
    uh with this data in markdown and then
    your version controlling git now what’s
    really cool about this is that it
    provides you a way to standardize and
    bring all of your standards for your
    program of work together but it
    democratizes the process in that if
    people want to
    you know
    propose a change to it
    they just open a pull request you know
    they they propose their changes uh and
    then the team around them reviews it
    provides feedback and then when it’s
    merged uh by the the consensus of the
    team um it’s then the standard now
    the other cool thing about is there’s
    some um
    there’s some tooling around it to help
    you sort of work with the the files in
    there you you don’t have to use them
    they’re pretty simple but um if you look
    at the
    page it does sort of frame some of the
    tools they’re all bash pretty much so
    yeah it really does help um and this
    book doesn’t exist i just made that up
    because i didn’t know what else to put
    here so
    so what i’ll do is i’ll show you what
    one looks like for
    one of the the projects that i’m on so
    one of the biggest problems that we had
    was that everybody was committing code
    uh the repositories differently so every
    pr was different the commit messages
    were you know you could really tell who
    was good at get and who wasn’t very
    early on because of you know the types
    of commits and the pr’s that were coming
    in so one of the the first ones is
    actually the number number three was
    this adr we put in for git standards and
    what we wanted to do was we wanted
    everybody to be consistent with the way
    that they committed code to the code
    bases and so you know we call out the
    context of why we’re we’re
    putting this standard in place
    and then we propose
    the implementation standards of what we
    expect for
    for this particular adr so in this
    context you know we want to make sure
    that you know the branch names are
    consistent that contains the jiras that
    the code is referencing
    for a feature or a bug
    we want to make sure that there’s clear
    commit messages that explain what’s
    going on
    we want to make sure that they’re
    attributed to an individual not unknown
    um and you know we wanted to squash them
    before we open the prs
    we do have a little bit of an example
    around how to set this up
    and then we also have a similar thing
    for the prs themselves so you know make
    sure that in github it’s got a title
    it’s clear and untruncated um it has a
    clear description you know why we’re
    doing this uh what’s been tested
    um and you know update the doco as you
    go and then we have the consequences
    right this is where we call out you know
    if we don’t put this in place what is
    going to happen
    you know in this case you know we call
    that you know we’re struggling to scale
    if we don’t do this
    what happens then is that we put these
    adrs in we open a pull request uh we
    bring them together
    uh after team the teams had a chance to
    review them
    uh and then once it goes in and it’s
    merged to the the master branch like
    it’s set in stone now what we normally
    do is we communicate out to the team
    this new adr is in place we generally
    have a cutoff for a cool off period for
    in-flight changes that don’t align to to
    be merged um
    but once it’s in
    we change the way we work now the other
    thing that we needed to do when we
    communicate this out is to let everybody
    know in advance that these processes are
    are in place they are part of our way of
    working now
    but then what we did was we then
    analyzed the environment for you know
    quick wins
    that were causing problems now one of
    the cool things about this um just to
    talk a little bit about the get standard
    specifically is once the standards went
    in and everybody started working this
    we were able to then put ci checks into
    the pipelines to say if you don’t align
    to the standards here
    um then just reject the changes or like
    straight away so it enabled a lot of
    automation early on
    but once we started getting these in we
    built a backlog of
    priority adrs to be created
    and then when they were defined it
    you know emerged we changed the way we
    worked we communicated them out and then
    we looked to enforce them further
    through automation
    now on this particular project um the
    the quick wins for us were the git and
    pr standards like that one day and night
    probably shaved off
    half of the pain points on this project
    you know of you know
    understanding changes setting the
    expectations we put a template in for
    pull requests you had to tick all the
    for that that are called out in the adr
    uh and it really helped now the other
    quick queens were aws resourcing names
    you know we wanted to make sure that we
    had consistency on the resource naming
    so that it let us do compliance checks
    and security consistently and things
    like that but we also put in place you
    know lambda language standards so you
    know we standardize for the platform on
    python unless we
    we couldn’t use it um
    for many different reasons and then we
    also set in standards around unit test
    expectations and the standard tooling
    that we expect or code to fall under
    when we started looking at some of the
    modules that we’re writing for terraform
    uh we also look to advocate you know
    sender alignments and things like that
    for releasing code
    uh the other one is that this one’s in
    progress at the moment is you know where
    we’re in we’re kind of stuck on a uh an
    older version of terraform we’re
    trending towards 1.0 or one.x and so
    what we’re trying to do is introduce
    some new standards in our environment to
    get us through that and provide a
    set of standards that we want to um
    align to over time to maintain it so it
    doesn’t fall a bit into disarray
    so i guess the results of this were
    pretty profound um
    so you know the main one is increased
    developer cadence um
    there’s a lot more time spent writing
    code than reviewing um different code
    and different uh changes being
    introduced uh you know we’re really able
    to shift left and implement a lot of
    automation into this so you know we’ve
    got ci now that
    enforces adrs and throws common errors
    to say hey you don’t align to this
    standard go check out the adr this is
    sort of our expectation
    we’re also able to take a lot of
    ambiguity out of what development
    standards were we’ve actually been able
    to pass this adr repo down to other
    teams and say we recommend that you do
    um you know and and they’re sort of
    following our lead uh people are a lot
    happier when things are written down
    um and it’s also when it comes to you
    know new people coming into the program
    it also
    like draws a line around clear
    expectations around where the quality
    is for introducing code into the
    environment uh and sort of
    you know having a checklist of have you
    done this have you done this are you
    jiras updated things like that um and
    also the big one who’s around the
    history as well is
    like there’s a lot of standards that you
    might encounter in your environments
    and you’re like what like why are we
    doing this
    and so
    this gives you a really clear history of
    of why things are as they are it might
    not always be great
    but it sort of made me think about like
    a mate of mine he worked for an isp and
    he decided he was going to like
    change the way the network worked
    because he thought it would be better
    and he did it and all hell broke loose
    and he ended up having to roll it back
    and when he was rolling it back he found
    a post
    that seemed exactly like his problem
    and it was by the guy
    who used to work there
    had um tried the same thing had the same
    problems posted on using that
    and rolled it back and he found it in
    the middle of his change so um you know
    writing stuff down understanding the
    history is a
    is a uh it’s a really good you know good
    thing overall for everybody especially
    at scale
    and as a result people are happier like
    this program
    has many different people permis
    different consultancies and the friction
    is actually pretty low for uh
    for for a project this size when it
    comes to
    collaboration and things so
    yeah it’s pretty good
    that’s it
    i’m just over but i tried my best so
    look thanks for listening i’m gonna stop
    sharing to see if there are any
    questions uh
    i think i’ve already done my bit about
    we’re hiring but
    you know uh do to reach out i’m gonna
    sharing great thanks kieran any
    questions for karen while we’re working
    out how to stop the screen sharing
    where’d it go
    i have a question
    um hey karen it’s uh jesse reynolds um
    oh excellent talk this sounds
    like it’s the answer to everything and
    i’m just wondering if there are any kind
    limitations into applicability that are
    sort of top of mind
    um it does apply to everything and you
    know what’s really interesting it’s it’s
    um i went out and looked at lots of
    different programs adrs and i would
    really recommend you do it i’ll tell you
    what a really good one is the home
    assistant project so if you do any home
    that’s actually a really mature project
    and i found that their adrs were very
    interesting um
    people focus on different things but
    um you know for us it was actually low
    hanging fruit was probably our biggest
    biggest challenge but
    um you do see some some teams implement
    at different levels um or they all they
    tell other teams look we cover this bit
    and go look at other teams adrs for
    everything else does that answer your
    yeah i mean uh
    i guess
    could you i suppose it’s
    it’s really applicable and this is
    probably gonna sound just a bit stupid
    but and obvious but i guess it’s really
    applicable to a development team’s
    methodologies right but i’m just
    wondering if you could extend it beyond
    that to
    more organizational
    um things you definitely could so one of
    the things that actually went in first
    is uh team principles and
    it was kind of one of i reflect more on
    the team principles for this project a
    lot more now
    team principles was probably
    one of the things we needed to get the
    team aligned on first so team principles
    in the context of this team is
    we make small
    small frequent changes or
    we um
    i’m trying to think of the other one we
    do everything as code
    and so you can have higher level things
    like that i do think they trend towards
    principles though which should
    absolutely be recorded as an adr i
    you could apply this to a business you
    definitely could
    whether or not that’s a good outcome or
    not i don’t know i i work with a bunch
    of business development dudes as you
    definitely do as well and
    they hate
    using jira
    like they’re a sales course team and um
    as much as i think they’re wrong and
    they should learn because it’s easy and
    it’s great for us to track things um
    using adrs
    for higher levels up the chain you may
    but i look forward to hearing how you go
    yeah thank you
    i’ve got a question if i may go for it
    um kiran so did you did you encounter
    any pushback from
    the developers and if you did what what
    are the main criticism of not using adr
    or what
    or did they come up with any good
    excuses for
    awarding to use an adr for example
    we’ve been pretty lucky um i i think
    that everybody acknowledged that there
    were problems and i think that a lot of
    people were
    happy to jump on board if it meant that
    their pull requests were going to get
    reviewed more consistently
    and i think also quick feedback like we
    had a lot of
    tooling problems to be honest the
    customer picked some pretty horrible
    tools which were now off
    which prevented us from sort of
    enforcing some of these checks but
    people have been pretty receptive
    one thing that i found we needed was to
    have our senior management our
    stakeholders kind of
    encourage people to get on board often
    what i say is a good idea isn’t always
    like accepted um until like influences
    above me saying this is a good idea you
    should do it um but
    you know hearts and minds i think
    internal brown bags on the on the
    approach um
    is probably one one thing i would have
    would recommend
    um and you know showing people how it
    can get them moving faster uh is the
    other one you’ll notice as well that
    that pit one is actually superseded we
    actually use it as a stepping stone to
    moving to conventional kit commits um so
    you don’t have to do big bang all of the
    time to get everybody on board
    so you know conventional commits are a
    predefined standard and it’s a lot
    easier for us to
    to go to that with an intermediary step
    where it could be easier for people to
    understand what we’re trying to achieve
    lucky for us terrifying enterprise
    doesn’t support conventional commits for
    releases so
    that was a logical
    step for us
    thanks got one other question that was
    asked in the chat uh from jim he said hi
    kieran do you think that the adr is
    suitable for small scale teams like less
    than ten devs and designers great
    presentation by the way thanks
    thank you um i think so um
    i think that
    as soon as you have more than
    you know a couple of people
    um it gets hard to scale
    you know some of the decisions and if
    people want to go on leave and things
    like that um i actually was looking
    around at other adr
    presentations and i saw this really
    great slide that i wanted to steal when
    i was like no i’ll make it all my own
    content and it was like one of these
    cavemen pointing to um
    to you know a cave uh to a painting on
    the wall and there were a whole bunch
    like there was like three other people
    on the ground watching him you know he
    was drawing some like sql databases and
    how they go together on you know and i
    was like it’s that tribal knowledge i
    the sooner you can probably
    write stuff down the better
    because it’s like three people sure four
    or five it gets challenging
    awesome all right any last questions for
    karen before we wrap this up
    all right sounds like we are done with
    questions well again karen thank you so
    much it was a really fantastic
    presentation and if you look in the chat
    uh things have gone off there so lots of
    uh lots of great conversation happening
    man that mythical man month is the thing
    that that spawned this conversation at
    this customer
    um the one we used to say was you know
    you can’t have um
    you can visit nine women can’t have a
    baby in one month it was the other one
    that used to uh to come up a lot so
    so yeah amen
    all right thank you very much kieran i
    am gonna
    kick you off and i’m just gonna add one
    last thing before we wrap up um so if
    you’ve been inspired by any of the talks
    tonight and you’re keen to maybe give
    your own presentation about adr or
    devsecops or any other devops related
    topic uh our next meetup is going to be
    in october so we’re always on the third
    thursday of the month i would love to
    have you speak if you’re keen to do that
    please hit me up either through meetup
    or via email and love to get you locked
    all right that’s it for tonight thank
    you so much for coming and i’ll see you
    online in october
    thank you
    cheers karen