Making the Most of Automation in DevOps

In the context of DevOps, automation is key for making processes scalable, consistent and secure. That’s why DevOps teams should constantly be on the lookout for ways that they can automate workflows – and to make the automations that they are already using even more efficient.

Below, we explain the role automation plays in DevOps, which benefits automation brings to DevOps, examples of DevOps automation and best practices for getting started with the automation of DevOps processes.

What is DevOps automation?

DevOps automation is the use of tools to automate any type of workflow or process that a DevOps team needs to perform. In other words, when you take a DevOps process – such as software testing or deployment – and use a tool to automate the process in part or in full, you’re using DevOps automation.

DevOps automation examples

To contextualize what DevOps automation means in practice, let’s look at some common examples of DevOps workflows that can be automated. This list is not comprehensive; indeed, virtually every process that DevOps team members perform has the potential to be automated. However, the processes described below are among the most common and important DevOps workflows. They are therefore good places to begin for organizations looking to get started with DevOps automation.

Infrastructure provisioning

Infrastructure provisioning is the process of configuring the infrastructure that hosts DevOps workloads. For example, if you run workloads on cloud-based VM instances, the provisioning process would entail deciding which types of instances to use, then configuring them with your desired operating system, packages, libraries and so on.

You could complete this process manually. Engineers could launch each VM instance by hand, then configure it as needed. But a more efficient and scalable approach would be to apply DevOps automation to infrastructure provisioning.

You can do so using Infrastructure-as-Code (IaC) tools, like Terraform. IaC tools let DevOps teams define configurations for infrastructure, then apply those configurations automatically across their environments. For example, you could use CloudFormation, the IaC tool built into the AWS cloud, to configure AWS EC2 VM instances automatically based on IaC code like the following (which is part of a template that configures IP and port settings):

“Resources” : {

    “EC2Instance” : {

      “Type” : “AWS::EC2::Instance”,

      “Properties” : {

        “UserData” : { “Fn::Base64” : { “Fn::Join” : [ “”, [ “IPAddress=”, {“Ref” : “IPAddress”}]]}},

        “InstanceType” : { “Ref” : “InstanceType” },

        “SecurityGroups” : [ { “Ref” : “InstanceSecurityGroup” } ],

        “KeyName” : { “Ref” : “KeyName” },

        “ImageId” : { “Fn::FindInMap” : [ “AWSRegionArch2AMI”, { “Ref” : “AWS::Region” },

                          { “Fn::FindInMap” : [ “AWSInstanceType2Arch”, { “Ref” : “InstanceType” }, “Arch” ]

} ] }

      }

    },

 

    “InstanceSecurityGroup” : {

      “Type” : “AWS::EC2::SecurityGroup”,

      “Properties” : {

        “GroupDescription” : “Enable SSH access”,

        “SecurityGroupIngress” : 

          [ { “IpProtocol” : “tcp”, “FromPort” : “22”, “ToPort” : “22”, “CidrIp” : { “Ref” : “SSHLocation”} }]

      }

    },

 

    “IPAddress” : {

      “Type” : “AWS::EC2::EIP”

    },

 

    “IPAssoc” : {

      “Type” : “AWS::EC2::EIPAssociation”,

      “Properties” : {

        “InstanceId” : { “Ref” : “EC2Instance” },

        “EIP” : { “Ref” : “IPAddress” }

      }

    }

  }

 

Using this approach, you avoid having to log into each VM instance and configure its network settings manually.

Software testing

Software testing is the process of evaluating software to determine whether it meets usability, performance, security and other requirements. DevOps teams usually run software tests after they’ve built a new application release but before deploying it into production. If the application passes necessary tests, it’s deployed; if not, bugs are fixed and it’s rebuilt and re-tested.

DevOps teams can perform software tests manually by evaluating each application release by hand. But a much more efficient approach is to automate software tests using an automated testing framework. Automated testing frameworks allow engineers to write scripts that define which aspects of an application to test. Then, the scripts can be executed automatically, and the results can be evaluated to determine whether the application meets the requirements necessary to be deployed.

There are a variety of test automation frameworks available. They vary with regard to which programming languages they support and which types of tests they can run. If you’re getting started with DevOps software testing automation, however, a good framework to consider is Selenium, which is open source and supports a variety of testing use cases.

With Selenium, you can write a test like the following to evaluate your application automatically:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.chrome.service import Service as ChromeService

from webdriver_manager.chrome import ChromeDriverManager

 

def test_eight_components():

driver = webdriver.Chrome(service=ChromeService(executable_path=ChromeDriverManager().install()))

driver.get(“https://www.selenium.dev/selenium/web/web-form.html”)

 

title = driver.title

assert title == “Web form”

 

driver.implicitly_wait(0.5)

 

text_box = driver.find_element(by=By.NAME, value=”my-text”)

submit_button = driver.find_element(by=By.CSS_SELECTOR, value=”button”)

 

text_box.send_keys(“Selenium”)

submit_button.click()

 

message = driver.find_element(by=By.ID, value=”message”)

value = message.text

assert value == “Received!”

 

driver.quit()

 

As you can see, this test is a Python script that imports the webdriver package. Webdriver is a tool that runs scripted interactions with a Web browser, making this a useful script for DevOps teams that want to test Web apps automatically.

CI/CD

The processes necessary to move software down the CI/CD pipeline – such as pushing source code into build tools or beginning test automation processes – can be performed using DevOps pipeline automation.

For example, here is a simple configuration file for JFrog Pipelines, a CI/CD automation tool, that defines a series of pipeline steps (named p1_s1, p1_s2 and so on in the example) to perform within a CI/CD pipeline:

pipelines:

  – name: my_first_pipeline

steps:

– name: p1_s1

type: Bash

configuration:

inputResources:

# Sets up step to be triggered when there are commit events to myFirstRepo

– name: myFirstRepo

execution:

onExecute:

# Data from input resources is available as env variables in the step

– echo $res_myFirstRepo_commitSha

# The next two commands add variables to run state, which is available to all downstream steps in this run

# Run state documentation: https://www.jfrog.com/confluence/display/JFROG/Creating+Stateful+Pipelines#CreatingStatefulPipelines-RunState

– add_run_variables current_runid=$run_id

– add_run_variables commitSha=$res_myFirstRepo_commitSha

# This variable is written to pipeline state in p1_s3.

# So this will be empty during first run and will be set to prior run number in subsequent runs

# Pipeline state documentation: https://www.jfrog.com/confluence/display/JFROG/Creating+Stateful+Pipelines#CreatingStatefulPipelines-PipelineState

– echo “Previous run ID is $prev_runid”

 

– name: p1_s2

type: Bash

configuration:

inputSteps:

– name: p1_s1

execution:

onExecute:

# Demonstrates the availability of an env variable written to run state during p1_s1

– echo $current_runid

 

– name: p1_s3

type: Bash

configuration:

inputSteps:

– name: p1_s2

outputResources:

– name: myPropertyBag

execution:

onExecute:

– echo $current_runid

# Writes current run number to pipeline state

– add_pipeline_variables prev_runid=$run_id

# Uses an utility function to update the output resource with the commitSha that triggered this run

# Dependent pipelines can be configured to trigger when this resource is updated

# Utility functions documentation: https://www.jfrog.com/confluence/display/JFROG/Pipelines+Utility+Functions

– write_output myPropertyBag commitSha=$commitSha runID=$current_runid

 

By executing a series of steps like these using a tool such as JFrog Pipelines, DevOps teams can automate processes that they would otherwise have to perform manually, creating the potential for inefficiency and delays.

The benefits of DevOps automation

DevOps refers to a software delivery strategy that integrates development and IT operations. 

While it is possible to deliver a DevOps approach without automation, it’s difficult to perform DevOps processes efficiently or consistently without itshelp. It’s only through automation that DevOps teams can achieve benefits such as:

• The ability to accomplish more with fewer DevOps team members.
• The reduction in the amount of “toil” – meaning tedious, manual tasks – that DevOps engineers have to perform.
• The achievement of consistent, reliably repeatable processes (because automation tools will always yield the same way as long as they are configured in the same way).
• The ability to keep processes consistent no matter who is performing them.
• Reduced risk that a staff member’s departure or promotion will disrupt DevOps operations because no one else knows how to perform the work of that person. When DevOps is automated, processes can continue regardless of which engineers are on the team.

DevOps automation tools

Broadly speaking, DevOps automation tools fall into two main categories.

The first consists of tools designed to automate a particular type of DevOps process, but not the entire DevOps software delivery operation. For example, you could use an IaC tool like Terraform to automate infrastructure provisioning, or a test automation framework to execute automated software tests. But because each of these tools addresses a specific and narrow need, you’d need to deploy multiple tools to automate most or all aspects of DevOps.

The other category of DevOps tools consists of DevOps platforms that are designed to enable end-to-end DevOps automation, or something close to it. A DevOps platform includes tooling to automate all core DevOps processes and workflows – such as CI/CD automation, security scanning, software delivery and artifact management. A DevOps platform can also tie together the additional tools and processes that exist across your supply chain to help disparate automation tools work seamlessly together.

Best practices for automating DevOps

Every organization is different, and there is no one-size-fits-all approach to DevOps automation. But in general, core best practices for making the most of DevOps automation include:

• Start with low-hanging fruit: Get started with DevOps automation by identifying processes that are the easiest to automate and expand from there by automating additional processes. This is important because you don’t want to lose support or buy-in for a DevOps automation initiative by biting off more than you can chew as you get started.
• Consider partial automation: DevOps automation doesn’t have to be all-or-nothing. In cases where it is too difficult to automate a process fully, you can automate it partially. For example, you could use pipeline automation to prep code to move down the pipeline automatically, but still require an engineer to sign off on changes manually before they are implemented. That way, you get the efficiency of automation as well as the assurance of having a human’s oversight.
• Continuously improve: Automation shouldn’t be a one-and-done affair. In addition to looking for manual processes that you can automate, look for ways to increase the efficiency or scalability of the automations you already have in place, in order to get even more value out of them. For instance, if you’re automating some of your software tests today, consider how you can automate even more.
• Track automation metrics: Ensure you can quantify how much value DevOps automation is creating – and find situations where it’s falling short – by tracking metrics related to automation. A good starting point here are the often cited DORA metrics: Deployment frequency, lead time for change, time to recovery and change failure rate. You can also track how much time it takes to deploy an application automatically versus manually, or how much you reduce the time required to find and fix a bug when you use automated testing instead of manual tests.

Implementing DevOps automations can take time, and the payoff may not be immediately clear. It’s not until your team has worked through many software release cycles, and engineers gain a sense of how much time they’re saving by minimizing manual processes, that the full benefits of DevOps automation start to feel real. For this reason, it’s important not to dismiss automation initiatives as a failure just because they don’t appear to effect transformative change overnight.