DevOps: 8 Reasons for DevOps to Use a Binary Repository Manager in the Cloud
INTRODUCTION
For today’s businesses, cloud computing is now the default model for applications, storage, and compute. The cloud enables organizations to innovate fast, and to make bold digital bets at minimum cost and risk while being able to scale elastically. Cloud is the key to business modernization.
At the same time, developing software has transformed from an activity of small, tightly-knit groups to one performed by many interdisciplinary teams, oftentimes sprawled across the globe. An iterative and incremental approach to software development is the norm at the modern enterprise, which is now cloud-enabled, software-driven, and virtual.
Organizations that once issued a significant new release every year or twice a year now more commonly release a new version to customers monthly or weekly. Many enterprises now deploy updates to production multiple times a day.
As releases become more frequent, expectations from end-users keep pace — customers demand frequent software updates to fix bugs, remediate vulnerabilities, and provide new features.
This is why enterprises are increasingly focussed on DevOps — the tools and best practices that enable creating and deploying more releases, more frequently, and at high quality into production environments. To succeed, DevOps practices aim to reduce friction at all points in the software development lifecycle, or SDLC, and automate workflows (through CI and CD servers and more) to greatly accelerate release velocity.
We’ll explain why a binaries-centric approach to DevOps is the best way to be successful. Then we’ll show the top 8 ways that Artifactory, the binary repository manager at the heart of the JFrog DevOps Platform, gives SREs and DevOps teams the tools to execute it through efficient management of an ever-growing matrix of binaries, environments, and geographically distributed sites.
Along with the supporting solutions in the JFrog Platform, Artifactory can help ensure a smooth and secure application development workflow from end to end, as well as enable the hybrid and multi-cloud DevOps architectures you need to speed innovation, stay best-in-class, and reduce the risk of cloud vendor lock-in.
DEVOPS SUCCESS LIVES IN YOUR BINARIES
You’re probably asking, “How can DevOps be about binaries? Isn’t software development all about code?” Of course it is, and developers need to be able to focus on writing innovative, bug-free source code that creates value for customers. To most developers, source code is what matters.
But as important as coding is, source code isn’t what your customers use — they run an application, which is a composite of many binaries from many sources. The quality of the software you deliver to your customers depends on the quality of the binaries that go into it.
Software development is about code, and quality code comes from smart people who know how to write it.
Software delivery is about binaries — ensuring quality builds and getting them swiftly into your customers’ hands. Quality binaries come from smart systems that know how to manage and distribute them.
We can see this in the standard diagram of the DevOps loop for a best practice SDLC:
What’s clear? Source code plays a foundational, yet proportionally small part of the total SDLC. Most processes in the DevOps loop don’t deal directly with code at all, but instead center on the binaries that are built, tested, deployed, and run.
ARTIFACTS: WHERE DEVELOPMENT MEETS DELIVERY
Artifacts are the building blocks of software, from your packages and configuration files to the binaries that are the deployable runtime components of your application. Every delivered application is composed of a set of artifacts that connect to each other.
Your binaries are what travel through your software delivery pipeline(s) from end to end as they pass checkpoints of testing, validation, and security before final deployment to production.
While every application begins with source code, each source file both produces and relies on many artifacts (binaries and dependencies) through many build and test steps before resulting in a deliverable piece of software.
DevOps aims to speed the delivery of quality software releases. It does this by removing friction in the software delivery flow, which is most often found where software lifecycle processes meet. The consistency and certainty this brings enables automation for fast delivery.
Your ecosystem of artifacts — deployable binaries and their associated files — occupy the space where development meets delivery. The JFrog Platform’s ability to control, ease, and automate the flow of your binaries is the single most powerful key to successful DevOps.
FROM CLOUD NATIVE TO CLOUD NIMBLE
Systems architects must make sure their systems are cloud native, with applications that are optimized for scalable cloud technology infrastructure.
What’s become clear is that cloud native solutions were only the start for running agile, scalable operations in cloud and hybrid environments. Forward-looking DevOps engineers must also seek solutions that are cloud nimble: operationally consistent platforms that empower you to choose, and enable you to spread critical workloads across multiple domains both public and private.
Hybrid cloud and multi-cloud are fast becoming the most popular IT architectures for cloud service delivery, as enterprises seek even greater agility for their virtual operations through divestment from a single cloud provider.
Cloud nimble enterprises demand the ability to run in any cloud, anywhere, so they can:
- Match cloud choice to best-in-class services for given tasks
- Allocate workloads to the lowest-cost provider for each
- Shift workloads for load balancing or to adjusting to demand pricing
- Keep data and compute services in geo proximity
- Maintain redundancy to help ensure five nines availability
- Employ a hybrid strategy to comply with security and/or regulatory requirements
A cloud nimble enterprise is better able to adapt to change without losing balance or speed.
Maintain a Consistent Security Posture
Protect your software supply chain with consistent permissions and monitoring across cloud domains.
Same Feature Set Everywhere
Interoperate across clouds with a consistent operating model through the same best DevOps practices..
Distribute Across Best-in-Class Services
Choose best-fit cloud providers for different segments of your SDLC.
Multi-Domain Automation
With reliable geo-replication and interoperability, securely deliver payloads from one cloud to another.
Transparent Cloud Migration
Maintain operational continuity migrating packages, builds, and metadata from one instance to another.
Choose Your Cloud For SaaS
Host SaaS accounts on multiple cloud providers for data proximity to the cloud ecosystems your teams use.
A SINGLE SOURCE OF TRUTH FOR DEVOPS
Adhering to a single set of workflow best practices becomes progressively harder as an organization adds languages and technologies to its tool chain.
As the power behind the JFrog Platform, Artifactory is a universal repository manager for all of your artifacts: dependencies, binaries, and configuration files. With native support for over 30 package types (including generic repositories), it’s your single source for storing, securing, and tracing the entire artifact ecosystem of your development organization. Developers can store and retrieve their packages or images in Artifactory repositories through the same package management services they use every day.
Artifactory’s universal binaries management serves all the developers in your organization. Whether they develop programs in Java, JavaScript, Python, Go, C++, or C#, Swift, Rust, or more, Artifactory is the central home for everyone’s packages and builds.
“We haven’t hit a kind of thing we can’t push into Artifactory yet. Our technology choices aren’t ever limited by Artifactory.” – Graham Bucknell, CI/CD Lead at Monster
When all of your binaries are governed through this one common tool, your entire enterprise can align onto the same SDLC workflows and best practices that will assure quality and accelerate release velocity. This is why Artifactory is the central component of a fully-automated software distribution pipeline that powers the JFrog DevOps Platform.
TRACK A SOFTWARE BILL OF MATERIALS (SBOM)
Making builds is easy. Knowing about everything you’ve built is hard. Especially when you make many new builds every day or every hour.
Artifactory stores new metadata — what we call “build info” — with every build you make, linking to the package metadata of your open source and proprietary dependencies along with build artifacts and environment settings. With detailed build info, you can trace every build back to where it came from and out to every place it’s been staged for service.
Your build info is the basis of a Software Bill of Materials (SBOM) — a machine-readable inventory detailing all the items included in an application and their origin — for every release put into production or delivered to a customer.
As a growing number of governments and regulated industries require an SBOM to help combat cyberattacks, the JFrog Platform is your turnkey solution for compliance.
Artifactory’s build info helps ensure that you are never in the dark about where a build came from, how it was created, or where it was deployed.
“A big bonus for our developers is build metadata – when log4j hit, it was the easiest thing to generate a report of what apps had that vulnerable dependency, fix it, and we were good to go.” – Caio Trevisan, DevOps Service Owner, Bendigo and Adelaide Bank
ARTIFACTORY QUERY LANGUAGE
Artifactory Query Language (AQL) empowers DevOps engineers to quickly identify relevant packages and builds within growing repositories. They can employ AQL in their CI/CD servers to automate procedures that would otherwise require human intervention, or in analytics to improve workflow performance.
CACHE REMOTE REPOSITORIES BY PROXY
The open source software (OSS) dependencies drawn from remote resource repositories such as npm, Maven, Conan, and others can easily be the more significant portion of code in today’s applications. Ensuring site reliability and speedy access is a vital key to maintaining release velocity, but can face several challenges:
Network latency
Inherent latency from physical distance – a remote site can be on the other side of the globe.
Heavy loads
Delays produced by heavy demand for a service.
Poor connectivity
Network outages, jitter, poor bandwidth, unstable connections.
Site downtime
The remote site where your dependencies are stored may suffer a service disruption from failure, attack, or loss of service.
A remote repository in Artifactory is a local proxy that caches your remote resource dependencies on the same infrastructure where you keep your local repositories. Developers never access a remote resource directly, but build using an on-demand copy of the dependency in Artifactory. When Artifactory is hosted on the same cloud environment as your compute, your builds can run at top speed.
“Imagine a couple thousand developers going through a single proxy to be able to access, say, Maven Central. That unnecessary traffic is now negated because of Artifactory. We’re able to build our solutions without going outside of our firewall.”
— Mike Smith, Technology Architect at Kroger
Eliminating network latencies inherent in physical distance or across cloud service regions, helps keep builds running as fast as possible. The proxy also protects against disruption if the remote site itself is unavailable.
Equally important, the cache in Artifactory helps maintain your dependencies as immutable versions — after the first on-demand pull from the remote resource, the package in the cache never changes. This guards against any force-push overwrite into a remote resource (possibly with malicious intent) and guarantees deterministic builds.
VIRTUAL REPOSITORIES
A virtual repository encapsulates any number of local and remote repositories, and represents them as a unified repository accessed from a single URL. It gives you a way to manage which repositories are accessed by developers since you have the freedom to mix, match and modify the actual repositories included within it.
You can also optimize artifact resolution by defining the underlying repository order so that Artifactory will first look through local repositories, then remote repository caches, and only then will Artifactory request the artifact directly from the remote resource. For the developer it’s simple. Just request the package, and Artifactory will safely and optimally access it according to your organization’s policies.
BUILD ONCE AND PROMOTE EVERYWHERE
Under the agile methodology of continuous integration, every new software version must pass several quality gates in an SDLC. A candidate version is promoted successively to different teams for integration, testing, and staging before it is released for general use.
But what passes through these gates makes the difference between a speedy or a plodding path to release. When version promotion is done through source code, each receiving team must perform its own deterministic build of the code in their own runtime environment, potentially leading to a different set of binaries. This lacks certainty that the software evaluated at each stage is exactly the same as the last.
Artifactory’s binary repository management offers a more consistent and reliable method that carries a single, immutable binary through the entire SDLC. With a repository for each SDLC stage, a build with its metadata can be promoted in the JFrog Platform simply by shifting it to the next repo in sequence.
In this “build once and promote” approach, the same build is evaluated at every stage, assuring absolute consistency through the DevOps pipeline.
Once free of having to perform their own builds or manage build environments, teams can apply the hours they recover to conducting more exhaustive tests and delivering feedback more quickly.
At each stage, teams can add additional metadata about stability, security, and more to the candidate’s build info. By passing critical metadata learned from one stage that can be used by the next, the JFrog Platform accumulates a comprehensive record about the lineage of every build put into production.
CHECKSUM-BASED STORAGE
The JFrog Platform optimizes storage by ensuring that any binary is only stored once on the file system. When a binary is stored in a local or remote repository, Artifactory calculates a unique checksum (both MD5 and SHA1 are supported) of the file and renames the file to its checksum. Repositories hold only references to files and their metadata, so when a binary is copied or promoted to another repository only the references are changed. The physical file is never duplicated, and its checksum can be used to verify the binary’s integrity.
ACCELERATE CLOUD NATIVE DEVELOPMENT
Increasingly, software development is cloud native development: applications written to effectively utilize cloud technology infrastructure, and enable the inherent best characteristics of running in the cloud.
This means producing container-based microservices, relying on established cloud native standards like OCI and cloud native tools like Docker and Kubernetes.
Docker repositories in the JFrog Platform fully support all Docker Registry APIs, so they can function natively with the Docker CLI. WIth local repositories, you can maintain as many private Docker registries in Artifactory as you need, to distribute and share container images within your organization.
Artifactory reveals the layers within every Docker or OCI image that compose it, and links the metadata for a fully traceable path back to the origins of all its parts.
When combined with the JFrog Platform’s fine-grained access control, you can maintain secure, private Docker repositories that exceed the security offered by Docker Trusted Registry. Using Artifactory’s local repositories instead of private repositories on Docker Hub avoids all internet connectivity concerns, providing reliable and consistent access to images.
Artifactory also supports Helm chart repositories, so you can also manage your Kubernetes orchestration manifests alongside your Docker images. In this way, the JFrog Platform can serve as your comprehensive Kubernetes registry, a central, traceable home for everything deployed to your clusters.
NO LIMITS WITH DOCKER HUB
JFrog’s partnership with Docker exempts JFrog Cloud users of Artifactory from Docker Hub’s image pull rate limits. Ordinarily, anonymous free users are limited to 100 pulls per six hours, and authenticated free users to 200 pulls per six hours. But Docker Hub waives these limits when the image pull request is from a JFrog Cloud account.
By setting up an Artifactory remote repository to proxy Docker Hub, cloud users gain unlimited, high-performant access to Docker Hub and to Docker Official Images, simplifying cloud native application development.
By leveraging JFrog Xray, developers can also gain continuous, comprehensive vulnerabilities scanning of the images they pull from Docker Hub.
PROTECT YOUR SOFTWARE SUPPLY CHAIN
When you run an enterprise, protecting your software supply chain — your application’s ingredients — from mistakes and attacks is critical to protecting what’s most precious: your business.
Supply chain attacks like Tyupkin (2014), NotPetya (2017), Operation ShadowHammer (2019), and SolarWinds (2020) have spurred a heightened awareness and inspired the World Economic Forum to rank them among the top cybersecurity challenges.
As the single source of truth for all of your binaries, a binary repository manager makes an attractive target for a supply chain cyberattack. That’s why security is a top priority for the JFrog Platform.
Authentication
All services of the JFrog Platform require authentication through secure credentials such as a password or access token.
Checksum Verification
Every artifact’s computed checksum is integral to how it is stored in Artifactory repositories, and is used to verify its integrity.
Permissions Management
With the JFrog Platform’s fine-grained access control, administrators can ensure that developers and groups can access only the repositories through the CRUD operations they are authorized to.
SSO
Support for LDAP, SAML, OAuth and SCIM protocols empower admins to integrate with Single Sign-On services such as Active Directory, Crowd and others for secure, organization-wide credentials management.
JFrog Artifactory’s fine-grained permissions and its integration with single sign-on through multiple protocols empower organizations to maintain the same high-quality protection of their software supply chain across multiple cloud domains.
VULNERABILITY SCANNING
In a recent survey, only 18% of organizations reported being extremely confident in their open source components, which can typically be 60-80% of an application’s code. Nearly twice as many were either not very or not at all confident.
The companion security solution JFrog Xray performs deep recursive scanning of binaries in your JFrog Platform repositories to identify all open source components that have known vulnerabilities. As part of the JFrog Platform, Xray is tightly integrated with Artifactory, maintaining additional security metadata and providing impact analysis so you can quickly remediate all binaries where a vulnerable dependency has been used.
With Xray, you can prevent high-risk builds from being deployed into production, and enable recall of builds with risks that are newly discovered.
In addition, Xray can also monitor the license types of your open source components, to alert you of all that are out of compliance with your organization’s policies.
With JFrog Xray, you can operate with consistent monitoring, reporting, and remediation of vulnerabilities and compliance scanning across all cloud environments.
CONNECT TO YOUR UNIVERSE OF TOOLS
For developers, DevOps is a set of pipelines flowing in one direction toward a production-quality release.
But for the site reliability engineer, DevOps is an organizational fabric, weaving tools and procedures together to create a strong, resilient infrastructure.
Integrating a complex DevOps tool stack well is critical for reliable operations, and keeping this ecosystem running can consume far too much engineering time.
It starts with the tools that automates your builds: your CI/CD servers must integrate smoothly with the systems where they’ll be stored. The JFrog Platform can connect to your choice for CI/CD automation through:
CLI
A command line interface (CLI) tool enables developers to store and retrieve binaries and metadata to and from repositories through a command window, shell script, or CI pipeline.
REST APIs
Developers can manage binary repositories using REST commands through curl or custom DevOps tools.
Webhooks
Trigger an action in another service in response to an event in Artifactory, either to notify users of the event or initiate an automated flow.
Build Integrations
Ready-made plugins and extensions for major CI tools such as Jenkins, Circle CI, TeamCity, Bitbucket, Pipelines, and Azure DevOps accelerate integration with your CI pipelines.
Of course, that’s only the start of your integration needs. Tools for automated testing, collaboration, ITSM, observability and analytics are all part of your DevOps infrastructure fabric. You’ll want to push the information these tools produce into your repositories, and connect that information to other tools, too.
With a large family of JFrog partner integrations with top industry providers, the JFrog Platform can help quickly build a strong, tight weave of your DevOps tool stack through this single source of truth.
NATURALLY INTEGRATED CI/CD
JFrog Pipelines CI/CD is the fastest way to build your CI/CD ecosystem. As part of the JFrog DevOps Platform, Pipelines is naturally joined with Artifactory. Using Pipelines’ declarative pre-built steps, you can focus on what goes into your repositories, not how to get it there.
With the remaining components of the JFrog Platform, Xray, and Distribution, the unified tools at the heart of your DevOps universe are pre-integrated at installation.
Pipelines also provides out-of-the-box integrations for the most popular tools in your ecosystem, including GitHub, GitLab, BitBucket, Slack, Jira, AWS, GCP, Docker, Kubernetes, and many more. Integrating most of these services with Pipelines is as simple as entering a URL endpoint and user credentials.
SCALE TO INFINITY
For the modern enterprise, software development is now a highly collaborative endeavor of packages shared by intersecting teams across multiple sites and cloud regions spread across the globe.
Even small teams should expect to grow, and the best practices of DevOps have to stretch seamlessly along with them.
These binaries-centric best practices for DevOps – a single source of truth, metadata, build promotion, security, etc. – have been proven to scale smoothly. Every day, they enable JFrog Platform users to release quality software at speed whether it’s developed by 5 people in one room, or 500 around the globe.
As your mission-critical tool, you’ll need to ensure continuous and responsive access even as your department grows.
On-Premises High Availability
Self-manage as a BYOL installation in clusters in your own cloud account, or on servers in your own secure datacenter on-premises.
A High Availability configuration of replicated nodes in your cluster helps spread the load to accommodate large load bursts and ensures there is no single point of failure. This maximizes your uptime, even during system updates, up to “five nines” level of availability.
Cloud (SaaS)
Subscribe to a JFrog cloud managed service hosted in the major cloud provider of your choice (AWS, GCP, or Azure), and leverage the “any time, from anywhere” availability and elasticity of the cloud.
Employ a multicloud strategy by maintaining multiple SaaS accounts hosted on different cloud providers. This helps avoid vendor lock-in, and enables allocating workloads to the most cost-effective provider.
Hybrid
A hybrid strategy empowers you to employ the scalability of the cloud for your dynamic workloads, while keeping sensitive workloads in clusters within your secure on-premises data center.
The JFrog Platform’s “same here, same there” promise of feature equivalency in every environment means you can divide your workloads between cloud and on-prem however you need.
Multi-Site Geo-Replication
The JFrog Platform enables multi-site geo-replication through several push/pull replication topology options, or bidirectional federation of repositories.
JFrog’s unique set of multi-site capabilities ensure locality in any global cloud topology. This empowers geographically distributed teams to work on the same artifacts (binaries and their metadata), with minimal latency so that every build, at every site, in every cloud region, can complete fast, without fail.
DISTRIBUTE TO THE FARTHEST EDGES
Once you have a fully validated release, where does it need to go? Enterprises must distribute software and updates to a growing number of global endpoints, for delivery to the clusters, devices, and desktops where it will benefit users.
As part of the JFrog DevOps Platform, JFrog Distribution empowers DevOps teams to efficiently package release bundles (binaries, artifacts, and metadata) from Artifactory and automate trusted software delivery to dozens or thousands of remote sites across the globe. Using JFrog Edge Nodes in multiple cloud regions — cost-effective read-only Artifactory instances — you can transmit signed, immutable releases through a private data network that are verified at each destination, while maintaining fine-grained access controls.
ARTIFACTORY PUTS YOUR BINARIES TO WORK
The evidence is clear: the key to successful DevOps — minimizing or eliminating the friction in your pipelines — lies in effective management of your binaries through your entire SDLC. A binary repository manager is your key agent of digital transformation.
The 8 reasons we’ve given here are the essential practices of this binaries-centered approach to DevOps. They’re also the core values that the JFrog Platform is built around, to better assure your success.
Artifactory’s design and features empower developers with the best practices of DevOps that reduce friction in key lifecycle stages, and accelerate software delivery.
The effectiveness of this binaries-centered approach to agile digital transformation is proven by thousands of JFrog customers every day, every hour.
JFROG PLATFORM
Artifactory is the core component that powers the JFrog DevOps Platform, a comprehensive end-to-end platform solution for one-stop DevOps.
The cloud native JFrog Platform provides the full market-leading feature set in every operating environment, enabling you to operate your SDLC across clouds with the same best practices and consistent security posture for DevOps success.
The JFrog Platform is available at multiple subscription levels for self-hosted installations or as a managed service in the cloud. Each level includes a set of JFrog components appropriate to your organization’s needs. Whether you’re a small development team for internal applications or a global enterprise delivering secured software to the edge, JFrog has the right set of solutions for you.
