What is ModelOps?

Overview

Leveraging maximum value from a machine learning (ML) model requires not just developing and deploying an effective model, but also implementing the necessary governance controls to align the model’s operations with the goals and priorities of the business.

This is the purpose of ModelOps. By providing a framework for systematically managing models, ModelOps helps bridge the gap between the technical facets of ML development on the one hand, and business needs on the other.

What is ModelOps?

At a high level, ModelOps can be defined as the set of practices and tools a business employs to maximize the value of its AI resources.

That said, defining ModelOps with precision can be challenging because different people use the term to refer to somewhat different concepts and practices. There are two main strands of thinking surrounding what ModelOps means.

The first is the ModelOps definition promoted by IBM researchers who coined the term “ModelOps” in a 2018 presentation. In their talk, they envisioned it as a way to drive “reusable, platform-independent and composable AI workflows.” Viewed from this perspective, ModelOps primarily focuses on bringing speed and efficiency to ML development and deployment workflows. It builds upon concepts born out of the DevOps movement, which aims to bring speed and efficiency to software development in general.

However, other ModelOps definitions, such as Gartner’s, focus more on the role of ModelOps in ML model governance and lifecycle management than on the efficiency of ML development. These definitions treat ModelOps as a way to align the technical aspects of ML models with business priorities by, for example, ensuring that models include the necessary governance controls to meet a business’s AI compliance obligations.

These varying definitions converge in the sense that they position ModelOps as a strategy organizations can use to manage their AI investments and optimize the value they create – so no matter how one chooses to define ModelOps, the ultimate goal is to help organizations make more effective use of AI and ML.

ModelOps vs. MLOps

While there is some inconsistency in definitions of ModelOps, all definitions distinguish ModelOps from MLOps.

MLOps, short for machine learning operations, is an approach to ML model development that breaks the process into consistent, repeatable stages – such as model design, development, testing, and deployment. The main purpose of MLOps is to make ML development reliable and predictable.

In contrast, ModelOps focuses on managing a business’s approach to AI and ML resources at a higher level. It includes considerations like model governance and lifecycle management, which are not common components of most organizations’ approach to MLOps.

It is possible to argue that ModelOps builds upon or is an extension of MLOps. This is because having a systematic, predictable approach to ML development (which is the main goal of MLOps) is a prerequisite for achieving ModelOps objectives like the ability to enforce effective governance controls over models. Alternatively, one could view ModelOps as a complement to MLOps, in the sense that ModelOps helps ensure not just that businesses can develop and deploy models effectively, but that they can also use them in a way that maximizes value for the organization.

Regardless of how one chooses to think about the relationship between ModelOps and MLOps, what really matters is understanding that MLOps is a narrower process that centers on ML design, development, and deployment. In contrast, ModelOps addresses AI and ML strategy at an organization-wide level.

Components of ModelOps

Due to the varying definitions of ModelOps, different opinions exist on precisely which processes and types of tools organizations need to implement to adopt ModelOps. In general, however, key ModelOps components include:

• ML development: In machine learning, development is the process of creating and refining models.
• Data management: Ensuring that models are trained on the right data, and that the data is of appropriate quality, is key for aligning ML initiatives with business goals.
• Training: Training exposes models to data that helps them identify patterns and “learn” how to make decisions.
• Testing and performance analysis: By analyzing how well models excel at performing a given task, organizations can determine their fitness for supporting intended use cases.
• Governance: Governance controls in models reduce organizational risks by ensuring that models are developed, trained, and deployed in ways that mitigate business risks (such as biased decision-making).
• Versioning: Model version controls make it possible to manage changes to models over time.
• Scaling: Scaling ensures that models can support growing numbers of users or data volumes over time.

These various components require the participation of multiple types of stakeholders in the ModelOps process. In addition to ML developers, who are responsible for developing and deploying models, representatives from across the business – such as risk analysts, who can establish which governance controls should be in place, and executives, who understand which high-level business priorities models should support – have a role to play in ModelOps.

The Importance of ModelOps

ModelOps matters because investing in AI and ML technology is often costly, complex, and risky, and there is no guarantee that these investments will ultimately create business value. For example, an organization could build and deploy an expensive AI model to automate a critical business process only to discover that the model makes decisions that lead to compliance risks. Or, it might find that it is not updating the model frequently enough based on novel training data to keep it in sync with evolving business priorities.

ModelOps helps mitigate risks like these by providing a structured, consistent way for businesses to manage their various AI and ML investments. It helps ensure that projects like the development of a new model are conceived and implemented in alignment with business goals, and it encourages organizations to take a proactive stance in managing the risks that may arise from ML investments.

ModelOps Examples

As examples of how organizations apply ModelOps in practice, consider the following two sample scenarios.

#1. Ensuring fairness in finance

Banks and other finance organizations have ethical and regulatory obligations to ensure that they treat customers fairly. This includes avoiding making biased decisions, such as refusing a loan based on where an applicant lives or his or her age. To meet these obligations when using ML models to process loan applications, banks must ensure that the models include controls designed to mitigate the risk of bias.

A ModelOps approach can help in this scenario by looping risk management and compliance teams into the ML design, development, and deployment process. By allowing these stakeholders to provide input to ML developers about which anti-bias controls should be in place within models, organizations can get ahead of potential business risks that may arise when depending on a model to automate decision-making.

#2. Mitigating cybersecurity risks for AI-assisted coding

AI-assisted software development tools can substantially speed up the development process. However, they also introduce novel risks, such as the possibility that a model will “hallucinate” the name of a software package or dependency. When this happens, threat actors can potentially exploit the issue to launch a package hallucination attack.

To mitigate this risk, an organization could leverage ModelOps to implement AI security controls capable of identifying fictitious packages within AI-generated code. For example, developers could create an output filter that checks package names within the code against lists of actual packages to catch instances where the model has hallucinated a dependency. This safeguard would significantly reduce the risk that AI-generated code could become a vector for software supply chain attacks that exploit hallucinated packages.

Building a ModelOps Platform

Because ModelOps remains a relatively new concept, the software ecosystem surrounding it is still developing. A handful of platforms exist that are designed specifically to help organizations implement ModelOps, and adopting one of them is one way to implement ModelOps.

However, teams can also adopt ModelOps by integrating governance and lifecycle management capabilities into their existing MLOps tools and workflows. For instance, they could modify processes related to model design to ensure that stakeholders other than developers are part of the process, and they could add steps during the development process for implementing governance controls.

Integrating ModelOps into existing ML workflows may work better for organizations that already have a well-established set of ML development tools and processes, and that want to avoid having to rebuild their MLOps tooling and processes from scratch to accommodate ModelOps.

ModelOps with JFrog

By providing a unified experience for building, training, securing, serving, and monitoring ML models, JFrog delivers the foundational capabilities necessary to adopt a holistic, ModelOps-based approach to managing ML investments. See for yourself by getting started for free.