ML Model Registry: The Ultimate Guide

Imagine you are the only data scientist on your team, you start working on a machine learning project and perform a series of experiments that produce various ML models (and artifacts) that you “track” through non-standard naming conventions. Since the naming conventions you used for your model files were unclear, it took you a while to find the most optimal model you trained. 

When you finally did, you decided to either hand the raw model artifacts over to the operations team or, worse, deploy it yourself. You now hand over the raw artifacts to the operations team, and they reverted for more information like: 

Because all you did was build the model and hand it off, it was perhaps difficult to collaborate with them for a successful deployment.

Now imagine another scenario. 

Your company is planning on shipping out more ml-based products/features. A data scientist, engineer, and maybe even a product manager joins the team. When you were working alone, although your model was a pain to deploy (if it was deployed), your workflow worked. Now that you have new teammates, and they started to ask you about your model versions, you realize that storing the ML models in files is not that manageable after all. 

This is the moment when you really feel the pain of not having an efficient way to share your model versions, model artifacts, and model metadata. And the thing is, at this point, there is no easy way to go back in time and set up something proper. 

Your new and improved cross-functional team is asking you about:

• Where can we find the best version of this model so we can audit, test, deploy, or reuse it?
• How was this model trained?
• How can we track the docs for each model to make sure they are compliant and people can know the necessary details about it including the metadata?
• How can we review models before they are put to use or even after they have been deployed?
• How can we integrate with tools and services that make shipping new projects easier?

Can you blame them?

They want to understand what is running in production, and how to improve it or roll back to previous versions. It makes total sense. 

So with all that pretty experience you gained, you start your next project and look for a tool that deals with it. And you find this article about the machine learning model registry. 

So let’s dive in and tell you all about it!

What is model registry?

An ML model registry serves as a centralized repository, enabling effective model management and documentation. It allows for clear naming conventions, comprehensive metadata, and improved collaboration between data scientists and operations teams, ensuring smooth deployment and utilization of trained models.

How does machine learning model registry work?

A data scientist can push trained models to the model registry. Once in the registry, your models are ready to be tested, validated, and deployed to production in a workflow that is similar to the one below:

While Model Registry serves as a repository of your machine learning model artifacts, there are however some differences, as a repository is a subset feature of a registry. Let’s understand the distinctions between Model Registry and other similar-sounding terms.

Model registry vs model repository

Model Repository is a storage location for machine learning models, while a Model Registry is a more comprehensive system that tracks and manages the full lifecycle of machine learning models.

However, both are often used interchangeably, and the specific definitions may vary depending on the context or the tools and platforms being used. 

In a nutshell, broad differences on a conceptual level look like this:

Model registry vs model store

Model store is a superset of a Model registry. Meaning within a model store, you can find the model registry component. The store is a service other services can interface with to retrieve models from the registry. 

In a model registry, you may store and fetch models (like a docker registry), but in a model store, you can have the full package of logging, discovery, assets, pipelines, metadata, all model information, and even blueprints to use for building new models. Examples of typical model stores are ModelStore, neptune.ai, or ClearML.

Model registry vs experiment tracking

Model registry has to integrate with the Experiment management system (which tracks the experiments) to register ML models from various experiment runs to make them easier to find and work with. Let’s take a look at some of the key differences between a model registry and an experiment management system.

Model registry key features and functionalities

Acts as centralized storage for effective collaboration

Model registry provides a central storage unit that holds ML models (including model artifacts) for easy retrieval by an application (or service). Without the model registry, the model artifacts would be stored in files that are difficult to track and saved to whatever source code repository is established.

The centralized storage also enables data teams to have a single view of the status of all ML models, making collaboration easier. Here is an example showing a single view of different models with model artifacts stored in a model registry.

Bridges the gap between experiment and production activities

Model registry acts as a glue between ML experimentation and Operations, enabling model development, software development, and operational teams to collaborate.

Providing a central UI 

Model registry provides teams with visibility over their ML models. With a central interface, teams can: 

• Search for models,
• View the status of models (if they are being staged, deployed, or retired),
• Approve or disapprove models across different stages,
• And view the necessary documentation. 

This makes model discovery easier for everyone on the team. If a model needs to be deployed, the operations teams can easily:

• Search for it,
• Look up the validation results and other metrics,
• Package the model (if needed),
• And move it from the staging environment to the production environment. 

This improves the way cross-functional teams collaborate on ML projects.

Provides an interface for downstream systems to consume models

Model registries provide interfaces that enable downstream services to consume the model through API integration. The integration can also track offline and online evaluation metrics for the models. 

This makes it easy to build an automated setup with CI/CD/CT with ML pipelines. The downstream service could be either a model user, an automated job, or a REST serving that can consume the most stable—or any—version of the model. 

Enables model versioning by tracking different versions of a model

Model registry enables model versioning by tracking the different versions of a model as they are developed and improved. This allows you to compare different versions of the model, track its performance, and select the best version for deployment.

Here’s how it typically works:

• Model Registration: When a new model is developed or trained, it is registered in the model registry.
• Version Control: The model registry maintains a history of all registered models and their versions. 
• Model Comparison: The model registry allows users to compare performance metrics, model architectures, hyperparameters, and other relevant information in different versions of a model.
• Model Tracking: As new versions of the model are developed or trained, they are registered in the model registry as well, incrementing the version number.
• Retention and Archiving: The model registry typically retains older versions of the models, ensuring a complete history and traceability.

By enabling model versioning, the model registry ensures that different iterations of a model can be stored, tracked, compared, and accessed conveniently.

Integrates with experiment management systems or training pipelines

Model registry integrates with systems that output the trained ML models. They could be the raw model artifacts (model weights, configuration, and metadata) or models that have been serialized into a file (e.g., an ONNX file) for compatibility with the production environment or containerized (using Docker) to be exported to the production environment.

Model registries:

• Register the model, 
• Assign a version to it,
• Note the version of the dataset the model was trained on,
• Add annotations and tags,
• Retrieve the parameters, validation results (including metrics and visualizations), and other relevant model metadata on the model from the experiment management system.

To make collaboration easier, the registry also includes details such as:

• The model owner or developer, 
• Experiment run id the model was trained under, 
• Versioned model source code, 
• Environment runtime dependencies used to train the model (and the versions),
• Comments and model change history, 
• And the model documentation.

Integrates with the staging environment for your trained ML models

Model Registry provides the functionality for integrating with the staging environment for running all types of checks and balances on the model. These checks can include integration testing (with other applications) and other QA tests before the model can be promoted to the production environment. 

Sharing and collaboration should be enabled for ML models in this environment so that deployment engineers can work with data scientists to test models and ensure they are good to deploy.

In the staging environment, the model reviewers should also be able to perform fairness checks on the model to ensure it:

• Outputs explainable results, 
• Complies with regulatory requirements, 
• And provides useful business benefits.

Generally, the governance and approval workflows should be configured in this environment. There should also be access level control and secure authorization to models in this environment, especially models trained on data with sensitive information.

Integrate with model delivery (CI/CD) tools and services for automation

Automation is a critical part of building any scalable software. In machine learning, building automated pipelines will allow you to spend more time building new products rather than maintaining old models. 

A Model registry integrates with pipeline automation tools and provides custom APIs that can allow you to plug into custom workflow tools. For example, using webhooks to trigger downstream actions based on predefined events in the registry. 

You should also be able to configure model promotion schemes through different environments like development (training), staging (testing), and production (serving).  Performance is a crucial requirement for building automated pipelines. Model registries should be highly available for automated jobs that are event or schedule-based to enable continuous training and delivery of the model.

Integrate with model deployment tools

Eventually, ML models have to be deployed, and the more efficient the deployment process, the better. Model Registries:

• Integrate with downstream services and REST serving services that can consume the model and serve it in the production environment.
  
• Collect real-time (or aggregated) metrics on the production model to log performance details of the model. This will be helpful for comparison between models (deployed and staged), as well as auditing the production model for review

Why do you need a model registry?

Model registry enables machine learning operations and fits well with the pillars of MLOps:

ML model registry enables faster deployment of your models

As we discussed earlier, one of the ways a model registry enables collaboration is by bridging the gap between experimentation and production activities. This results in a faster rollout of your production ML models. 

In addition, model registries store trained models for fast and easy retrieval by any integrated application or one of the model deployment tools, which is ultimately what you want in an ideal automation setup. 

With a model registry, software engineers and reviewers can easily identify and select only the best version of the trained ML models (based on the evaluation metrics), so the model can be tested, reviewed, and released to production. This makes it a good unifying component for both training and deployment pipelines, as there is less friction in the hand-off of production-ready models from experiment to production environments.

ML model registry simplifies model lifecycle management

When you work in a large organization with lots of experiments running, many ML models, and cross-functional teams, managing the lifecycle of these models is often a challenging process. 

Model registry helps tackle this challenge by simplifying the management of your model lifecycle. With the registry, you can:

• Register, track, and version your trained, deployed, and retired models in a central repository that is organized and searchable.
• Store the metadata for your trained models, as well as their runtime dependencies so the deployment process is eased.
• Build automated pipelines that make CI/CD and training of your production model possible.
• Compare models running in production (champion models) to freshly trained models (or challenger models) in the staging environment.

Here is an example of a registered and versioned trained model in a model registry, with the model training summary and relevant metadata included:

The registry can also track and store online and offline evaluation metrics for the models. With this functionality, you can easily look up models that are in production to detect a drop in the performance of the model (or concept drift). 

You can also compare their online and offline performance to see which of the production models need to be reviewed, maintained, or archived.

Not only can you track evaluation metrics for the model both in production and training, but you can also track the system metrics to understand which models are consuming the most application resources (CPU, memory, and GPU usage). 

Here is an example of tracking the offline system and evaluation metrics for a top-performing model in Neptune:

ML model registry enables model governance

One thing that the model registry does really well is centralizing ML models and organizing their relevant details. With the registry, you have a central source of truth for your models throughout different stages of their lifecycle, including: 

• Development, 
• Validation,
• Deployment,
• And monitoring.

This helps create visibility and model discovery which is very crucial for ML models that require thorough regulatory complaint processes in specific industries such as health, finance, and law.

A user in charge of ensuring legal compliance should be able to easily review the models in the registry and understand: 

• How the model was trained,
• What version of data the model is trained on,
• The conditions a model performs best and produces consistent results at, being well-informed of the model’s capabilities and limitations,
• Exposure to any compliance risk in case of regulated industries like financial institutions

A standard model registry will also enforce the documentation and reporting of models, ensuring results are repeatable and can be reproduced by any auditing user. 

Review, Approve, Release, and Rollback are all steps in the model launch process that the registry may help with. These choices are based on a variety of factors, including offline performance, bias, fairness measures, and the results of online experiments.

ML model registry improves model security

ML models, as well as the underlying packages used to build them, must be scanned for vulnerabilities, especially when a large number of packages are used to develop and deploy the models. 

A model registry can manage specific versions of the packages, and you can scan and remove security vulnerabilities that may pose a threat to the system.

Models are likewise vulnerable to adversarial attacks, and as a result, they must be maintained and secured. In some cases, the least privilege access security concept must be employed so that only authorized users have access to specified model information, data privacy, and protecting PII and other resources.

Where does a model registry fit in the MLOps stack?

If you want to run machine learning projects efficiently and at scale, you would most likely need to add a model registry to your MLOps stack. Depending on what level of implementation you are in your MLOps stack, your needs and requirements for a model registry would differ. Where does it fit? 

Well, recall we learned earlier that the model registry sits between machine learning model development and deployment. 

Model registry in MLOps level 0

If you are at level 0 implementation of MLOps, your workflow with a model registry could look like this: 

The output from the experimentation step is fed into the model registry. This involves a manual process where the data scientist prepares the model artifact and metadata and could also package them (serialization, containerization) before registering them. 

The operations team can push the packaged model to the staging environment for testing before deploying it to a prediction service engine that can integrate with other applications.

Model registry in MLOps level 1

As opposed to level 0 (where the workflow is a manual process), the goal of the workflow in level 1 is to perform continuous training of the model by automating the ML pipeline. This is one process a model registry enables well because of its ability to integrate with the pipeline. 

At this level, the entire pipeline is deployed, and when ML models are trained on the provided dataset, the output (trained model and metadata) is fed into the model registry where it can be staged, and if it passes the necessary tests and checks, it can be fed to the continuous delivery pipeline for release.

Model registry in MLOps level 2

The role of the model registry in level 2 of the MLOps workflow is also the same as that of level 1—the automated pipeline delivers the trained model to the model registry where it is staged, may be passed through QA checks, and sent to the continuous delivery pipeline:

The model registry serves as a crucial component in any automated pipeline because event triggers can be integrated with it to promote models with good metrics upon re-training on fresh data or archive models.

Setting up machine learning model registry: build, maintain, or buy?

Setting up a model registry for your MLOps workflow will require you to decide on either Building one, Maintaining one, or Buying one. Let’s take a close look at each one of these decisions and the factors to consider before making a choice.

Building a model registry solution

Like any software solution, if you understand the key functionalities and requirements, you can build a system yourself. This is the case with a model registry. You may want to set up the following:

• Object storage for models and artifacts.
• Database for logging model details.
• API integration for both receiving models, promoting models across various environments, and collecting model information from the different environments.
• User interface (UI) for ML teams to interact with a visual workflow. 

While building the solution yourself might seem ideal, you should consider the following factors:

• Incentive: What’s the incentive to build out your solution? Is it for customization or for owning a proprietary license to the solution?
• Human resources: Do you have the talents and skills to build out your solution?
• Time: How long would it take you to build out a solution and is it worth the wait?
• Operations: When the solution is eventually built out, who would maintain its operations?
• Cost: What would it cost you to build a solution, including the maintenance of the solution?

Maintaining a self-hosted model registry

Another option to consider—if you do not want to build out a solution—is to maintain an existing solution yourself. In this scenario, the solution has been built out already, but you might have to manage some features, such as object storage and the database. Most of these existing solutions are open-source solutions.

The following are the factors to consider:

• Type of solution: Are you going to opt for an open-source solution with no license cost or a closed-source solution with license cost?
• Operations: Who is going to manage the solution? Does the solution support consistent maintenance and software updates?
• Cost: What is the cost of operating the solution in terms of the infrastructure to host it and the running cost? 
• Features: What features have already been implemented, and what features do you have to build and manage yourself? Is it worth adopting compared to building out your solution?
• Support: What type of support is available in case things break during operations? Is there a community or dedicated customer support channel? For open-source solutions, while you might have a community, you will likely lack the necessary developer support required to fix things.
• Accessibility: How easy is it to get started with the solution? Is the documentation comprehensive enough? Can everyone from the model reviewers to the model developers and software engineers intuitively use the solution?

Purchase the license to a fully-managed solution

The final option to consider is subscribing to a fully managed solution where the operations and management of the registry are handled by the solution vendor. In this case, you do not have to worry about building or maintaining a solution. You just have to ensure your systems and services can integrate with the registry.

Here are the factors to consider: 

• Industry type: What type of industry is the model built for? What sensitive information have the models learned? Are there data privacy compliance measures? Is the model only allowed to stay on-premise?
• Features: Are the key features and functionalities of any model registry available in this solution? What extra features are available, and how relevant are they to your workflow?
• Cost: What’s the cost of purchasing a license, and do the features justify the cost?
• Security: How secure is the platform hosting the solution? Is it resistant to third-party attacks?
• Performance: Is the registry highly performant? For situations where models are too large, can the registry provide models for services to consume at low latency?
• Availability: What’s the uptime of the solution, and does it meet your required service level agreement (SLA)?
• Support: What level of support is available in case things go south?
• Accessibility: How easy is it to get started with the solution? Are the documentation and learning support decent enough? What’s the learning curve in terms of usage?

That being said, there are lots of options available out there, and it totally depends on the team/organization’s requirements to choose. 

Now, let’s take a look at some of the model registry solutions on the market.

ML model registry solutions out there

Here are the most popular ones available out there that offer a host of Model Registry features.

Recommended

Comparison of experiment tracker and model registry tools on a feature-by-feature basis

Learn more

neptune.ai model registry

Type: Proprietary, with free and paid offerings.

Options: Managed (self-hosted), fully-managed offering.

Neptune is primarily an MLOps stack component for experiment tracking. But it provides model registry functionality to some extent. 

It’s a lightweight solution that fits well into any workflow and lets you track, compare, and share all kinds of metadata about your experiments and models. 

On the model registry front, it will prove to be the most useful in the model handover phase. Since you can store your model versions in Neptune and attach any metadata or artifacts to them – it can serve as a connection between the development and deployment phases. ML Engineers can access model artifacts in Neptune instantly, and they can get all the information about how the model was trained, what data was used, who ran the experiment, etc. 

Learn more

How to Store and Manage ML Models Using Model Registry With Neptune

See video

Neptune model registry features

Neptune’s model registry lets you:

• Register production-ready models;
• For any registered model, create as many versions as you want to;
• Attach any metadata or artifacts to models and model versions 
• Save hash, location, and other model artifact metadata (by the way, you don’t have to upload the model to Neptune, you can keep track of the model reference to local or S3-compatible storage);
• Assign stages to models (None/Staging/Production/Archive);
• Access models and model versions via web app and API.

Editor’s note

Do you feel like experimenting with neptune.ai?

• 

  Create a free account right away and give it a go

• 

  Try it out first and learn how it works (zero setup, no registration)

• 

  See the docs or watch a short product demo (20 min)

MLflow model registry

Type: Open-source/Managed

Options: Managed (self-hosted), fully-managed offering.

The MLflow Model Registry component is a centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of an MLflow Model. It provides model lineage (which MLflow experiment and run produced the model), model versioning, stage transitions (for example, from staging to production), and annotations. The model registry component was one of the clamored needs of MLflow users in 2019. 

The MLflow Model Registry is one of the few open-source model registries available in the market today. You can decide to manage this on your infrastructure or use a fully-managed implementation on a platform like Databricks.

MLflow model registry features

MLflow provides 

• Annotation and description tools for tagging models, providing documentation and model information such as the date the model was registered, modification history of the registered model, the model owner, stage, version, and so on.
• Model versioning to automatically keep track of versions for registered models when updated.
• An API integration to serve ML models as RESTful APIs for online testing, dashboard updates, etc.
• CI/CD workflow integration to record stage transitions, request, review, and approve changes as part of CI/CD pipelines for better control and governance.
• A model stage feature to assign preset or custom stages to each model version, like “Staging” and “Production” to represent the lifecycle of a model.
• Configuration for promotion schemes to easily move models across different stages.

MLflow stores:

• The model artifacts,
• Metadata,
• Parameters,
• Metrics.

The pricing will depend on the option you are opting for—a self-hosted, open-source solution or a fully-managed offering.

You can learn more about the workflow here and get started with MLflow Model Registry here.

Amazon Sagemaker model registry 

Type: Bundled with SageMaker’s free-tier and on-demand pricing.

Options: Fully managed offering

Amazon SageMaker is a fully managed service that developers can use for every step of ML development, including model registry. The model registry is part of the suite of MLOps offerings in SageMaker that helps users build and operationalize machine learning solutions by automating and standardizing MLOps practices across their organization.

AWS Sagemaker model registry features

With the SageMaker Model Registry you can do the following:

• Catalog models for production.
• Manage model versions.
• Associate metadata, such as training metrics, with a model.
• Manage the approval status of a model.
• Deploy models to production.
• Automate model deployment with CI/CD.

You can make a model group to keep track of all the ML models you’ve trained to solve a specific problem. Each model you train can then be registered, and the model registry will add it to the model group as a new model version. A typical workflow might look like the following:

• Create a model group.
• Create an ML pipeline with SageMaker Pipelines that trains a model.
• For each run of the ML pipeline, create a model version that you register in the model group you created in the first step.

The cost of using the Model Registry is bundled with the SageMaker pricing tiers. You can learn more about the SageMaker Model Registry in the documentation.

Verta.ai model registry

Type: Proprietary, with Open-source, SaaS, and Enterprise offerings.

Options: Fully managed offering.

The Verta.ai Model Registry helps you manage your ML models in one place. It provides features that enable you to package, validate, and reliably promote release-ready models and apply safe release and governance practices.

Verta.ai model registry features

• It provides a unified hub to publish release-ready models by allowing you to:
  • Connect to an experiment management system for end-to-end information tracking.
  • Publish all the model metadata, documentation, and model artifacts in one central repository.
  • Select the best-fit models from model experiments and stage them for release.
  • Record state transitions and manage the release lifecycle from development, staging, and production to archive.
    
• It enables model validation and CI/CD automation by allowing you to:
  • Integrate with existing CI/CD pipelines like Jenkins, Chef, and so on.
  • Use webhooks to trigger downstream actions for model validation and deployment.
  • Automatically track model versions and tagged releases.
• Set up granular access control editors, reviewers, and collaborators.
• Access detailed audit logs for compliance.
• Release models once they pass basic security and fairness checks.

You can learn more about the open-source version and pricing tiers available on this page. You can learn more about the Verta Model Registry on this page.

You’ve reached the end!

By now, we’ve established the fact that model registries enable the successful operationalization of machine learning projects, and it does so by:

• helping you speed up your roll-out process
• making your ML experiments easy to manage
• making collaboration easier
• creating a seamless hand-off for your team
• increasing security and governance

The ideal next step for you is to pick up a solution and see if it improves your MLOps workflow. 

Let this article be your guide to making a concrete decision based on your level of MLOps implementation and the factors worth taking into account. 

Here are some additional resources you might want to read for the same.

• Practitioners guide to MLOps: A framework for continuous delivery and automation of machine learning
• MLflow Model Registry – Databricks
• MLflow Model Registry — MLflow 1.21.0 documentation
• Model Registry – neptune.ai
• neptune.ai– Model Registry documentation
• ML Pipelines on Google Cloud 
• MLOps: Continuous delivery and automation pipelines in machine learning (google.com)
• Model Registry Makes MLOps Work – Here’s Why – neptune.ai
• MLOps Model Stores: Definition, Functionality, Tools Review – neptune.ai
• Introducing Verta Model Registry
• Verta Model Registry | Manage all your AI‑ML models in one place
• Top 3 Reasons You Need a Model Registry (verta.ai)
• Simplifying MLOps with Model Registry

That’s it for now, stay tuned for more! Adios!