Blog » ML Tools » Best Alternatives to MLflow Model Registry

Best Alternatives to MLflow Model Registry

In the Spark + AI Summit in Amsterdam, MLflow Model Registry, a brand new feature in the MLflow platform, was announced. MLflow Model Registry expands upon MLflow’s existing capabilities as a platform used to track the machine learning lifecycle, and now has 800,000 monthly downloads. But why is it so popular, and how can you implement this into your own machine learning workflow? 

Consider this: have you ever encountered any of these pesky machine learning bugs before: 

  • Bias: are elements in a dataset more represented than others, leading to prejudice on datasets? 
  • Drift: after your models are built and deployed, they must be recalibrated and resynced according to usage and data to maintain accuracy – if not, it will drift and deteriorate over time 
  • Fragility: can your model cope with issues such as unavailable data, null vale, etc? 

These bugs are notoriously difficult to identify pre-production. However, monitoring and observability, and identifying the business value that your ML components impact can help you approach them early on. 

The ML lifecycle establishes what steps a company should take to best use machine learning and artificial intelligence to derive practice business value.

However, as with traditional development, the ML lifecycle comes with problems such as version control, packaging, deployment, collaboration, and serving. Rather than forcing software development solutions into this ecosystem, custom products such as MLflow are providing tools for training and serving models. MLflow Model Registry lets you manage the full lifecycle of MLflow Models. 

Model Registry allows you to track:

  • chronological model lineage (which MLflow experiment and run produced the model at a given time)
  • model versioning
  • stage transitions (for example, from staging to production or archived)
  • model version annotations and descriptions

Today we will explore the functionality of MLFlow Model Registry while also discussing more effective alternatives. 

Here’s what you’ll find in this article:

  • A quick overview of MLflow Model Registry 
    • The Model Registry UI 
    • The Model Registry API
  • The problems with MLflow Model Registry
  • A list of alternatives to MLflow 

What is MLflow Model Registry? 

MLflow Model Registry is a centralized model store and set of APIs and UI that allow you to collaboratively manage the full lifecycle of an MLflow model. 

MLflow Model Registry features

  • Central Repository allows you to register MLflow models; each registered model has a unique name, version, stage, and other metadata
  • Model Versioning lets you automatically track versions for registered models as they are updated
  • Model Stage is the assigned preset/ custom stages for each model version, such as “staging” and “production” that represents a model lifecycle 
  • Model Stage Transitions record new registration events or changes as activities that automatically log users, changes, and other metadata
  • CI/CD Workflow Integration allows you to record stage transitions, request, review and approve changes as part of CI/CD pipelines for better control and governance
  • Model Serving lets you quickly serve machine learning models as RESTful APIs for online testing, dashboard updates, etc. 

What problem does it solve? 

Large businesses can have thousands of ML models during experimentation, testing, and production. 

MLFlow Model Registry provides an organized approach by allowing teams to:

  • Share ML models
  • Work together from experimentation, online testing, production, etc. 
  • Integrate with approval and governance workflows
  • Monitor ML deployments and their performance

You must use a database-backed backend store to access the model registry via the UI or API when running your own MLflow server. Once you have logged your model to the Model Registry, you can add, modify, update, transition, or delete the model through the UI or the API.

UI Workflow

Within the registered Models page, you can find the model properties: 

MLflow model registry

Each model’s overview page shows the active versions: 

MLflow model registry

You can click a version to navigate to the version detail page: 

MLflow model registry

The version detail page shows the model version details and current stage of the model version. 

The Stage drop-down at the top right lets you transition the model version to one of other valid stages:

MLflow model registry

API Workflow

The Mlflow model flavor or MLflow Client Tracking API interface is another way to interact with Model Registry. You can register a model during an MLflow experiment run or after. 

Here is an example of adding an MLflow Model to the Model Registry: 

result = mlflow.register_model(
    "runs:/d16076a3ec534311817565e6527539c0/artifacts/sklearn-model",
    "sk-learn-random-forest-reg"
)

Here is how you could fetch an MLflow model from the Model Registry: 

import mlflow.pyfunc

model_name = "sk-learn-random-forest-reg-model"
stage = 'Staging'

model = mlflow.pyfunc.load_model(
    model_uri=f"models:/{model_name}/{stage}"
)

model.predict(data)

Here is how you could add or update Mlflow Model Descriptions: 

client = MlflowClient()
client.update_model_version(
    name="sk-learn-random-forest-reg-model",
    version=1,
    description="This model version is a scikit-learn random forest containing 100 decision trees"
)

The problems with MLflow Model Registry

Although MLflow is a full-fledged MLOps framework, it fails to meet all the experiment tracking needs of a team:

  • MLflow does not let you manage user permissions
  • The MLflow UI lags when you have thousands of runs
  • MLflow does not let you save different experiment dashboard views
  • MLflow does not allow you to monitor hardware 
  • MLflow does not let you track exploratory analysis
  • MLflow lacks many integrations with other ML tools

Other tools such as Neptune AI provide a more comprehensive framework with an intuitive UI that lets you manage user access within a team and shows team progress. Read through the list of better alternatives we have generated for you.


READ ALSO
👉 MLOps: What It Is, Why it Matters, and How To Implement it (from a Data Scientist Perspective)
👉 The Best MLOps Tools You Need to Know as a Data Scientist


List of alternatives to MLflow Model Registry

Neptune

Build reproducible

Neptune is an extensive, lightweight machine learning experiment management tool that is very flexible and easy to integrate. It has two components: You can install the server on your own hardware or use it is as a service, or the client libraries

One of the major advantages of Neptune is its easy integration with all workflows, and team management; although all team members may use various ML libraries and platforms, they can share results and collaborate on a single dashboard. 

You can even use their software-as-a-service offering so that you do not have to deploy it on to your own hardware. 

Neptune’s main features include: 

  • Experiment Management: You can keep track of all your team’s experiments, and tag, filter, group, sort, and compare them 
  • Notebook versioning and diffing: You can compare two notebooks or checkpoints in the same notebook; similar to source code, you can do a side-by-side comparison 
  • Team Collaboration: You can add comments, mention teammates, and compare experiment results 

Neptune is a feature-rich and collaborative platform that allows for easy machine learning model management, placing it over MLflow. 

👉 Check Neptune vs. MLflow comparison

Azure Machine Learning 

Azure

Azure Machine Learning is a cloud MLOps platform that lets you manage and automate the whole ML lifecycle, including model management, deployment, and monitoring. The following MLOps capabilities are included in Azure: 

  • Create reproducible ML pipelines.
  • Create reusable software environments for training and deploying models.
  • Register, package, and deploy models from anywhere.
  • Deals with data governance for the end-to-end ML lifecycle.
  • Notify and alert on events in the ML lifecycle.
  • Monitor ML applications for operational and ML-related issues.
  • Automate the end-to-end ML lifecycle with Azure Machine Learning and Azure Pipelines.

Azure is helpful if you would like to make your whole ML infrastructure cloud-based, or already are. 

ModelDB

ModelDB

Model DB is a complete system that accounts for every aspect of ML model management: logging, version control, and a central dashboard. It’s powerful backend can run docker containers, a pluggable storage system, and integrations with other powerful ML tools like Tensorflow, Pytorch, and scikit-learn. 

Model DB allows you to make your ML models reproducible, and manage your ML experiments with build performance dashboards. You can track models across their lifecycle including development, deployment, and live monitoring. 

Learning more about Neptune…

As you can see, Neptune provides a more comprehensive set of tools for your machine learning and model registry needs. Through integrating Neptune with MLflow, you can even use Neptune as a central place for managing your experiments and collaborating on them with your team.

If you want to learn the details about Neptune, check out the official documentation. If you want to try it out, you can create a free account and start tracking your machine learning experiments with Neptune.


NEXT STEPS

How to get started with Neptune in 5 minutes

1. Create a free account
Sign up
2. Install Neptune client library
pip install neptune-client
3. Add logging to your script
import neptune.new as neptune

run = neptune.init('Me/MyProject')
run['params'] = {'lr':0.1, 'dropout':0.4}
run['test_accuracy'] = 0.84
Try live notebook

MLflow Neptune

MLflow vs. Neptune: How Are They Actually Different?

Read more
Experiment tracking Experiment management

15 Best Tools for ML Experiment Tracking and Management

Read more

The Best MLflow Alternatives (2021 Update)

Read more
MLflow share and collaborate

How to Make your MLflow Projects Easy to Share and Collaborate on

Read more