Best DataRobot Alternatives for Model Registry

DataRobot is an enterprise AI platform that provides tools for building, managing, and deploying models at scale for businesses, and automating end-to-end ML lifecycles. It contains different kinds of models, data, latest open-source algorithms that can be used on-premise, or as a fully managed AI service, DataRobot gives you the power of AI to drive better business outcomes. One important component in DataRobot is its model registry.

The DataRobot Model Registry is the central organizational hub for the variety of models used on the DataRobot platform. It houses each model and registers them as a deployment-ready model package. 

DataRobot model registry offers the following: 

• A central hub to register all your models regardless of their origin or deployment location.
• It processes each package functions the same way, regardless of the origin of its model.
• It provides a unified deployment experience for your machine learning models across the entire enterprise.  You can also deploy models within DataRobot to external sources.
• DataRobot model registry has a Custom Model Workshop, where you can create and deploy custom models.
• You can also create custom model inferences and add external model packages for use in the model registry.
• DataRobot provides a leaderboard for comparing all the models in the model registry. This leaderboard also provides explainability on the features and techniques used during model training and allows to learn more details about how much data it trained on, and its overall accuracy scores.

The main aim of the DataRobot Model Registry is to store all model metadata needed for reproducibility and deployment across the business enterprise. In turn, this would allow data science teams to easily find all model-related metadata whenever they need them.

However, the DataRobot model registry might not be the best option for your team, and here’s why:

• The DataRobot Model Registry is one of the components of the DataRobot platform, so to use it your team has to move the entire machine learning infrastructure to the DataRobot platform. 
• The API integrations of DataRobot with other frameworks and libraries are not so easy to use.
• DataRobot only stores deployment-ready models, i.e. experimental models that require continuous training are not stored.
• When it comes to data handling, DataRobot handles only small data unless you’re using the enterprise version. As you know, every model is as good as the amount of data it is being trained on.
• Also, data cannot be edited within DataRobot. i.e No data cleansing or ETL capabilities. The data has to be edited outside the system and then be input again.
• You cannot export the underlying code used in DataRobot.
• DataRobot is less suitable for unsupervised learning and scenarios where enough data is not available.
• On DataRobot there are constraints to how many models can be run at the same time. Not more than two per time.

So it is imperative that we look at some alternative tools for model registry.

1. Neptune

Neptune is a metadata store for MLOps, built for research and production teams that run a lot of experiments. It provides a central store to log, store, display, organize, compare, and query all metadata generated during the machine learning lifecycle.

• It supports and stores many ML model-related metadata types and you can version, display, and query most metadata generated during model building. 
• Neptune model registry helps you store and log metadata on your models, dataset, environment configuration versions, hyperparameters, and model metrics.

Features

• From the Neptune dashboard you can:
  • search based on metric and parameter values, 
  • find runs with user-defined tags, 
  • group model training runs based on the parameter or tag values.
• Neptune versions, and stores all your model training runs in its central registry repository. 
• Neptune versions and stores all types of model metadata such as pkl files, data metadata, images, data, charts, etc.
• Neptune allows you to log model information via the UI or the client library.
• Combined with its model registry capability, Neptune allows you to track changes in models metadata. 
• You can make comparisons between model training runs and get insights into the differences between them.
• Neptune has integrations with ML frameworks and libraries including PyTorch, TensorFlow, Keras, XGBoost, etc.
• Neptune model registry allows collaboration for ML teams as it gives access for teams to explore all the model-related metadata from its central store.

Pricing

Neptune provides pricing plans for Cloud and your private infrastructure. There are a few plans available:

• Individual: Free (+ usage above free quota)
• Academia: Free
• Team: Paid

Check out Neptune’s pricing to learn more.

2. SageMaker Model Registry

SageMaker is a fully managed tool that can be used for every stage of ML development, including a model registry. It helps users build, debug, deploy, and basically everything one needs to adequately monitor ML models all within a unified visual interface.

SageMaker Model Registry provides the following:

• A searchable centralized store for models and model-related metadata for reproducibility.
• Versioning and tracking for every experimental training run.
• Deploying a model from the model registry into an existing Sagemaker endpoint.

Sagemaker Model Registry allows you to connect to Amazon SageMaker Model Monitor to continuously monitor the quality of your machine learning model in real-time after deployment.

Features

• Sagemaker has a system that helps to catalog models for production. Sorting and storing them according to model type and content.
• In the Sagemaker model registry, models can be quickly moved to production. It has Automated model deployment with CI/CD.
• Sage maker allows you to switch from one hardware configuration to another with its ability to provide and manage the hardware infrastructure of your model’s environment
• Jupyter notebooks are easily created and shared.
• It has over 150 preset models available for several cases.
• Model groups can be created to monitor models that have already been trained to solve certain problems. These models can then be stored and made into new model versions.
• SageMaker Studio supports several other frameworks such as Tensorflow, PyTorch, MXNet, etc

Pricing

With Sagemaker, it is pay-per-use. And there are two available payment options:

• On-demand pricing: This is billed by the second, with no minimum fees or upfront commitments.
• The SageMaker savings plan: This offers a flexible, usage-based pricing model in exchange for a commitment to a consistent amount of usage.

The AWS pricing calculator can be used to manage your billing. Also, check out the pricing page for detailed info.

3. MLflow Model Registry

MLflow model registry is a self-sufficient system that comprises a UI, set of APIs, concepts that enable the efficient management of the life cycle of MLflow models. Stage transitions, model versioning, and annotations are included. MLflow model registry stores model artifacts, Metadata, Parameters, and Metrics.

Features

• MLflow Model Registry works hand in hand with the MLflow tracking component. This allows you to backtrack to the original source, from which the model and data artifacts were generated, including its source code version, thereby producing a thorough lineage of the lifecycle for all models and data transformation.
• MLflow Model Registry comprises APIs and smart UI that enable easy registration and sharing of new model versions. And of course conduct management of the life cycle on existing models.
• MLFlow has the ability to reproduce experiments and reports/results.
• With MLFlow you can Instantly version the data stored in the data lake as it is stored into the Delta table or directory. 
• Each model version has an allocated preset. For example, “Production” depicts the life cycle of a model.

Pricing

It is free.

4. Comet

Comet is a machine learning, cloud-based experiment management platform. Comet model registry provides a system for logging, registering, versioning, and deploying machine learning models, via its Python SDK Experiment, as well as registering, versioning, and deploying them. 

Comet simplifies the process of documenting the history of the experiments and model versions.

Model training runs are registered in Comet via:

• Comet.ml Experiment’s Asset Tab User Interface
• Programmatically through the Comet Python SDK

Features

• Comet is compatible with the majority of platforms and machine learning libraries. 
• Comet makes it easy to compare experiments, by making code, hyperparameters, metrics, and dependencies in one user interface, accessible to you.
• The automated notification system enables users to constantly monitor the model’s performance. Thereby, boosting the overall quality of the project.
• With its in-built features for visualization and reporting, comet allows for both individual developers and professional teams to communicate, as well as run and track experiments as they occur.

Pricing

Comet offers the following pricing plans:

• Individual: Free (+ usage above free quota)
• Academia: Free
• Team: Paid

You can read about their pricing in detail here.

5. Verta

Verta is an AI/ML model management and operations tool with model registry features, where you can manage and deploy your machine learning models in a central space.

Verta enables you to streamline your data science and ML workflows. It also facilitates faster deployment of models into production, while ensuring real-time model health.

Features

• Verta model registry has a central base for the publishing of release-ready models. 
• Verta model registry is connected to the experiment component making it easy to reuse trained models.
• Verta model registry helps you with managing model governance.
• Verta works well on Docker and Kubernetes. Can be integrated with several ML tools like Tensorflow, PyTorch, Spark.ml, and R. 
• Verta can be easily integrated with CI/CD pipelines such as Jenkins, Chef, and GitOps. 
• Verta is also known for its Git-like environment. Making it very convenient and familiar to experienced Git users (and other developers).

Pricing

Verta offers the following pricing plans:

• Open-source: Free
• SaaS: Paid
• Enterprise: Paid

You can read about their pricing in detail here.

6. Dataiku

Dataiku provides a central solution for the design, deployment, and management of ML models for businesses. It provides self-service analytics and collaboration for teams for efficient work. 

When models are designed and trained on Dataiku DSS, they are stored in Lab, which poses as its model registry. 

Dataiku model registry is the central store for models in Dataiku. It allows the storage of external models via MLflow. Dataiku Model registry provides a central point to: 

• Explore Models and their versions.
• Track the state of a model version
• Track simple input drifts over time.
• Store model parameters and data lineage.

Features

• Dataiku provides a dashboard for monitoring model operations and training.
• Dataiku provides both model and data lineage.
• Dataiku provides a set of interactive features for data cleaning, analysis, and validation.
• It provides end-to-end management for models from model building to deployment.
• Data DSS is data agnostic.
• Dataiku has a robust deployment API that ships models to production.
• Data flow is very easy to visualize. Data models can be represented in picture and table form, as opposed to having different files everywhere.
• Freedom for you to choose your database format, available options include SQL, NoSQL, and HADOOP. 
• Dataiku integrates with frameworks like Tensorflow, Keras, XGboost, and external libraries( H2O, Data, etc) using its API.

Pricing

Dataiku offers services for both Online or Installed on-premise or on your Cloud stack. It provides a 14-day trial version after that it starts at $499/ month. Read more about the pricing here.

Comparison of DataRobot alternatives for Model Registry

	DataRobot Model Registry	Neptune	Dataiku	MLflow Model Registry	SageMaker Model Registry	Comet	Verta
Artifact Store
Model Metadata Store
Dataset Metadata Store
Metric Store
Environment Config Store
Centralized Storage
Collaboration
Integration With Model Delivery Tools
Integration With Model Deployment Tools
Pricing	Paid	Paid	Offers both free and paid versions	Free (open-source)	Paid	Paid	Paid

Conclusion

The DataRobot model registry is a good component in the DataRobot platform that promotes reproducibility and collaboration within teams but as stated above it might not always be the best fit for your team.

Alternative model registry tools like Neptune, Sagemaker Model registry,  Dataiku, MLflow would allow ML teams to perform better model registry and governance.

Also tools like Azure Model Registry, Tensorflow, Determined AI all have model registries on their platforms, you can also check them out. 

Happy experimenting!