The Best MLflow Alternatives (2020 Update)

MLflow is an open-source platform that helps manage the whole machine learning lifecycle. This includes experimentation, but also reproducibility and deployment. Each of these three elements represented by one MLflow component: Tracking, Projects, and Models.

That means a data scientist who works with MLflow is able to track an experiment, organize it, describe it for other ML engineers and pack it into a machine learning model. It’s been designed to enable scalability from one person to big organization, however, it works best for an individual user. You can find a good example of using MLflow in this article.

While MLflow is a great tool, some things could be better especially when working on a larger team and/or the number of experiment you run is very large:

  • Missing user management capabilities make it difficult to deal with access permissions to different projects/users or roles (manager/machine learning engineer).
  • Tracking UI, though improved recently, doesn’t give you full customizability when it comes to saving experiment dashboard views or grouping runs by experiment parameters (model architecture) or properties (data versions). Those are very useful when you have multiple people working on the same project or you are running thousands of experiments.
  • Speaking of large numbers of experiments, the UI can get quite slow when you really want to explore all your runs.
  • Unless you want to use Databricks platform you need to maintain MLflow server yourself. That comes with typical hurdles like access management, backups, and so on. The open-source community is vibrant but there is no dedicated user support to hold your hand when you need it.
  • MLflow is great for running experiment via Python or R scripts but the Jupyter notebook experience is not perfect, especially if you want to track some additional segments of machine learning lifecycle like exploratory data analysis or results exploration.
  • Some functionalities like logging resource consumption (CPU, GPU, Memory) or scrolling through large numbers of image predictions or charts are not there yet.

ML experiment tracking capabilities in MLflow give a ton of value to individual users or teams that are willing to maintain experiment data backend, tracking UI server and are not running huge numbers of experiments.

If some of the things mentioned before are important to you and your team you may want to look for complementary or alternative tooling. Luckily, there are many tools that offer some or most of those missing pieces.

In this article, building on top of the previous discussions like Hady Elsahar’s medium post, recent reddit discussion, and our comparison, we present top alternatives to MLflow.

In our opinion, the following are the best alternatives to MLflow:

  1. Neptune
  2. Weights & Biases
  3. Comet.ml
  4. Valohai
  5. Polyaxon

1. Neptune

Neptune is an experiment management and collaboration tool.

Neptune offers an open-source Python library that lets users log any experiments whether those run in Python scripts, Jupyter Noteboks, Amazon SageMaker Notebooks or Google Colab.

Projects in Neptune can have multiple members with different roles (viewer, contributor, admin), so all machine learning experiments that land in Neptune can be viewed, shared and discussed by every team member. Neptune is meant to provide an easy-to-use and quick-to-learn way to keep track of your experiments.

Moreover, Neptune integrates with MLflow letting you send all your experiments to Neptune and have them backed-up, organized in a nice UI, and easy to access at any time.

Neptune – summary:

  • Provides user management and organization with different organization, projects and user roles
  • Enjoy fast and beautiful UI with a lot of capabilities to organize runs in groups, save custom dashboard views and share them with the team
  • You can use a hosted app to avoid all the hussle with maintaining yet another tool (or have it deployed on your on-prem infrastructure)
  • Your team can track experiments which are executed in scripts (Python, R, other), notebooks (local, Google Colab, AWS SageMaker) and do that on any infrastructure (cloud, laptop, cluster)
  • Enjoy extensive experiment tracking and visualization capabilities (resource consumption, scrolling through lists of images)

Read more about how Neptune approaches experiment management in this blog post.

2. Weights & Biases

Weights & Biases a.k.a. WandB is focused on deep learning. Users track experiments to the application with Python library, and – as a team – can see each other experiments.

Unlike MLflow, WandB is a hosted service allowing you to backup all experiments in a single place and work on a project with the team – work sharing features are there to use.

Similarly to MLflow, in the WandB users can log and analyse multiple data types.

Weights & Biases – summary:

  • Deals with user management
  • Great UI allows users to visualize, compare and organize their runs nicely.
  • Sharing work in a team: multiple features for sharing in a team.
  • Integrations with other tools: several open source integrations available
  • SaaS/Local instance available: Yes/Yes
  • Bonus: WandB logs the model graph, so you can inspect it later.

 

3. Comet.ml

Comet is a meta machine learning platform for tracking, comparing, explaining and optimizing experiments and models.

Just like many other tools – for example Neptune (neptune-client specifically) or WandB – Comet proposes open source Python library to allow data scientists integrate their code with Comet and start tracking work in the application.

As it’s offered both cloud-hosted and self-hosted, users can have team projects and save backup of experimentation history.

Comet is converging towards more automated approaches to ML, by predictive early stopping (not available with free version of the software) and Neural architecture search (in the future).

Comet.ml – summary:

  • Deals with user management 
  • Sharing work in a team: multiple features for sharing in a team.
  • Integrations with other tools: should be developed by the user manually
  • SaaS/Local instance available: Yes/Yes
  • Bonus: Display parallel plots to check patterns in the relationships between parameters and metrics

 

4. Valohai

Valohai takes a slightly different approach when it comes to tracking and visualizing experiments.

Platform proposes orchestration, version control and pipeline management for machine learning – simply speaking they cover what MLflow is doing in terms of logging and additionally manage your compute infrastructure.

As was the case in MLflow users can easily check and compare multiple runs. At the same time, differentiator is the ability to automate starting and shutting down cloud machines used for training.

Valohai lets you develop in any programming language – including Python and R – which can be handy in a team working in a fixed technological stack.

Valohai – summary:

  • Deals with user management
  • Sharing work in team: multiple features
  • Integrations with other tools: examples of integrations provided in the documentation
  • SaaS/Local instance available: Yes/Yes
  • Bonus: WIth the infrastructure for training you can run experiments on the environment managed by Valohai.

5. Polyaxon

Polyaxon is a platform that focuses on both, the whole life cycle management of machine learning projects as well as the facilitation of the ML team collaboration.

It includes a wide range of features from tracking and optimization of experiments to model management and regulatory compliance. The main goal of its developers is to maximize the results and productivity while saving costs. It’s worth mentioning, however, that Polyaxon needs to be integrated into your infra/cloud before it’s ready to use.

Polyaxon – summary:

  • Deals with user management
  • A lot of features around team collaboration.
  • Focuses on productionalization of machine learning models
  • It is integrated with most popular deep learning frameworks and ML libraries
  • SaaS Enterprise/Local instance available: Yes/Yes
  • Bonus: it is designed to serve different groups of interests including data scientists, team leads and architects

See the comparison between Polyaxon and Neptune.

Conclusion

It’s a good idea to choose one of the MLflow alternatives if you’re not sure if the tool is suitable for you. Additionally, you make take advantage of the extra features Mlflow doesn’t have.

A good alternative will help you keep transparency in your projects, make collaboration with team easier, and improve your machine learning experiments.

Experiment Tracking tool for machine learning projects