Blog » ML Tools » The Best MLflow Alternatives (2021 Update)

The Best MLflow Alternatives (2021 Update)

MLflow is an open-source platform that helps manage the whole machine learning lifecycle. This includes experimentation, but also reproducibility, deployment, and storage. Each of these four elements is represented by one MLflow component: Tracking, Projects, Models, and Registry.

That means a data scientist who works with MLflow is able to track an experiment, organize it, describe it for other ML engineers and pack it into a machine learning model. In this article, we focus mostly on the experiment tracking capabilities of MLflow and overview the best alternatives for that.

While MLflow is a great tool, some things could be better especially when working in a larger team and/or the number of experiments you run is very large. 

What are the main concerns you may have? What are the main MLflow weaknesses?

  • Missing user management capabilities make it difficult to deal with access permissions to different projects or roles (manager/machine learning engineer). Because of that, and no option to share UI links with other people, team collaboration is also challenging in MLflow. 
  • Tracking UI, though improved recently, doesn’t give you full customizability when it comes to saving experiment dashboard views or grouping runs by experiment parameters (model architecture) or properties (data versions). Those are very useful when you have multiple people working on the same project or you are running thousands of experiments.
  • Speaking of large numbers of experiments, the UI can get quite slow when you really want to explore all your runs.
  • Unless you want to use the Databricks platform you need to maintain the MLflow server yourself. That comes with typical hurdles like access management, backups, and so on, not to mention that it’s very time-consuming.
  • The open-source community is vibrant but there is no dedicated user support to hold your hand when you need it.
  • MLflow is great for running experiments via Python or R scripts but the Jupyter notebook experience is not perfect, especially if you want to track some additional segments of the machine learning lifecycle like exploratory data analysis or results exploration.
  • Some functionalities like logging resource consumption (CPU, GPU, Memory) or scrolling through large numbers of image predictions or charts are not there yet.

ML experiment tracking capabilities in MLflow give a ton of value to individual users or teams that are willing to maintain experiment data backend, tracking UI server, and are not running huge numbers of experiments.

If some of the things mentioned above are important to you and your team you may want to look for complementary or alternative tooling. Luckily, there are many tools that offer some or most of those missing pieces.

In this article, building on top of some reddit discussions, and our comparison, we present top alternatives to MLflow.

In our opinion, the following are the best alternatives to MLflow:

1. Neptune

Source

Neptune is a metadata store—it serves as a connector between different parts of the MLOps workflow from data versioning, experiment tracking to model registry and monitoring. Neptune makes it easy to store, organize, display, and compare all metadata generated during the ML model lifecycle.

Neptune makes it easy to store, organize, display, and compare all metadata generated during the ML model lifecycle.

You can log metrics, hyperparameters, interactive visualizations, videos, code, data versions, and more, and organize it in a customized structure. Once logged, everything is visible in an intuitive and clean UI, where you can analyze and compare it.

You can also create custom dashboards that include all this metadata and share them with your colleagues, team manager, or even external stakeholders. Here’s an example of such a dashboard:

Metadata dashboard artifacts
Example dashboard in Neptune | See in the app

 

There are four different comparison views available – charts, parallel coordinates, side-by-side tabular dashboard, and artifacts comparison section. So you can easily evaluate models and select the best-performing ones.   

Neptune can also be extremely useful in the production phase. With all the logged metadata, you know how the model was created and how to reproduce it. 

Neptune—summary:

If you want to see Neptune in action, check this live Notebook or this example project (no registration is needed) and just play with it. 

MLflow vs Neptune

The main difference between these tools is that MLflow is an open-source solution while Neptune is a managed cloud service. It affects various aspects of how MLflow and Neptune work. If you’re looking for a free, open-source tool, that covers a wide range of ML lifecycle steps, MLflow might be the right choice for you. But you should keep in mind, that even though MLflow is free to download, it does generate costs related to maintaining the whole infrastructure.

If you prefer to focus on the ML process and leave hosting to someone else, Neptune is the way to go. For the monthly fee, you get excellent user support, quick & easy setup, you don’t have to worry about maintenance, and the tool scales well. Plus, Neptune has user management features, so it will work better in the team environment. 

Want to dig deeper?

👉 See an in-depth comparison between Neptune and MLflow.

👉 Read the case study of Zoined to learn why they chose Neptune over MLflow.

2. Weights & Biases

Weights & Biases a.k.a. WandB is focused on deep learning. Users track experiments to the application with Python library, and – as a team – can see each other’s experiments.

Unlike MLflow, WandB is a hosted service allowing you to backup all experiments in a single place and work on a project with the team – work sharing features are there to use.

Similarly to MLflow, in the WandB users can log and analyze multiple data types.

Weights & Biases—summary:

  • Deals with user management
  • Great UI allows users to visualize, compare and organize their runs nicely.
  • Sharing work in a team: multiple features for sharing in a team.
  • Integrations with other tools: several open source integrations available
  • SaaS/Local instance available: Yes/Yes
  • Bonus: WandB logs the model graph, so you can inspect it later.

MLflow vs Weights & Biases

Similar to Neptune, Weight & Biases offers a hosted version of its tool. In opposite to MLflow, which is open-sourced, and needs to be maintained on your own server. Weights & Biases provides features for experiment tracking, dataset versioning, and model management, while MLflow covers almost the entire ML lifecycle. Finally, WandB offers user management features that are probably important for you when working in a team. 

Want to check other comparisons?

3. Comet

Comet is a meta machine learning platform for tracking, comparing, explaining, and optimizing experiments and models.

Just like many other tools – for example, Neptune (neptune-client specifically) or WandB – Comet proposes an open-source Python library to allow data scientists to integrate their code with Comet and start tracking work in the application.

As it’s offered both cloud-hosted and self-hosted, users can have team projects and save the backup of experimentation history.

Comet is converging towards more automated approaches to ML, by predictive early stopping (not available with the free version of the software) and Neural architecture search (in the future).

Comet—summary:

  • Deals with user management 
  • Sharing work in a team: multiple features for sharing in a team.
  • Integrations with other tools: should be developed by the user manually
  • SaaS/Local instance available: Yes/Yes
  • Bonus: Display parallel plots to check patterns in the relationships between parameters and metrics

MLflow vs Comet

Comet comes with user management features and allows for sharing projects within the team—something that is missing in MLfow. It also offers both, hosted and on-premises setup, while MLflow is only available as an open-source solution that requires you to maintain it on your own server.  

Want to check other comparisons?

4. Valohai

Valohai takes a slightly different approach when it comes to tracking and visualizing experiments.

The platform proposes orchestration, version control, and pipeline management for machine learning – simply speaking they cover what MLflow is doing in terms of logging and additionally manage your compute infrastructure.

As was the case in MLflow users can easily check and compare multiple runs. At the same time, the differentiator is the ability to automate starting and shutting down cloud machines used for training.

Valohai lets you develop in any programming language – including Python and R – which can be handy in a team working in a fixed technological stack.

Valohai—summary:

  • Deals with user management
  • Sharing work in team: multiple features
  • Integrations with other tools: examples of integrations provided in the documentation
  • SaaS/Local instance available: Yes/Yes
  • Bonus: With the infrastructure for training you can run experiments on the environment managed by Valohai.

MLflow vs Valohai

As per Valohai’s own comparison, Valohai provides MLflow-like experiment tracking without any setup. Similar to MLflow, Valohai covers a big part of the MLOps landscape (including experiment tracking, model management, machine orchestration, and pipeline automation), but it is a managed platform, rather than an open-source solution. 

5. TensorBoard

TensorBoard is an open-source visualization toolkit for TensorFlow that lets you analyze model training runs. It’s often the first choice of TensorFlow users. TensorBoard allows you to visualize various aspects of machine learning experiments, such as metrics or model graphs, as well as view tensors’ histograms and more.

Apart from the popular, open-source version of TensorBoard, there’s also TensorBoard.dev which is available on a managed server as a free service.

TensorBoard.dev lets you upload and share your ML experiment results with anyone. It’s an important upgrade in comparison to TensorBoard, ad the collaboration features are missing there. 

TensorBoard—summary:

  • Well-developed features related to working with images
  • The What-If Tool (WIT), that’s an easy-to-use interface for expanding understanding of black-box classification and regression ML models
  • Strong and big community of users that provide community support.

MLflow vs TensorBoard

Both tools are open-source and supported by their respective communities in terms of dealing with any issues and questions. The main difference seems to be the range of features each of them provides. TensorBoard is described as a visualization kit for TensorFlow, so it serves well for visualizations, it allows you to track experiments and compare them (limited capability). MLflow, on the other hand, proves to be useful in many more stages of the ML lifecycle. Both tools are lacking user management and team sharing features (sharing is available in TensorBoard.dev, but there’s no possibility to manage the privacy of data there).

Want to check other comparisons?

Conclusion

MLflow is a great tool, but there are certain capabilities it doesn’t have. So it’s worth checking what else is available out there. In this overview, we mentioned 5 tools that could be good alternatives and check the missing boxes.

If your main reason to look for MLflow alternatives are missing collaboration and user management features, you should check Neptune, Weights & Biases, Comet, or Valohai. All of them are also available as hosted applications if you don’t want to maintain the experiment tracking tool yourself. 

If you want to stick to open-source tools, TensorBoard may be the tool for you, but you should keep in mind that it’s less advanced than MLflow in terms of features. 

Finally, if you don’t need a tool that covers almost an entire ML lifecycle (like MLflow or Valohai), we recommend you to check Neptune, Weight & Biases, or Comet. 

In any case, make sure that the alternative solution matches your needs and improves your workflow. Hope this article helps you find it. Good luck!


READ NEXT

Zoined Case Study: Open-source or Paid Hosted Solution?

6 mins read | Updated October 25th, 2021

Zoined offers Retail and Hospitality Analytics as a cloud-based service for different roles from top management to manager level. The service collects sales data from stores and venues including inventories, time and attendance, and visitor tracking systems as well as webstores. The data is analyzed and presented in a very accessible, visual format for business owners so they can get real-time, actionable insights for their business and select the preferred time frames they want to report on. The product also allows businesses to filter and group their data easily and create custom views, grasp trends quickly with charts and graphs.

Zoined case study
Zoined dashboard | Source: Zoined

With Zoined® businesses have access to a fully-managed, off-the-shelf solution with ready-made dashboards and analytics for retail and wholesale, especially fashion, food retail, coffee shops’, and restaurants’ needs.

Problem

Running lots of experiments, especially in a start-up with few scientists and engineers solving problems, can get daunting. Tracking the experiments, versioning the datasets as they inevitably get larger, and generally taking procedures to get reproducible results can be very tricky to navigate. This was the problem Kha faced when he first joined Zoined.

“When I joined this company, we were doing quite many different experiments and it’s really hard to keep track of them all so I needed something to just view the result or sometimes or also it’s intermediate results of some experiments like what [does] the data frame look like? What [does] the CSV look like? Is it reasonable? Is there something that went wrong between the process that resulted in an undesirable result? So we were doing it manually first but just writing… some log value to some log server like a Splunk.”

Kha Nguyen, Zoined

Kha Nguyen

Senior Data Scientist at Zoined

Continue reading ->
Experiment tracking Experiment management

15 Best Tools for ML Experiment Tracking and Management

Read more
Best MLOps tools

The Best MLOps Tools and How to Evaluate Them

Read more
MLflow share and collaborate

How to Make your MLflow Projects Easy to Share and Collaborate on

Read more
MLOps guide

MLOps: What It Is, Why It Matters, and How to Implement It

Read more