Blog » ML Tools » The Best MLflow Alternatives (2021 Update)

The Best MLflow Alternatives (2021 Update)

MLflow is an open-source platform that helps manage the whole machine learning lifecycle. This includes experimentation, but also reproducibility, deployment, and storage. Each of these four elements is represented by one MLflow component: Tracking, Projects, Models, and Registry.

That means a data scientist who works with MLflow is able to track an experiment, organize it, describe it for other ML engineers and pack it into a machine learning model. In this article, we focus mostly on the experiment tracking capabilities of MLflow and overview the best alternatives for that.

While MLflow is a great tool, some things could be better especially when working in a larger team and/or the number of experiments you run is very large. 

What are the main concerns you may have? What are the main MLflow weaknesses?

  • Missing user management capabilities make it difficult to deal with access permissions to different projects or roles (manager/machine learning engineer). Because of that, and no option to share UI links with other people, team collaboration is also challenging in MLflow. 
  • Tracking UI, though improved recently, doesn’t give you full customizability when it comes to saving experiment dashboard views or grouping runs by experiment parameters (model architecture) or properties (data versions). Those are very useful when you have multiple people working on the same project or you are running thousands of experiments.
  • Speaking of large numbers of experiments, the UI can get quite slow when you really want to explore all your runs.
  • Unless you want to use the Databricks platform you need to maintain the MLflow server yourself. That comes with typical hurdles like access management, backups, and so on, not to mention that it’s very time-consuming.
  • The open-source community is vibrant but there is no dedicated user support to hold your hand when you need it.
  • MLflow is great for running experiments via Python or R scripts but the Jupyter notebook experience is not perfect, especially if you want to track some additional segments of the machine learning lifecycle like exploratory data analysis or results exploration.
  • Some functionalities like logging resource consumption (CPU, GPU, Memory) or scrolling through large numbers of image predictions or charts are not there yet.

ML experiment tracking capabilities in MLflow give a ton of value to individual users or teams that are willing to maintain experiment data backend, tracking UI server, and are not running huge numbers of experiments.

If some of the things mentioned above are important to you and your team you may want to look for complementary or alternative tooling. Luckily, there are many tools that offer some or most of those missing pieces.

In this article, building on top of some reddit discussions, and our comparison, we present top alternatives to MLflow.

In our opinion, the following are the best alternatives to MLflow:

1. Neptune

Source

Neptune is a metadata store—it serves as a connector between different parts of the MLOps workflow from data versioning, experiment tracking to model registry and monitoring. Neptune makes it easy to store, organize, display, and compare all metadata generated during the ML model lifecycle.

Neptune makes it easy to store, organize, display, and compare all metadata generated during the ML model lifecycle.

You can log metrics, hyperparameters, interactive visualizations, videos, code, data versions, and more, and organize it in a customized structure. Once logged, everything is visible in an intuitive and clean UI, where you can analyze and compare it.

You can also create custom dashboards that include all this metadata and share them with your colleagues, team manager, or even external stakeholders. Here’s an example of such a dashboard:

Metadata dashboard artifacts
Example dashboard in Neptune | See in the app

 

There are four different comparison views available – charts, parallel coordinates, side-by-side tabular dashboard, and artifacts comparison section. So you can easily evaluate models and select the best-performing ones.   

Neptune can also be extremely useful in the production phase. With all the logged metadata, you know how the model was created and how to reproduce it. 

Neptune—summary:

If you want to see Neptune in action, check this live Notebook or this example project (no registration is needed) and just play with it. 

MLflow vs Neptune

The main difference between these tools is that MLflow is an open-source solution while Neptune is a managed cloud service. It affects various aspects of how MLflow and Neptune work. If you’re looking for a free, open-source tool, that covers a wide range of ML lifecycle steps, MLflow might be the right choice for you. But you should keep in mind, that even though MLflow is free to download, it does generate costs related to maintaining the whole infrastructure.

If you prefer to focus on the ML process and leave hosting to someone else, Neptune is the way to go. For the monthly fee, you get excellent user support, quick & easy setup, you don’t have to worry about maintenance, and the tool scales well. Plus, Neptune has user management features, so it will work better in the team environment. 

Want to dig deeper?

2. Weights & Biases

Weights & Biases a.k.a. WandB is focused on deep learning. Users track experiments to the application with Python library, and – as a team – can see each other’s experiments.

Unlike MLflow, WandB is a hosted service allowing you to backup all experiments in a single place and work on a project with the team – work sharing features are there to use.

Similarly to MLflow, in the WandB users can log and analyze multiple data types.

Weights & Biases—summary:

  • Deals with user management
  • Great UI allows users to visualize, compare and organize their runs nicely.
  • Sharing work in a team: multiple features for sharing in a team.
  • Integrations with other tools: several open source integrations available
  • SaaS/Local instance available: Yes/Yes
  • Bonus: WandB logs the model graph, so you can inspect it later.

MLflow vs Weights & Biases

Similar to Neptune, Weight & Biases offers a hosted version of its tool. In opposite to MLflow, which is open-sourced, and needs to be maintained on your own server. Weights & Biases provides features for experiment tracking, dataset versioning, and model management, while MLflow covers almost the entire ML lifecycle. Finally, WandB offers user management features that are probably important for you when working in a team. 

Want to check other comparisons?

3. Comet

Comet is a meta machine learning platform for tracking, comparing, explaining, and optimizing experiments and models.

Just like many other tools – for example, Neptune (neptune-client specifically) or WandB – Comet proposes an open-source Python library to allow data scientists to integrate their code with Comet and start tracking work in the application.

As it’s offered both cloud-hosted and self-hosted, users can have team projects and save the backup of experimentation history.

Comet is converging towards more automated approaches to ML, by predictive early stopping (not available with the free version of the software) and Neural architecture search (in the future).

Comet—summary:

  • Deals with user management 
  • Sharing work in a team: multiple features for sharing in a team.
  • Integrations with other tools: should be developed by the user manually
  • SaaS/Local instance available: Yes/Yes
  • Bonus: Display parallel plots to check patterns in the relationships between parameters and metrics

MLflow vs Comet

Comet comes with user management features and allows for sharing projects within the team—something that is missing in MLfow. It also offers both, hosted and on-premises setup, while MLflow is only available as an open-source solution that requires you to maintain it on your own server.  

Want to check other comparisons?

4. Valohai

Valohai takes a slightly different approach when it comes to tracking and visualizing experiments.

The platform proposes orchestration, version control, and pipeline management for machine learning – simply speaking they cover what MLflow is doing in terms of logging and additionally manage your compute infrastructure.

As was the case in MLflow users can easily check and compare multiple runs. At the same time, the differentiator is the ability to automate starting and shutting down cloud machines used for training.

Valohai lets you develop in any programming language – including Python and R – which can be handy in a team working in a fixed technological stack.

Valohai—summary:

  • Deals with user management
  • Sharing work in team: multiple features
  • Integrations with other tools: examples of integrations provided in the documentation
  • SaaS/Local instance available: Yes/Yes
  • Bonus: With the infrastructure for training you can run experiments on the environment managed by Valohai.

MLflow vs Valohai

As per Valohai’s own comparison, Valohai provides MLflow-like experiment tracking without any setup. Similar to MLflow, Valohai covers a big part of the MLOps landscape (including experiment tracking, model management, machine orchestration, and pipeline automation), but it is a managed platform, rather than an open-source solution. 

5. TensorBoard

TensorBoard is an open-source visualization toolkit for TensorFlow that lets you analyze model training runs. It’s often the first choice of TensorFlow users. TensorBoard allows you to visualize various aspects of machine learning experiments, such as metrics or model graphs, as well as view tensors’ histograms and more.

Apart from the popular, open-source version of TensorBoard, there’s also TensorBoard.dev which is available on a managed server as a free service.

TensorBoard.dev lets you upload and share your ML experiment results with anyone. It’s an important upgrade in comparison to TensorBoard, ad the collaboration features are missing there. 

TensorBoard—summary:

  • Well-developed features related to working with images
  • The What-If Tool (WIT), that’s an easy-to-use interface for expanding understanding of black-box classification and regression ML models
  • Strong and big community of users that provide community support.

MLflow vs TensorBoard

Both tools are open-source and supported by their respective communities in terms of dealing with any issues and questions. The main difference seems to be the range of features each of them provides. TensorBoard is described as a visualization kit for TensorFlow, so it serves well for visualizations, it allows you to track experiments and compare them (limited capability). MLflow, on the other hand, proves to be useful in many more stages of the ML lifecycle. Both tools are lacking user management and team sharing features (sharing is available in TensorBoard.dev, but there’s no possibility to manage the privacy of data there).

Want to check other comparisons?

Conclusion

MLflow is a great tool, but there are certain capabilities it doesn’t have. So it’s worth checking what else is available out there. In this overview, we mentioned 5 tools that could be good alternatives and check the missing boxes.

If your main reason to look for MLflow alternatives are missing collaboration and user management features, you should check Neptune, Weights & Biases, Comet, or Valohai. All of them are also available as hosted applications if you don’t want to maintain the experiment tracking tool yourself. 

If you want to stick to open-source tools, TensorBoard may be the tool for you, but you should keep in mind that it’s less advanced than MLflow in terms of features. 

Finally, if you don’t need a tool that covers almost an entire ML lifecycle (like MLflow or Valohai), we recommend you to check Neptune, Weight & Biases, or Comet. 

In any case, make sure that the alternative solution matches your needs and improves your workflow. Hope this article helps you find it. Good luck!


READ NEXT

15 Best Tools for ML Experiment Tracking and Management

10 mins read | Author Patrycja Jenkner | Updated August 25th, 2021

While working on a machine learning project, getting good results from a single model-training run is one thing. But keeping all of your machine learning experiments well organized and having a process that lets you draw valid conclusions from them is quite another. 

The answer to these needs is experiment tracking. In machine learning, experiment tracking is the process of saving all experiment-related information that you care about for every experiment you run. 

ML teams implement experiment tracking in different ways, may it be by using spreadsheets, GitHub, or self-built platforms. Yet, the most effective option is to do it with tools designed specifically for tracking and managing ML experiments.

In this article, we overview and compare the 15 best tools that will allow you to track and manage your ML experiments. You’ll get to know their main features and see how they are different from each other. Hopefully, this will help you evaluate them and choose the right one for your needs. 

How to evaluate an experiment tracking tool? 

There’s no one answer to the question “what is the best experiment tracking tool?”. Your motivation and needs may be completely different when you work individually or in a team. And, depending on your role, you may be looking for various functionalities. 

If you’re a Data Scientist or a Researcher, you should consider: 

  • If the tool comes with a web UI or it’s console-based;
  • If you can integrate the tool with your preferred model training frameworks;
  • What metadata you can log, display, and compare (code, text, audio, video, etc.);
  • Can you easily compare multiple runs? If so, in what format – only table, or also charts;
  • If organizing and searching through experiments is user-friendly;
  • If you can customize metadata structure and dashboards;
  • If the tool lets you track hardware consumption;
  • How easy it is to collaborate with other team members – can you just share a link to the experiment or you have to use screenshots as a workaround?

As an ML Engineer, you should check if the tool lets you: 

  • Easily reproduce and re-run experiments;
  • Track and search through experiment lineage (data/models/experiments used downstream); 
  • Save, fetch, and cache datasets for experiments;
  • Integrate it with your CI/CD pipeline;
  • Easily collaborate and share work with your colleagues.

Finally, as an ML team lead, you’ll be interested in:

  • General business-related stuff like pricing model, security, and support;
  • How much infrastructure the tool requires, how easy it is to integrate it into your current workflow;
  • Is the product delivered as commercial software, open-source software, or a managed cloud service?
  • What collaboration, sharing, and review feature it has. 

I made sure to keep these motivations in mind when reviewing the tools that are on the market. So let’s take a closer look at them. 

Continue reading ->
MLflow vs. Tensorboard vs. Neptune

MLflow vs. TensorBoard vs. Neptune – What Are the Differences?

Read more
Best MLOps tools

The Best MLOps Tools and How to Evaluate Them

Read more
MLflow share and collaborate

How to Make your MLflow Projects Easy to Share and Collaborate on

Read more
MLOps guide

MLOps: What It Is, Why it Matters, and How To Implement It

Read more