MLflow is an open-source platform that helps manage the whole machine learning lifecycle. This includes experimentation, but also reproducibility, deployment, and storage. Each of these four elements is represented by one MLflow component: Tracking, Projects, Models, and Registry.
That means a data scientist who works with MLflow is able to track an experiment, organize it, describe it for other ML engineers and pack it into a machine learning model. In this article, we focus mostly on the experiment tracking capabilities of MLflow and overview the best alternatives for that.
While MLflow is a great tool, some things could be better especially when working in a larger team and/or the number of experiments you run is very large.
What are the main concerns you may have? What are the main MLflow weaknesses?
- Missing user management capabilities make it difficult to deal with access permissions to different projects or roles (manager/machine learning engineer). Because of that, and no option to share UI links with other people, team collaboration is also challenging in MLflow.
Read also
- Tracking UI, though improved recently, doesn’t give you full customizability when it comes to saving experiment dashboard views or grouping runs by experiment parameters (model architecture) or properties (data versions). Those are very useful when you have multiple people working on the same project or you are running thousands of experiments.
- Speaking of large numbers of experiments, the UI can get quite slow when you really want to explore all your runs.
- Unless you want to use the Databricks platform you need to maintain the MLflow server yourself. That comes with typical hurdles like access management, backups, and so on, not to mention that it’s very time-consuming.
- The open-source community is vibrant but there is no dedicated user support to hold your hand when you need it.
- MLflow is great for running experiments via Python or R scripts but the Jupyter notebook experience is not perfect, especially if you want to track some additional segments of the machine learning lifecycle like exploratory data analysis or results exploration.
- Some functionalities like logging resource consumption (CPU, GPU, Memory) or scrolling through large numbers of image predictions or charts are not there yet.
ML experiment tracking capabilities in MLflow give a ton of value to individual users or teams that are willing to maintain experiment data backend, tracking UI server, and are not running huge numbers of experiments.
If some of the things mentioned above are important to you and your team you may want to look for complementary or alternative tooling. Luckily, there are many tools that offer some or most of those missing pieces.
In this article, building on top of some reddit discussions, and our comparison, we present top alternatives to MLflow.
In our opinion, the following are the best alternatives to MLflow:
1. Neptune
Neptune is a metadata store—it serves as a connector between different parts of the MLOps workflow from data versioning, experiment tracking to model registry and monitoring. Neptune makes it easy to store, organize, display, and compare all metadata generated during the ML model lifecycle.
Neptune makes it easy to store, organize, display, and compare all metadata generated during the ML model lifecycle.
You can log metrics, hyperparameters, interactive visualizations, videos, code, data versions, and more, and organize it in a customized structure. Once logged, everything is visible in an intuitive and clean UI, where you can analyze and compare it.
You can also create custom dashboards that include all this metadata and share them with your colleagues, team manager, or even external stakeholders. Here’s an example of such a dashboard:
There are four different comparison views available – charts, parallel coordinates, side-by-side tabular dashboard, and artifacts comparison section. So you can easily evaluate models and select the best-performing ones.
Neptune can also be extremely useful in the production phase. With all the logged metadata, you know how the model was created and how to reproduce it.
Neptune—summary:
- Team and user management features that allow you to create workspaces and projects and assign roles within these projects.
- Available in the on-premises setup but also as a hosted app if you want to avoid all the hussle with maintaining yet another tool.
- Fast, responsive, scalable, and highly customizable UI with a lot of capabilities to organize runs in groups, save custom dashboard views and share them with the team (e.g. via sharable UI links).
- Out-of-the-box integrations with 25+ with MLOps tools, including Jupyter Notebooks, PyTorch ecosystem, Optuna, Kedro, and more.
If you want to see Neptune in action, check this live Notebook or this example project (no registration is needed) and just play with it.
MLflow vs Neptune
The main difference between these tools is that MLflow is an open-source solution while Neptune is a managed cloud service. It affects various aspects of how MLflow and Neptune work. If you’re looking for a free, open-source tool, that covers a wide range of ML lifecycle steps, MLflow might be the right choice for you. But you should keep in mind, that even though MLflow is free to download, it does generate costs related to maintaining the whole infrastructure.
If you prefer to focus on the ML process and leave hosting to someone else, Neptune is the way to go. For the monthly fee, you get excellent user support, quick & easy setup, you don’t have to worry about maintenance, and the tool scales well. Plus, Neptune has user management features, so it will work better in the team environment.
Want to dig deeper?
See an in-depth comparison between Neptune and MLflow.
Read the case study of Zoined to learn why they chose Neptune over MLflow.
2. Weights & Biases
Weights & Biases a.k.a. WandB is focused on deep learning. Users track experiments to the application with Python library, and – as a team – can see each other’s experiments.
Unlike MLflow, WandB is a hosted service allowing you to backup all experiments in a single place and work on a project with the team – work sharing features are there to use.
Similarly to MLflow, in the WandB users can log and analyze multiple data types.
Weights & Biases—summary:
- Deals with user management
- Great UI allows users to visualize, compare and organize their runs nicely.
- Sharing work in a team: multiple features for sharing in a team.
- Integrations with other tools: several open source integrations available
- SaaS/Local instance available: Yes/Yes
- Bonus: WandB logs the model graph, so you can inspect it later.
MLflow vs Weights & Biases
Similar to Neptune, Weight & Biases offers a hosted version of its tool. In opposite to MLflow, which is open-sourced, and needs to be maintained on your own server. Weights & Biases provides features for experiment tracking, dataset versioning, and model management, while MLflow covers almost the entire ML lifecycle. Finally, WandB offers user management features that are probably important for you when working in a team.
Want to check other comparisons?
See an in-depth comparison between Neptune and Weights & Biases.
3. Comet
Comet is a meta machine learning platform for tracking, comparing, explaining, and optimizing experiments and models.
Just like many other tools – for example, Neptune (neptune-client specifically) or WandB – Comet proposes an open-source Python library to allow data scientists to integrate their code with Comet and start tracking work in the application.
As it’s offered both cloud-hosted and self-hosted, users can have team projects and save the backup of experimentation history.
Comet is converging towards more automated approaches to ML, by predictive early stopping (not available with the free version of the software) and Neural architecture search (in the future).
Comet—summary:
- Deals with user management
- Sharing work in a team: multiple features for sharing in a team.
- Integrations with other tools: should be developed by the user manually
- SaaS/Local instance available: Yes/Yes
- Bonus: Display parallel plots to check patterns in the relationships between parameters and metrics
MLflow vs Comet
Comet comes with user management features and allows for sharing projects within the team—something that is missing in MLfow. It also offers both, hosted and on-premises setup, while MLflow is only available as an open-source solution that requires you to maintain it on your own server.
Want to check other comparisons?
See an in-depth comparison between Neptune and Comet.
4. Valohai
Valohai takes a slightly different approach when it comes to tracking and visualizing experiments.
The platform proposes orchestration, version control, and pipeline management for machine learning – simply speaking they cover what MLflow is doing in terms of logging and additionally manage your compute infrastructure.
As was the case in MLflow users can easily check and compare multiple runs. At the same time, the differentiator is the ability to automate starting and shutting down cloud machines used for training.
Valohai lets you develop in any programming language – including Python and R – which can be handy in a team working in a fixed technological stack.
Valohai—summary:
- Deals with user management
- Sharing work in team: multiple features
- Integrations with other tools: examples of integrations provided in the documentation
- SaaS/Local instance available: Yes/Yes
- Bonus: With the infrastructure for training you can run experiments on the environment managed by Valohai.
MLflow vs Valohai
As per Valohai’s own comparison, Valohai provides MLflow-like experiment tracking without any setup. Similar to MLflow, Valohai covers a big part of the MLOps landscape (including experiment tracking, model management, machine orchestration, and pipeline automation), but it is a managed platform, rather than an open-source solution.
5. TensorBoard
TensorBoard is an open-source visualization toolkit for TensorFlow that lets you analyze model training runs. It’s often the first choice of TensorFlow users. TensorBoard allows you to visualize various aspects of machine learning experiments, such as metrics or model graphs, as well as view tensors’ histograms and more.
Apart from the popular, open-source version of TensorBoard, there’s also TensorBoard.dev which is available on a managed server as a free service.
TensorBoard.dev lets you upload and share your ML experiment results with anyone. It’s an important upgrade in comparison to TensorBoard, ad the collaboration features are missing there.
TensorBoard—summary:
- Well-developed features related to working with images
- The What-If Tool (WIT), that’s an easy-to-use interface for expanding understanding of black-box classification and regression ML models
- Strong and big community of users that provide community support.
MLflow vs TensorBoard
Both tools are open-source and supported by their respective communities in terms of dealing with any issues and questions. The main difference seems to be the range of features each of them provides. TensorBoard is described as a visualization kit for TensorFlow, so it serves well for visualizations, it allows you to track experiments and compare them (limited capability). MLflow, on the other hand, proves to be useful in many more stages of the ML lifecycle. Both tools are lacking user management and team sharing features (sharing is available in TensorBoard.dev, but there’s no possibility to manage the privacy of data there).
Want to check other comparisons?
See an in-depth comparison between Neptune and TensorBoard.
Conclusion
MLflow is a great tool, but there are certain capabilities it doesn’t have. So it’s worth checking what else is available out there. In this overview, we mentioned 5 tools that could be good alternatives and check the missing boxes.
If your main reason to look for MLflow alternatives are missing collaboration and user management features, you should check Neptune, Weights & Biases, Comet, or Valohai. All of them are also available as hosted applications if you don’t want to maintain the experiment tracking tool yourself.
If you want to stick to open-source tools, TensorBoard may be the tool for you, but you should keep in mind that it’s less advanced than MLflow in terms of features.
Finally, if you don’t need a tool that covers almost an entire ML lifecycle (like MLflow or Valohai), we recommend you to check Neptune, Weight & Biases, or Comet.
In any case, make sure that the alternative solution matches your needs and improves your workflow. Hope this article helps you find it. Good luck!
