We Raised $8M Series A to Continue Building Experiment Tracking and Model Registry That “Just Works”
Neptune vs MLflow
Scalable MLflow alternative with a hosted UI built for teamwork
vs
Looking for an experiment tracking and/or model registry solution? Not sure how Neptune.ai and MLflow are actually different? Or maybe you’re using MLflow but wanted to check some alternatives?
You’re in the right place. On this page we break down:
- what are the differences between MLflow and Neptune
- why do people switch from MLflow to Neptune
- what are the cases when you should go with Neptune (and when you shouldn’t)
When should you go with Neptune?
Why others choose Neptune over MLflow?
MLflow is open-source and free to use. You can customize it if you want to, but you have to set it up and maintain the infrastructure.
Neptune.ai is a hosted SaaS solution (an on-prem option is also available). It’s easy to plug it into your workflow and it comes with user support.
So, in this aspect, it’s the good, old “buy vs use open-source” dilemma. Many ML people we talked to have a general impression that open source tools are good enough for very small teams or individual contributors with a need for basic tracking. After some time you just get to the point when it’s not enough.
Now when you look into what parts of the ML lifecycle both tools cover, the scope is quite similar.
MLflow does experiment tracking and model registry, but also comes with two additional components: Projects (packaging format) and Models (general format for sending models to deployment tools).
Neptune.ai focuses on experiment tracking and model registry, but it doesn’t limit its functionality. Within those features, it covers also dataset versioning, model versioning, or monitoring model training live. On top of that, thanks to the flexible API you can package models however you like and attach any metadata in any structure to a model version.
This flexibility is often brought up by Neptune users, as one of the first advantages they notice. The tool easily hooks into multiple frameworks, so no matter what solution you use for training, deploying, or monitoring models, Neptune will fit in.
There’s one more thing that will probably be important when comparing Neptune and MLflow, especially for the teams.
Neptune comes with out-of-the-box collaboration features.
- You can share persistent links to the UI with other team members, or with external people if needed.
- And you can create multiple projects and manage who can access them.
There are teams that use the open-source MLflow, but the collaboration is much more challenging there. Team management features are available in the Managed MLflow which is a paid solution developed by Databricks. So probably this kind of functionality won’t be added to the free, open-source version.
MLflow is a solid solution, and it checks many boxes when the team needs a simple experiment tracking and model registry. But it doesn’t scale well. Many ML teams quickly reach its limits and look for something more advanced.
Here’s what Neptune users say
“For now, I’m not using MLflow anymore ever since I switched to Neptune because I feel like Neptune is a superset of what MLflow has to offer.” [Read full case study]
Senior Data Scientist at Zoined
“Previously used tensorboard and azureml but Neptune is hugely better. In particular, getting started is really easy; documentation is excellent, and the layout of charts and parameters is much clearer.”
AI Engineer and Data Scientist
“(…) thanks for the great tool, has been really useful for keeping track of the experiments for my Master’s thesis. Way better than the other tools I’ve tried (comet / wandb).
I guess the main reason I prefer neptune is the interface, it is the cleanest and most intuitive in my opinion, the table in the center view just makes a great deal of sense. I like that it’s possible to set up and save the different view configurations as well. Also, the comparison is not as clunky as for instance with wandb. Another plus is the integration with ignite, as that’s what I’m using as the high-level framework for model training.”
Data Science and AI student, Kranenburg, Germany
“This thing is so much better than Tensorboard, love you guys for creating it!”
Junior Researcher at Rényi Alfréd Institute of Mathematics in Budapest, Hungary
“While logging experiments is great, what sets Neptune apart for us at the lab is the ease of sharing those logs. The ability to just send a Neptune link in slack and letting my coworkers see the results for themselves is awesome. Previously, we used Tensorboard + locally saved CSVs and would have to send screenshots and CSV files back and forth which would easily get lost. So I’d say Neptune’s ability to facilitate collaboration is the biggest plus.”
Computer Science Undergraduate at Saint Louis University
“Such a fast setup! Love it:)”
PhD student in Music Information Processing at Télécom Paris
“For me the most important thing about Neptune is its flexibility. Even if I’m training with Keras or Tensorflow on my local laptop, and my colleagues are using fast.ai on a virtual machine, we can share our results in a common environment.”
Senior NLP/ML Engineer
Give Neptune a try
2
Install Neptune client library
pip install neptune-client
3
Add logging to your script
import neptune.new as neptune
run = neptune.init('Me/MyProject')
run['params'] = {'lr':0.1,
'dropout':0.4}
run['test_accuracy'] = 0.84 Dig deeper into the differences between Neptune and MLflow
Commercial Requirements
Standalone component or a part of a broader ML platform?
Standalone component. ML metadata store that focuses on experiment tracking and model registry
Open-source platform which offers four separate components for experiment tracking, code packaging, model deployment, and model registry
Is the product available on-premises and / or in your private/public cloud?
Tracking is hosted on a local/remote server (on-prem or cloud). Is also available on a managed server as part of the Databricks platform
Is the product delivered as commercial software, open-source software, or a managed cloud service?
Managed cloud service
The standalone product is open-source, while the Databricks managed version is commercial
What is the pricing model?
SLOs / SLAs: Does the vendor provide guarantees around service levels?
Email and chat support for individuals and teams. Well defined support SLAs for enterprise customers
Community support for the open-source version, various support plans for the databricks managed version
SSO, ACL: does the vendor provide user access management?
General Capabilities
What are the infrastructure requirements?
No special requirements other than having the neptune-client installed and access to the internet if using managed hosting. Check here for infrastructure requirements for on-prem deployment
No requirements other than having mlflow installed if using a local tracking server. Check here for infrastructure requirements for using a remote tracking server
How much do you have to change in your training process?
Minimal. Just a few lines of code needed for traking. Read more
Minimal. Just a few lines of code needed for traking. Read more
Does it integrate with the training process via CLI/YAML/Client library?
Yes, through the neptune-client library
Yes, they have SDK clients in various languages, and a CLI
Serverless UI
No
Yes
How can you access model metadata?
– Julia SDK
No
No
Supported operations
Logging modes
Mobile support
No
No
Experiment Tracking
Dataset
– Dataset slicing support
No
No
Code versions
Metrics and losses
Rich format
– Neural Network Histograms
No
No
– Prediction visualization (tabular)
No
No
– Prediction visualization (image)
No
No
Hardware consumption
– TPU
No
No
System information
– Execution command
No
Yes
Environment config
Files
– Any file
– External file reference (s3 buckets)
No
No
Parameters and metrics
– Parameter Importance plot
No
No
– Slice plot
No
No
– EDF plot
No
No
Rich format (side by side)
– Video
No
No
– Audio
No
No
– Plots
No
No
– Interactive visualization (HTML)
No
No
– Neural Network Histograms
No
No
– Prediction visualization (image, video, audio)
No
No
Code
– Git
No
No
– Source files
No
No
– Notebooks
Yes
No
Environment
– pip requirements.txt
No
No
– conda env.yml
No
No
– Docker Dockerfile
No
No
Hardware
System information
– Execution command
No
No
Data versions
– Dataset diff
No
No
– External reference version diff (s3)
No
No
Files
– Models
No
No
– CSV
No
No
Custom compare dashboards
Experiment table customization
– Renaming columns in the UI
Yes
No
– Adding colors to columns
Yes
No
Experiment filtering and searching
– Query language vs fixed selectors
Query language
Query language
Custom dashboards for a single experiment
– Custom widgets and plugins
No
No
Experiment lineage
– List of datasets used downstream
No
No
– List of other artifacts (models) used downstream
No
No
– Downstream artifact dependency graph
No
No
Saving/fetching/caching datasets for experiments
No
No
Interactive project-level reports
No
No
Model Registry
Models/experiments created downstream
No
No
History of evaluation/testing runs
No
No
Support for continuous testing
No
No
Users who created a model or downstream experiments
No
No
Custom stage tags
No
No
Locking model version and downstream runs, experiments, and artifacts
No
No
Adding annotations/comments and approvals from the UI
No
Compliance audit (datasets used, creation process approvals, results/explanations approvals)
No
No
Support for continuous testing
No
No
Integrations with CI/CD tools
No
No
Active models
No
No
Search query language
No
No
Integrations and Support
FBProphet
No
No
Hyperopt
No
No
This table was updated on 14 June 2022. Some information may be outdated.
Report outdated information here.
It only takes 5 minutes to integrate Neptune with your code
Have more questions? Let’s talk
Vovik Tsybulski
Head of Sales
Schedule a call