We Raised $8M Series A to Continue Building Experiment Tracking and Model Registry That “Just Works”
Neptune vs MLflow
vs
Commercial Requirements
Standalone component or a part of a broader ML platform?
Standalone component. ML metadata store that focuses on experiment tracking and model registry
Open-source platform which offers four separate components for experiment tracking, code packaging, model deployment, and model registry
Is the product available on-premises and / or in your private/public cloud?
Tracking is hosted on a local/remote server (on-prem or cloud). Is also available on a managed server as part of the Databricks platform
Is the product delivered as commercial software, open-source software, or a managed cloud service?
Managed cloud service
The standalone product is open-source, while the Databricks managed version is commercial
What is the pricing model?
SLOs / SLAs: Does the vendor provide guarantees around service levels?
Email and chat support for individuals and teams. Well defined support SLAs for enterprise customers
Community support for the open-source version, various support plans for the databricks managed version
SSO, ACL: does the vendor provide user access management?
General Capabilities
What are the infrastructure requirements?
No special requirements other than having the neptune-client installed and access to the internet if using managed hosting. Check here for infrastructure requirements for on-prem deployment
No requirements other than having mlflow installed if using a local tracking server. Check here for infrastructure requirements for using a remote tracking server
How much do you have to change in your training process?
Minimal. Just a few lines of code needed for traking. Read more
Minimal. Just a few lines of code needed for traking. Read more
Does it integrate with the training process via CLI/YAML/Client library?
Yes, through the neptune-client library
Yes, they have SDK clients in various languages, and a CLI
Serverless UI
No
Yes
How can you access model metadata?
– Julia SDK
No
No
Supported operations
Logging modes
Mobile support
No
No
Experiment Tracking
Dataset
– Dataset slicing support
No
No
Code versions
Metrics and losses
Rich format
– Neural Network Histograms
No
No
– Prediction visualization (tabular)
No
No
– Prediction visualization (image)
No
No
Hardware consumption
– TPU
No
No
System information
– Execution command
No
Yes
Environment config
Files
– Any file
– External file reference (s3 buckets)
No
No
Parameters and metrics
– Parameter Importance plot
No
No
– Slice plot
No
No
– EDF plot
No
No
Rich format (side by side)
– Video
No
No
– Audio
No
No
– Plots
No
No
– Interactive visualization (HTML)
No
No
– Neural Network Histograms
No
No
– Prediction visualization (image, video, audio)
No
No
Code
– Git
No
No
– Source files
No
No
– Notebooks
Yes
No
Environment
– pip requirements.txt
No
No
– conda env.yml
No
No
– Docker Dockerfile
No
No
Hardware
System information
– Execution command
No
No
Data versions
– Dataset diff
No
No
– External reference version diff (s3)
No
No
Files
– Models
No
No
– CSV
No
No
Custom compare dashboards
Experiment table customization
– Renaming columns in the UI
Yes
No
– Adding colors to columns
Yes
No
Experiment filtering and searching
– Query language vs fixed selectors
Query language
Query language
Custom dashboards for a single experiment
– Custom widgets and plugins
No
No
Experiment lineage
– List of datasets used downstream
No
No
– List of other artifacts (models) used downstream
No
No
– Downstream artifact dependency graph
No
No
Saving/fetching/caching datasets for experiments
No
No
Interactive project-level reports
No
No
Model Registry
Models/experiments created downstream
No
No
History of evaluation/testing runs
No
No
Support for continuous testing
No
No
Users who created a model or downstream experiments
No
No
Custom stage tags
No
No
Locking model version and downstream runs, experiments, and artifacts
No
No
Adding annotations/comments and approvals from the UI
No
Model compare (current vs challenger etc)
No
No
Compliance audit (datasets used, creation process approvals, results/explanations approvals)
No
No
Support for continuous testing
No
No
Integrations with CI/CD tools
No
No
Active models
No
No
Search query language
No
No
Integrations and Support
FBProphet
No
No
Hyperopt
No
No
This page was updated on 13 August 2021. Some information may be outdated.
Report outdated information here.
What are the key advantages of Neptune, then?
- Easy, one-time setup to have all projects in one place
- Hosted and sharable environment
- Intuitive and user-friendly interface built for teamwork
- User management features
See these features in action
1. Create a free account
Sign up2. Install Neptune client library
pip install neptune-client
3. Add logging to your script
import neptune.new as neptune
run = neptune.init('Me/MyProject')
run['params'] = {'lr':0.1, 'dropout':0.4}
run['test_accuracy'] = 0.84
4. Or see how it works in a notebook (no registration)
Try live notebookAlready using MLflow?
Migrate mlruns in one command:
neptune mlflow /PATH/TO/MLflow_PROJECT
--project USER_NAME/PROJECT_NAME
Or connect a new project in 5 minutes.
You can use a cloud-hosted service or have Neptune on your own servers.
Want to know why others chose Neptune?
Zoined, a company behind an analytics solution for retailers, restaurants, and wholesalers, evaluated both Neptune and MLflow when searching for the experiment management solution.
Read about the biggest challenges they faced with MLflow, and why they decided to go with Neptune in the end.
Thousands of ML people already chose their tool
“For now, I’m not using MLflow anymore ever since I switched to Neptune because I feel like Neptune is a superset of what MLflow has to offer.”
Senior Data Scientist at Zoined
“Previously used tensorboard and azureml but Neptune is hugely better. In particular, getting started is really easy; documentation is excellent, and the layout of charts and parameters is much clearer.”
AI Engineer and Data Scientist
“(…) thanks for the great tool, has been really useful for keeping track of the experiments for my Master’s thesis. Way better than the other tools I’ve tried (comet / wandb).
I guess the main reason I prefer neptune is the interface, it is the cleanest and most intuitive in my opinion, the table in the center view just makes a great deal of sense. I like that it’s possible to set up and save the different view configurations as well. Also, the comparison is not as clunky as for instance with wandb. Another plus is the integration with ignite, as that’s what I’m using as the high-level framework for model training.”
Data Science and AI student, Kranenburg, Germany
“This thing is so much better than Tensorboard, love you guys for creating it!”
Junior Researcher at Rényi Alfréd Institute of Mathematics in Budapest, Hungary
“While logging experiments is great, what sets Neptune apart for us at the lab is the ease of sharing those logs. The ability to just send a Neptune link in slack and letting my coworkers see the results for themselves is awesome. Previously, we used Tensorboard + locally saved CSVs and would have to send screenshots and CSV files back and forth which would easily get lost. So I’d say Neptune’s ability to facilitate collaboration is the biggest plus.”
Computer Science Undergraduate at Saint Louis University
“Such a fast setup! Love it:)”
PhD student in Music Information Processing at Télécom Paris
“For me the most important thing about Neptune is its flexibility. Even if I’m training with Keras or Tensorflow on my local laptop, and my colleagues are using fast.ai on a virtual machine, we can share our results in a common environment.”
Senior NLP/ML Engineer
