We Raised $8M Series A to Continue Building Experiment Tracking and Model Registry That “Just Works”
Neptune vs Kubeflow
vs
Commercial Requirements
Standalone component or a part of a broader ML platform?
Standalone component. ML metadata store that focuses on experiment tracking and model registry
Part of the Kubernetes environment
Is the product available on-premises and / or in your private/public cloud?
Almost all popular cloud providers maintain their own distribution of Kubeflow. It can also be installed on-premises manually. Read about the differejnt installation options available here
Is the product delivered as commercial software, open-source software, or a managed cloud service?
Managed cloud service
The base product is open source, with managed distributions made available by cloud providers
What is the pricing model?
The open source version is free. Maintainers have their own priving plans for their managed distributions
SLOs / SLAs: Does the vendor provide guarantees around service levels?
Email and chat support for individuals and teams. Well defined support SLAs for enterprise customers
Community support for the open source version. Different options available for managed distributions
Support: Does the vendor provide 24×7 support?
Not for the open source version. Different options available for managed distributions
SSO, ACL: does the vendor provide user access management?
Different options available for managed distributions. Read more here
Security policy and compliance
Not for the open source version. Different options available for managed distributions
General Capabilities
What are the infrastructure requirements?
No special requirements other than having the neptune-client installed and access to the internet if using managed hosting. Check here for infrastructure requirements for on-prem deployment
Advanced setup required for manual install. Installation requirements for the managed distribution varies depending on the cloud provider. Read more here
How much do you have to change in your training process?
Minimal. Just a few lines of code needed for tracking. Read more
Extensive code and infrastructure changes are required. Check out a few examples here
Does it integrate with the training process via CLI/YAML/Client library?
Yes, through the neptune-client library
N/A
How can you access model metadata?
– gRPC API
No
No
– Java SDK
No
No
– Julia SDK
No
No
Supported operations
Logging modes
Mobile support
No
No
Webhooks and notifications
No
No
Experiment Tracking
Dataset
– Dataset slicing support
No
No
Code versions
Metrics and losses
Rich format
– Neural Network Histograms
No
N/A
– Prediction visualization (image)
No
No
Hardware consumption
– TPU
No
No
System information
– Execution command
No
No
Environment config
Files
– External file reference (s3 buckets)
No
N/A
Parameters and metrics
– Parameter Importance plot
No
No
– Slice plot
No
No
– EDF plot
No
No
Rich format (side by side)
– Video
No
No
– Audio
No
No
– Plots
No
No
– Interactive visualization (HTML)
No
No
– Neural Network Histograms
No
No
– Prediction visualization (image, video, audio)
No
No
Code
– Git
No
No
– Source files
No
No
– Notebooks
Yes
No
Environment
– pip requirements.txt
No
No
– conda env.yml
No
No
– Docker Dockerfile
No
No
Hardware
System information
– Execution command
No
No
Data versions
– Dataset diff
No
No
– External reference version diff (s3)
No
No
Files
– Models
No
No
– CSV
No
No
Custom compare dashboards
Experiment table customization
– Renaming columns in the UI
Yes
No
– Adding colors to columns
Yes
No
Experiment filtering and searching
– Query language vs fixed selectors
Query language
N/A
Custom dashboards for a single experiment
One-command experiment re-run
No
No
Experiment lineage
– List of datasets used downstream
No
No
– List of other artifacts (models) used downstream
No
Yes
– Downstream artifact dependency graph
No
Yes
Reproducibility protocol
Limited
N/A
Saving/fetching/caching datasets for experiments
No
N/A
Interactive project-level reports
No
N/A
Model Registry
Environment versions
No
N/A
Model files (packaged models, model weights, pointers to artifact storage)
No
N/A
Models/experiments created downstream
No
Yes
History of evaluation/testing runs
No
N/A
Support for continuous testing
No
N/A
Users who created a model or downstream experiments
No
N/A
Main stage transition tags (develop, stage, production)
No
N/A
Custom stage tags
No
N/A
Locking model version and downstream runs, experiments, and artifacts
No
N/A
Adding annotations/comments and approvals from the UI
No
N/A
Model compare (current vs challenger etc)
No
N/A
Compatibility audit (input/output schema)
No
N/A
Compliance audit (datasets used, creation process approvals, results/explanations approvals)
No
N/A
Webhooks
No
N/A
Model accessibility
No
N/A
Support for continuous testing
No
N/A
Integrations with CI/CD tools
No
N/A
Registered models
No
N/A
Active models
No
N/A
By metadata/artifacts used to create it
No
N/A
By date
No
N/A
By user/owner
No
N/A
By production stage
No
N/A
Search query language
No
N/A
Native packaging system
No
N/A
One model one file or flexible structure
No
N/A
Integrations and Support
FBProphet
No
No
Gluon
No
No
H2O
No
No
Paddle
No
No
Spacy
No
No
Spark MLlib
No
No
Statsmodel
No
No
Ray Tune
No
No
Netron
No
No
This page was updated on 24 October 2021. Some information may be outdated.
Report outdated information here.
What are the key differences, then?
- Neptune is a paid hosted metadata store with the main focus on experiment tracking and model registry
- Kubeflow is an open-source project created to enable easier deployment of ML workflows on Kubernetes
Neptune and Kubeflow are not mutually exclusive. In fact, Neptune can serve as a great solution for experiment management and model registry inside the Kubeflow Pipelines.
Check how to start using it.
See Neptune features in action
1. Create a free account
Sign up2. Install Neptune client library
pip install neptune-client
3. Add logging to your script
import neptune.new as neptune
run = neptune.init('Me/MyProject')
run['params'] = {'lr':0.1, 'dropout':0.4}
run['test_accuracy'] = 0.84
4. Or see how it works in a notebook (no registration)
Try live notebookThousands of ML people already chose their tool
“Previously used tensorboard and azureml but Neptune is hugely better. In particular, getting started is really easy; documentation is excellent, and the layout of charts and parameters is much clearer.”
AI Engineer and Data Scientist
“(…) thanks for the great tool, has been really useful for keeping track of the experiments for my Master’s thesis. Way better than the other tools I’ve tried (comet / wandb).
I guess the main reason I prefer neptune is the interface, it is the cleanest and most intuitive in my opinion, the table in the center view just makes a great deal of sense. I like that it’s possible to set up and save the different view configurations as well. Also, the comparison is not as clunky as for instance with wandb. Another plus is the integration with ignite, as that’s what I’m using as the high-level framework for model training.”
Data Science and AI student, Kranenburg, Germany
“This thing is so much better than Tensorboard, love you guys for creating it!”
Junior Researcher at Rényi Alfréd Institute of Mathematics in Budapest, Hungary
“While logging experiments is great, what sets Neptune apart for us at the lab is the ease of sharing those logs. The ability to just send a Neptune link in slack and letting my coworkers see the results for themselves is awesome. Previously, we used Tensorboard + locally saved CSVs and would have to send screenshots and CSV files back and forth which would easily get lost. So I’d say Neptune’s ability to facilitate collaboration is the biggest plus.”
Computer Science Undergraduate at Saint Louis University
“Such a fast setup! Love it:)”
PhD student in Music Information Processing at Télécom Paris
“For me the most important thing about Neptune is its flexibility. Even if I’m training with Keras or Tensorflow on my local laptop, and my colleagues are using fast.ai on a virtual machine, we can share our results in a common environment.”
Senior NLP/ML Engineer