Feel in control of your models and experiments by having all metadata organized in a single place.
Focus on ML, leave metadata bookkeeping to Neptune.
Get started in 5 minutes.
Metadata store for MLOps, built for research and production teams that run a lot of experiments
What is a metadata store for MLOps?
ML metadata store is an essential part of the MLOps stack that deals with model building metadata management.
It makes it easy to log, store, display, organize, compare and query all metadata generated during ML model lifecycle.
What ML metadata are we talking about?
Experiment and model training metadata
You can log anything that happens during ML run including:
- Metrics
- Hyperparameters
- Learning curves
- Training code and configuration files
- Predictions (images, tables, etc)
- Diagnostic charts (Confusion matrix, ROC curve, etc)
- Console logs
- Hardware logs
- And more
Artifact metadata
For datasets, predictions or models you can log:
- Paths to the dataset or model (s3 bucket, filesystem)
- Dataset hash
- Dataset/prediction preview (head of the table, snapshot of the image folder)
- Description
- Feature column names (for tabular data)
- Who created/modified
- When last modified
- Size of the dataset
- And more
Model metadata
For trained models (production or not) you can log:
- Model binary or location to your model asset
- Dataset versions
- Links to recorded model training runs and experiments
- Who trained the model
- Model descriptions and notes
- Links to observability dashboards (Grafana)
- And more
What can you use an ML metadata store for?
Experiment tracking
Track, organize, and compare everything you care about in your ML experiments.
- Monitor experiments as they are running
- Keep track of metrics, parameters, diagnostic charts, and more
- Search, group, and compare experiments with no effort
- Drill down to every experiment information you need
- Share results with your team and access all experiment details programmatically
Model registry
Have your models versioned, traceable and easily accessible.
- Know exactly how every model was built
- Record dataset, code, parameters, model binaries, and more for every training run
- Get back to every model building metadata even months after
- Share models with your team and access them programmatically
Neptune Metadata Store for MLOps = Client + Database + Dashboard
# log
experiment.log_metric('accuracy', 0.9)
experiment.log_image('predictions', image_pred)
experiment.log_artifact('model_weights.h5')
# query
project.get_experiment(id='EXP-123').download_artifacts()
Metadata Database
It’s a place where experiment, model, and dataset metadata are stored. They can be logged and queried efficiently.
Dashboard
A visual interface to the metadata database. A place where you can see metadata of your experiment, models, and datasets.
Get started in 5 minutes
1. Create a free account
Sign up2. Install Neptune client library
pip install neptune-client
3. Add logging to your script
import neptune
neptune.init('Me/MyProject')
neptune.create_experiment(params={'lr':0.1, 'dropout':0.4})
# training and evaluation logic
neptune.log_metric('test_accuracy', 0.84)
Try live notebook
Dive deeper into Neptune features
Log and display any ML metadata you care about
How can you start logging?
To log things to the Neptune experiment, you need
an experiment 🙂.
- Connect Neptune to your script (pass credentials)
- Create an experiment (start logging context)
- Log whatever you want
neptune.init('YOUR_ORG/PROJECT', 'YOUR_API_TOKEN') # Credentials
neptune.create_experiment('great-idea') # Start logging context
neptune.log_metric('accuracy', 0.92) # Log things
What can you log and display?
Neptune allows you to log most metadata related to ML models, experiments, and datasets, including:
- Training code and configuration files
- Parameters and metrics
- Hardware consumption and console logs
- Performance charts and interactive visualizations
- Model weights
- and more!
Log more with less code with our integrations!
Don’t implement the loggers yourself. We have 25 + integrations with all major ML frameworks to make things even easier.
- PyTorch and PyTorch Lightning
- TensorFlow / Keras and TensorBoard
- Scikit-Learn, LightGBM, and XGBoost
- Optuna, Scikit Optiimize, and Keras Tuner
- Bokeh, Altair, Plotly, and Matplotlib
- and more!
#Tensorflow/Keras
model.fit(..., callbacks=[NeptuneMonitor()])
# PyTorch lightning
trainer = pl.Trainer(..., logger=NeptuneLogger())
# lightGBM
gbm = lgb.train(... callbacks = [neptune_monitor()])
# XGBoost
xgb.train(..., callbacks=[neptune_callback()])
# Optuna
study.optimize(…, callbacks=[NeptuneCallback()])
Compare experiments and models with no extra effort
Experiment auto-diff in a table
Compare metrics and parameters in a table that automatically finds what changed between experiments.
See the difference between metrics, parameters, text, and more with no extra effort!
Compare learning curves
Overlay learning curves and drill down to experiment details like interactive performance charts to get a better picture.
Get all that automatically just because you logged metadata to Neptune.
Organize experiments and model training runs in a single place
Run experiments everywhere, keep the results in one place
You can execute experiment code on your laptop, cloud environment, or a cluster. All ML metadata you care about will be logged to the central storage hosted by us or deployed on-premises.
It works out-of-the-box with Python, R, Jupyter notebooks, and other languages and environments.
Structure your teams’ ML work in workspaces and projects
Clean up your ML teamwork by grouping your experiment into projects and workspaces.
If you are working on a model that does a particular feature, just create a project for it. All the experiments in that project will be easy to compare and search through.
If your team is working for multiple clients or departments, you can create a workspace for each client and separate projects in each organization.
Filter, sort, and compare experiments in a dashboard
Search through your experiments quickly with a dashboard built for ML experiments and models.
- Filter experiments by metric and parameter values
- Display min/max/last value of a metric or loss series like validation loss
- Compare everything with no extra effort
See (only) the information you want: customize and save dashboard views
Choose which metrics, parameters, or other information you want to see and customize your dashboard.
Create multiple dashboard views and save them for later.
See your ML experiments live as they are running
See learning curves live
Take a look at the learning curves and training metrics of your models and compare them to past runs as they are running.
Detect unpromising training runs and react quickly.
See hardware consumption whenever you want
You can monitor the hardware for your experiment runs automatically. See how much GPU/CPU and memory your model training runs are using.
See performance problems quickly, even for multi-gpu training.
Look at model predictions, console logs, and anything else during training
You can log model predictions after every epoch or console logs from a remote machine and have full control over your training process.
Make experiments and model training runs reproducible and traceable
Every time you train your model, you can automatically record:
- Code and environment versions
- Data Versions
- Model parameters
- Model weights and other binaries
- Evaluation metrics,
- Model explanation charts,
- Who trained the model,
- and anything else you need
You will not forget to commit your changes because it happens automatically.
neptune.create_experiment(
params={'lr':0.21,
'data_version': get_md5('data/train.csv')},
upload_source_files=['**.*.py',
'requirements.yaml')
neptune.log_metric('acc', 0.92)
neptune.log_image('explanations', explanation_fig)
neptune.log_artifact('model.pkl')
Share any result or visualizations with your team by sending a link
Want to discuss what you see right now in the application?
Just share a link. It’s that simple.
Neptune has a system of persistent links that makes sharing experiment details, comparisons, dashboard views, or anything else straightforward.
Query experiment and model training metadata programmatically
You can download everything you logged to Neptune programmatically (or from the UI).
- Experiment table with all metrics and parameters
- Experiment object itself and access all the metadata logged to it
- You can update experiment objects with new information even after training!
project = neptune.init('Project')
project.get_leaderboard()
exp = project.get_experiments(id='Proj-123')[0]
exp.get_parameters()
exp.download_artifact('model.pkl')
exp.download_sources()
Neptune integrates with your favourite frameworks and tools
With Neptune you can get more out of tools you use
every day. Neptune comes with 25+ integrations with
libraries used in machine learning, deep learning and
reinforcement learning.
See what our users are saying
“Neptune allows us to keep all of our experiments organized in a single space. Being able to see my team’s work results any time I need makes it effortless to track progress and enables easier coordination.”
VP, Machine Learning @Zesty.ai
“For me the most important thing about Neptune is its flexibility. Even if I’m training with Keras or Tensorflow on my local laptop, and my colleagues are using fast.ai on a virtual machine, we can share our results in a common environment.”
Senior NLP/ML Engineer @reply.ai
“Within the first few tens of runs, I realized how complete the tracking was – not just one or two numbers, but also the exact state of the code, the best-quality model snapshot stored to the cloud, the ability to quickly add notes on a particular experiment. My old methods were such a mess by comparison.”
Enterprise Data Scientist @Intel
“Neptune is making it easy to share results with my teammates. I’m sending them a link and telling what to look at, or I’m building a View on the experiments dashboard. I don’t need to generate it by myself, and everyone in my team have access to it.”
Head of Data Science @New Yorker
“Neptune is making it easy to share results with my teammates. I’m sending them a link and telling what to look at, or I’m building a View on the experiments dashboard. I don’t need to generate it by myself, and everyone in my team have access to it.”
Resarch Lead @Banacha Street
“Without information, I have in the Monitoring section I wouldn’t know that my experiments are running 10 times slower than they could. All of my experiments are being trained on separate machines which I can access only via ssh. If I would need to download and check all of this separately I would be rather discouraged. When I want to share my results I’m simply sending a link.”
Machine Learning Researcher @TensorCell
“Previously used tensorboard and azureml but Neptune is hugely better. In particular, getting started is really easy; documentation is excellent, and the layout of charts and parameters is much clearer.”
AI Engineer and Data Scientist
