Compare

Neptune vs ClearML

Neptune

ClearML

Commercial Requirements

Commercial requirements

Standalone component or a part of a broader ML platform?

Standalone component

Standalone open-source platform

Is the product available on-premises and / or in your private/public cloud?

Tracking is hosted on a managed server, and can also be deployed on-premises and in a public/private cloud server

Available both on-prem and on cloud

Is the product delivered as commercial software, open-source software, or a managed cloud service?

Managed cloud service

Available both as an open-source platform, and a managed cloud service

What is the pricing model?

Free individual and paid team/enterprise plans available

SLOs / SLAs: Does the vendor provide guarantees around service levels?

Email and chat support for individuals and teams. Well-defined support SLAs for enterprise customers

Community support for the free plan, email/slack for teams, white glove service for enterprises. SLAs only for paying teams and enterprises

Support: Does the vendor provide 24×7 support?

Only for enterprise customers

SSO, ACL: does the vendor provide user access management?

Only for enterprise customers

Security policy and compliance

Yes

Limited for the free plan, available for paying teams and enterprises

General Capabilities

Setup

What are the infrastructure requirements?

No special requirements other than having the neptune-client installed and access to the internet if using managed hosting. Check here for infrastructure requirements for on-prem deployment.

No special requirements other than having the clearml python package installed if using managed hosting. Check requirements here for self-hosting.

How much do you have to change in your training process?

Minimal. Just a few lines of code needed for tracking. Read more

Does it integrate with the training process via CLI/YAML/Client library?

Yes, through the neptune-client library

Yes, through the ClearML python package

Does it come with a web UI or is it console-based?

Web UI

Serverless UI

Flexibility, speed, and accessibility

Customizable metadata structure

Yes

How can you access model metadata?

– gRPC API

– CLI / custom API

– REST API

– Python SDK

– R SDK

– Java SDK

– Julia SDK

Supported operations

– Search

– Update

– Delete

– Download

Distributed training support

Yes

Pipelining support

Yes

Logging modes

– Offline

– Debug

– Asynchronous

– Synchronous

Live monitoring

Mobile support

Webhooks and notifications

Limited

Experiment Tracking

Log and display of metadata

Dataset

– location (path/s3)

– hash (md5)

– Preview table

– Preview image

– Preview text

– Preview rich media

– Multifile support

Code versions

– Git

– Source

– Notebooks

Parameters

Metrics and losses

– Single values

Yes

– Series values

Yes

– Series aggregates (min/max/avg/var/last)

Yes

Limited (only avg, min, and max)

Tags

Yes

Descriptions/comments

Yes

Rich format

– Images (support for labels and descriptions)

Yes

– Plots

Yes

– Interactive visualizations (widgets and plugins)

Yes

– Video

Yes

– Audio

Yes

– Neural Network Histograms

Only for Keras models

– Prediction visualization (tabular)

– Prediction visualization (image)

– Prediction visualization (image – interactive confusion matrix for image classification)

Yes

– Prediction visualization (image – overlayed prediction masks for image segmentation)

Only for the paid version

– Prediction visualization (image – overlayed prediction bounding boxes for object detection)

Only for the paid version

Hardware consumption

– CPU

Yes

– GPU

Yes

– TPU

– Memory

Yes

System information

– Console logs (Stderr, Stdout)

Yes

– Error stack trace

Yes

– Execution command

– System details (host, user, hardware specs)

Yes

Environment config

– pip requirements.txt

Yes

Can be logg e d, but not previewed

– conda env.yml

Yes

Can be logged, but not previewed

– Docker Dockerfile

Yes

Can be logged, but not previewed

Files

– Model binaries

Yes

– CSV

Yes

– Any file

All files can be logged, but some formats might not be rendered

All files can be logged, but some might not be rendered

– External file reference (s3 buckets)

Yes

Comparing experiments

Table format diff

Yes

Overlayed learning curves

Yes

Parameters and metrics

– Groupby on experiment values (parameters)

Yes

– Parallel coordinates plots

Yes

– Parameter Importance plot

Rich format (side by side)

– Image

– Video

– Audio

– Plots

– Interactive visualization (HTML)

– Text

Yes

– Neural Network Histograms

– Prediction visualization (tabular)

Yes

– Prediction visualization (image, video, audio)

Limited

Code

– Git

Yes

– Source files

Yes

– Notebooks

Yes

Environment

– pip requirements.txt

– conda env.yml

– Docker Dockerfile

Hardware

– CPU

Yes

– GPU

Yes

– Memory

Yes

System information

– Console logs (Stderr, Stdout)

Yes

– Error stack trace

Yes

– Execution command

– System details (host, owner)

Yes

Limited

Data versions

– Location

Yes

– Hash

Yes

– Dataset diff

Yes

– External reference version diff (s3)

Yes

Files

– Models

Yes

– CSV

Yes

Custom compare dashboards

– Combining multiple metadata types (image, learning curve, hardware)

Yes

– Logging custom comparisons from notebooks/code

– Compare/diff of multiple (3+) experiments/runs

Yes

Organizing and searching experiments and metadata

Experiment table customization

– Adding/removing columns

Yes

– Renaming columns in the UI

Yes

– Adding colors to columns

Yes

– Displaying aggregate (min/max/avg/var/last) for series like training metrics in a table

Yes

– Automagical column suggestion

Yes

Experiment filtering and searching

– Searching on multiple criteria

Yes

Limited

– Query language vs fixed selectors

Query language

Fixed selectors

– Saving filters and search history

Yes

Custom dashboards for a single experiment

– Can combine different metadata types in one view

Yes

– Saving experiment table views

Yes

– Logging project-level metadata

Yes

– Custom widgets and plugins

Tagging and searching on tags

Yes

Nested metadata structure support in the UI

Yes

Reproducibility and traceability

One-command experiment re-run

Experiment lineage

– List of datasets used downstream

Yes

– List of other artifacts (models) used downstream

Yes

– Downstream artifact dependency graph

Limited

Reproducibility protocol

Limited

Yes

Is environment versioned and reproducible

Yes

Saving/fetching/caching datasets for experiments

Yes

Collaboration and knowledge sharing

User groups and ACL

Only in paid plans

Only for enterprise customers

Sharing UI links with project members

Yes

Sharing UI links with external people

Yes

Commenting

Limited

Yes

Interactive project-level reports

Yes

Model Registry

Model versioning

Code versions (used for training)

Environment versions

Parameters

Dataset versions

Results (metrics, visualizations)

Yes

Explanations (SHAP, DALEX)

Limited

Model files (packaged models, model weights, pointers to artifact storage)

Yes

Model lineage and evaluation history

Models/experiments created downstream

Limited

History of evaluation/testing runs

Support for continuous testing

Users who created a model or downstream experiments

Access control, model review, and promoting models

Main stage transition tags (develop, stage, production)

Yes

Custom stage tags

Locking model version and downstream runs, experiments, and artifacts

Adding annotations/comments and approvals from the UI

Limited

Model compare (current vs challenger etc)

Limited

Compatibility audit (input/output schema)

Compliance audit (datasets used, creation process approvals, results/explanations approvals)

CI/CD/CT compatibility

Webhooks

Model accessibility

Support for continuous testing

Integrations with CI/CD tools

Limited

Model searching

Registered models

Yes

Active models

Yes

By metadata/artifacts used to create it

By date

By user/owner

By production stage

Search query language

Yes

Fixed selectors

Model packaging

Native packaging system

Yes

Compatibility with packaging protocols (ONNX, etc)

One model one file or flexible structure

One model one file

Integrations with packaging frameworks

Integrations and Support

Languages

Java

Julia

Python

Yes

Limited

REST API

Yes

Model training

Catalyst

CatBoost

fastai

FBProphet

Gluon

HuggingFace

Yes

H2O

LightGBM

Yes

Paddle

PyTorch

PyTorch Ignite

PyTorch Lightning

Scikit Learn

Skorch

Spacy

Spark MLlib

Statsmodel

TesorFlow / Keras

Yes

XGBoost

Yes

Hyperparameter Optimization

Hyperopt

Keras Tuner

Limited

Yes

Optuna

Yes

Ray Tune

Scikit-Optimize

Limited

Model visualization and debugging

DALEX

Yes

Netron

SHAP

TensorBoard

Yes

IDEs and Notebooks

JupyterLab and Jupyter Notebook

Google Colab

Deepnote

AWS SageMaker

Data versioning

DVC

Yes

Orchestration and pipelining

Airflow

Yes

Argo

Kedro

Yes

Kubeflow

ZenML

Yes

Experiment tracking tools

MLflow

Limited

Sacred

Yes

TensorBoard

Yes

CI/CD

GitHub Actions

Yes

Gitlab CI

CircleCI

Travis

Jenkins

Yes

Model serving

Seldon

Cortex

Databricks

Model versioning

Seldon

Fiddler.ai

Arthur.ai

LLMs

LangChain

Yes

This table was updated on 11 September 2023. Some information may be outdated.
Report outdated information here.

What are the key advantages of Neptune then?

Customizable metadata structure
Integrations with all the important visualization libraries that allow you to log interactive visualizations to Neptune
More advanced features for organizing and searching experiments and metadata

Explore features

See these features in action

pip install neptune

import neptune

run = neptune.init_run()
run["params"] = {
    "lr": 0.1, "dropout": 0.4
}
run["test_accuracy"] = 0.84

import neptune

model = neptune.init_model()
model["model"] = {
    "size_limit": 50.0,
    "size_units": "MB",
}
model["model/signature"].upload(
    "model_signature.json")

Thousands of ML people already chose their tool

Previously used tensorboard and azureml but Neptune is hugely better. In particular, getting started is really easy; documentation is excellent, and the layout of charts and parameters is much clearer.

Simon Mackenzie AI Engineer and Data Scientist

(…) thanks for the great tool, has been really useful for keeping track of the experiments for my Master’s thesis. Way better than the other tools I’ve tried (comet / wandb).

I guess the main reason I prefer neptune is the interface, it is the cleanest and most intuitive in my opinion, the table in the center view just makes a great deal of sense. I like that it’s possible to set up and save the different view configurations as well. Also, the comparison is not as clunky as for instance with wandb. Another plus is the integration with ignite, as that’s what I’m using as the high-level framework for model training.

Klaus-Michael Lux Data Science and AI student, Kranenburg, Germany

This thing is so much better than Tensorboard, love you guys for creating it!

Dániel Lévai Junior Researcher at Rényi Alfréd Institute of Mathematics in Budapest, Hungary

While logging experiments is great, what sets Neptune apart for us at the lab is the ease of sharing those logs. The ability to just send a Neptune link in slack and letting my coworkers see the results for themselves is awesome. Previously, we used Tensorboard + locally saved CSVs and would have to send screenshots and CSV files back and forth which would easily get lost. So I’d say Neptune’s ability to facilitate collaboration is the biggest plus.

Greg Rolwes Computer Science Undergraduate at Saint Louis University

Such a fast setup! Love it:)

Kobi Felton PhD student in Music Information Processing at Télécom Paris

For me the most important thing about Neptune is its flexibility. Even if I’m training with Keras or Tensorflow on my local laptop, and my colleagues are using fast.ai on a virtual machine, we can share our results in a common environment.

Víctor Peinado Senior NLP/ML Engineer