We Raised $8M Series A to Continue Building Experiment Tracking and Model Registry That “Just Works”

Read more

The Best Tools for Machine Learning Model Visualization

The phrase “Every model is wrong, but some are useful” is especially true in Machine Learning. When developing machine learning models, you should always understand where it works as expected and where it fails miserably.

An important method of understanding a model is through model visualization. Visualizing the model architecture is important for:

  • Model explainability
  • Results interpretation
  • Model debugging
  • Model comparison

Read more about model visualization

Once you get some decent understanding of one model, you are good, right? Wrong 🙂

Typically, you need to do some or a lot of experimenting with model improvement ideas, and visualizing differences between various ML experiments becomes crucial. 

There are many methods that you can use to get that understanding:

  • Look at evaluation metrics (also, you should know how to choose an evaluation metric for your problem)
  • Look at performance charts like ROC, Lift Curve, Confusion Matrix, and others
  • Look at learning curves to estimate overfitting
  • Look at model predictions on best/worst cases
  • Look how resource-intensive is model training and inference (they translate to serious costs and will be crucial to the business side of things) 
  • Use model interpretation tools and techniques to vet predictions (further reading)

You can do all of those (or most of those) yourself, but today there are tools that you can use. If you’re looking for the best tools that will help you visualize machine learning experiments and models, you’ve come to the right place.

Tools for Machine Learning experiments visualization

1. Neptune

Metadata-dashboard-experiments
Example of model training metadata displayed in a dashboard | See in the app

Neptune is a metadata store for MLOps, built for teams that run a lot of experiments.‌ It gives you a single place to log, store, display, organize, compare, and query all your model-building metadata.

Neptune is used for:‌

  • Experiment tracking: Log, display, organize, and compare ML experiments in a single place.
  • Model registry: Version, store, manage, and query trained models and model building metadata.
  • Monitoring ML runs live: Record and monitor model training, evaluation, or production runs live.

How can Neptune help you visualize experiments and models?

2. Weights & Biases

Weights & Biases is a machine learning platform for developers to build better models faster. It lets you quickly track experiments, version and iterate on datasets, evaluate model performance, reproduce models, visualize results and spot regressions, and share findings with colleagues.

How can Weights & Biases help you visualize experiments and models?

  • Monitor training runs information like loss, accuracy (learning curves)
  • View histograms of weights and biases (no pun intended), or gradients
  • Log rich objects like, charts, video, audio or interactive charts during training
  • Use various comparison tools like tables showing auto-diffs, parallel coordinates plot and others
  • Interactive prediction bounding box visualization for object detection models
  • Interactive prediction masks visualization for semantic segmentation models
  • Visualize live metrics like GPU and CPU utilization
  • Build dataset dependency graphs
  • Visualize parameter importance

Read more

3. Comet

Comet is a meta machine learning platform for tracking, comparing, explaining, and optimizing experiments and models. It also lets you monitor, detect and alert, and debug production models in real time. You can also build your own or use community provided custom visualizations and applications based on experiments and model data.

How can Comet help you visualize experiments and models?

  • Visualize samples with dedicated modules for vision, audio, text and tabular data to detect overfitting and easily identify issues with your dataset
  • You can customize and combine your visualizations
  • You can monitor your learning curves
  • Comet’s flexible experiments and visualization suite allow you to record, compare, and visualize many artifact types
  • Build your own, or use community built, “panels” to visualize your models and experiments
  • Monitor production models in real-time

Read more

4. TensorBoard

TensorBoard provides the visualization and tooling needed for machine learning experimentation. It’s open-source and offers a suite of tools for visualization and debugging of machine learning models. TensorBoard is the most popular solution on the market; thus, it’s widely integrated with many other tools and applications.

What’s more, it has an extensive network of engineers using this software and sharing their experience and ideas. This makes a powerful community ready to solve any problem. The software, itself, however, is best suited for an individual user.

How can TensorBoard help you visualize experiments and models?

  • Tracking and visualizing metrics such as loss and accuracy
  • Visualizing the model graph (ops and layers)
  • Viewing histograms of weights, biases, or other tensors as they change over time
  • Projecting embeddings to a lower-dimensional space
  • Displaying images, text, and audio data
  • Profiling TensorFlow programs

5. Sacred + Omniboard

The Sacred + Omniboard stack actually comprises two open-source components – Sacred, an experiment management tool, and Omniboard, a web dashboard for Sacred. Sacred logs experiment data to a MongoDB backend which Omniboard connects to visualize experiments.

Sacred gives you a powerful command-line interface, observers to log experiment information, and automatic seeding to control experiment randomness so that results are reproducible. Omniboard adds to this with its suite of experiment management, drill down, and comparison functionalities.

How can Sacred + Omniboard help you visualize experiments and models?

  • The Sacred CLI can be used to update parameters and run different variants of the same experiment
  • Save experiment configurations in a MongoDB database
  • Compare different experiments on the Omniboard dashboard
  • Roll-up or drill-down experiments to either view aggregates or individual details

6. MLflow

MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It is currently by far the most widely used ML experiment tracker and as such, is widely supported by languages, frameworks, and platforms. MLflow supports logging through Python, R, Java, and REST APIs.

Being an open-source project, there are no direct costs involved with MLflow for personal use. However, infrastructure costs will add up if you want to use MLflow for collaboration between large teams, as you will need to have your own remote tracking servers. Databricks’ managed MLflow takes care of hosting for you.

How can MLflow help you visualize experiments and models?

Comparison table: tools for ML experiments visualization

Experiment tracking is the core use-case of all the abovementioned products, and as such, most offer similar functionality. However, a few features are not supported in all the tools. The below table highlights a few differences in the functionality offered by the abovementioned tools.

Neptune
W&B
Comet
TensorBoard
Sacred + Omniboard
MLflow

Supported languages

Neptune:

Python, R

W&B:

Python, Java

Comet:

Python, Java

TensorBoard:

Python, R

Sacred + Omniboard:

Python

MLflow:

Python, R, Java

Can aggregate logged metrics?

Neptune:

Yes

W&B:

Yes

Comet:

Yes

TensorBoard:

No

Sacred + Omniboard:

No

MLflow:

Yes

Can log hardware consumption?

Neptune:

Yes

W&B:

Yes

Comet:

Yes

TensorBoard:

No

Sacred + Omniboard:

Yes

MLflow:

No

Can log console output?

Neptune:

Yes

W&B:

Yes

Comet:

Yes

TensorBoard:

No

Sacred + Omniboard:

Yes

MLflow:

No

Table format diff for experiment comparison?

Neptune:

Yes

W&B:

Yes

Comet:

Yes

TensorBoard:

No

Sacred + Omniboard:

No

MLflow:

No

Groupby on experiment parameters?

Neptune:

Yes

W&B:

Yes

Comet:

Yes

TensorBoard:

No

Sacred + Omniboard:

Yes

MLflow:

No

Parallel coordinate plots?

Neptune:

Yes

W&B:

Yes

Comet:

Yes

TensorBoard:

No

Sacred + Omniboard:

No

MLflow:

Only in the managed version

Side-by-side comparison of data?

Neptune:

Image, table, text

W&B:

Audio, HTML, image, table, text, video

Comet:

Plots, table, text

TensorBoard:

No

Sacred + Omniboard:

Text

MLflow:

Table

Machine Learning model visualization tools

1. dtreeviz

dtreeviz is a python library for decision tree visualization and model interpretation. It currently supports scikit-learn, XGBoost, Spark MLlib, and LightGBM trees. Versions 1.3 onwards also support feature space illustrations for any model which have a predict_proba().

How can dtreevix help you visualize models?

  • Visualize tree-based models
  • Illustrate feature space for supported classifiers
  • Highlight prediction path for a single observation in the tree, and get plain English explanations for the same
  • Visualize leaf metrics like purity, samples, and target distribution
  • Visualize classification boundaries in the feature space.

2. Netron

Netron can be used to produce interactive visualizations of neural network models. It is available as a Python package, a standalone application, and a browser application.

How can Netron help you visualize models?

  • Produce interactive visualizations of neural networks 
  • Expand nodes to view layer details
  • Supports almost all widely used frameworks
  • No installation required to use the browser application

3. NN-SVG

NN-SVG is a hosted application for creating Neural Network (NN) architecture drawings parametrically rather than manually. It also provides the ability to export those drawings to Scalable Vector Graphics (SVG) files, suitable for inclusion in academic papers or web pages.

How can NN-SVG help you visualize models?

  • Fully hosted application, no installation required
  • Supports FCNN, LeNet, and AlexNet styles
  • Highly customizable
  • Export generated schematics as an SVG image

4. TensorBoard

TensorBoard is mostly an experiment visualization tool, but it also shows model graphs for TensorFlow models.

How can TensorBoard help you visualize models?

  • Visualize TensorFlow model graphs
  • Track and log TensorFlow experiments
  • Provides a fully-managed version – TensorBoard.dev – which lets you host, track, and share your ML experiments 

Summary

In this blog, we have covered and compared some of the most popular model and visualization tools available currently. As with most of the tools in the ML space, there is no one-size-fits-all, and which tool to use will depend a lot on your specific use case. 

What kind of models do you want to visualize? And what is the purpose of your visualizations?

Most of the model visualization tools are built targeting specific architectures and use cases.

  • For tree-based models, dtreeviz is a good choice.
  • TensorBoard integrates very well with Tensorflow/Keras models.
  • If you just want to create a non-interactive visualization of your neural network model for illustration purposes, NN-SVG might save you some time.
  • Netron produces detailed interactive visualizations of neural network models, which can help a lot with debugging and optimization. 

Are you a small team working on a limited budget?

If yes, some of the open-source and free tools might be good to start with. Keep in mind, though, that as your team grows, your self-hosting costs will grow as well. You may also consider SaaS solutions with pricing that doesn’t depend on the number of users (e.g., Neptune). This way, you’ll be better prepared for the time when your needs scale.

Are you a larger team looking for collaborative development?

Most SaaS solutions have rich collaboration features like URL-based sharing, collaborative dashboards, and reports, user management, etc. 

In any case, hope this article will help you make that choice, and you’ll find what you were looking for.

Happy visualizing!


READ NEXT

15 Best Tools for ML Experiment Tracking and Management

10 mins read | Author Patrycja Jenkner | Updated August 25th, 2021

While working on a machine learning project, getting good results from a single model-training run is one thing. But keeping all of your machine learning experiments well organized and having a process that lets you draw valid conclusions from them is quite another. 

The answer to these needs is experiment tracking. In machine learning, experiment tracking is the process of saving all experiment-related information that you care about for every experiment you run. 

ML teams implement experiment tracking in different ways, may it be by using spreadsheets, GitHub, or self-built platforms. Yet, the most effective option is to do it with tools designed specifically for tracking and managing ML experiments.

In this article, we overview and compare the 15 best tools that will allow you to track and manage your ML experiments. You’ll get to know their main features and see how they are different from each other. Hopefully, this will help you evaluate them and choose the right one for your needs. 

How to evaluate an experiment tracking tool? 

There’s no one answer to the question “what is the best experiment tracking tool?”. Your motivation and needs may be completely different when you work individually or in a team. And, depending on your role, you may be looking for various functionalities. 

If you’re a Data Scientist or a Researcher, you should consider: 

  • If the tool comes with a web UI or it’s console-based;
  • If you can integrate the tool with your preferred model training frameworks;
  • What metadata you can log, display, and compare (code, text, audio, video, etc.);
  • Can you easily compare multiple runs? If so, in what format – only table, or also charts;
  • If organizing and searching through experiments is user-friendly;
  • If you can customize metadata structure and dashboards;
  • If the tool lets you track hardware consumption;
  • How easy it is to collaborate with other team members – can you just share a link to the experiment or you have to use screenshots as a workaround?

As an ML Engineer, you should check if the tool lets you: 

  • Easily reproduce and re-run experiments;
  • Track and search through experiment lineage (data/models/experiments used downstream); 
  • Save, fetch, and cache datasets for experiments;
  • Integrate it with your CI/CD pipeline;
  • Easily collaborate and share work with your colleagues.

Finally, as an ML team lead, you’ll be interested in:

  • General business-related stuff like pricing model, security, and support;
  • How much infrastructure the tool requires, how easy it is to integrate it into your current workflow;
  • Is the product delivered as commercial software, open-source software, or a managed cloud service?
  • What collaboration, sharing, and review feature it has. 

I made sure to keep these motivations in mind when reviewing the tools that are on the market. So let’s take a closer look at them. 

Continue reading ->
The Best Tools to Visualize Metrics and Hyperparameters of Machine Learning Experiments

The Best Tools to Visualize Metrics and Hyperparameters of Machine Learning Experiments

Read more
Plotly Python Tutorial for Machine Learning Specialists

Plotly Python Tutorial for Machine Learning Specialists

Read more
pyLDAvis: Topic Modelling Exploration Tool That Every NLP Data Scientist Should Know

pyLDAvis: Topic Modelling Exploration Tool That Every NLP Data Scientist Should Know

Read more
Pandas Plot: Deep Dive Into Plotting Directly With Pandas

Pandas Plot: Deep Dive Into Plotting Directly With Pandas

Read more