The Best Tools to Visualize Metrics and Hyperparameters of Machine Learning Experiments

Posted May 21, 2020

Evaluating your model on the key metrics is a crucial first step in understanding your model quality. Keeping track of hyperparameters and corresponding evaluation metrics is important because small changes in hyperparameters can sometimes have a big impact on model quality.

And so, understanding which hyperparameters have an impact and which do not affect evaluation metrics can lead to valuable insights. This is why you should visualize what impact those parameters have on your metrics and know what is your model performance across all of your ML experiments.

To help you, I’ve gathered a list of recommended tools that will do the tedious work for you.

Here are the best six tools to visualize metrics and hyperparameters of machine learning experiments.

1. Neptune

Neptune is a light-weight experiment management tool that helps to keep track of machine learning experiments in a team.

You can use Neptune to log hyperparameters and output metrics from your runs, then visualize and compare results. Automatically transform tracked data into a knowledge repository, then share and discuss your work with colleagues.

Neptune – summary:

  • Easily keep track of metrics, hyperparameters
  • Visualize losses and metrics as your model is training (monitor learning curves)
  • Compare learning curves across various models/experiments
  • Use interactive comparison table that automatically shows diffs between experiments
  • Fetch experiment data and visualize parameters and metrics in notebooks. You can use the HiPlot integration or do custom analysis
  • It has other visualization features that are not parameter-metric related

SEE ALSO
The Best Tools for Machine Learning Model Visualization


2. WandB

Weights & Biases a.k.a. WandB is focused on deep learning. Users can track experiments to the application with Python library, and – as a team – can see each other’s experiments.

The tool lets you record and visualize every detail of your research and collaborate easily with teammates. You can easily log metrics from your script to visualize results in real-time as your model trains. You can also see what your model is producing at each time step.

WandB – summary:

  • Monitor training runs information like loss, accuracy (learning curves)
  • Compare runs with a dashboard table showing auto-diffs
  • Visualize parameters and metrics via parallel coordinates plot
  • Explore how parameters affect metrics with feature (parameter) importance visualization (this I think is experimental)
  • It has other visualization features that are not parameter-metric related

3. Comet

Comet app

Comet is a meta machine learning platform for tracking, comparing, explaining, and optimizing experiments and models. It allows you to view and compare all of your experiments in one place. It works wherever you run your code with any machine learning library, and for any machine learning task.

Comet is suitable for teams, individuals, academics, organizations, and anyone who wants to easily visualize experiments and facilitate work and run experiments.

Comet – summary:

  • You can customize and combine your visualizations
  • You can monitor your learning curves
  • Comet’s flexible experiments and visualization suite allow you to record, compare and visualize many artifact types
  • It has other visualization features that are not parameter-metric related

4. TensorBoard

TensorBoard is a visualization toolkit for TensorFlow that lets you analyze model training runs. It’s open-source and offers a suite of tools for visualization and debugging of machine learning models.

What’s more, it has an extensive network of engineers using this software and sharing their experience and ideas. This makes a powerful community ready to solve any problem. The software, itself, however, is best suited for an individual user.

TensorBoard – summary:

  • Tracking and visualizing metrics such as loss and accuracy
  • Comparing learning curves of various runs
  • Parallel coordinates plot to visualize parameter-metric interactions
  • It has other visualization features that are not parameter-metric related

See also: The Best TensorBoard Alternatives (2020 Update)

5. Optuna

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning.

Additionally, Optuna Integrates with libraries such as LightGBM, Keras, TensorFlow, FastAI, PyTorch Ignite, and more.

Optuna – summary:

  • Visualizations in Optuna let you zoom in on the hyperparameter interactions and help you decide on how to run your next parameter sweep
  • plot_contour: plots parameter interactions on an interactive chart. You can choose which hyperparameters you would like to explore
  • plot_optimization_history: shows the scores from all trials as well as the best score so far at each point
  • plot_parallel_coordinate: interactively visualizes the hyperparameters and scores
  • plot_slice: shows the evolution of the search. You can see where in the hyperparameter space your search went and which parts of the space were explored more

6. HiPlot

Hiplot is a straightforward interactive visualization tool to help AI researchers discover correlations and patterns in high-dimensional data. It uses parallel plots and other graphical ways to represent information more clearly.

HiPlot can be run quickly from a Jupyter notebook with no setup required. The tool enables machine learning (ML) researchers to more easily evaluate the influence of their hyperparameters, such as learning rate, regularizations, and architecture. It can also be used by researchers in other fields, so they can observe and analyze correlations in data relevant to their work.

HiPlot – summary:

Creates an interactive parallel plot visualization to easily explore various hyperparameter-metric interactions
Based on selection on the parallel plot the experiment table is updated automatically
It’s super lightweight and can be used inside notebooks or as a standalone web server

Final words

Now that you have all the list of the best tools, you can visualize metrics and hyperparameters of your ML experiment. Test them yourself and see which one works best for you. We, of course, recommend Neptune – the most lightweight of them all 😉

And which tool is your favorite?


NEXT STEPS

Get started with Neptune in 5 minutes

If you are looking for an experiment tracking tool you may want to take a look at Neptune. 

It takes literally 5 minutes to set up and as one of our happy users said:

“Within the first few tens of runs, I realized how complete the tracking was – not just one or two numbers, but also the exact state of the code, the best-quality model snapshot stored to the cloud, the ability to quickly add notes on a particular experiment. My old methods were such a mess by comparison.” – Edward Dixon, Data Scientist @intel

To get started follow these 4 simple steps. 

Step 1

Install the client library.

pip install neptune-client

Step 2

Connect to the tool by adding a snippet to your training code. 

For example:

import neptune

neptune.init(...) # credentials
neptune.create_experiment() # start logger

Step 3

Specify what you want to log:

neptune.log_metric('accuracy', 0.92)

for prediction_image in worst_predictions:
    neptune.log_image('worst predictions', prediction_image)

Step 4

Run your experiment as you normally would:

python train.py

And that’s it!

Your experiment is logged to a central experiment database and displayed in the experiment dashboard, where you can search, compare, and drill down to whatever information you need.

Get your free account ->