There is a broad catalog of tools you can use as helping hands to increase your understanding of machine learning models. They come in different categories:
In this article, I’ll briefly tell you about as many tools as I can, to show you how rich the ML tools ecosystem is.
1. Interactive web app tools
This open-source ML tool lets you build customized web apps for your models. You can showcase your model in a very interactive and understandable way so that anybody can use it easily.
In just a few minutes, you can build and deploy beautiful and powerful data apps.
To use Streamlit, you just need to install it with pip by using the command:
pip install streamlit
Streamlit’s UI will ask if you’d like to rerun the app and view the changes. This allows you to work in a fast iterative loop: you write some code, save it, review the output, write some more, and so on until you’re happy with the results.
Check out the Streamlit docs to see exactly how it works.
One of the most popular, lightweight Python frameworks for building web apps. You can use it to develop a web API for your model. It’s based on the Werkzeug WSGI toolkit and Jinja2 template engine.
Flask has a simple architecture, and you can learn it very easily. It’s highly recommended for building small applications. You can quickly deploy the model and set up a REST API.
To start using Flask, set up the virtual environment, and install it using the command:
pip install flask
For detailed information, check out Flask documentation.
Shiny is an R package for building interactive web apps. If you already know the R language, it will be really easy to use to build an app and share your work. It has a really interactive and intuitive design.
To start using Shiny use the command:
2. Data analysis tools
DABL stands for Data Analysis Baseline Library. You can use it to automate repetitive processes that happen in the early stages of model development, like data-cleaning, preprocessing, analysis, dealing with missing values, or converting data to different formats.
It’s a new Python library, so its functions are limited, but it’s very promising.
To start using DABL use the command:
pip install dabl
For more information about DABL, check out this article.
Open-source data analysis tool for creating data science applications and building machine learning models.
You can integrate various components for machine learning and data mining through its modular data pipelining concept. KNIME has been used in areas like CRM, text mining, and business intelligence.
It provides an interactive GUI to create workflows using a drag and drop builder. It supports multi-threaded in-memory data processing, and KNIME server supports team-based collaboration.
To get started with KNIME, visit the documentation.
Data science platform that helps you prepare and analyze data. Very user-friendly, you can just drag and drop pieces of code.
It has several data exploration features that you can use to gain valuable insights from your data. It provides more than 14,000 operators for data analysis.
To get started with RapidMiner, follow this link.
SAS stands for Statistical Analysis System, and it’s used to analyze statistical data. It helps you do data analysis using SAS SQL and automatic code generation. You can easily integrate it with MS tools like Excel.
SAS lets you create interactive dashboards and reports to better understand complex data.
To get started with SAS, check out this tutorial.
3. Model explainability tools
Eli5 is a Python package that lets you explain predictions of machine learning classifiers. It provides support for the following packages and frameworks:
- XGBoost – explains predictions of XGBClassifier, XGBRegressor, and also helps validate feature importance.
- CatBoost – explains predictions of CatBoostClassifier, CatBoostRegressor, and also helps validate feature importance.
- Scikit-learn – explains weights and predictions of scikit-learn linear classifiers and regressors, and also helps validate feature importances of decision trees.
To learn more about Eli5, check out the documentation.
It stands for SHapley Additive exPlanations. It’s based on Shapley values. SHAP values show the impact of each feature by breaking down the predictions, which may result in powerful model insights.
Some applications of SHAP values are:
- A model says a bank shouldn’t loan someone money, and the bank is legally required to explain the basis for each loan rejection.
- A healthcare provider wants to identify what factors are driving a patient’s risk of a disease, in order to directly address those risk factors with targeted health interventions.
To learn more about SHAP, check out this tutorial.
It stands for Descriptive Machine Learning Explanations. It’s an R package, built mainly for model interpretability.
Machine learning interpretability is becoming more important. This package helps you extract insights and get clarity on how your algorithm works, and why one prediction is made over another.
Dalex makes it convenient to compare performance across multiple models. There are several advantages of using Dalex:
- It includes a unique and intuitive approach for local interpretation,
- It can customize predicted outputs,
- It provides convenient ways for result comparisons.
To learn more about Dalex, check out the official GitHub page.
4. Model debugging tools
A visual debugging tool for machine learning, built by Uber’s team. It was developed to make the model iteration process more informed and actionable.
Data scientists can use it to look at a general summary and detect the inaccurately predicting part of data. Manifold also explains the potential causes of poor model performance by surfacing the feature distribution differences between better and worse-performing subsets of data.
Manifold serves the majority of the ML models, which includes most classification and regression models.
To learn more about Uber Manifold, check out the official page.
5. Model performance debugging tools
MLPerf is becoming a staple in ML workloads for interesting experiments, comparing different types of specialized infrastructures or software frameworks. It’s used for building useful standards for measuring training performance of ML hardware, software, and services.
The goals of MLPerf are to serve both commercial and academic communities, to ensure reliable results, to accelerate progress in ML, and to enable fair comparison of competing systems, yet to encourage innovation to improve state-of-the-art ML.
To learn more about MLPerf, check out this introductory article.
6. Experiment tracking tools
Lightweight and powerful experiment tracking tool for data scientists. It easily integrates with your workflow and offers an extensive range of tracking features.
You can use it to track, retrieve, and analyze experiments, or to share them with your team and managers. Neptune is flexible, works with many frameworks, and thanks to its stable user interface, it enables great scalability (to millions of runs).
It also lets you store, retrieve, and analyze large amounts of data.
To learn more about Neptune, check the website.
It stands for Weights and Biases. It’s a Python package that lets you monitor model training in real-time. It easily integrates with popular frameworks like Pytorch, Keras, and Tensorflow.
Additionally, it lets you organize Runs into Projects, where you can easily compare them and identify the best performing model.
Learn more about WandB, in this introductory article.
👉 Check the comparison between WandB and Neptune.ai
Comet helps data scientists manage and organize machine learning experiments. It lets you easily compare experiments and keep a record of collected data, as well as collaborate with other team members. It can easily adapt to any machine and works well with different ML libraries.
To learn more about Comet, check out the official documentation.
👉 Check the comparison between Comet and Neptune.ai
Open-source platform for tracking machine learning experiments, and also for deploying models. Each element is represented by one MLflow component: Tracking, Projects, and Models.
This means that if you’re working with MLflow, you can easily track an experiment, organize it, describe it for other ML engineers, and pack it into a machine learning model.
MLflow has been designed to enable scalability from one person to a big organization, but it works best for an individual user.
To learn more about MLflow, check the official documentation.
👉 Check which tool is better: Neptune vs MLflow
👉 Explore other tools in the article 15 Best Tools for Tracking Machine Learning Experiments
7. Production monitoring tools
Kubeflow makes it easier to deploy machine learning workflows. It’s known as the machine learning toolkit for Kubernetes and aims to use the Kubernetes potential to facilitate scaling of ML models.
The team behind Kubeflow is constantly developing its features, and doing its best to make life easier for data scientists. Kubeflow has some tracking capabilities, but they’re not the main focus of the project. It can be easily used with other tools on this list as a complementary tool.
To learn more about Kubeflow, check out the official documentation.
👉 Check the comparison between Kubeflow and Neptune.ai
This concludes our list of different machine learning tools. As you can see, the ecosystem is wide, and the list doesn’t cover all the tools out there.
Whatever are your needs, you don’t need to do things manually. Use these tools to speed up your workflows, and make your life as a data scientist easier.
15 Best Tools for ML Experiment Tracking and Management
10 mins read | Author Patrycja Jenkner | Updated August 25th, 2021
While working on a machine learning project, getting good results from a single model-training run is one thing. But keeping all of your machine learning experiments well organized and having a process that lets you draw valid conclusions from them is quite another.
The answer to these needs is experiment tracking. In machine learning, experiment tracking is the process of saving all experiment-related information that you care about for every experiment you run.
ML teams implement experiment tracking in different ways, may it be by using spreadsheets, GitHub, or self-built platforms. Yet, the most effective option is to do it with tools designed specifically for tracking and managing ML experiments.
In this article, we overview and compare the 15 best tools that will allow you to track and manage your ML experiments. You’ll get to know their main features and see how they are different from each other. Hopefully, this will help you evaluate them and choose the right one for your needs.
How to evaluate an experiment tracking tool?
There’s no one answer to the question “what is the best experiment tracking tool?”. Your motivation and needs may be completely different when you work individually or in a team. And, depending on your role, you may be looking for various functionalities.
If you’re a Data Scientist or a Researcher, you should consider:
- If the tool comes with a web UI or it’s console-based;
- If you can integrate the tool with your preferred model training frameworks;
- What metadata you can log, display, and compare (code, text, audio, video, etc.);
- Can you easily compare multiple runs? If so, in what format – only table, or also charts;
- If organizing and searching through experiments is user-friendly;
- If you can customize metadata structure and dashboards;
- If the tool lets you track hardware consumption;
- How easy it is to collaborate with other team members – can you just share a link to the experiment or you have to use screenshots as a workaround?
As an ML Engineer, you should check if the tool lets you:
- Easily reproduce and re-run experiments;
- Track and search through experiment lineage (data/models/experiments used downstream);
- Save, fetch, and cache datasets for experiments;
- Integrate it with your CI/CD pipeline;
- Easily collaborate and share work with your colleagues.
Finally, as an ML team lead, you’ll be interested in:
- General business-related stuff like pricing model, security, and support;
- How much infrastructure the tool requires, how easy it is to integrate it into your current workflow;
- Is the product delivered as commercial software, open-source software, or a managed cloud service?
- What collaboration, sharing, and review feature it has.
I made sure to keep these motivations in mind when reviewing the tools that are on the market. So let’s take a closer look at them.Continue reading ->