There is a broad catalog of tools you can use as helping hands to increase your understanding of machine learning models. They come in different categories:
In this article, I’ll briefly tell you about as many tools as I can, to show you how rich the ML tools ecosystem is.
1. Interactive web app tools
This open-source ML tool lets you build customized web apps for your models. You can showcase your model in a very interactive and understandable way so that anybody can use it easily.
In just a few minutes, you can build and deploy beautiful and powerful data apps.
To use Streamlit, you just need to install it with pip by using the command:
pip install streamlit
Streamlit’s UI will ask if you’d like to rerun the app and view the changes. This allows you to work in a fast iterative loop: you write some code, save it, review the output, write some more, and so on until you’re happy with the results.
Check out the Streamlit docs to see exactly how it works.
One of the most popular, lightweight Python frameworks for building web apps. You can use it to develop a web API for your model. It’s based on the Werkzeug WSGI toolkit and Jinja2 template engine.
Flask has a simple architecture, and you can learn it very easily. It’s highly recommended for building small applications. You can quickly deploy the model and set up a REST API.
To start using Flask, set up the virtual environment, and install it using the command:
pip install flask
For detailed information, check out Flask documentation.
Shiny is an R package for building interactive web apps. If you already know the R language, it will be really easy to use to build an app and share your work. It has a really interactive and intuitive design.
To start using Shiny use the command:
2. Data analysis tools
DABL stands for Data Analysis Baseline Library. You can use it to automate repetitive processes that happen in the early stages of model development, like data-cleaning, preprocessing, analysis, dealing with missing values, or converting data to different formats.
It’s a new Python library, so its functions are limited, but it’s very promising.
To start using DABL use the command:
pip install dabl
For more information about DABL, check out this article.
Open-source data analysis tool for creating data science applications and building machine learning models.
You can integrate various components for machine learning and data mining through its modular data pipelining concept. KNIME has been used in areas like CRM, text mining, and business intelligence.
It provides an interactive GUI to create workflows using a drag and drop builder. It supports multi-threaded in-memory data processing, and KNIME server supports team-based collaboration.
To get started with KNIME, visit the documentation.
Data science platform that helps you prepare and analyze data. Very user-friendly, you can just drag and drop pieces of code.
It has several data exploration features that you can use to gain valuable insights from your data. It provides more than 14,000 operators for data analysis.
To get started with RapidMiner, follow this link.
SAS stands for Statistical Analysis System, and it’s used to analyze statistical data. It helps you do data analysis using SAS SQL and automatic code generation. You can easily integrate it with MS tools like Excel.
SAS lets you create interactive dashboards and reports to better understand complex data.
To get started with SAS, check out this tutorial.
3. Model explainability tools
Eli5 is a Python package that lets you explain predictions of machine learning classifiers. It provides support for the following packages and frameworks:
- XGBoost – explains predictions of XGBClassifier, XGBRegressor, and also helps validate feature importance.
- CatBoost – explains predictions of CatBoostClassifier, CatBoostRegressor, and also helps validate feature importance.
- Scikit-learn – explains weights and predictions of scikit-learn linear classifiers and regressors, and also helps validate feature importances of decision trees.
To learn more about Eli5, check out the documentation.
It stands for SHapley Additive exPlanations. It’s based on Shapley values. SHAP values show the impact of each feature by breaking down the predictions, which may result in powerful model insights.
Some applications of SHAP values are:
- A model says a bank shouldn’t loan someone money, and the bank is legally required to explain the basis for each loan rejection.
- A healthcare provider wants to identify what factors are driving a patient’s risk of a disease, in order to directly address those risk factors with targeted health interventions.
To learn more about SHAP, check out this tutorial.
It stands for Descriptive Machine Learning Explanations. It’s an R package, built mainly for model interpretability.
Machine learning interpretability is becoming more important. This package helps you extract insights and get clarity on how your algorithm works, and why one prediction is made over another.
Dalex makes it convenient to compare performance across multiple models. There are several advantages of using Dalex:
- It includes a unique and intuitive approach for local interpretation,
- It can customize predicted outputs,
- It provides convenient ways for result comparisons.
To learn more about Dalex, check out the official GitHub page.
Neptune’s integration with Dalex 👇
4. Model debugging tools
A visual debugging tool for machine learning, built by Uber’s team. It was developed to make the model iteration process more informed and actionable.
Data scientists can use it to look at a general summary and detect the inaccurately predicting part of data. Manifold also explains the potential causes of poor model performance by surfacing the feature distribution differences between better and worse-performing subsets of data.
Manifold serves the majority of the ML models, which includes most classification and regression models.
To learn more about Uber Manifold, check out the official page.
5. Model performance debugging tools
MLPerf is becoming a staple in ML workloads for interesting experiments, comparing different types of specialized infrastructures or software frameworks. It’s used for building useful standards for measuring training performance of ML hardware, software, and services.
The goals of MLPerf are to serve both commercial and academic communities, to ensure reliable results, to accelerate progress in ML, and to enable fair comparison of competing systems, yet to encourage innovation to improve state-of-the-art ML.
To learn more about MLPerf, check out this introductory article.
6. Experiment tracking tools
Lightweight and powerful experiment tracking tool for data scientists. It easily integrates with your workflow and offers an extensive range of tracking features.
You can use it to track, retrieve, and analyze experiments, or to share them with your team and managers. Neptune is flexible, works with many frameworks, and thanks to its stable user interface, it enables great scalability (to millions of runs).
It also lets you store, retrieve, and analyze large amounts of data.
To learn more about Neptune, check the website.
It stands for Weights and Biases. It’s a Python package that lets you monitor model training in real-time. It easily integrates with popular frameworks like Pytorch, Keras, and Tensorflow.
Additionally, it lets you organize Runs into Projects, where you can easily compare them and identify the best performing model.
Learn more about WandB, in this introductory article.
Check the comparison between WandB and Neptune.ai.
Comet helps data scientists manage and organize machine learning experiments. It lets you easily compare experiments and keep a record of collected data, as well as collaborate with other team members. It can easily adapt to any machine and works well with different ML libraries.
To learn more about Comet, check out the official documentation.
Check the comparison between Comet and Neptune.ai.
Open-source platform for tracking machine learning experiments, and also for deploying models. Each element is represented by one MLflow component: Tracking, Projects, and Models.
This means that if you’re working with MLflow, you can easily track an experiment, organize it, describe it for other ML engineers, and pack it into a machine learning model.
MLflow has been designed to enable scalability from one person to a big organization, but it works best for an individual user.
To learn more about MLflow, check the official documentation.
– Check which tool is better: Neptune vs MLflow
– Explore other tools in the article 15 Best Tools for Tracking Machine Learning Experiments
7. Production monitoring tools
Kubeflow makes it easier to deploy machine learning workflows. It’s known as the machine learning toolkit for Kubernetes and aims to use the Kubernetes potential to facilitate scaling of ML models.
The team behind Kubeflow is constantly developing its features, and doing its best to make life easier for data scientists. Kubeflow has some tracking capabilities, but they’re not the main focus of the project. It can be easily used with other tools on this list as a complementary tool.
To learn more about Kubeflow, check out the official documentation.
Check the comparison between Kubeflow and Neptune.ai.
This concludes our list of different machine learning tools. As you can see, the ecosystem is wide, and the list doesn’t cover all the tools out there.
Whatever are your needs, you don’t need to do things manually. Use these tools to speed up your workflows, and make your life as a data scientist easier.
15 Best Tools for Tracking Machine Learning Experiments
7 mins read | Author Pawel Kijko | Updated July 14th, 2021
While working on a machine learning project, getting good results from a single model-training run is one thing, but keeping all of your machine learning experiments organized and having a process that lets you draw valid conclusions from them is quite another. That’s what machine learning experiment management helps with.
In this article, I will explain why you, as data scientists and machine learning engineers, need a tool for tracking machine learning experiments and what is the best software you can use for that.
Tools for tracking machine learning experiments – who needs them and why?
- Data Scientists: Currently many organizations are either including ML in their products as an added value or are AI-first companies. These organizations have to adhere to MLOps processes and tools such as experiment tracking to ensure better collaborations between individuals, teams, and the success of the ML project. Without them when a data scientist or data science team wants to come back to an idea, re-run a model from a couple of months ago, or simply compare and visualize the differences between runs, the need for a system or tool for tracking ML experiments becomes (painfully) apparent.
- Machine Learning Engineers: Right after the data scientist or data science team finalizes model development and is ready to launch the said model, the MLEs are the ones who take all the research/dev code, model and turn it all into a production-ready version as well as deploy it. But in between this handoff process from DS’ to MLEs, there is more information about the model than just the weights that are needed to ensure a successfully deployed solution (one that is debuggable, maintainable, reproducible, and comparable). That’s when experiment metadata is very important – from the dataset, code, and model version to hyperparameters and other configurations used to train the model.
- Managers/Business people: tracking software creates an opportunity to involve other team members like managers or business stakeholders in your machine learning projects. Thanks to the possibility to prepare visualizations, add comments and share the work, managers and co-workers can easily track the progress and cooperate with the machine learning team.
Here is an in-depth article about experiment tracking for those of you who want to learn more.Continue reading ->