TL;DR
Weights & Biases (W&B or WandB for short) is a popular experiment-tracking platform with a broad range of features.
The lack of scalability, the pricing model, difficulties with self-hosting and lacking documentation lead data science teams to explore alternatives.
neptune.ai, Comet ML, Aim, MLflow, and ClearML Experiment are among the leading Weights & Biases competitors, each offering unique advantages.
Google Vertex AI is a viable alternative for teams already committed to the respective cloud platform provider.
Weights & Biases (W&B or WandB) is a platform for tracking, visualizing, and comparing machine-learning experiments. It provides features to log experiment metadata – parameters, metrics, and outcomes – critical in the iterative ML model development process.
Weights & Biases enhances the efficiency of ML projects and streamlines the data science team’s workflow. Over the years, it has become an established tool within the ML community, appreciated for its wide range of capabilities.
The platform is available in managed and on-premises variants catering to different groups of users. Furthermore, Weights & Biases offers a free tier for personal use, making it accessible to individuals and small teams.
However, Weights & Biases is just one of many experiment tracking platforms on the market, and while it works great for many users, it’s not the best fit for every team and project.
The Weights & Biases platform
Weights & Biases (W&B) offers features to enhance and streamline the machine learning development process:
- Interactive dashboard: A central, user-friendly dashboard for viewing experiments and tracking their performance. This dashboard serves as a command center for monitoring all aspects of the machine-learning process.
- Experiment tracking: W&B excels at tracking every detail of the model training process. It visualizes models and facilitates easy comparison of different experiments, simplifying the identification of the most effective strategies and models.
- Automated hyperparameter tuning: With its ‘Sweeps’ feature, W&B automates the hyperparameter tuning process. It explores a range of hyperparameter combinations, aiding in optimizing model performance and deepening the understanding of how different parameters impact results.
- End-to-end ML pipeline tracking: W&B allows users to version the entire machine learning pipeline. This includes data preparation, model training, and deployment, ensuring a clear lineage of all artifacts produced.
- Framework integration: W&B is designed to be compatible with numerous machine learning frameworks, including TensorFlow, PyTorch, and Keras, as well as libraries like Hugging Face Transformers and LangChain.
- Collaboration features: The W&B platform is built with collaboration in mind, offering various features that support teamwork, such as sharing experiments, results, reports, and insights with others. These features make W&B an ideal tool for small teams and large organizations alike.
Learnings From Teams Training Large-Scale Models: Challenges and Solutions For Monitoring at Hyperscale
Drawbacks of Weights & Biases and reasons to explore alternatives
While W&B is a popular and well-regarded platform, like every software, it comes with limitations. There are scenarios where W&B stands out as the optimal choice, yet there are also instances where it might not meet the needs or expectations of users.
Lack of scalability
As the use of AI technology becomes more widespread, teams are training more and larger models. Thus, they are looking for experiment trackers that can handle an influx of tens of thousands of data points in parallel and allow data scientists to analyze hundreds of runs simultaneously.
Weights & Biases does not scale well when it comes to tracking, displaying, and retrieving large amounts of data.
One common complaint about Weights & Biases is that logging data can take a long time, slowing down the training process:
[It] is really frustrating when you log a lot of data […] and you have to wait for it to upload after each run. I do a lot of runs when optimizing so if im doing 500 training runs and […] have to wait tens of seconds after each run for it to upload, it slows me down. Former Weights & Biases user on Reddit
The user interface of Weights & Biases is known to struggle with displaying larger amounts of data and is generally slow.
[I] especially [like] the manipulation of plots and reports as they simplify and visualize many metrics and parameters. […] [But] sometimes the web UI is a bit slow.
Senior Algorithm Architect at a large enterprise
[The] UI can get so slow that it borders on unusable. ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ Weights & Biases user on Reddit
Users also experience issues with retrieving data from Weights & Biases for analysis:
The user interface is slow but it is acceptable. [But] [r]etrieving runs data […] using the wandb.Api() takes forever (e.g., 30h for around 30 000 runs of hyperparameter in several environments).
I would like to be able to download all data from a set of runs selected from filters in a single [API] call. Since it represents less than 100 [MB] of data, it should be feasible in a few minutes maximum, right? Weights & Biases user at a small business
Pricing model
Weights & Biases has a unique pricing model based on the number of users and the tracked hours. This means that the more time you spend training a model, the more expensive it gets to log metadata to Weights & Biases, even if you make just a few API requests.
The Teams plan includes 5,000 tracked hours, with each additional hour costing $1. For many teams using the SaaS version under this plan, this quickly leads to high costs:
WandB’s pricing was way too much for us to swallow once we ramped up experiments – we went over the tracked hours limit almost immediately. Former Weights & Biases user on Reddit
The costs associated with tracked hours become particularly problematic for teams training many models in parallel:
I have a few powerful GPUs, and run multiple experiments at a time on each one. This results in “tracked hours” of many multiples of realtime, for each GPU, which doesn’t seem right. This is OK for me now as an academic, on the personal plan with unlimited tracked hours, but discourages me from using this for commercial projects in the future, where cost would quickly become prohibitive. Weights & Biases user at a small biotech company
The Enterprise plan is not subject to tracked hours or storage limitations but priced on a per-user basis. While Weights & Biases does not publish list prices, users report a price range of between $200 and $400 per user. For larger teams with members who only occasionally need access to the tracked data, this can become expensive.
Self-hosting is difficult
Many organizations, especially those dealing with sensitive data or conducting large-scale training, look to self-host their experiment tracker in a private cloud environment.
While Weights & Biases offers self-hosting, it is not recommended by Weights & Biases and requires that users configure several infrastructure components (such as databases) on their own. (Weights & Biases offers Terraform scripts for deploying on AWS, GCP, and Azure.)
This makes it difficult to set up and maintain:
We had on-prem (for security reasons) and it was a mess.‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ Former Weights & Biases user on Reddit
In addition, while operating the on-premises variant of Weights & Biases is subject to a similar pricing model as the SaaS version, it comes with additional costs:
I thought [the on-premise version] was the same price but there was a huge NRC [non-recurring charge]Â upfront for on-premise. Former Weights & Biases User on Reddit
Aside
We initially aimed for a GKE deployment for our experiment tracking tool. However, the alternative solution we explored had a rigid installation process and limited support, making it unsuitable for our needs.
Thankfully, Neptune’s on-premise installation offered the flexibility and adjustability we required. The process was well-prepared, and their engineers were incredibly helpful, answering all our questions and even guiding us through a simpler deployment approach. Neptune’s on-prem solution and supportive team saved the day, making it a win for us.Krzysztof, DevOps Engineer at Cradle
Support and documentation
Comprehensive and up-to-date documentation is essential to enable data scientists to use and manage an experiment tracker effectively.
While Weights & Biases provides an extensive documentation, users frequently report that it is difficult to figure out how to accomplish a task and have to go to online communities for support:
It is most of the time hard to find the relevant information you are seeking […] in the documentation, hence help comes from issues dealt online by users on different platforms (github, stackoverflow, etc.) Researcher using Weights & Biases at a large enterprise
Alternatives to Weights & Biases
Of course, Weights & Biases is not the only experiment tracking and machine learning platform out there.
On the one hand, there is a range of managed experiment tracking platforms with a comparable range of features. We’ll review neptune.ai, Comet ML, and Aim as leading competitors of Weights & Biases, focusing on key differences and unique strengths.
On the other hand, open-source platforms like MLflow and Clear ML provide experiment-tracking capabilities as part of an end-to-end ML lifecycle management solution.
Finally, the large cloud providers each offer experiment tracking capabilities as part of their ML platforms. For teams already committed to a cloud platform, solutions like Google Vertex AI are worth exploring as alternatives to Weights & Biases.
neptune.ai
Neptune is an experiment tracker built with a strong focus on scalability. The tool is known for its fast, user-friendly interface and the ability to handle model training monitoring at a really large scale (think foundation model scale).
Neptune enables data scientists and AI researchers to log, monitor, visualize, compare, and query all their model metadata in a single place. It handles data such as model metrics and parameters, model checkpoints, images, videos, dataset versions, and visualizations. It also gives users a lot of flexibility when defining metadata tracking structures. Furthermore, Neptune improves team collaboration by making sharing results with team members and stakeholders easy.
- Scalability: Neptune easily tracks tens of thousands of data points and handles up to a thousand times more throughput than Weights & Biases. The UI allows users to compare more than 100,000 runs with millions of data points.
- Pricing: Neptune’s pricing model is based on the number of users, allowing them to collaborate on as many projects as they like. Storage is priced on usage.
- Self-hosting: Neptune is available for self-hosting, which is a first-class offering in the Enterprise tier. Designed to be hosted in a private cloud environment, Neptune integrates with common authentication solutions like SAML or LDAP, allowing seamless integration while keeping sensitive data protected.
- Support and documentation: All plans (including the Free tier) provide access to chat and email support, with SLAs reserved for the Enterprise plan. Neptune’s documentation is comprehensive and includes many examples. (Important note: Teams migrating to Neptune from Weights & Biases can use a utility script to transfer their data).
Neptune takes experiment tracking to the next level with its ability to fork experiment runs from any intermediate step. This is particularly important for large-scale deep learning experiments – such as training foundational models – where training failures due to hardware or network issues are unavoidable. It’s also common to try different parameters and training configurations over the course of a month-long training process.
In deep learning experiments, especially when training foundational models, there are two challenges that often disrupt workflows:
- Failures and restarts: Hardware faults, network interruptions, or orchestration errors can cause experiment crashes. Restarting from a saved checkpoint often leads to mismatched or incomplete data, which can jeopardize the integrity of the experiment. With Neptune, users can restart from any saved step while preserving historical data, maintaining data accuracy across the board.
- Branches and parallel exploration: Training foundational models often spans weeks or months. Researchers frequently tweak parameters and configurations mid-experiment to optimize performance. Forking allows users to branch out from the best-performing step of an experiment and try new configurations—all while keeping the original run intact. This is great for teams to explore multiple directions in parallel without losing insight into historical progress.
Comparison Between Neptune vs. Weights & Biases
Comet ML
Comet is a managed machine-learning platform that provides tools for tracking, comparing, and collaborating on machine-learning experiments.
With features like real-time metrics, parameter visualization, experiment management, and a clean UI, Comet is worth exploring as an alternative to Weights & Biases. Beyond experiment tracking, Comet ML ships with monitoring features and an integrated model registry.
Since Comet ML’s experiment tracking functionality is deeply integrated with the platform, it is difficult to use as a standalone tool.
- Scalability: Users report that the UI is much faster than Weights & Biases’ but Comet’s UI tends to become slow for large numbers of experiments.
- Pricing: Comet’s Starter plan is billed based on the number of users and the training hours, potentially leading to similar cost issues as with Weights & Biases. For teams with more than ten members or in need of more training capacity, Comet offers an Enterprise plan.
- Self-hosting: Comet can be self-hosted, either on bare metal or virtual machines, on Kubernetes, or via the AWS Marketplace and Google Cloud Marketplace.
- Support and documentation: Comet’s Experiment Management documentation covers the experiment tracking features. Users can support each other and reach the developers via Comet’s Slack community. The Enterprise Plan includes access to support and SLAs.
Aim
Aim is an open-source experiment tracker developed by AimStack. It supports popular deep-learning frameworks but does not include a specific interface for scikit-learn. However, it offers a dedicated integration with the NLP framework spaCy. Teams migrating from Weights & Biases to Aim can transfer their data using a built-in converter.
In October 2023, AimStack released AimOS as the successor. It remains to be seen whether Aim will be actively developed further. Initially, they released AimOS as a direct replacement but later said they would continue the development of Aim.
- Scalability: Aim states in their GitHub README that their UI can “handle several thousands of metrics at the same time smoothly with 1000s of steps” but that the UI “may get shaky when you explore 1000s of metrics with 10000s of steps each.”
- Pricing: As an open-source platform, Aim can be used free of charge
- Self-hosting: The Aim experiment tracker can be run locally or hosted on Kubernetes for multiple users. No managed hosting options are available.
- Support and documentation: Aim’s documentation covers the essential parts of the application. Users can reach out for support on GitHub and in a Discord community. AimStack, the company behind Aim, offers enterprise support for Aim.
MLflow
MLflow is an open-source platform for managing the entire machine-learning lifecycle from experimentation to deployment. It comprises four key components: Tracking, Model Registry, Projects, and Models.
The MLflow Tracking component provides an API for logging ML metadata (including parameters, code versions, metrics, and output files) and a UI for visualizing and analyzing the results.
Since it’s available free of charge, MLflow is one of the experiment-tracking solutions many data scientists explore first, and several other tools offer a compatible API, elevating it to a de facto standard. However, it lacks many crucial features, such as user access management, leading teams to explore MLflow alternatives.
MLflow can also be used in managed environments such as Amazon SageMaker AI and Azure Machine Learning, both of which offer MLflow client compatibility. In these setups, users log experiments using the MLflow API, but the data is sent to proprietary backends maintained by the respective cloud provider:
- Amazon SageMaker introduced managed MLflow tracking in mid-2024, replacing its earlier SageMaker Experiments module. SageMaker provides pre-configured MLflow tracking servers in three sizes, with the largest option supporting up to 100 transactions per second (200 in burst mode). This MLflow integration enables greater flexibility and interoperability with external tools in the MLflow ecosystem, while still benefiting from the managed infrastructure and autoscaling capabilities of the AWS ecosystem.
- Azure Machine Learning (or AzureML) also supports experiment tracking via the MLflow client, but the backend is not based on the open-source MLflow server. You can use an Azure Machine Learning workspace as the tracking backend for any MLflow code, even if the code runs outside of Azure. Simply configure the MLflow client to point to your Azure ML workspace, and it will log runs there. AzureML integrates well with Microsoft Entra and Azure’s RBAC model, providing enterprise-grade security and access management.
These integrations make it easy to adopt MLflow syntax within cloud workflows, but they don’t deliver the full capabilities of a self-hosted MLflow stack.
- Scalability: MLflow reportedly faces challenges when tracking a large number of experiments or machine-learning models.
- Pricing: MLflow is available open-source, which allows anyone to run it without incurring license fees. However, hosting an MLflow instance comes with costs for the infrastructure and maintenance. The lack of access and authentication features requires teams to develop and integrate their own.
- Self-hosting: Since it is an open source platform, MLflow has to be self-hosted, which is possible in different setups and configurations. Managed variants of MLflow are offered by Databricks and as part of Amazon SageMaker.
Support and documentation: MLflow comes with solid documentation. Thanks to its broad community, users can find support in forums and discussion groups. Compared to support plans offered by vendors, there is no guarantee for timely responses and typically users have to share their questions publicly.
The Real Cost of Self-Hosting MLflow
ClearML Experiment
ClearML (formerly Allegro Trains) is an open-source MLOps solution for automating and streamlining the machine learning lifecycle from prototyping over training to deployment and monitoring.
ClearML Experiment is the integrated experiment-tracking component. It can log code, notebooks, configuration files, and containers. ClearML Experiment captures experiment information automatically and is framework-agnostic. Since it is tightly integrated with the other ClearML platform components, it is difficult to use as a standalone tool.
- Scalability: ClearML’s backend can spawn multiple processes, making full use of the hardware resources a user provides. Users have reported that the web app can become slow as the number of experiments approaches 1 million.
- Pricing: As an open-source platform, ClearML is not subject to license fees. The managed offering’s pricing model is predominantly based on resource consumption in the Pro plan, while the Scale and Enterprise plans are subject to individual negotiation.
- Self-hosting: The backbone of the ClearML platform is the ClearML Server, which—like the other platform components—is open source. It is Docker-based and can be hosted on Linux, MacOS, Windows, and Kubernetes.
User management features like access rules, user groups, centralized permissions management, and integrations with identity providers are only available in the SaaS version under the Enterprise plan. - Support and documentation: ClearML comes with extensive documentation covering the various components, accompanied by a collection of video tutorials. Users can connect with the developers and fellow community members via a dedicated Slack community and GitHub issues. Support and SLAs are available as part of the Scale and Enterprise plans.
One particularly interesting feature of ClearML is the ability to launch remote sessions. This enables data scientists and ML engineers to develop models and debug training scripts directly on the machines that their model training eventually runs on.
Google Vertex AI
Vertex AI is the end-to-end ML solution integrated into the Google Cloud Platform (GCP). Vertex AI combines Google’s AI offerings into a cohesive environment, facilitating the end-to-end ML workflow from data analysis and model development to deployment and monitoring.
Its integration with GCP’s databases and orchestration tools makes Vertex AI an interesting option for teams working within the Google ecosystem. Via GCP’s Identity and Access Management (IAM) capabilities, Vertex AI provides enterprise-grade permissions management.
Vertex ML Metadata is the experiment tracking component integrated into Vertex AI. Tracking is not a first-class feature. Based on the ML Metadata (MLMD) library maintained by the TensorFlow Serving team, it tracks metrics and artifact lineage for any machine-learning framework. For visualization and analysis, Vertex AI integrates the open-source TensorBoard.
- Scalability: Google does not provide information regarding scalability or the underlying resources.
- Pricing: Users tracking experiments with Vertex ML Metadata pay a fixed price per gigabyte of storage consumed.
- Self-hosting: Vertex ML Metadata is exclusively available as part of the cloud-hosted Vertex AI platform. All data, models, and metadata are stored on GCP.
- Support and documentation: Documentation for Vertex ML Metadata is included in the Vertex AI documentation. GCP offers support for Vertex AI through different channels, including support packages with access to technical experts.
Table comparison of alternatives to Weights & Biases based on core experiment tracking features
In the table below, I’ve summarized the key capabilities of alternatives to Weights & Biases discussed in this article:
N/A
N/A
SaaS version’s pricing model is based on resource consumption
Based on storage utilization
Possible
Not possible
Extensive documentation, multiple community channels, enterprise support available
Documentation and support through Google
Wrapping up
Weights & Biases is a popular and widely-known ML experiment-tracking platform. However, teams often encounter limitations and missing features as they scale and build out their ML efforts.
Whether Weights & Biases is the right choice for your team depends on its size, the type of machine-learning models you train, and the usage frequency. By reviewing Weights & Biases’ widely lauded features and limitations, you’ve learned the key considerations when evaluating alternatives, including pricing models, onboarding, administration, and integration concerns.
For the same reason that Weights & Biases is not a good fit for every team, none of the discussed competitors is the clear best alternative. But by carefully analyzing your requirements and a tools’ capabilities, you can find the ideal option for your team.
More about The Best Weights & Biases Alternatives
Check out our product resources and related articles below:
Related article
From Research to Production: Building The Most Scalable Experiment Tracker For Foundation Models
Related article
Learnings From Teams Training Large-Scale Models: Challenges and Solutions For Monitoring at Hyperscale
Related article
How to Migrate From MLflow to Neptune
Product resource
How Neptune Helps Artera Bring AI Solutions to Market Faster
