In this post, we’ll show you the best alternatives to Sacred + Omniboard. Since you came across this article, you may already know what these tools are made for. But let’s analyze them shortly as it could be helpful in better defining your needs and choosing the best alternative for you.
It would be a sin to mention one tool without the other. And that’s because they kind of cooperate.
Let’s begin with Sacred. Sacred is a tool developed at the Swiss AI Lab IDSIA (Dalle Molle Institute for Artificial Intelligence Research) that helps to configure, organize, log, and reproduce experiments.
Here’s Sacred in a nutshell:
- Track any model training developed with any Python library
- Access all parameters of your configuration from every function
- You have access to a powerful command-line interface: for each experiment, you can use it to change parameters and run different variants
- Provides “Observers” for various database backends that log all kinds of information about the experiment, its dependencies, the configuration used, the machine it is run on, and the result. These can be saved to a MongoDB, filesystem, hosted experiment tracking tool and other things
- Automatic seeding helps to control the randomness in experiments, such that the results remain reproducible
- Doesn’t come with its proper UI but there are a few dashboarding tools that you can connect to it (like Omniboard).
Now, let’s get to Omniboard. Omniboard is a web dashboard for Sacred. It connects to the MongoDB database used by Sacred and helps to visualize the experiments and metrics/logs collected for each experiment. Omniboard is written in React, Node.js, Express and Bootstrap.
And here is a short overview of what you can do with Omniboard:
- You have access to experiment management features such as a list of all experiment runs in a customizable tabular format with columns
- You can compare experiments in the UI
- You have access to detailed data on experiments ( metric graphs, source files, artifacts, git hash/version control info, and more)
- Only supports connecting to MongoDB >= 4.0
These two are a great combination but sometimes, it’s not enough or you may just not like it. There’s a way to combine tools or replace them with solutions that are better for you. You only need to know what’s out there. Here are the best Sacred + Omniboard alternatives. Let’s dig deeper.
Omniboard alternatives
Omniboard is the main front-end (UI) for Sacred (which is an experiment database). But there are other options you can use, especially, if you want more flexibility.
Some people may want to stay with Sacred logging API but use some other UI. Here are two great options.
1. Neptune
Neptune is a metadata store for MLOps built for research and productions teams that run a lot of experiments. Neptune offers an open-source Python library that lets users log any experiments, so it’s not restricted to deep learning.
Projects in Neptune can have multiple members with different roles (viewer, contributor, admin), so all machine learning experiments that land in Neptune can be viewed, shared and discussed by every team member. Neptune is meant to provide an easy-to-use and quick-to-learn way to keep track of all the metadata generated during the ML life cycle.
Neptune fits any workflow and gives a lot of flexibility. When it comes to Sacred, Neptune has Sacred integration that replaces Omniboard and MongoDB backend with its own Observer. With the integration, you can log Sacred experiments and manage them directly in Neptune:
- You don’t have to set your DB up,
- All your data is logged to the cloud or, if you prefer, on-premise
- All you have to do is simply add NeptuneObserver to the experiment:
# Create sacred experiment
ex = Experiment('image_classification', interactive=True)
# Add NeptuneObserver
ex.observers.append(NeptuneObserver(run=neptune_run))
Read more
2. Sacredboard
Just like Omniboard, Sacredboard is a web dashboard for Sacred. It lets you access and monitor all your past ML experiments.
The tool connects to the MongoDB database used by Sacred and displays a list of experiments, their state, Sacred configuration, and the standard output from the running program.
If you want it to work properly, you’ll need Python 3.5 and a modern web browser.
It may have a bit limited functionality in comparison to Omniboard as its improvement is a slow process. But if you’re looking for a different solution, you will find Sacredboard sufficient enough.
Here are some of the main features:
- You can run Tensorboard directly from the web console to see detailed information, charts, and Tensorflow graph visualizations
- You can Filter experiments
- You get an overview of running and finished experiments in a table
Sacred alternatives
Sacred and Omniboard make a great couple but you can also replace Sacred entirely and use a different logging API. Take a look at these two examples.
1. MLflow
MLflow is an open-source platform that helps to manage the whole machine learning lifecycle—experimentation, reproducibility, deployment, and a central model registry.
The tool is library-agnostic. You can use it with any machine learning library and in any programming language.
MLflow—summary:
- You can log parameters, code versions, metrics, and artifacts when running machine learning code and for later visualize and compare the results
- Reusable, reproducible form of packaging ML code allows you to share it with other data scientists or transfer it to production
- You can manage and deploy models from different ML libraries to a variety of model serving and inference platforms
- A central model store lets you collaboratively manage the full lifecycle of an MLflow Model, including model versioning, stage transitions, and annotations
Check also
2. Guild AI
Guild AI is a tool for running, tracking, and comparing experiments. Guild AI is cross-platform and framework independent — you can train and capture experiments in any language using any library. Guild AI runs your unmodified code so you get to use the libraries you want. The tool doesn’t require databases or other infrastructure to manage experiments — it’s simple and easy to use.
Guild AI – summary:
- Lets you track experiment of any model training and any programming language
- Has automated machine learning process
- Integrated with any language and library
- Remote training and backup possibility
- You can reproduce your results or recreate experiments
May be useful
To wrap it up
Working on your machine learning experiments in Sacred + Omniboard is efficient for sure. But combining different tools gives you more flexibility and freedom so you can get the best out of your work. It’s worth trying out different combinations so you can stick to the one that is most effective.
We hope you found something of interest. Happy experimenting!
