Gone are the days of drawn-out spreadsheets detailing chaotic ML experiments. With platforms such as Sacred, you can log experiments with peace of mind. Sacred does the good work for you: it keeps track of your parameters, model architectures, dataset changes, training jobs, artifacts, metrics, meta, debugging, and more.
In essence, Sacred is a Python module that lets you configure, organize, log, and reproduce experiments. This way, you can:
- easily manage your experiment parameters
- play around with settings for your experiments
- save configurations for individual runs in a MongoDB database
- replicate results
Sacred comes with a robust command-line interface, where you can change parameters and carry out different experiments and variants. Through its “observers,” it can log a variety of details about experiments – like dependencies, configurations, machines used, or results. This info can go to a database or another experiment tracking tool.
Is there a catch? Well, for all of its worth, Sacred doesn’t have a thorough UI or key experiment tracking features.
This is where Omniboard comes in. Omniboard is Sacred’s twin – you can’t use one without the other. Omniboard is a NodeJS server that connects to Sacred’s database and visualizes experiments + metrics/logs. You can:
- Access experiment management features through a table or list view
- Compare experiments
- Review detailed data regarding experiments, such as metric graphs, source files, artifacts, or git hash/version control info.
The combination of Sacred and Omniboard can be powerful, but they lack valuable functionalities that can be crucial for large ML teams. For example:
- Features for detailed team collaboration
- UI that shows notebook versioning or notebook auto snapshots
- UI that can save experiment views or group experiments
- UI that can scale to millions of runs
- Dedicated user support
However, don’t lose hope yet. You can easily get all these features, along with the Sacred functionalities you know and love, with a Neptune integration.
Neptune is a lightweight ML experiment management tool. It’s flexible and easy to integrate with different workflows. Your teammates can use any ML libraries and platforms, share results, and collaborate on a single Neptune dashboard. If you don’t want to deploy it on your own hardware, you can use their web platform.
Neptune’s main features are:
- Experiment Management: keep track of all your team’s experiments, also tag, filter, group, sort, and compare them
- Notebook versioning and diffing: compare two notebooks or checkpoints in the same notebook; similarly to source code, you can do a side-by-side comparison
- Team Collaboration: add comments, mention teammates, and compare experiment results
For more details, see -> Neptune vs. Sacred + Omniboard
Neptune + Sacred Integration
Omniboard is the popular front-end for Sacred – however, as we discussed above, it lacks many key features, especially for collaboration. Neptune, on the other hand, lets you stay with the Sacred logging API, while providing its sleek, intuitive UI. It’s the more functional alternative to Omniboard.
When integrating with Sacred, Neptune replaces the MongoDB backend with its own Observer. This way, you don’t have to set up your database. All your data can be logged to the cloud or locally, however you like.
Now lets walk through how to sync them:
Neptune Setup (skip if you already have a Neptune account)
- Sign up for a Neptune AI account first. It’s free for individuals and non-organizations, and you get a generous 100 GB of storage.
- Get your API token by clicking the top right menu.
- Create a NEPTUNE_API_TOKEN environment variable and run it in your console. export NEPTUNE_API_TOKEN=’your_api_token’
- Create a project. In your Projects dashboard, click “New Project” and fill in the following information. Pay attention to the privacy settings!
First, you will need to install your neptune-client:
pip install neptune-client
Make sure to create an experiment as well:
ex = Experiment('iris_rbf_svm')
Then, pass the experiment object as the first argument:
from neptunecontrib.monitoring.sacred import NeptuneObserver ex.observers.append(NeptuneObserver(api_token='ANONYMOUS', project_name='shared/sacred-integration'))
Make sure to replace ‘ANONYMOUS’ with your own API token (see above!) and project name.
After that, simply run your experiments through Sacred as you normally would. Now, they will train in Neptune!
Collaborate with Neptune
Your experiment metadata should now be stored in Neptune, and you can view it in your experiment dashboard:
You can customize the dashboard by adding tags and grouping experiments with custom filters.
Neptune lets you share ML experiments simply by sending a link. It can be:
Neptune also comes with workspaces, a central hub where you can manage projects, users, and subscriptions; there are individual and team workspaces.
In the team workspace, team members can browse the content that’s related to their assigned role. You can assign various roles in projects and workspaces. In a team workspace, you can invite people either as admin or member, each with different privileges.
Workplace settings can be changed in the workspace name on the top bar:
Under the Overview, Projects, People and Subscription tabs, you can see workplace settings:
There are three roles in a project: owner, contributor, and viewer. Depending on the role, users can run experiments, create notebooks, modify previous stored data, etc.
For more details, see -> User Management
Learning more about Neptune
As you can see, Neptune complements Sacred nicely. You can easily replace Omniboard and add the Neptune Observer to unlock more functionality. Neptune can act as a central hub to manage your experiments and collaborate on them with your team.