Gone are the days of drawn-out spreadsheets detailing chaotic ML experiments. With platforms such as Sacred, you can log experiments with peace of mind. Sacred does the good work for you: it keeps track of your parameters, model architectures, dataset changes, training jobs, artifacts, metrics, meta, debugging, and more.
In essence, Sacred is a Python module that lets you configure, organize, log, and reproduce experiments. This way, you can:
- easily manage your experiment parameters
- play around with settings for your experiments
- save configurations for individual runs in a MongoDB database
- replicate results
Sacred comes with a robust command-line interface, where you can change parameters and carry out different experiments and variants. Through its “observers,” it can log a variety of details about experiments – like dependencies, configurations, machines used, or results. This info can go to a database or another experiment tracking tool.
Is there a catch? Well, for all of its worth, Sacred doesn’t have a thorough UI or key experiment tracking features.
This is where Omniboard comes in. Omniboard is Sacred’s twin – you can’t use one without the other. Omniboard is a NodeJS server that connects to Sacred’s database and visualizes experiments + metrics/logs. You can:
- Access experiment management features through a table or list view
- Compare experiments
- Review detailed data regarding experiments, such as metric graphs, source files, artifacts, or git hash/version control info.
The combination of Sacred and Omniboard can be powerful, but they lack valuable functionalities that can be crucial for large ML teams. For example:
- Features for detailed team collaboration
- UI that shows notebook versioning or notebook auto snapshots
- UI that can save experiment views or group experiments
- UI that can scale to millions of runs
- Dedicated user support
However, don’t lose hope yet. You can easily get all these features, along with the Sacred functionalities you know and love, with a Neptune integration.
Neptune is a lightweight ML experiment management tool. It’s flexible and easy to integrate with different workflows. Your teammates can use any ML libraries and platforms, share results, and collaborate on a single Neptune dashboard. If you don’t want to deploy it on your own hardware, you can use their web platform.
Neptune’s main features are:
- Experiment Management: keep track of all your team’s experiments, also tag, filter, group, sort, and compare them
- Notebook versioning and diffing: compare two notebooks or checkpoints in the same notebook; similarly to source code, you can do a side-by-side comparison
- Team Collaboration: add comments, mention teammates, and compare experiment results
For more details, see -> Neptune vs. Sacred + Omniboard
Neptune + Sacred Integration
Omniboard is the popular front-end for Sacred – however, as we discussed above, it lacks many key features, especially for collaboration. Neptune, on the other hand, lets you stay with the Sacred logging API, while providing its sleek, intuitive UI. It’s the more functional alternative to Omniboard.
When integrating with Sacred, Neptune replaces the MongoDB backend with its own Observer. This way, you don’t have to set up your database. All your data can be logged to the cloud or locally, however you like.
Now lets walk through how to sync them:
Neptune Setup (skip if you already have a Neptune account)
- Sign up for a Neptune AI account first. It’s free for individuals and non-organizations, and you get a generous 100 GB of storage.
- Get your API token by clicking the top right menu.
- Create a NEPTUNE_API_TOKEN environment variable and run it in your console. export NEPTUNE_API_TOKEN=’your_api_token’
- Create a project. In your Projects dashboard, click “New Project” and fill in the following information. Pay attention to the privacy settings!
First, you will need to install your neptune-client:
pip install neptune-client
Make sure to create an experiment as well:
ex = Experiment('iris_rbf_svm')
Then, pass the experiment object as the first argument:
from neptunecontrib.monitoring.sacred import NeptuneObserver ex.observers.append(NeptuneObserver(api_token='ANONYMOUS', project_name='shared/sacred-integration'))
Make sure to replace ‘ANONYMOUS’ with your own API token (see above!) and project name.
After that, simply run your experiments through Sacred as you normally would. Now, they will train in Neptune!
Collaborate with Neptune
Your experiment metadata should now be stored in Neptune, and you can view it in your experiment dashboard:
You can customize the dashboard by adding tags and grouping experiments with custom filters.
Neptune lets you share ML experiments simply by sending a link. It can be:
Neptune also comes with workspaces, a central hub where you can manage projects, users, and subscriptions; there are individual and team workspaces.
In the team workspace, team members can browse the content that’s related to their assigned role. You can assign various roles in projects and workspaces. In a team workspace, you can invite people either as admin or member, each with different privileges.
Workplace settings can be changed in the workspace name on the top bar:
Under the Overview, Projects, People and Subscription tabs, you can see workplace settings:
There are three roles in a project: owner, contributor, and viewer. Depending on the role, users can run experiments, create notebooks, modify previous stored data, etc.
For more details, see -> User Management
Learning more about Neptune
As you can see, Neptune complements Sacred nicely. You can easily replace Omniboard and add the Neptune Observer to unlock more functionality. Neptune can act as a central hub to manage your experiments and collaborate on them with your team.
InstaDeep Case Study: Looking for Collaboration Features and One Central Place for All Experiments
5 mins read | Updated November 22th, 2021
InstaDeep is an EMEA leader in delivering decision-making AI products. Leveraging their extensive know-how in GPU-accelerated computing, deep learning, and reinforcement learning, they have built products, such as the novel DeepChain™ platform, to tackle the most complex challenges across a range of industries.
InstaDeep has also developed collaborations with global leaders in the AI ecosystem, such as Google DeepMind, NVIDIA, and Intel. They are part of Intel’s AI Builders program and are one of only 2 NVIDIA Elite Service Delivery Partners across EMEA. The InstaDeep team is made up of approximately 155 people working across its network of offices in London, Paris, Tunis, Lagos, Dubai, and Cape Town, and is growing fast.
About the BioAI team
The BioAI team is the place at InstaDeep where Biology meets Artificial intelligence. At BioAI, they advance healthcare and push the boundaries of medical science through a combination of biology and machine learning expertise. They are currently building DeepChain™, their platform for protein design. They are also working with their customers in the bio sector to tackle the most challenging problems with the help of bioinformatics and machine learning.
They apply the DeepChain™ protein design platform to engineer new sequences for protein targets using sophisticated optimization techniques such as reinforcement learning and evolutionary algorithms. They also leverage Language Models pre-trained on millions of protein sequences and train their own in-house protein language models. Finally, they use machine learning to predict protein structure from sequence.
Building complex software like DeepChain™, a platform for protein design, requires a lot of research with different moving parts. Customers demand various types of solutions that require new experiments and research every time. With several experiments running for different customers, it will be unavoidably daunting for a team of any size to keep track of the experiments while ensuring they remain productive.
Fazed with the thought of managing numerous experiments, Nicolas and the BioAI team encountered a series of challenges:
- 1Experiment logs were all over the place
- 2It was difficult to share experiment results
- 3Machine learning researchers were dealing with infrastructure and operations