Blog » ML Tools » How to Make Your Sacred Projects Easy to Share and Collaborate On

How to Make Your Sacred Projects Easy to Share and Collaborate On

Gone are the days of drawn-out spreadsheets detailing chaotic ML experiments. With platforms such as Sacred, you can log experiments with peace of mind. Sacred does the good work for you: it keeps track of your parameters, model architectures, dataset changes, training jobs, artifacts, metrics, meta, debugging, and more. 

In essence, Sacred is a Python module that lets you configure, organize, log, and reproduce experiments. This way, you can: 

  • easily manage your experiment parameters
  • play around with settings for your experiments
  • save configurations for individual runs in a MongoDB database
  • replicate results

Sacred comes with a robust command-line interface, where you can change parameters and carry out different experiments and variants. Through its “observers,” it can log a variety of details about experiments – like dependencies, configurations, machines used, or results. This info can go to a database or another experiment tracking tool. 

Is there a catch? Well, for all of its worth, Sacred doesn’t have a thorough UI or key experiment tracking features. 

This is where Omniboard comes in. Omniboard is Sacred’s twin – you can’t use one without the other. Omniboard is a NodeJS server that connects to Sacred’s database and visualizes experiments + metrics/logs. You can: 

  • Access experiment management features through a table or list view
  • Compare experiments 
  • Review detailed data regarding experiments, such as metric graphs, source files, artifacts, or git hash/version control info. 

The combination of Sacred and Omniboard can be powerful, but they lack valuable functionalities that can be crucial for large ML teams. For example:

  • Features for detailed team collaboration 
  • UI that shows notebook versioning or notebook auto snapshots
  • UI that can save experiment views or group experiments
  • UI that can scale to millions of runs
  • Dedicated user support

However, don’t lose hope yet. You can easily get all these features, along with the Sacred functionalities you know and love, with a Neptune integration. 

Introducing Neptune

Logging metadata

Neptune is a lightweight ML experiment management tool. It’s flexible and easy to integrate with different workflows. Your teammates can use any ML libraries and platforms, share results, and collaborate on a single Neptune dashboard. If you don’t want to deploy it on your own hardware, you can use their web platform. 

Neptune’s main features are: 

  • Experiment Management: keep track of all your team’s experiments, also tag, filter, group, sort, and compare them 
  • Notebook versioning and diffing: compare two notebooks or checkpoints in the same notebook; similarly to source code, you can do a side-by-side comparison 
  • Team Collaboration: add comments, mention teammates, and compare experiment results

For more details, see -> Neptune vs. Sacred + Omniboard

Neptune + Sacred Integration 

Omniboard is the popular front-end for Sacred – however, as we discussed above, it lacks many key features, especially for collaboration. Neptune, on the other hand, lets you stay with the Sacred logging API, while providing its sleek, intuitive UI. It’s the more functional alternative to Omniboard. 

When integrating with Sacred, Neptune replaces the MongoDB backend with its own Observer. This way, you don’t have to set up your database. All your data can be logged to the cloud or locally, however you like. 

Now lets walk through how to sync them: 

Neptune Setup (skip if you already have a Neptune account)

  1. Sign up for a Neptune AI account first. It’s free for individuals and non-organizations, and you get a generous 100 GB of storage. 
  2. Get your API token by clicking the top right menu. 
Neptune Sacred get started
  1. Create a NEPTUNE_API_TOKEN environment variable and run it in your console. export NEPTUNE_API_TOKEN=’your_api_token’
  2. Create a project. In your Projects dashboard, click “New Project” and fill in the following information. Pay attention to the privacy settings!
Neptune Sacred new project

Integration

First, you will need to install your neptune-client

pip install neptune-client

Make sure to create an experiment as well: 

ex = Experiment('iris_rbf_svm')

Then, pass the experiment object as the first argument: 

from neptunecontrib.monitoring.sacred import NeptuneObserver
ex.observers.append(NeptuneObserver(api_token='ANONYMOUS',
                                   project_name='shared/sacred-integration'))

Make sure to replace ‘ANONYMOUS’ with your own API token (see above!) and project name. 

After that, simply run your experiments through Sacred as you normally would. Now, they will train in Neptune!

Neptune Sacred monitoring

Collaborate with Neptune

Your experiment metadata should now be stored in Neptune, and you can view it in your experiment dashboard:

Sacred Neptune integration

You can customize the dashboard by adding tags and grouping experiments with custom filters. 

Neptune lets you share ML experiments simply by sending a link. It can be:

Mlflow experiment links
Mlflow experiment charts
MLflow group of experiments
MLflow diagnosis plots

Neptune also comes with workspaces, a central hub where you can manage projects, users, and subscriptions; there are individual and team workspaces. 

In the team workspace, team members can browse the content that’s related to their assigned role. You can assign various roles in projects and workspaces. In a team workspace, you can invite people either as admin or member, each with different privileges. 

Workplace settings can be changed in the workspace name on the top bar: 

Neptune workspaces

Under the Overview, Projects, People and Subscription tabs, you can see workplace settings: 

Neptune workplace settings

There are three roles in a project: owner, contributor, and viewer. Depending on the role, users can run experiments, create notebooks, modify previous stored data, etc. 

For more details, see -> User Management

Learning more about Neptune

As you can see, Neptune complements Sacred nicely. You can easily replace Omniboard and add the Neptune Observer to unlock more functionality. Neptune can act as a central hub to manage your experiments and collaborate on them with your team. 

If you want to learn more about Neptune, check out the official documentation. If you want to try it out, create your account and start tracking your machine learning experiments with Neptune.


READ NEXT

InstaDeep Case Study: Looking for Collaboration Features and One Central Place for All Experiments

5 mins read | Updated November 22th, 2021

InstaDeep is an EMEA leader in delivering decision-making AI products. Leveraging their extensive know-how in GPU-accelerated computing, deep learning, and reinforcement learning, they have built products, such as the novel DeepChain™ platform, to tackle the most complex challenges across a range of industries. 

InstaDeep has also developed collaborations with global leaders in the AI ecosystem, such as Google DeepMind, NVIDIA, and Intel. They are part of Intel’s AI Builders program and are one of only 2 NVIDIA Elite Service Delivery Partners across EMEA. The InstaDeep team is made up of approximately 155 people working across its network of offices in London, Paris, Tunis, Lagos, Dubai, and Cape Town, and is growing fast.

About the BioAI team

The BioAI team is the place at InstaDeep where Biology meets Artificial intelligence. At BioAI, they advance healthcare and push the boundaries of medical science through a combination of biology and machine learning expertise. They are currently building DeepChain™, their platform for protein design. They are also working with their customers in the bio sector to tackle the most challenging problems with the help of bioinformatics and machine learning.

Deepchain dashboard
DeepChain dashboard | Source

They apply the DeepChain™ protein design platform to engineer new sequences for protein targets using sophisticated optimization techniques such as reinforcement learning and evolutionary algorithms. They also leverage Language Models pre-trained on millions of protein sequences and train their own in-house protein language models. Finally, they use machine learning to predict protein structure from sequence.

Problem

Building complex software like DeepChain™, a platform for protein design, requires a lot of research with different moving parts. Customers demand various types of solutions that require new experiments and research every time. With several experiments running for different customers, it will be unavoidably daunting for a team of any size to keep track of the experiments while ensuring they remain productive.

Fazed with the thought of managing numerous experiments, Nicolas and the BioAI team encountered a series of challenges:

  • 1Experiment logs were all over the place
  • 2It was difficult to share experiment results
  • 3Machine learning researchers were dealing with infrastructure and operations
Continue reading ->
The Best Sacred + Omniboard Alternatives

The Best Sacred + Omniboard Alternatives

Read more
Experiment tracking Experiment management

15 Best Tools for ML Experiment Tracking and Management

Read more
How to Fit Experiment Tracking Tools Into Your Project Management Setup

How to Fit Experiment Tracking Tools Into Your Project Management Setup

Read more
MLOps guide

MLOps: What It Is, Why It Matters, and How to Implement It

Read more