Case Study

How ailslab Uses Neptune’s Standardized Logging to Cut out Miscommunication

I would say the main argument for using Neptune is that you can be sure that nothing gets lost, everything is transparent, and I can always go back in history and compare.

Thore Bürgel

PhD Student at AILS Labs

Before

Lack of standardized practices when building and analyzing models

After

Standardized logging methods allowing for easy comparison of results

More time for the actual research work thanks to automated tracking

Optimized project management

ailslab is a small bioinformatics research group on a mission to make humanity healthier. That mission is to build models which might someday save your heart from illness. It boils down to applying machine learning to predict cardiovascular disease development based on clinical, imaging, and genetics data.

Research is so intense that it required a custom infrastructure (which took about a year to build) to extract features from different types of data, including electronic health records (EHR), image (convolutional neural networks), or structured data and ECG.

The problem

ailslab started as a small group of developers. Collaboration became more challenging, and new problems began to appear along with the inflow of new team members. They noticed those problems quickly, and started to look for a solution.

Scaling up Machine Learning research with Neptune

Keeping the data safe

Due to limited data, Ailslab has to work with NDA-protected private data. Models can only be trained locally, and the data can’t be uploaded anywhere, so there was a lot of additional work to ensure the data’s safety.

ailslab separated the data workflows from the analysis workflows with Neptune in between.
Neptune keeps sensitive data safe, it just receives logged information that researchers decide to share, and the training part can happen on a local machine or anywhere else. It gives ailslab maximum control.

Standardizing the workflow

As a small team, ailslab didn’t need standardized practices. But as the team kept growing and new developers brought along different programming styles, it became harder to manage code development.

Thanks to Neptune, researchers use a standard library to build models, which is much easier than writing custom code. In addition, they have a standardized view of logged information with the PyTorch Lightning integration. All team members use the same infrastructure.

Neptune unifies how everyone presents results, so there’s less miscommunication.

Neptune works flawlessly, and integrating it with PyTorch Lightning was very smooth.

Jakob Steinfeldt Physician Researcher at ailslab

Simplifying metadata logging

With a custom logger, it was challenging to answer essential questions when analyzing experiments. Plus, a custom logger comes with the burden of managing the logger long-term and adding new features when necessary. When bugs happened, that’s even more time to build internal tools instead of doing research.

Neptune can group experiments for comparison. It’s easy to get a link to share the results with another researcher or stakeholder.

Even if a researcher leaves the project and is no longer available, Neptune saves all information about their experiments.

Neptune automatically logs each experiment through the API. All experiments are visible to all team members, making the whole project transparent.

Efficient methods for feature and model selection

Detailed patient records consist of multimodal data, which means multimodal training for ailslab researchers. It’s a highly complex process with an enormous number of moving parts to test. With as many experiments as the ailslab team does, it was difficult to keep track of all models trained on different versions of a dataset and its features.

Selecting the best features for their models is easier because comparing them in Neptune is quick and straightforward. Researchers compare model performance, detailed parameters, and even hardware resource consumption. Scale is no issue. Neptune handles any amount of experiments that ailslab researchers throw at it.

Smooth experiment management

ailslab researchers had to do manual tasks like creating checkpoints by hand or figuring out how to change one or more hyperparameters to do another experiment.

ailslab leaders can supervise researchers and compare their experiments all in one dashboard. The team no longer cares about organizing experiments, as Neptune does it quite elegantly. Plus, Neptune versions data for better control of experiments.

So I would say the main argument for using Neptune is that you can be sure that nothing gets lost, everything is transparent, and I can always go back in history and compare.

Thore Bürgel PhD Student at ailslab

The results

If you’re a researcher, you know that managing multiple experiments is challenging. With such complex objectives and workflows, the ailslab team has to do a lot of tedious work to stay on the right track. With Neptune, that’s not the case.

The most important benefits for aislab are:

Automated tracking gives the team more time for the actual research work.
A centralized and standardized place for all results leaves less room for mistakes.
Out-of-the-box comparison capabilities allow for iterating much quicker.
Building and reproducing complex models is much easier, as Neptune stores all the necessary data (environment setup, the underlying code, and the model architecture).
Integrating Neptune’s sharable URLs into Kanban boards significantly improved project management.

To learn more about ailslab, check out their full story.

Thanks to Jakob Steinfeldt and Thore Bürgel for their help in creating this case study!