Case Study

TH Köln

My productivity in collaborating with students and also my own research speed increased dramatically. I wouldn’t know how to do my work without Neptune.

Jan Bollenbacher

Research Assistant at TH Köln

TH Köln is Germany’s biggest University of Applied Sciences in the Electrical Engineering Department. Jan works there as a member of the staff and is also pursuing his PhD alongside. Therefore, it is no surprise that he is regularly engaged in multiple projects collaborating with both students and colleagues. One of the ideal goals of Jan’s department is to pave the way for students to understand, experiment, and leverage great tools that could potentially accelerate technological projects.

*Deutz Campus, TH Köln | Source: TH Köln*

When it comes to Artificial Intelligence and Machine Learning, the department is focused on meta-learning research with standard ML frameworks such as TensorFlow and PyTorch.

Problem

As most academic researchers will agree, working with colleagues on research papers while managing multiple student projects is not an easy task, especially in research departments that handle large-scale projects.

Like many research teams, Jan and his team managed experiment-tracking across multiple servers by manually creating CSV files to record details generated during the experiment run such as loss or f2 score. On top of the CSV file that recorded loss and metrics, there were additional files with hyperparameters and other configurations. This made the analysis of past experiments extremely challenging and prone to errors due to the manual management of multiple files.

It was horrible times. We ran experiments on several servers. I saved the data generated during the run in a CSV file and created an excel file where I stored hyperparameters. You would need to download every CSV from every server when you wanted the data, and then load our experiment and Python and plot it. It was painful and very prone to mistakes because it was so much manual work.”

Jan Bollenbacher Research Assistant at TH Köln

Some of the recurrent problems that Jan’s research teams were facing were:

1 Multi-server project management
2 Problems with access control management
3 Multi-step process for result comparison and presentation
4 When students leave we lose experiment history

Since Jan’s department handles large research projects, they run across multiple servers. This implies that data is generated by each of those servers and must be recorded accordingly. Not having a centralized location for data logging and analysis is time-consuming and exhausting for the researchers. Also, it is risky if the data generated by each server isn’t backed up at different points.

“Once I tried to log into our servers, and all servers were gone. There was a problem with our power supply due to heavy rain in Germany and water was coming into the server room. It was lucky we didn’t lose any hardware, but if something like that happens, I would be happy to have all my data backed up in a service like Neptune.” – Jan Bollenbacher, Research Assistant at TH Köln
Jan’s team works across multiple servers, and every user has an account on the servers. However, only Jan, the project administrator, can view and get information from every file. The students who do not have administrator access cannot access Jan’s data or projects and are prone to repeat mistakes that Jan might have fixed. To avoid this, the team has to set up something like a dropbox which is not ideal for research collaboration.

“I’m an administrator so I can access the files. But if the student wants to look into my experiments, it will be a problem. Due to lack of visibility, when I don’t do mistakes, he might repeat the mistakes. You would need to have something like a Dropbox folder which doesn’t work if you want to collaborate.” – Jan Bollenbacher, Research Assistant at TH Köln
Before Neptune, the usual process used by the department for experiment tracking was to record data in multiple CSV files and upload them to the respective servers. So, when the data is required, every CSV file from every server has to be downloaded and organized carefully for comparison, and eventually, presentation.

“We have several servers and run experiments on all of them. When you want to look into the data, you would need to download every CSV from every server. This is very prone to mistakes because of so much manual work.” – Jan Bollenbacher, Research Assistant at TH Köln
Students play a big role in academic research projects. This can often make the projects unstable since students move around and are often temporary project participants. When students leave, it could be challenging to retain and maintain the work they did or track the experiment data that they previously managed.

“Sometimes the projects students do for master thesis, bachelor thesis or research projects overlap with my work. Therefore it’s necessary for me to have access to all experiment data because if the students left, maybe the data has also left.” – Jan Bollenbacher, Research Assistant at TH Köln

To find a one-stop solution that could serve all the above problems, Jan started looking for open-source software (OSS) solutions.

Solution

Why we didn’t choose open-source

When Jan started looking into OSS solutions, he came across metadata collection services like Sacred. However, Jan stresses that such options meant a lot of work on the administrative side. It could steer the researcher’s focus from core ML experiments such that they end up devoting significant time to organizing and maintaining administrative work.

Since I’m a researcher in machine learning, I can do the administrative things, but I don’t like to do this.

Jan Bollenbacher Research Assistant at TH Köln

Some such inconveniences include the following:

Installation hiccups: Complex installations can consume massive chunks of the researcher’s time. As Jan shared, if you don’t follow the complex installation instructions for an OSS, the software will not work the right way, and even when it does, you may hit problems if the software automatically updates – the system has to be reset again.

I really liked the open-source community and the open-source thinking. But when it comes to working with it, everything cracks if you don’t do exactly how they planned the installation. You’d spend a week with multiple installations to get everything up and running.

Jan Bollenbacher Research Assistant at TH Köln

Security concerns: Jan shares a dominant concern on security threats that could arise from delayed open source updates impacting all the servers behind a very restrictive firewall. Such updates can be postponed by researchers in the interest of time and experiment prioritization, given that resets and installations take up considerable time.

After spending days or weeks with installation, there might be an update coming which blows up everything. Therefore, you wouldn’t do updates frequently which could be a security concern.

Jan Bollenbacher Research Assistant at TH Köln

Resource consumption: The open-source software, once installed will run on a project server. For Jan, this will mean dedicating resources to experiment-tracking which he would rather use for training models. It will be ideal to use the full capacity of resources such as CPU, GPU, and RAM to run experiments, given that most experiments demand 100% resource consumption.

The open-source software would run on one of our servers, yet we have a lot of calls and we need it for our experiments which are always on 100% resource consumption.

Jan Bollenbacher Research Assistant at TH Köln

Why we chose Neptune

Narrowing down the research further, Jan started looking for solutions that were less taxing from an administrative point of view.

Jan stumbled upon Neptune on a tech forum while looking for a suitable solution. What got his attention was the minimal setup time and a zero-hassle integration to his workflow. As the forum said, Neptune was supposed to be ready in 5-10 minutes, and it didn’t take Jan a minute longer.

Jan chose Neptune during the first few minutes of trial and was a happy Neptune user ever since which at the time of writing was two years.

Often, the most challenging aspect of any new integration is to get other people on board. Yet, it has never been easier to onboard collaborators or colleagues with Neptune’s one-time integration and smart tracking and presentation capabilities. Below are some first-hand experiences of Jan’s students and colleagues when they crossed paths with Neptune:

I did some research, then I wrote to Florian that we will use Neptune. I showed him how easy it was to implement into the code and how you can access the data and in two or three minutes, he was in love with it.

The Student

We were participating… in a hackathon. The team wrote something like Neptune on their own server to notify us and to have an automated generation of graphs. Then I showed them Neptune and they realized what a time saver it would be. Without Neptune, a lot of things they needed to do and implement on their own. They were very happy to be acquainted with Neptune.

The Team

We had a post-doc visiting us, and I showed live on Neptune my results and we wrote a few queries and looked into the data. After the meeting, he wrote to me asking what was the software I was using which was super simple and really lite. I’m very sure that today he’s also using it.

External Reviewers

Results

After choosing Neptune, these are some of the benefits that Jan and his research teams could leverage:

There’s a record of everything that has happened during the research project. Neptune diffuses the problem of lost data when a team member leaves and acts as a common repository for multiple projects that share the same backbone.

“Neptune was very helpful in collaborating with a student who was doing his master’s thesis with me. Everything was stored in one database, and I could still use the information even after he left.” – Jan Bollenbacher, Research Assistant at TH Köln
Neptune captures and processes metadata and experiment details, making it possible to track patterns that are easily missed on manual records. A feature that Jan uses with his students is Neptune’s ability to generate links corresponding to views and visualizations that can convey the exact snapshots.

Easy link creation and sharing options on neptune.ai | Source: Neptune.ai

“That’s a function I use often with students. They send me a link and when I click on the link, I know what they are saying and which experiment they are talking about.” – Jan Bollenbacher, Research Assistant at TH Köln
Neptune’s quick and easy query language and search options on the UI save research teams tons of time since they can easily find data on the go, even during live meetings.

Saving filter views on neptune.ai | Source: Neptune.ai

“For each item, I write search queries in Neptune and get all the results into one graph. I also query the Neptune database during the meetings because sometimes you need additional info on the go. With the new search, it’s very easy to query data, and I extensively use the tags and the tagging function is really helpful for me.” – Jan Bollenbacher, Research Assistant at TH Köln
When running multiple experiments, a significant amount of time is eaten up by clerical tasks such as copying queries from older to newer experiments and recreating presentation views. Reproducing query runs and visualizations becomes an autonomous process with Neptune’s ability to save filter views and to store metadata and queries in a central repository.

Search or filter run views on neptune.ai | Source: Neptune.ai

“In the past, I copied queries to a text file to have them for a different experiment. But with Neptune, it would be much easier to select a different view. You can just go back to a view and have the same process done with the new data- all in one chart without setting anything or writing anything new.” – Jan Bollenbacher, Research Assistant at TH Köln
Without a common repository with easily manageable access controls like view or edit access, experiment- and file-sharing could become a frequent nightmare impacting effective collaboration. Neptune’s one-stop platform was Jan’s choice for easy collaboration with his students.

“I’m an administrator so I can access the student’s files but he, can’t. It was difficult to collaborate. So, I did some research, then I wrote to Florian that we will use Neptune. I showed him how easy it was to implement into the code and how you can access the data and in 2 or 3 minutes, he was in love with it.” – Jan Bollenbacher, Research Assistant at TH Köln
While open-source solutions are a no-go for Jan due to difficult installations, Neptune’s 5-10 minute setup, only a few lines of central code, and fewer points of integration were ideal for Jan and his students. Neptune’s on-point documentation is prepared to exactly cater to the user’s need for a quick and hassle-free setup.

“Neptune is well documented and very easy to use. In five or ten central lines of code that you need to implement your project, you will have something like a logger right away. Then you just would need to have about five different places where you need to implement Neptune.” – Jan Bollenbacher, Research Assistant at TH Köln
Neptune support staff is extremely attentive and is always on the lookout for distressed users. It is not unnatural for a user to face some bumps during the early days of platform integration. The support team is always prepared to resolve such issues with pre-existing or quick solutions from Neptune’s active developers.

“I like it, that if there is a problem the very competent support tackles this and solves this instant. The support sees the problem, consults with the engineering team, and gets back within an hour or so.” – Jan Bollenbacher, Research Assistant at TH Köln
It is challenging to find platforms that offer highly detailed interactive visuals that are user-friendly and easily customizable. Neptune ensures that your tracking data is automatically well-represented through exhaustive visuals along with informative data. This feature, even though subtle, makes a huge difference in making a researcher’s tasks easier.

Using filters on metrics or loss charts on neptune.ai | Source: Neptune.ai

“If you hover over the graph, it is possible to select fields which are shown. So this was one thing I like. It’s a very small feature but it’s super helpful and you miss it when it’s not there. People are talking about this.” – Jan Bollenbacher, Research Assistant at TH Köln
Neptune is flexible and can adjust what you log and also enable you to customize your visualizations. It gives the users high control over their tracking methods.

“With the latest version of Neptune, you’re allowed to do the whole folder structure as you like in the experiment. Therefore, you have a lot of flexibility. It’s very easy and straightforward and there’s no magic. You just need to read in the documentation.” – Jan Bollenbacher, Research Assistant at TH Köln