Case Study

How Neptune Let Continuum Industries Move on From Maintaining Their Clunky In-House Tool

Gone are the days of writing stuff down on google docs and trying to remember which run was executed with which parameters and for what reasons. Having everything in Neptune allows us to focus on the results and better algorithms.

Andreas Malekos

Head of Artificial Intelligence at Continuum Industries

Before

Struggled to maintain increasingly complex in-house solution

After

Have an out-of-the-box tracking tool that lets the team focus on the main product

Continuum Industries specializes in automating and optimizing the design of linear infrastructure assets like water pipelines and power lines, overhead transmission lines, subsea power lines, or telecommunication cables.

Their core product, Optioneer, uses optimization algorithms to generate efficient connections based on geospatial data and engineering assumptions.

The challenge

Each infrastructure project, such as an offshore power cable, represented a massive undertaking with significant financial stakes. Head of AI, Andreas Malekos explains: “The more reasonable designs you see, the better decision you can make. Optioneer can get you design assets in minutes at a fraction of the cost of traditional design methods.”

Since people rely on this optimization engine with million-dollar infrastructure design decisions, the team had to set up a proper validation and testing.

But the nature of the problem they are trying to solve presents additional challenges:

You cannot automatically tell whether an algorithm’s output is correct or not. This is not like in ML, where you label data to compute accuracy or recall on your evaluation set.
You need a set of example problems that is representative of the kind of problem that the algorithm will be asked to solve in production. Furthermore, these problems need to be versioned so that repeatability is as easily achievable as possible.

The team needed a robust system to manage and validate countless design variations efficiently.

From clunky in-house tool to out-of-the-box solution

Initially, the team has developed a relatively simple in-house solution. But with time, it became pretty complex and extremely clunky. It turned into a “product within a product” that they did not have time to maintain or fix when it broke.

So they started to look for a managed solution.

They realized that even though they do not use ML in the product, they face many of the same challenges that ML-in-production faces. That’s when they decided to properly investigate the MLOps solutions.

The team ended up choosing Neptune because they wanted a tool that:

Could easily track and visualize different types of data;

Allow them to track both local and cloud runs in the same way;
Wouldn’t require self-hosting or maintenance;

We’re still a fairly small team (10 devs or so), so we’d rather avoid having to manage this system ourselves, so we can focus on building our product and improving the AI. We had to do that with our previous system, and it was a huge time sink.

Andreas Malekos Head of Artificial Intelligence at Continuum Industries

Comparing, monitoring and debugging

After integrating Neptune into their codebase, the team started recording quality metrics (objective value, constraint violation), the final value of the design variables, and all the input parameters for production runs.

They can now compare plots of different metrics across runs.
They monitor and debug production runs: If the optimization crashes, recording the input parameters and the code version makes it fairly easy to replicate the error and find out why things crashed.
Having all parameters and results tracked in Neptune lets them quickly review what they have tried and plan the next steps relatively easily.

The ability to compare runs on the same graph was the killer feature, and being able to monitor production runs was an unexpected win that has proved invaluable.

Andreas Malekos Head of Artificial Intelligence at Continuum Industries

Flexible metadata structure

What plays a significant role in Neptune’s adoption within Continuum Industries is its flexible approach to the metadata structure. It makes it easy to use Neptune for pretty much anything:

When researching algorithm improvements, they can record results in their custom structure and compare them easily.
They can monitor and record production runs in a way that is convenient for debugging.
They can use Neptune as part of their CI pipeline by setting up many batch jobs, all writing metric data to the same Neptune run, and then comparing things to the current version of the master to make sure that they’ve not broken anything.

We are huge fans of the way data is structured in Neptune runs. The fact that we can basically design our own file structure effortlessly gives us enormous flexibility.

Andreas Malekos Head of Artificial Intelligence at Continuum Industries

For example, in the case of the engine tests, where multiple jobs write to a single run, the structure would look like this:

metrics: top-level folder where all the metrics are stored
- {TEST_CASE_NAME}
  - {INDEX}: each case is run multiple times with different seeds
    - {STAGE_NAME}: there are multiple stages during the optimization
      - metric0
      - metric1
      - metric2

Here’s a visual example of how the metadata structure in Neptune works:

Connecting Neptune to the CI/CD pipeline

Neptune is not only in the experimentation phase but also sits at the core of Continuum Industries’ version of the production MLOps pipeline, executed through GitHub actions.

When someone submits a Pull Request, it triggers a CI/CD pipeline via GitHub Actions. Each step evaluates the newest version of the algorithm on a baseline problem in a separate process.

The team was afraid that organizing all of those results for each CI/CD pipeline execution would be a nightmare, but thanks to Neptune’s custom run ID functionality, they can log all of the evaluations to the same run and keep it nice and clean.

We benefited a lot from using Neptune for experimentation and then got even more value out of it when we connected it to production!

Andreas Malekos Head of Artificial Intelligence at Continuum Industries

Support

Finally, one more aspect convinced Continuum Industries to Neptune. Adopting any tool may involve some bumps. But what is unique about Neptune is that you can really count on the team to help you through it, share your feedback and improvement ideas, and see them implemented. You get to be a part of the journey.

As Andreas shares – “We had some issues with using the new Neptune API at first, and we’ve received an incredible amount of support from the team since then. Talk to the Neptune team if you run into any issues, as they are incredibly helpful.”

The results

With Neptune, the Optioneer team:

Understands the performance of their algorithm at any given time with all the engine-related metadata recorded to Neptune through their weekly Quality Assurance CI/CD pipelines
Has much more confidence in the results generated by the Optioneer engine and knows exactly how the new versions of the engine were built.

Thanks to the whole team behind the Optioneer engine (Andreas Malekos, Miles Gould, Daniel Toth, Ivan Chan) for their help in creating this case study!