How to Set Up Continuous Integration for Machine Learning with Github Actions and Neptune: Step by Step Guide

In software development, Continuous Integration (CI) is a practice of merging code changes from the entire team to the shared codebase often. Before any new code can be merged it is tested and checked for quality automatically. 

CI makes the codebase up-to-date, clean, and tested by design and helps to find any problems with it quickly. 

But what does Continuous Integration mean for machine learning?

The way I see it:

Continuous Integration in machine learning extends the concept to running model training or evaluation jobs for each trigger event (like merge request or commit).

This should be done in a way that is versioned and reproducible to ensure that when things are added to the shared codebase they are properly tested and available for audit when needed.  

Some examples of CI workflows in machine learning could be:

Good news is today there are tools for that and in this article, I will show you how to set up Continuous Integration workflow with two of those: 

You will learn

How to set up a CI pipeline that automates the following scenario.

On every Pull Request from branch develop to master:

See this Pull Request on Github

CI for machine learning: Step-by-step guide

Before you start

Make sure you meet the following prerequisites before starting the how-to steps:

Step 1: Add Neptune logging to your training scripts

In this example project, we will be training a lightGBM multiclass classification model. 

Since we want to properly keep track of models we will also save the learning curves, evaluation metrics on testset, and performance charts like the ROC curve. 

1. Add Neptune tracking to your training script

Let me show you first and explain later.

import os

import lightgbm as lgb
import matplotlib.pyplot as plt
import neptune
from neptunecontrib.monitoring.lightgbm import neptune_monitor
from scikitplot.metrics import plot_roc, plot_confusion_matrix, plot_precision_recall
from sklearn.datasets import load_wine
from sklearn.metrics import f1_score, accuracy_score
from sklearn.model_selection import train_test_split

PARAMS = {'boosting_type': 'gbdt',
         'objective': 'multiclass',
         'num_class': 3,
         'num_leaves': 8,
         'learning_rate': 0.01,
         'feature_fraction': 0.9,
         'seed': 1234
         }
NUM_BOOSTING_ROUNDS = 10

data = load_wine()
X_train, X_test, y_train, y_test = train_test_split(data.data,
                                                   data.target,
                                                   test_size=0.25,
                                                   random_state=1234)
lgb_train = lgb.Dataset(X_train, y_train)
lgb_eval = lgb.Dataset(X_test, y_test, reference=lgb_train)

# Connect your script to Neptune
neptune.init(api_token=os.getenv('NEPTUNE_API_TOKEN'),
            project_qualified_name=os.getenv('NEPTUNE_PROJECT_NAME'))

# Create an experiment and log hyperparameters
neptune.create_experiment('lightGBM-on-wine',
                         params={**PARAMS,
                                 'num_boosting_round': NUM_BOOSTING_ROUNDS})

gbm = lgb.train(PARAMS,
               lgb_train,
               num_boost_round=NUM_BOOSTING_ROUNDS,
               valid_sets=[lgb_train, lgb_eval],
               valid_names=['train', 'valid'],
               callbacks=[neptune_monitor()],  # monitor learning curves
               )
y_test_pred = gbm.predict(X_test)

f1 = f1_score(y_test, y_test_pred.argmax(axis=1), average='macro')
accuracy = accuracy_score(y_test, y_test_pred.argmax(axis=1))

# Log metrics to Neptune
neptune.log_metric('accuracy', accuracy)
neptune.log_metric('f1_score', f1)

fig_roc, ax = plt.subplots(figsize=(12, 10))
plot_roc(y_test, y_test_pred, ax=ax)

fig_cm, ax = plt.subplots(figsize=(12, 10))
plot_confusion_matrix(y_test, y_test_pred.argmax(axis=1), ax=ax)

fig_pr, ax = plt.subplots(figsize=(12, 10))
plot_precision_recall(y_test, y_test_pred, ax=ax)

# Log performance charts to Neptune
neptune.log_image('performance charts', fig_roc)
neptune.log_image('performance charts', fig_cm)
neptune.log_image('performance charts', fig_pr)

It is a typical model training script with a few additions:

Now when you run your script:

python train.py

You should get something like this:

See this experiment in Neptune

2. Add a snippet that is run only in the CI environment. 

Add the following snippet at the bottom of your training script.

if os.getenv('CI') == "true":
   neptune.append_tag('ci-pipeline', os.getenv('NEPTUNE_EXPERIMENT_TAG_ID'))

What this does is:

Step 2: Create an environment file

Having an environment setup file which makes it easy to create your training or evaluation environment from scratch is generally a good practice.

But when you are training models in the CI workflow (like Github Actions) this is a must. The environment where the workflow is executed (and models are trained) will be created from scratch every time your workflow is triggered. 

There are a few choices when it comes to environment setup files. You can use:

Let’s go with the simplest solution and create a requirements.txt file with all the packages we need:

requirements.txt

lightgbm==2.3.1
neptune-client==0.4.125
neptune-contrib==0.24.8
numpy==1.19.0
scikit-learn==0.23.1
scikit-plot==0.3.7

Now, whenever you need to run your training install all the packages with:

pip install -r requirements.txt

Step 3: Set up Github Secrets

GitHub Secrets allow you to pass sensitive information like keys or passwords to the Github CI workflow runners so that your automated tests can be executed. 

In our case, two sensitive things are needed:

Without those, Github wouldn’t know where to send the experiments and Neptune wouldn’t know who is sending them (and whether this should be allowed). 

To set up GitHub Secrets:

Step 4: Create .github/workflows directory and a .yml action file

Github will run all workflows that you define in the .github/workflows directory with .yml configuration files. Which means you need to:

1. Create .github/workflows directory:

Go to your project repository and create both .github and .github/workflows directories.

mkdir -p .github/workflows

2. Create neptune_action.yml

Workflow configs that define actions are .yml files of a certain structure that you put in the .github/workflows directory. You can have multiple .yml files to fire multiple workflows. 

I will create just one neptune_action.yml

touch .github/workflows/neptune_action.yml

As a result, you should see:

your-repository
├── .github
│   └── workflows
│       └── neptune_action.yml
├── .gitignore
├── README.md
├── requirements.txt
└── train.py

Step 5:  Define your workflow .yml config

Workflow configs are .yml files where you specify what you want to happen and when. 

In a nutshell, you define:

In our machine learning CI workflow we need to run the following sequence of steps:

Here is the neptune_action.yml that does all that. I know it seems complex but in reality, it’s just a bunch of steps that run terminal commands with some boilerplate around it.  

neptune_action.yml

name: Neptune actions

on:
 pull_request:
   branches: [master]

jobs:
 compare-experiments:
   runs-on: ubuntu-latest
   strategy:
     matrix:
       python-version: [3.7]
   env:
     NEPTUNE_API_TOKEN: ${{ secrets.NEPTUNE_API_TOKEN }}
     NEPTUNE_PROJECT_NAME: ${{ secrets.NEPTUNE_PROJECT_NAME }}

   steps:
     - name: Set up Python
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}

     - name: Checkout pull request branch
       uses: actions/checkout@v2
       with:
         ref: develop

     - name: Setup pull request branch environment and run experiment
       id: experiment_pr
       run: |
         pip install -r requirements.txt
         export NEPTUNE_EXPERIMENT_TAG_ID=$(uuidgen)
         python train.py
         echo ::set-output name=experiment_tag_id::$NEPTUNE_EXPERIMENT_TAG_ID

     - name: Checkout main branch
       uses: actions/checkout@v2
       with:
         ref: master

     - name: Setup main branch environment and run experiment
       id: experiment_main
       run: |
         pip install -r requirements.txt
         export NEPTUNE_EXPERIMENT_TAG_ID=$(uuidgen)
         python train.py
         echo ::set-output name=experiment_tag_id::$NEPTUNE_EXPERIMENT_TAG_ID

     - name: Get Neptune experiments
       env:
         MAIN_BRANCH_EXPERIMENT_TAG_ID: ${{ steps.experiment_main.outputs.experiment_tag_id }}
         PR_BRANCH_EXPERIMENT_TAG_ID: ${{ steps.experiment_pr.outputs.experiment_tag_id }}
       id: compare
       run: |
         pip install -r requirements.txt
         python -m neptunecontrib.create_experiment_comparison_comment \
           --api_token $NEPTUNE_API_TOKEN \
           --project_name $NEPTUNE_PROJECT_NAME \
           --tag_names $MAIN_BRANCH_EXPERIMENT_TAG_ID $PR_BRANCH_EXPERIMENT_TAG_ID \
           --filepath comment_body.md
         result=$(cat comment_body.md)
         echo ::set-output name=result::$result

     - name: Create a comment
       uses: peter-evans/commit-comment@v1
       with:
         body: |
           ${{ steps.compare.outputs.result }}

You can just copy this file and paste it into your .github/workflows directory and it will work out of the box. 

That said, there are some things that you may need to adjust to your setup:

Step 6: Push it to Github 

Now you need to push this workflow to GitHub. 

git add .github/workflows train.py requirements.txt;
git commit -m "added continuous integration"

Since our workflow will be triggered on every Pull Request to master, nothing will happen just yet. 

Step 7: Create a Pull Request 

Now everything is ready and you just need to create a PR from branch develop to master.

git checkout -b develop

2. Change some parameters in train.py

train.py

PARAMS = {'boosting_type': 'gbdt',
          'objective': 'multiclass',
          'num_class': 3,
          'num_leaves': 15, # previously 8
          'learning_rate': 0.01,
          'feature_fraction': 0.85, #previous 0.9
          'seed': 1234
          }

3. Add, commit and push your changes to the previously created branch develop

git add train.py;
git commit -m"tweaked parameters"
git push origin develop

4. Go to Github and create a Pull Request from branch develop to master.

The workflow is triggered and it goes through all the steps one by one. 

Explore the result

If everything worked correctly you should see a Pull Request comment that shows:

See this Pull Request on Github

Final thoughts

Ok, so in this how-to guide, you learned how to set up a Continuous Integration workflow that creates a comparison table for every Pull Request to master. 

Hopefully, with this information, you will be able to create the CI workflow that works for your machine learning project!