Building MLOps Pipeline for Time Series Prediction [Tutorial]

In this tutorial, we’ll present a simple example of a time-series-based ML project and build an MLOps pipeline for that. Every step will be executed following the best practices from MLOps, and the whole project will be explained step by step.

This time-series project is based on the Binance trading app, but similar logic is also applicable to other ML projects as well.

Note: This article is not intended for financial advice, and it’s written for educational purposes only. Also, the main purpose of this article is to present MLOps architecture with end-to-end ML project flow and not to present a profitable trading strategy. 

We’ll go through some basics first, but if you want, you can skip that and just jump to the tutorial right away.

MLOps 101

MLOps stands for Machine Learning Operations, and it’s the process of managing the ML project pipeline. The role of MLOps is to connect different parts of an ML project as one structure that works in harmony with all its components and is intended to keep that functionality in the future. In order to achieve maximum robustness, MLOps applies some practices from DevOps, especially continuous integration and continuous delivery (CI/CD):

1. Continuous integration makes sure that the whole ML pipeline runs smoothly whenever a piece of code or data is updated. This is done through code and data versioning which allows the code to be shared and run across different teams. Re-running might include training and testing but it’s also a good practice to have and run code tests to ensure that the input and output data follow a certain format and everything works as expected.

2. Continuous delivery allows automatic deployment of the new ML model. With the CD process, it’s possible to deploy a new model in the existing environment using triggers, such as retrained model on a new set of data, new hyperparameters or a new model architecture.

In general, for someone with programming experience, the easiest way to understand MLOps is to compare it with DevOps. But they are not completely the same in terms of tools and processes. Both DevOps and MLOps attempt to integrate development, testing, and operational principles; however, DevOps focuses on conventional software development while MLOps only focuses on ML projects.

In ML projects, the code is not the only component that goes under version control management. Input data, hyperparameters, metadata, logs, and models can change over time and thus need to be controlled and monitored. One more difference is that conventional software doesn’t degrade, while ML models do. There is a possibility that once a model is deployed in production, it may start to produce errors and inaccurate results. This is because of the fact that the input data changes over time while the model remains the same, trained with old data.

Because of that, besides CI/CD, MLOps also includes:

• Continuous Training (CT) – a process that automatically retrains ML models in production.
• Continuous Monitoring (CM) – an idea of continuously monitoring data and models in production in order to notice potential data drift or model staleness. 

MLOps phases

There is no unique way or architecture for solving MLOps problems, especially nowadays when more than hundreds of tools and packages related to MLOps exist. This is because of the diverse nature of ML projects and the fact that the concept of MLOps is fairly young, we can propose some steps to help build the MLOps pipeline, but most likely, they might not be exhaustive.

Design and scope

Before we start developing an ML project and writing the code, we need to make sure that the business goal is clear and that we are competent enough to solve the problem. Here comes the phase of design and scope. 

In this phase, we need to make sure that we understand the problem statement and business goals of our project. Also, we need to check the availability of all resources such as suitable architecture, computation resources, competent team, and similar.

Development

After design and scope, comes project development. It includes:

• Research – gathering new information about potential input features, data preprocessing steps, ML model, new tools, and similar.
• Data engineering – data ingesting, developing ETL pipelines, data warehousing, database engineering, etc.
• Exploratory data analysis (EDA) – understanding our data using data analysis and visualization techniques.
• Experiment development – may include data preprocessing, features engineering, ML model development, hyperparameters tuning, and similar.
• Experiment tracking – lastly, we want to compare all experiments and draw conclusions.

Operations

After a model is developed and ready for production comes the operations phase. The primary goal of operations is to put this developed model into production using some MLOps practices such as testing, versioning, CI/CD, monitoring, and others.

Time series 101

A time series is a sequence of data points that are ordered in time. It is a series of observations of the same variable at different points in time. A time-series data is often presented as a line on a graph with time on the x-axis and the value of each data point on the y-axis. Also, every time series is composed of four components:

Learn more

Time Series Projects: Tools, Packages, and Libraries That Can Help

Examples of time series projects

Time series are represented in many industries, and there are a lot of real-world examples of time series projects. Here, we’ll mention only some of them.

Website traffic prediction

The main idea is to predict the amount of traffic for a specific website and, based on that, optimize resource allocation. It might help load balancer to distribute network or application traffic across a number of servers. Besides that, it’s possible to develop ML solutions for anomaly detection in web traffic in order to find potential hacker attacks. 

Many organizations such as Facebook, Amazon, eBay, and others use similar applications in order to predict and monitor internet traffic. 

Time series projects in healthcare

Time series data in healthcare, such as electronic health records (EHR) and registries, represent valuable sources of information about patients and can be used in many ways for patient benefits. Using this data, it’s possible to develop ML models that provide a much deeper understanding of individual trajectories of health and disease such as cancer, Alzheimer, cardiovascular diseases, COVID-19, and many others.

With time series and ML models, it is possible to find out the risks of mortality, relapse, and complications in the future and to recommend prescriptions and precautions for the same.

Stock market prediction

Stock market forecasting is a very challenging task in which the main objective is to create various strategies for forecasting future stock prices. The stock market has very volatile and chaotic behavior because of many factors, such as the global economy, financial reports, politics, and others.

In general, there are many ways to apply stock market forecasting methods. Some of them are trend predicting, long-term price forecasting, volatility predicting, daily, hourly, or high-frequency trading, and similar.

Most of them are based on two approaches; fundamental analysis and technical analysis. Fundamental analysis takes into consideration some factors such as financial reports, industry trends, inflation rate, GDP, and similar. Technical analysis uses technical indicators calculated from historical stock data and, based on that, predicts how stocks will perform in the future.

Bitcoin trading

Bitcoin is a digital asset whose price is determined by supply and demand. Similarly, as the stock market, it has very volatile and chaotic behavior, and bitcoin price predicting is a complex challenge. It is known that bitcoin correlates with some tech company stocks, and therefore many techniques from stock market predicting can be used.

In this article, we will use bitcoin price trading as an example of a time series project. This article is not financial advice, and it’s written for educational purposes with a focus on time series and MLOps architecture. Because of that, only a simple trading strategy will be developed using Python.

In order to write a Python program that is able to automatically place orders, we’ll use Binance exchange and Binance API with its own wrapper package python-binance. Also, Binance provides a demo account that allows traders to trade with “paper money” in a simulated environment. More about that is presented in the article below.

Introduction to Crypto Bitcoin Trading with Python and Binance

MLOps pipeline for time series prediction: design and scope

All phases that were previously mentioned and briefly described will be practically implemented below, step by step, using our example with bitcoin trading. 

Problem statement

First of all, we need to make sure that the problem is clear to us. In this particular case, there are no external factors such as clients or stakeholders with whom we need to communicate about it. To make it simpler, we’ll try to predict hourly bitcoin movement. It means that we want to predict whether bitcoin will go up or down for the next hour, which indicates that it’s a classification problem. Based on that prediction, we’ll take a long or short position (buy or sell a specific amount of bitcoin). 

For instance, if we predict that the price of bitcoin will increase in the next hour and if it does from 100$ to 105$, our profit will be 5$. Otherwise, if the price decreases from 100$ to 95$, we’ll lose 5$.

Roughly, for prediction will be used XGBoost model, and the whole ML project will follow best MLOps practices.

In this stage, we can approximately propose the blueprint for MLOps structure as follows:

Business goal

A business goal is clear here which is to make a profit. This is a very speculative strategy; besides that, we would also need to define what kind of losses we can handle. For example, at any moment, if the strategy goes under -20% in cumulative return, the trading will be halted. 

Available resources

As our project is pretty small and experimental in nature, most likely, we wouldn’t need to spend any money on tools. Also, real money won’t be invested since the project will be deployed to Binance simulated environment. 

All historical bitcoin price data for features engineering and training will be downloaded from Binance on AWS S3. Since the historical data is not big in size, we’ll also download it locally, and model development will be done locally as well. 

Lastly, we would need to have access to all tools that will be used in our projects, this will be done along the way.

MLOps pipeline for time series prediction: model development

Research

As we mentioned before, a good practice of ML project development is to start with research. In our case, we can simply Google some terms related to our project like “machine learning bitcoin trading python” or  “bitcoin technical indicators”. Also, a good source of information is youtube, especially if some more complicated concepts need to be explained.

Some quality sources might not be easily accessible with a Google search. For instance, notebooks from Kaggle are rarely in the top results. Also, some advanced courses in the specific domain might appear on Udemy, Udacity, or Coursera.

There are some popular websites for ML, like Machine Learning Mastery, Medium, and Towards Data Science. A good place for quality MLOps content search is for sure the neptune.ai blog. There’s also the MLOps community on Slack, a super active group of practitioners discussing and sharing knowledge. You can join it here (and if you do join, come to the #neptune-ai channel to talk about MLOps, this article, or just say hi!).

Lastly, for some state-of-the-art solutions, we can search Google Scholar or ResearchGate.

Data engineering

In our case, the only source of input data is Binance. Using the python-finance package it is possible to get data directly with existing methods. Also, there is an alternative way of getting historical prices and saving them directly as zipped .csv files organized per month.

For that, we need to use a binance-public-data repository. To download zipped .csv files, we use the command:

`python download-kline.py -s BTCUSDT -i 1h -startDate 2017-08-01 -endDate 2022-06-01`

where `download-kline.py` is a script from the `python` directory. More about this python script and its arguments can be found here.

Since we’re going to use AWS cloud architecture, the data will be ingested in S3 storage. New AWS account can get 5GB of S3 storage for 12 months for free. In order to upload the data, we’ll need to create a bucket, which is a container for objects stored on S3. How to create a bucket and upload a file in the bucket is explained in this video.

Besides uploading data using the AWS web console, it’s possible to upload data from the AWS EC2 instance. Basically, with EC2 it is possible to create an instance or virtual machine, where we can download data and copy from there directly to the S3 bucket using a simple command:

`aws s3 cp LOCAL_DIRECTORY/ s3://BUCKET_NAME/DIRECTORY/ --recursive`

With a new AWS account, it is possible to get some smaller instances for 750 hours monthly, and 12 months for free. In addition, we’ll deploy our project on an EC2 instance as well.

For more complicated projects, there are several other data engineering services on AWS and some of the most used are explained in this video.

ML model development

Now when the data is ready, we can start with exploratory data analysis (EDA). The main goal of EDA is to explore and visualize data to discover some insights and plan how to develop models. Usually, EDA is done using Jupyter notebook with some packages for data manipulation and analysis such as pandas, numpy, matplotlib, and similar.

Next, we can start with feature engineering. For the sake of this tutorial, we’re going to use 9 technical indicators as features (moving average for 5, 10, and 20 hours, RSI and MFI for 7, 14, and 21 hours). All technical indicators will be calculated using TA-Lib and they are mostly related to financial time series but in general, package tsfresh is a good choice for calculating time-series features.

For the ML model, we’ll try XGBoost and Optuna for optimizing hyperparameters. By default, Optuna uses the Tree-structured Parzen Estimator algorithm but there are a few more that can be selected. The optimization algorithm will try to maximize the cumulative return of our strategy. 

The data set will be divided into two parts:

• In-sample (from 2018-01-01 until 2021-12-31, used for hyperparameter search and backtesting)
• Out-of-sample (from 2022-01-01 until 2022-05-31, used to double-check the selection strategy, to make sure we haven’t overfitted the model)

Backtesting will be done using time series cross-validation with a fixed sliding window because we want to keep our training set of the same size in every iteration.

Experiment tracking

Experiment tracking will be done using neptune.ai’s integration for Optuna. It’s a very convenient integration that lets you track all metadata from model training with several plots with just a few lines of code. 

In our case, we’ll use Neptune-Optuna integration to log and monitor the Optuna hyperparameter tuning for the XGBoost model. Usually, time series models are not big in comparison to some convolutional neural networks and, as an input, have a few hundred or thousand numerical values, so models train pretty fast.

For financial time series, it’s especially important to have a convenient way of tracking model hyperparameters since we would need to run a lot of different experiments. This is because time series in finance tend to have very chaotic movements and require a lot of tuning.

By having a suitable tool for hyperparameter tracking, we would be able to recognize how optimization progresses by observing the optimization history plot. In addition to the run time and hardware consumption logs, we would be able to conclude whether we need to increase the optimization trials (iterations) or not.

Neptune-Optuna integration provides visualizations for hyperparameter importance, parallel coordinate plots showing the relationship between different hyperparameter values and the value of the objective function, and many more useful features.

To do that, firstly, we’ll define `objective` method in our main class:

def objective(self, trial):

    params = {
        'n_estimators': trial.suggest_int('n_estimators', 350, 1000),
        'max_depth': trial.suggest_int('max_depth', 3, 10),
        'learning_rate': trial.suggest_uniform('learning_rate', 0.01, 0.10),
        'subsample': trial.suggest_uniform('subsample', 0.50, 0.90),
        'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.50, 0.90),
        'gamma': trial.suggest_int('gamma', 0, 20),
    }
    look_back = trial.suggest_int('look_back', 30, 180)
    self.apply_strategy(params)

    return self.get_score()

where `trial` is a parameter used in Optuna, `apply_strategy` prepares data and trains the model, and `get_score` returns in our case cumulative return as a metric that we want to optimize. After that, we need to connect Optuna with neptune.ai

import optuna
import neptune
import neptune.integrations.optuna as optuna_utils

run = neptune.init_run(
	project="enes.zvornicanin/optuna-test",
	api_token="YOUR_API_TOKEN",
)

neptune_callback = optuna_utils.NeptuneCallback(run)

# our main class for training and optimization
hb = HourlyBacktester(dt.data, trader_config)
n_trials = 20

study = optuna.create_study(direction="maximize")
study.optimize(hb.objective, n_trials=n_trials, callbacks=[neptune_callback])

For more information, please follow this Neptune-Optuna integration guide. 

MLOps pipeline for time series prediction: automated testing

During model development, we will use GitHub as a source code management tool. Also, based on a project and its requirements, it’s a good practice to implement some automated tests.

Read more

Automated Testing in Machine Learning Projects [Best Practices for MLOps]

As an example of automated tests, we’ll create a simple smoke test that will run the whole project using GitHub actions. It’s a simple automated test that checks if our main python script can run successfully. 

In order to test the project, in our `main.py` script, besides the main function, we’ll create a function with the same name but with the prefix “test_”:

def main():
	pt = PaperTrader(config)
	pt.execute_trade()

def test_main():
	pt = PaperTrader(config, debug=True)
	pt.test_execute_trade()

if __name__ == '__main__':
	main()

In that way, if we run `main.py` with Pytest as `pytest main.py`, Pytest will automatically run all functions that have the “test_” prefix. In the `test_main` function we provide a `debug` parameter that controls if we are going to use real data or dummy data generated only for testing. Also, when `debug=True`, the program doesn’t log into the Binance account but creates a mock object that simulates one method of the Binance client object:

def log_in(self):

    	if self.debug:
        	self.client = Mock()
        	self.client.create_order = lambda symbol, side, type, quantity:{
            	"executedQty":1,
            	"cummulativeQuoteQty":np.random.rand()+1
        	}
        	return

    	self.client = Client(api_key = os.environ.get('BINANCE_TESTNET_API'),
                         	api_secret = os.environ.get('BINANCE_TESTNET_SECRET'),
                         	tld = 'com',
                         	testnet = True)

where client.create_order is used later in the code.

Next, we’ll set up GitHub in a way such that it automatically runs our test on the Ubuntu virtual machine after every push to the repository. The first step is to create a .yml file in the directory `.github/workflows` or directly in the GitHub repository by going to:

Actions -> New workflow -> Set up workflow yourself:

After that, a new workflow template will appear with some basic commands:

The updated .yml file created for our use case is available in the repository and it looks like this:

This will make the GitHub action with the name “Tests” run the commands below on a push using the “master” branch. The workflow will be run on an Ubuntu machine and the first two steps, which are common in most projects, are related to check-out repository and setup of python. After that, it installs requirements and package talib (used for technical indicators) that require special installation. Lastly, it runs our test.

If action completes all steps successfully, there will be a green sign under the Action tab:

MLOps pipeline for time series prediction: deployment and continuous delivery

Deployment will be done following CI/CD practices using GitHub actions, Docker and AWS ECS. Elastic Container Service (ECS) is a container orchestration service that makes it easy to deploy, manage, and scale containerized applications. In our example, we’ll use it as CI/CD service that will host an EC2 instance where docker will run.

Access to AWS services will be provided using AWS Identity and Access Management (IAM) service. Also, before deployment, we’ll ingest input data on AWS S3. 

Check also

Continuous Integration and Continuous Deployment (CI/CD) Tools for Machine Learning

Briefly, deployment steps are as follows:

• When code is pushed on a defined branch, GitHub actions activate and start the CD process. Firstly, actions configure AWS credentials in order to have access to the services.
• Actions build Docker image and push it on AWS ECR. Elastic Container Registry is a repository for container images, where images can be easily pushed, accessed, and distributed with other AWS services.
• Actions deploy the defined image on ECS service.
• ECS service will operate one EC2 instance where a Docker container will be deployed. Inside the Docker container, the cron job will run our main python script every one hour, using the Miniconda environment.
• Output data will be stored on AWS S3. Also, some output results and metadata will be stored on neptune.ai as a monitoring service.

IAM and S3

Steps for creating an IAM user

1. AWS -> IAM -> Users -> Add user

2. Write user name and under AWS access type select “Access key – Programmatic access” and click next to the permission tab.

3. Select the following policies:

• AmazonS3FullAccess
• AmazonEC2FullAccess
• AmazonEC2ContainerRegistryFullAccess
• AmazonECS_FullAccess
• EC2InstanceProfileForImageBuilderECRContainerBuilds

4. Click the next button twice and click create user. Now, user credentials will appear and make sure to download and save them, because you won’t be able to see them again.

Steps for creating S3 bucket

1. Go to AWS -> Amazon S3 -> Buckets -> Create bucket

2. Write bucket name and optional, enable bucket versioning. Click on create bucket and it should be created.

3. Now click on your bucket name and try to upload a file from your computer.

ECR and ECS

Steps for creating ECR repository

1. Go to AWS ECR -> Get started (create repository)

2. In Visibility settings select Private.

3. Define repository name.

4. Click Create repository.

For ECS service, we would need to create:

• Task definition (required to run Docker containers in Amazon ECS)
• Cluster (defines infrastructure where Docker will run)

Steps for creating ECS task definition

1. Go to AWS ECS -> Task definition -> Create new task definition

2. Select EC2 for launch type compatibility.

3. Specify a name for task definition.

4. Under container definition, click on Add container button.

5. For container name, add ECR repo name previously defined.

6. For the image add the ECR repo URI link (it can be found in ECR services, next to the repository name). Memory limit and port mappings fill as in the image below. Click Add and after that click Create button. 

Steps for creating ECS cluster

1. Go to AWS ECS -> Clusters -> Create cluster

2. Select EC2 Linux + Networking and click next step.

3. Define a cluster name, choose EC2 instance type (t2.micro is eligible for free tier) and create a key pair (or choose existing ones). Click on the Create button.

4. When the cluster is created, click on the cluster name and under the Services tab, click Create button.

5. Select EC2 for launch type, define service name and number of tasks (1).

6. Click on the Next step a few times and finish by clicking on the Create Service button.

Dockerfile

Dockerfile can be found In this repository and we’ll explain it briefly here.

• For Docker containers, we’ll use the Ubuntu 22.10 image. We set up the environment variable PATH for miniconda, environment variables that will be used by the python script, and install some packages for Ubuntu. After that, we download, install and create a miniconda python environment with the name “env”.

• Command `COPY . binance_trading/` copies all files from our project into Docker “binance_trading” directory. The next set of commands are related to installing Ta-Lib Python package which we use for technical indicators. After that we install all other packages from the requirements.txt file.

• The next few commands are related to configuring cron job inside Docker. For that, we need to have a “cron-job” file in the repository which we’ll copy into the Docker container. The last command that will be activated on container startup redirects environment variables into /etc/environment file in order to be visible to the cron job. Also, it activates the cron job.

• Cron job file is defined in our repository, and it’s supposed to run our main script every hour. 

Check also

Best Practices When Working With Docker for Machine Learning

GitHub Actions

Next, we’ll explain how to deploy our project to ECS as part of continuous deployment (CD) workflow. For that, we’ll use GitHub actions. Yaml file that defines this process is available in the repository and here we’ll explain it step by step.

We’ll follow the framework available in GitHub Docs. This action will be activated on push to the main branch. 

1. First, we define some environment variables that will be used later. Some of them such as repository, service, cluster, and container name was defined in previous steps. The AWS region is related to the AWS profile.

2. ECS task definition can be stored as .json file in the repository and is available under ECS -> Task Definition -> Your definition name -> JSON tab

3. After that, in the yaml file, we define steps which will be run on the latest Ubuntu OS and the first step is to check out the repository.

4. Next, we need to configure AWS credentials. We created them under the IAM user. Also, they will be stored as actions secrets and we can define them in our GitHub repository under: Settings tab -> Secrets -> Actions -> New repository secret

All secrets can be accessed using ${{ secrets.VARIABLE_NAME }} notation.

5. After login into Amazon ECR, we build and push the Docker image to the ECR repository. Here we define local environment variables from our secrets. Some of them are processed into a Docker container during the build command as build arguments. In that way, our python script is able to reach them using the method.

import os
aws_access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')

6. Lastly, we fill a new image ID in the task definition and deploy it on ECS.

Now, if we push changes in our repository, GitHub actions will start this process. An example of a successful run is attached below:

You may also like

How to Build MLOps Pipelines with GitHub Actions [Step by Step Guide]

MLOps pipeline for time series prediction: monitoring

Instead of building a Flask or Fast API web server where our results, metadata, and charts would be presented, we can easily get the same functionality using neptune.ai. The idea is that after every script run i.e. after every trade, we upload results with charts on a neptune.ai project. With a few lines of code and two minutes of work, we would be able to store and track our Matplotlib charts and .csv result file in the neptune.ai project.

To get started with monitoring performance metrics, run the following code:

import neptune
run = neptune.init_run(
        	project=os.environ.get('NEPTUNE_PROJECT'),
        	api_token=os.environ.get('NEPTUNE_API_TOKEN'),
    	)

define your matplotlib plot:

fig = plt.figure(figsize =(4, 4))

and upload it on the project as a static or interactive image:

run["static-img"].upload(neptune.types.File.as_image(fig))
run["interactive-img"].upload(neptune.types.File.as_html(fig))

In order to upload a pandas data frame with results, add one more line of code:

run['data/results'].upload(neptune.types.File.as_html(df_results))

This would automate the process of logging metadata, results, in this case. Simultaneously you’d also be able to draw a comparison between different runs of your experiment over defined metrics, as indicated in the next screenshot.

Besides simply keeping track of our results in order to present them to our colleagues and clients, there are more benefits to building a monitoring system for an ML project:

• For instance, model staleness is a common issue in financial time series. Although in this project, the logic for model retraining is implemented in the code and will be triggered if the model starts to perform badly, it might be useful to monitor model staleness directly.

• Similarly, data drift can be a subject for monitoring, where it’s possible to create dashboards that will show some statistics about real-time production data at every model prediction.

Besides Neptune, there are a few more tools that are more specialized for ML monitoring. For example, Arize AI enables ML practitioners to better detect and diagnose model issues by helping understand why a machine learning model behaves the way it does when deployed in the real world.

Similarly, WhyLabs is a model monitoring tool that helps ML teams with monitoring data pipelines and ML applications. More about ML monitoring tools can be found in this article: Best Tools to Do ML Model Monitoring.

Conclusion

In this tutorial, we’ve presented a simple end-to-end ML time series project following MLOps practices. 

Our project is located on the github repository with defined smoke tests which are triggered on code push. That is a simple example of CI practices. In addition to that, CD functionality is implemented with GitHub actions, Docker, and AWS service. CM example is simply provided using neptune.ai and CT logic is integrated in the python code itself, where the model will be retrained and optimized if the results of the last N runs are below the T threshold.

For any additional questions regarding this project, feel free to reach out to me on Linkedin.

References

• https://docs.github.com/en/actions/deployment/deploying-to-your-cloud-provider/deploying-to-amazon-elastic-container-service 
• https://kubesimplify.com/cicd-pipeline-github-actions-with-aws-ecs