ARIMA vs Prophet vs LSTM for Time Series Prediction

Assuming we subscribe to a linear understanding of time and causality, as Dr. Sheldon Cooper says, then representing historical events as a series of values and features observed over time provides the foundations for learning from the past. However, time series are somewhat different from other datasets, including sequential data like text or DNA sequences.

The time component provides additional information that can be useful when predicting the future. Thus, there are many different techniques designed specifically for dealing with time series. Such techniques range from simple visualization tools that show trends evolving or repeating over time to advanced machine learning models that utilize the specific structure of time series.

In this post, we will discuss three popular approaches to learning from time-series data:

We will then show how to compare the results across the three models using neptune.ai and its powerful features.

Let’s start with a brief overview of the three methods.

Overview of the three methods: ARIMA, Prophet, and LSTM

ARIMA

ARIMA is a class of time series prediction models, and the name is an abbreviation for AutoRegressive Integrated Moving Average. The backbone of ARIMA is a mathematical model that represents the time series values using its past values. This model is based on two main features: 

1. Past Values: Clearly, past behaviour is a good predictor of the future. The only question is how many past values we should use. The model uses the last p time series values as features. Here p is a hyperparameter that needs to be determined when we design the model.
2. Past Errors: The model can use the information on how well it has performed in the past. Thus, we add as features the most recent q errors the model made. Again, q is a hyperparameter.   

An important aspect here is that the time series needs to be standardized such that the model becomes independent from seasonal or temporary trends. The formal term for this is that we want the model to be trained on a stationary time series. In the most intuitive sense, stationarity means that the statistical properties of a process generating a time series do not change over time. It does not mean that the series does not change over time, just that the way it changes does not itself change over time.

There are several approaches to making a time series stationary, the most popular being differencing. By replacing the n values in the series with the n-1 differences, we force the model to learn more advanced patterns. When the model predicts a new value, we simply add the last observed value to it in order to obtain a final prediction. Stationarity can be somewhat confusing if you encounter the concept for the first time, you can refer to this tutorial for more details.

Parameters

Formally, ARIMA is defined by three parameters p, d, and q that describe the three main components of the model. 

• Integrated (the I in ARIMA): The number of differences needed to achieve stationarity is given by the parameter d. Let the original features be Y_t where t is the index in the sequence. We create a stationary time series using the following transformations for different values of d.

For d=0

In this case the series is already stationary and we have nothing to do.

For d=1

This is the most typical transformation.

For d=2

Observe that differencing can be seen as a discrete version of differentiation. For d=1 the new features represent how the values change. While for d=2 the new features represent the rate of the change, just like the second derivative in calculus.  The above can be generalized to d>2 as well but this is rarely used in practice.

• AutoRegressive (AR): The parameter p tells us how many past values to consider for the expression of the current value. Essentially, we learn a model that predicts the value at time t as:

•  Moving Average (MA): How many of the forecast errors in the past should be considered. A new value is computed as:

The past prediction errors:

The combination of the three components gives the ARIMA(p, d, q) model. More precisely, we first integrate the time series, and then we add the AR and MA models and learn the corresponding coefficients.

Prophet

Prophet FB was developed by Facebook as an algorithm for the in-house prediction of time series values for different business applications. Therefore, it is specifically designed for the prediction of business time series.

It is an additive model consisting of four components:

Let us discuss the meaning of each component:

1. g(t): It represents the trend and the objective is to capture the general trend of the series. For example, the number of advertisements views on Facebook is likely to increase over time as more people join the network. But what would be the exact function of increase?
2. s(t): It is the Seasonality component. The number of advertisement views might also depend on the season. For example, in the Northern hemisphere during the summer months, people are likely to spend more time outdoors and less time in from of their computers. Such seasonal fluctuations can be very different for different business time series. The second component is thus a function that models seasonal trends. 
3. h(t): The Holidays component. We use the information for holidays which have a clear impact on most business time series. Note that holidays vary between years, countries, etc. and therefore the information needs to be explicitly provided to the model.
4. The error term ε_t stands for random fluctuations that cannot be explained by the model. As usual, it is assumed that ε_t follows a normal distribution N (0, σ²) with zero mean and unknown variance σ that has to be derived from the data.

LSTM recurrent neural networks

LSTM stands for Long short-term memory. LSTM cells are used in recurrent neural networks that learn to predict the future from sequences of variable lengths. Note that recurrent neural networks work with any kind of sequential data and, unlike ARIMA and Prophet, are not restricted to time series. 

The main idea behind LSTM cells is to learn the important parts of the sequence seen so far and forget the less important ones. This is achieved by the so-called gates, i.e., functions that have different learning objectives such as: 

1. a compact representation of the time series seen so far
2. how to combine new input with the past representation of the series
3. what to forget about the series
4. what to output as a prediction for the next time step. 

See Figure 1 and the Wikipedia article for more details.

Designing an optimal LSTM based model can be a difficult task that requires careful hyperparameter tuning. Here is the list of the most important parameters an LSTM based model needs to consider:

• How many LSTM cells are to use in order to represent the sequence? Note that each LSTM cell will focus on specific aspects of the time series processed so far. A few LSTM cells are unlikely to capture the structure of the sequence while too many LSTM cells might lead to overfitting.
• It is typical that first, we convert the input sequence into another sequence, i.e. the values h_t. This yields a new representation as the h_t states capture the structure of the series processed so far. But at some point, we won’t need all htvalues but rather only the last h_t. This will allow us to feed the different h_t’s into a fully connected layer as each h_t corresponds to the final output of an individual LSTM cell. Designing the exact architecture might require careful finetuning and many trials.

Finally, we would like to reiterate that recurrent neural networks are a general class of methods for learning from sequential data and they can work with arbitrary sequences such as natural text or audio.  

Experimental evaluation: ARIMA vs Prophet vs LSTM

Dataset

We are going to use stock exchange data for Bajaj Finserv Ltd, an Indian financial services company in order to compare the three models. The dataset spans the period from 2008 until the end of 2021. It contains the daily stock price (mean, low, and high values) as well as the total volume and the turnover of traded stocks. A subsample of the dataset is shown in Figure 2. 

We are interested in predicting the Volume Weighted Average Price (VWAP) variable at the end of each day.  A graph of the time series VWAP values is presented in Figure 3.

For the evaluation, we divided the time series into a train and test time series where the training series consists of the data until the end of 2018 (see Figure 4). 

Total number of observations: 3201 

Training observations: 2624

Test observations: 577

Implementation 

In order to work properly, machine learning models require good data and for this, we will do a little Feature engineering. The objective behind feature engineering is to design more powerful models that exploit different patterns in the data. As the three models learn patterns observed in the past, we create additional features that thoroughly describe the recent trends of the stock movements. 

In particular, we track the moving average for the different trade features over a period of 3, 7, and 30 days. In addition, we consider features such as the month, the week number, and the weekday. Thus, the input to our models is multidimensional.  A small example of the used feature engineering looks as follows:

lag_features = ["High", "Low", "Volume", "Turnover", "Trades"]
df_rolled_7d = df[lag_features].rolling(window=7, min_periods=0)
df_mean_7d = df_rolled_7d.mean().shift(1).reset_index().astype(np.float32)

The above code excerpt shows how to add the running mean over the last week of several features describing the sales of the stock. Overall, we create a set of exogenous features:

Now, let’s get started with our main models:

ARIMA

We implemented the ARIMA version from the publicly available package pmdarima. The function auto_arima accepts as an additional parameter a list of exogenous features where we provide the features created in the feature engineering step. The main advantage of auto_arima is that it first performs several tests in order to decide if the time series is stationary or not. Also, it employs a smart grid search strategy that determines the optimal parameters for p, d, and q discussed in the previous section. 

from pmdarima import auto_arima
model = auto_arima(
	df_train["VWAP"],
	exogenous=df_train[exogenous_features],
	trace=True,
	error_action="ignore",
	suppress_warnings=True)

The grid search over different values of the parameters p, d, and q is shown below. In the end, the model with the smallest AIC value is returned. (The AIC value is a measure of model complexity that simultaneously optimizes the accuracy and the complexity of a prediction model.) 

Predictions on the test set are then obtained by

forecast = model.predict(n_periods=len(df_valid),  exogenous=df_valid[exogenous_features])

Prophet  

We use the publicly available Python implementation of Prophet. The input data must contain two specific fields: 

1. Date:  should be a valid calendar date from which the holidays can be computed
2. Y: the target variable we want to predict.

We instantiate the model as:

from prophet import Prophet
model = Prophet()

The features created during feature engineering have to be explicitly added to the model as follows:

for feature in exogenous_features:
	model.add_regressor(feature)

Finally, we fit the model:

model.fit(df_train[["Date", "VWAP"] + exogenous_features].rename(columns={"Date": "ds", "VWAP": "y"}))

And the forecast for the test set is obtained as:

forecast = model.predict(df_test[["Date", "VWAP"] + exogenous_features].rename(columns={"Date": "ds"}))

LSTM

We used the Keras implementation of LSTMs:

import tensorflow as tf
from keras.layers import Dropout
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from tensorflow.keras.metrics import RootMeanSquaredError, MeanAbsoluteError
from tensorflow.keras.models import Sequential

The model is defined by the following function.

def get_model(params, input_shape):
	model = Sequential()
	model.add(LSTM(units=params["lstm_units"], return_sequences=True, input_shape=(input_shape, 1)))
	model.add(Dropout(rate=params["dropout"]))

	model.add(LSTM(units=params["lstm_units"], return_sequences=True))
	model.add(Dropout(rate=params["dropout"]))

	model.add(LSTM(units=params["lstm_units"], return_sequences=True))
	model.add(Dropout(rate=params["dropout"]))

	model.add(LSTM(units=params["lstm_units"], return_sequences=False))
	model.add(Dropout(rate=params["dropout"]))

	model.add(Dense(1))

	model.compile(loss=params["loss"],
              	optimizer=params["optimizer"],
              	metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

	return model

Then we instantiate a model with a given set of parameters. We use the past 90 observations in the time series as a sequence for the input to the model. The other hyperparameters describe the architecture and the specific choices for training the model. 

params = {
	"loss": "mean_squared_error",
	"optimizer": "adam",
	"dropout": 0.2,
	"lstm_units": 90,
	"epochs": 30,
	"batch_size": 128,
	"es_patience" : 10
}

model = get_model(params=params, input_shape=x_train.shape[1])

The above results in the following Keras model (see Figure 5):

We then create a callback to implement early stopping i.e. to stop training the model if it yields no improvement on the validation dataset for a given number of epochs (in our case 10):

es_callback = tf.keras.callbacks.EarlyStopping(monitor='val_root_mean_squared_error',
                                           	mode='min',
patience=params["es_patience"])

The parameter es_patience refers to the number of epochs for early stopping.

Finally, we fit the model using the predefined parameters:

model.fit(
	x_train,
	y_train,
	validation_data=(x_test, y_test),
	epochs=params["epochs"],
	batch_size=params["batch_size"],
	verbose=1,
	callbacks=[neptune_callback, es_callback]
)

Experiment tracking and model comparison 

Since in this blog post, we want to answer the simple question of which model yields the most accurate predictions for the test dataset, we will need to see how these three models fare against each other.

There are many different approaches for model comparisons such as creating tables and charts that record the evaluation of different metrics, creating graphs that plot the predicted values vs the true values on a test set, etc. However, for this exercise, we will be using neptune.ai.

It’s a metadata store for MLOps, built for teams that run a lot of experiments.‌ It gives you a single place to log, store, display, organize, compare, and query all your model-building metadata.

We first create a Neptune project and record the API of our account. You can check a detailed tutorial on how to do it in the Neptune documentation.

import neptune

# Create a Neptune run object
run = neptune.init_run(
    project="your-workspace-name/your-project-name",  
    api_token="YourNeptuneApiToken",  
)

The variable run can be seen as a folder in which we can create subfolders containing different information. For example, we can create a subfolder called model and record in it the name of the model:

run["model/name"] = "Arima"

We will compare the accuracy of these models with respect to two different metrics: 

1. The root mean square error (RMSE)

2. The mean absolute error (MAE)

Note that these values can be logged into Neptune by setting the corresponding values, for example, setting:

run["test/mae"] = mae
 run["test/rmse"] = mse

The mean square error and the mean average error for the three models can be seen next to each other in the runs table:

The comparison of the three algorithms can be then seen side by side in Neptune, as shown in Figure 7.

We see that ARIMA yields the best performance, i.e., it achieves the smallest mean square error and mean absolute error on the test set. In contrast, the LSTM neural network performs the worst of the three models. 

The exact predictions plotted against the true values can be seen in the following images. We observe that all three models capture the overall trend of the time series but the LSTM appears to be running behind the curve, i.e. it needs more to adjust itself to the change in trend. And Prophet appears to lose against ARIMA in the last few months of the considered test period where it underestimates the true values.

A deeper look into the performance of the models 

ARIMA grid-search

When doing grid-search over different values for p, d, and q in ARIMA, we can plot the individual values for the mean squared error. The colored dots in Figure 11 show the mean square error values for different ARIMA parameters over a validation set.

Trends in Prophet

We collect in Neptune the parameters, forecast data frames, residual diagnostic charts, and other metadata while training models with Prophet. This is achieved using a single function that captures Prophet training metadata and logs it automatically to Neptune.

In Figure 12, we show the change of the different components of the Prophet. We observe that the trend follows a linear increase while the seasonal components exhibit fluctuations.

Why did LSTM fare the worst?

We collect in Neptune the mean absolute error while training the LSTM model over several epochs. This is achieved using a Neptune callback which captures Keras training metadata and logs it automatically to Neptune. The results are shown in Figure 13. 

Observe that while the error on the training dataset decreases over subsequent epochs, this is not the case for the error on the validation set which reaches its minimum in the second epoch and then fluctuates. This shows that the LSTM model is too advanced for a rather small dataset and is prone to overfitting. Despite adding regularization terms such as dropout, we can’t still avoid overfitting.

Conclusions

In this blog post, we presented and compared three different algorithms for time series prediction. As expected, there is no clear winner and each algorithm has its own advantages and limitations. Below we summarize our observations for each algorithm:

1. ARIMA is a powerful model and as we saw it achieved the best result for the stock data. A challenge is that it might need careful hyperparameter tuning and a good understanding of the data. 
2. Prophet is specifically designed for business time series prediction. It achieves very good results for the stock data but, speaking from anecdotes, it can fail spectacularly on time series datasets from other domains. In particular, this holds for time series where the notion of calendar date is not applicable and we cannot learn any seasonal patterns. Prophet’s advantage is that it requires less hyperparameter tuning as it is specifically designed to detect patterns in business time series.
3. LSTM-based recurrent neural networks are probably the most powerful approach to learning from sequential data and time series are only a special case. The potential of LSTM based models is fully revealed when learning from massive datasets where we can detect complex patterns. Unlike ARIMA or Prophet, they do not rely on specific assumptions about the data such as time series stationarity or the existence of a Date field.  A disadvantage is that LSTM based RNNs are difficult to interpret and it is challenging to gain intuition into their behaviour. Also, careful hyperparameter tuning is required in order to achieve good results.

Future directions

So I hope you enjoyed reading this article and now you must have a better understanding of the time-series algorithms that we discussed here. If you want to dig deeper, here are some links to some useful resources. Happy experimenting!

1. PMD ARIMA. The documentation for the respective Python package.
2. Prophet. Documentation and tutorial for Facebook Prophet.
3. Keras LSTM. Documentation and examples for LSTM RNNs in Keras.
4. Neptune. The Neptune website with tutorials and documentation.
5. A blog post on ML experiment tracking with neptune.ai. 
6. A deeper overview of ARIMA models.
7. A tutorial on time series prediction with LSTM RNNs.
8. The original Prophet research paper.

Was the article useful?

Yes No

Thank you for your feedback!

What topics would you like to see for your next read

Let us know what should be improved

Your message (optional)

Submit

Δ

Thanks! Your suggestions have been forwarded to our editors

More about ARIMA vs Prophet vs LSTM for Time Series Prediction

Check out our product resources and related articles below:

Related article

Deep Learning Optimization Algorithms

Read more Related article

Track and Visualize Information From Your Pipelines: neptune.ai + ZenML Integration

Read more Related article

Product Updates September ’23: Scatter Plots, Airflow Integration, and More

Read more Related article

Train, Track, and Deploy Your Models: Neptune + Modelbit Integration

Read more

Explore more content topics:

Computer Vision General LLMOps ML Model Development ML Tools MLOps Natural Language Processing Product Updates Reinforcement Learning Tabular Data Time Series