Volatility Forecasting with Machine Learning and Intraday Commonality* (2024)

Forecasting and modeling stock return volatility has been of interest to both academics and practitioners. Recent advances in high-frequency trading (HFT) highlight the need for robust and accurate intraday volatility forecasts. For example, Deutsche Börse, one of the world’s leading data and technology service providers, launched the “Intraday Volatility Forecast” project in 2015 to provide intraday volatility forecasts up to 30-min for DAX, EURO STOXX 50, and Euro-Bund.

Engle and Sokalska (2012) pointed out that intraday volatility forecasts are important for managing risk, pricing derivatives, and devising quantitative strategies, especially in HFT. Stroud and Johannes (2014) also demonstrated that intraday measures are useful for market makers, HFT, and option traders. Specifically, intraday volatility forecasts may support traders in assessing the likelihood of price changes and therefore better understanding the risk involved in certain automated trading strategies (see Bates 2019). Screen traders could leverage volatility forecasts to support live trading and enhance the pre-trade transaction cost analysis from the risk assessment of price slippage. Intraday volatility forecasts are also helpful for practitioners screening for high-volatility opportunities and trading corresponding option strategies (see Ni, Pan, and Poteshman 2008).

To the best of our knowledge, unlike daily volatility forecasting, intraday volatility has not yet received much attention in the research literature. It is pointed out by Andersen and Bollerslev (1997) that conventional parametric models, such as Generalized AutoRegressive Conditional Heteroskedasticity (GARCH) and stochastic volatility (SV) models, may fail to reveal certain features of intraday returns. In Andersen et al. (2003) and Corsi (2009), high-frequency data are used to estimate daily realized volatility (RV) by summing squared intraday returns. Some related literature on this aspect will be reviewed briefly in the next section. These methods may reveal important information about characteristics of daily returns and volatilities, but do not easily lend themselves applicable to the task of forecasting intraday volatility.

In this article, we study several non-parametric machine learning (ML) models for forecasting multi-asset intraday volatility by leveraging high-frequency data from the U.S. equity market. We first propose a measure for evaluating the commonality in intraday volatility. The results demonstrate that, by taking advantage of commonality in intraday volatility, the forecasting performance of these ML models improves significantly. Neural networks (NNs) yield both statistically and economically significant improvements in out-of-sample performance over linear regressions and tree-based models, due to their ability to uncover the non-linearity and model complex latent interactions among variables. The improvements remain robust when we apply trained models to new stocks that have not been included in the training set, thus alleviating the overfitting concerns of NNs and providing new evidence toward a certain universality phenomenon in modeling volatility. In the end, our findings reveal that past intraday volatilities provide additional useful information for forecasting daily volatility, and reveal subtle time-of-day effects that aid the forecasting mechanism.

We augment our proposed methodology with a very thorough set of numerical experiments. The data covered in this work span the period from July 2011 to June 2021, and include the top 100 most liquid components of the Standard & Poor’s (S&P) 500 index, and the 10-min, 30-min, 65-min, and daily (without overnight information) forecasting horizons are analyzed.

More specifically, a measure for evaluating the commonality in intraday volatility is proposed, which is the adjusted R-squared value from linear regressions of the RVs of a given stock against the market RVs. It is demonstrated that commonality over the daily horizon is turbulent over time, although the commonality in intraday RVs is strong and stable. The analysis of the high-frequency data from the real market reveals the following interesting phenomena. During a trading session, commonality achieves a peak near closing sessions, in contrast to the diurnal volatility pattern.

Second, in order to assess the benefits of incorporating commonality into models used to predict intraday volatility, multiple ML algorithms (including autoregressive integrated moving averages [ARIMA], heterogeneous autoregressive [HAR], ordinary least squares [OLS], least absolute shrinkage and selection operator [LASSO], XGBoost, multilayer perceptron [MLP], and long short-term memory [LSTM]) are implemented under three different schemes: (i) Single: training specific models for each asset; (ii) Universal: training one model with pooled data for all assets; (iii) Augmented: training one model using pooled data with an additional predictor which takes into account the impact of market RV. It is revealed that for most models, the incorporation of intraday commonality likely leads to better out-of-sample performance, based on the pooled data together with additional information of the market volatility.

The empirical results we present in the article demonstrate that NNs can be superior to other techniques. Empirical evidence is provided to demonstrate the capability of NNs for capturing complex interactions among predictors. Furthermore, to alleviate the concerns of over-fitting, a stringent out-of-sample test is conducted, where the trained models are evaluated on completely new stocks which have not been included in the training sample. Our results reveal that NNs can outperform other approaches. By comparing the result with the performance obtained by OLS models trained for each new stock, we show the validity of a universal volatility mechanism among stocks. Similar findings are reported in Sirignano and Cont (2019) concerning universal features of price formation in equity markets.

We conclude the article by proposing a new approach for predicting daily volatility, in which the past intraday volatilities rather than the past daily volatilities are used as predictors. This approach fully utilizes the available high-frequency data, and therefore contributes to the improvement over traditional methods of modeling daily volatilities. The results presented in this article demonstrate that ML models, where past intraday volatilities are used as predictors, tend to outperform the traditional models with past daily volatilities (e.g., HAR of Corsi [2009], SHAR of Patton and Sheppard [2015], and HARQ of Bollerslev, Patton, and Quaedvlieg [2016]). We believe that our approach brings a novel perspective on research that studies the effectiveness of past intraday volatilities in forecasting future daily volatility, providing new insights into the understanding of volatility dynamics.

The remainder of this article is structured as follows. We begin with Section 1 by reviewing some closely related literature, which however should not be considered as a comprehensive survey of the subject. In Section 2, the data and the definition of RV are described. In Section 3, we discuss the commonality in intraday volatility. Various ML models and three training schemes for predicting future intraday volatility are introduced in Section 4. Section 5 provides the forecasting results and discusses the empirical findings. In Section 6, a new approach to forecasting daily volatility using past intraday volatility as predictors is proposed. Finally, we summarize our study and discuss further avenues of investigation in Section 7.

1 Related Literature

Our study is built upon several research streams proposed by various authors over the recent years. The first stream is related to the research on the commonality in financial markets. Chordia, Roll, and Subrahmanyam (2000) have recognized the existence of commonality in liquidity and Karolyi, Lee, and Van Dijk (2012) have suggested that commonality in liquidity is related to market volatility, in particular, the presence of international investors and trading activity. Dang, Moshirian, and Zhang (2015) have made an observation that the news commonality is associated with stock return co-movement and liquidity commonality.

The co-movement in daily volatility is well known from the previous literature. Traditional GARCH and SV models (e.g., Andersen et al. 2006; Calvet, Fisher, and Thompson 2006) all make use of the volatility spillover effects. Herskovic et al. (2016) have provided empirical evidence of the co-movement in volatility across the equity market. Bollerslev et al. (2018) have observed strong similarities in daily RV and have utilized them to forecast the daily RV. Engle and Sokalska (2012) have emphasized that pooled data are useful for intraday volatility forecasting and Herskovic et al. (2020) have reported that volatilities co-move strongly over time. However, there is still a void of research related to commonality in intraday volatility and its implications for managing intraday risks, especially for forecasting purposes.

Second, there are numerous contributions in the existing literature on the topic of forecasting daily volatility. However, most methods proposed by various researchers for modeling and forecasting return volatility largely rely on the parametric GARCH or SV models, which provide forecasts of daily volatility from daily return. As pointed out by Andersen et al. (2003, 2006) and Engle and Patton (2007), these models employed to predict daily volatility cannot take advantage of high-frequency data, and suffer from the curse of high-dimensionality when dealing with multiple assets simultaneously. Due to the availability of high-frequency data, RV, computed from summing squared intraday returns, has gained popularity in recent years. Andersen et al. (2003) have proposed an AutoRegressive Fractionally Integrated Moving Average (ARFIMA) model for forecasting daily RVs, which outperforms conventional GARCH and related approaches. Corsi (2009) has put forward a parsimonious AR-type model, termed HAR, for predicting daily RVs using various RV components over different time horizons. Recently, Izzeldin et al. (2019) have made a comparison investigation for the forecasting performance of ARFIMA and HAR, and have concluded that their performance is essentially indistinguishable. See Section 6 for more models to predict daily volatility.

Nonetheless, little attention has been paid to the role of forecasting intraday volatility. Taylor and Xu (1997) have proposed an hourly volatility model based on an ARCH specification and Engle and Sokalska (2012) have constructed a GARCH model for intraday financial returns, by specifying the variance as a product of daily, diurnal, and stochastic intraday components. These models, such as traditional GARCH and SV, are potentially restrictive due to their parametric nature, and are not able to effectively take into account the non-linear and highly complex relationships among different financial variables.

Third, ML models have demonstrated great potential in finance, such as their applications in asset pricing. The high-dimensional nature of ML methods allows for better approximations to unknown and potentially complex data-generating processes, in contrast with traditional economic models. Gu, Kelly, and Xiu (2020) have pointed out the superior performance of ML models for empirical asset pricing. Recently, Xiong, Nichols, and Shen (2015) have applied LSTMs to forecast S&P 500 volatility, with Google domestic trends as predictors, and Bucci (2020) has demonstrated that recurrent NNs (RNNs) are able to outperform all the traditional econometric methods in forecasting monthly volatility of the S&P index. Rahimikia and Poon (2020) have compared ML models with HAR models for forecasting daily RV by using variables extracted from limit order books and news. Li and Tang (2020) have proposed a simple average ensemble model combining multiple ML algorithms for forecasting daily (and monthly) RV and Christensen, Siggaard, and Veliyev (2021) have examined the performance of ML models in forecasting 1-day-ahead RV with firm-specific characteristics and macroeconomic indicators.

2 Data and RV

2.1. Data

We use the Nasdaq ITCH data from LOBSTER1 to compute intraday returns via mid-prices. We select the top 100 components of S&P 500 index, for the period between July 1, 2011 and June 30, 2021. After filtering out the stocks for which the dataset does not span the entire sample period, we are left with 93 stocks. Table1 presents the number of stocks in each sector, according to the Global Industry Classification Standard (GICS) sector division.2

2.2. Realized Volatility

In a general form, $Pi,t$ denotes the price process of a financial asset i and it follows:where $μi$ is the drift, $σi,t$ is the instantaneous volatility, and $Wt$ is the standard Brownian motion. The theoretical integrated variance (IV) of stock i during $(t−h,t]$ is estimated aswhere h is the look-back horizon, such as 10 min, 30 min, 1 day, etc.

In this article, we consider the minutely logarithmic mid-price return for asset i during $(t−1,t]$ as

Here, $Pi,t$ is the mid-price at time t, that is, $Pi,t=Pi,tb+Pi,ts2$ and $Pi,tb$ (respectively, $Pi,ta$⁠) represents the best bid (respectively, ask) price.

Andersen et al. (2001) and Barndorff-Nielsen and Shephard (2002) showed that the sum of squared intraday returns is a consistent estimator of the unobservable IV. Because of the availability of high-frequency intraday data, we choose to compute RV as a proxy for the unobserved IV (see Andersen et al. 2001; Bollen and Inder 2002; Hansen and Lunde 2006). To reduce the impact of extreme values, we consider the logarithm, in line with Andersen et al. (2003), Bucci (2020) and Herskovic et al. (2016). Specifically, during a period $(t−h,t]$⁠, the RV is defined as follows3:

As pointed out by Pascalau and Poirier (2021), there are no conclusive methods to incorporate the overnight session’s information content into the daily volatility. In line with Engle and Sokalska (2012), overnight information is excluded from our empirical analysis of daily volatility. For simplicity, we refer to this daily scenario (excluding the overnight) as the “1-day” scenario, throughout the rest of this article.

2.3. Summary Statistics

To mitigate the effect of possibly spurious data errors, for each stock, we set the values of return/volatility below the 0.5 percentile equal to the respective 0.5 percentile, and the values above the 99.5 percentile is set equal to the 99.5 percentile, a process commonly referred to as winsorization. Figure1 illustrates the pairwise Pearson and Spearman correlations of returns and realized volatilities. This figure depicts the empirical distribution of pairwise correlation coefficients over the entire sample period. We generally observe higher correlations in RV than the counterparts in return. Figure1 also reveals that, on average, as the horizon gets longer, RV’s correlations increase from 0.598 (10-min) to 0.731 (30-min) to 0.766 (65-min). However, when turning to daily RV, correlations in RVs become weaker, with an average of 0.514. This indicates that the connections between stocks in terms of intraday volatility may be more stable and tight than the ones in daily volatility.

Figure2 plots the daily RV over time. Stocks demonstrate similar time-series patterns, consistent with Herskovic et al. (2016) and Bollerslev et al. (2018). Additionally, the width shrinks during the periods of higher volatility, such as, stock market crashes in August 2011 (European sovereign debt crisis), between June 2015 and June 2016 (Chinese stock market turbulence and Brexit), in March 2018 (China–U.S. trade war), in March 2020 (COVID-19). Figure3 shows that the diurnal volatility forms a so-called reverse-J-shape, namely larger fluctuations near the open and close (see Harris 1986; Engle and Sokalska 2012).

3 Commonality Estimation

Inspired by prior studies (e.g., Chordia, Roll, and Subrahmanyam 2000; Morck, Yeung, and Yu 2000; Karolyi, Lee, and Van Dijk 2012; Dang, Moshirian, and Zhang 2015), we follow an analogous procedure to estimate the commonality in volatility. Specifically, we use the average adjusted R² value from the following regressions across stocks, as a measure of commonality in volatility (denoted as $R(h)2$⁠)4 where $RVM,t(h)$ (see Bollerslev et al. 2018) is the contemporaneous market volatility during $(t−h,t]$ for stock i, which is calculated as the equally weighted average5 of all individual stock volatilities during $(t−h,t]$⁠, that is,

Figure4 presents the commonality in RV, averaged across stocks for each month. To create this figure, we use the observations in each month to obtain the R² value from Equation (5). We notice that commonality effects in intraday scenarios (especially 30-min and 65-min) are substantially larger than the daily ones. For example, as reported in Table2, the average commonality in 65-min data is around 74.3%, while only 35.5% in daily data. Moreover, $R(h)2$ is much more turbulent at the daily frequency. The last column in Table2 also reports the results of the relation between the average commonality and the market volatility. As the horizon extends, the average commonality has a higher correlation with the market volatility.6

Figure5 reports the averaged values and standard deviations (black vertical lines) of commonality for each half-hour in the trading session. To create this figure, we use the observations in a given interval, such as [09:30, 10:00], to fit Equation (5). We observe a gradual increase in commonality throughout the trading session as we get closer to market close, in sharp contrast to the diurnal volatility pattern in Figure3.

4 Methodology

In this section, we leverage the commonality for the task of predicting cross-asset volatility. We construct the prediction model as follows:where $RVi,t+h(h)$ is the volatility of asset i during $(t,t+h]$⁠. $u$ represents the input features, which can be further separated into two categories: (i) a multi-dimensional vector of predictor variables for a specific stock i available up to time t, denoted as individual features, such as $(RVi,t(h),…,RVi,t−(p−1)h(h))′$ and (ii) a vector of features for all stocks in the studied universe up to t, denoted as market features, such as $(RVM,t(h),…,RVM,t−(p−1)h(h))′$⁠. $θ$ refers to the parameters that need to be estimated. Whenever is clear from the context and no ambiguity arises, we use also use $θ$ to denote the forecasting model. We are aiming to find a function of variables that minimizes the out-of-sample errors for future RV.

4.1. Models

This section summarizes the collection of ML models employed in our numerical experiments.

4.1.1 Seasonal ARIMA

The ARIMA model is a popular forecasting method for univariate time-series data, where an initial differencing step can be applied one or more times to eliminate the non-stationarity of the trend. An ARIMA$(p,d,q)$ is given bywhere $φ(L)=1−∑k=1pφiLi$ and $ρ(L)=1−∑j=1qρjLj$ are the AR and MA lag polynomials, and $ϵi,t$ is the error which is distributed as $N(0,σi2)$⁠. Following Christensen and Prabhala (1998) and Ribeiro et al. (2021), we adopt ARIMA$(1,1,1)$ to model the daily RV.

When the time series exhibits seasonality, the seasonal differencing could be applied to eliminate the seasonal component, which is denoted as seasonal ARIMA (SARIMA). As revealed in Figure3, intraday volatility time series possesses a seasonal component. For modeling intraday RV, we choose the SARIMA, where the seasonal period is the corresponding number of intraday time buckets in a day and other parameters related to the seasonal pattern are set as zero (for more details about SARIMA, see Sheppard 2010).

4.1.2 HAR with diurnal effects

Corsi (2009) proposed a volatility model, named as HAR, which considers realized volatilities over different interval sizes. HAR has shown remarkably good forecasting performance on daily data (Patton and Sheppard 2015; Izzeldin et al. 2019). For day t, the forecast of HAR is based onwhere $RVi,t(d)$ denotes the daily RV in the past day, and $RVi,t(w)=15∑l=15RVi,t−l(d)and RVi,t(m)=121∑l=121RVi,t−l(d)$ denote the weekly and monthly lagged RV, respectively. The choice of a daily, weekly, and monthly lag is aiming to capture the long-memory dynamic dependencies observed in most RV series.

However, very little attention has been paid to forecasting intraday volatility with HAR. One closely connected model is that of Engle and Sokalska (2012), who proposed an intraday volatility forecasting model, where they interpret that conditional volatility of high-frequency returns is a product of daily, diurnal, and stochastic intraday components. After the decomposition of raw returns, the authors apply a GARCH model (Engle 1982) to learn the stochastic intraday volatility components.

Following the spirit of Engle and Sokalska (2012), we extend the daily HAR model to intraday scenarios by adding diurnal effect and previous intraday component, as follows7:where $Di,τt+h$ denotes the average diurnal RV in the bucket-of-the-day $τt+h$ computed from the last 21 days. For example, when $t=$ 10:30 and $h=30$ min, then $τt+h$ corresponds to the bucket 10:30–11:00. $RVi,t(h)$ represents the lag = 1 intraday RV. $RVi,t(d)$ (⁠$RVi,t(w)$⁠, $RVi,t(m)$⁠) denotes the aggregated daily (weekly, monthly) RV. When we consider the daily scenarios, Equation (10) becomes the standard HAR model (Equation9), by removing the diurnal term and the intraday component.

4.1.3 Ordinary least squares

Instead of using aggregated RV, we apply OLS to original features, as follows, with its loss function being the sum of squared errors. Recall $u=(u1,…,up)′$ represent the vector of input features, such as past intraday RVs, and (perhaps) market RVs. Notice that the model only incorporating the past intraday RVs as features is actually an autoregressive (AR) model:

4.1.4 Least absolute shrinkage and selection operator

When the number of predictors approaches the number of observations, or there are high correlations among predictor variables, the OLS model tends to overfit noise rather than signals. This is particularly burdensome for the volatility forecasting problem, where the features could be highly correlated.

LASSO is a linear regression method that can avoid overfitting via adding a penalty of parameters to the objective function. As pointed out by Hastie, Tibshirani, and Friedman (2009), LASSO performs both variable selection and regularization, therefore enhances the prediction accuracy and interpretability of regression models. The objective function of LASSO is the sum of squared residuals and an additional $l1$ constraint on the regression coefficients, as shown in Equation (12). Here, the hyperparameter $λ$ controls the penalty weight. In our experiments, we provide a set of hyperparameter values and then choose the one with the best performance on the validation data, as our forecasting model:

4.1.5 XGBoost

Linear models are unable to capture the possible non-linear relations between the dependent variable and the predictors, and the interactions among predictors. As pointed by Bucci (2020), RVs are subject to structural breaks and regime-switching, hence the need to consider non-linear models. One way to add non-linearity and interactions is the decision tree, see more in Hastie, Tibshirani, and Friedman (2009).

XGBoost is a decision-tree-based ensemble algorithm, implemented under a distributed gradient boosting framework by Chen and Guestrin (2016). There is abundant empirical evidence showing the success of XGBoost, such as in a large number of Kaggle competitions. In this work, we only review the essential idea behind XGBoost—tree boosting model. For more details about other important features of XGBoost, such as the scalability in various scenarios, parallelization, distributed computing, feature importance to enhance interpretability, etc., the reader may refer to Chen and Guestrin (2016). Let $u$ represent the vector of input features,where $F$ is the space of regression trees. An example of the tree ensemble model is depicted in Figure6. The tree ensemble model in Equation (13) is trained sequentially. Boosting (see Friedman 2001) means that new models are added to minimize the errors made by existing models, until no further improvements are achieved.

4.1.6 Multilayer perceptron

Another non-linear method is the NN, which has become increasingly popular for ML problems, for example, in computer vision and natural language processing, due to its flexibility to learn complex interactions. Hill, O’Connor, and Remus (1996) and Zhang (2003) suggest that NN is a promising non-linear alternative to the traditional linear methods in time-series forecasting. We introduce two commonly implemented NNs in the following sections.

MLP is a class of feedforward NNs and a “universal approximator” that can learn any smooth functions (see Hornik, Stinchcombe, and White 1989). MLP has been applied to many fields, for example, computer vision and natural language processing. MLPs are composed of an input layer to receive the raw features, an output layer that makes forecasts about the input, and in-between those two, an arbitrary number of hidden layers that are non-linear transformations. MLPs perform a static mapping between an input space and an output space (Bucci 2020). The parameters in MLPs can be updated via stochastic gradient descent. In this work, we use Adam (see Kingma and Ba 2014), which is based on adaptive estimates of lower-order moments. Let $u∈Rp$ represent the input variableswhere $θ:=(W1,W2,…,WL,b1,b2,…,bL)$ represents the parameters in the NN. $Wl∈Rnl×nl−1$⁠, $bl∈Rnl×1$ for $l=1,2,…,L$⁠, and $n0=p$⁠. For the activation function $σ(⋅)$⁠, we choose the rectified linear unit, that is, $σ(x)=max(x,0)$⁠.

4.1.7 Long short-term memory

A RNN requires that historic information is retained to forecast future values. Therefore, RNN is well-suited for processing, classifying, and making predictions based on time-series data (Bucci 2020). LSTM, proposed by Hochreiter and Schmidhuber (1997), is an extension of the RNN architecture by replacing each hidden unit in RNNs with a memory block to capture the long-term effect. LSTMs have received considerable success in natural language processing, time series, generative models, etc.

For simplicity, we consider the time series for a given stock and remove the subscript for stock identity. The standard transformation in each unit of LSTM is defined as follows. For a more detailed discussion, we refer the reader to Hochreiter and Schmidhuber (1997):where $ut$ is the input vector, $ft$ is the forget gate’s activation vector, $it$ is the update gate’s activation vector, $ot$ is the output gate’s activation vector, $c˜t$ is the cell input activation vector, $ct$ is the cell state vector, and $ht$ is the hidden state vector, that is, output vector of the LSTM unit. $°$ is the Hadamard product function. $σg$ is the sigmoid function, and $σc,σh$ are hyperbolic tangent function. $Wf(i,o,c),bf(i,o,c)$ refer to weight matrices and bias vectors that need to be estimated.

To summarize, we first consider a traditional time-series model ARIMA, then include three linear regression models, that is, HAR(-D), OLS, and LASSO. To account for the nonlinear impact of individual predictors on future volatilities and the interactions among predictors, we choose an ensemble tree model XGBoost and two NNs, that is, MLP and LSTM. The primary difference between MLP and LSTM is that LSTM has feedback connections, which allow learning the dependencies in the input sequences.

4.2. Training Scheme

Motivated by the strong commonality in volatility across stocks, we consider the following three different schemes for model training.

• Single denotes that we train customized models $Fi$ for each stock i, as in Bucci (2020) and Hansen and Lunde (2005). We use a stock’s own past RVs only as predictor features, namely

  $u=(RVi,t(h),…,RVi,t−(p−1)h(h))′$

  and no market features, where p represents the number of lags.

• Universal denotes that we train models with the pooled data of all stocks in our universe. That is, $Fi$ is same for all stocks in Equation (7). As in the Single scheme, we use a stock’s own past RVs only as predictor features and no market features. Sirignano and Cont (2019) showed that the model trained on the pooled data outperforms asset-specific models trained on time series of any given stock, in the sense of forecasting the direction of price moves. Bollerslev et al. (2018) and Engle and Sokalska (2012) suggested that models estimated under the Universal setting yield superior out-of-sample risk forecasts, compared with models under the Single setting, when forecasting daily RV.

• Augmented denotes that we train models with the pooled data of all stocks in our universe, but in addition, we also incorporate a predictor that takes into account the impact of the market RV (e.g., Bollerslev et al. 2018) in order to leverage the commonality in volatility shown in Section 3. Namely, $Fi$ is same for all stocks in Equation (7). We use both individual features and market features as predictors, that is, $u=(RVi,t(h),…,RVi,t−(p−1)h(h),RVM,t(h),…,RVM,t−(p−1)h(h))′$⁠. Note that for HAR with diurnal (HAR-D) models under the Augmented setting, we include aggregated market features as additional features and use the OLS to estimate the parameters.

In summary, compared with the benchmark Single setting, we gradually incorporate cross-asset and market information into the training of models. The hyperparameters for each model are summarized in Appendix B.

4.3. Performance Evaluation

To assess the predictive performance for RV forecasts, we compute the following metrics on the rolling out-of-sample tests (see Patton and Sheppard 2009; Patton 2011; Engle and Sokalska 2012; Bollerslev et al. 2018; Bucci 2020; Rahimikia and Poon 2020; Pascalau and Poirier 2021). Both functions measure losses, so lower values are preferred. Patton and Sheppard (2009) demonstrate that QLIKE has the highest power in the Diebold–Mariano (DM) test. Consequently, we focus more on the QLIKE rather than the MSE:

• $Mean-squared error(MSE):1N∑i=1N1#Ttest∑t∈Ttest(RVi,t(h)−RV^i,t(h))2,$

• $Quasi-likelihood(QLIKE):1N∑i=1N1#Ttest∑t∈Ttest[ exp(RVi,t(h)) exp(RV^i,t(h))−(RVi,t(h)−RV^i,t(h))−1],$

where $RV^i,t(h)$ represents the predicted value of $RVi,t(h)$⁠, the RV for stock i during $(t−h,t]$⁠. N is the number of stocks in our universe, $Ttest$ is the testing period, and $#Ttest$ is the length of the testing period.

4.3.1 Model confidence set

MCS was proposed by Hansen, Lunde, and Nason (2011) to identify a subset of models $M*$ with significantly superior performance from model candidates $M0$⁠, at a given level of confidence. The iterative elimination is based on sequentially testing the following hypothesis:where $Lij,t$ is the loss difference between models i and j at day t in terms of a specific loss function L, such as MSE and QLIKE. The model confidence set (MCS) procedure renders it possible to make statements about the statistical significance from multiple pairwise comparisons. For additional details, we refer to the studies of Hansen, Lunde, and Nason (2011).

4.4. Utility Benefits

We have demonstrated that one can assess the out-of-sample statistical performance for each model via the above metrics and tests. However, in such an approach, the economic magnitude of the gain from complex risk models is ignored. Bollerslev et al. (2018) have proposed a utility-based framework, which gauges the utility benefits of an investor with mean–variance preferences investing in an asset with time-varying volatility and a constant Sharpe ratio. We implement this framework to measure the volatility forecasts. For a more detailed description, we refer the reader to Bollerslev et al. (2018).

The expected utility of a mean–variance investor at t can be approximated aswhere $Wt$ denotes the wealth and $γA$ denotes the absolute risk aversion of the investor.

Assume that the investor allocates a fraction $xt$ of the current wealth to a risky asset with return $rt+1$ and the rest to a risk-free money market account earning $rtf$⁠. Then, the wealth at $t+1$ becomes $Wt+1=Wt(1+rtf+xtrt+1e)$⁠, where $rt+1e≡rt+1−rtf$⁠. After dropping constant terms, the expected utility in Equation (17) amounts towhere $γ≡γAWt$ denotes the investor’s relative risk aversion.

Suppose the conditional Sharpe ratio $SR≡Et(rt+1e)/Et(exp(RVt+1))$ is constant, so that the expected utility is

The optimal portfolio that maximizes this utility is obtained by investing the following fraction of wealth to the risky asset:

To determine the utility gains based on different risk models, the expectation based on model $θ$ is denoted by $Etθ(⋅)$⁠. Assuming that the investor uses model $θ$⁠, then the position $xtθ=SR/γEtθ(exp(RVt+1))$ is chosen. By plugging $xtθ$ into Equation (19) and replacing $Et(exp(RVt+1))$ with the RV $exp(RVt+1)$⁠, the expected utility per unit of the wealth (called realized utility, or in short RU) is given by

If a risk model is ideal, that is, it predicts perfectly the realized volatilities $Etθ(exp(RVt+1))=exp(RVt+1)$⁠, then its realized utility is $SR22γ$⁠. Alternatively, the investor is willing to give up $SR22γ$ of the wealth in order to utilize the perfect risk model instead of investing only in the risk-free asset. In this article, the same Sharpe ratio (SR = 0.4) and the same coefficient of risk aversion (⁠$γ=2$⁠) are applied as in Bollerslev et al. (2018) and Li and Tang (2020).

The previous comparisons are based on a frictionless setting, ignoring the trading cost. The case of incorporating the effect of transaction costs is also considered. Following Bollerslev et al. (2018) and Li and Tang (2020), we assume that transaction costs are linear in the absolute magnitude of the change in the positions, and use the full median bid–ask spread for each of the assets over the last 90 trading days. The realized utility with trading costs deducted, denoted as RU-TC, is simply the realized utility after subtracting the simulated costs. We evaluate this realized utility (with and without trading cost) empirically by averaging the corresponding realized expressions over stocks and the same rolling out-of-sample forecasts.

5 Experiments

5.1. Implementation

For each data set, we divide the observations into three non-overlapping periods and maintain their chronological order: training, validation, and testing. For a given trading day t, the training data, including the samples in the first period [July 1, 2011$,t−251]$⁠, are used to estimate models subject to a given architecture. Validation data, including the recent 1-year samples $[t−250,t]$⁠, are deployed to tune the hyperparameters of the models. Finally, testing data are samples in the next year $[t+1,t+251]$⁠; they are out-of-sample in order to provide objective assessments of the models’ performance. Due to limited computational resources, models are updated annually. In other words, when we retrain the models in the next calendar year, the training data expand by one year, whereas the validation samples are rolled forward to include the samples in the most recent 1-year period, following Gu, Kelly, and Xiu (2020). To examine the effect of model update frequency, we perform a robustness check for HAR-D models in Appendix D and we conclude that the update frequency has insignificant effect on the model’s performance. Our testing period starts from July 1, 2015 until June 30, 2016, and the corresponding training and validation samples are [July 1, 2011, June 30, 2014] and [July 1, 2014, June 30, 2015], respectively. When we predict the RV in [July 1, 2016, June 30, 2017], the training and validation samples are [July 1, 2011, June 30, 2015] and [July 1, 2015, June 30, 2016], respectively. Therefore, our testing sample includes 6 years, from July 2015 to June 2021.

For HAR-D and OLS, we use both the training and validation data for training, due to no requirement of hyperparameter tuning. Given the stochastic nature of NNs, we apply an ensemble approach to MLPs and LSTMs for improving their robustness (see Hansen and Salamon 1990; Gu, Kelly, and Xiu 2020). Specifically, we train multiple NNs with different random seeds for initialization, and construct final predictions by averaging the outputs of all networks. For more information on the model settings, see Appendix B.

In all of the models, the features are based on the observations in the last 21 days. Prior to inputting variables in the models, at each rolling window estimation, we normalize them by removing the mean and scaling to unit variance.

5.2. Main Results

Table3 presents the results of each model under three training schemes.8 Due to limited computation power, MLPs and LSTMs are only performed under the Universal and Augmented settings. We draw the following conclusions from Table3.

We begin by comparing the SARIMA with HAR-D under the Single setting.9 The results show that the ARIMA model achieves similar performance as HAR over the 1-day horizon, consistent with Izzeldin et al. (2019). However, HAR-D yields more accurate out-of-sample forecasts than SARIMA across intraday horizons, that is, 10-min, 30-min, and 65-min.

Regarding linear models, we observe that for HAR-D, Universal shows no improvement in forecasting, compared with Single. HAR-D models trained under Augmented significantly outperform the ones trained under the other two schemes, across all horizons in our study. The average reduction in QLIKE of Augmented compared with Single is 0.031, −0.005, 0.004, and 0.010 over 10-min, 30-min, 65-min, and 1-day, respectively.

Generally speaking, there are significant improvements when moving from HAR-D models to OLS models, over 10-min, 30-min, and 65-min horizons. For example, QLIKEs are reduced from 0.453 (respectively, 0.227, 0.186) with the best HAR-D model (i.e., under Augmented) to 0.430 (respectively, 0.204, 0.171) with the best OLS model (i.e., under Augmented), across the three horizons (i.e., 10-, 30-, and 65-min), respectively. Within the OLS models, conclusions are similar with HAR-D models, that is, no benefits from Universal while significant benefits from Augmented. We also observe similar findings in LASSO as in OLS, suggesting that regularization does not further aid performance. On the other hand, MLPs and LSTMs achieve state-of-the-art accuracy across all measures and intraday horizons (i.e., 10-, 30-, and 65-min), implying the complex interactions between predictors. Further analysis is provided in Section 5.3.

Interestingly, linear models slightly outperform MLPs and LSTMs at the 1-day horizon. This is perhaps expected, and might be due to the availability of only a small amount of data at the 1-day horizon, rendering the NNs to underperform due to lack of training data.

Echoing the findings from Panel A, OLS based on the 21-day rolling daily RVs deliver the higher utility than the HAR-type models, consistent with Bollerslev et al. (2018). NNs still perform the best, with the highest realized utility achieved by LSTMs.

Let us now consider the OLS model as an illustrative example for understanding the relative reduction in error. We compare its QLIKEs under these three schemes, at a monthly level, as shown in Figure7. For better readability, we report the reduction in error of Universal relative to Single (denoted as Univ–Single), the reduction of Augmented relative to Universal (denoted as Aug–Univ), and the reduction of Augmented relative to Single (denoted as Aug–Single). Note that Aug–Single = (Aug–Univ) + (Univ–Single). Negative values of $ΔQLIKE$ indicate an improvement on out-of-sample data and positive values indicate degradation. To arrive at this figure, we average the $ΔQLIKE$ values in each month, across stocks. Figure7 reveals that the improvement of Universal compared with Single is relatively small but consistent. In terms of the benefits of Augmented, it is typically the case that incorporating the market volatility as an additional feature helps improve the forecasting performance, especially for turmoil periods.

An interesting question to investigate is whether the improvements of Universal or Augmented for individual stocks are associated with their commonality with the market volatility. To this end, we present the results in Figure8 for each quintile bucket, sorted by stock commonality (computed from Equation5). From this figure, we observe that the reduction of Augmented in out-of-sample QLIKE relative to Universal is explained by commonality to a large extent. Generally, the out-of-sample QLIKE is expected to decline steadily for stocks with higher commonality. Another interesting result arises from Figure8(a), where we observe that Universal and Augmented actually reduce the out-of-sample QLIKE more for stocks (in the Q1 bucket) that are most loosely connected to the market in terms of 10-min volatility. Further research should be undertaken to investigate this finding.

5.3. Variable Importance and Interaction Effects

This section provides intuition for why NNs perform as strongly as they do, with an eye toward explainability. Due to the use of non-linear activation functions and multiple hidden layers, NNs enjoy the benefit of allowing for potentially complex interactions among predictors, albeit at the cost of considerably reducing model interpretability. To better understand such a “black-box” technique, we provide the following analysis to help illustrate the inner workings of NNs and explain their competitive performance.

5.3.1 Relative importance of predictors

In order to identify which variables are the most important for the prediction task at hand, we construct a metric (see Sadhwani, Giesecke, and Sirignano 2021) based on the sum of absolute partial derivatives (Sensitivity) of the predicted volatility. In particular, to quantify the importance of the k-th predictor, we compute

Here, F is the fitted model under the Augmented scheme, $u$ represents the vector of predictors, and $uk$ is the k-th element in $u$⁠. $ui,t$ represents the input features of stock i at time t. We normalize the sensitivity of all variables such that they sum up to one. In a special case of linear regression, the sensitivity measure is the normalized absolute slope coefficient.

Considering the 65-min scenario as an example, Figure9 reveals that for both OLS and MLP, there has been a tendency of the lagged features to decline in terms of sensitivity, as the lag increases. Additionally, we observe that the sensitivity values rise to a high point at every six lags, corresponding to one day. A distinct difference between the sensitivity values implied by OLS and the ones implied by MLP is that the latter places more weight on the lag = 1 individual RV (Sensitivity = 0.90) and less on the lag = 1 market RV (Sensitivity = 0.059). On the other hand, for OLS, the sensitivities of lag = 1 individual (respectively, market) RV are 0.081 (respectively, 0.069).

5.3.2 Interaction effects

To analyze the interactions between the two most significant features implied by NNs, we adopt an approach (e.g., Gu, Kelly, and Xiu 2020; Choi, Jiang, and Zhang 2021) that focuses on the partial relations between a pair of input variables and responses, while fixing other variables at their mean valueswhere q represent the quantile values for the i-th predictor $ui$⁠.

Figure10 illustrates how predicted volatility (i.e., the fitted values) varies as a function of the pair of predictor variables $RVi,t(h)$ and $RVM,t(h)$⁠, over their support.10 In particular, we analyze the interaction of the lag = 1 individual RV (⁠$RVi,t(h)$⁠), with the lag = 1 market RV (⁠$RVM,t(h)$⁠). As shown in Figure10(a), a parallel shift between the different curves occurs if there are no interaction effects between $RVi,t(h)$ and $RVM,t(h)$⁠. Figure10(b) first reveals that the predicted volatility is non-linear in $RVi,t(h)$ and the slope of that relationship becomes higher after $RVi,t(h)$ exceeds a certain threshold (around 0.5). Furthermore, it demonstrates clear interaction effects between $RVi,t(h)$ and $RVM,t(h)$⁠. As it can be observed from the rightmost region of Figure10(b), the distances between the curves become relatively smaller, conveying the message that, when an individual stock is very volatile, the market effect on it weakens.

5.4. Forecasting RVs of Unseen Stocks

To examine the ability to generalize and address concerns regarding overfitting, we perform a stringent out-of-sample test, that is, using the existent trained models to forecast the volatility of new stocks that have not been included in the training sample, in the spirit of Sirignano and Cont (2019) and Choi, Jiang, and Zhang (2021). For better distinction, we denote the stocks used for estimating ML models as raw stocks and those new stocks not in the training sample as unseen stocks.11 We follow the procedure of training, validation, and testing periods described in Section 5.1. Specifically, to predict the RVs of unseen stocks in a particular year, we train and validate the models using the past data of raw stocks exclusively.

In this experiment, we choose OLS models trained for each unseen stock as the baseline. The results are shown in Table4. Note that models trained under Single cannot be applied to forecast unseen stocks, since they are trained for each specific raw stock individually. From Table4, we conclude that NNs trained on the pooled data of raw stocks have better forecasting performance compared with baselines, across all horizons. This presents new empirical evidence for a universal volatility mechanism among stocks. Furthermore, NNs significantly outperform other methods across three metrics, over 10-min, 30-min, and 65-min forecasting horizons, thus validating their robustness. Concerning the 1-day scenario, NNs obtain comparable results (QLIKE = 0.252) to the best non-NN model (MSE = 0.249, attained by LASSO). The realized utility of the different risk models echoes that of the out-of-sample QLIKE.

6 Forecasting Daily RVs with Intraday RVs

Given the fact that intraday volatility exhibits a high and stable commonality (see Sections 2 and 3), we are interested in the potential benefits of using past intraday RVs to forecast daily RVs.

6.1. Closely Related Literature

Generally speaking, there are two broad families of models used to forecast daily volatility: (i) GARCH and SV models that employ daily returns and (ii) models that use daily RVs. Previous well-established studies have shown that due to the utilization of available intraday information, daily RV is a superior proxy for the unobserved daily volatility, when compared with the parametric volatility measures generated from the GARCH and SV models of daily returns (see Barndorff-Nielsen and Shephard 2002; Andersen et al. 2003; Izzeldin et al. 2019). It is worth noting that in these traditional forecasting daily RV models (e.g., ARFIMA of Andersen et al. 2003; HAR of Corsi 2009; SHAR of Patton and Sheppard 2015; and HARQ of Bollerslev, Patton, and Quaedvlieg 2016), only past daily RVs (or their alternatives) are included as predictors. Even though this is a mainstream approach in the literature, it does not benefit to the full extent from the availability of intraday data. In the presidential address of SoFiE 2021, Bollerslev (2022) also pointed out that “semivariation measured over shorter interday time intervals may afford additional useful information.”

Intraday RV information is also studied for forecasting the 1-day-ahead volatility in several previous works, for example, the mixed data sampling (MIDAS) approach of Ghysels, Santa-Clara, and Valkanov (2005, 2006) and the “Rolling” approach of Pascalau and Poirier (2021). In particular, the classic MIDAS approach uses smooth-distributed lag polynomials of high-frequency predictors to forecast the low-frequency target variables, in the form $RVi,t+1(d)=βi,0+βi,1[a(1)−1a(L)]RVi,t(d)+ϵi,t+1$⁠, where the $a(L)$ lag polynomial is defined by scaled beta functions. Ghysels, Santa-Clara, and Valkanov (2006) find that the direct use of high-frequency data does not improve volatility predictions compared with the forecasts from a model based on daily RVs only. We reckon it is due to the restricted flexibility of MIDAS models, usually with one or two parameters determining the pattern of the weights, therefore missing the time-of-day effect of intraday RVs. Pascalau and Poirier (2021) increase the training samples by rolling a fixed window of intraday returns over consecutive trading days by adding and dropping one intraday return at each end. They claim that their proposed “Rolling” approach could potentially capture the changing dynamics of serial correlation throughout the trading day; thus, leading to improved volatility forecasts.

6.2. Proposed Approach

In Section 4.1, we introduced a set of commonly used models, where the daily variables (lagged daily RVs) are employed as predictors when forecasting 1-day RVs. For simplicity, we refer to these models as traditional approaches in this section. Previous sections, such as Figure9, concluded that the most recent RV plays a more important role in forecasting future volatility. Motivated by the fact that intraday volatility has a high and stable commonality, we propose a new prediction approach for forecasting daily volatility using past intraday RVs as predictors, denoted by Intraday2Daily approach.

In contrast to Ghysels, Santa-Clara, and Valkanov (2006) and Pascalau and Poirier (2021), our Intraday2Daily approach takes the time-of-day effect into account in an explicit way and posits the model

Here, $(RVi,t(h),…,RVi,t−(k−1)h(h))$ represent the past RVs for stock i computed over shorter intraday time horizons h at day t and $(RVi,t−1(d),…,RVi,t−(p−1)(d))$ are past daily RVs of stock i up to day $t−1$⁠. Departing from traditional models where all the variables are computed in the daily frequency, we decompose the lag-one daily $RVi,t(d)$ to sub-sampled RVs, that is, $(RVi,t(h),…,RVi,t−(k−1)h(h))$⁠. Under the Augmented training scheme, we also incorporate the market volatilities into models. Figure11 illustrates the comparison between the traditional approach and our Intraday2Daily approach.