Can Machine Learning Help Predict Crises?

Forecasting financial crises is difficult. The data comprises a limited set of observed instances and signals a warning only when it is already too late to intervene. Moreover, the early warning models are complex and difficult to distil into simple, transparent indicators for macroprudential authorities. But can we use machine learning to model the crises? A new staff working paper from the Bank of England tries to find out.

 

The Set Up

They use a database that covers 2,499 observations across 17 countries between 1870 and 2016. For the purpose of testing, they exclude the crisis year itself and the following four years, as well 1914-1918 and 1930-1945 (the Great Depression and the two World Wars). This allows them to more cleanly use non-crisis period to predict the crisis. For their models to be successful, it has to predict the crisis one to two years before it erupts.

For the predictive indicators, they look at:

• Yield curve (local and global)
• Credit/GDP ratio (local and global
• Debt-service ratio
• Investment
• Public debt
• Broad money
• Current accounts
• CPI
• Consumption
• Stock market

 

Using Machine Learning

The major innovation of the paper is that they use machine learning. The conventional approach is to run regressions. They do run one as their baseline model for comparison purposes. But the advantage of machine learning models is that they more easily account for non-linearity and interactions between the indicators.

In terms of the inner workings of machine learning. At a basic level, you throw in your data – without specifying the rules in the form of an equation – and let the algorithm discover patterns. With a regression you would specify the rules upfront.

Moreover, there are different types of machine learning models. These include:

(1) Decision tree splits data into subsets across a series of binary nodes. The first node is “start” which splits data into “equal or greater than 8.5” and “less than 8.5”. At the terminal “leaves” a probability of the outcome is calculated based on the data.

(2) Random forest consists of hundreds of different decision trees trained on different subsets of data and averages them. A different, randomly selected subset of data is used to construct each tree.

(3) Extremely randomised trees (“extreme trees”) are like random forests except that each tree is trained on the complete data rather than a subset.

(4) Support vector machines learn a linear function of the inputs (i.e. similar to a regression) but the inputs are transformed first in order to model nonlinear classification problems.

(5) Neural network maps the inputs (yield slope, CPI, etc) to various outputs (probability of recession) via intermediate “hidden” nodes. These hidden nodes present a partial transformation of the input nodes

From these models, the predicted crisis probability for each individual observation can be decomposed into the sum of contributions from each predictor using something called  Shapley values. For a linear regression, the Shapley value of a predictor is simply the product of its regression coefficient and the difference between the predictor value and its mean:.

 

The Best Machine Learning Model is Extreme Trees

It turns out the best model for predicting crises was to use the extreme tree machine learning models. This was based on its ratio of hit-rate versus false positives: it generated an 80% hit rate and 19% false positives when the model is calibrated to identify a crisis with a predicted probability of 9.6%.The extreme trees model also performs best in alternative experiments where some of the data has been transformed.

The extreme tree model missed out on 6 out of 49 crisis events fully, and another 6 partly. It does have a large number of false alarms although many of these occur more than two years prior to a crisis, providing an early warning signal; cluster when other countries experience crises; and the number of false alarms drops after WWII.

 

Results

As for the best predictive indictors signalled by the models, it was the global yield curve slope and global credit (Fig 1 and 2). Indeed, most crises occurred during periods of high credit growth and globally flat/inverted yield curves, including the 2007 crisis

There were some caveats. The domestic yield curve is particularly informative in the absence of a recession. This is because the onset of a recession typically causes the yield curve to steepen and leads to a lower predicted crisis probability.  A flat or inverted yield curve is of greater concern when nominal yields are low. And global credit shows a particularly strong nonlinearity: only very high global credit growth beyond a certain point influences the prediction of the models.

 

Bottom Line

The study is useful in introducing new modelling approaches to crises prediction. They also uncovered a new indicator – the global (rather than local) yield curve. The credit growth measure has been identified in earlier crisis literature. But confirmation of this is reassuring. More importantly, they do find that these indicators are affected by non-linearities, which more conventional models do not reveal.

 

Figure 1: Mean Absolute Shapley Values of Individual Variables Across Different Models

Source: Page 33 of  “Credit growth, the yield curve and financial crisis prediction: evidence from a machine learning approach“

 

Figure 2: Mean Difference of Shapley Values (Crisis Minus Non-crisis Observations) For Different Periods

Source: Page 45 of  “Credit growth, the yield curve and financial crisis prediction: evidence from a machine learning approach“