Download References - ACS - United Nations Economic Commission for Africa

Tracking Forecasters for ECA Country Profiles
Concept Note
Draft: 8 October 2014
1.
Background
In 2013, the United Nation Economic Commission for Africa (ECA) reaffirmed its commitment
to embark on the production of quarterly and yearly country profiles (CPs) that track and monitor
development in African countries, using a set of key economic and social indicators. Following
the meetings on “Coordinating Approaches for the Delivery of Country Profiles” held in May
and November 2013 in Rwanda and Morocco, respectively, a Task Force comprising mainly of
the Sub Regional Offices (SROs), the African Center for Statistics (ACS) and the Macroeconomic
Policy Division (MPD) was formed. This Task Force drafted a proposal outlining why ECA CPs
are important and how they should be developed as corporate products.
ECA CPs provide a vehicle for ECA to produce and disseminate country-and region-specific
policy analyses and recommendations to support African countries’ efforts to: (i) promote
sustainable growth and social development; (ii) strengthen regional integration; (iii) facilitate
development planning and economic governance; and (iv) mitigate against potential risks. Indeed
by providing quality and timely data, information and forecasts on key economic and social
indicators, ECA CPs serve as a compass to assist member States in policy analysis, leading to
evidence-based planning and policy making.
2.
Justification and objectives
To ensure the accuracy, reliability and cross-country comparability of the ECA CPs, it was
recommended that the data provided by different African sub-regions through SROs and the ACS
should be harmonized. It was also recommended that forecasts produced by ECA CPs be
comparable to those provided by other selected institutions such as the International Monetary
Fund’s (IMF), the World Bank (WB), United Nations Development Programme (UNDP), and the
Economist Intelligence Unit (EIU) among others. The objective of such comparisons is to assess
and provide policy makers with a better understanding of the accuracy, and usefulness of forecasts
provided by different institutions, which they can use to inform their strategic policy decisions
(ECA and AUC, 2013; Melet 2014).
While accurate forecasts can lead to improved designing of macroeconomic policies (Bai et al,
2008; Rapach et al, 2010; Bernanke et al, 2005), inaccurate forecasts can be a recipe for disaster
as evident from the unexpected oil shock of the early 1970s and the 2008-2009 Global Financial
Crisis (Burda et al, 2009). By tracking different forecasts generated by different institutions, ECA
CPs will help member States understand the degree of uncertainty surrounding forecasts.
Evaluating forecasts using well established techniques will help one to judge whether or not a
1
forecast produced by any of the institutions is closer to reality. In turn, the evaluation will be
instrumental to identify alternative sources of data and forecasts that are reliable and accurate
which will be used for better monitoring, evaluation and analysis of macro- and socioeconomic
phenomena. The variables of interest are: GDP growth rate, inflation rate, current account balance,
fiscal balance and exchange rate.
3.
Methodology and Data
For illustration, the project will use Morocco as a case study to assess the accuracy of the country
forecasts on five key macroeconomic variables: real GDP growth rate, inflation rate, current
account balance, fiscal balance and exchange rate of Dirham to USD((Dh:USD). The Concept
Note focuses on forecasts provided by institutions which have established reputation in forecasting
economic fundamentals, namely: the International Monetary Fund, the Economist Intelligence
Unit (EIU), African Development Bank (AfDB), UNDESA, and the Morocco High Commission
for Planning (HCP).
More specifically, the exercise involves collecting actual and forecasted data on the selected
variables on Morocco from the institutions mentioned above, and use reliable statistical methods
to assess the accuracy of the forecasted values of the different variables over the last 5 years (i.e.
2009-2013). For illustrative purposes and due to the limited availability of forecasted values for
our variables of interest, the 5 year window is deemed appropriate. More forecasters will be
included in the final concept note once we get hold of more pertinent data1.
Indeed, assessing the accuracy of forecasts provided by private institutions is important to check
whether their forecasts are better than those of public institutions (HCP for example) and
International Organizations (AfDB, UNDESA, IMF etc). However, access to private forecasts
data is a challenge for several reasons: First access to private data requires expensive subscription
and most private forecasters such as INVESTEC, NED BANK GROUP, etc, are predominantly in
South Africa, not in Morocco. Second, despite Morocco is a country in the Oxford sample, there
is a need for ECA to subscript in order to have access to their actual and forecasts data. Therefore,
the only private institution used in this concept note is EIU which could be consider as a
representative sample of private forecasters. Moreover, the Economic Report on Africa (ERA)
relies on some data provided by EIU regarding fiscal balance and exchange rate for example.
To demonstrate the relative performance of forecasts by the various institutions, we adopt the Root
Mean Square Error (RMSE) and decomposition method for Mean Square Error (MSE). The RMSE
method has the advantage of being easy to use given that it provides information about the existing
distance between the outcomes and projections or forecast values.. Hence, the smaller is the
RMSE, the better is the forecast. However, its weakness comes from its inability to provide a
benchmark against which to evaluate the performance and accuracy of forecasts. Theil (1966)
provided a decomposition of the MSE into three main components in order to assess the bias, the
variance and the covariance in relation to the deviation of the actual value from the forecast value.
Hence, the bias provides information on how far the mean of the forecast is from the mean of the
1
Table 2 summarizes data availability and sources (see annex)
2
actual series, while the variance assesses how far the variation of the actual series is from the
variation of the forecast. The covariance component finally captures the remaining unsystematic
forecast error. Therefore, if the forecast is accurate, the bias, the variance or their joint effect,
should be small while the covariance captures most of the error term.
4. Illustration
Assessing the accuracy of forecasts generated by the forecaster is carried out using the different
methods outlined above. It is worth emphasizing that the Root Mean Square Error (RMSE) is scale
dependent and therefore, does not provide a robust benchmark to evaluate how accurate the
forecasts generated are. Hence the illustration places emphasis on the Theil’s decomposition
technique to assess the accuracy of the forecasts provided by the selected institutions. Table 1
presents the summary results of our analysis.
Regarding the real GDP growth rate, since the results from the bias and the variance could not
give a conclusive idea of the relative merit of a given data, the joint effect (which is the sum) of
the bias and variance is used. And the result shows that the smallest joint effect value is found with
data from HCP (0.22) followed by AfDB (0, 25), UNDESA (0.27), IMF (0.33) and EIU (0.35).
Therefore, with regard to real GDP growth rate, forecasts provided by HCP are more accurate (i.e.
closer to reality) than those provided by the other four institutions. This is possibly due to the fact
that HCP as a home institution came up with forecasts close to reality mainly due to its
advantageous position to know the movements of the economy compared to external forecasters.
With respect to the inflation rate, the bias component of the decomposition of MSE is the same
and relatively the smallest for both EIU and UNDESA (0.03), in comparison to the other
institutions’ values. In addition, the variance component is the same and smallest for UNDESA
and AfDB (0.00). Therefore, since UNDESA has the smallest values for both the bias and the
variance, the current year forecasts of UNDESA, with regard to inflation rate, are more accurate
than those provided by the other institutions in the specific case of Morocco based on the sample
used.
With regard to current account balance (CAB/GDP), the joint effect of the bias and variance
component is smallest for AfDB (0.13) compared to the rest of the institutions (excluding
UNDESA)2. Hence, AfDB forecasts, with regard to current account balance data, are more
accurate than those provided by the other institutions.
2
See table 2 in annex for more details
3
Regarding the fiscal balance (internal balance), the joint effect of the bias and variance component
is smallest for HCP (0.26) compared to other forecasters (excluding UNDESA). Hence, forecasts
provided by HCP with regard to fiscal balance, are more accurate than those of other institutions.
With respect to exchange rate, both the bias and the variance components are lower for HCP (0.02)
than EIU, 0.03 and 0.05 respectively (except other institutions)3. As a result, the joint effect is
smaller for HCP (0.04) than EIU (0.08). Therefore, forecasts provided by HCP with regard to
exchange rate are more accurate than those of EIU.
Table 1: Accuracy of IMF, EIU, HCP, AfDB, UNDESA current year forecasts
IMF
EIU
HCP
GDP
Grow
th
Inflati
on
rate
CAB/G
DP
Intern
al
Balan
ce
GDP
Grow
th
Inflati
on
rate
CAB/G
DP
Intern
al
Balan
ce
Exchan
ge Rate
GDP
Grow
th
Inflati
on
rate
CAB/G
DP
Intern
al
Balan
ce
Exchan
ge Rate
Root Mean Square
Error (RMSE)
0.33
2.20
1.64
0.71
0.75
1.58
4.07
2.09
0.33
1.09
1.11
2.35
1.19
0.26
Decomposition of MSE
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Bias
0.30
0.36
0.20
0.37
0.29
0.03
0.05
0.24
0.05
0.21
0.19
0.36
0.24
0.02
Variance
0.03
0.22
0.34
0.02
0.06
0.19
0.11
0.09
0.03
0.01
0.03
0.07
0.02
0.02
Covariance
0.67
0.43
0.46
0.61
0.65
0.78
0.84
0.67
0.92
0.78
0.78
0.57
0.74
0.96
Table 1: Accuracy of IMF, EIU, HCP, AfDB, UNDESA current year forecasts (cont…)
AfDB
GDP Growth
Inflation rate
CAB/GDP
2.49
1.00
0.18
0.07
0.75
1.36
1.00
0.15
0.00
0.85
3.65
1.00
0.06
0.07
0.87
Internal
Balance
UNDESA
GDP Growth
Inflation rate
0.88
0.91
0.69
1.00
1.00
1.00
0.06
0.15
0.03
0.73
0.12
0.00
0.20
0.73
0.98
Source: Authors’ calculation from IMF-WEO (2003-2013), EIU country reports, HCP, AfDB ( African Economic Outlook,
2009-2014) and UNDESA
Root Mean Square Error (RMSE)
Decomposition of MSE
Bias
Variance
Covariance
3
See table 2 in annex for more details
4
Pessimistic and optimistic data sources
Finally, we assessed whether forecasts provided by EIU, IMF, HCP, UNDESA and AfDB for
Morocco are more optimistic4 or pessimistic by comparing the actual and forecasts values of the
variables.
It is interesting to note that four forecasters (IMF, HCP, AfDB and UNDESA) were optimistic
when it comes to the country’s growth rate, with the IMF being most optimistic, followed by HCP,
AfDB and UNDESA while EIU forecasts were pessimistic, i.e predicting lower values than actual
values. Regarding the inflation rate four institutions (EIU, HCP, AfDB and UNDESA) were
pessimistic (i.e. predicting higher inflation rates than the actual rate) while IMF was optimistic
.With regard to the current account balance (CAB/GDP) and fiscal balance, the results show that
all the institutions were optimistic except UNDESA, while forecasts were pessimistic when it
comes to country’s exchange rate.
Conclusion
This note is aimed at assessing the accuracy of the macroeconomic forecasts provided by selected
international and national institutions using Morocco as a case study. Based on statistical
diagnostic tools, the note highlights three main findings. First, even if most of the forecasts
provided by different institutions are close to reality, their degree of accuracy varied considerably.
For example, real GDP growth rate, fiscal balance and exchange rate forecasts provided by HCP
are more accurate than those provided by other institutions. However, for the inflation rate,
UNDESA forecasts are better than those of the other institutions considered, while AfDB has better
forecasts for the current account balance. It is also important to note that the growth forecasts
provided by different institutions tended to be generally more optimistic. Conversely, there is a
general observation of pessimism for inflation. Inflation is a variable which is often and quickly
felt by most observers and yet notoriously difficult to forecast. The findings underscore the need
for policymakers to be informed of the trend as well as relative accuracy of forecasts provided by
different institutions when it comes to policy analysis and decision making.
Annex
4
Forecasts could be optimistic, neutral or pessimistic however this note only focuses on the optimistic and
pessimistic forecast.
5
Table 2: Summary of data used
Variables/Indicators, 2009-2013
GDP
Growth
EIU
Yes
Inflation
rate
Yes
CAB/GDP
Yes
Fiscal Balance
Exchange Rate
(Dh:USD)
Yes
Yes
IMF
Yes
Yes
Yes
Yes
No (forecasts
values are not
available)
HCP
Yes
Yes
Yes
Yes
Yes
AfDB
UNDESA
Yes
Yes
Yes
Yes
Yes
Yes
No (values are
No (forecasts
No (actual
values are
available but
forecasts values
are not
available)
No (forecasts
only available
values are not
values are not
for 2009)
available)
available)
Data source
Morocco country
reports, 2008-2014
(last update
October 2014)
World Economic
Outlook, 20082014(last update
October 2014).
Morocco High
Commission for
planning and
survey reports,
2008-2013
African Economic
Outlook, 20092014
UNDESA
References
Bai, J., & Ng, S. (2008). Forecasting economic time series using targeted predictors. Journal of
Econometrics, 146(2), 304–317.
Bernanke, B., Boivin, J., & Eliasz, P. (2005). Measuring the effects of monetary policy: a factoraugmented vector autoregressive (FAVAR) approach. The Quarterly Journal of Economics,
120(1), 387–422.
Burda, M., & Wyplosz, C. (2009). Macroeeconomics, A European Text. (5th ed.). New York:
Oxford University Press Inc
ECA and AUC (2013) “Economic Report on Africa: Making the most of Africa’s Commodities –
Industrializing for growth, jobs and economic transformation”. ECA, Addis Ababa. Ethiopia.
Jiahan Li and Weiye Chen (2014). Forecasting macroeconomic time series: ASSO-based
approaches and their forecast combinations with dynamic factor models, International Journal of
Forecasting 30 (2014) 996–1015
6
Mellet. André (2014). Analysis of Macro Economic Forecasting Accuracy of South African
National Treasury, Mediterranean Journal of Social Sciences, Vol 5 No 21, ISSN 2039-2117
(online)
Rapach, D. E., Strauss, J. K., & Zhou, G. (2010). Out-of-sample equity premium prediction:
combination forecasts and links to the real economy. Review of Financial Studies, 23(2), 821–862.
Rob J. Hyndman, Anne B. Koehler. (2006). Another look at measures of forecast accuracy,
International Journal of Forecasting 22 (2006) 679– 688
7