What's the Significance?


Caution: Technical Discussion to Follow—READ AT YOUR OWN RISK.

As the old saying goes, there are three kinds of lies: lies, damned lies and statistics.

It is true data can be tortured with statistical tools that will make it confess to almost anything. On the other hand, the average investor's ignorance about basic statistics and the role of chance might pose an even bigger problem.

But that is a topic for another day. The objective of the following discussion is to bolster your understanding of a critical concept — statistical significance — so you don't blindly believe the lies.

Dimensional Fund Advisors develops investment strategies based on empirical research conducted through a scientific process. This approach generally requires drawing inferences from "noisy" historical data.

This noise often leads to confusion and erroneous conclusions. However, investors who understand statistical significance will have a much better frame of reference for interpreting results that help us hone in on our best guess.

Historical data is essentially all we have to form expectations. Although it represents what actually happened, the past is only one of many possible outcomes. From an empirical standpoint, there are numerous parallel universes, so to speak, of different outcomes drawn from the same underlying capital market process.

An analysis of historical data should aim to identify risk-and-return relationships without placing too much weight on a period-specific result. Consequently, empirical research should always incorporate statistical tests designed to calculate the likelihood that the results occurred by chance.

This is referred to as "statistical significance," and it will be discussed in the context of what many investors care about first and foremost — expected returns and risk premiums.

Statistical Significance and Expected Returns

The expected return tells us what returns we can expect going forward, or more precisely, it reflects the mean of the distribution. Since expected returns are unobserved, we often estimate them using historical averages.

In an uncertain world, we can never know the "true value" of a result drawn from historical data. But computing a t-statistic can help us determine whether the value is "statistically significant," and if so, it enables us to rule out that the true value is zero.

Over small samples, average returns won't equal zero even if the expected value is zero because returns are noisy. However, we can figure out how likely the historical average return is under the assumption that the true expected return is zero. To do that, we compare the t-stat (more on this later) to a t-distribution.

Assuming the expected return has a zero mean, the t-distribution in Chart 1 shows that the probability of getting a t-stat beyond 1.0 is 32%.

At that level, we don't have condemning evidence that the expected return is not really zero since 32% is a pretty high probability. In other words, if we assume the expected return is different from zero, there is a 32% chance that we'd be wrong.

Chart 1: T-distribution highlighting a t-stat range beyond 1

Source: Dimensional Fund Advisors and Brad Steiman

So how large of a t-stat do we need before we're comfortable saying the expected return is not zero? The conventional approach is to draw that line at 5% statistical significance. As shown in Chart 2, this corresponds to a t-stat of 2. That is, there is only a 5% chance of getting a t-stat of 2 or more if the expected return is really zero, or said another way, you can be 95% confident the expected return is not zero.

Chart 2: T-distribution highlighting t-stat range beyond 2

Source: Dimensional Fund Advisors, Brad Steiman 

If you can't live with a 5% chance that you could be wrong, then you simply require bigger t-statistics to be confident the expected return is not zero. You can be 99% sure with a t-stat of 2.6!

Keep in mind the normal distribution goes on forever, and there will always be some tiny probability that the true mean is zero even if we get a large t-stat, but we have to draw the line somewhere. 

T-stats: Why Bigger Is Better

Understanding why a t-stat of 2 or more is considered statistically significant is important. However, it is vital to simply grasp why bigger t-stats mean the value is more "reliably" different from zero. To begin with, refer to the following equation defining a t-stat:

t-stat = (√T x average) / standard deviation

Decomposing the elements of this equation can demonstrate what leads to bigger t-stats and help instill the intuition behind why a bigger t-stat implies that the observed value is less likely to have a true value of zero.

"Average" is the average of all observations in the sample. This parameter is in the numerator, so as the average increases, so does the t-stat. To illustrate, consider the two data series below:

Series A: 1, 2, 1, 2, 1, 2, 1, 2, 1, 2

Series B: 9, 10, 9, 10, 9, 10, 9, 10, 9, 10

Both have the same number of observations and the same standard deviation. But series A has an average of 1.5 and series B has an average of 9.5. As the average increases, so does the t-stat, meaning it is less likely the true average from series B is actually zero.

The intuition here is that a mean further from zero makes it less likely that the true value is in fact zero.

"√T" is the square root of the number of observations. This parameter is also in the numerator, so as the number of observations increases, the t-stat does as well. Consider the two data series below:

Series A: 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2

Series B: 1, 2, 1, 2

Both have the same average of 1.5 and the same standard deviation of 0.5, but series A has 20 observations and series B only has 4. As the number of observations increases, so does the t-stat, and the observed average becomes more reliable. In this example, series A has a t-stat of 13.4 and series B has a t-stat of 6 due to the difference in the number of observations. This means series A is more reliably different from zero than series B.

The intuition here is that a larger number of observations results in more reliability.

"Standard deviation" is a measure of how much the individual observations in the sample vary from the average. This parameter is in the denominator, so as the standard deviation decreases, the t-stat increases. Consider the two data series below:

Series A: 9, 10, 9, 10, 9, 10, 9, 10, 9, 10

Series B: 18, 0, -18, 32, 10, -20, 40, 15, 8, 10

Both have the same 9.5 average and the same number of observations, but series A has much less volatility and a lower standard deviation than series B. As the standard deviation increases, the t-stat decreases, so the average from series B is less reliably different from zero than the same average from series A. Said differently, there is a greater likelihood the 9.5 average from series B happened by chance due to the volatility of the data series.

The intuition here is that a more volatile data series results in a mean that is less reliably different from zero. 

Hypothetical Examples: Risk Premiums

Illustrating how this applies to real data might help solidify the concept. We regularly discuss three types of expected returns: the U.S. market premium, the U.S. value premium and the U.S. size premium.

An important consideration for investors is the likelihood that these risk "premiums" are actually zero (i.e., there is no premium) despite a historical mean that is positive. As discussed, the starting point is calculating a t-stat for each return series as outlined in Table 1 below.

(Note: Presentation of the next five tables and associated discussion is intended for illustrative purposes only and represents an extended sample period to show how an analysis of t-stat data might be accomplished. Results will differ over different timeframes and different data sets. The intended takeaway below is not tied to any specific results, but rather how such an analysis generally takes place.) 

Table 1: U.S. equity risk premiums: t-stats


U.S. Equity Premium (RM — RF) 3.52
U.S. Value Premium (HML) 3.22
U.S. Size Premium (SMB) 2.41

The t-stats above are all considered statistically significant (i.e., greater than 2), and we can almost be 99% sure that all three risk premiums are positive, with only the SMB t-stat being marginally lower than the required 2.6 for that level of significance.

All three data series have the same number of observations, so differences in their t-stats will be a function of different means and standard deviations, as illustrated in Table 2 below.

Table 2: U.S. equity risk premiums: mean and standard deviation


  Mean Standard Deviation
U.S. Equity Premium (RM — RF) 8.04 20.95
U.S. Value Premium (HML) 4.89 13.94
U.S. Size Premium (SMB) 3.75 14.25

As you can see, the equity premium is the most reliable (i.e., different from zero) despite having the highest volatility because it has a significantly higher mean to go with it. Conversely, the size premium is less reliable than the value premium despite having nearly the same volatility because it has a lower historical mean.

Solving for T

Fama and French often remark that it requires an investment lifetime for risk premiums to be reliable. Why? We can input values for the average premium and standard deviation from Table 2 into the equation for t-stats but rearranged to solve for T, and find the T that gives us a t-stat of 2.

T = √[(t-stat x standard deviation) / average return]

For example, the historical average US equity premium was about 8% with a standard deviation of 20%. If we plug those values into the equation above and set the t-stat equal to 2, then the result for T is 26 years! This means it takes twenty-six years of data before we can say the equity premium is reliably different than zero. Table 3 contains the value of T that results in a t-stat of 2 for all three risk premiums.

Table 3: U.S. equity risk premiums: minimum years of data for statistical significance (t-stat > 2)


U.S. Equity Premium (RM — RF) 26
U.S. Value Premium (HML) 26
U.S. Size Premium (SMB) 61

Many advisors are initially surprised to learn the number of years required for historical observations of risk premiums to be considered "statistically significant." Uneasiness may set in from the notion that their clients must invest for twenty-six years in order to earn a positive return.

This is no doubt a daunting proposition, considering that most clients don't interpret results over an investment lifetime but over intervals more likely in the range of three to five years. Unfortunately, a period of five years amounts to nothing more than noise because, as explained above, a small number of observations will likely result in a small t-stat.

The consequence: Few, if any, inferences can ever be drawn from periods that many investors consider long term.

It is important to distinguish between (a) the number of observations required to determine whether a risk premium is likely to be different from zero (t-stat > 2) and (b) how long an investor must wait for positive risk premiums to be realized.

The latter can materialize over short periods of time; however, the length of time demanded of the former supports the notion that investing is a risky proposition and that investors can experience prolonged periods in which they are not rewarded for the risk they took. Table 4 outlines the longest historical periods without a positive equity, value, and size premium, respectively.

Table 4: U.S. equity risk premiums: longest period without a positive risk premium


  Duration Time Period
U.S. Equity Premium (RM — RF) ~16 years November 1958–August 1974
U.S. Value Premium (HML) ~16 years December 1983–January 2000
U.S. Size Premium (SMB) ~27 years July 1983–March 2010

Confidence Intervals

Finally, it is important to remember that if the premium is considered statistically significant, it does not imply that the historical mean is a precise estimate of the true mean of the distribution. Statistical significance simply suggests it is likely that the true mean of the distribution is not zero. The confidence interval, on the other hand, can tell us the range of values the actual premium could be. The equation for confidence intervals is as follows:

Average 2 x (std dev / √T)

Below are the values for the equity, size, and value premiums in the US with 95% confidence intervals. Here, we assume that expected values aren't moving around so that the past predicts the future. However, this might not be the case, as risk premiums could be higher or lower than what we've seen in the past.

Table 5: U.S. equity risk premium ranges: 95% confidence interval


U.S. Equity Premium (RM — RF) 3.51–12.26
U.S. Value Premium (HML) 2.13–7.53
U.S. Size Premium (SMB) 0.47–5.99


Many of you probably vowed not to revisit these concepts after closing the book on your last statistics exam in college, but if you are still reading at this point, hopefully you have gleaned a few key points from this exhaustive overview.

Despite the propensity of investors to look at results over short horizons, an understanding of the statistical properties of market returns reinforces the notion that these periods amount to mostly noise. Investors without a grasp of basic statistics could draw erroneous conclusions by simply looking at the point estimates from a short sample period.

This is particularly important when setting expectations with clients because risk premiums have been (and will be) zero, negative, or lower than expected over time frames that investors consider to be long periods.

While we do know these periods are bound to occur, unfortunately we don't know when. For this reason, the contributions an advisor makes to setting the right expectations and bringing discipline to the process are both crucial and extremely valuable.

If you take anything away from this discussion, the points to remember are:

  • Periods that your clients define as "long term" are mostly noise.
  • Maintaining discipline requires that you prevent clients from extrapolating the recent past and drawing erroneous conclusions from short periods.
  • The formula for successful investing is simple, but "noisy" data means it isn't easy to implement. This makes the role of an advisor challenging but also extremely valuable.
  • Don't trust any number that is not accompanied by a t-stat!

(The contributions of Marlena Lee are gratefully acknowledged.)

This is not to be construed as an offer, solicitation, recommendation, or endorsement of any particular security, product or service. There is no guarantee investment strategies will be successful.  Investing involves risks, including possible loss of principal. Performance may contain both live and back-tested data. Data is provided for illustrative purposes only, it does not represent actual performance of any client portfolio or account and it should not be interpreted as an indication of such performance. IFA Index Portfolios are recommended based on time horizon and risk tolerance. Take the IFA Risk Capacity Survey (www.ifa.com/survey) to determine which portfolio captures the right mix of stock and bond funds best suited to you.  For more information about Index Fund Advisors, Inc, please review our brochure at https://www.adviserinfo.sec.gov/ or visit www.ifa.com.