Caution: Technical Discussion to Follow—READ AT YOUR OWN RISK.
As the old saying goes, there are three kinds of lies: lies, damned lies, and statistics. It is true data can be tortured with statistical tools that will make it confess to almost anything. On the other hand, the average investor's ignorance about basic statistics and the role of chance might pose an even bigger problem, but that is a topic for another day. The objective of the following discussion is to bolster your understanding of a critical concept—statistical significance—so you don't blindly believe the lies.
Dimensional develops investment strategies based on empirical research conducted through a scientific approach. This process generally requires drawing inferences from "noisy" historical data. This noise often leads to confusion and erroneous conclusions. However, investors who understand statistical significance will have a much better frame of reference for interpreting results that help us hone in on our best guess.
Historical data is essentially all we have to form expectations. Although it represents what actually happened, the past is only one of many possible outcomes. From an empirical standpoint, there are numerous parallel universes, so to speak, of different outcomes drawn from the same underlying capital market process.
An analysis of historical data should aim to identify riskandreturn relationships without placing too much weight on a periodspecific result. Consequently, empirical research should always incorporate statistical tests designed to calculate the likelihood that the results occurred by chance. This is referred to as statistical significance, and it will be discussed in the context of what many investors care about first and foremost—expected returns and risk premiums.
Statistical Significance and Expected Returns
The expected return tells us what returns we can expect going forward, or more precisely, it reflects the mean of the distribution. Since expected returns are unobserved, we often estimate them using historical averages. In an uncertain world, we can never know the "true value" of a result drawn from historical data. But computing a tstatistic can help us determine whether the value is "statistically significant," and if so, it enables us to rule out that the true value is zero.
Over small samples, average returns won't equal zero even if the expected value is zero because returns are noisy. However, we can figure out how likely the historical average return is under the assumption that the true expected return is zero. To do that, we compare the tstat (more on this later) to a t distribution.
Assuming the expected return has a zero mean, the t distribution in Chart 1 shows that the probability of getting a tstat beyond ±1.0 is 32%. At that level, we don't have condemning evidence that the expected return is not really zero since 32% is a pretty high probability. In other words, if we assume the expected return is different from zero, there is a 32% chance that we'd be wrong.
Chart 1. tdistribution highlighting a tstat range beyond ±1
So how large of a tstat do we need before we're comfortable saying the expected return is not zero? The conventional approach is to draw that line at 5% statistical significance. As shown in Chart 2, this corresponds to a tstat of ±2. That is, there is only a 5% chance of getting a tstat of ±2 or more if the expected return is really zero, or said another way, you can be 95% confident the expected return is not zero.
Chart 2. tdistribution highlighting tstat range beyond ±2
If you can't live with a 5% chance that you could be wrong, then you simply require bigger tstatistics to be confident the expected return is not zero. You can be 99% sure with a tstat of ±2.6!
Keep in mind the normal distribution goes on forever, and there will always be some tiny probability that the true mean is zero even if we get a large tstat, but we have to draw the line somewhere.
tstats: Why Bigger Is Better
Understanding why a tstat of ±2 or more is considered statistically significant is important. However, it is vital to simply grasp why bigger tstats mean the value is more "reliably" different from zero. To begin with, refer to the following equation defining a tstat:
tstat = (√T x average) / standard deviation
Decomposing the elements of this equation can demonstrate what leads to bigger tstats and help instill the intuition behind why a bigger tstat implies that the observed value is less likely to have a true value of zero.
"Average" is the average of all observations in the sample. This parameter is in the numerator, so as the average increases, so does the tstat. To illustrate, consider the two data series below:
Series A: 1, 2, 1, 2, 1, 2, 1, 2, 1, 2
Series B: 9, 10, 9, 10, 9, 10, 9, 10, 9, 10
Both have the same number of observations and the same standard deviation. But series A has an average of 1.5 and series B has an average of 9.5. As the average increases, so does the tstat, meaning it is less likely the true average from series B is actually zero.
The intuition here is that a mean further from zero makes it less likely that the true value is in fact zero.
"√T" is the square root of the number of observations. This parameter is also in the numerator, so as the number of observations increases, the tstat does as well. Consider the two data series below:
Series A: 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2
Series B: 1, 2, 1, 2
Both have the same average of 1.5 and the same standard deviation of 0.5, but series A has 20 observations and series B only has 4. As the number of observations increases, so does the tstat, and the observed average becomes more reliable. In this example, series A has a tstat of 13.4 and series B has a tstat of 6 due to the difference in the number of observations. This means series A is more reliably different from zero than series B.
The intuition here is that a larger number of observations results in more reliability.
"Standard deviation" is a measure of how much the individual observations in the sample vary from the average. This parameter is in the denominator, so as the standard deviation decreases, the tstat increases. Consider the two data series below:
Series A: 9, 10, 9, 10, 9, 10, 9, 10, 9, 10
Series B: 18, 0, 18, 32, 10, 20, 40, 15, 8, 10
Both have the same 9.5 average and the same number of observations, but series A has much less volatility and a lower standard deviation than series B. As the standard deviation increases, the tstat decreases, so the average from series B is less reliably different from zero than the same average from series A. Said differently, there is a greater likelihood the 9.5 average from series B happened by chance due to the volatility of the data series.
The intuition here is that a more volatile data series results in a mean that is less reliably different from zero.
Examples: Risk Premiums
Illustrating how this applies to real data might help solidify the concept. We regularly discuss three types of expected returns: the US market premium, the US value premium, and the US size premium.
An important consideration for investors is the likelihood that these risk "premiums" are actually zero (i.e., there is no premium) despite a historical mean that is positive. As discussed, the starting point is calculating a tstat for each return series as outlined in Table 1 below.
Table 1. US equity risk premiums: tstats
1927–2010

tstat 
US Equity Premium (RM — RF) 
3.52 
US Value Premium (HML) 
3.22 
US Size Premium (SMB) 
2.41 
The tstats above are all considered statistically significant (i.e., greater than 2), and we can almost be 99% sure that all three risk premiums are positive, with only the SMB tstat being marginally lower than the required 2.6 for that level of significance.
All three data series have the same number of observations, so differences in their tstats will be a function of different means and standard deviations, as illustrated in Table 2 below.
Table 2. US equity risk premiums: mean and standard deviation
1927–2010

Mean  Standard Deviation 
US Equity Premium (RM — RF) 
8.04 
20.95 
US Value Premium (HML) 
4.89 
13.94 
US Size Premium (SMB) 
3.75 
14.25 
As you can see, the equity premium is the most reliable (i.e., different from zero) despite having the highest volatility because it has a significantly higher mean to go with it. Conversely, the size premium is less reliable than the value premium despite having nearly the same volatility because it has a lower historical mean.
Solving for T
Fama and French often remark that it requires an investment lifetime for risk premiums to be reliable. Why? We can input values for the average premium and standard deviation from Table 2 into the equation for tstats but rearranged to solve for T, and find the T that gives us a tstat of 2.
T = √[(tstat x standard deviation) / average return]
For example, the historical average US equity premium was about 8% with a standard deviation of 20%. If we plug those values into the equation above and set the tstat equal to 2, then the result for T is 26 years! This means it takes twentysix years of data before we can say the equity premium is reliably different than zero. Table 3 contains the value of T that results in a tstat of 2 for all three risk premiums.
Table 3. US equity risk premiums: minimum years of data for statistical significance (tstat > 2)
1927–2010

Years 
US Equity Premium (RM — RF) 
26 
US Value Premium (HML) 
26 
US Size Premium (SMB) 
61 
Many advisors are initially surprised to learn the number of years required for historical observations of risk premiums to be considered "statistically significant." Uneasiness may set in from the notion that their clients must invest for twentysix years in order to earn a positive return.
This is no doubt a daunting proposition, considering that most clients don't interpret results over an investment lifetime but over intervals more likely in the range of three to five years. Unfortunately, a period of five years amounts to nothing more than noise because, as explained above, a small number of observations will likely result in a small tstat. The consequence: Few, if any, inferences can ever be drawn from periods that many investors consider long term.
It is important to distinguish between (a) the number of observations required to determine whether a risk premium is likely to be different from zero (tstat > 2) and (b) how long an investor must wait for positive risk premiums to be realized. The latter can materialize over short periods of time; however, the length of time demanded of the former supports the notion that investing is a risky proposition and that investors can experience prolonged periods in which they are not rewarded for the risk they took. Table 4 outlines the longest historical periods without a positive equity, value, and size premium, respectively.
Table 4. US equity risk premiums: longest period without a positive risk premium
1927–2010

Duration  Time Period 
US Equity Premium (RM — RF) 
~16 years 
November 1958–August 1974 
US Value Premium (HML) 
~16 years 
December 1983–January 2000 
US Size Premium (SMB) 
~27 years 
July 1983–March 2010 
Confidence Intervals
Finally, it is important to remember that if the premium is considered statistically significant, it does not imply that the historical mean is a precise estimate of the true mean of the distribution. Statistical significance simply suggests it is likely that the true mean of the distribution is not zero. The confidence interval, on the other hand, can tell us the range of values the actual premium could be. The equation for confidence intervals is as follows:
Average ± 2 x (std dev / √T)
Below are the values for the equity, size, and value premiums in the US with 95% confidence intervals. Here, we assume that expected values aren't moving around so that the past predicts the future. However, this might not be the case, as risk premiums could be higher or lower than what we’ve seen in the past.
Table 5. US equity risk premium ranges: 95% confidence interval
1927–2010

Interval 
US Equity Premium (RM — RF) 
3.51–12.26 
US Value Premium (HML) 
2.13–7.53 
US Size Premium (SMB) 
0.47–5.99 
Conclusion
Many of you probably vowed not to revisit these concepts after closing the book on your last statistics exam in college, but if you are still reading at this point, hopefully you have gleaned a few key points from this exhaustive overview.
Despite the propensity of investors to look at results over short horizons, an understanding of the statistical properties of market returns reinforces the notion that these periods amount to mostly noise. Investors without a grasp of basic statistics could draw erroneous conclusions by simply looking at the point estimates from a short sample period.
This is particularly important when setting expectations with clients because risk premiums have been (and will be) zero, negative, or lower than expected over time frames that investors consider to be long periods.
While we do know these periods are bound to occur, unfortunately we don't know when. For this reason, the contributions an advisor makes to setting the right expectations and bringing discipline to the process are both crucial and extremely valuable.
If you take anything away from this discussion, the points to remember are:
 Periods that your clients define as "long term" are mostly noise.
 Maintaining discipline requires that you prevent clients from extrapolating the recent past and drawing erroneous conclusions from short periods.
 The formula for successful investing is simple, but "noisy" data means it isn't easy to implement. This makes the role of an advisor challenging but also extremely valuable.
 Don't trust any number that is not accompanied by a tstat!
The contributions of Marlena Lee are gratefully acknowledged.
Index Fund Advisors, Inc. (IFA) is a feeonly advisory and wealth management firm that provides riskappropriate, returnsoptimized,
globallydiversified and taxmanaged investment strategies with a fiduciary standard of care.
Founded in 1999, IFA is a Registered Investment Adviser with the U.S. Securities and Exchange Commission that provides investment
advice to individuals, trusts, corporations, nonprofits, and public and private institutions. Based in Irvine, California, IFA manages
individual and institutional accounts, including IRA, 401(k), 403(b), profit sharing, pensions, endowments and all other investment accounts.
IFA also facilitates IRA rollovers from 401(k)s and 403(b)s.
To find out more about the value of IFA
click here.
To determine your risk capacity, take the Risk Capacity Survey.