Why beating your index is difficult
Most retail investors probably have heard how hard it is to beat the market. Institutional investors know it even more. It’s a well known empirical fact that most active investors in North America underperform the market, understood as the largest index in the US, the S&P 500. But why is this the case? Let's examine some of the reasons.Reason 1: Transaction Costs
The first thing to note is that S&P 500 does not include transaction costs, but real investments do. Any investor must pay a cost, whether directly or indirectly to participate in the stock market. However, when S&P 500 is rebalanced quarterly, transaction costs are not subtracted from the index. This means if that one were to replicate the S&P 500, one would – somewhat paradoxically - still trail the S&P 500.
Reason 2: Indices require deep market knowledge
Another important yet often overlooked aspect is that constructing an index requires deep market knowledge. As discussed in a previous post indices can be designed along multiple dimensions. However, the construction of an index is not merely an abstract design process that concludes once it’s set up; it involves continuous, active maintenance. What may seem like a "passive" product from an investor's perspective actually encompasses numerous decisions and actions. Taking the S&P 500 as an example, it employs strict inclusion criteria such as market capitalization, liquidity, domicile, public float, and financial visibility. "Financial visibility" requires that for a company to be included in the index, both its most recent quarterly earnings and the sum of its last four consecutive quarters’ earnings must be positive. Thus, the criteria for inclusion in the S&P 500 extend beyond mere market capitalization to include aspects of quality, derived from earnings reports, and the ease of trading—indicated by liquidity and public float. This qualitative assessment by S&P is likely why the S&P 600 Small Cap index has consistently outperformed the Russell 2000, even though both indices aim to capture the U.S. small-cap market. The S&P 600 includes a profitability criterion, which the Russell 2000 lacks. In a market that is moderately efficient, we can expect that qualitative selection criteria will correlate with above-average returns over the long term.
To further illustrate the expertise required to create indices, consider MSCI, a company with over 5000 employees which generates revenue by licensing indices to ETFs and selling them to institutional investors for benchmarks or research purposes. Market participants are willing to pay premium prices for real-time access to these indices, partly because of brand value in the case of ETFs, and because the customers lack the technical capacity to develop similar products. Therefore, given that the construction and maintenance of indices demand both technical proficiency and extensive market knowledge, it is not surprising that the average investor typically underperforms relative to a given index.
Reason 3: The Nature of Competition
A third reason why the average investor with a given investment universe may often find him or herself underperforming an index of the universe I believe is found in the nature of competition. We have probably heard public markets being described as a highly competitive environment, a global game with rational participants dueling for alpha. So, the reasoning goes, this is why investors fail to consistently beat the benchmark: its simply not possible to keep having an edge over other resourceful participants who are constantly trying to gain and maintain an edge over you. Let’s call the argument ”underperformance from competition”. This argument is akin to the well known efficient market hypothesis. However, the argument does not hold on closer scrutiny. The difficulty of a task does not render it impossible. Indeed, we see the same phenomena of top out performance and average under performance in many other fields beside investing, from science, over art to the competitive activity par excellence: sports.
In sports we find a much cleaner rule-based competitive environment, and we still see a similar distribution of out and underperformance. In investing, the average investor underperforms their benchmark, whereas a select few outperforms over the long run. In sports the same phenomena is observed: most goals in soccer are scored by a minority of attacking players in each league, a couple of marathon runners dominate the world records, and the world’s best chess player is the same person today as it was in 2010. What I think is at stake is that competition itself fosters outliers in the top, as outliers in the top gain incrementally valuable experience and knowledge, and outliers in the bottom stop to compete. Said in another way, in investment as in many other domains, the median will typically underperform the average, and the total distribution of performance is right skewed.
Summary
Beating your index is hard, and should for the majority of investors be seen as a stretch goal. But saying it's impossible is a stretch too far.I currently work for a capital association named Symmetry that has performed well over the last decade relative to the S&P 500. If you would like more information about the association or just want to discuss the points in this post, please feel free to reach out.