How Accurate Is Morningstar’s Rating System?

While Morningstar, Inc. (NASDAQ: MORN), the mutual fund and exchange-traded fund (ETF) rating agency, is highly regarded for its investment research, that doesn’t necessarily mean its ratings are always the most accurate. Most investors are not experts, so they rely on third-party ratings to compare and contrast possible investments for their retirement portfolios, none more so than Morningstar.

Even the Financial Industry Regulatory Authority (FINRA) mutual fund analyzer relies on Morningstar. But the system is not infallible, and investors can get carried away by the simple, intuitive five-star Morningstar rating system.

The rating company is a veritable kingmaker among funds. Research from Strategic Insight indicates funds highly rated by Morningstar, at four-star and five-star, showed net positive investment flow every year between 1998 and 2010. Conversely, funds rated average or poor, at between one and three stars, by Morningstar showed net negative investment flow every year over the same period. This is clear evidence that funds lose money unless Morningstar likes them.

However, there is a big difference between net mutual fund flows and fund performance. It is very possible, even commonplace, for a fund to perform well for a few years, receive a large inflow of investor dollars, and then fail to live up to expectations. Even Morningstar warns investors not to rely too heavily on the firm’s star ratings, which are based on past performances relative to similar funds.

These warnings are well heeded. It turns out a majority of highly rated funds in 2004 did not score so highly in 2014. Many mutual fund investors have horizons well beyond 10 years, so staying power matters. Even more intriguing, the lowest-rated funds may produce the greatest excess returns when compared to their style benchmarks.

How the System Works

Conceptually, there are plenty of holes in the Morningstar method. If you boil it all down, the Morningstar star system is entirely dependent on average past returns. This means the system cannot account for outliers, such as when fund managers have one abnormally good or bad year to fudge their trailing average performances. Even worse, the star system cannot tell you if the fund had consistent leadership or if new managers arrived every two years.

Morningstar assigns a one- to five-star ranking to each mutual fund or ETF on a peer-adjusted basis. Every single metric is relative and risk-adjusted. Peer adjustment is achieved by grouping funds with similar assets together and comparing their performances. By “risk-adjusted,” this means all performances are measured against the level of risk a manager assumed to generate fund returns.

The top 10% of funds in a certain category are awarded five stars. The next 22.5% receive four stars, the middle 35% get three stars, the next 22.5% get two stars, and the final 10% get one star. Every mutual fund wants to receive and boast about a higher rating, and Morningstar often charges a fee for the right to advertise its scores.

Naturally, investors prefer to have their money in five-star funds and not in one- or two-star funds. It is for this reason that many rely heavily on Morningstar’s evaluations when making investment decisions. There is a glaring flaw with this approach; by the time the fund receives a five-star rating for past performances, it might be too late to participate. In effect, Morningstar, and its dedicated followers, often show up late to the party.

What Does the Data Say?

In 2014, The Wall Street Journal requested that Morningstar produce a comprehensive list of five-star funds over 10 years starting in 2004. The publication discovered that 37% of funds lost one star, 31% lost two stars, 14% lost three stars, and 3% dropped down to one star. Only 14%, or 58 out of 403, retained their premium ratings.

To express it a different way, investors invest money in a five-star mutual fund in the hopes of achieving five-star results moving forward, yet only 14% of such funds proved worthy of those hopes. If an investor was willing to accept a four- or five-star performance, the results were more palatable, since 51% of Morningstar’s five-star funds in 2004 received a four-star or above rating in 2014.

Given the turmoil of 2007-2009, there may be some recession-created distortions in The Wall Street Journal’s decade-long performance report. However, recessions tend to occur more than once every 10 years (1.6 per decade since the 1960s), so it is rare for a decade without a downturn interrupting mutual fund performances.

Low-cost fund provider Vanguard ran an analysis in 2013 to see how Morningstar-rated funds performed relative to a style benchmark over three-year periods. The goal was to identify excess returns compared to the benchmark, and group those returns by star rating.

The Vanguard study produced two critical findings, the first being “an investor had a less than a 50-50 shot of picking a fund that would outperform regardless of its rating at the time of selection.” This is different than saying five-star funds tend to outperform one-star funds in each category, which is generally true. What it means is that star ratings are not a good predictor of performance when measured against a benchmark.

The more surprising finding was that one-star funds had the greatest excess returns. Vanguard found that funds in the five-, four-, three-, and two-star rating groups outperformed their benchmarks by 37% to 39%, but one-star funds produced excess returns of 46%.

Expense Ratios Have Better Track Records

Russel Kinnel, director of mutual fund research at Morningstar, published a study in 2010 comparing the predictive accuracy of star ratings against simple expense ratios for each fund. He set up three possible measures of performance, which he deemed success ratio, total returns, and subsequent star ratings. The results spoke for themselves.

As Kinnel pointed out, “in every asset class over every time period, the cheapest quintile produced higher Total Returns than the most expensive quintile.” He added that for every “data point tested, low-cost funds beat high-cost funds.” The trend was unchanged for success ratio and subsequent star ratings.

Star ratings did not perform as well as expense ratios. Kinnel noted, “5-star mutual funds beat 1-star funds on our three measures, although there were exceptions.” His data suggests a higher-star fund beats a lower-star fund approximately 84% of the time.

The Bottom Line

Morningstar acknowledges its rating system is a quantitative measure of a fund’s past performance that is not intended to accurately predict future performance. Instead, the company recommends investors use the rating system to evaluate a fund’s track record compared to its peers. It can be the first step in a multi-step process investors can employ to analyze funds before making a purchase.