Projecting Uncertainty: A Roadmap to the Projected Standings

There's a 90 percent chance Detroit's projected 85 wins will be wrong. (via Keith Allison)

There’s a 90 percent chance Detroit’s projected 85 wins will be wrong. (via Keith Allison)

Editor’s Note: When this article was written, the FanGraphs Projected Standings projected the Detroit Tigers for 85 wins and the Cleveland Indians for 84. During the editing process, those two flipped, and then just before publication, the two teams each landed at 84 wins. In addition, at the time this was written, no team was projected to win more than 91 games, though just before publication the Nationals were up to 94 wins. The precise numbers are fluid, but that fluidity does not affect the overall point of this piece.

On Sept. 25, 2008, the Dodgers clinched the NL West title with an Arizona loss in St. Louis. That night, L.A. lost to San Diego to fall to 83-76 on the season. The Dodgers would go on to drop two of three to San Francisco, finishing the year at 84-78. That was the last time a team won its division with fewer than 86 wins.

Why, then, does FanGraphs have the Indians projected to win the AL Central with just 85 wins? As we’ve seen, a team can win its division at 85-77, but it is far from expected. It hasn’t happened in the last six years, and it only seldom happens at all. What’s more, save for the strike-shortened 1994 and ’81 seasons, no team has led the majors with fewer than 95 wins since the schedule expanded to 162 games in the early 1960s, yet FanGraphs’ projected standings have the Dodgers and Nationals leading major league baseball with 91 wins each.

This is one of the most common complaints I see with sabermetric analysis. Why are projections so conservative? Do they really think 85 wins will win the AL Central, or that no team will win more than 91 games? That can’t be right, can it?

Before going into this any further, I strongly recommend you read Tom Tango’s Forecasting 2006. He gives a fantastic overview of this issue, and understanding what Tango is saying there is critical to understanding what is going on here. The key point there is that these projections are not exact predictions, but rather our best estimate of the center of a wide range of possible outcomes.

With that in mind, why does FanGraphs project Detroit to win the division with just 85 wins? And why do the projections show 91 wins leading the majors? The short answer is, FanGraphs really isn’t projecting that at all.

As for the long answer: first, let’s focus on the AL Central, and we’ll get into the Dodgers/Nationals issue later. Before we do that, though, let’s break this down into three distinct statements:

  1. Detroit is projected to win 85 games in 2015.
  2. Detroit is projected to win the A.L. Central in 2015.
  3. Detroit is projected to win the A.L. Central with 85 wins in 2015.

The problem here is that only the first statement accurately describes what the projections are doing. While it seems natural to infer the other two from the projected standings page, we can’t actually do that. To see why, let’s take a closer look at each of these statements.

1) Detroit is projected to win 85 games in 2015

This is true, but keep in mind what we learned from the Tango article. This does not mean we found an almanac lying in Doc Brown’s DeLorean and know that Detroit will win exactly 85 games. There is a chance the Tigers will win 90 or 95 games, or that they’ll win 75. They could even win 100 games, or 60-something. All those outcomes are possible, but the most likely outcomes are the win totals in and around the mid-80s. The farther you get away from that center, the less likely it is the Tigers will end up there.

If you graph the likelihood of the Tigers ending up with any given win total based on an 85-win projection, you’ll get something like this:

Detroit 2015 W distribution
When you see the number “85” on the projected standings page, this distribution, or something like it, is what that number really represents. As you can see from the graph, while 85 is the average result, there is only about a six percent chance that Detroit will win exactly that many games. It’s much more likely the Tigers actually end up winning some number of games a bit more or a bit less than 85 games.

So even if we completely trust the projections, we should still expect the exact number given in the projection to be wrong over 90 percent of the time. We should even expect some of the projected win totals to miss by a fairly large margin.

I know that sounds underwhelming, but that uncertainty is an inherent limitation in trying to project future outcomes. If you don’t believe me, give it a try. Write down however many games you think each team will win and then compare that to how they end up at the end of the season. And keep doing that year after year. You might have a year or two when your predictions do really well, but most likely you will be surprised by how much you miss overall.

This applies to any projection: team wins, a player’s on-base percentage, his HR total, anything. Whenever you see a projection, that number really represents the average of a wide distribution of possible outcomes. (The average given is generally the mean of the distribution, but you could also give projections as the median.) Short of finding that almanac, this is the best we can do.

A Hardball Times Update
Goodbye for now.

2) Detroit is projected to win the AL Central in 2015

Yes, the Tigers have the highest projected win total of any team in the AL Central. Going from that to “the Tigers are projected to win the AL Central” is a subtle deviation, but we have to be careful about making that leap. This is because of the uncertainty in the projected win totals we saw above.

Recall that even when we project Detroit for 85 wins, we still expect the Tigers to finish with something other than 85 wins most of the time. This applies to every other team in the division as well. If we take the above graph showing the range of possible outcomes for Detroit and add the other four teams in the division, we get this:

AL Central 2015 W distribution

There is considerable overlap in the range of possibilities for all five teams in the division. Any of these teams conceivably could finish ahead of Detroit, though it is a lot less likely for Minnesota than for Cleveland. While Detroit may be the most likely of the five teams to win the division, having four other teams that could finish ahead of the Tigers means there is still a good chance they won’t.

And this isn’t even because we might be wrong about Detroit having the most talent in the division. Even if we assume that the projections tell us the exact probability of a team winning each of its games, there is still this much uncertainty in projecting its season win total.

To see why, let’s step back for a minute and forget about the AL Central. Instead of having five baseball teams, let’s say we have five coins. All we are going to do is flip each coin 162 times. Each time a coin lands on heads, it gets a win, and each time it lands on tails, it gets a loss. The coin with the most wins after 162 flips wins the division. Simple enough, right?

Now, let me ask a few questions about our five-coin division:

  1. How many wins would you project for each coin (assuming we have five identical, fair coins)?
  2. How many wins would you project for the coin that ends up winning the division, whichever coin that might be?
  3. Which coin do you project to win the division?

The first question is fairly simple: your best bet is to just project 81 wins for each coin, because that is the expected value for the number of heads in 162 coin flips.

The second question is actually more complicated, though. No coin by itself is going to have an expected value of more than 81 wins, but it is extremely likely that at least one out of the five coins will end up with more than 81 wins just by chance. It turns out that if you repeat this experiment a bunch of times, the coin that wins the division will end up with about 88 wins, on average.

That brings us to the third question. We expect, on average, at least one coin will end up around 88 wins, but which one? There is no way to predict that. All five coins have an equally likely chance of winning the division, and no one coin is expected to win it. We could arbitrarily pick one coin and project it for 88 wins, and occasionally we would be right, but more often than not we would just be making our projections worse by missing more heavily on the one we projected to get lucky. This is why the FanGraphs Coin Flip Mode for playoff odds essentially has every team at 81 wins right now.

Now, back to Detroit and the AL Central. Obviously, baseball teams aren’t coins, and baseball games aren’t random coin flips. What this simplified example does show, however, is that even if we know exactly what the probabilities are, uncertainty in our projections is still going to play a significant role in the final standings. The fact that baseball games are driven by unpredictable human elements only adds to that uncertainty.

With that in mind, how many wins do the FanGraphs projections imply for the eventual winner of the AL Central? Even though we don’t project any individual team for more than 85 wins, it is still really likely that at least one team will win more than that. Based on the distribution of possible outcomes for each team from the graphs above, we’d actually expect the division winner to end up with about 89 wins, on average. That is, of course, more games than Detroit (or anyone else in the division) is expected to win on its own.

Which brings us to the third, most complicated question from our coin-flipping example: if none of the teams is actually expected to win enough games to win the division, who is projected to win the division? With the coins, we couldn’t really project any one coin to “win” the division, because all five coins were equally (un)likely to end up on top. In this case, however, Detroit is more likely to win the division than any other team. Can’t we say, then, that Detroit is projected to win the division?

In a sense, we could, but as I said earlier, we have to be careful about making that leap. When we say Detroit is projected win 85 games, recall that that means we estimate 85 as the average of all possible outcomes for Detroit’s season. If we think of the Tigers’ position in the division standings in the same way, where would we expect them to finish the season, on average?

It turns out, it’s actually not first. We can simulate several seasons for the five teams in the AL Central, and Detroit wins (or ties for) the division about 42 percent of the time, but the Tigers’ average finish is second (mean standing position = 2.06, median = 2).

We run into a similar issue to the coin-flipping example, where no one team is truly expected to finish in first place. So while Detroit is projected as the most likely team to win the division, we should still be careful about saying, “Detroit is projected to win the division,” because “projected” in that sense doesn’t mean the same thing as saying, “Detroit is projected to win 85 games.” The latter is the average result we would expect from the projections. The former is not.

First place is not the mean or median of possible standing positions for Detroit, but it is the mode, which means first place is the most likely position for Detroit to finish. Here’s the kicker with that interpretation, though: Cleveland’s most likely finish also is first place. Cleveland will finish (or tie for) first about 34-35 percent of the time, which is less than Detroit but still more often than the Inidans will finish in any other position in the standings. Even if you break ties for first with a playoff in the simulation, first place is still Cleveland’s most likely finish by a small margin. So we have to be careful with this reasoning as well.

This is probably counter-intuitive, but it is an important subtlety of dealing with probability and statistics. You can have a division in which no team is expected to finish first, on average. You can have a division in which a first-place finish is the most likely outcome for two different teams. You can even have both of those scenarios in the same division at the same time.

3) Detroit is projected to win the AL Central with 85 wins in 2015.

We’ve seen there are some problems with saying, “Detroit is projected to win the division.” It would not be completely without justification, but we have to be clear that “projected” in this sense does not mean the same thing as what the projections are officially doing when they give a projected win total. As a result, we absolutely cannot take that statement and combine it with the projected win total to say those two things are projected to happen in conjunction. They aren’t.

Recall that when we project Detroit for 85 wins, in truth our expectation is that the precise number will be wrong over 90 percent of the time. The projection really represents a wide range of possible outcomes, with the most likely outcomes centered on 85 wins. When we start talking about Detroit winning the division, though, 85 is not the center of that range of possibilities.

Returning to our coin-flipping example for a moment, let’s say that we did want to arbitrarily pick one coin and predict it to win the division. We might predict something like, “this coin will win the division with 88 wins,” even though we know 81 wins is a better projection for that coin. What we wouldn’t do is predict the coin to end up with 81 wins and still win the division anyway, because we know that in all likelihood, it will have to win more than expected to end up ahead of each of the other four coins.

Similarly, in the subset of possible outcomes where Detroit wins the division, the Tigers will average around 90 wins. We can see this by keeping track of their win totals when they win the division and when they don’t in our simulation (the vertical lines represent Detroit’s average win totals in the sim):

Detroit simulated seasons
It is possible for Detroit to win the Central with 85 wins, but chances are it will have to over-perform its projection in order to win the division. This unlikelihood of Detroit winning the division with 85 wins that we intuitively understand is in fact an implication of the projected standings as well.

The Dodgers, Nationals and the Best Record in Baseball

We expect whoever wins the AL Central to end up around 89 wins. What about the other five divisions? Using the same process, here are the number of wins we would project for each division winner:

Projected Win Totals for Division Winners
Division Wins
AL East 92
AL Central 89
AL West 92
NL East 92
NL Central 91
NL West 92

Most division winners are expected to end up around 91-92 wins. Which brings us right back to the Dodgers/Nationals issue. Even if we project all the division winners rather than the individual teams, we still don’t get anyone winning more than 92 games.

The reason for this, of course, is the same reason we didn’t have any one team in the AL Central projected for more than 85 wins. Just as that 85-win Detroit projection represented the average of a range of possible outcomes, these numbers also represent a range of possible outcomes (look, for example, at the range of orange dots in the previous graph). And just as we expect at least one of the five teams in the AL Central probably will win more than 85 games, we expect at least one of the six division winners will win more than 92 games.

In practice, we would expect to see the six division winners spread out more along these lines (each vertical line represents the records for the six division winners in one simulated season):

Division winners
On average, the team that winds up with the best record in the majors wins 98 games. This goes back to something we touched on earlier: “Even if we completely trust the projections, we should still expect the exact number given in the projection to be wrong over 90 percent of the time. We should even expect some of the projected win totals to miss by a fairly large margin.”

We are now looking at 30 different teams, and it is a near certainty some of the projected win totals will miss the actual win total by a fairly big margin. It isn’t as likely to happen within a five-team division, but across the majors, we’re more likely to see teams winning something like 10-15  games more than we project, even if our projections are good.

This is all implicit in the projected win totals on the projected standings page. And, as I said before, it’s not just because the projections could be wrong about exactly how good each team is. They could, of course, be wrong. Maybe Detroit is not quite as good as Cleveland. Maybe Miguel Cabrera and David Price are not quite as good coming into 2015 as their past performance indicates, or maybe Michael Brantley and Corey Kluber are even better than we think. Or maybe we got the player talent levels right, but Cabrera’s or Victor Martinez’s injuries end up keeping him out of the lineup, or a team makes a big trade and gets better or worse than we thought it would be.

There are a million ways the projections could be off by some amount. All that adds uncertainty to the projected win totals, which adds to the likelihood that some team ends up way over its projection. Even without any of that, though, even if we assume the projected win totals are a perfect estimate of each team’s probability of winning each game, everything I wrote about above still happens.

In fact, the graphs and numbers above were all created with the assumption that the projections do perfectly describe each team’s chances of winning each game. This isn’t really the proper way to do this if you want more precise estimates of these distributions and probabilities, but for illustrative purposes, it establishes a minimum amount of uncertainty that we cannot avoid and shows the consequences of that uncertainty.

Again, I realize this sounds underwhelming, but it’s the reality. It’s something anyone who has ever tried to comprehensively project (and keep track of) these things has come to appreciate.

You might then ask, if we know that some team is going to win more than 91 games, why don’t we just bump up the projections for the Dodgers or Nationals to 98 wins so that the projected standings take on a more sensible spread?

For one, the projections are intended to tell us about the estimated talent level of each individual team. We could bump up the Dodgers or Nationals projection based on where we think the league leader will finish, but then we are really projecting the league as a whole, not the Dodgers or Nationals. Our projection would no longer be a best, or even necessarily a useful, estimate for the team we are projecting. The expected error in the projections would increase, and we would have biases where the teams projected for the best records would systematically perform worse than their inflated projections.

On top of that, we don’t know which team to bump up. And I don’t just mean which of the Dodgers and Nationals — it could end up being any number of teams. The whole reason we expect someone to reach the mid-to-upper 90s at all is that there are so many teams that conceivably could do it. After all, how many would have picked the Angels as that team going into last season?

Let’s say we pick the Dodgers and project them for 98 wins just because we expect someone to get to 98. Now where do we project the Nationals? We might put them at 94-95 wins since we project the next-best team to end up around there, and we don’t necessarily expect two different teams to get to 98 wins.

Except now we have a problem. We project that the Nationals and Dodgers are of equal strength, but we now have one team projected as three or four wins better than the other anyway. Not only that, but which team we choose to list as better than the other is arbitrary. One use of the projected standings is to compare teams to each other, but those comparisons no longer would be meaningful once you start adjusting the projections up or down to alter the spread.

If we just see the projections for the Dodgers and Nationals at 91 games each, and so on for every other team in the league, then with a proper understanding of probability and statistics, we can work out from that what range of values to expect from the division winners or from the best record in the majors, and they end up being pretty reasonable. If we alter the initial projections to put one of the Dodgers/Nationals at 98 wins and the other at 95, etc.then there is no way to work backwards from that and get the actual projected team strengths. We can’t see from these new forecasts that the Dodgers and Nationals are roughly equal or that both have about as good a chance at finishing under 91 wins as above.

Furthermore, the adjustments would depend on whether we are projecting a team in the context of all major league baseball, just the AL or NL, or just its division. If we are projecting how the Dodgers would do baseball-wide MLB, we might predict 98 wins. If we just want to project how they’ll do in the NL, we would put them at 96 wins. Within the NL West, 92 wins. By themselves, 91.

Simply put, projecting what one specific team will do individually and projecting what a group of teams will do collectively are two different things. This is because with one team, we have no way to know if it will over- or under-perform its projection. Either possibility is about equally likely. With five teams, however, we can be reasonably confident that at least one will over-perform its projection. With thirty teams, we can be reasonably confident that at least one will over-perform its projection by a lot. We know it will happen, but we have no way to tie that over-performance to any particular team.

As tempting as it is to treat the projected standings as pinpoint forecasts and infer things about the spread of the final standings from the spread of the individual projections, we can’t do that. If you want to infer how many wins it will take to win the division from the projected standings, you have to take into account the likelihood that any particular team will have to over-perform its projection to win the division, and the same with all of major league baseball.

Whatever number you come up with, it always will be higher than the highest projected win total for any individual team. Most likely, it also will be wrong.

References & Resources


Adam Dorhauer grew up a third-generation Cardinals fan in Missouri, and now lives in Ohio. His writing on baseball focuses on the history of the game, as well as statistical concepts as they apply to baseball. Visit his website, 3-D Baseball.
11 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Dennis Bedard
9 years ago

What about the 1973 Mets? They won their division with 83 wins and took the A’s to 7 games. Does that count?

JP
9 years ago

is there a winners (or losers) bias in actual results? Do teams with higher expectations outperform the mean on average because they make moves in season to win more often(and vice versa) ?

Calvin Liu
9 years ago

Not a very satisfactory answer.
For one thing – if in fact the projections are based on “most likely” results – what is the accuracy level of the projections vs actual record?
Are they 50% accurate? 50% accurate within 1 game? 2 games?
Merely saying a statistical projection system is projecting to a “most likely” result doesn’t communicate any reinforcement that the projection systems have any “skill” – which is to say, projections having some resemblance to reality.
This is redoubled because we don’t have unconstrained outcomes. A team CANNOT win more than 162 games nor lose more than 162 games with the statistical record much more constrained at plus minus 120. Sure, a team that outperforms its projections will affect other teams, but it won’t affect ALL other teams, and it is a fallacy to assume that there cannot be offset by an opposite performance.
Equally the nature of a “most likely” means there is some range to the “most likely” performance – hence the note above about how well the projection systems compare to reality in 1 game swing increments.

AC
9 years ago
Reply to  Calvin Liu

When you roll a pair of 6-sided dice, 7 is the most likely outcome, but it only happens one time in six. You’re far more likely to roll “not a 7” than you are a 7, but if you have to nail down one number to project as the outcome, it’s the best bet.

You could, perhaps, go look at the last few years of projections for the different systems, and see how far each were off to get an idea of what the accuracy is, but there are a lot of moving parts that no projection system can foresee. Trade, catastrophic injuries, etc. At the best of times, the system is saying that right now (pre-season) Team X has roughly Y wins worth of talent, given a certain set of parameters (playing time, mainly).

Calvin Liu
9 years ago
Reply to  AC

That’s exactly the point: while a short term rolling of dice may show a ‘7’ less than the statistical average, there will be a convergence towards that number over time.
Is this occurring with projections?
If not, does it mean that the projections are fundamentally not useful?
After all, anyone can predict the weather by simply saying that tomorrow will be the same as today. From a percentage basis, it will be correct most of the time – but it will never improve because there is no actual skill involved.

BMarkham1
8 years ago
Reply to  Calvin Liu

Man, you must be helpless. This article is not about how accurate the projections are, you can find that elsewhere. As a spoiler though I’ll let you know that they’re more accurate than anything else we have. What this article is about (and explained brilliantly I might add) is how to think about the projected standings and what they actually mean, and why they’re called PROJECTIONS and not PREDICTIONS, and why no system could accurately PREDICT every team’s record because of the large levels of inherent variance involved in the wonderful world of baseball.

What the projections give us is the mean, the average outcome. It’s amazing really that the projections can even come close because of all the random variation:

1) player injuries
2) actual changes in talent level
3) BABIP and HR/FB luck
4) sequencing offensive outcomes into runs (BaseRuns)
5) sequencing defensive outcomes into runs saved (DRS and UZR)
6) sequencing pitching outcomes into runs saved (LOB%)
7) sequencing runs scored and runs surrenders into wins (Pythag)

As you see there are many layers to the onion of baseball variance. That is why no projections system can ever be that close to exact.

John DiFool
9 years ago

Bill James did a study on this years ago, in the old “unofficial” available only by mailorder 1990 Abstract.

He found that division winning teams tend to exceed their “true talent” (he simply gave each team a winning % for team quality) by around 6-8 games or so. I replicated that BTW using an old Basic compiler.

TAngotiger
9 years ago

Fantastic piece!

AC
9 years ago

Excellent.
I’ve thought for a long time that projected win totals should include error bars right along with them. Perhaps at the 1 SD mark on either side of the number.

Take two hypothetical teams, projected at 86 and 87 wins. Since not every team is going to have the same distribution of possible outcomes, what this might actually mean is 86 ±8 and 87 ±5.

You’d get the same idea that team 2 has the higher mean (or median) projection, but team 1 has higher variability and the associated potential for breakout or collapse.

This would also do good in terms of exactly what this article is explaining: getting people to pay a little less attention to a single number rather than a range of possibilities. Naturally, some teams will continue to fall outside the given range, but it may be an easier sell to the less math savvy to explain why a team was outside that projection. Since the differences are bigger, there is more likely to be an obvious hero/goat.

pft
9 years ago

Maybe if the guys who did the projections reported like they do in the sciences. You know, 85 +/-6. Quantifying the uncertainty in projections or estimators is something I have been clamoring for.

PackBob
9 years ago
Reply to  pft

There are different ways to show this that would be more informative than a single, bald number. One would be to give a range of wins with a probability attached: say, for Detroit, a 30% chance at 87-91 wins, and for Cleveland a 30% chance at 86-90 wins (or maybe a 29% chance at 87-91 wins). Giving a single number makes it seem like it is projected standings since that is how standings are represented.

If writers think that a more accurate and detailed description of what the numbers mean will encumber the reader and send them somewhere else, add a more descriptive section at the end of the article for those interested.

It’s strange that those developing the metrics strive for rigorous results which are then presented in the simplest, and least rigorous, form possible. If researchers presented their statistical analysis results in this fashion to a publication, they would not be published.