Last February, we released our first real pre-season book, The Hardball Times Season Preview 2008. Though I am especially proud of the fantastic team essays and player commentary in the book, I know that a lot of people’s interest lies in the projections published in the Season Preview, and now that the season is over, some of you have probably been wondering how they stacked up against other major systems.
Well, I’ve gathered the numbers, and I can tell you right off the bat that we did pretty well. Today, we’ll compare The Hardball Times projections to the three best publicly-available projection systems I know of: Baseball Prospectus’s PECOTA, Sean Smith’s CHONE, and Dan Szymborski’s ZiPS. All these systems are designed by very smart people, and if we can measure up, I’ll be very happy.
The versions I used of these projections were those published by early February, which is when the Season Preview was released and fantasy drafts generally start happening.
Let’s start with hitter projections. I selected all players with at least 200 plate appearances in 2008 that were projected by all four systems, a total of 329. I could have tried to do something with every player in the major leagues, but that would have involved a lot more work and probably changed the results very little.
The first, most basic, and perhaps most important test is to find each system’s average error for those players. To do that, I compared each player’s projected OPS to his actual OPS, adjusting the numbers so that averages were equal. So if we had three players with OPSs of .800, .850, and .900, and their projections had been .825, .875, and .925, the projections would each be adjusted down 25 points, as they are on average 25 points higher, and the average error would be 0. Here are the four systems ranked by average error:
We start off on a good note for THT, which had the most accurate hitter projections of the year. Now what happens if we restrict our sample to the hitters the systems disagreed on most? I calculated the coefficient of variation for each player’s projections, and selected the 65 hitters (or about 20 percent) with the most varied projections. Here’s how the systems stacked up:
CHONE wins this round, though the top-three systems are very close. ZiPS lags clearly behind. The player the systems most disagreed on, by the way, was Blake Dewitt. Here were his projections:
CHONE: .656 OPS
PECOTA: .698 OPS
THT: .625 OPS
ZiPS: .556 OPS
ZiPS is the big outlier here, and Dewitt ended up outperforming everyone’s expectations, posting a .728 OPS on the year.
We can also split players up by age. What if we look at how the projection systems did predicting the numbers for young and old hitters? Note that young hitters are defined here as those 24 years or younger in 2008, while old hitters are 34 years old or more. Both categories encompass a little less than 20 percent of qualified hitters.
Here are the average errors for young hitters:
CHONE wins another category, though I’m quite happy to come in front of PECOTA and ZiPS in projecting this difficult-to-gauge group. What if we look at the old guys?
As group, older hitters were much easier to project, and THT and CHONE end up at the top.
Let us now turn our attention to pitchers. The process I followed for pitchers was much the same as what I did for hitters, except instead of OPS I used ERA, and my minimum baseline was 50 innings pitched in 2008. Let’s start off with the overall results:
PECOTA has always been known for being particularly good with pitchers, and this year was no exception, as it ever-so-slightly beat out THT for first place. Something crazy, however, happens if we look only at the 55 (roughly 20 percent of the whole sample) pitchers on whom the systems disagreed the most:
PECOTA really shines here, and I’m going to examine these results in more details to see just where our system can be improved. Jonathan Papelbon, by the way, was the pitcher on which the systems disagreed most, with his projections ranging from very good (THT) to otherworldly (ZiPS).
Let’s take a look at young and old pitchers now, with the definitions of young and old slightly tweaked so that each group includes about 20 percent of all qualified pitchers. Young pitchers are those who are 23 years of age or young; older pitchers must be at least 35. Let’s start with the young:
I’m happy to say that THT did the best out of these systems in projecting young pitchers, since that is a group that is generally very difficult to project. Here are the results for older pitchers:
And here we see another slight victory for PECOTA.
Overall, I am very happy with the results for THT. We did better than anyone else in projecting hitters, and were a close second with pitchers. Does that necessarily make our system better than any other? I don’t think so, but we definitely are at least as good as any other projection system.
Moreover, our goal is not just to correctly project a player’s OPS or ERA—though that’s part of it—but also to give you numbers that other systems don’t even bother with, like projected defensive ratings based on play-by-play data, projected depth charts, and fantasy dollar values.
If those are things that interest you, or if you just want to read great writing from the best baseball bloggers on the internet, make sure to pre-order The Hardball Times Season Preview 2009 today.