The death of superman

It was the forecast heard ’round the world: PECOTA forecast a .311/.395/.546 batting line for Orioles uber-prospect Matt Wieters in 2009. Essentially, that forecast would make Wieters one of the top 10-15 hitters in baseball right now. And that’s not accounting for the fact that he’s a catcher, too. It is, to put it lightly, the highest PECOTA has ever projected a rookie player.

This has inspired no small amount of running gags on the Internet, the culmination of which was surely, “Matt Wieters took batting practice this morning. There were no survivors.”

So, is Matt Wieters really one of the top five players in baseball right now? Let’s start by looking at what Wieters did last year.

His pro debut

Nobody denies that Wieters had an impressive season at the dish last year, his first season in the minors after the Orioles drafted him out of Georgia Tech. After hitting .345/.446/.576 in 280 PAs at High-A Frederick of the Carolina League, he was promoted to Double-A, only to hit an even more impressive .365/.460/.625 at Bowie in the Eastern League.

But we know that those numbers are from playing against leagues that aren’t as tough as the major leagues. Wieters’ PECOTA card also helpfully lists equivelents for his minor league production, based upon the Davenport Translations.

These translations, similar to Major League Equivalencies, are just as robust looking as the projection itself. The card gives us a translated batting line of .301/.396/.513 for his High-A performance, and a translated batting line of .349/.436/.627 for his Double-A—and yes, you are reading that correctly, his translated line says he would have had a higher slugging percentage playing in the majors than he did at Double-A!

Those figures are unbelievable, by which I mean literally that I do not believe them. Consider that the listed Equivelent Averages, Baseball Prospectus’ key rate stat for offense, essentially a rate version of linear weights, average out to a .330 EqA in 530 appearances. In other words, the suggestion is that if the Orioles hadn’t sent Wieters to the minors at all, the top hitters in the majors last season would have been:

Rank
Player
EqA
1.
Albert Pujols
0.372
2.
Chipper Jones
0.362
3.
Milton Bradley
0.341
4.
Lance Berkman
0.333
5.
Matt Wieters
0.330

In essence, the conclusion one gets from a literal reading of the Davenport Translations is that Wieters was one of the five-best hitters in all of baseball last year. You may be curious as to how they came to that conclusion; I certainly was.

How the Davenport Translations Work

The essential premise of the Davenport Translations is that you can compare players either historically (between eras) or between different leagues by placing them all in the same context. The crux of the matter is that each league, in each season, is assigned a “difficulty rating,” which measures how a player’s EqA would change if he moved between leagues.

In the case of translations for minor league players, these difficulty ratings are created by the average change in EqA of players who change levels. Since there are rarely players who go from High-A to the majors, chaining is used; to translate a player’s EqA from Double-A to the majors, it is first translated into a Triple-A EqA, based upon how players performed going from Double-A to Triple-A (and vice versa). It is then further translated into a Major League batting line by looking at players who moved from Triple-A to the majors (and vice versa).

Here are my estimates of the league difficulty ratings used by Baseball Prospectus for 2008, relative to their reference league of 1.00:

League
Level
Adj.
AL
Maj.
1.02
NL
Maj.
1.01
Eastern
AA
0.96
Int’l
AAA
0.94
PCL
AAA
0.93
Carolina
A+
0.91
Southern
AA
0.87
Texas
AA
0.87
California
A+
0.86
Florida State
A+
0.85

In order to convert between levels, simply take a player’s EqA and multiply it by the ratio of the league he is leaving to the league he is going to. A player with a .300 EqA at the Southern League, for example, would end up with a translated .256 EqA in the AL—for the answer, take .300 * .87/1.02.

The bolded entries are the leagues Wieters played in in 2008. What jumps out is how the difficulty ratings for those leagues vastly outstrips the other leagues at that level. You could argue that, based upon these ratings, the Eastern League could either be considered a third Triple-A league or a third major league; the Carolina League might take the Eastern League’s place at Double-A or it may join Triple-A itself.

This is a new phenomenon—compare that chart to these estimates of BP’s difficulty ratings for 2007:

League
Level
Adj.
AL
Maj.
1.02
NL
Maj.
1.02
PCL
AAA
0.96
Int’l
AAA
0.95
Eastern
AA
0.92
Southern
AA
0.91
Texas
AA
0.91
Florida State
A+
0.87
California
A+
0.86
Carolina
A+
0.84

We’re talking about a substantial one-year jump in the difficulty ratings for the two leagues that Wieters played in. Can this be true?

Players jumping leagues

Let’s take a look at position players who played both in Double-A and Triple-A, and how their OBP and SLG changed between the two levels. And let’s divide them up by which Double-A league they played in. We’ll weight the result by at-bats, specifically the at-bats the player had in the league where he played the least. A player who had 1 at-bat in a league tells us little about the change in league quality, regardless of whether he had 50 or 500 at-bats in the other league. A player with 200 at-bats in both leagues tells us quite a bit more.

League
AB
AA_OBP
AA_SLG
AAA_OBP
AAA_SLG
OBP%
SLG%
Eastern
5894
0.345
0.419
0.320
0.392
92.5%
93.6%
Southern
4446
0.353
0.428
0.323
0.388
91.7%
90.5%
Texas
4092
0.358
0.432
0.324
0.392
90.5%
90.7%

The first thing to note is that the Eastern League players, while retaining more of their production in Triple-A than their counterparts, still lost production. This is not what we would expect if BP’s difficulty ratings were correct. In fact, all of the Double-A leagues look pretty similar to each other in level of difficulty. If we were constructing our own MLEs, at this point we’d be interested in looking at how park effects could change these figures, and we’d also be interested in breaking it down further than just OBP and SLG. But for our purposes right now, this should serve just fine, as there’s little to suggest that the Eastern League is radically more difficult than the other Double-A leagues.

But let’s look at this from another angle, just to be sure. If the Eastern League had a significant boost in quality, we would expect repeaters (players who played in the league in 2007 and 2008) to decline. Our normal expectation would be for those players to improve, because they’re a year older and more more experienced, and that should show up in their minor league numbers. In 16,490 at-bats, here’s what we see:

Year
AVG
OBP
SLG
2007
0.259
0.327
0.383
2008
0.263
0.336
0.406
Change
101.5%
102.6%
106.0%

Our repeaters see an improvement, again not what we’d expect to see if BP’s difficulty ratings were accurate. What about our Carolina League repeaters?

Year
AVG
OBP
SLG
2007
0.255
0.332
0.382
2008
0.252
0.326
0.392
Change
98.7%
98.1%
102.7%

Again, no significant change between years – a slight decrease in on-base skill, a slight increase in extra-base hits. Not at all what we would see if the Carolina League suddenly shifted from being a High-A and a Triple-A league in quality.

Should we try to adjust for league and park? Why not? Let’s look at BP’s “real stats” EqA, which is adjusted for park and league average but not league difficulty. Players in the Eastern League who were promoted to Triple-A saw, on average, a drop from a .270 EqA to a .244 EqA in 5,748 at-bats. Meanwhile, players in the Carolina League who were promoted to Double-A saw, on average, a drop from a .274 EqA to a .252 EqA in 3,186 at-bats.

It’s not for me to tell you where Baseball Prospectus came up with this year’s difficulty ratings – quite frankly, I can’t figure it out. What I can tell you is that they don’t appear to be supported by the data itself. To put it bluntly – they’re wrong.

Making sense of it all

Another good question is why Baseball Prospectus, in fact, touts the forecast, like Steven Goldman does here:

The key for Wieters is just how well he makes out, despite the discounts on his performance wrought by these various translations. It is the rare young player who towers above his league to the degree that Wieters did last season and then goes further by making the difficult jump to Double-A, not only maintaining production, but actually accelerating it. It is the rare 22-year-old who bats .345/.448/.576 in High-A ball, let alone .365/.460/.625 at Double-A. No matter how aggressively you reduce Wieters’ numbers through translation, you simply can’t make all of that go away.

Any young player looking to receive a Wieters-style PECOTA projection and become an instant fantasy (and real-world) darling should follow a simple formula. First, dominate your league. Second, be the appropriate age for your league; PECOTA is properly skeptical of four-year college players who beat up on high school pitchers as 24-year-old players in short-season rookie ball. Third, do not rely too much on your ballpark to help you. Oh, and if it’s not too much trouble, fourth, do your best to show no weaknesses in any phase of your game. Just like Wieters did last season.

Injury is a particular risk for a young catcher, as an errant foul tip can mangle a finger and a contact play at the plate can mangle an entire body. What makes Wieters’ projection all the more impressive is that PECOTA is aware of the toll catching can take on a backstop’s offensive skills, and yet it still sees such impressive short-term results for the Orioles tyro.

No, you can’t make all of his minor-league performance go away through translation, but you can certainly go further than BP did and still call yourself a member of the reality-based community. It’s the second point that I’m more interested in now: Is PECOTA really accounting for the riggors of catching?

It helps to look at exactly what PECOTA is doing. Like almost all projection systems, it follows a two-step process:

  1. Produce a baseline forecast, using a weighted average of the past several seasons and some regression to the mean.
  2. Adjust for the effect of aging.

PECOTA is certainly more complex than other forecasts, but the two-step model still mostly holds true.

We already have reason to think that step one went awry here. Given the inflated inputs, we would expect an inflated output. But what about step two?

PECOTA differs significantly from other projection systems in that each player is given their own aging curve, called a “career path adjustment,” by looking at the performance of similar players. But who is similar to a 22-year-old catcher that hits like Lance Berkman in a good year? Check the similarity index on Wieters’ PECOTA card for the answer: It’s a big fat zero. There’s nobody very similar to Wieters, at least not if you’re looking at his inflated translation for guidance.

This shows up in his list of top comps:

Rank
Hitter
Year
Age
Pos
PA
wOBA
Score
1
Mark Teixeira
2003
23
1B
594
0.349
35
2
J.D. Drew
1999
23
CF
430
0.350
30
3
Alex Gordon
2007
23
3B
604
0.325
30
4
Ben Grieve
1999
23
LF
560
0.368
30
5
Joe Mauer
2006
23
C
629
0.398
29
6
Travis Lee
1998
23
1B
635
0.345
27
7
Austin Kearns
2003
23
RF
339
0.366
27
8
Ryan Zimmerman
2007
22
3B
725
0.342
27
9
Joe Borchard
2002
23
LF
37
0.273
24
10
Evan Longoria
2008
22
3B
512
0.372
22
11
Jeff Burroughs
1974
23
RF
674
0.404
22
12
Darryl Strawberry
1985
23
OF
483
0.427
21
13
Michael Cuddyer
2002
23
RF
123
0.323
21
14
Brad Komminsk
1984
23
OF
334
0.299
21
15
Ken Griffey
1993
23
OF
716
0.438
19
16
Bob Horner
1980
22
3B
498
0.367
19
17
Eddie Murray
1979
23
1B
696
0.376
18
18
Prince Fielder
2007
23
1B
702
0.416
17
19
Tom Brunansky
1983
22
OF
615
0.338
16
20
Albert Pujols
2003
23
LF
697
0.460
16

wOBA is a rate measure of offense, on the scale of on-base percentage; .340 or so is typically average. “Score” refers to the similarity score assigned by PECOTA, where 100 is identical and 0 means not similar at all.

Only one of those players was a catcher. Most of them are corner types, either in the infield or in the outfield. The effect is modest in the short run, judging by the graphs on Wieters’ PECOTA card. But it could be a major factor in why PECOTA doesn’t see Wieters’ odds of being a superstar drop below 65% in any season in the next seven years, rather bullish given the increased stress on a catcher noted above.

But mostly I bring it up because it’s simply breathtaking how someone at BP can write an article with claims like that, when looking at the PECOTA card itself tells a very different story.

So what’s a good projection for Wieters?

If you’re a fantasy owner or an Orioles fan (or even just a curious onlooker) and you can’t trust PECOTA, what can you expect Wieters to do next season? Let’s look at what four other projection systems have to say:

AVG
OBP
SLG
THT
.286
.396
.491
CHONE
0.274
0.352
0.439
ZiPS
0.291
0.361
0.467
Oliver
0.294
0.373
0.487

He still projects as a very good hitter, especially for a catcher. The point of this exercise isn’t to take a way from Wieters, it’s simply to get the most accurate look at his performance possible.

One last note: I’ve spent the entire article discussing Matt Wieters’ forecast, so I just want to reemphasize that this applies to any player’s forecast that played in the Eastern or Carolina Leagues in 2008. The effect on his forecast is simply the most pronounced, because this included the entirety of his professional career.

References & Resources
All minor league data in the article comes from Fangraphs.

Estimates of difficulty ratings for major leagues are made by comparing the “adjusted for season” and “adjusted for all time” Equivelent Average values on the Davenport Translation player cards. Estimates of difficulty ratings for minor leagues in 2008 are taken by comparing the Equivelent Averages from the “real statistics” pages on the minor league translations with the values on the PECOTA player cards. Estimates of difficulty ratings for minor leagues in 2007 were taken from comparing the regular translation EqAs to the real stats EqAs.

A note on difficulty ratings for minor league players – the Davenport Translations dock a player for being too old for his level. In other words, a 27-year-old at Double-A will have a different adjustment than a 22-year-old. The values presented are based upon players of the appropriate age for their level. This seems to lead to a smaller drop in production that what we would expect from a translation factor – looking at all players presents a more modest looking set of league difficulty factors. (These values are not presented due to the difficulty in obtaining enough data to present both charts for comparison.) The upshot – young players, like Wieters, are aparently presented with a more optimistic set of translations than older players.

For further reading on the Davenport Translations and PECOTA, look for Baseball Prospectus’ Baseball Between The Numbers. Also helpful in preparing this article were the Baseball Prospectus 2001 and Baseball Prospectus 2003 editions of their annual. Also helpful were the following articles:

Whenever I mention EqA, I feel obligated to link to Patriot’s critique of the metric. The salient point for the way the EqA is used in the Davenport Translations is that since EqA is not linear, the percentage change of EqA between levels doesn’t tell us the same thing for different players – in other words, if two players both change levels, and their new EqAs both are 80% of their old EqAs, those players did not lose the same amount of production.

If you are up for some dense, technical reading on the subject of MLEs, I cannot recommend this discussion enough.

This article was greatly aided by conversations with Dan Szymborski, Sean Smith, Brian Cartwright, Matt Swartz, Kevin Goldstein and Eric Seidman. Thanks to all of them. Also, thanks to the Lounge for putting up with my rantings on this topic for this long.


Comments are closed.