A treatise on true talentby David Gassko
July 30, 2009
Was Neifi Perez a better hitter than Barry Bonds? It seems an unlikely proposition, but is it impossible? That has been the topic of discussion in an interesting (and very long) recent thread at Baseball Think Factory. In the thread, Tom Tango, a respected statistical researcher and a consultant to the Seattle Mariners, argues that “[Barry] Bonds 100 percent performed better than Neifi [Perez]. We are 99.9999 percent sure that he was in fact better than Neifi.” In other words, Tango is arguing that even knowing the paths of Bonds’ and Perez’s careers, we cannot say with total certainty that Bonds’ “true talent” was greater than Perez’s.
Most of the posters on the thread disagree. This post is fairly representative: “I don’t think you can go around talking about a player’s ‘true talent’ as if it really exists…Consider how absurd this sounds when applied to other areas of human endeavor. Would you say that there is only 99.99(...)% chance that Mozart had more ‘true musical talent’ than, say, William Hung?” So where does the truth lie? To answer that, we must first understand what we mean by “true talent.”
True talent is an ephemeral term, but it can perhaps be best defined as our probabilistic expectations of a player’s output at a given point in time, given that we know everything there is to know about that player—which we never do. In other words, true talent is only something we can estimate. With a lot of data we can do it fairly precisely—after all if a player has hit a home run in five percent of his plate appearances each of the past few years, he will likely continue to hit a home run in roughly five percent of his plate appearances if nothing changes—but without knowing everything—how the player’s muscles are feeling, what he had for breakfast, whether or he has some song stuck in his head—we cannot precisely guess at his true talent.
Moreover, based on all those factors—and of course many others—a player’s true talent level changes from moment-to-moment. Ichiro may have a 30 percent chance of getting a hit in one at-bat, but if his jock strap starts to itch, perhaps that goes down to 29 percent the next. On the other hand, if someone in the dugout makes a funny joke that puts Ichiro in a good mood, his true talent could go up to 31 percent so long as that good mood lasts.
Because we can never know all these minutiae, we can only attempt to estimate a player’s true talent. Now with Ichiro, that happens to be a relatively easy task: As of this writing, Ichiro has had 6,358 plate appearances in the major leagues and gotten a hit in 1,953, or 30.7 percent. If Ichiro’s average true talent over his career is equal to his true talent today—an assumption that cannot safely be made for most players, but one which I am fairly comfortable making for Ichiro, at least in the interest of making our lives a bit simpler for the time being—that would mean that our best estimate of Ichiro’s odds of getting a hit the next time he comes to plate would be 30.7 percent.
Actually, that is not exactly true, and this is where the second point of contention and confusion comes into play. The issue is that even if we are sure that Ichiro’s true talent today is equal to his average true talent throughout his major league career, even 6,358 plate appearances is still just a sample of Ichiro’s ability. True, it is a very large sample, but a sample nonetheless.
Let’s say, for example, that I asked you to predict the odds that a batter will get a hit in his next plate appearance, but the only information you had was about his last plate appearance, in which he got a hit. What odds would you estimate? Well, knowing nothing about the guy other than that he is a major league hitter and that he got a hit his last time at-bat, you would be best-off hedging and guessing that he was around average. The average player gets a hit in 23.5 percent of his plate appearances, so your best bet would be to guess that his odds are 23.5. Actually, though one plate appearance is very little information, it is some, so the best guess you could give me would be closer to 23.7 percent.
Okay, now what if I told you that the player had gathered 40 hits in his past 100 plate appearances—what would you guess his odds of getting a hit were now? You certainly wouldn’t say 23.5 or 23.7 percent—after all, a guy who can gather 40 hits in 100 times at-bat is probably a pretty good hitter. Sure, he could be a poor hitter who got lucky—every year, some hapless batter goes on a huge hot streak before returning back to earth—or he could be an average hitter who’s had a little luck, but more likely than not, he handles a bat better than most major leaguers. I don’t know exactly what you would guess, but statistically you’d be best-off guessing something like 27.2 percent. Sounds reasonable, right?
So how is 6,358 different from one plate appearance or 100 plate appearances? Well, it’s obviously a much bigger number, but still it is just a number, and any number—no matter how big—is just a sample. Now, if all I told you about a given hitter was that he had gathered 1,953 hits in his past 6,358 plate appearance, what would be your best guess of our expectations for his next time at-bat? Well, if you guessed 30.7 percent, you certainly wouldn’t be far off, but you would be incorrect. The right answer is more like 30.3 percent.
How can that be? The important thing to remember is that statistics are just a sampling of an athlete’s true ability; actually, they’re less than that since that true ability constantly varies. But even if we forget about that variation, no number of plate appearances will tell us exactly how good that player is. At a trillion plate appearances, we might have to go out many, many decimal points before the player’s sample numbers and our best estimate of his true talent diverge, but eventually they would.
The reason for this is simple. Again, remember that all statistics know is what they show. If all we know about a player is that he (1) plays in the major leagues, and (2) got a hit in his last plate appearance, we have very little to distinguish him from every other major leaguer. The odds are roughly equal of his being above average or below, though it is ever-so-slightly more likely that he is above. That is why our best estimate of his true talent is 23.7 percent, versus an average of 23.5—that is essentially the weighted average of all his potential talent levels. What are the odds he’s Matt Stairs? What are the odds he’s a high school scrub? What are the odds he’s Ted Williams? Mario Mendoza? Take those odds, multiply them by the corresponding talent level, and add them all up, and you’ll get 23.7.
If all we know about the hitter is that he (1) plays in the major leagues, and (2) has gathered 40 hits in his last 100 plate appearances, then we still can’t be too sure about his talent level. Sure, most players who go on such tears are well above-average—but not all by any stretch. The odds that the player in question is Ted Williams (or a player of equivalent talent) are definitely higher than the odds that he is Mario Mendoza, but we can’t know for sure. Even Mendoza had some hot stretches of hitting in his career. So again, if we weight the odds of that player being at each given true talent level, we come to the conclusion that our best guess at his true odds of getting a hit are 27.2 percent.
So now we come to the player who has 1,953 hits in 6,358 plate appearances. It is true that even if all we know is this one piece of information (and of course that he is a hitter in the major leagues), we already know a lot about him. But we do not know all. We know that his true talent is almost certainly somewhere near 30.7 percent—the sample is too large for us not to be. His talent might be 30.6 percent or 30.8 percent, but it’s somewhere around there. However, there does exist a small possibility that even such a large sample has not given us the proper impression of the player’s talent. Now, this happens to be a very small possibility but it could be that this is the most talented hitter of all time and it could be that he is of merely average talent, and has simply gotten exceptionally lucky. The odds may be 1-in-a-million, but they are not zero.
Statistically, the next question to ask is, which is more likely? Well, we know that there are many average hitters out there—thousands in the history of the MLB—but by definition, only one man can be the greatest hitter of all-time (or, for that matter, the worst). A simpler way of putting this is that a major league player is much more likely to be about average than he is to be at an extreme, whether that extreme is greatness or mediocrity. (Note that technically this is not quite true, since there are obviously many more mediocre players than there are great hitters or even average ones. However, if you weight these things by playing time, playing time is distributed fairly normally, with about average players getting the most in aggregate while extreme players get much less—extremely bad players because major league teams try to avoid playing them, and extremely good ones because there are so few.)
Since our player has demonstrated well above-average performance, this means is that he is slightly more likely to be a worse hitter than his performance has thus far shown him to be than he is to be better—simply because there are more such hitters. If he had shown a below-average track record, conversely, our best guess would be that he was slightly better than he had played, even over 6,358 plate appearances.
Now, what this does not mean is that Ichiro’s 1,953 major league hits somehow do not all count, just because our best guess—only knowing his numbers—is that if Ichiro re-played all the games he’s played in his career all over again, he’d end up with 1,930 hits. The 1,953 hits are real, and what we estimate his true talent is does not affect that. Moreover, our estimate of his true talent has thus far been confined to only two facts: (1) He is a major league baseball player, and (2) In his career, Ichiro has gathered 1,953 hits and 6,358 plate appearances. If we knew how fast Ichiro was, how good his bat control was, and any other pertinent facts, we could get a better estimate of his true talent—if the average player with Ichiro’s speed and bat control gets a hit in 30.7 percent of his plate appearances, our best guess about Ichiro’s talent would also be 30.7. Statistics know only what you tell them.
Now, statistically we still have to quantify Ichiro’s pertinent traits, and we still have to calculate what hit probability those traits correspond to. To say that you’ve observed Ichiro being a good hitter so you know his true talent is 30.7 percent is not enough—the logic is circular and the thinking minimally rigorous. But I do want to make this point clear: The more information we have, the better and more exactly we can estimate how good a player is. Even that one player with one hit in one at-bat—if the scouts tell us he’s a superstar, his odds of getting a hit in the next at-bat are much better than 23.7 percent.
But in the end, Tango’s point is correct. We are 99.9 percent sure that Barry Bonds was a more talented hitter than Neifi Perez. We could, in fact, carry out the nine many more decimal places than that and the statement would still be accurate. But, if all we know is that Barry Bonds hit 762 home runs in his career whereas Perez hit 64, we cannot state with 100 percent certainty that Bonds was the more talented hitter. We know more than that—we know that Bonds had great pitch recognition while Perez did not; we know that Bonds was a big, strong guy while Perez was not; we know that Bonds could generate incredible power while Perez could not—but the numbers know only themselves. So unless we can quantify all of these intangibles, and moreover show that a Perez-type player could never (i.e., not 1-in-a-million, not even 1-in-a trillion, but the odds have to be exactly zero) be better than a Bonds-type, we cannot claim with 100 percent certainty that Bonds is the more talented hitter.
Now, because we can be 99.9 percent sure, this specific argument is really just pedantic, but it does shed useful light on the science of estimating a player’s talent, which is what every fan, general manager, and fantasy baseball player is always working hard to do.
David Gassko is a former consultant to a major league team. He welcomes comments via e-mail.