Moneyball and Efficient Efficiencyby JC Bradbury
July 26, 2005
Yes, this is another Moneyball article. But it's not a rehash of who's right, who's wrong, and let's just hold hands in the mushy middle. I'm as sick of those articles as everyone else. Sure, maybe we need more of this analysis, but I think we deserve a break right now. What I want to do is add a little tangent to the debate that has escalated since the publication of Moneyball.
One of the central lessons of Moneyball is this: to get the most output from your inputs in order to maximize the return on the dollars that your organization spends on running team. A GM must be efficient in running his organization. In economic terms, he's attempting to put all of his resources to their most highly valued uses. If the market overvalues a particular baseball talent–for example, saves–then a team should liquidate its assets in this area. If the market undervalues a talent–for example, OBP–you acquire it while it's cheap. It's all very simple in theory, but difficult in practice.
This is clearly something all GMs try to do, and we're told that Oakland A's GM Billy Beane excels at it. Whether it's hitting, pitching, fielding, aging, speed, or intelligence; the A's can value it. And once the values are in place, Beane is able to do the wheeling and dealing to make sure the club prospers from this knowledge. As a result, the A's win. Moneyball, the philosophy, involves properly allocating scarce resources, explaining the book's popularity among economists–and the A's play it better than anyone.
But, there is one aspect of in the book that often gets overlooked. It deals with efficiency, but a type of efficiency different from the type commonly attributed to the book. Efficiency is also a term statisticians use to judge what are known as estimators. Estimators are theoretical tools for predicting a numerical estimate from a sample of data. Rather than being a number, an estimator as a method of predicting an estimate; it's a function of the data.
There are lots of different methods we could chose predict estimates from a group of data. For example, let's say I was asked to predict the SAT score of a randomly selected student at a university. I would have many tools at my disposal. I could ask the next student who walked down the street what his score was, and go with that score. Or I could go with the mean or the median of the student population. These choices represent estimators that will generate a prediction.
Statisticians try to determine properties of competing estimators that will minimize the mistakes of estimates. To me, this sounds a lot like the "stat-heads versus scouts" debate that has raged since the book's publication. It's an argument over estimators, except the estimators are not necessarily purely known mathematical functions of the data. Yet, each camp makes predictions based about the same population of players based on these different estimators.
In Moneyball, Lewis pits traditional scouting methods of personal observation against performance scouting via statistics. In reality, no matter which way it leans on the traditional/performance scouting spectrum, no organization completely ignores the other method. Sabermetric clubs have plenty of guys with left-arm only tans from driving, just as the traditional clubs have pasty-white computer nerds stashed in their basements. So, the actual debate is over the slant or preference each organization has on the scouting spectrum. Lewis portrays the push toward sabermetric methodology to identify talent as reason for the A's success in winning on a tight budget.
It's tempting to say the method the A's employ—of which sabermetrics play a large role—is superior to all other methods, and that's why A's have been winning as of late. This upsets a lot of people. After all, haven't teams been successful without employing sabermetric methods? There is no arguing that Beane and DePodesta used a sabermetric mind-set to stay ahead of their competitors.
But, in looking at the success of baseball teams during Oakland's recent run of success, the A's clearly aren't the only team that's been winning, even in terms of their limited budget. Below is a list of teams ranked on total budgets as a percent above/below the league average payroll and the number of playoff appearances by team for the past five seasons.
Rank Team Payroll Playoff Rank Team Payroll Playoff 1 FLO -44.91% 1 16 BAL -1.21% 0 2 KC -41.92% 0 17 COL -0.59% 0 3 MON -40.74% 0 18 LAA 1.08% 2 4 MIN -40.27% 3 19 HOU 2.78% 2 5 MIL -40.27% 0 20 CHC 11.11% 1 6 PIT -34.91% 0 21 SF 11.36% 3 7 TB -34.53% 0 22 STL 16.04% 5 8 OAK -32.61% 4 23 SEA 21.19% 2 9 SD -28.81% 0 24 TEX 25.57% 0 10 CIN -24.57% 0 25 ARI 26.08% 2 11 DET -17.10% 0 26 ATL 38.22% 5 12 CH -15.36% 1 27 BOS 49.39% 2 13 PHI -5.05% 0 28 NYM 50.39% 1 14 TOR -4.93% 0 29 LAD 57.84% 1 15 CLE -1.66% 1 30 NYY 94.13% 5
But the message of Moneyball isn't that statistical analysis is superior to all other methods in projecting major league talent, either. It's a method that's successful because the teams that have employed it have been using it with great success on a particular subset of talent that includes: major leaguers, minor leaguers, and college players. In particular, the Moneyball GMs have been criticized and praised heavily for their draft strategy of focusing on college players. I believe the focus on college players has to do with the usefulness of the estimators these clubs have chosen to use to evaluate talent. Playing Moneyball is not just about finding bargains, but it's also a method (a set of estimators) for finding bargains. It just so happens that the statistics-driven method of evaluating talent is more efficient, in a statistical sense, than old methods in evaluating college players.
Estimators are judged on two properties: bias and consistency. First, we want our estimator to be unbiased–that is not consistently above or below the true value we're estimating. Second, we want to minimize the size of any predicting mistakes; a quality known as consistency. When we choose from several unbiased estimators to predict a true outcome from population, we want the one that minimizes the variance of the errors of the prediction. Fewer and smaller mistakes are preferred to more and larger mistakes, right? In the language of statistics, the estimator with the greatest consistency, or smallest variance, is said to be the most efficient of all estimators. In this sense, being efficient means making fewer and smaller mistakes compared to other available estimation methods.
The figure below shows a the variance of estimates of a pitching prospect's true talent using two hypothetical methods, A and B. Both methods estimate the player's talent to be the same. The difference is in the consistency. If we can predict a pitcher's true talent level with more accuracy (method A), then this will result in fewer mistakes in evaluating talent than with method B. These methods are estimators, and A is more efficient than B. This, in turn, increases the economic efficiency of the organization, because it now needs fewer resources to devote to scouting talent.
The A's focus on college players not because of a bias of stat-heads in thinking college players are superior to high school players; but because they are more predictable based on the statistical tools the A's favor. A technological innovation in performance scouting, such as DIPS, can increase the efficiency in evaluating talent. DIPS ERA can be used to better predict a college player, but maybe not a high-school player.
As a result, the A's are going to have a higher confidence when drafting from this talent pool. And if a technology can be employed in one area but not another, it's no surprise that the A's would concentrate on a talent pool where this new technology is useful. Just as the cotton gin caused southern farmers to switch to cotton farming, where the technological innovation could be used, so too did the A's turn to the college talent pool where it's inventions were useful. This reason for this choice is well-documented in Lewis's book.
From Paul's point of view, that was the great thing about college players: they had meaningful stats. They played a lot more games, against stiffer competition, than high school players. The sample size of their relevant statistics was larger, and therefore a more accurate reflection of some underlying reality.
And, as it just so happens, the stat-head methodology happens to be a cheap operation to run if you do it right. Rather than scattering the country with scouts to report back on thousands of personal observations (from which personal impressions could be quantified), why not target those who exhibit qualities of successful ballplayers? You can afford to limit the talent pool you draw from the more efficient you are at evaluating it, because you will have more confidence in the predictions you make.
Other organizations that have been successful in evaluating talent through traditional methods have their own innovations for evaluating talent that have caused them to be successful on a small budget, too. Moneyball speaks to the success of the implementation of one methodology; a method that happened to come to baseball from the outside, which makes it interesting. And just as traditional scouting organizations would be wise to adopt innovations from sabermetric organizations, so too can stat-savvy clubs learn from the innovations in traditional scouting–innovations are innovations. And clubs that wish to win will shift their resources to take advantage of these new methods.
I guess my point is, I don't understand why Moneyball upsets so many people. And as much as I hate to admit it, maybe sociology or psychology can explain the reason for the backlash. The book is about one method, comprised of many complicated parts, that one team has used to win baseball games. It's possible that the A's recent successes are a product of dumb luck, and it's really too early to say it's not with any certainty. But, I don't believe it's luck. I find sabermetric methods to be powerful methods elicit many powerful truths about the game that the conventional wisdom seems to have missed. For this reason, I expect that organizations that employ these methods for evaluating talent will benefit from their use.
JC Bradbury runs the weblog Sabernomics: Economic Thinking about Baseball. You can contact him with your comments and questions via e-mail.