The Truth About Productive Outsby Larry Mahnken
May 05, 2004
In a parallel universe...
October 6, 2002
The Twins were three defensive outs away from toppling the 102-win Athletics today, taking a 2-1 lead into the ninth. Dustan Mohr drew a walk against right-hander Billy Koch, with lefty-hitting catcher A.J. Pierzynski coming up; Denny Hocking and Jacque Jones were to follow.
Twins' manager Ron Gardenhire could have had Pierzynski swing away and try for the big inning -- he was hitting .400 for the series -- but the Twins didn't necessarily require a big inning. One more run would have felt like 100 to the A's, with Eddie Guardado looming in the bottom of the inning. Gardenhire, a smart manager, knew this and signaled for his catcher to make the productive out, bunting Mohr over to second.
Hocking and Jones couldn't come through, and in the bottom of the ninth, disaster struck. Eric Chavez led off the bottom of the ninth with a single, Jermaine Dye forced him out at second, and after David Justice doubled Dye to third, second baseman Mark Ellis clubbed a dramatic, series-winning three-run homer.
Oakland went on to knock off the Angels in the ALCS, and win the World Series over their cross-bay rivals, the Giants, as chronicled in the bestselling book, "Moneyball".
After the game, Gardenhire was unapologetic about his decision to bunt the hot-hitting Pierzynski. "Yeah, the stats looked good, but they only tell you what's happened before, not what's going to happen. It's not like you could have expected him to hit a homer there or anything, Koch is a great closer. With the ninth hitter coming up, we wanted to avoid the double play and try to scratch out an insurance run. It didn't cost us the game, those guys over in that other clubhouse played hard and earned this win. I'm proud of my team."
Over in another corner of the clubhouse, Twins' owner Carl Pohlad muttered to reporters, "Give me another $50 million and we wouldn’t have lost today."
Good statistical analysis is not about disproving conventional wisdom. A good sabermetrician doesn't seek to prove that what they believe is true, they seek to discover what actually is true, and believe that.
A recent article by Buster Olney at ESPN.com epitomizes the exact opposite of that ideal. Olney writes about a new statistic being tracked by the Elias Sports Bureau: productive outs.
Productive outs are defined as advancing the runner with the first out of the inning, scoring a runner with the second out, or when a pitcher sacrifice bunts with the second out. Successfully making these types of outs, Olney posits, is an overlooked aspect of winning baseball games, particularly close ones.
Olney's assertion is backed by very little data, relying largely on the opinions of coaches, ballplayers and broadcasters, and accepting them as fact:
"That's how games are won and lost -- productive outs, advancing baserunners and getting guys in from third with less than two out," (Detroit Hitting Coach Bruce Fields) told John Lowe of the Detroit Free Press earlier this week.
Olney provides very little data, period, and what data he does provide is presented in a manner which will make the non-skeptical reader believe it supports him. The rate of productive outs is given for only 12 teams this season, the top six and bottom six. The top includes some teams that have surprised thus far, the bottom includes teams that have disappointed. The implication being that making or not making productive outs is the cause of their success or failure.
The only "Productive Out Percentage" numbers given for past years are the POP numbers for the Florida Marlins and Anaheim Angels last season, both of whom ranked in the top five. The implication is, of course, that making productive outs is the reason these teams won the last two World Series (over teams that currently rank in the bottom five).
Ignored is the fact that Florida's POP during the regular season last year is not particularly relevant to their postseason success, and that Anaheim's POP last season, when they finished 77-85, is not even close to being relevant to their postseason success in 2002.
It's clear that Olney did very little research for his article, and what research he did do was data mining, trying to find stats that supported his claims.
Because the data is compiled by the Elias "You'll Know What We Want You To Know" Sports Bureau, productive out data is impossible to find, making an independent study of regular season productive outs almost impossible. However, for the sake of discovering and spreading truth, rather than dogma, I did an independent study of the past two postseasons using the game logs available at Retrosheet. The study was long and tedious, but I believe the results were worth it.
The first thing that I discovered was that Olney had described Productive Out Percentage incorrectly. It's not a percentage of total outs, but rather outs made in productive out opportunities. Going through the game logs, I collected the total number of productive out opportunities for each team in each game, the number of productive outs they made in those opportunities, and the number of times they reached base in those opportunities.
I found that, indeed, Anaheim and Florida did very well in making their outs productive in the past two postseasons. Anaheim's POP in 2002 was .388, while Florida's was .369 in 2003. Both teams, however, ranked third in this category among all playoff teams each season, Anaheim behind St. Louis (.526) and San Francisco (.393) in 2002, Florida behind Atlanta (.500) and San Francisco (.450) in 2003.
What does this tell us? Nothing conclusive, only that the past two World Champions made productive outs at a good rate. It doesn't tell us that's why they won, and the fact that two teams with astonishingly great productive out rates were knocked out in the first round last season casts some doubt on that theory.
Did San Francisco and Atlanta lose because they didn't have very many productive out opportunities? No, Atlanta had 20 such opportunities in five games, and San Francisco had 36 opportunities in four games -- more than New York and Minnesota had combined for in their first-round series last year.
Atlanta only got on base 20% of the time in their opportunities, so that could explain their defeat, but San Francisco had an on-base percentage of .444 in those opportunities. The reasons for San Francisco's defeat can probably be found in their ability to drive home runners rather than just get on, and in making two-out hits, two things that go beyond the scope of this study.
A look at all teams shows that while productive outs tend to come at higher rates for teams that win, getting on base seems to be much more important. Winning teams over the past two postseasons had a .360 POP compared to the .301 POP of their opponents, a difference of .059. But winning teams also had a .364 OBP in those situations, compared to a .266 OBP for the losers, a difference of .099. Taking productive outs as a percentage of all opportunities, and not just outs made, you find that the difference between winning teams and losing team is only .008 -- .229 to .221.
This last statistic indicates that making productive outs is not an important part of winning ballgames. The correlation to winning percentage drives the nail in the coffin: POP has a .463 correlation to winning percentage, OBP in those situations has a .750 correlation, while the rate of productive outs has a mere .283 correlation.
Overall, OBP, SLG, OPS and GPA correlate even better: OBP -- .841, SLG -- .855, OPS -- .874, GPA -- .877. Of course, these are in very small samples, but if the strategy of making productive outs doesn't work in a small sample, then how is it a useful substitute to the "Moneyball" style of play, which emphasizes playing in a fashion that will be more effective over the long haul?
POP doesn't work for one-run games, either. Teams that win one-run games have a .348 POP compared to the .303 of their opponents, a .045 difference. But again, the OBP difference dwarfs it: .373 to .277. And perhaps most shocking of all, the rate of productive outs per opportunity for teams that win one-run games is .218, the rate for teams that lose is .219. That's right -- the team that makes a higher rate of productive outs is slightly more likely to lose.
There are many problems with productive outs. The first, and most obvious, is how it credits a one-out sac bunt as a productive out if it's by the pitcher, but not by any other player. An out by a pitcher that advances a runner is certainly more productive than one would normally expect, but to include it in a team statistic is ridiculous. If POP is supposed to help teams win, does that mean you should pinch-hit a pitcher every time you have a runner on and one out, so they can sac bunt?
Another major problem is that it does not indicate how many runs it creates. Though I didn't record the rates, in reading the game logs, I noticed that a very high percentage of productive outs did not create any runs, and a very high percentage of those that did were followed by hits that negated the usefulness of the productive out.
There is a very small value to tracking productive outs, if altered from it's current form, and using it as a value statistic, rather than a skill statistic, since a productive out is worth more than a non-productive one. However, it shouldn't be viewed as a shortcut to victory, or emphasized as a strategy. It doesn't work.
At the end of Olney's article, he cited Yankees' broadcasters Jim Kaat and Paul O'Neill regarding the Yankees and productive outs:
As club broadcasters Jim Kaat and Paul O'Neill noted last weekend, the team's offense is built much differently than in the championship years; in those seasons, the Yankees advanced runners, put runners in motion, bunted occasionally. While they didn't always have an overpowering offense -- the notable exception being the 125-win season of 1998 -- they had an efficient offense that provided the team's typically strong pitching enough runs to win.
Over the past two postseasons, the Yankees have had a POP of .310. From 1998-2000, when the Yankees won three consecutive championships, they had a POP of .268.
Larry Mahnken is a staff writer for The Hardball Times, and co-editor of the Replacement Level Yankees Weblog. You can contact him with your comments, questions, romantic propositions and incoherent rantings at DLMahnken@hardballtimes.com.