Monday, August 17, 2009
Measuring the umpire’s effect on the game
Posted by Nick SteinerWe all know umpires make mistakes, especially when calling balls and strikes. While some people will argue that those mistakes are part of the game, there are few who are able to make a convincing argument to support that. On the other hand, the blogosphere is littered with arguments against umpires and for a computerized zone.
I'm not here to take sides (I'm actually firmly in the camp for human umpires), instead, I wanted to take a look at how much of an impact umpires actually have. First, I wanted to see how accurate umpires are. To do that, I went into my Pitch f/x data and grabbed all pitches that were called strikes and all pitches that were called balls. Then, I mapped them out onto an approximation of a major league strike zone.
I used the average top and bottom hitter zones provided by Gameday, 1.6 and 3.4 feet above ground respectively, as the vertical ends of the strike zone. Than I used the official major league horizontal zone (17 inches) and added two inches of leeway to each side. I also normalized the vertical position of each pitch ball to batter height.
Here is what I got:
Remember that this is from the catchers point of view.
As you can see, there is significant overlap. While umpires are pretty good at judging the high end of the strike zone, they are absolutely dreadful at judging the bottom and the sides, especially the third base side. Overall, 9.1% of pitches that were called balls were inside of the strike zone, and 21.7% of pitches that were called strikes were outside of the strike zone. I find that second figure outstanding, especially given that I am already giving the umps a pretty lenient strike zone.
If I change the perimeters of the zone to 2 feet both ways, than those percentages become 16.5% and 11.6% respectively. So, assuming that the umpires have no bias towards hitters or pitchers, the "real" zone is likely somewhere in between that.
John Walsh already did some great work a couple of years ago on figuring out the "real" strike zone, and I may try to update that later having the benefit of more accurate Pitch f/x data. However, for now, I wanted to take a look at this from another angle.
Those percentages I quoted above are huge numbers. Any way you swing it, it appears that the umpires are only about 85% accurate, at least this year. That leaves a lot of room for random variation among players. How much? Well, let's find out.
I queried all pitchers this year who have thrown at least 500 pitch in baseball this year, 329 in total, and sorted each pitcher by the number of pitches called strikes that were outside of the strike zone minus the number of pitches called balls that were inside of the strike zone. Then I divided by total pitches to get it on a rate stat. Then I multiplied that by 100 pitches, or roughly one game, and named that "Gift Rate". Here are the results shown graphically:
In case it isn't clear, the x axis is all pitchers who've thrown at least 500 pitches this year.
You can read that as the number of "gifts" minus the number of "squeezes" each pitcher receives per game. You can see that despite the old adage, it does not all even out. Some of that may be due to measurement error, as I don't profess my strike zone to be very thorough and there still may be problems with the Pitch f/x data (namely park effects), and there may be some sampling error as well; however, it's clear that umpires effect some pitchers more than others.
The standard deviation of Gift Rate among pitchers this year is about 1.6, which means that 68% of pitchers will have up to 1.5% difference in their strike rate based on umpires alone. That may not sound like a lot, but consider that, based off of this years data alone, there is an R^2 of about .62 on strike% vs. BB/9. The average difference in walk rate among guys with a 1.6% difference in their strike% is about .4 which is pretty significant.
Going by the FIP formula, if you added .4 and subtracted walks per 9 to a league average pitcher, their FIP would rise by about .20 points. Obviously this doens't consider how strike% effects K Rate, and other factors. So in order to get more actionable numbers, a more rigorous study needs to be applied. However, it serves a reasonable illustration of the impact that umpires can have.
Now, here are the pitchers who have been getting the biggest help this year:
1) Derek Lowe: 5.6
2) David Weathers: 5.5
3) Javier Vazquez: 5.1
4) Mariano Rivera: 5.0
5) Livan Hernandez: 4.9
And here are the guys who have been hurt the most:
1) Dontrelle Willis: -3.6
2) Brandon League: -3.4
3) Charlie Morton: -3.2
4) Ryan Rowland-Smith: -3.1
5) Dana Eveland: -3.0
It's hard to see any sort of bias in those lists. Among the leaders, you have two of the best pitcher in baseball (Mariano and Vazquez) and two of the worst (Hernandez and Weathers). The trailers are filled with guys with abysmal control, like Willis, and guys with good control, like League. For those who want it, here is the complete list (pitchers are labeled by their Elias ID and my SQL is acting wonky right now, so you'll have to do some translating).
The next step, along with creating a more accurate strike zone, is finding how much of an impact those missed calls have. We all know that there a certain missed calls more significant than others; however, as I showed earlier, the potential impact of a lost or gained strike may be pretty significant in itself.
We use FIP, tRA and other such metrics to eliminate defense and other kinds of luck from pitcher ability. However, it's possible that umpires themselves may have as big, if not more, of an effect.
Nick is a Cardinals fan living in LA. He also writes for Viva El Birdos. He welcomes questions and comments via email
Given the placement of some of those dots either
a) There is measurement error, or
b) Some umpires should be fired immediately
Good work.