Print this Article
 E-mail this Article

The eye of the umpire

How accurate are umpires in calling the strike zone? How well can they locate a ball flashing towards them at 95 mph? Or unexpectedly swooping down and, perhaps, nicking the lower outside extremity of the strike zone? Two inches? One? One-tenth of an inch? Here's what Ted Williams wrote about his ability to judge where a pitched ball actually goes, from his book The Science of Hitting:

It's very likely that once you've made yourself sensitive to the strike zone, you'll be a little more conscious of what you think are bad calls by the umpire ... I would say umpires are capable of calling a ball within an inch of where it is. As a hitter, I felt I could tell within a half-inch.

Today I'm going to build on some work I did last time (Strike zone: fact vs. fiction) on determining the size of the strike zone using pitch data. As we'll see in a few moments, we can infer from that data how well an umpire can locate the incoming pitch. First, though, I want to go back and make some small improvements to the measurements of the strike zone that I did last time.

That was a ball?!?

One of the loose ends of that analysis was some question about the quality of the data. Here's a snippet from that article:

I've already mentioned the fact that the ball fraction for pitches right down the middle of the plate is not zero, in fact it's about 5-6%. Can umpires be missing these easy calls so frequently? It seems hard to believe. The alternative explanation is that there is some problem with the data.

After viewing some other pitches on video, it became clear that the MLB system for tracking pitches was just getting some pitches wrong. Of course, this shouldn't be surprising. This is a very complex system that is still in the course of being rolled out in all major league parks, we should not expect the data to be perfect. But, we do need to understand its limitations and see how it affects what we are trying to do with the data.

So, I have tried to determine how often the system mis-tracks a pitch. First, let's recall the ball fraction graphic I produced last time. This graph shows the fraction of balls called by the umpire as you move across the strike zone.

In fact, the sharpness of the ball-strike transition is a direct measure of the accuracy of the system, although it should be kept in mind that I'm referring to the pitch-tracking system and umpire pitch-locating ability combined. The graphic below shows how the ball fraction curve is modified for different accuracies. I generated these curves analytically using a simple model (see the Resource section for details).

Do any of these colored curves look like the real data shown above? Not really: the green or cyan curves seem to have the right shape in the transition region, but they do not show the non-zero ball fraction at the center. It turns out that no value of the accuracy number can reproduce what we see in the data. However, if I modify my model a bit, I can get this plot:

A big word of caution: I am not claiming that 5% of the pitches gathered thus far are mis-measured. Mine is just one hypothesis that happens to qualitatively describe the data, but it doesn't mean it's correct. My little model does not rule out other possibilities, it simply shows how one hypothesis is indeed plausible.

The main point here is that, while there is some small level of noise in the data, its presence doesn't affect our ability to measure the strike zone.

Calling the high strike, or not

After my previous article appeared there were lively discussions on the results both on Ballhype and over at The Book Blog. Sabermetrician Mitchel Lichtman was fairly (OK, very) certain that there was something wrong with my estimation of the vertical strike zone for right-handed batters. I had found the the umps were calling the high strike correctly, as shown in this plot (taken directly from my previous article):

In any case, there is NO WAY IN HECK that the average umpire calls a rule book strike at the top of the zone for RHB!!!!!!!!!!

...Something is wrong. I have watched 300 games a year for 20 years. The average top of the strike zone is well below the rule book. This is almost unequivocable.

My mistake was in reporting the size of the rulebook strike zone. I did not add in the radius of the ball to either end of the vertical strike zone as I had for the horizontal dimension. OK, that's easy to fix, but the second problem was more difficult to solve. It has to do with the MLB-supplied limits of the vertical strike zone.

The height of Jeter's knee

While the horizontal size of the strike zone is defined by the width of the plate and is the same for everybody, the vertical dimension of the zone is tied to each individual batter. A nice feature of the MLB pitch data, is that they include, for each pitch, their estimate of the lower and upper limits of the strike zone, based on the batter's stance. The operator of the pitch-f/x system sets those limits on a video screen as the batter assumes the hitting position.

This data, then, allows us to know if a pitch was actually in the strike zone. However, I have found some problems with these strike zone limits that come with the pitch data, namely, they seem to vary a quite a bit, even for the same batter on different days. As an example, here are the lower and upper limits of the strike zone for Derek Jeter on three different occasions:

Limits of Jeter's strike zone (inches)
Game         Low    High    Diff
Tex, 5/3    23.6    53.0    29.4
Sea, 5/12   23.3    46.4    23.0
Chi, 5/16   20.4    40.5    20.1
Diff: High minus Low; the vertical size of the strike zone

Let me say that I don't think this is particularly surprising. As I mentioned above, this is a complicated system that has just begun operating. There is surely a learning curve for the system's operators and I'm confident that the strike zone data will improve as time goes on.

But in the meantime, what shall we do? Do we abandon our idea of measuring the vertical strike zone using the pitch data? Actually, I don't think we have to do that. What we can do is assume that on average the system's operators are getting it right. So, for each batter, I calculate his average strike zone lower and upper limits, based on the pitch data. Then I apply each batter's average strike zone for all pitches thrown to him, instead of the pitch-by-pitch values that come with the data. Make sense?

The results for both right-handed and left-handed batters, is shown in the graph below:

From these plots, it now appears that umpires are not really calling the vertical strike zone as they should, although they are doing just as poorly on the low strike as they are on the high strike. Here are updated versions of a plot and table I ran last time:

Actual vs. Rulebook Strike Zone Dimensions (inches)
            Left  Right   Lower*  Upper  Total Area+
RHB        -12.0   12.1    21.6    42.0      492
LHB        -14.6    9.9    21.5    40.8      475
Rulebook    -9.9    9.9    17.7    44.2      527
* vertical strike zone mapped to average
+ total area in square inches

It's hard to judge the height of a pitch on TV

But are we seeing what we think we're seeing? I'm not sure we are. When we watch a pitch on television, we generally see if from the center field camera, so we have no depth perception along a line from the pitcher's mound to home plate. We necessarily judge the location of a pitch from where it hits the catcher's glove. However, since the pitch is moving at a downward angle and the catcher is positioned well back of home plate, the pitch drops significantly from the point it passes through the strike zone to the point where the catcher receives it.

The amount of drop will depend on the speed and the type of pitch, it can be a foot or more for a slow curve, but even hard fastballs will drop 3-4 inches between home plate and catcher's glove. As I mentioned, watching on TV we cannot discern this drop, we can't tell how high the pitch was when it crossed the plate.

Note that this same illusion is present even when viewing a pitch from the side, which is the view on some replays. In that case, we tend to judge the pitch as it passes the batter, but almost all batters take their stance well back in the batter's box and the distance from the front of home plate to the batter (middle of chest, let's say) can easily be two feet. Again, many pitches will drop several inches over that distance, and we will think the pitch is lower than it actually was.

In other words, it is virtually impossible to judge the vertical position of where a pitch crosses the strike zone by watching on TV.

Final thoughts

So what about Ted Williams and his claim that umpires can call pitches to an accuracy of one inch, what does my study say about that? Well, the nice curve I calculated for the third graphic in this article assumed an accuracy of 2.5 inches. Now that number represents a combination of the average accuracy of the umpires and the accuracy of the pitch-f/x system. The latter is reported to have an accuracy of one inch, but keeping with my skeptical nature, I will assume that this is the best-case scenario. This would imply that the contribution of the umps to the overall accuracy is, at most, a little over two inches (see the Resources section if you're curious about how I get this number). Two inches is not as good as Williams' estimate, but I think it's pretty darn good.

• Analytical ball fraction curves—I used a simple simulation to generate these curves. The first step is to choose a random number between -2 and two feet. This is the true position of a pitch. To that I add a small number, the uncertainty, the result being the apparent position of the pitch. The uncertainty is normally distributed with mean zero and sigma set to one, two or three inches, etc. The pitch is a strike if its apparent position is within the strike zone. I generate thousands of pitches this way, and the ball fraction as a function of the true position gives the curves shown above.

  To reproduce the actual data, I had to add about 5% of pitches where the uncertainty is very large (around two feet) instead of two or three inches.

• Accuracy of umpire's eye — Our measured accuracy is a combination of the accuracy of the pitch-tracking system and umpire accuracy. When there are multiple contributions to an uncertainty, the total uncertainty is not the sum of the individual contributions, but rather the square of the total is the sum of the squares. Thus, given total uncertainty (s_tot) and pitch-tracking uncertainty (s_track), the umpire uncertainty (s_ump) can be estimated as s_ump = sqrt(s_tot^2 - s_track^2).