Unlisted

It’s January, the hot stove is burning, and it’s prospect list season. I’m not a fan. Non-statistical rankings of any kind are arbitrary enough as it is, and prospect rankings try to compare things that we just don’t know enough about.

There are two key differences between prospect lists and most other forms of hot stove rankings: Prospectors try to look far into the future using a very limited amount of data, and their lists attempt to compare radically different players on the same scale.

Such characteristics make prospect rankings all but useless. While some list makers try to mitigate these problems by telling us what they look for—their voting criteria, if you will—some just assert that low Single-A pitcher X is better than Double-A catcher Y, make a generic reference to each player’s ceiling, and leave it at that.

Ultimately, does it matter? Of course the whole baseball journalism industry (myself, however tangentially, included) is just entertainment, and if you like the lists, it would be silly to begrudge you your enjoyment. Assuming, however, that baseball writers and fans have an end goal of better understanding how the game works, both on the field and in the front office, we have a long way to go before we can make really useful comparisons among minor leaguers.

What We Can Project

Baseball analysts have become pretty good at looking one year into the future. There are plenty of variables at play that we may never be able to understand fully, but when it comes to predicting what a veteran player will do for his team next year, that problem is largely solved.

After one year, though, projections get pretty messy. Crudely speaking, an algorithm can project two years into the future by projecting one year into the future, then using those numbers to repeat the process for the following year. You can, of course, do that as far into the future as you like. On the other hand, given that players tend to drift toward the mean, a projection more than a couple years into the future will remove much of the uniqueness of the player being projected. Of course, accuracy suffers, too.

What does this have to do with prospect rankings? Those who evaluate prospects are, by definition, looking far into the future. No organization sends a scout to a Florida State League game solely to determine whether a trade target is going to make a big-league impact the following year. While scouts have developed a sort of specialized system for measuring young players, they are forced to use tools less precise than in-game performance numbers. Then they are expected to use that system to forecast performance in ways that the in-game statistics cannot.

It’s a tall order. While every scout and prospect-watcher can agree on a few dozen can’t-miss prospects (some of whom, of course, miss), and each one can point to a few sleepers they championed when no else did, it’s well-nigh impossible to look at a baseball player who hasn’t yet played in the majors and predict what he’ll be doing more than a year or two down the road.

Stats and Scouts

At this point in my discussion, you’re probably expecting a full-on attack on the scouting community, but that’s not my point. By now, the distinction between quantitative and qualitative analysis is artificial and unhelpful.

What needs to be better understood (and in this case, “understood” probably means “quantified”) by both scouts and statistical forecasters is the degree to which our predictions mean anything. That’s not a new idea: Tom Tango includes a reliability score with the Marcels, and a percentile system, like PECOTA, does a similar thing by showing multiple sets of projections.

The focus in prospect discussions is usually what a young player will accomplish at the big-league level. I believe that, far more often than we do already, we ought to direct our attention more closely to whether the player ever will surpass replacement level. Sure, there are a few dozen hitters in the minors who we can be fairly certain will turn into something valuable someday. (On the other hand, I should point out: In Chris Constancio’s prospect rankings, which I’ll discuss below, even Alex Gordon has only a 77% chance of becoming a major league regular.)

Beyond that small number of well-developed position players in the high minors, everybody else is a question mark. We need to know how big that question mark is.

Back to Prospect Lists

To some extent, any minor league analyst attempting to rank players is trying to take most of those factors into account. We know that, all else equal, a younger player in a tougher league has a higher ceiling; hitters are less likely to be destroyed by injuries than pitchers; different things can be expected from high school, junior college and college draftees.

Fine. Then, in addition to weighing all the factors that go into a player’s forecast (or ceiling or upside or major-league readiness), the analyst has to consider the value of different types of careers, possibly at undetermined positions. In many ways, comparing players across all levels and ages in the minor leagues (or even within a single organization) is like voting on Hall of Fame candidates before the candidates log a single major-league appearance.

Sound hyperbolic? I don’t think it is. A prospect’s value to his organization is, essentially, the weighted mean of his range of conceivable career projections, at least through his first six years of major league service time. (For the vast majority of minor leaguers, that’s more than they’ll ever get.) It’s tough enough to measure the value of a career after the fact. Comparing two players who provided value in different ways is even tougher. Now try to do the same comparisons, but with nothing except three years of minor-league stats and some video. I’m sure there’s plenty to be gleaned from that data, but its value is severely limited.

Good Starts

There may be a logical way to rate Billy Butler just above Adam Miller, as Bryan Smith did in his recent WTNY prospect list, but most such decisions seem awfully arbitrary. The further down the list, the more arbitrary it seems. The community prospect project at MinorLeagueBall.com shows just how bizarre such decision-making can be. Eric Patterson versus Dellin Betances versus Nolan Reimold? (Among others.) As I said in my introduction, this kind of thing may be great for entertainment value, but it can’t possibly mean anything.

So, it should come as no surprise that I’m far from the first person to recognize this as a problem. My particular favorite solution, incremental as it is, is Nate Silver’s “Upside.” He introduced it in this article, making many of the same points I’ve made here about the limitations of traditional prospect lists. Upside measures the degree to which a player is projected to turn into an above-average producer over a relatively long period of time. This way, at least theoretically, we could compare Patterson and Betances. (Though Nate would be the first to tell you we shouldn’t put much stock into anything a system has to say about Betances at this point in his career.)

Another approach that’s already headed in the right direction is Chris Constancio’s, which he described in his chapter, “Prospecting for Ballplayers,” in The Hardball Times Baseball Annual 2007. Using his analysis of comparable players, he analyzes the likelihood that any given minor leaguer will turn into, among other things, a major league regular. The system is limited in the sense that it doesn’t differentiate between different degrees of stars or regulars. Such an approach may give a short shrift to players like Alex Gordon, whose ceiling is higher than that of many of the other potential stars.

Moving Forward

Nate’s and Chris’s approaches give proper consideration to the possibility that even a solid minor league player will never make an impact above Triple-A. Constancio’s 13th best prospect, Humberto Sanchez, is given only a 60% chance of becoming a part-time player in the majors, though Sanchez makes up for it with a very high likelihood of becoming a star.

A projection system based on comparable players (like either of the two discussed so far) does consider the possibility of injury, but it doesn’t do so in a very player-specific way. It’s a great start to consider age, size and statistical profile, but things like a pitcher’s delivery or even a player’s attitude (in the case of someone like Elijah Dukes) can seriously affect his chances of turning into something valuable. Perhaps the biggest challenge, then, is to incorporate into forecasting systems what is now only anecdotal evidence.

As I suggested above, the factors prospect analysts use when ranking players are not flawed, they just don’t have nearly the level of sophistication they need to make judgments among atypical players. Ultimately, a young player’s value is represented by three things:

1. How good he’ll be (runs, or wins, above replacement)
2. When he’ll contribute (estimated time of major league arrival)
3. How much variance is in his projection

PECOTA is likely the best—and certainly the most ambitious—at tackling No. 1. Constancio’s approach is geared more toward No. 3. No. 2 is probably not an enormous challenge, though it’s risky business to predict when a player will have a chance to play full time, as Ryan Howard and Ryan Shealy can attest. It may be better to view No. 2 as “When he’ll be ready to contribute,” to represent a player’s potential value to the organization, whether he is promoted, traded or squandered.

As I see it, the holy grail of prospect analysis would be to combine those three aspects into one number—preferably a dollar amount. To be most accurate, the number would have to be generated in the context of a team’s success cycle; it’s arguable that Ryan Zimmerman’s six years under team control will have been wasted while the Nationals figure out how to become a winning team someday. The success cycle is what makes No. 2 so key: The Royals don’t want Alex Gordon to peak in 2007 (as fearsome as that might be); they want him to do so in, say, 2011 or 2012, when more of their promising youngsters will be contributing as well.

Put that all together in one number and you’ve got it all:

1. Team forecasts years in advance, helping teams pinpoint what needs they will have when, thus aiding free-agent decisions
2. The value of minor leaguers in trade
3. The potential of Rule 5 selections

And, of course, you could create the most sophisticated prospect list the world has ever seen.

That is, if you’re into that sort of thing.