State of sabermetrics: Insights from the 2010 Sloan Sports Analytics Conferenceby Sal Baxamusa
March 11, 2010
I want to talk about two people: Kevin Kelley and Mark Cuban.
Most of you probably know that Mark Cuban is the brash billionaire owner of the Dallas Mavericks. Kevin Kelley is just a high school football coach from Little Rock, Ark. He has become something of a minor celebrity since he adopted a punt-free approach to football offense and led Pulaski Academy High School to multiple state titles. On the surface it might seem a little gimmicky. I disagree. He didn't just look at the numbers; he was willing to follow them to their logical conclusion. He is one of the important innovators in North American sports of the last ten years.
|The numbers tell Kevin Kelley to go for it on fourth down, so he does. Photo credit: John Marcus, MIT Sloan Sports Analytics Conference.|
Don't believe me? At the 2010 MIT Sloan Sports Analytics Conference, held last weekend in Boston, Kelley was featured on a panel alongside Avery Johnson, Buck Showalter, and Nate Silver. And with plenty of sports brainpower on stage, it was Kelley who made the most important statement. After an hour-long panel discussion on the influence of analytics on coaches, a conference attendee asked if there was a method of presentation that made them more receptive to analytics. Showalter and Johnson, two very strong personalities and master showmen, dominated the conversation. Kelley, barely managing to get a word in edgewise, said, "It's not just the method in which it's said, it's who says it."
Why won't anyone listen to the stathead?
Well, fellow statheads, that's just it, isn't it? We can bang the WAR drum all day. We can refine our PITCHf/x studies until we find the one pitch that even Pujols can't hit. We can play all the fancy analytical tricks we want. But when it comes to using these analytics, teams have to do more than just hire a few quants to sit under the stadium and code all day. Lots of teams have lots of good sabermetricians working for them. A decent number of teams have analytically minded general managers. But in talking with and hearing from various front office personnel, one recurring theme kept coming back: integrating serious, rigorous sabermetric thinking into a formalized decision-making framework is hard.
John Dewan, of Baseball Info Solutions and pioneer of fielding statistics, said that his organization meets with lots of baseball teams, not just the sabermetric-friendly ones. He was candid in his assessment that some teams that he speaks with don't understand how to effectively use the defensive data that's available. John Abbamondi, the assistant GM of the St. Louis Cardinals, gave an example of a colleague who wanted platoon splits for relief pitchers over a one-week period—data with such small sample sizes so as to be rendered irrelevant as a predictive tool.
These are smart people, and you can hear it in the language they use. Abbamondi views vesting options on player contracts as an arbitrage opportunity. Tom Tippett, director of baseball information services for the Red Sox, produced the world's best baseball simulator when he built Diamond Mind. Shiraz Rehman, director of baseball operations for the Arizona Diamondbacks, even dropped a Black-Scholes reference on the audience. But it's not enough to be a smart person, even in an organization receptive to these ideas. The whole culture of an organization has to immerse itself not just in the latest and greatest stats, but in the rigorous processes that precede them. It was the otherwise unimpressive panelist Dan Duquette (the former Red Sox GM) who told the audience, "Teams that have effective decision-making processes are the ones that are at the top of the division." It sounds obvious, but as a sabermetric conceit, it's hard to say that it has been successfully implemented in all but a few major league organizations.
|Buck Showalter, haunted by the trophies of the teams that followed his. Photo credit: John Marcus, MIT Sloan Sports Analytics Conference.|
Showalter paid lip service to the goal of successfully blending scouting observation with statistical innovation. But he delighted in telling the anecdote of the analysts who suggested that Nelson Cruz be placed in center field. Showalter obliged but predicted defensive disaster, and after a miserable two games in center (as Showalter told the story), the analysts came back with their tails between their legs. In this particular instance, it's quite possible that the analysts were dead wrong and Showalter correct (despite a sample size of two games). And I'm not going to blame Showalter—who is not only human but by my account is a terrific person in my short interaction with him—for feeling vindicated. But his story highlights the nature of the problem: The decision makers need to be convinced, and the people convincing them are bright but they are not in a position of real authority.
Worse still, GMs are fired, assistants are promoted, analysts come and go. There's organizational turnover, especially in an industry where teams will make a change just to say they made a change. The organizational culture can't come from a GM, no matter what kind of fiefdom he has the freedom to run in the baseball operations department. What teams need is something like The Dodger Way for their management.
Which brings me back to Mark Cuban and Kevin Kelley. Kelley, the high school football coach, noted that it's not what the geeks say, or even how they say it, but who they are. And Mark Cuban, to my mind, is the first true stathead owner in North American sports. His approach to thinking about basketball is couched in the analytical approach (one gets the feeling that if his entrepreneurial ventures hadn't been successful, he'd be sitting in his mother's basement blogging about sports like the rest of us losers).
Maybe baseball analysis really is dead
I derided Gary Huckabay's proclamation that "baseball analysis is dead." On its face, it is simply untrue. But one-liners aside, his true message referred to the state of baseball analysis within major league organizations. Quoth Huckabay:
That means that if you want to change something within an organization, you need to make the case. And making the case means showing a compelling reason why the organization should commit to some investment, or overhauling some existing process. The bar for ‘compelling’ in that sentence is usually set extremely high. ... They need to develop a culture receptive to adopting the changes implied by this newfound wisdom."
There are many ways to make that happen, but the simplest, and likely the most effective, way that I can think of for baseball analytics is to have a stathead buy a team. That's why the prospect of having Cuban purchase a major league team should excite statheads. (Cuban, to his credit, stated that the Pirates were not put on sale because there is no financial disincentive to losing games in MLB, as there is in the NBA.) I think the orthodoxy suggesting that owners should be check-writing vehicles for the real brains in the baseball operations department is incorrect. Daryl Morey, conference chair and unabashed stathead GM of the Houston Rockets ("we don't make any decisions based on the belief in clutch"), said that an active owner is a good owner, and that a franchise with an absentee owner is going to have a harder time being successful. I agree, given the caveat that the owner should be dedicated to sound and rigorous decision-making as an organization.
|Somebody give this man a baseball team. Photo credit: John Marcus, MIT Sloan Sports Analytics Conference.|
The process can show up directly on the field. If there is an organizational buy-in, from the owner all they way on down, on the importance of understanding leverage when deciding when to bring in your bullpen ace, I imagine (hope?) there would be a lot less kvetching from relief pitchers about their inability to perform when they "don't know their roles."
Okay, so maybe it's asking a lot for a stathead to buy a team. The fabulous wealth that Tom Tango generated from sales of The Book won't make him the winning bidder for a Montreal-based expansion franchise anytime soon. But it is fitting that the Sports Analytics Conference was hosted by a management school, because it is poor management, and not poor analytics, that prevents baseball teams from making sabermetrically sound decisions.
The frontiers of baseball analysis
So where can proper management of analytics make a difference? Rob Neyer, moderator of the baseball analytics panel, posed a direct question to the participants: What is something that you don't know that you want to know? Tippett nailed it when he said, "We want to get career tracks on players on an individual basis." (In fairness, he first snarked that he wanted to know how to prevent the Yankees from ever winning a World Series.) I would take the liberty to make it even more specific: Given a pool of minor leaguers or even college players, can you identify which ones will produce at a star level in their pre-arbitration years and return huge surplus value to their teams?
It sounds almost too simplistic, but lots of the cutting-edge analytical work that is being done contributes to this goal. PITCHf/x is a great tool for putting a quantitative slant on traditional scouting. The real breakthrough will come when we can use PITCHf/x to tease out the good young pitchers from the future great ones. A similar argument could be made for a rigorous injury-analysis, prevention, and treatment program.
A recurring theme at the conference was that teams wanted an organized way to understand player psychology—in scoutspeak, "desire" or "make-up." That goes directly to the heart of understanding career tracks of individual athletes. Abbamondi noted that when a player's make-up is "off the charts," then it is time to get a bigger chart. As it stands, a player can get a bad rap—a Milo hung around him, as in Moneyball—simply by how many yessirs and nosirs he gives to an amateur scout. Rehman agreed.
Bill Polian, president of the Indianapolis Colts, succinctly organized the task on three levels: One, isolate the characteristics that are unique to your sport and your organization; two, nurture those characteristics in young athletes as they mature from post-adolescents to adults; and three, improve any identified deficiencies in your athletes through coaching and training. Morey, Cuban, and Patriots executive Jonathan Kraft echoed the sentiment to varying degrees, and Cuban mentioned that the Mavericks have an organizational system in place for psychological testing and evaluation that he believes is way ahead of the NBA curve.
Psychology isn't the first thing that comes to mind when you think of advanced analytics. Indeed, "heart," "grit," and "determination" are snark targets among the sabermetric set. And at the major league level, that may be partly true. There is a huge selection bias for major league psychology—all major league players must have a minimum level of "heart" in order to parlay their talent into a major league career, and the spread in desire-as-talent is likely small. But the real value lies is in understanding and predicting which young or amateur players are going to turn into stars, and it's not out of the question that psychological talent hasn't been selected for at that level.
Maybe it is just random, and there is no way to increase your odds. But dollar for dollar, pouring resources into finding out is quite likely a better use of resources than, say, bringing the error bars on fielding metrics down from +/-5 to +/- 1. In the latter case, the marginal improvement is a couple of million dollars. In the former case, the marginal improvement is tens or perhaps hundreds of millions of dollars in surplus value.
Out of the basement, into the office
So I left the conference feeling both re-energized and depressed about baseball analytics. There is a lot to be done, but I am not certain that today's practitioners of sabermetrics are the ones who will do it. The innovations of the last 30 years were driven by maverick researchers, working alone or in small groups and inventing and re-inventing the framework of baseball analytics. These are the hard-working and bright writers and analysts that populate this and other popular sabermetric websites. Front-office analysts with whom I spoke last weekend were confident that their internal metrics were better than the freely available ones of the sabermetric community, perhaps because of better data or better methodologies. But the differences can be only marginal at best. The online sabermetric community is indeed the foundry for the analytics in modern baseball front offices. (Why else do they hire our best analysts?)
The future of analytical innovation in baseball, in my opinion, is going to move away from basement bloggers to the teams themselves, or companies founded specifically to consult teams on these issues. Industry events like the Sloan Sports Analytics Conference are going to be the germ for such analyses. MLB teams are far better equipped to deal with issues of mental and physical health than the amateurs, both in terms of competency and ability to invest intellectual and financial resources.
Understanding and predicting the career trajectories of individual players, if possible, is going to happen at a team level. As Morey put it, "If you can figure this out, I have a job waiting for you." In other words, don't expect the next revolution to happen at THT. Don't expect Michael Lewis to write a best-seller about it. Teams are going to clamp down on it and protect their competitive advantage as best they can.
Just hope that your team that figures it out first.
References and Resources
I thank conference co-chair Daryl Morey and student volunteer Ben Schlesinger for arranging a media pass for THT. This was the fourth year that THT has covered the event, and it is growing quickly in both scope and quality.
Sal Baxamusa is a graduate student in chemical engineering. He can be reached here.