State of sabermetrics: Insights from the 2010 Sloan Sports Analytics Conference

I want to talk about two people: Kevin Kelley and Mark Cuban.

Most of you probably know that Mark Cuban is the brash billionaire owner of the Dallas Mavericks. Kevin Kelley is just a high school football coach from Little Rock, Ark. He has become something of a minor celebrity since he adopted a punt-free approach to football offense and led Pulaski Academy High School to multiple state titles. On the surface it might seem a little gimmicky. I disagree. He didn’t just look at the numbers; he was willing to follow them to their logical conclusion. He is one of the important innovators in North American sports of the last ten years.

image
The numbers tell Kevin Kelley to go for it on fourth down, so he does. Photo credit: John Marcus, MIT Sloan Sports Analytics Conference.

Don’t believe me? At the 2010 MIT Sloan Sports Analytics Conference, held last weekend in Boston, Kelley was featured on a panel alongside Avery Johnson, Buck Showalter, and Nate Silver. And with plenty of sports brainpower on stage, it was Kelley who made the most important statement. After an hour-long panel discussion on the influence of analytics on coaches, a conference attendee asked if there was a method of presentation that made them more receptive to analytics. Showalter and Johnson, two very strong personalities and master showmen, dominated the conversation. Kelley, barely managing to get a word in edgewise, said, “It’s not just the method in which it’s said, it’s who says it.”

Why won’t anyone listen to the stathead?

Well, fellow statheads, that’s just it, isn’t it? We can bang the WAR drum all day. We can refine our PITCHf/x studies until we find the one pitch that even Pujols can’t hit. We can play all the fancy analytical tricks we want. But when it comes to using these analytics, teams have to do more than just hire a few quants to sit under the stadium and code all day. Lots of teams have lots of good sabermetricians working for them. A decent number of teams have analytically minded general managers. But in talking with and hearing from various front office personnel, one recurring theme kept coming back: integrating serious, rigorous sabermetric thinking into a formalized decision-making framework is hard.

John Dewan, of Baseball Info Solutions and pioneer of fielding statistics, said that his organization meets with lots of baseball teams, not just the sabermetric-friendly ones. He was candid in his assessment that some teams that he speaks with don’t understand how to effectively use the defensive data that’s available. John Abbamondi, the assistant GM of the St. Louis Cardinals, gave an example of a colleague who wanted platoon splits for relief pitchers over a one-week period—data with such small sample sizes so as to be rendered irrelevant as a predictive tool.

These are smart people, and you can hear it in the language they use. Abbamondi views vesting options on player contracts as an arbitrage opportunity. Tom Tippett, director of baseball information services for the Red Sox, produced the world’s best baseball simulator when he built Diamond Mind. Shiraz Rehman, director of baseball operations for the Arizona Diamondbacks, even dropped a Black-Scholes reference on the audience. But it’s not enough to be a smart person, even in an organization receptive to these ideas. The whole culture of an organization has to immerse itself not just in the latest and greatest stats, but in the rigorous processes that precede them. It was the otherwise unimpressive panelist Dan Duquette (the former Red Sox GM) who told the audience, “Teams that have effective decision-making processes are the ones that are at the top of the division.” It sounds obvious, but as a sabermetric conceit, it’s hard to say that it has been successfully implemented in all but a few major league organizations.

image
Buck Showalter, haunted by the trophies of the teams that followed his. Photo credit: John Marcus, MIT Sloan Sports Analytics Conference.

Showalter paid lip service to the goal of successfully blending scouting observation with statistical innovation. But he delighted in telling the anecdote of the analysts who suggested that Nelson Cruz be placed in center field. Showalter obliged but predicted defensive disaster, and after a miserable two games in center (as Showalter told the story), the analysts came back with their tails between their legs. In this particular instance, it’s quite possible that the analysts were dead wrong and Showalter correct (despite a sample size of two games). And I’m not going to blame Showalter—who is not only human but by my account is a terrific person in my short interaction with him—for feeling vindicated. But his story highlights the nature of the problem: The decision makers need to be convinced, and the people convincing them are bright but they are not in a position of real authority.

Worse still, GMs are fired, assistants are promoted, analysts come and go. There’s organizational turnover, especially in an industry where teams will make a change just to say they made a change. The organizational culture can’t come from a GM, no matter what kind of fiefdom he has the freedom to run in the baseball operations department. What teams need is something like The Dodger Way for their management.

Which brings me back to Mark Cuban and Kevin Kelley. Kelley, the high school football coach, noted that it’s not what the geeks say, or even how they say it, but who they are. And Mark Cuban, to my mind, is the first true stathead owner in North American sports. His approach to thinking about basketball is couched in the analytical approach (one gets the feeling that if his entrepreneurial ventures hadn’t been successful, he’d be sitting in his mother’s basement blogging about sports like the rest of us losers).

Maybe baseball analysis really is dead

I derided Gary Huckabay’s proclamation that “baseball analysis is dead.” On its face, it is simply untrue. But one-liners aside, his true message referred to the state of baseball analysis within major league organizations. Quoth Huckabay:

That means that if you want to change something within an organization, you need to make the case. And making the case means showing a compelling reason why the organization should commit to some investment, or overhauling some existing process. The bar for ‘compelling’ in that sentence is usually set extremely high. … They need to develop a culture receptive to adopting the changes implied by this newfound wisdom.”

There are many ways to make that happen, but the simplest, and likely the most effective, way that I can think of for baseball analytics is to have a stathead buy a team. That’s why the prospect of having Cuban purchase a major league team should excite statheads. (Cuban, to his credit, stated that the Pirates were not put on sale because there is no financial disincentive to losing games in MLB, as there is in the NBA.) I think the orthodoxy suggesting that owners should be check-writing vehicles for the real brains in the baseball operations department is incorrect. Daryl Morey, conference chair and unabashed stathead GM of the Houston Rockets (“we don’t make any decisions based on the belief in clutch”), said that an active owner is a good owner, and that a franchise with an absentee owner is going to have a harder time being successful. I agree, given the caveat that the owner should be dedicated to sound and rigorous decision-making as an organization.

image
Somebody give this man a baseball team. Photo credit: John Marcus, MIT Sloan Sports Analytics Conference.

The process can show up directly on the field. If there is an organizational buy-in, from the owner all they way on down, on the importance of understanding leverage when deciding when to bring in your bullpen ace, I imagine (hope?) there would be a lot less kvetching from relief pitchers about their inability to perform when they “don’t know their roles.”

Okay, so maybe it’s asking a lot for a stathead to buy a team. The fabulous wealth that Tom Tango generated from sales of The Book won’t make him the winning bidder for a Montreal-based expansion franchise anytime soon. But it is fitting that the Sports Analytics Conference was hosted by a management school, because it is poor management, and not poor analytics, that prevents baseball teams from making sabermetrically sound decisions.

The frontiers of baseball analysis

So where can proper management of analytics make a difference? Rob Neyer, moderator of the baseball analytics panel, posed a direct question to the participants: What is something that you don’t know that you want to know? Tippett nailed it when he said, “We want to get career tracks on players on an individual basis.” (In fairness, he first snarked that he wanted to know how to prevent the Yankees from ever winning a World Series.) I would take the liberty to make it even more specific: Given a pool of minor leaguers or even college players, can you identify which ones will produce at a star level in their pre-arbitration years and return huge surplus value to their teams?

It sounds almost too simplistic, but lots of the cutting-edge analytical work that is being done contributes to this goal. PITCHf/x is a great tool for putting a quantitative slant on traditional scouting. The real breakthrough will come when we can use PITCHf/x to tease out the good young pitchers from the future great ones. A similar argument could be made for a rigorous injury-analysis, prevention, and treatment program.

A recurring theme at the conference was that teams wanted an organized way to understand player psychology—in scoutspeak, “desire” or “make-up.” That goes directly to the heart of understanding career tracks of individual athletes. Abbamondi noted that when a player’s make-up is “off the charts,” then it is time to get a bigger chart. As it stands, a player can get a bad rap—a Milo hung around him, as in Moneyball—simply by how many yessirs and nosirs he gives to an amateur scout. Rehman agreed.

Bill Polian, president of the Indianapolis Colts, succinctly organized the task on three levels: One, isolate the characteristics that are unique to your sport and your organization; two, nurture those characteristics in young athletes as they mature from post-adolescents to adults; and three, improve any identified deficiencies in your athletes through coaching and training. Morey, Cuban, and Patriots executive Jonathan Kraft echoed the sentiment to varying degrees, and Cuban mentioned that the Mavericks have an organizational system in place for psychological testing and evaluation that he believes is way ahead of the NBA curve.

Psychology isn’t the first thing that comes to mind when you think of advanced analytics. Indeed, “heart,” “grit,” and “determination” are snark targets among the sabermetric set. And at the major league level, that may be partly true. There is a huge selection bias for major league psychology—all major league players must have a minimum level of “heart” in order to parlay their talent into a major league career, and the spread in desire-as-talent is likely small. But the real value lies is in understanding and predicting which young or amateur players are going to turn into stars, and it’s not out of the question that psychological talent hasn’t been selected for at that level.

Maybe it is just random, and there is no way to increase your odds. But dollar for dollar, pouring resources into finding out is quite likely a better use of resources than, say, bringing the error bars on fielding metrics down from +/-5 to +/- 1. In the latter case, the marginal improvement is a couple of million dollars. In the former case, the marginal improvement is tens or perhaps hundreds of millions of dollars in surplus value.

Out of the basement, into the office

So I left the conference feeling both re-energized and depressed about baseball analytics. There is a lot to be done, but I am not certain that today’s practitioners of sabermetrics are the ones who will do it. The innovations of the last 30 years were driven by maverick researchers, working alone or in small groups and inventing and re-inventing the framework of baseball analytics. These are the hard-working and bright writers and analysts that populate this and other popular sabermetric websites. Front-office analysts with whom I spoke last weekend were confident that their internal metrics were better than the freely available ones of the sabermetric community, perhaps because of better data or better methodologies. But the differences can be only marginal at best. The online sabermetric community is indeed the foundry for the analytics in modern baseball front offices. (Why else do they hire our best analysts?)

The future of analytical innovation in baseball, in my opinion, is going to move away from basement bloggers to the teams themselves, or companies founded specifically to consult teams on these issues. Industry events like the Sloan Sports Analytics Conference are going to be the germ for such analyses. MLB teams are far better equipped to deal with issues of mental and physical health than the amateurs, both in terms of competency and ability to invest intellectual and financial resources.

Understanding and predicting the career trajectories of individual players, if possible, is going to happen at a team level. As Morey put it, “If you can figure this out, I have a job waiting for you.” In other words, don’t expect the next revolution to happen at THT. Don’t expect Michael Lewis to write a best-seller about it. Teams are going to clamp down on it and protect their competitive advantage as best they can.

Just hope that your team that figures it out first.

References & Resources
I thank conference co-chair Daryl Morey and student volunteer Ben Schlesinger for arranging a media pass for THT. This was the fourth year that THT has covered the event, and it is growing quickly in both scope and quality.


15 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Mike W
14 years ago

Nice write up Sal,
a lot interesting tidbits there.

salb918
14 years ago

Thanks Mike. 

There were a few “fly-on-the-wall” moments I didn’t write about.  My favorite was overhearing Daryl Morey and Brent Barry organize a post-conference basketball game with Mark Cuban.

Second favorite was hearing someone complain that Michael Lewis had to rush out to fly to LA for the Oscars.  Can you really blame Lewis for wanting to leave a bunch of geeks behind for the glitz and glamor of the red carpet?

Sky
14 years ago

Awesome, thanks, Sal.

Dave Studeman
14 years ago

Great wrap, Sal.  Not sure I agree with your conclusion, however.

To the extent future analytics are based on proprietary data, such as psychological testing, then the future does seem to lie with teams. But if much of the critical data remains free to the public, such as Pitchf/x, then I think the Internet will still spawn plenty of revolutionary analysis.

edk
14 years ago

There’s another way forward: other sports need your help. There are opportunities for researchers to contribute in, say, basketball, that don’t exist for baseball. I keep pitching this to baseball statheads: if you want to make a real difference, find a sport which hasn’t been “solved” to the degree that baseball has—not only will you find a largely uncharted territory for you to explore, you’ll find people in front offices who are willing to listen to anything that gives them an edge, as Morey suggests above.

Josh Fisher
14 years ago

I’m not all that good at conspiracy theories, so I don’t even know how to begin wondering what some teams know that we don’t. But I’m with Dave…I don’t see the well running dry; there are too many ridiculously talented people motivated by nothing more than a desire to learn more about the game. Have we already hit the big stuff? Largely, yes. But there’s all sorts of things left.

Speaking of things we want to know…can we measure the lateness of pitches’ breaks yet? Or, I should say: have we?

Sean Smith
14 years ago

“Front-office analysts with whom I spoke last weekend were confident that their internal metrics were better than the freely available ones of the sabermetric community, perhaps because of better data or better methodologies.”

This I am very skeptical about.  How can you tell?  One is how they value players. For every free agent signing, how much do the statheads think he’s worth?  And how much do teams pay him?  And then compare to what the players actually do next year.  More often than not in recent years, the statheads (posting on Tango’s yearly offseason transactions thread) did a better job valuing players.

salb918
14 years ago

Sean: I am skeptical too, which is why I think any advantages they have over (for example) fangraphs are marginal.  If I were a cheap owner, I’d pay a Primate to scour Fangraphs instead of paying for a “true” analyst.

Dave Studeman
14 years ago

An example: One front office exec emailed me asking why we won’t be running Revised Zone Ratings next year.  If their data is so good, why are they also relying on sites like ours?

salb918
14 years ago

Dave: data validation, I would guess.

To be fair, I spoke to people from two of the more analytically-minded FOs.  But yeah, without seeming too arrogant I would guess that the sabermetric community is at least 90% of the way to whatever the best MLB FOs have, maybe more in offense.

Dave Studeman
14 years ago

Yeah, I think you’re giving these clubs a bit too much credit, Sal.  I’m sure there are some advanced sabermetric-thinking organizations, but I’ll bet most of them still get the majority of their sabermetrics off sites like ours, Fangraphs, BJOL, Baseball Ref, etc. etc.  Like you said, why wouldn’t they?

salb918
14 years ago

Dave: fair enough.  I don’t disagree.  But, as I said, I think the MLB teams have bigger fish to fry in the minors.

Pizza Cutter
14 years ago

Did someone say psychology?

Mike Fast
14 years ago

I love the article, Sal, as I told you already.

@edk, I can’t speak for everyone, but it’s not my goal to work my way into a sports career or a front office in whatever way I can.  I love the game of baseball.  At the age of 37, I’m more fascinated by it than ever.  Some of the older analysts and fans give me confidence that that will still be true in my old age.

I enjoy watching a football, basketball, or hockey game here and there, but I can’t imagine investing myself in them the way I do in baseball or finding the same joy in analyzing them.

@Josh, yes, we know how pitches break thanks to PITCHf/x.  The constant-acceleration model is a very good fit to the measured trajectory data.

@Sal, Dave, Sean…regarding front office capabilities, I have been very impressed with the data integration that some front offices have.  If I had to pick between the capabilities of all 30 MLB teams’ baseball ops analytical people and all the free-lance people saber-heads, I’d choose the public domain people in a heartbeat.  But if you added in all the scouting and coaching resources that the baseball ops people also have at their disposal, I’d be inclined to lean that direction. 

The key, in any case, is making that data, whatever its source, useful to the decision makers, and that takes someone smart, with good understanding of the game, and good communication/presentation/organization skills.

Robert Dudek
14 years ago

I realize that this was far from the intent of the article, but a reader may come away with the impression that sabermetrics is all about prediction modelling.

While there has been increased emphasis on prediction for the past decade or so, sabermetrics is really about learning about the game of baseball by studying its past. Whatever use that knowledge has for a major league baseball team is of secondary consideration (in a scientific sense).