Risk, consistency and uncertainty

During the latest Baseball HQ podcast, Dylan Hedges was talking about the consistency tools Baseball HQ uses to help fantasy players in head-to-head fantasy play. He noted that some consistent players this year have been Magglio Ordonez, Lance Berkman, Chipper Jones and Dan Uggla. He especially noted that Uggla has been surprisingly consistent, in contrast to 2007. Among inconsistent players who had delivered quite a few “disaster” weeks he mentioned Kenji Johjima, Asdrubal Cabrera and Corey Patterson.

If you look at the two lists, you’ll notice that the consistent players have been having good seasons and the inconsistent players have been having bad seasons. I do not think we need a consistency tool to measure this. And should we be surprised that Uggla has been much more consistent this year? Probably not since he’s hitting 42 points higher than last year and has almost equaled last year’s home run output. I’m not even sure if there is much skill in week-to-week consistency given the small sample sizes of weekly performances. In most cases, if you get good players, you will have consistent players.

Baseball HQ also uses a reliability score to measure the risk of a player. The least risky players are those “who receive regular playing time, are healthy, are in a stable age range and have displayed consistent performance over the past three years (using runs created per game).” Sounds great, right? I would say no.

What the consistency tool and reliability score have in common are that they use past “risk” results to forecast future risk measurement. While in baseball we use past data all the time to come up with performance forecasts, I would recommend being wary of trying to measure risk and consistency using past data. Just because a player has had consistent results in the past does not mean his projection is now safer. The results of all players will follow a binomial distribution, so naturally, due to a survivorship effect, some players will appear more consistent than others.

For example, John Lackey has a 96 reliability score (the reliability scores are given on a scale of 0-100, with 100 being the least risky). Now, I’d feel pretty safe with various pitching projections for Lackey, since he has thrown an average of 210 innings the past five years. In other words, we have a pretty good estimate of Lackey’s true talent.

I do not trust, though, measuring Lackey’s risk using his past data. Lackey gets a high reliability score because he has thrown an average of 210 innings the past five years. However, couldn’t one say that Lackey is at a high risk of burnout because he has thrown so many innings? Obviously it is easy for me to be saying this since Lackey was hurt earlier this year, but the point still remains. Note that this should not be considered an attack on Baseball HQ. The analysts there are doing tremendous work, and they are in my opinion one of the top sabermetric web sites out there.

“Burnout” is probably the wrong term to use in regard to Lackey and risk. I believe that when we talk about risk in baseball, we should be using Knightian terms. Frank Knight, a University of Chicago economist, has defined risk as a situation where unknown outcomes have knowable probabilities while uncertainty is a situation where we cannot calculate the probabilities of unknown outcomes. I would argue that the measurement of Lackey’s true talent is a risk.

For example, we can calculate how much we need to regress his stats to the mean using the reliability measurement given in a Marcels forecast. I would also argue that at this stage in sabermetrics, the chance of Lackey getting hurt is an uncertainty. In other words, we cannot safely put a probability on what the chances are that Lackey lands on the DL.

Now, don’t get me wrong. I do think health is a skill. I just don’t think we’re very good at quantifying that skill right now. Forecasting systems like PECOTA may come up with numbers that give an attrition rate, but these have not been empirically tested, and it’s still questionable how much information Will Carroll’s injury projection system adds. As Knight argued as an economist, when we can quantify risk, we can diversify and essentially become “risk free.” Because of this, uncertainty is where a large part of profits will come from. In baseball, I feel this uncertainty lies within risk management.

One important point when dealing with uncertainty is Nassim Taleb’s central idea of uncertainty: To make a decision, you need to focus on the consequences, which you have a better chance of understanding than the probabilities, which by the definition of uncertainty you do not know.

For example, how many different answers do you think I would have gotten if I asked at the beginning of the year, “How many innings do you think Rich Harden will pitch?” Now, I think it would be very difficult for forecasters to estimate that. However, I think most forecasters would agree that Harden is a very good pitcher. Note that I am not advocating that teams stack themselves with only highly talented but injury-prone players. I do advocate distinguishing between risk and uncertainty when we forecast.

Another important point Taleb makes: When we deal with risk, often what matters is not the actual forecast but the variability of the forecast. In other words, when we have small samples of information on a player, the actual projection for the player isn’t as important when we consider the variability around that projection.

As projection systems become more advanced, forecasters are becoming better at projecting player performance, especially when players have large samples. Because of this, some forecasters are using various statistical tools to try to gain an edge by managing these projections along with the risk and consistency of these projections. However, I feel these measurements are just in the beginning stages and as a result, when some analysts claim we are dealing with risks, we are actually dealing with uncertainty.

I hope that at some point in the near future we will be able to make baseball decisions in terms of risk, but by trying to do so now, forecasters will make many more mistakes than they think.