Factoring risk and uncertainty into player evaluation

Most player forecasts these days give projections in single number values. Baseball Prospectus’ PECOTA forecast system does give a range of estimates but uses a weighted average system to come up with single number estimates. Now this is fine if the decision maker using the forecast is risk neutral, but most people are risk averse. Measuring risk and uncertainty is of course a very important topic in baseball forecasting, but in this article I will just be looking at using risk preference to value a player. I talked briefly about risk preference with prospects in my last article and will expand upon that with free agents. Also introduced in my last article were utility curves, which are a key part of risk analysis. A utility curve essentially measures how much satisfaction one receives as a certain variable is added or subtracted. Now let’s look at how this can be used to value free agents.

Let’s look at the utility curves of two hypothetical teams. Both teams are looking to sign a shortstop. One team appears to be a legitimate playoff contender and has an excellent core returning. We will call this team “Team A.” The other team returns a group of solid veterans but lacks the elite level talent to separate them as a playoff contender. This will be “Team B.”

Team A Utility Curve

Team B Utility Curve

The X-axis shows a player’s value measured by WAR (Wins Above Replacement). The Y-axis shows each team’s utility. Utility is measured in utils. A 0 WAR and 5 WAR player are used as reference points where a 0 WAR player brings back 0 utility and a 5 WAR player brings the team 1 util. From these utility curves we can see that these teams have different aversions to risk. Team A receives relatively high utility once it signs a player who is above replacement level. This is because team A is already a major playoff contender. Almost anything they do that does not hurt them will be a plus. Their risk aversion can be signified by a concave-shaped curve. Meanwhile, team B needs high quality players if it is to be a playoff contender because of the returning players. This means there is little difference for the team between a replacement level player and an average player. Therefore, the team’s utility curve has a convex shape.

Now let’s say there are two shortstops on the free agent market that these teams are interested in. Both shortstops have a true talent of 2.5 WAR. However, shortstop A has a 20 percent chance of being a 2 WAR player, a 60 percent chance of being a 2.5 WAR player, and a 20 percent of being a 3 WAR player. Shortstop B has a 20 percent chance of being a 1 WAR player, a 30 percent chance of being a 2 WAR player, a 30 percent chance of being a 3 WAR player, and a 20 percent chance of being a 4 WAR player. If we used a single number expected value forecast each player would be evaluated as a 2.5 WAR player. Each shortstop would be evaluated as being worth $11.4 million (1 WAR = $4.4 million, plus the addition of the $.4 million minimum salary). However, these teams, when factoring in risk, will find the value of each player by taking utility of each WAR possibility multiplied by the chance of each WAR possibility. For example:

Team A Evaluation

Shortstop A: .2*.7 + .6*.8 + .2*.9 = .8 utils
Shortstop B: .2*.4 + .3*.7 + .3*.9 + .2*.975 = .755 utils

Team B Evaluation

Shortstop A: .2*.25 + .6*.35 + .2*.45 = .35 utils
Shortstop B: .2*.1 + .3*.25 + .3*.45 + .2*.7= .37 utils

So from this risk analysis we see that team A prefers shortstop A and team B prefers shortstop B. This makes sense given their risk aversion and the risk of each player. However, besides preference, expressing their value in utils does not really mean anything. If you tell a general manager this shortstop is worth .8 utils he will stare back at you with a blank face. However, we can convert these utils back to a risk-adjusted WAR equivalent. All we have to do is go across from a player’s utility on the y-axis until we hit the graphed line. Then we go down from that point until we reach the X-axis. This point we have reached is a player’s risk-adjusted WAR. So for team A, shortstop A is equivalent to a 2.5 WAR player and worth $11.4 million and shortstop B is equivalent to a 2.1 WAR player and worth $9.64 million. For team B, shortstop A is worth $10.5 million and shortstop is worth $12.28 million. Thus, two players are projected to have the same true talent level actually have different values to teams based on their risk preference.

We can do the same evaluation for draft picks. Some teams are noted for drafting college players with only moderate upside while others are known to take high-risk, high-reward high school prospects. While these teams may have different evaluations of the future production of these players, this also occurs because these teams have different preferences towards risk. A team whose preferences lean toward safer college players would have a utility curve that looks similar to the first free agent utility curve while a team that prefers more volatile high school players would have a utility curve that looks similar to the second free agent utility curve.

While factoring in risk isn’t too hard, projecting risk is a lot more difficult. Right now if you were to value players while including risk you would have two options. Option one would be to use PECOTA and its range of forecasts along with a risk preference to come up with a risk adjusted player value. Option two would be to use whatever forecasting system you prefer and use the mean projection for whatever player you are evaluating. Then you would have to subjectively determine a player’s range of outcomes. I hope to look into projecting volatility in the future, but for now I hope I was able to show how a decision maker can factor in risk in player evaluation.