What you need to know about player raters

Two weeks ago, we covered the innings pace projection as something that fantasy league providers like Yahoo, CBS Sports, and ESPN give to customers, often with the result of creating a psychological imperative towards a goal and shaping the way that those in leagues manage their pitching staffs.

Another tool that fantasy league providers give out is a “player rater”—a mathematical algorithm that takes accumulated statistics to date and spits out how players measure up to each other in overall fantasy value for the current season.

The psychological impact of seeing how one player compares to the next should not be underestimated. Raise your hand if you’ve ever offered or considered a trade in consult with your league provider’s player rater.

We believe there are a few basic things that everybody needs to know about player raters:

First, not all player raters are the same.

For example, Yahoo’s player rater is a pretty dominant feature in its service. You can’t make a trade without seeing it. And if you are using it, you’ll see the top current five batters so far in the 2009 season listed this way:

1. Albert Pujols
2. Ian Kinsler
3. Evan Longoria
4. Raul Ibanez
5. Adam Jones (yes, almost unbelievably, he’s No. 5)

You’ll have to search a bit harder for CBS Sports’ player rater. But once you find it, you’ll see a different order:

1. Ian Kinsler
2. Albert Pujols
3. Carl Crawford
4. Raul Ibanez
5. Kevin Youkilis

Part of the reason for the difference is that the gurus behind CBS Sports, for whatever reason, favor the Head-to-Head points scoring format, and in the past few seasons, has juggled its point calculation formula on “standard leagues.” They’ve constructed their player rater to place more weight on metrics like walks and negative value on metrics like losses and blown saves.

Second, highly scarce statistics like stolen bases and saves tend to make a big impact on player raters.

Obviously, categorical scarcity should play a role in determining value, but at the start of a season, things like a single save or stolen base can be a little deceiving. Take these two players and figure out who is rated as more valuable:

Felix Hernandez: 34 innings, 4 wins, 36 strikeouts, 2.38 ERA, 1.09 WHIP, 0 saves
Jason Frasor: 10.1 innings, 4 wins, 6 strikeouts, 0.0 ERA, 0.48 WHIP, 1 saves

For those who guessed Frasor, congratulations.

It almost goes without saying that player raters measure retrospective value much better than prospective value. A pitcher with a poor strikeout rate may luck his way into a few wins and a good position on the player rater. That’s not to say that luck will continue.

Consider Ramon Ramirez, reliever for the Boston Red Sox, who currently has 2 wins and a 0.0 ERA despite having a strikeout rate of 4.8 per 9 innings pitched and a FIP (a measure of adjusted ERA based on peripherals and minus defense) of 3.1.

A Hardball Times Update
Goodbye for now.

Right now, Yahoo’s player rater judges Ramon Ramirez to be the 15th most valuable pitcher in baseball, ahead of Josh Johnson, among others. Strip away the vulture wins and adjust the ERA and Mr. Ramirez wouldn’t crack the top 200 pitchers.

On the other hand, don’t totally discount the value of a good middle reliever with solid ratios. Year after year, those in fantasy baseball leagues express some disbelief that middle relievers are indeed valuable in leagues that don’t count holds. But in this case, statistics don’t lie.

Yes, in 2008, Grant Balfour computed as a Top 100 fantasy player, ahead of Michael Young, Corey Hart, and Scott Kazmir. Considering the season that Balfour is having now, it may be tough to figure out those rock-solid middle relievers, but they appear high up on player raters for a good reason: A sub-2.0 ERA in 80 innings is indeed quite valuable. That’s a topic for another day, however.

Finally, the longer the season goes on, as sample sizes become larger and larger, player raters gain more credence, but also less worth.

At the start of the season, we all want players who are going to contribute to overall categorical success. As the season moves on, our position in some categories becomes fixed while the opportunity to move ahead in other categories offers promise. Making a player move becomes more dependent on the context of a particular league’s standings.

If there’s a conclusion to be drawn from all of this, it’s that going for the player who is rated as having the highest overall value is not always the smartest move.


5 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Briks, Philly
14 years ago

I use my own player rater system, and this might be a good place to fund out if people think i’ve used a sound methodology. I’m in a weekly head-to-head 5×5 league with normal cats except OBP instead of average.

My league setting have been the same for the past couple years and i have been tracking how many of each stat each team has scored each week over that time. I then take the standard deviation of each category to figure out their value against each other. My idea is that the highest standard deviation would mean the lowest value to each marginal stat in that category. My results up to this point has been that RBI’s have the least value, so I assign each RBI to have a value of 1, and base the other stats from that. Runs are 1.15, homers 2.4, sb’s 2.6, and each point of OBP is .25.

Then, when applying that formula to the players, since OBP is a full year stat, I divide it by how much of the season has been played so far. (so after 1 game your obp is divided by 162, after 16 games divided by 10, at the end of the season divided by 1, etc.).

Does this methodology make sense? I’m no stat head obviously, but it made sense to me. My top 10 players as of Monday morning were (this is unadjusted for points above replacement at the each position:

Pujols
Kinsler
Longoria
Ibanez
Crawford
Jones
Utley
Pena
Youk
Hunter

Connecticut Mike
14 years ago

I have always been of the opinion that player raters are very flawed.

The yahoo player rater seems to be the same regardless of the scoring system in the league you are in.

For instance, if I am in a league with non standard categories (esp walks or OBP) it would stand to reason that some players would be more valuable in that context than the standard.
 
Further, positional requirements would seem to matter.  Brian McCann would seem to be more valuable in a two catcher league than a one catcher league, and would have very little value in a no catcher league (if that exists). 

I wish the player rater was a better tool, it could be very useful.

whitty
14 years ago

Briks, that method makes sense to me, although there should probably be some adjustment made for plate appearances when determining value for OBP—a guy with 30 PA/week is going to have more of an impact on your team’s OBP than someone with 15 or 20 PA/week.

The question, of course, is how you go about determining the correct value for plate appearances. If you just use total PA to date you’ll end up undervaluing players who have been on the DL earlier in the season or who weren’t on the roster. Maybe something like average PA per game spent on the 25-man roster would be useful. I have no idea how much work would go into calculating that—I don’t know if there’s an easily accessible source for games spent on the 25-man.

digglahhh
14 years ago

Good point, whitty.

The most overlooked aspect of influencing rate stats is volume. Ichiro has not only been a huge asset in BA because of his actual performance, but because of his approach as well. He hits atop the order, is durable, and never walks. In short, he’s led the league in Abs in 6 of his 8 MLB seasons. Per 162, he averages nearly 100 more ABs than Pujols (picked him b/c he’s one of the few who can post an Ichiro-in-is-prime like BA). So, if both guys were to hit .330, Ichiro’s would be far more valuable, as it would hold more weight as percentage of your team’s total ABs.

Similarly, in reverse, Adam Dunn rarely accumulates 550 ABs, even playing 155+ games. Due to all those walks, his .250-ish BA is a bit more palatable than some think, especially when paired with a 700 AB, .300-ish hitter. In fact, I had one team in 2007 which rostered both Ichiro and Dunn. While the mathematical mean of their .264 and .351 BAs is a .307, the extra weight of Ichiro’s ABs pushed their combined BA to .313.

Of course, if you are counting SLG, then the opposite applies for this pair. Ichiro will drag Dunn down.

I highly doubt that the generic player raters take this dynamic into account.

Tyson Trautmann
14 years ago

The other frustrating thing about player raters is that most don’t take into account league specific settings, position scarcity, etc.  For example my baseball league is roto 6×6 (we add Holds and OPS) and folks are still using YRank for deals even though it doesn’t take those categories into account.  Position scarcity can be an even bigger factor, my 12 team baseball league added a second catcher and most people didn’t even stop to consider the enormous impact the change had on catcher values going into the draft.

I’ve actually been working for several years now on a Dynamic Fantasy Sports Player Rater that takes all of these things into account (including volume, which the previous poster pointed out).  If you’re interested feel free to take a look, it’s free to use.