Beyond Moneyball: Player development, Part 5

(Previously Part 1, Part 2, Part 3, Part 4)

Finding prospects is the lifeblood of any professional sport. A significant portion of any major league baseball organization’s resources and activities is dedicated to this problem. From a job description for a baseball scout:

“This knowledge allows them to recognize young players who have astonishing athletic talent and skill.”

How do scouts actually recoginze such players? They do it through a process of objective-subjective analysis, being quantifiably objective by using a system that attempts to identify the critical components of baseball “talent and skill”—specifically how the player hits, (swings?), pitches (throws?), fields and runs.

Q: How do your scouts rate prospects?
A: We have basic generic things that apply to both position players and pitchers and then we have specific things for position players and pitchers. We grade position players on hitting ability, power, running speed, arm strength and fielding. We use a scale of 2-8 in each category to grade our players and come up with an OFP (Overall Future Potential). A total range of 40-80 covers major league prospects.

Forty is the minimum for our category of a major league prospect. We have fringe, average and definite prospect (categories). As the number gets higher, the better the prospect is. We grade pitchers on fastball, curveball, slider and other (if the pitcher throws a knuckler or split-finger). If a pitcher doesn’t throw an “other,” he gets graded on the three he throws. But that’s when a scout’s instinct comes into play. If a pitcher only throws two pitches, but the scout sees he has the arm instinct to develop a slider, he’ll grade him higher. A scout can move the number up if he feels the potential is higher. The generic qualities that apply to both position players and pitchers are things like aggressiveness, instinct, dedication, work ethic.

http://mlb.mlb.com/mlb/official_info/about_mlb/scouting_overview.jsp

The scouts’ numeric system is based largely upon their subjective opinion.

For example:

{exp:list_maker}(Bat Speed) the ability to swing the bat quickly
The ability to consistently hit the ball hard
Knowledge of the strike zone
The ability to turn on a major league fastball
The ability to hit breaking pitches
The ability to hit to all fields
The ability to make adjustments at the plate when fooled {/exp:list_maker}

They form opinions using their own personal experiences as opposed to specific formal training. The only real quantitative information is obtained from either the radar gun or stopwatch.

Pitchers Velocity:
{exp:list_maker}8: 98 mph +
7: 93-97 mph
6: 90-92 mph
5: 88-89 mph
4: 85-87 mph
3: 83-84 mph
2: 82 mph – {/exp:list_maker}

Which often leads to:

One of the amazing things about baseball talent is the fact that different, great scouts can see different things in different ballgames. Someone might see a young player when he’s especially hot while another scout may see the same ball player when he’s when he’s in a slump. How have you settled those kinds of differences?

Subjectivity is the very thing that sabermetrics and its application by major league organizations attempts to eliminate; i.e., differences in player evaluation caused by differences in scout/organizational opinions. The argument, statistics versus scouting, is really an argument of objectivity versus subjectivity. Both attempt to answer the same question: How does one identify top performers?

From Dr. John Sullivan, a human resources expert:

The Performance Differential Percentage Between Average and Top Performers

Whether your focus is on hiring or on retention, it’s critical that you measure and quantify the “performance differential” between average and top-quality performers in order to determine whether it’s worth the time and money to recruit and retain top performers.

The Container Store found a 300 percent differential, while Jack Welch at GE found a 500 to 1,000 percent differential. The Corporate Executive Board did extensive research on the topic and came up with a 1,200 percent differential across all industries, while Bradford Smart’s Top Grading research found a differential as high as 2,400 percent.

Alan Eustace, VP of Engineering at Google, concluded that there was a 30,000 percent differential between recruiting a top performer over an average one, a perception that may identify why Google is willing to invest so many resources into a recruiting model that is ultra-selective. As Google has proven, building a top talent “recruiting machine” requires unrelenting execution of a well-designed process and the resources to power it.

Defining a Top Performer

Some managers think top-performer status is related to educational level, others a candidate’s past experience. Intelligent managers often counter that individuals with the most qualifications do not always produce the greatest output.

What defines a top performer is an ability to consistently produce above-average results and to work effectively within the organization’s culture. Thus, top performers are not “stars” or even the most qualified; instead, they are individuals who produce the top 1 percent of results and innovations while they are in the job. A top performer in one organization may be a bottom performer in another.

http://www.ere.net/articles/db/F72234B64CEF412E9B5438E8D4D4D8DF.asp

From Wikipedia, the free encyclopedia:

Sabermetrics is the analysis of baseball through objective evidence, especially baseball statistics. The term is derived from the acronym SABR, which stands for the Society for American Baseball Research. It was coined by Bill James, who was among its first proponents and has long been its most prominent advocate known to the general public.

From David Grabiner’s Sabermetric Manifesto:

Bill James defined sabermetrics as “the search for objective knowledge about baseball.” Thus, sabermetrics attempts to answer objective questions about baseball, such as “which player on the Red Sox contributed the most to the team’s offense?” or “How many home runs will Ken Griffey Jr. hit next year?” It cannot deal with the subjective judgments that also are important to the game, such as “Who is your favorite player?”

It may, however, attempt to settle questions such as “Was Willie Mays faster than Mickey Mantle?” by establishing several possible parameters for examining speed in objective studies (how many triples each man hit, how many bases each man stole, how many times each was caught stealing) and then reaching a tentative conclusion on the basis of these individual studies.

Moneyball started a firestorm of controversy by pitting the sabermetrician versus the MLB scout. The desired results are the same, and decisions regarding prospects are reached the same way. Information and data are collected, the information is compared to a standard of excellence and then a decision is made regarding the value of the prospect.

Sabermetrics works on numbers, processed according to predetermined formulas. The challenge is finding the right numbers and the right formulas. But players do not develop in a continuous, linear fashion.

Scouts work on what they see, hear, feel. This information is processed based upon their abilities to “see” and distinguish what is important and what is not, and then, based upon their life experiences, to process that information and predict the future.

From “The Great Debate,” by Alan Schwarz in Baseball America , Jan. 7, 2005:

For the past two years, the scouting and statistics communities have feuded like members of rival families. Baseball lifers who evaluate players with their eyes are derided as over-the-hill beanbags who don’t understand the next frontier. Numbers-oriented people are cast as cold, computer-wielding propellerheads with no appreciation for scouting intangibles. Not surprisingly, the camps have grown so polarized that they have retreated to their respective bunkers rather than engage in open and intelligent debate.

The four participants were Gary Hughes, the Cubs’ assistant general manager and a scout for more than 30 years with many clubs; Eddie Bane, the Angels’ scouting director and a former top pitching prospect himself; Gary Huckabay, one of the lead analysts for Baseball Prospectus and a statistical consultant for the Athletics; and Voros McCracken, another top numbers man who also consults for the Red Sox.

http://www.baseballamerica.com/today/features/050107debate.html

A sampling from the debate:

ALAN SCHWARZ: To start out… how would you characterize the relationship between the scouting community and the statistics community?

EDDIE BANE: It is adversarial right now. Our guys, the so-called old-school guys, the thought is out there that we don’t know how to handle a computer and we wouldn’t know how to use that stuff. I’m very comfortable with a computer. Our people are very comfortable with a computer. We do have to drag some of our old-time guys through it. But the main adversarial thing is that some of our old-time guys are losing jobs that we didn’t feel they should be losing. It was due to cutbacks. Maybe the cutbacks were due to money or whatever. But we correlate it to the fact that some of the computer stuff is causing that. And we resent it.

ALAN SCHWARZ: Is there less of a need for scouts today, compared to 25 years ago?

EDDIE BANE: My point would be that the reason to have at least as many scouts, if not more, is when you’re drafting Marquis Grissom, as Gary Hughes did with Montreal from Florida A&M, he doesn’t cost $100,000 anymore, he costs a million maybe. And his stats at Florida A&M can be thrown out the window, because you need to see him in the two games a year that he plays against a pitcher that might have any ability whatsoever. That would be my reasoning to have more evaluators see this guy, because the bonus money is going to be astronomical on a guy like that if you have the guts to take him that high. Gary didn’t care what his stats were.

A player at UConn, his stats, compared to a guy that I’m watching in the Pac-10, mean almost nothing to me. I’m in the middle of a negotiation right now (with Jered Weaver) where a guy wants to compare our first-round pick’s stats to Mark Prior’s. And to me, there’s no correlation whatsoever.

VOROS McCRACKEN: My response to that would be that those sorts of things, say the difference between playing at Cal and playing at Florida A&M or UConn, you can study those sorts of things and find out what do the stats mean at UConn, what do they mean at Florida A&M, what do they mean at Cal? It’s not as if we treat a guy like Rickie Weeks, his stats at Southern—he had ridiculous stats at Southern, in a weak conference—the same as if he was playing for USC or Arizona State. Those kinds of things are studied. You can find out information.

Obviously, I don’t think it’s useful to draft players simply based on their stats. The issue I would bring up is that for all of these issues—level of play, the type of pitchers, his raw abilities like his speed, his strength, his size—these are all things that can be, to an extent, measured. Six-foot-one is a measurement. Five-foot-seven is a measurement. Hitters who are 6-1, do they turn out better than hitters who are 5-7, with similar stats at similar schools? These are the sorts of things that people can analyze, and I think it could provide useful information.

GARY HUGHES: All your statistics are going to tell you is what a guy has done. Somebody has got to make the decision on what the guy’s going to do.

VOROS McCRACKEN: I have no idea what the guy’s going to do. But my point would be, the scouts also have only a limited idea of what the guy’s going to do. He might do this, he might do that, he might be somewhere in the middle. What you’re trying to do is you’re trying to take the guys who you think have the best chance. I fully admit that you can’t tell the future via stats. My point is that scouting has that equal amount of unpredictability. You can only know so much. You’re scouts, you’re not fortunetellers.

GARY HUCKABAY: I think it’s important to understand that a lot of people have overclaimed what you can do by statistical analysis. It’s a tool. A car is a tool as well—you can use it to drive to the store, or you can use it to drive into a tree. I think there’s more of a dichotomy between good statistical analysis and bad statistical analysis. But all the information you can get your hands on—as long as you understand what it’s good for, and what its quality is—is always a good thing.

We’re all after the same thing here: We’re out to build a great baseball team. As long as you have X number of pieces of information, whether it’s performance data—a term I prefer to use rather than statistics, because these things are records of what happened on the field—and then also, if you’ve got people who have tremendous insight who are well trained, they know how to scout a guy, give me that information too. I want both of it. What I don’t want is someone going, “I want this guy because he had 120 RBIs.”

The difference between sabermetrics (statistical scouting) versus flesh and blood scouting is demonstrated in the following diagrams.

Sabermetrics attempts to capture the baseball talent area through statistical analysis. In essence, what sabermetrics is attempting to do is quantify the confluence of physical abilities, movement pattern (mechanics) and mental makeup of the player. How effectively it does this is illustrated by the amount of “player talent” that is captured by the analysis.

Sabermetrics is “blind” to the player’s actual physical abilities, movement patterns and mental makeup. But these are actually included in the sabermetrics analysis by attempting to capture the baseball talent that results from this combination of player attributes illustrated by the overlay of “sabermetrics analysis” and baseball talent.

It is important to understand that for any given player or group of players, the analysis is a “one-shot” deal. This is the allure of sabermetrics: Numbers don’t lie. But do they?

Using the same pictorial process, scouting can be illustrated as follows.

What each attempts to do is quantify the amount of baseball talent a prospect possesses. From the diagram you can see the problem that human nature introduces: No two scouts see the same thing. They cannot because what we see is based upon our life experiences, and, as with fingerprints, it is impossible for two people to have the same life experience.

Attempts are made to “average out” this variability through the use of cross checkers and the final decision-maker (the general manager?). This variability is what sabermetrics attempts to mitigate. But does it really?

This diagram represents sabermetrics analysis, which illustrates that the metric analysis depends upon who’s doing the analysis.

While numbers may have less susceptibility to interpretation, the importance accorded numbers in terms of predicting a player’s performance is subjective.

Ideally a “super scout” is the sabermetrician or flesh-and-blood MLB scout whose ability to “see” baseball talent would look like this:

Bill James, quoted in Moneyball:

Think about it. One absolutely cannot tell, by watching, the difference between a .300 hitter and a.275 hitter. The difference is one hit every two weeks. It might be that a reporter, seeing every game the team plays, could sense that difference over the course of the year if no records were kept, but I doubt it. Certainly the average fan, seeing perhaps a tenth of the team’s games, never could gauge two performances that accurately…. The difference between a good hitter and an average hitter is simply not visible, it is a matter of record.

Statistics cannot tell you what the player looks like. The physical attributes, height, weight, percent body fat, etc. can be measured. The value of the scout is in his ability to see how the body swings and throws, the ability to see how the player creates statistics. The scouting problem is, in its subjectivity: Beauty is in the eye of the beholder. Today there exists no absolute way to quantify “goodness or badness” of how a player actually uses his body to swing or throw. This is where a scout’s experience and “gut feelings” potentially conflict with the statistics; the scout’s “gut feeling” is not quantifiable except for the scout’s record for evaluating prospects.

From Moneyball, this excerpt:

Here’s an astonishing fact: Prince Fielder is too fat for even the Oakland A’s. Of no other baseball player in the whole of North America can this be said.”

Sabermetrics couldn’t tell A’s general manager Billy Beane, or should I say the stats Oakland used, that Prince Fielder has a very sweet swing, specifically in his ability to adjust to off-speed and breaking pitches. And, to be fair, the player Oakland did have its eye on instead, Nick Swisher, has done very well. But so has Fielder.

From “6-4-3, For What You Are About to Receive” by Gary Huckabay in Baseball Prospectus, Sept. 4, 2007:

Baseball analysis is dead.

Baseball analysis is dead because its utility has pretty much vanished. Analysis and information are only interesting and useful if they inform a decision, and even then, there really needs to be some sort of advantage or gain present relative to competitors in order for the time investment to be worthwhile. At this point in history, baseball analysis really has very little to offer on that front.

Any club that actually wants to use baseball analysis now to develop and maintain an advantage relative to their competitors has a tough task in front of them. They need to expand the scope of the data used for the analysis. They need to identify real changes that can be made in their operations if real phenomena are unearthed. They need to have people of sufficient skill to find these new discoveries. They need to develop a culture receptive to adopting the changes implied by this newfound wisdom. And finally, they need to find a way to keep other organizations from discovering the formula to their secret sauce. That’s a reasonable description of what clubs need from their search on the datafields of the game, and it’s precisely what baseball analysis cannot provide. Because baseball analysis is dead.

http://www.baseballprospectus.com/article.php?articleid=6666

Sabermetrics is a tool. And, as with any tool, its utility depends on the skill of the craftsman using that tool. And a good craftsman not only knows how to use each tool but also when.

Metrics: measures used to indicate progress or achievement.

Scouts typically use two numbers when grading a player, such as 4/6 or 3/5. The first number is the player’s current rating on the two to eight scale; the second is his projected professional baseball rating. Of course, those numbers are based on the individual scout’s opinion. When only one number is given it is almost always that scout’s projection of the player’s professional baseball potential.

Other than measurable parameters such as throwing speed, running speed and possibly bat speed, all the measurements are subjective.

This rating system might be termed “external metrics.” It attempts to qualify and quantify based on the result of body movement. The same can be said for sabermetrics except that it doesn’t take into account physical aspects of the movement. The next step in the evolution of player scouting and evaluation is what I would call “motor skill metrics”—the ability to evaluate players based upon how skillfully they use their body to swing and throw.

“The most difficult judgment of all scouting categories will be your appraisal of who will hit and who will not,” our scouting manual tells us. “Certain attributes are found in most quality hitters.”

Those attributes are:
{exp:list_maker}Strength
Starting the bat, generating bat speed
Full arm extension and follow-through after making contact
Head stays on ball
Lack of fear
Short stride
Top hand is evident upon making contact and follow-through
Head of bat does not lag
Aggressive
Short stroke, yet ball jumps off bat
Bat goes to ball {/exp:list_maker}

Complicating matters further, not all good hitters meet these criteria. “Some hitters are natural hitters. They may do some strange things, but they can and do hit,” the manual says. “Don’t be concerned with their uniqueness. They can just hit.”
(Baseball America, “The Scouting Department Diary of a Wannabe Scout,” http://www.baseballamerica.com/online/columnists/scout0928.html)

From Wikipedia:

Biometrics (ancient Greek: bios =”life”, metron =”measure”) is the study of methods for uniquely recognizing humans based upon one or more intrinsic physical or behavioral traits….

Biometric characteristics can be divided in two main classes: Physiological are related to the shape of the body. The oldest traits, that have been used for more than 100 years, are fingerprints. Other examples are face recognition, hand geometry and iris recognition. Behavioral are related to the behavior of a person. The first characteristic to be used, still widely used today, is the signature. More modern approaches are the study of keystroke dynamics and of voice. Other biometric strategies are being developed such as those based on gait (way of walking), retina, hand veins, ear.

The diagram shows a simple block diagram of a biometric system. The main operations a system can perform are enrollment and test. During the enrollment, biometric information from an individual is stored. During the test, biometric information is detected and compared with the stored information. The first block (sensor) is the interface between the real world and our system; it has to acquire all the necessary data. Most of the time it is an image acquisition system, but it can change according to the characteristics desired.

The second block performs all the necessary pre-processing: it has to remove artifacts from the sensor, to enhance the input (e.g. removing background noise), to use some kind of normalization, etc. In the third block features needed are extracted. This step is important, as the correct features need to be extracted in the optimal way. A vector of numbers or an image with particular properties is used to create a template. A template is a synthesis of all the characteristics extracted from the source, in the optimal size to allow for adequate identifiability.

If enrollment is being performed, the template is simply stored somewhere (on a card or within a database or both). If a matching phase is being performed, the obtained template is passed to a matcher that compares it with other existing templates, estimating the distance between them using any algorithm. The matching program will analyze the template with the input. This will then be output for any specified use or purpose (e.g. entrance in a restricted area).

A scout is a very sophisticated player-biometric identification system employing the following functions:

• Sensor. This is the scout’s eyes, ears, nose, etc. It is everything that takes in raw information regarding the player: How the player looks, moves, swings, throws, fields, carries himself.
• Preprocessing. Certain criteria must be met before the scout will engage in a specific biometric analysis. Typical are preset thresholds of performance, such as throwing speed, 60-yard dash time or hitting or throwing statistics.
• Feature extractor: Specific player attributes such as swinging and/or throwing—attributes perceived to be of greatest importance in terms of long-term projection.
• Template generator: Player attributes are assembled in a standard format for comparison.
• Stored templates: Visual images that are characteristics of successful players—physical characteristics and how they swing, throw and move.
• Matcher: A comparison is made between the prospect template and stored templates. The scout determines how they match based on his like experience.
• Application device: If there is a match, the scout makes a recommendation.

The difference between art and science:

In their most intellectual form, all things are Art. In their most practical form, all things are Science.
In this statement, I am taking the term Science to mean a process of following a set of predetermined guides in order to achieve a result. An Art, by contrast, takes the power of metaphor and independent thought inherent in the human mind to bear in order to create something.

A model, assembled from a kit by following a detailed list of instructions, is not a work of art. That is not to say it is not something to be proud of—and indeed the directions always need some degree of interpretation, substitution, etc.—which could be construed as Art. In other words, anything that could be theoretically done by a computer, given enough time, is a science rather than an art.

The greatest of scientists and mathematicians, the ones who come up with completely novel approaches to problems, are truly artists. But the rest are mere scientists, following rules discovered by greater minds in an attempt to be like them. http://everything2.com/index.pl?node_id=683344)

The present state of player development is more an art than science. Typing the words “the art of pitching” into Google returns hundreds of links referring to pitching as an art. Developing a high-level pitcher is historically perceived as finding ways to deceive the batter. You often will encounter the word “style” in baseball. Typically it is used to describe differences in the way that players throw or swing, an attempt to explain the unexplainable. What’s unexplainable is why a player who is successful at deceiving the batter or defeating the pitcher looks different than the accepted definition of good pitching or hitting mechanics.

Baseball is very technique-driven. USA Today rated hitting a baseball as the hardest thing to do in sports. But only over the last 30 years have there been scientific attempts to better understand how the body swings and throws.

“The Science of Hitting,” written by Ted Williams and John Underwood, and Bob Shaw’s book entitled “Pitching; The Basic Fundamentals and Mechanics of Successful Pitching” appeared at about the same time (1970, 1971). They were the first real attempts at hitting and pitching science, defined by predictability of cause and effect. Shaw’s was the first widely read book to use the word “mechanics” to describe the throwing process.

“Mechanics” (in physics the term is used to describe the study of the motion of objects) caught on and now is used to describe anything and everything having to do with how the body swings or throws, hits or pitches. Instructional literature on pitching and hitting mechanics is based almost entirely on individual opinion about the hitting and pitching process. Because there is no peer-reviewed consensus on what constitutes good mechanics, there can be no such thing.

Pitching mechanics is based upon a specific “look.” Good mechanics are often described in the context of looking controlled, looking like you are throwing free and easy, looking like you are balanced, etc.

The “right” science may tell us how to make better decisions and solve problems.

Motor learning and theory is a scientific approach to understanding how voluntary movement occurs. Swinging and throwing are voluntary movements. Because of their ballistic nature they are considered autonomous—theyhappen without much thought.

The field of motor learning and control is not a precise science. In many respects, it is still in its infancy. But it is the only game in town with respect to understanding how skills are developed. And more importantly the science of motor learning and control provides a formal framework to build more effective instructional and coaching information and methods.

One prevailing theory of learning is called the two-stage model of motor learning. The first stage is acquiring the basic movement pattern. The second stage is learning how to use that pattern under all conditions.

How this applies to the major league draft is quite straightforward. Major league organizations try to identify players who have established movement patterns that are capable of dealing with all game conditions. Motor learning terminology calls these constraints. What characterizes high-level swinging and throwing performance are movement patterns that are constraint-independent; i.e., minimal change in the swing or throw movement is required to handle any condition.

For a hitter, this means that no matter what the game situation, the movement pattern of the swing required to successfully hit the ball requires a minimal amount of change. The same applies for the pitcher attempting to get the batter out.

This principle also applies to golf. The goal is to take the same swing as often as possible. The constraints are the location of the ball on the golf course. The golfer deals with these constraints by selecting the appropriate club. Different clubs allow the golfer to approximate the same swing no matter what the constraint.

This same principle works against player development.

Why? Because if you have selected the right player with respect to their movement patterns for swinging and throwing, player development becomes a process of maturation, being able to effect this “right” movement under all conditions. Player maturation is the domain of coaching, because 95 percent of professional player development is to provide an environment, in the form of game competition, for player maturation. The hope is that players will develop the knowledge and experience necessary to deal with the increased demands of major league competition.

The player who has mechanical deficiencies must engage in a swing and throw adjustment process which takes place through trial and error. Game performance, success or failure, becomes his instructional process. And his development success is often in spite of well-meaning yet inappropriate coaching and instruction.

The “metrics problem” is how does one determine constrained-independent mechanics? And if you are attempting to develop players, how does one differentiate between constrained-independent mechanics versus constraint-driven mechanics?

Next time: The search for predictability.