Building UZR and Catch Probability for the KCL

Author: Noah Lippman: Data Science Major at Indiana University and Normal CornBelters Baseball Analytics Intern

0. Introduction

Defense has long been one of the most important aspects of a player’s ability, but has also been the most difficult to properly evaluate. For a long time, errors and fielding percentage were the only ways that fielders were evaluated via statistics. Errors are an inherently flawed stat because they are determined subjectively and do not account for range, which is the most important part of playing defense. In the early 1980s Range factor was introduced, which simply evaluated how many plays a player is responsible for per inning. Range factor was a step in the right direction, but it still had fundamental issues. A shortstop on a team with a lot of ground ball pitchers will likely have a much higher range factor than a shortstop with the same defensive ability on a team with a lot of fly ball pitchers, simply because they get more opportunities. UZR was then developed in 2002 by Mitchel Lichtman to estimate the probability of each ball being caught and thus paint a fuller and more accurate picture of a player’s defensive ability.

1. What is UZR

UZR is an advanced defensive statistic that uses machine learning to estimate the probability that a ball is caught by each defensive position. Each position is assigned a probability of catching a batted ball; If the ball is dropped, each position is penalized its projected probability of catching the ball. For example, if a batted ball is assigned the following positional catch probabilities: (1B: .1, 2B: .1, 3B: 0, SS: .005, LF: 0, CF: .1, RF: .695) and the ball is not caught, the positions will be penalized .1,.1,0,.005,0,.1, and .695 respectively. One might assume that a player earns the complement of their catch probability 1 – P(CatchByPosition); however, UZR instead credits the player who made the catch with 1 – P(CatchOverall). This distinction is designed to minimize added credit for ball hogging. For example, if a ball is hit straight to the left fielder with a 7-second hang time, the left fielder will likely be given a catch probability greater than 95%; the centerfielder, however, will be given a very small catch probability. The centerfielder is not assigned a low catch probability because of their proximity to the ball, but rather the fact that the leftfielder will almost always make the catch. If we gave the centerfielder credit for the complement of their probability of making the catch (1 – a very small number), we would be giving them credit for hogging a ball from the leftfielder instead of making a truly difficult catch. It is important to note that UZR is traditionally expressed in terms of runs saved. Because we do not have the KCL’s exact run environment, we will express UZR in the scale of outs above average.

2. Data Collection and Feature Selection

2.1 Data Collection

For this project, I used YakkerTech data that was collected for each KCL game at the Corn Crib this summer. After collecting data for each game, I then hand-annotated each dataset with the positional lineups for each game. I also marked which position was responsible for recording each out on balls in play. Throughout the summer, I noticed Yakkertech struggled to provide highly accurate distance measurements as distances were often heavily underestimated or overestimated as compared to where they landed or were caught. To be fair to Yakkertech, distance is impossible to measure without radar because of the effect of weather on the flight of the ball. Due to this issue, I decided to train one model that used X coordinate, Y coordinate, and hangtime, and another model that used Exit Speed, Launch Angle, bearing (horizontal exit angle), and hangtime.

2.2 Feature Selection

I chose to use multi-class log loss as my evaluation metric for choosing the best model. Multi-class log loss takes the average of the natural logs of each true class’s predicted probability. For a single observation, the log loss is where is 1 if the class is the true class and 0 otherwise, and where is the class’s projected catch probability. If our model predicts that the centerfielder has a 40% catch probability for a certain ball and they end up catching it, then the log loss value for that ball will be -1 * (1 * ln(.4)) = .9163. If our model instead predicts the same ball to have a 95% catch probability by the centerfielder, then the log loss value for that ball will be -1 * (1 * ln(.95)) = .0512. This evaluation metric penalizes our model for assigning low probabilities to the true class.

After running a grid search cross-validation on both the exit velo/launch angle model and the coordinate/hangtime model, I found them to be extremely close in accuracy. The coordinate model had a best log loss of .604, while the exit velo/launch angle model had a best log loss of .609. These values led me to choose the coordinate model. This was an easy choice because the accuracy was the same as the exit velo/launch angle model, and coordinates and hangtime are much more interpretable and easier to display graphically. If we assigned all 8 outcomes (No catch, caught by 1B, caught by 2B, caught by SS, etc.) the same probability of .125 for each ball in play, we would get a log loss of 2.08. Both models are much better than randomly guessing who makes the catch. A log loss value of .6 means that the model, on average, assigns a probability of to the correct class (0 or the position that makes the catch). A perfect model would have a log loss of 0 and would assign a value of 1 to the correct class for each ball in play. Of course, nothing in life, especially baseball, can be perfectly predicted, and I am pleased with a value of .6 as it means that the defense model can be used as a viable tool.

As displayed in the graph above, all features are relatively important to determining catch probability. A model that uses X coordinate, Y coordinate, and hangtime as features is clearly the best choice both in terms of interpretability and accuracy.

2.3 Final Model

X coordinates, Y coordinates, and hangtime are highly interactive variables. An X coordinate of 0 (2nd base, home plate, centerfield) changes from a routine flyout to the centerfielder to a sure base hit when you change the Y coordinate from 300 to 100 and the hangtime from 5 seconds to 2 seconds. The relationship between X and Y coordinates and the probability that a certain position catches a ball is also nonlinear. A centerfielder’s catch probability for an individual ball will be negatively impacted if the X coordinate moves too far in a positive or negative direction. Additionally, a centerfielder’s catch probability for an individual ball will be negatively impacted if the Y coordinate is too small or too large. Due to the interactive nature and nonlinearity of the predictor variables, I chose to use XGBoost as my model. XGBoost excels at handling interactions between variables as well as the nonlinearity of predictor variables. I used grid search cross-validation to select the best model and hyperparameters. Grid search cross-validation is a brute force algorithm that tests models of all combinations of a selected number of hyperparameters. After running the grid search cross-validation with possible values of max depth = 2, 3, 4, or 5, learning rate = .01, .1, .2, or .3, and n_estimators = 50, 100, or 200.  I chose to fit an XGBoost model with a learning rate of .3, a max depth of 2, and 50 estimator trees. The final model is fed a play with an X coordinate, Y coordinate, hangtime, result, and the position that made the catch, or 0 if no catch is made. The model then outputs a dictionary with UZR values for each position. If the play result was a home run, the outputted dictionary will be all 0s; if the play result was a catch, then the dictionary will be all 0s except for the position that caught it, which will be given 1 – total catch probability. If the play is a hit or error, then each position is assigned -1 * the positional catch probability.

3. Graphics and Shiny App

One goal I had for this internship was to strengthen my abilities in shiny web development. After completing the catch probability model, I decided to develop a BaseballSavant style web app that allows users to view the overall ability of a defender, the ability of a defender in different directions, the ability of a defender at different positions, and the catch probability of specific plays.

Determining Average Fielder position

One limitation of Yakkertech is that it does not provide us with data on the positioning of all fielders on the diamond for a given play. Due to this limitation, I decided to estimate the average X and Y coordinate of each position by finding the coordinates where the position had the highest catch probability on balls with a hang time of 3 seconds. This approach is not perfect and is a very rough estimation of the average coordinates of each defensive position. Individual positioning also varies heavily among different players. All positioning graphics should be viewed as more of a best estimate as opposed to an exact measurement.

Catch Probability by HangTime and Direction Chart

This chart displays all batted balls that a fielder was deemed responsible for (their position’s catch probability was greater than 5%) during the KCL season. The teal background represents the catch probability of balls hit at certain distances from the position’s average starting coordinates and the hang time of the batted ball. The teal catch probability background is based on a different model than the one previously discussed and was built using a logistic regression model with the log odds formula:

and the probability formula of:

It is important to note that the original XGBoost model produces probabilities for each position and thus weighs the value of a play against the average range of each position. Center fielders will be given less of a boost for covering a lot of ground than right fielders covering the same amount of ground because center fielders generally have more range than right fielders. Contrary to the XGBoost model, the hangtime/distance from average start model does not account for position. This chart is meant to show overall range in relation to all defenders in the league as opposed to specific positions.

Catch Probability Interactive Spray Chart

This chart shows the overall catch probabilities for all batted balls with a similar hang time. This spray chart uses data from the original XGBoost model and is specific to each position. To avoid displaying ball-hogged catches as more difficult than they truly are, I chose to make the blue background bubbles representative of the total catch probability instead of the positional catch probability. This graph is also interactive with the previously mentioned catch probability by hangtime/distance chart. By clicking a point in the hang time/distance chart, the catch probability spray chart will be generated for that point. Because these two charts use different models, it is possible that a point clicked in a certain zone i.e., (.2 – .4) in the hangtime/direction chart, may lie in a different zone in the catch probability spray chart.

UZR table

This table displays a player’s total UZR for each position and in each direction (relative to estimated league average positioning). Grayson Schnierle’s 2.81 UZR means that he caught 2.81 more balls than expected during the 2025 KCL season. The table also shows how many outs the player would be on pace to add in 100 chances. As you can see from the table, Grayson Schnierle accumulated the majority of his defensive value playing in the corner outfield and was an average to plus defender in all directions as an outfielder. Schnierle was also very valuable in his brief time at 2nd base, with approximately half an out above average in just 6 innings. Schnierle’s defensive versatility and overall range resulted in him grading out as the one of the best overall defenders in the KCL in terms of UZR.

Range Percentile Chart

Raw numbers are often difficult to understand without context. Obviously, 1 UZR in a certain direction is better than -1 UZR in the same direction, but by how much? The range percentile chart illustrates what percentile the player’s range is in each direction (from the batter’s view). As we see from the chart above, Grayson Schnierle is elite in almost all directions, especially to his left, where he grades out in the upper 90th percentile of KCL defenders. Schnierle does however, struggle on balls to his back right. This does not necessarily mean that Schnierle lacks range over his right shoulder or that he is the rangiest player in the league going to his left. We do not know exactly where Schnierle positions himself for every pitch, so it is very likely that his apparent lack of range over his right shoulder and extreme range to his left is because he is positioned further right from the batter’s view than the average fielder.

Defensive Spray Chart

The defensive spray chart shows all plays where the selected fielder had at least a 1% catch probability. Underneath the graph, the user can filter both the hang time of balls and what position the player was playing, as shown in the video above. The Dark Blue Triangles illustrate the estimated league average starting position of KCL defenders at the specific position.

4. Limitations

The most obvious limitation of the XGBoost model is that it does not factor defensive positioning into each play’s catch probabilities. This issue is slightly minimized in the KCL because defenders often play in very similar positions for different hitters. This means that although the accuracy of each batted ball’s catch probability vector will suffer from a lack of positioning data, over a large enough sample size, it should even itself out. For example, If a player is shifted heavily to the right compared to league average, they will likely be penalized heavily for balls to their left because they are farther than the league average defender would be. While the player will be penalized for missing catches the model views as easy, the player will also be rewarded for balls to his right that the model deems difficult because he is starting in a more advantageous position for those balls.

The model is also trained on only 1162 balls in play, which is a small sample size for a model that is attempting to predict the probabilities of 8 different classes (all defensive positions besides catcher and pitcher and the ball dropping). Throughout the summer, I saw the log loss of the model drop as I annotated more data. Another X number of games may have improved the accuracy of the model.

The model also only calculates the catch probability of balls in the air, while UZR and OAA (Statcast’s defensive model) calculates the out probability of all batted balls, including ground balls. OAA uses sprint speed in their out-probability calculations for ground balls because fast players have a much better chance of beating out ground balls than slow players. Unfortunately, Yakkertech does not track sprint speed, which greatly hinders the accuracy of a potential groundball model. The lack of a ground ball model means that infielders will only be evaluated on line-drives and pop-ups. My model, therefore, does not accurately describe an infielder’s overall defensive ability.

5. Conclusion and Next Steps

I am proud of the model and website that I developed. Considering the often-fleeting accuracy of the Yakkertech data and the small sample size, I believe that I have built a viable and accurate catch probability model.  With that being said, the XGBoost model likely needs a larger training dataset and Statcast quality data to increase its accuracy and usability. Due to these inherent, unavoidable limitations, players should not be evaluated on their defensive ability based on my model. Yakkertech often did not read balls when it was too dark out, so many plays may be excluded from a player’s overall defensive rating. As mentioned earlier, Yakkertech also often miscalculated the X and Y coordinates of balls in play. This means that a player could be heavily penalized because Yakkertech said a ball dropped right in front of him, even though it landed 40 feet in front of him. A player could also be given false credit for catching a ball that Yakkertech calculated as far away, even though it was a routine catch. Players also played a maximum of 28 games, which is far too small of a sample size to make a serious judgment of a player’s defensive ability. Piper Slowinski said it best in her article about UZR for FanGraphs: “Beware of sample sizes! If a player only spent 50 innings at a position last season, it’d be a good idea not to draw too many conclusions from their UZR score over that time. Like with any defensive statistic, you should always use three years of UZR data before trying to draw any conclusions on the true talent level of a fielder.”

My next goal is to develop a ground ball model. A ground ball model would be heavily constrained by the lack of sprint speed data. One potential workaround for this is using the batter’s stolen bases and even air UZR as parameters in the model. Both statistics are at least slightly correlated with sprint speed, so they may improve a ground ball model.

I am extremely grateful to Jarrett Rodgers and the entire Normal Cornbelters organization for the opportunity to work with the in-depth data that Yakkertech provided. This project helped me grow my machine learning, data visualization, and Shiny web development skills. I would also like to thank all the other analytics interns: Charlie Keglovitz, Max Quirk, Ari Goldberg, Wyatt Sherman, and Cameron Cischke for making this experience even more rewarding.

6. Code and Website

• Code: https://github.com/NoahLippman/Kernelytics-Projects/tree/main/KCL_Defense
• Website (Best in full screen): https://nlippman.shinyapps.io/kcl_defense/

7. Sources

• Slowinski, Piper. “UZR.” Sabermetrics Library, library.fangraphs.com/defense/uzr/.
• Lichtman, Mitchel. “The Fangraphs UZR Primer.” FanGraphs Baseball, 7 Jan. 2013, blogs.fangraphs.com/the-fangraphs-uzr-primer/.
• https://baseballsavant.mlb.com/