Revisiting Which Statistics Matter for Evaluating Collegiate Quarterbacks

A few years ago here on numberFire, we looked at the college statistics of quarterbacks drafted in the first round to see if there was a formula for success. The overall conclusion was that, yes, college stats could be used in evaluating how they would fare in the pros.

Although there was some actionable information in there, it still left a bit of an unsatisfied taste in my mouth. This study required placing quarterbacks in arbitrary "tiers" based on their production in the NFL. It was loosely based on numberFire's Net Expected Points (NEP), but it also required a few judgement calls on my part. It's hard to get around this, but it still bothered me a bit after the fact.

Over the past few years, these trepidations have gotten even more profound. I placed quarterbacks that averaged 15.00 Total NEP per season in the successful tier of quarterbacks. However, because of changes in scheme and the way penalties are called, a quarterback worth 15.00 Total NEP over the course of a season really isn't quality at all anymore. That would have been the 32nd best total among quarterbacks this year.

It's time to shake things up a bit.

That's what we're going to do today. Instead of evaluating quarterbacks based on their raw Total NEP, we're going to look at their year-to-year rankings among other quarterbacks in that category. Using this strategy, hopefully we can get some more definitive takeaways in order to evaluate future incoming rookie signal-callers.

In case you're new to the site, here's how NEP works. On each play, there's an expected number of points the offense will score on its current drive. A positive play (such as a five-yard completion on 3rd-and-4) will increase that, while a negative play (such as a five-yard completion on 3rd-and-9) will decrease it. NEP tracks the fluctuations over the course of the year and gives us a measuring stick of efficiency for both teams and players.

First, let's go through which statistics we'll be studying and the method for the study. Then, we'll go into the results to see what information comes from this line of thinking. Finally, we'll wrap things up by spinning that forward to see which college stats matter most for quarterbacks (if any) and what we should be looking for each year.

The Method

From 1995 on, there have been 52 different quarterbacks selected in the first round. For each quarterback, I noted which overall pick they were, the number of games they played in their collegiate career (defined as the number of games in which they attempted at least 10 passes), the conference in which they played, and the passing efficiency rating, adjusted yards per attempt, and Adjusted QBR they posted in their final season at school. Adjusted yards per attempt (AY/A) lumps touchdowns and interceptions into a yards-per-attempt-esque number, and Adjusted QBR is a stat created by ESPN that accounts for rushing abilities and is adjusted based on opponent.

For every quarterback, I looked at the percentage of seasons in which they had finished in the top 5, top 10, and top 15 in Total NEP. Only seasons in which they recorded at least 200 drop backs were included so as to try not to taint the numbers much with injuries.

The quarterbacks drafted over the past two seasons were excluded from this because of issues with sample sizes and career arcs. For what it's worth, of those five, only Jameis Winston has logged a top-15 finish so far, finishing 14th during his rookie season.

By basing the analysis on seasonal rankings as opposed to raw numbers, we can better account for the quarterback's competency relative to his era. But could it help us better decide which numbers are relevant? Let's take a look.

College Stats Matter

Before we dive into the collegiate stats, it seems important to note something first: first-round quarterbacks bust. A lot.

Of the 47 quarterbacks drafted in the first round from 1995 to 2013, 20 of them never finished a single season in the top 15 in Total NEP. These teams spent heavy capital on these players, and over two-fifths were never able to finish in the top 15 at their position.

That said, we have seen that most successful quarterbacks are drafted in the first round, so we can't simply write the position off. Instead, let's try to find more efficient ways to separate gems from duds.

Let's break the quarterbacks into those two different groups first: those who did have at least one top-15 Total NEP finish and those who didn't. The table below shows the split between the two groups with the average collegiate stats of the quarterbacks in each category.

Just in case you're worried about sample sizes, I've also included this table which shows the median totals as opposed to the averages. Either way, it's the same story: college stats are relevant.

The average collegiate AY/A of quarterbacks with at least one top-15 finish was 9.6 percent higher than those with none, and the median was 12.0 percent higher. I'd say that's significant.

The biggest splits here, though, came in the games played and draft pick categories. This is in line with what we saw two years ago in that quarterbacks with less collegiate experience or picked later in the first round carry a greater risk. The draft pick gap makes complete sense as scouts know what they're doing, but the games played discrepancy should raise red flags about draft prospects with fewer games under their belt.

When we're sorting quarterbacks into two categories such as these, it's best to have skepticism about the numbers you see. I wanted to dig a bit deeper just to make sure we weren't getting any bad data that was skewing the presentation.

To do that, I instead looked at the percentage of seasons in which quarterbacks ranked in the top 10 and top 15 of Total NEP. I then found the correlation coefficients between those numbers and their collegiate stats.

The intent here wasn't to see particularly high correlation numbers. It was just to validate that there was a non-zero relationship between the pieces of data to make sure we're on the right track here. It could also give us some insights into which statistics were more valuable than others. Those correlation coefficients are presented in the table below.

Once again, it appears that our two most useful categories are games played and draft pick. We definitely should also consider passing efficiency rating and AY/A, too. Total QBR appears to be the one category that lagged behind a bit.

This table -- combined with what we saw above and two years ago -- is enough to convince me that we should be using collegiate stats to evaluate quarterback prospects. The next question, though, becomes what are the optimal numbers in each category? Let's take a peek.

The Collegiate Blueprint

The main flaw with the first two tables is that a quarterback only needed one top-15 finish to count in the optimal grouping. This means that guys like Jason Campbell, David Carr, and Vince Young would be success stories. They had their moments, but that's not likely what you want out of your first-round pick.

A realistic, lofty goal for a quarterback taken in the first round would seem to be a guy who is consistently one of the 10 best in the league. If you can find this, you'll be putting yourself in a great spot to contend on a year-by-year basis.

In our sample of 47 quarterbacks, 10 of them were able to finish in the top 10 in Total NEP in at least half of their relevant seasons. This includes guys like Aaron Rodgers, Peyton Manning, Cam Newton, and Ben Roethlisberger -- championship-caliber quarterbacks who would make any team blush.

To try to develop our blueprint, let's look at how those 10 quarterbacks profiled coming out of college to see if we can glean anything. Because of the limited sample size, the numbers below are the medians of the data, comparing these shininig pillars of efficiency to the rest of the pack.

If you were looking for a quick blueprint, I think we've found it here.

To me, the most striking piece is the difference between the passing efficiency ratings. We didn't see as large of a gap in the initial section, but it does show up when we narrow the focus to the players with the highest upside.

Even with this, the percentage gap between the top group and the rest was still larger in AY/A than it was for passing efficiency rating. Therefore, I think I would give mild preference to a player with a higher AY/A than one with a high passing efficiency rating if all else were equal. The split is big enough, though, where both should be considered.

The one aspect of this study we haven't addressed yet is the conference in which the player played his college ball. Considering the 2016 draft class includes several candidates for which this issue could arise, we should address it before wrapping up.

To give a broader perspective, let's loosen the guidelines a bit. Instead of looking at quarterbacks who were in the top 10 half the time, let's lower that to one-third of their seasons. This expands our sample to 13 quarterbacks as opposed to 10, perhaps giving us better insights into whether the player's conference matters.

Here are the conferences where these 13 played collegiately. These are all based on the conference the school was in during the player's final year at the school.

Yes, even the Southwestern Athletic Conference made the list with Steve McNair repping Alcorn State. There was Daunte Culpepper at then-independent Central Florida, with Roethlisberger and Chad Pennington from the MAC. I don't think you need to worry much about conference.

You'll note that the Big Ten is not on that list. That's only because the conference hasn't had a quarterback taken in the first round since Penn State's Kerry Collins in 1995. There have been 35 instances since 2000 in which a former Big Ten quarterback finished in the top 10 in Total NEP, more than any other conference (the SEC was second with 23). That's not an obstacle, either.

So, we don't need to factor conference into our blueprint. What should we include?

Based on this data, I would say we would want a quarterback who has played -- at minimum -- three seasons as a starter. That would get them somewhere in the range of 36 to 40 starts, which is the sweet spot for the quarterbacks in the sample.

For their efficiency, you'd likely want to see a passing efficiency rating of 165 with an AY/A of 9.3 or higher. There have been quarterbacks who have been successful with worse numbers, but they are far less plentiful than the disappointments.

As far as predicting whether or not that player will be a successful quarterback, combining all of these factors with their draft slot seems to be the most effective method. Players taken outside of the top 10 are far less likely to be regulars in the top flight of quarterbacks than their highly-drafted counterparts.

There's no foolproof blueprint for predicting whether a quarterback will be successful. You'll have your Sam Bradford types, who post quality college numbers but fail to succeed in the pros, and you'll also have guys like Matt Ryan, who find success despite sub-par collegiate stats. Still, considering both efficiency and experience can better help you decide who is best set up to man the franchise.