Why Our Preseason Projections Work: An Open Letter To Profane Commenters

August 23, 2012 - by Tom Federico

Whenever we post our NFL projections and college football projections, which are data driven and incorporate no subjective opinions of our own (save the extremely rare manual fudge for huge impact scenarios like Penn State this year), we tend to get a lot of comments on them. Many of these comments are constructive, but we also invariably see a few of the following routines:

  • Hard core fan of Team X checks out our preseason predictions
  • Hard core fan sees we’re projecting Team X to get worse this year (maybe a lot worse, sacrebleu!)
  • Hard core fan is convinced Team X has made brilliant moves in the offseason to improve and fix problems that were evident last year.
  • Hard core fan leaves a comment letting us know that we are stupid worthless idiots, perhaps with an expletive thrown in for good measure, as well some highly suspicious spelling. (Real example from the pride of Chicago: “ur f–king stupid thinking the bears goin 7 and 9 go f–k urself”!)
  • We are also occasionally accused of having a deep personal vendetta against Team X, and apparently programming our computers to hate them too. (Example #2: “You must be a Packer fan!”)

This stuff provides some nice comic relief during a hard day of number crunching. We don’t take it personally, as we’ve clearly had to develop some pretty thick skin to do the sports predictions thing for over a decade now.

We certainly understand that when devoted fans are concerned, it’s the nature of the beast. Fans are highly passionate, and several of those fans also enjoy taking large doses of steroids before they start trolling the blogosphere.

However, some more insight into our process would probably help to reduce some of this vitriol. In short, we don’t have it in for you, profane commenter dude. We probably don’t even care a lick about your team, which actually may be even worse in your mind, but that’s another story.

Let’s be clear though. Unless you went to UC-Berkeley, it’s extremely doubtful that any key decision maker at Team Rankings actually hates your team.

3 Realities You Need To Come To Grips With

With that said, profane commenter dude, you need to be aware of three realities:

  1. Every year, some teams get better and some get worse. Your team is not always going to get better, even though almost every year you’re convinced it will.
  2. Every prediction system has strengths and weaknesses. You need to critique the performance of the system as a whole, and not just one of many data points (i.e. just your team’s prediction)
  3. Over the long term, we’re probably a lot better than you at predicting sports, profane commenter dude. I didn’t say we were amazing, or the best in the world, or always right. Just better than you.

Why Our Predictions Are Usually Solid, Despite Some Flaws

Here’s some brief insight into how we approach preseason predictions, using college football as the lens.

As you would expect from Team Rankings, we crunch a ton of hard data. Our analysis is largely based on looking at historical variables (turnover margins, recruiting class strength, production lost to graduation/the draft, QB strength, etc.) and trying to establish a statistically significant connection between those data and a team’s performance during a season. We not only look at how a team did last year and its schedule strength in the coming year, but also at some variables that go back several years.

In terms of predicting season wins and losses, this fully algorithmic process has both pluses and minuses, compared to the likely predictive skills of an intelligent hard core fan who watches a ton of football.

Negatives

There are certainly some dynamics (e.g. a drastic change of coaching staff, game strategy, or offensive/defensive schemes, or major injuries/personnel changes such as Tyrann Mathieu getting the boot from LSU) that are very difficult to model quantitatively and with a high level of confidence. Provided they can remain fairly objective, a relatively smart superfan may have an edge over our models at projecting whether these types of factors will have a slight, moderate, or major impact on a team’s future success.

Positives

Where we do have enough information gathered to model factors with decent confidence (e.g. how good a team has been in the recent years preceding the upcoming season), our ability to process that information is far, far superior to the average hard core fan.

For example, an LSU superfan may be better than our computers at understanding how Mathieu’s dismissal will affect the team’s performance, based on a deep knowledge of how the LSU coaches may alter their defensive schemes, the quality of the replacement player(s) who will see more playing time, and even the likely morale impact on the team. But our models will kick that fan’s butt at quantifying the relative strength of all of LSU’s non-conference and SEC opponents, and then coming up with a probability distribution of how many games LSU is likely to win this season.

Doing that complex calculation process well is a lot more important than accurately assessing one lost defensive player’s impact.

Secondly, a surprising amount of conventional wisdom about projecting a team’s end of season record/performance ends up being just plain wrong. We’ve examined lots of different factors that many fans would assume make an impact in projecting future results (e.g. Rivals.com ratings of each team’s recruiting class for the past two or three years in college football) and as it turns out, many of these instinctively important-sounding factors just don’t matter as much as most people think they do when you actually crunch the data and control for them.

So while we may be guilty of not incorporating some data that may indeed make a difference in a team’s future success, our approach also doesn’t shoot itself in the foot by drawing false conclusions from meaningless data. Many humans and unsophisticated projection systems regularly make that mistake.

Objectivity Can Hurt Sometimes

On balance, we expect our preseason projections to be much better than the vast majority of projections that non-computer-savvy humans come up with. However, we also know and expect that a few of our preseason predictions could be way off, primarily for teams that carry high uncertainty due to scenarios like the ones mentioned in the first bullet point above.

To that point, are our projections for the 49ers this year much too pessimistic, because Jim Harbaugh really is a motivational genius and our models aren’t accounting for that? Perhaps. It’s definitely a plausible theory. And if you believe it, feel free to just ignore that projection and we’ll see how things shake out by January.

Does that mean the entire system we’ve built for preseason NFL projections is worthless? Heck no…because our approach is much, much better at evaluating a bunch of very important factors than you are, profane commenter dude. Even though we just started doing these algorithmic preseason projections last year, and despite them provoking similar incredulous tirades last year from the occasional commenter, the 2011 projections did very well. And we’re going to continue to refine them every year from here on out.

We’ll never claim to be perfect or infallible. But don’t hate us for being algorithmical!