August 23, 2012 - by Tom Federico
Whenever we post our NFL projections and college football projections, which are data driven and incorporate no subjective opinions of our own (save the extremely rare manual fudge for huge impact scenarios like Penn State this year), we tend to get a lot of comments on them. Many of these comments are constructive, but we also invariably see a few of the following routines:
This stuff provides some nice comic relief during a hard day of number crunching. We don’t take it personally, as we’ve clearly had to develop some pretty thick skin to do the sports predictions thing for over a decade now.
We certainly understand that when devoted fans are concerned, it’s the nature of the beast. Fans are highly passionate, and several of those fans also enjoy taking large doses of steroids before they start trolling the blogosphere.
However, some more insight into our process would probably help to reduce some of this vitriol. In short, we don’t have it in for you, profane commenter dude. We probably don’t even care a lick about your team, which actually may be even worse in your mind, but that’s another story.
Let’s be clear though. Unless you went to UC-Berkeley, it’s extremely doubtful that any key decision maker at Team Rankings actually hates your team.
With that said, profane commenter dude, you need to be aware of three realities:
Here’s some brief insight into how we approach preseason predictions, using college football as the lens.
As you would expect from Team Rankings, we crunch a ton of hard data. Our analysis is largely based on looking at historical variables (turnover margins, recruiting class strength, production lost to graduation/the draft, QB strength, etc.) and trying to establish a statistically significant connection between those data and a team’s performance during a season. We not only look at how a team did last year and its schedule strength in the coming year, but also at some variables that go back several years.
In terms of predicting season wins and losses, this fully algorithmic process has both pluses and minuses, compared to the likely predictive skills of an intelligent hard core fan who watches a ton of football.
There are certainly some dynamics (e.g. a drastic change of coaching staff, game strategy, or offensive/defensive schemes, or major injuries/personnel changes such as Tyrann Mathieu getting the boot from LSU) that are very difficult to model quantitatively and with a high level of confidence. Provided they can remain fairly objective, a relatively smart superfan may have an edge over our models at projecting whether these types of factors will have a slight, moderate, or major impact on a team’s future success.
Where we do have enough information gathered to model factors with decent confidence (e.g. how good a team has been in the recent years preceding the upcoming season), our ability to process that information is far, far superior to the average hard core fan.
For example, an LSU superfan may be better than our computers at understanding how Mathieu’s dismissal will affect the team’s performance, based on a deep knowledge of how the LSU coaches may alter their defensive schemes, the quality of the replacement player(s) who will see more playing time, and even the likely morale impact on the team. But our models will kick that fan’s butt at quantifying the relative strength of all of LSU’s non-conference and SEC opponents, and then coming up with a probability distribution of how many games LSU is likely to win this season.
Doing that complex calculation process well is a lot more important than accurately assessing one lost defensive player’s impact.
Secondly, a surprising amount of conventional wisdom about projecting a team’s end of season record/performance ends up being just plain wrong. We’ve examined lots of different factors that many fans would assume make an impact in projecting future results (e.g. Rivals.com ratings of each team’s recruiting class for the past two or three years in college football) and as it turns out, many of these instinctively important-sounding factors just don’t matter as much as most people think they do when you actually crunch the data and control for them.
So while we may be guilty of not incorporating some data that may indeed make a difference in a team’s future success, our approach also doesn’t shoot itself in the foot by drawing false conclusions from meaningless data. Many humans and unsophisticated projection systems regularly make that mistake.
On balance, we expect our preseason projections to be much better than the vast majority of projections that non-computer-savvy humans come up with. However, we also know and expect that a few of our preseason predictions could be way off, primarily for teams that carry high uncertainty due to scenarios like the ones mentioned in the first bullet point above.
To that point, are our projections for the 49ers this year much too pessimistic, because Jim Harbaugh really is a motivational genius and our models aren’t accounting for that? Perhaps. It’s definitely a plausible theory. And if you believe it, feel free to just ignore that projection and we’ll see how things shake out by January.
Does that mean the entire system we’ve built for preseason NFL projections is worthless? Heck no…because our approach is much, much better at evaluating a bunch of very important factors than you are, profane commenter dude. Even though we just started doing these algorithmic preseason projections last year, and despite them provoking similar incredulous tirades last year from the occasional commenter, the 2011 projections did very well. And we’re going to continue to refine them every year from here on out.
We’ll never claim to be perfect or infallible. But don’t hate us for being algorithmical!
Printed from TeamRankings.com - © 2005-2018 Team Rankings, LLC. All Rights Reserved.