Notes from the Sports Nerds

The Impact of Resting Starters On NFL Predictions

• 3 comments - NFL - posted by tom on 12.19.09

Now that we’re nearing the close of the 2009-10 NFL regular season, it’s a good time to investigate the impact of resting starters on our site’s algorithmic NFL picks and prediction models.

(Note: When we say “resting starters”, we use that term to refer to resting starters entirely, pulling starters early, or any other end-of-season inspired player usage strategy.)

If you don’t care to read all about this, here’s the quick summary. Based on historical performance over the last four years during the late season:

- Expect reduced performance from our models in terms of picking game winners and vs. money lines
- Be wary of applying Power Ratings model predictions to any game in which resting starters is expected
- The Similar Games (Odds) model has performed best ATS during weeks 16 & 17 (counting all games)

Here’s the long version:

Team power ratings either fully or partially drive our NFL prediction models. At its most basic level, a power ratings based game prediction assumes that a team will perform at the same level at which it has performed in the past.

There are variations on this theme, since some of our models look more at recent performance, others at full season performance, etc. But the bottom line is that if a coach suddenly decides to use players and/or strategies in a much different way than he has done in the past, the underlying data on which any power rating is based becomes less relevant.

When it comes to our NFL prediction models, we have one model (Power Ratings) that incorporates only power ratings when making game predictions, and another model (Similar Games) that incorporates power ratings, betting odds, and other stats and data.

To get a sense of if and how resting starters may affect prediction performance, we decided to do a quick exploration of model performance during NFL Weeks 16 and 17 over the last four years.

This isn’t a great proxy, because it’s a small sample of games and it’s also likely the case that, due to the randomness of team scheduling, some years feature more “resting starters games” than other years. But we were interested in the results nonetheless. Here’s what we found in terms of Week 16 and 17 results:

- Both models have done worse picking winners (58% and 57%) than they usually do (63-69%). It’s logical to conclude that games in which late season personnel changes were in play were the primary cause of that performance decline, but we can’t be 100% certain of that based on this analysis.

- ATS prediction performance dropped somewhat for the models primarily based on power ratings or expected scoring; the Power Ratings model was 52% ATS overall in Weeks 16-17 (although it did go 23-9 ATS during the last two weeks of 2006, it also had two horrible years) and the Similar Games (Points) method was 50% ATS.

- ATS prediction performance for the Similar Games (Odds) approach has been good (57.6% ATS), with fairly consistent performance across the last four years. This higher level of performance does make logical sense, as this model is largely driven by point spreads, which (at least to some degree) reflect the expected effects of resting starters

Given that game winner prediction performance has historically decreased during the last two weeks, it’s likely the case that money line prediction performance will decrease as well, as a result of resting starters games.

We want to share this information to make sure our users are aware of it. The fact is that predicting the impact of resting starters with a high degree of precision is extremely difficult. In most cases, it’s a crap shoot when a coach may pull his starters, and it often depends on a variety of intangible factors or the timing of team scoring throughout a game.

As a result, given the data available to us, we don’t play any guessing games by trying to alter our models in any way to “adjust” for clinched (or bad) teams emptying their benches near the end of the season.

If you believe resting starters will likely be a factor in a game, it’s probably best to focus on the Similar Games (Odds) model for that game, or use our NFL Game Predictor tool to build your own algorithm that emphasizes the Point Spread and Game Week factors much more than power ratings.

For games in which both teams should play to the best of their abilities to the bitter end, of course, none of the above precautions applies, for Week 16 at least. However, a future post will discuss the potential impact of a good team losing a “meaningless” game on league power ratings.

NFL Early Season Over-Reaction: Bettor Beware

• 0 comments - NFL - posted by tom on 09.23.09

In one of his recent posts on ESPN Insider, Chad Millman reported that most of his wiseguy friends love to bet NFL Week 2 games, primarily because most square bettors tend to over-react to Week 1 results.

We don’t have the data to prove that assertion as fact, but it seems like a reasonable hypothesis. We’ve certainly noticed that a lot of sports fans tend to read far too much into small sample sizes of information.

Example: The fact that a team is 3-0 under the lights in October is probably due as much (or more) to randomness than to having a team full of vampire cyborgs that draw superhuman energy from the dark of night.

This problem is only amplified by the sports media, who are always looking for a number (no matter how meaningless it may be) to back up their opinions. Similarly, many sports betting related sites and publications focus on identifying “trends” and implying that they mean something, when 99% of the time they absolutely don’t.

(That’s a huge pet peeve of ours, to be explored more in a future post.)

But given Chad’s column, I looked forward to seeing how our NFL picks from our two primary math models did this week. It was a crazy week in the NFL, with lots of close games, and games expected to be close.

In fact, closing lines for 10 of the 16 NFL games in Week 2 showed favorites with an expected advantage of 3.5 points or less — a much tighter range than Week 1.

As it turns out, our Similar Games model was on the wrong end of several close games and went a poor 6-10 picking straight up winners, 3 games behind the worst ESPN expert at 9-7. ATS Performance was better at 8-8 overall, including going 4-2 in picks with predicted cover odds of 55% or better.

However, our Power Ratings model — which, compared to Chad’s assessment of the betting public, tends to UNDER-react to recent games — went 10-6 picking winners, better than 6 of the 8 ESPN experts. It also went an impressive 11-5 ATS.

(Note: Neither model adjusts for recent injuries, so both favored Philadelphia over New Orleans despite McNabb being out. In the future, we plan to introduce “injury warnings” for the math model predictions.)

Again, these sample sizes are all too small to prove anything, but it was interesting to see our “tortoise” model significantly outperform the hare, so to speak, in NFL Week 2.

Soon we plan to launch model performance pages up that show a five year pick history with results, which will give users a much deeper look into how our math models have performed in various situations with more meaningful sample sizes.

Week 1 NFL Picks (SU) & Patriots-Bills Learnings

• 5 comments - NFL - posted by tom on 09.15.09

Early season predictions are always somewhat of a crapshoot for mathematically driven models, but it’s always fun to review how our NFL picks and college football picks do at the start of a new year.

Since we’ve recently launched our over under picks and money line value picks, we’re also keeping close tabs on how those models perform.

The most fun about this whole early season thing is that the average sports bar jock cannot believe that our math models, which incorporate literally zero knowledge about off-season player and coaching changes, could possibly perform well in the opening weeks. During some years, they’re right.

But the fact is, it’s difficult if not impossible to assess how things have changed in the off-season before you have at least several weeks of game results to substantiate your theories. Until then, you’re guessing. If you know football really well and also guess relatively well, you’ll probably outperform our models in the opening weeks of a season. But your assumptions and guesses also could be wrong — and you could underperform an 100% objective approach as a result.

A case in point was last nights Patriots-Bills game.

I spent some time on the phone yesterday with a friend who could not believe our Similar Games Model highly favored the Bills to cover a +13 spread. (The spread was +11.5 earlier in the day, and the model still strongly favored Buffalo.) That model also saw Buffalo +680 as one of the top three money line value plays of the week.

I went through the whole litany of early season warnings about our data-driven algorithms with my buddy. The model had no idea Tom Brady was back, or that the Bills had just fired their offensive coordinator. For Week 1, it essentially assumes that everything is the same as it was at the end of the season last year, until this season’s results begin to prove otherwise.

He was absolutely convinced of a Patriots blowout and discredited the model’s approach.

We all know what happened. At the last minute, the Pats barely escaped with a one-point win. Last year, New England beat Buffalo by 10 in Foxboro. At this point, this result could mean almost anything, including:

1) The Bills are significantly better than they were last year, despite a new OC.
2) The Patriots are actually worse than they were last year, despite Brady being back.
3) More random factors played a major role in the outcome of this game being so different than expectations.

Only time will shed light on the most likely explanation.

Overall for Week 1, our Power Ratings model was 13-3 picking straight up NFL winners and our Similar Games model was 12-4. In comparison, five of the eight ESPN experts had 4 losses, one had 3 losses, and two finished with 2 losses.

Not bad for not knowing anything about the off-season…