Under the TeamRankings Hood, Part 4: Models, Models, Everywhere

posted in > Site Help, College Football, MLB, NBA, NCAA Basketball, NFL

Looking for our 2011 NCAA bracket predictions?

[This post is the last of a four part series covering our ratings and models. Today we present an overview of each of our statistical models, which combine the ratings covered in Parts 1-3 with outside information in order to make game winner, spread, totals, and money line predictions. For background, see the previous posts. Part 1: Power Ratings Basics. Part 2: Defining Each Rating. Part 3: Pros And Cons]

In various places across our site, you’ll find game predictions. For example, our NCAA college basketball win picks page shows the win odds for all the current day’s games, and all our game matchup overviews have game winner, spread, totals, and money line picks.

These predictions are derived from six different statistical models, which range in complexity from simple historical win percentages to a machine learning algorithm that uses hundreds of input variables. To see an example of predictions from all six models, check out: Georgia vs Washington.

Below are detailed descriptions of all the models, along with pros and cons of each. We hope we’ve covered all the most important details, but if anything is still unclear, feel free to ask questions in the comment section.

Predictive Power Ratings Model

Our Predictive Power Ratings Model uses the Predictive power ratings described in the first three parts of this series. To summarize, a computer algorithm iteratively analyzes data on every Division I NCAA basketball game result this season. In the end, every team receives a simple numerical rating (e.g. 92.4 or 104.7) indicative of its actual, documented, proven track record at outscoring opponents.

Like fine wine, predictive power ratings tend to improve with age. The more game results data recorded for a given season, the lower the potential impact of luck on a team’s overall performance results, and the more “connections” the model can make between teams in conferences of varying strength.

By comparing the predictive ratings of any two teams, you can determine projections for the game winner and expected margin of victory in a hypothetical game between the two teams. Calculating win odds is more complex; a formula translates the differences in predictive ratings into respective win odds for each team.

If you went back and recreated the entire NCAA basketball season using the final predictive power ratings for every team and the prediction methods described above, all 300+ Division I teams would end up with margin of victory performance and a win-loss record equal or very close to what actually happened.

Outputs: win odds, margin of victory

Strengths: Predictive power ratings are relatively abstract measures that cut through media hype and rate teams based on actual scoring differentials, adjusted for game location and opponent strength. You can blab all you want about a team’s legendary coach, elegant offense and twin 7-footers — but who the heck cares if they still don’t outscore the average opponent better than 40% of the other teams in Division I?

Weaknesses: The Predictive Power Ratings Model is a dynamic and reactive system, but the sole inputs are game results. If an absolutely key player gets injured or a bad team all of a sudden gets fired up and starts playing well, it could take several games for the impact of those developments to make a big difference in predictive ratings. Abstraction has a downside too; if a huge situational mismatch exists between two particular teams (e.g. one team has an eight foot tall center and its opponent has no player over 5’5″), predictive ratings have no idea.

Similar Games Model

Our Similar Games Model uses data driven algorithms to identify historical NCAA tournament games that featured statistically alike teams competing under similar matchup circumstances.

For example, imagine that a first round matchup features a high scoring team from a weak conference playing a low scoring team that turns the ball over a lot. Both teams are traveling moderate distances to a neutral site arena. Similar matchup scenarios most likely have occurred in recent history, and the Similar Games Model identifies them and analyzes their outcomes.

Final predictions depend on the aggregate analysis of a number of data points about each identified similar historical game, such as which team won, by how much, how many points were scored, what results were implied by betting lines, and the relative degree of similarity to the current game.

Outputs: win odds, projected final score, point spread cover odds, over/under odds

Strengths: This model incorporates a range of power ratings and team stats as well as several contextual factors including Vegas line implications, travel distances, and game timing.

Weaknesses: This model does not explicitly consider several difficult-to-model factors such as recent injuries or days rest. If you feel one of those factors may have a material impact on the outcome of a given game, it may be wise to apply subjective adjustment to its predictions. Also, in certain cases, highly uncommon matchup scenarios make it impossible to find many relevant historical matchups.

Simulation Model

The theory behind our Simulation Model is to use possession based statistics (also known as ‘tempo-free’ or efficiency statistics) to project the likely outcome of a game.

Possession based stats are better measures of team performance than “per game” stats primarily because the pace of a basketball game is an important driver of the final outcome. For example, imagine two teams are playing each other. One team turns the ball over an average of 15 times a game, while the other only turns it over seven times a game. Which team takes better care of the ball?

Of course, that’s a trick question. What you really need to compare is how efficient each team is at handling the ball. If the first team typically has twice as many possessions per game as the second team, then in reality, these teams probably have about equal ball handling skills.

In our Simulation Model, we first generate what essentially are tempo-free power ratings for a team’s performance in major stat categories (blocks, rebounds, etc.). We then look at the number of possessions each of two opposing teams typically has, and estimate the number of possessions we expect in the game being modeled. Given that number, we can use the individual stat ratings for each team to project what will happen on every possession, leading to a score prediction and expected box score.

Outputs: win odds, margin of victory, projected final score

Strengths: Unlike the more abstract power ratings and similar games models, the Simulation Model takes a very “bottoms up” approach to analyzing the playing styles of two teams, how they match up, and how specific statistical differences (e.g. a strong rebounding team playing against a weak rebounding team) are likely to effect the final score. The data used is all from this season, and each statistical rating for each team is adjusted for opponent strength.

Weaknesses: There are more assumptions at play in this model than in other models. Primarily, we are assuming that a team will continue to play its same general style of basketball as its season stats so far imply. That’s usually a safe assumption, but if a coach makes a radical change in game strategy or play calling for a given opponent, the raw data on which this model is based loses relevance.

Decision Tree Model

Our Decision Tree Model is the output of a machine learning algorithm that views every college basketball game since 1999 through the lens of hundreds of input variables, ranging from contextual information like the distance traveled by squad to team statistics like effective field goal percentage.

The algorithm does what might be convenient to think of as complex, high volume, statistically significant trend analysis. It repeatedly partitions the games into smaller and smaller subsets based on the values of one or more variables. Each split is chosen so that the win probabilities of the teams in each group get further away from 50% and closer to 0% or 100%.

But it also takes care not to be too overzealous in the splitting, checking to be sure that the splits are meaningful, and not just the products of a small sample size. In the end, it’s left with a set of rules along the lines of: If Variable1 is greater than 100 and Variable2 is less than 7 and Variable3 equals “YES” (and so on) then the win probability of TeamA is 64%.

Complicated enough for you? OK, now here’s the twist: we actually have about 100 different decision tree models, each of which look at a different subset of variables. The results of all the individual models are averaged to get the overall win probability. This helps reduce the effect of crazy outlier results, and ensures that we have multiple reasons for picking a certain team to win.

Outputs: win odds, point spread cover odds, over/under odds

Strengths: This is our most complex model, incorporating the largest amount of information, so if there’s an obscure nugget of knowledge hidden deep within our database, this is the most likely place for it to show up. It also generally has proven so far to be our most accurate model, although prediction performance varies by sport.

Weaknesses: Since it is partly based on historical trends, changes in the way the game is played or in other associated data can lead the model down a new, unexplored path, where that trend no longer applies. The complexity of the model also makes it next to impossible to explain to an average fan. We know what the output is, but we never know exactly why the model gave us a specific number as an answer. We’re just shoveling data into a computer and trusting it, which is what most advanced quantitative prediction systems do, but it still makes some people nervous.

Vegas Implied Model

The Vegas Implied Model assumes that the betting lines offered by Las Vegas sports books are efficient, meaning that they are the market’s best prediction for the outcome of a contest.

Vegas offers up to three basic types of bets, which each describe a different aspect of a game. The spread tells us what the expected margin of victory is, the over/under tells us how many total points are expected to be scored, and the money line gives us implied win odds for each team.

When a money line is available, that’s what is used for this model. While the win odds implied by the money line have some “juice” added in, we can reduce the odds proportionally for both teams to come up with more realistic win probabilities.

In cases where no money line is available, we convert the point spread into expected win odds based on the historical winning percentages associated with particular lines. And when there is no spread available, we use our predictive power ratings to estimate what the spread would be, then use those same historical win odds.

Outputs: win odds, projected final score

Strengths: Because the betting market reacts dynamically to new information, the Vegas Implied model should adjust nearly immediately to the market’s anticipated effects of injuries, suspensions, and other contextual data that is, for all practical purposes, invisible to our other models.

Weaknesses: The biggest drawback to this method is that it relies on the assumption that the betting markets are efficient. In reality, some lines are probably affected by things unrelated to the outcome of the game: the public’s fetish for certain teams and their tendency to support favorites, or the activities of influential high-dollar bettors who may try to use their betting patterns to manipulate the market.

Seed Difference Model

The Seed Difference Model, which only applies to the NCAA basketball tournament, is simply a window into the past, created to satisfy the curiosity of ourselves and of our readers. We look back at all tournament games since 1998 to find what happened when teams with a matching seed difference played each other.

For example, when an 8-seed plays a 9-seed in round one, we find the winning percentages of the higher and lower seed in all previous 8-vs-9 games, as well as 2-vs-3, 1-vs-2, etc. We expand the search in this manner in order to increase the sample size of each possible combo. Otherwise there would be very little data for quite a few of the specific seed-vs-seed matchups.

Outputs: win odds

Strengths: Much of successful NCAA tournament pool strategy involves identifying inefficiencies in the likely picks of your opponents. They will often default to picking the higher seed, and this model can help you discover which they may be placing too much emphasis on a seed difference that shows little historical predictive power.

Weaknesses: While expanding our search to include any equivalent seed differences allows us to increase the sample size, it also leads to perhaps less relevant results being included. Though the seed difference for both is 3, the difference in quality between a 1-seed and a 4-seed should be larger than the difference between a 7-seed and a 10-seed. In addition, the NCAA selection committee’s tendencies may change over time, meaning the meaningfulness of seed differences could also change.

  • http://www.teamrankings.com mkoidin

    Which model should I use?

  • Mark

    Where do you get the raw stats the you use in your formulas? Do you buy them from another site? Do you get them free from another site? Or do you enter them in yourself

  • http://www.teamrankings.com TeamRankings.com

    Mark, we currently use a number of sources for our data, from publicly available databases like Retrosheet (baseball) to feeds from businesses like SportsDirect. So we get some for free, and others we pay for. Being tech guys, we kind of set the rule that we need to automate all of our data acquisition…no manual entry allowed!

  • Mark

    I noticed that some of your NCAAB models switch teams during the day. Is it because information, from games being played that day, are being thrown in the formulas? Also, Will you have the entire NCAAB season available like the way your NCAAF season is to look over? It is basically the entire season from day one. I really appreciate your help. This handicapping is not easy.

  • http://www.teamrankings.com TeamRankings.com

    Mark, every time data changes — the result of another game being played comes in, betting lines move, etc. — our model predictions are subject to change, since they consider so many variables. As for NCAAB, what are you looking for exactly?

  • Mark

    I first found your site on January 27th. I could only keep track of your picks, on a day to day basis, from that day. I would like to see the day to day picks from the start of the NCAAB season. Steve is right about when all three of your formulas agree on the same team. From January 27th through March 13th you were 57%. Which, as you know is great. I’m curious as to how the rest of the season went. One thing I noticed during the 1/27 through 3/13 time period is that the same teams were turning up a lot as unanimous picks. I’m wondering if that was because the books had a hard time putting out a good line for those teams.

  • Mark

    Hey TeamRankings, I told you what i was looking for exactly. Do you have time to continue this discussion? If not, let me know when you do okay.

  • http://www.teamrankings.com TeamRankings.com

    Mark, sorry for not replying to your earlier comment. We’ve recorded your request and it’s on the list. We actually have a number of potential to-dos on our list regarding posting more historical data (our daily predictions, prior season matchups & stats & rankings, etc.) and making it much more accessible.

    In terms of tracking records where all our models agree, it shouldn’t too hard to just add that to our Prediction Accuracy pages somewhere, it’s just a matter of scheduling in the work.

    After March Madness ends each year we typically spend a few weeks getting baseball up and running and also plotting out our priorities for the next 6-12 months (and taking a couple much needed days off). We’re in the middle of all that right now, so all I can say is that if we make the “better historical data” project a priority, we’ll probably be rolling out some of those features in the next 4-6 weeks. Thanks for the comments.

  • http://www.teamrankings.com TeamRankings.com

    Mark, I responded to the historical data request in your follow up question. As to the point about the same teams coming up as unanimous picks, your theory is as good as ours at this point — our Decision Tree model especially is sufficiently complex that it’s never going to be super obvious exactly why it’s choosing certain teams. But it certainly could be the case that for whatever reason, the lines are consistently undervaluing certain teams — especially if they are not public favorites — and our quantitative models are picking up on that.

  • Mark

    I figured that you were busy. I just didn’t want you to forget the conversation. Thanks for the quick reply. Enjoys the time off. I’ll also get back to you when I have more questions. There no rush, NCAAB doesn’t start until November. I’m busy watching your baseball information from the start.

  • brick

    Do your MLB picks take pitchers into account?

  • http://www.teamrankings.com TeamRankings.com

     Brick –

    Yes, our MLB picks from the Similar Games and Decision Tree models (along with our official picks) take into account the listed starting pitchers.

    The Power Ratings model does not base its predictions on starting pitching.

  • kyle

    I am a huge sports fan and absolutely love analyzing stats, and this is the motherload! I am addicted to this site. Thank you so much for making this available for free. It is incredible what you guys have come up with. I’ll never make a bet again without checking teamrankings.com first.

  • Norm Plante

    In your prediction accuracy breakdown page , what time period do the picks reflect in the tab ” all models picks” page, 

    Best stat site by far great job guys

  • http://www.teamrankings.com/ David Hess

    Norm — There’s no single answer to your question. It depends on which model and type of picks you’re viewing, since we created the models at different times in the past. For any given sport, model, and pick type (i.e. NFL Decision Tree picks against the spread, or MLB Power Ratings money line picks), you can find out the time period by viewing the appropriate Yearly Results page: http://www.teamrankings.com/nfl/betting-models/yearly-results/

    On that page, for example, if you select ATS & Similar Games, you’ll see those picks go back to 2005. If you select Money Line and Power Ratings, you’ll see the picks go back to 2007. If you select Decision Tree and Totals, you’ll see results back to 2008. Whatever time frame you see here will be the same one you get for “All Model Picks.”

  • Chris

    Would you describe these picks as coming from a computer simulation or human expert

  • http://www.teamrankings.com/ David Hess

    Chris — This page describes our predictive models, which are all implemented via computer programs, so definitely not “human expert.”. But only the Simulation Model is a computer *simulation*.

  • http://www.cameronaskew.com Cameron Askew

    Hey there TR team. I’m impressed with your guys Decision Tree results on NFL ATS bets. I’m doing a project building a backpropagation neural network for NFL ATS bets.

    I’ve had some success in data mining..I won 2nd place in a competition and was flown to Germany (http://www.sandiego.edu/insideusd/?p=18247).

    One thing I wanted to know was if you guys had NFL game/team stats nicely formatted into a database or CSV or XML files.

    I’ve gathered some data going back to 1978, which is about as far back as I wanna go considering the game has changed substantially since the 50’s/60’s. However, the data I have is pretty shallow (for example, I have turnovers but not fumbles lost vs. interceptions and the two probably shouldn’t be clumped together).

    Anyways, hope to hear from you. I’ll keep tabs on this post but you can also e-mail me at askew.cameron@gmail.com

  • Matt_mccoy

    Love this sight…. Mother load is right for stat lovers…

  • http://www.teamrankings.com/ David Hess

    Matt — Thanks for the high praise! We’re glad you find the site useful!

  • Rmortl

    I just started watching your site the last couple of weeks and am confused about your predictions grid if you would be so kind to help.  Today for example….Your 1st line on the predictions grid you have 2 stars for St.Louis as GAME WINNER but then you have 2 stars PIT +176 for the MONEY LINE.  I am looking at this saying they predict St. Louis to Win but then saying to Take PIT +176 to win straight up.  Can you help clarify what I am missing here?  I have tried to research your stuff without having to bother you but am lost.  Rich

  • http://www.teamrankings.com/ David Hess

    Rich, we get asked that a lot, so rather than answer you here, I added the question & answer to our FAQ: http://www.teamrankings.com/blog/ncaa-basketball/betting-predictions-faq

    Just look for the question “Why is the Game Winner pick sometimes different than the Money Line pick?”

  • NFL Fan


    I had a question about your predictive models…it seems like your algorithms don’t change much in % confidence even when the point spread changes significantly.

    For example, a couple days ago, your decision tree had Philadelphia Eagles -1 with 55.1% confidence. Now it has Philadelphia Eagles -3 with 55.2% confidence. And I’ve seen this pattern many times now.

    So…you’re more confident in the Eagles even though there’s more points against them? Doesn’t make sense to me…

  • http://www.teamrankings.com/ David Hess

    Well, the exact details of why that happens are complicated, but the basic idea is that when the spread moves, that’s new information about a game, and the models take it into account.

    This is oversimplifying, but it’s kind of like the market has revised it’s estimate of how good the Eagles are. When the models saw the Eagles as a “market value -1” team, the predicted cover odds for a -1 line were 55%. Now that the models see the Eagles as a “market value -3” team, the projected odds to cover -1 would be even higher than 55%.

  • http://www.facebook.com/james.rodgers.733 James Rodgers

    Hi there I am new to this site , just found it today, when you see a star rating you bet on them with how many stars its the lay off bit which i can’t understand could you clarify please

  • http://www.teamrankings.com TeamRankings.com

    James, thanks for the question and glad to have you on Team Rankings. To start off, we’ve built computer models that predict games against the betting lines, but we don’t give any betting advice in terms of if, when, and how you decide to bet games. If you’re interested in learning more there, I’d suggest Googling around to read about the Kelly Criterion and related topics as a start.

    In short, the star rating for our picks is just a simple translation of the confidence odds we list by every prediction. For point spread and totals picks, picks with over 52.4% confidence odds (expected profitability at -110 payout odds) get two stars, and over 55% odds gets three stars. It’s not meant to be a betting guide or system though, just a quick way to roughly categorize picks into buckets.

    For money line value picks, “LAY OFF” means that our models don’t find any value on either team’s money line.

  • bh3ka

    I am looking at the NBA numbers and they look great.

    Prediction Accuracy –> [Moneyline / TR Pick [2* or more] / 2012-13]

    Picking Away Teams = +21.6
    Underdogs = +21.4

    However I get a bit concerned checking previous years as the numbers are really bad! … tell me there has been some tinkering to the prediction model so I can get my feet wet in this.

  • http://www.teamrankings.com/ David Hess

    Yes, actually we made huge changes to the basketball and football TR Pick logic this season. We now combine the info from BOTH our spread models and our win models, and make one master prediction for each game, which is used to drive the ATS, money line, and game winner pick.

    That said, this season is still a pretty small sample size, so I would not expect the results to continue to be *that* good.

  • http://www.teamrankings.com TeamRankings.com

    One clarification: we did not make changes to college basketball point spread and totals prediction logic this year, though.

  • bh3ka

    Thanks. will thread carefully.

  • ed

    I am looking at the against the spreads pick for the NFL, specifically the power rating column. For the first one it has OAK +3.5 in green. Does that mean after calculating the numbers, the spread should be Oakland +3.5?

  • http://www.teamrankings.com/ David Hess

    It means the Power Ratings Model projects Oakland to beat the spread by 3.5 points.

  • Jack

    Would you say the decision tree rating or power rating is more important? Also what does it mean if the number is red.

  • http://www.teamrankings.com/ David Hess

    Decision Tree has historically performed better. However, we would suggest paying attention to the TR Pick and the associated Odds, as those combine info from multiple models to give our best estimate of win/cover odds. See here for more info: http://www.teamrankings.com/blog/site-updates/updated-mlb-pick-logic

    If a number is red, it means that model disagrees with the TR Pick.

  • Reese

    David why you haven´t picks about NHL?

  • http://www.teamrankings.com/ David Hess

    Adding a new sport is very time and labor intensive, and there’s only so many hours in the day. We don’t want to spread ourselves too thin. If we spent time adding a new sport, we wouldn’t be able to create new stuff like our custom pick tools for football and March Madness: