How Accurate Were Our 2015 NCAA Tournament Projections?

Last week, FiveThirtyEight published a post reviewing the accuracy of their 2015 NCAA tournament projections.

In that post, they compared their round-by-round survival odds against those from other analytically-oriented sites using a technique called Brier scores.

Since FiveThirtyEight didn’t include TeamRankings in the comparison (no hard feelings, as our advancement odds weren’t publicly available prior to the First Four play-in round, since they are part of our premium NCAA Bracket Picks package), we went ahead and added our 2015 results to the mix.

What We’re Measuring: Projections, Not Bracket Picks

What is being measured here is not the actual bracket picks we advised our customers to make; coming up with optimal picks for a bracket pool requires the analysis of several other types of data, such as pool size and public picking trends, in addition to team survival projections. (We’re going to publish another post soon on how our recommended brackets did this year.)

The table below simply compares how accurate various sites were at predicting how far teams were likely to make it in the 2015 tournament.

2015_ncaa_tournament_brier_scores

So How Did Our Projections Stack Up?

As you can see, our projected survival odds would have placed us tied with NumberFire for second place in the overall accuracy rankings, and barely (0.001) behind The Power Rank.

We think it’s fair to say that the three of us constituted the top tier in accuracy this year, at least when measuring by Brier scores.

However, our path to that top tier performance was fairly different:

  • We never had the absolute best performance in any given round
  • We made up for it with consistency, being the only competitor that ranked in the top half of accuracy scores in every single round
  • We had the second best Round of 32 projections, behind only FiveThirtyEight, and well ahead of the pack

Overall, we’re happy with this result. While ranking first obviously would have been nice, finishing in the top tier with consistently solid showings in every round is a sign that our score wasn’t overly reliant on a couple fluky results. That should also bode well for future tournaments.

Our 2015 NCAA Tournament Survival Odds

For reference, here were our 2015 NCAA Tournament round by round survival odds, before the First Four play-in round began, sorted by odds to win it all:

SeedTeamRegionMake R32Make S16Make E8Make F4Make FinalWin It All
1KentuckyMidwest99.2%92.7%86.5%73.9%52.8%40.7%
2ArizonaWest99.2%78.1%62.9%40.5%18.8%12.3%
1WisconsinWest96.9%81.0%60.2%33.1%13.7%8.3%
1DukeSouth97.1%80.7%57.9%36.1%21.5%8.3%
1VillanovaEast97.7%78.5%60.2%36.4%20.5%7.6%
2GonzagaSouth96.6%71.2%46.6%25.5%14.1%4.9%
2VirginiaEast94.1%59.9%39.5%22.7%12.2%4.2%
3Iowa StateSouth92.8%66.6%32.1%14.9%7.0%2.0%
3OklahomaEast92.1%64.1%29.3%14.3%6.4%1.8%
2KansasMidwest87.5%53.9%30.6%7.4%2.8%1.3%
5UtahSouth75.5%51.9%22.0%10.7%4.9%1.3%
4N CarolinaWest81.2%58.1%23.0%8.9%2.5%1.1%
7Michigan StEast67.0%29.6%17.3%8.6%3.9%1.1%
3Notre DameMidwest88.9%52.6%28.1%6.4%2.3%1.0%
3BaylorWest78.3%47.0%13.9%5.4%1.3%0.5%
10Ohio StateWest73.4%19.0%11.2%4.6%1.2%0.5%
4LouisvilleEast77.7%46.9%16.0%6.1%2.1%0.4%
7Wichita StMidwest67.5%32.7%16.6%3.4%1.1%0.4%
11TexasMidwest53.8%25.3%11.7%2.2%0.6%0.2%
5N IowaEast75.9%39.1%11.9%4.0%1.2%0.2%
7IowaSouth57.8%17.8%8.1%2.8%1.0%0.2%
4GeorgetownSouth80.5%34.0%10.3%3.7%1.2%0.2%
9PurdueMidwest53.9%4.1%2.2%0.8%0.2%0.1%
6XavierWest48.3%21.9%4.6%1.4%0.2%0.1%
6ButlerMidwest46.2%20.2%8.6%1.4%0.4%0.1%
6S MethodistSouth55.0%18.6%5.5%1.6%0.5%0.1%
8NC StateEast56.8%13.1%6.5%2.1%0.6%0.1%
8San Diego StSouth54.2%10.8%4.2%1.3%0.4%0.1%
6ProvidenceEast58.6%21.8%6.6%2.2%0.7%0.1%
5W VirginiaMidwest66.4%37.6%4.2%1.6%0.4%0.1%
4MarylandMidwest68.0%36.8%4.0%1.5%0.3%0.1%
10DavidsonSouth42.2%10.5%4.0%1.2%0.3%0.1%
10GeorgiaEast33.0%9.7%4.1%1.4%0.4%0.1%
11BYUWest32.7%16.4%4.1%1.4%0.3%0.1%
5ArkansasWest76.6%30.2%7.8%2.0%0.3%0.1%
9Oklahoma StWest52.0%9.9%4.0%1.0%0.2%0.0%
16HamptonMidwest0.1%0.0%0.0%0.0%0.0%0.0%
10IndianaMidwest32.5%10.7%3.7%0.4%0.1%0.0%
7VCUWest26.6%2.9%1.1%0.2%0.0%0.0%
9St JohnsSouth45.8%7.9%2.8%0.8%0.2%0.0%
11Boise StateEast15.9%4.3%0.9%0.2%0.0%0.0%
11DaytonEast25.4%8.3%2.1%0.6%0.2%0.0%
8CincinnatiMidwest46.1%3.0%1.5%0.5%0.1%0.0%
11MississippiWest19.0%8.0%1.5%0.4%0.1%0.0%
11UCLASouth45.0%13.3%3.5%0.9%0.2%0.0%
8OregonWest48.0%8.5%3.3%0.7%0.1%0.0%
12WyomingEast24.1%6.6%0.9%0.1%0.0%0.0%
12BuffaloMidwest33.6%13.7%0.8%0.2%0.0%0.0%
12WoffordWest23.4%4.2%0.4%0.0%0.0%0.0%
12Ste F AustinSouth24.5%10.9%2.3%0.6%0.1%0.0%
13HarvardWest18.8%7.5%1.1%0.2%0.0%0.0%
13ValparaisoMidwest32.0%11.8%0.6%0.1%0.0%0.0%
13E WashingtnSouth19.5%3.2%0.3%0.0%0.0%0.0%
13UC IrvineEast22.3%7.4%1.0%0.2%0.0%0.0%
14UABSouth7.2%1.5%0.1%0.0%0.0%0.0%
14Georgia StWest21.7%6.8%0.7%0.1%0.0%0.0%
14NortheastrnMidwest11.1%1.9%0.3%0.0%0.0%0.0%
14AlbanyEast7.9%1.6%0.1%0.0%0.0%0.0%
15N Dakota StSouth3.4%0.5%0.1%0.0%0.0%0.0%
15TX SouthernWest0.8%0.1%0.0%0.0%0.0%0.0%
15BelmontEast5.9%0.8%0.1%0.0%0.0%0.0%
15N Mex StateMidwest12.5%2.7%0.5%0.0%0.0%0.0%
16ManhattanMidwest0.7%0.1%0.0%0.0%0.0%0.0%
16N FloridaSouth1.9%0.4%0.1%0.0%0.0%0.0%
16Rob MorrisSouth1.1%0.2%0.0%0.0%0.0%0.0%
16LafayetteEast2.3%0.3%0.0%0.0%0.0%0.0%
16Coastal CarWest3.1%0.6%0.1%0.0%0.0%0.0%
9LSUEast43.2%8.0%3.5%0.9%0.2%0.0%