How Our NCAA Bracket Pool Picks Did In 2015

March 9, 2016 - by Tom Federico

I think we can agree that the primary goal of entering bracket pools is to win them.

If winning is the goal, then the best metric to measure the true success of bracket pool picks is clear — how often they win pools.

More precisely, if you happen to sell bracket advice, like we do, you should measure yourself by looking deeper than how often your picks win. What matters is how often they win, compared to the level of success that a customer could expect to achieve without your supposedly expert advice.

Last year we collected performance data on how our NCAA Bracket Picks customers did in 1,390 different bracket pools, and this post reports on that data.

How Not To Measure Bracket Pick Success

Before we go any further, it’s helpful to touch upon a few of the silly ways to measure bracket pick performance, because we often get asked about them.

Here are two commonly mentioned metrics that don’t matter much at all:

The percentage of picks you get correct
How you did last year

Let’s briefly explain why each one falls short.

First Bad Way To Measure: Percentage of Picks Correct

When it comes to bracket pools, achieving a high pick accuracy rate overall is by no means a guarantee that you’ll win a pool.

In some NCAA tournaments, like in 2008 when all four #1 seeds made the Final Four, many of the popular favorites keep winning. In years like those, you’ll need a very high rate of picking accuracy just to keep pace with the top 20% or 30% of your pool.

In other years, some big favorites lose early, and/or some unpopular teams make deep runs in the bracket (remember #7 UConn vs. #8 Kentucky in the 2014 final?). When that happens, your overall bracket pick accuracy can be quite low, compared to others in your pool, but you can still have a great shot at a prize if you nail a late-round pick or two that almost all of your opponents miss.

(Keep in mind that the early rounds account for over 70% of the games in the tournament, so your overall accuracy is heavily biased by how you pick in those games. Yet they’re worth the least points.)

And guess what? If you want to maximize your odds to get the highest percentage of picks right in your 2016 bracket, your strategy is quite simple. Just pick the favorite to win every single game. However, that’s rarely the strategy that maximizes your odds to win a pool.

In short, being a skilled bracket pool player is not determined by achieving some arbitrary level of overall picking accuracy. It’s a vanity metric that can sound impressive while meaning very little.

Second Bad Way To Measure: How You Did Last Year

We know what you’re thinking: “What the heck, isn’t this post called ‘How Our Bracket Picks Did Last Year’?”

Indeed it is.

Prospective customers who check out our NCAA Bracket Picks product often ask us how well our picks did the year before, which is a perfectly reasonable request. Being transparency obsessed data geeks, we do our best to collect and share that data. (And our picks did decently well last year, as you’ll see below.)

However, what needs to be better understood, as we’ve explained in our post on bracket pool win odds, is that luck will always be a factor in how a set of bracket picks does in any given year. It’s over the long term, not the short term, that truly skilled bracket pickers differentiate themselves from the masses.

The same thing goes for our bracket picks. Our level of skill is most accurately measured by how our picks perform across lots and lots of pools, so that the potential impact of luck on our performance results is greatly diminished.

How We Collect Bracket Pick Performance Data

The great news is that as far as we know, we’re the only site that objectively measures, on a decently large scale, how people using our bracket picks actually did in their pools. That’s because:

We have a lot of customers of our NCAA Bracket Picks product
Those customers enter details for every bracket pool they’re entering
After the tournament ends, we email them a survey asking how our brackets did in each specific pool

Admittedly, this isn’t a foolproof method for measuring success. Given all the friction involved in getting someone to receive, notice, open, and respond to a corporate-style email, our survey response rates typically hover around 10-15%.

Still, is this approach much more objective and comprehensive than what anyone else in our position is doing? Absolutely. As mentioned above, last year we collected data on how our picks performed in over 1,390 different pools.

In comparison, other sites or self-proclaimed “experts” that offer bracket picking advice may occasionally throw up a quote or two from someone saying, “I won my pool with your picks!” That’s simply not a representative sample. How many people used their picks and lost?

We don’t know how every single one of our customers did last year with our picks. But almost 1,400 bracket pools worth of data is likely to be at least directionally accurate.

Finally…How We Did Last Year, Measured Intelligently

The table below presents data on how often our customer survey respondents reported finishing “in the money” (aka winning a prize) in the most popular types of bracket pools.

It also compares their win rates to what an average pool participant would expect, in pools of equivalent sizes, and assuming if everyone in the pool was equally skilled.

Scoring System Type	Expected In_Money	Actual In_Money	Win Multiplier
Traditional (1-2-4-8-16-32)	10.0%	13.27%	1.3
Alternate Round-Based	9.5%	10.34%	1.1
Round and Seed-Based	11.8%	34.68%	2.9
Traditional Sweet 16 Pool	6.6%	10.13%	1.5

Some observations:

Customers in pools where the seed number factored into the base scoring scoring system (e.g. Round + Seed, Round x Seed) were the big winners in 2015; our picks for these pools won a prize over 1/3 of the time, almost 3x the win rate you’d expect from an average participant.
The second best performance came in Sweet 16 pools (often called “second chance” pools), which start fresh in the tournament’s second week. Our picks were about 50% more likely to score a prize in those pools than what you’d expect from an average player.
Our picks only provided a very slight edge in round-based scoring that wasn’t based on the traditional round values (1-2-4-8-16-32). However, they provided a modestly better 30% edge in the most popular bracket pool scoring system.

Analyzing The Results

In the old days (aka, before 2012), we published two or three recommended brackets each March, all of them designed for the traditional 1-2-4-8-16-32 scoring system. So it was pretty easy to reach a verdict on whether we did great, OK, or poorly with our picks in any given year.

Today, our bracket picking algorithms customize brackets for particular pool sizes and scoring systems, and our customers’ pools use hundreds of different scoring systems. So a simple statement about how well we did in a given year is no longer so easy to make.

The way we do things now, it’s highly unlikely there will ever be an unequivocally great or terrible year for all our bracket picks. Going forward, as we saw in 2015, our picks for some types of pools will likely do better than our picks for other types of pools, in the same tournament year.

In general, though, 2015 was a decent but not spectacular year for our picks. Thanks to a few tough breaks in the late rounds, we didn’t deliver as big an edge as we strive to achieve in the most popular 1-2-4-8-16-32 scoring system.

However, we did provide at least some level of positive edge across all four of the most popular types of pools. That’s significant, and it’s a testament to the capability of our technology to fine-tune bracket picks for wildly different types of scoring systems.

And About Those Customer Testimonials…

As we mentioned above, hard data about bracket pick performance across 1000+ pools beats anecdotal evidence from one or two people every day. Still, customer comments can also lend some additional context to the quantitative results.

Along those lines, we invite you to read what dozens of our 2015 NCAA Bracket Picks customers think of the product. Some of them have been subscribing to our bracket advice for years, and provide a historical perspective on how they’ve done over multiple years in this customer reviews post.

Alternatively, here’s where to learn more about our bracket picks for 2016.