TeamRankings.com
Get Smart About Sports
April 29, 2015 - by David Hess
This post contains a detailed review of how our bracket picks performed in 2015. In case you’re not familiar with it, we sell a premium product for NCAA bracket pool picks.
Our goal here is to quantify how effective our NCAA Bracket Picks product was in delivering value to its customers. This is actually a somewhat complex process, for a number of reasons, but we do our best given the constraints.
(If you want to learn more about the details, you can read the appendix at the end of this post entitled “How We Measure Bracket Pick Performance.”)
Note: These numbers assume that all competitors in our customers’ bracket pools were equally skilled. In addition, we have adjusted prize winning expectations to account for any cases where customers played multiple brackets in the same pool, and where customers played in multiple pools. If the baseline expectations for winning a prize seem high, that’s why.
These results are pretty solid, and they surprised us a bit, in a good way.
Based on how the tournament played out, we knew that 2015 wouldn’t be one of our best years for bracket pick recommendations. Our tournament predictions saw eventual champion Duke as the third mostly likely team to win it all (tied with Wisconsin at 8.3%), but the Blue Devils were also relatively popular (picked by 9.3% of the public to win the title).
When the dust settled, our algorithms didn’t see the Blue Devils as a particularly good champion pick this year, in most any scoring system. As a result, none of our Best Brackets for various pool types had Duke winning it all.
However, our customers ended up doing a good job leveraging the broader scope of advice that our NCAA Bracket Picks product provides. In particular, many of our customers played multiple brackets we recommended, either in the same pool (as a portfolio strategy), or in different types of pools.
That approach helped diversity their risk and increased their chances of winning a prize, and it clearly paid dividends.
At the most basic level, our goal is to help our customers win prizes in bracket pools. So, success to us means customers winning prizes.
However, our customers use our bracket picks and advice in a variety of ways. So if we had to measure our bracket pick performance by a single metric, it would not be any of these:
Here’s why. First, the only goal of competing in a bracket pool is to beat your opponents. There’s no prize tied to getting a certain number of points or picks right.
Second, most of our customers don’t just play a single bracket, and all of them have a choice of brackets to use. We identify our top bracket for each pool type, but we also provide Alternative brackets for customers to choose from, primarily to play along with the Best Bracket as a portfolio strategy.
Based on our survey, nearly 80% of our customers used multiple brackets we recommended, and nearly 60% entered more than one bracket per pool:
Entry Distribution % of Customers
1 pool, 1 entry 22%
1 pool, multiple entries 27%
Multiple pools, 1 entry per pool 19%
Multiple pools, multiple entries per pool 32%
And while half of our customers used 2, 3, or 4 brackets, a handful entered a dozen or more:
# of total brackets entered % of Customers
1 22%
2 26%
3 13%
4 11%
5 7%
6 8%
7 4%
8 3%
9 2%
10 2%
11 to 20 2%
20 to 30 1%
more than 30 1%
In case you’re wondering, the most brackets any single customer reported entering was 72 (!), across 12 different pools.
Customizing picks for specific types of pools and creating optimized portfolios of brackets are key features of our NCAA Bracket Picks product, and they directly impact customer value.
After Arizona lost to Wisconsin in the Elite 8, for example, we knew that our Best Brackets for the most popular scoring systems would suffer. We had projected Arizona as a bit more likely than Wisconsin to make the Final Four, and Arizona was also a less popular pick.
That’s a compelling profile, and we ended up having Arizona making the Final Four in over 99% of our Best Brackets for various types of pools.
At the same time, 34% of our Alternate brackets, which are designed to help diversify the key risks taken in our Best Brackets, had Wisconsin in the Final Four over Arizona. And 30% of those brackets had Wisconsin then beating Kentucky and making the championship game.
So when it comes to evaluating the value we delivered to customers, we can’t just look at how accurate our NCAA tournament survival odds were or how our top bracket for one particular scoring system did. It makes more sense to look at broader metrics that evaluate winning in general.
We use these two:
Past customer surveys we’ve done have indicated that our customers tend to prioritize winning something over coming in first place (or bust), so those metrics serve as good proxies for customer value.
We listed our overall results in “The Bottom Line” section at the top of the post. Below we’ll slice and dice them in a few different ways.
While we’re very interested in the overall frequency with which our customers win prizes, investigating performance in different types of pools is informative as well.
Our survey asked customers for information about every pool that they defined in our product. Consequently, we can review how our resulting bracket pick recommendations did based three specific factors:
For each of these breakdowns, we can report on the two primary measures:
The second calculation takes into account both pool size and the number of entries our customers submitted in each pool, and gives us a baseline for comparing actual performance against expectations. (For example, one would expect a customer submitting 10 entries in a 1,000 person pool to win 1% of the time in the long run.)
The results by pool size look about as we expected, given our pick recommendations and the way the tournament played out:
Pool Size Expected % in the money Actual % in the money Edge
10 or fewer entries 21% 38% 1.8x
11 to 30 entries 15% 22% 1.5x
31 to 50 entries 11% 21% 1.8x
51 to 100 entries 10% 11% 1.1x
101 to 250 entries 7% 7% 1.1x
251 to 1000 entries 4% 1% 0.2x
more than 1000 entries 2% 1% 0.5x
Grand Total 10% 14% 1.4x
Our bracket picks this year were fairly “chalky” compared to previous years, with a lot of favorites advancing. We got some pushback from customers about this, and fielded several questions about why our brackets didn’t feature more upsets. The short answer is because every year is different, and that’s what our numbers indicated was optimal in 2015.
Small Pools
In the end, the focus on favorites this year served our customers well in small pools. Even though most of our Best Brackets had Kentucky winning it all and got two Final Four teams correct, our picks still provided a decent edge in smaller pools, thanks to good earlier-round decisions.
In the end, our customers in pools with 50 or fewer entries cashed 50% to 80% more often than one would expect.
Large Pools
In larger pools, a decent showing doesn’t cut it. In order to win a prize, you need to take calculated risks, and your big gambles (or most of them, at least) need to come through.
Our larger pool picks this year made some big bets on undervalued teams like Arizona, Gonzaga, and Utah. In retrospect, those decisions gave us a nice shot; both Gonzaga and Utah made it farther than the public expected, and all three teams ended up losing to an eventual NCAA finalist.
But especially once Duke, the nation’s third most popular tournament champion pick, won it all, it was pretty much a given that we’d log a sub-par performance in big pools this year.
With so many scoring systems supported by our custom bracket picks tool, we have to group them into broad categories here.
In the table below, we grouped pools by whether the points awarded for each game (“Base scoring”) took into account a winning pick’s seed number or not. If not, we then subdivided based on whether they featured the traditional 1-2-4-8-16-32 round-based scoring system, or something else. Finally, we also split out pools that awarded upset bonus points.
It turns out when you look at it this way, there’s one type of pool where our customers did especially well this year: pools with seed based scoring, but without upset bonuses.
Base scoring Upset Bonus? Expected % in the money Actual % in the money Edge
Seed based no 12% 37% 3.1x
Round based (1-2-4-8-16-32) no 9% 11% 1.2x
Seed based YES 11% 11% 1x
Round based (other) no 10% 9% 0.9x
Round based YES 12% 8% 0.7x
Grand Total 10% 14% 1.4x
This outcome appears to be the result of a couple factors:
Also worth noting: Despite not picking Duke as tournament champion in any of our Best Brackets for the popular 1-2-4-8-16-32 scoring system, our customers still won those pools more often than expected, thanks to earlier round picks and using our Alternative brackets.
In theory, as a smart player enters more brackets into a specific pool, two things should happen:
To explain the second point, consider that for any type of bracket pool, there will be one combination of picks that gives you the absolute best chance to win (i.e. the maximum edge over your opponents). A smart player does their best to identify that bracket, and play it as their first entry.
By definition, then, any additional brackets the smart player enters, assuming those brackets have some different picks than the first bracket, are not quite as likely to win compared to the optimal, first bracket played.
Playing more brackets in a pool therefore should increase your overall odds of winning a prize, but your expected return on investment also decreases a bit, since you’re paying the same price to enter the pool with a second, third, etc. bracket that is not quite as good as the first bracket you entered.
As expected, we do see our customers’ relative edge decreasing as they enter more of our recommended brackets in a specific pool. Unfortunately, this year the percentage of our customers winning a prize (“Actual % in the money”) also decreased, once they entered more than two brackets in a pool:
Number of Brackets Entered Expected % in the money Actual % in the money Edge
1 7% 13% 1.8x
2 12% 18% 1.5x
3 14% 15% 1x
more than 3 16% 14% 0.9x
Grand Total 10% 14% 1.4x
However, there’s another angle to consider here. People entering multiple brackets into a pool are generally in larger pools, and as we saw above, our performance in smaller pools this year was much better than our performance in larger pools.
So let’s look at the numbers again, taking pool size into account:
Pool Size: 50 or fewer
Number of Brackets Entered Expected % in the money Actual % in the money Edge
1 11% 21% 2x
2 19% 31% 1.6x
3 25% 26% 1x
more than 3 36% 32% 0.9x
Pool Size: 51 to 250
Number of Brackets Entered Expected % in the money Actual % in the money Edge
1 4% 4% 1x
2 8% 10% 1.3x
3 12% 12% 1x
more than 3 20% 21% 1.1x
Pool Size: more than 250
Number of Brackets Entered Expected % in the money Actual % in the money Edge
1 2% 1% 0.6x
2 2% 0% 0x
3 3% 0% 0x
more than 3 6% 2% 0.3x
The data above is a bit noisy since some of these bins have a pretty small sample size; not many people enter more than 3 brackets in a pool that has fewer than 50 total entries. Still, these trends look closer to what we would expect to see.
When evaluating bracket pick performance, it’s imperative to understand the nature of bracket pools. You’re never expected to win, but when you do win, the return you earn makes up for years and years of trying.
And if you use the right strategies and commit to playing in pools for the long term, the expected returns are extremely compelling. That’s why we’ve made pools an area of focus for TeamRankings, even though we know that big wins aren’t going to come every year (or close to it), and we’ll never be able to promise that to customers.
As objectively as we can measure, our bracket picks delivered a nice edge to our customer base as a whole in 2015. Even though it wasn’t close to our best year, on average our customers were 60% more likely than expected to take home a bracket pool prize, and the year’s worth of bragging rights that come with it. Throw in the fact that we remove all stress and time from the pick-making process, even for pools with crazy scoring systems, and the overall value proposition passes the test.
More importantly, these results continue to validate all the work we’ve put in over the last 10+ years to use math, data, and technology to get an edge in bracket pools.
Of course, we realize that not all of our customers were happy with our picks this year. As much as it bothers us, that will always be the case, as it’s simply an unfortunate reality of our business. When one untimely bounce of a ball can spell doom for a critical pick, the year-to-year variance in our performance is likely to be substantial. In addition, in any given year, our picks for some types of pools (e.g. for seed-based scoring this year) may do quite well, while others don’t.
Regardless, our commitment is to keep improving and refining our methods every year, so that we always offer our customers their best possible chance to win.
We’re excited to improve next year’s product even more.
Measuring how our picks did overall isn’t as easy as you may think. Here’s why:
Given these constraints, the most effective and efficient way for us to measure our bracket pick performance is to have our customers fill out an online survey to tell us how they did in their pools. It’s not a perfect solution; there could be bias related to which customers actually end up responding, and intentionally or not, customers could report incorrect information.
Still, it’s the best we can do. To try to counter any potential response bias, we offered a chance at a nice prize (a $150 Amazon gift card) just for completing the survey, and we discarded any survey answers that seemed to conflict details that the same customer had already entered on our site when setting up his or her pool.
Printed from TeamRankings.com - © 2005-2024 Team Rankings, LLC. All Rights Reserved.