Tuesday, April 8, 2014

Auditing The 2013-2014 Projections

A little bit later on today, the projections for the 2015 season will go live. Before the 2015 projections go up, though, I wanted to take a look back at how the 2014 version panned out.

First of all, I think it's important to point out what these projections are. When you look at the list of teams, what you're seeing is the system's best guess at how that team will perform in the season in question. It's not an absolute prediction, though; the numbers listed are merely the middle of a whole spectrum of possibilities for each team. 

Projection Accuracy

The first and best way to test the effectiveness of the system is to identify where each team's 2014 season fell on that spectrum. A team that blew its preseason projection out of the water will have a score approaching 1; a team whose season was a complete failure relative to expectations would be a 0; and a team whose season was right in line with the projection would be 0.5.

This histogram plots the results of this analysis. With 351 teams divided into 10 buckets, each one should have 35 teams in it. As you can see, the spread is pretty even, with a bit of skew to the far right. There were 13 more teams than expected who exceeded their 90th percentile projection, but the ranges that were short were distributed seemingly randomly across the rest of the bunch. It's something I'll keep an eye on--it's possible that I'm double-dipping on some downside risk that's already built into the player projections or something like that--but I'm not making any changes just yet.

Wins & Losses

TAPE ratings and the numbers behind the numbers are all well and good, but what's really important is how those numbers translate into the real world: how did the system do at projecting wins and losses.

Before the season begins it's impossible to make a prediction about teams' overall wins and losses because so many games and matchups are unknown. Only the first games of holiday tournaments are known before hand, while postseason tournaments obviously aren't seeded and set until after the regular season ends. Conference games are known, however, and we can use those as a barometer.
For the 2014 season, the average error of projected conference wins was 2.13 wins. Of the 350 teams who played conference schedules, 222 finished within 2 wins either way of their preseason projection.

Postseason Honors

The system also assigns the odds for teams winning their conference and for receiving NCAA Tournament bids. For the conference winners, the value assigned is left a little bit ambiguous (intentionally so), but it's really just the odds of each team finishing with the highest TAPE rating in their league. On that front, it did better than expected, correctly picking 19 of the 32 teams finishing with that distinction. 
Conference Predicted Winner Reg. Season Champ Top TAPE Rank NCAA Auto-Bid
Amerca East Vermont (50%) Vermont Vermont Albany (7%)
American Athletic Louisville (52%) Cincinnati (1%) Louisville Louisville
Atlantic 10 Virginia Commonwealth (39%) Saint Louis (20%) Virginia Commonwealth St. Joseph's (1%)
Atlantic Coast Duke (32%) Virginia (28%) Virginia Virginia
Atlantic Sun Florida Gulf Coast (46%) Florida Gulf Coast Mercer (35%) Mercer
Big 12 Kansas (39%) Kasas Kansas Iowa State (10%)
Big East Marquette (31%) Villanova (7%) Villanova Providence (3%)
Big Sky Weber State (40%) Weber State Weber State Weber State
Big South Charleston Southern (49%) High Point (25%) Charleston Southern Coastal Carolina (4%)
Big Ten Michigan (31%) Wisconsin (19%) Michigan Michigan State (21%)
Big West UC Irvine (22%) UC Irvine UC Irvine Cal Poly (20%)
Colonial Northeastern (36%) Delaware (3%) Delaware Delaware
Conference USA Southern Miss (46%) Louisiana Tech (37%) Louisiana Tech Tulsa (3%)
Horizon Wright State (61%) Green Bay (17%) Green Bay Milwaukee (<1%)
Ivy League Harvard (75%) Harvard Harvard Harvard
Metro-Atlantic Iona (31%) Iona Manhattan (23%) Manhattan
Mid-American Buffalo (23%) Western Michigan (21%) Buffalo Western Michigan
Mid-Eastern Athletic Norfolk State (27%) N.C. Central (25%) N.C. Central N.C. Central
Missouri Valley Wichita State (69%) Wichita State Wichita State Wichita State
Mountain West New Mexico (40%) San Diego State (16%) San Diego State New Mexico
Northeast Wagner (33%) Robert Morris (16%) Robert Morris Mount St. Mary's (10%)
Ohio Valley Belmont (48%) Belmont Belmont Eastern Kentucky (30%)
Pacific 12 Arizona (48%) Arizona Arizona UCLA (16%)
Patriot Boston U. (41%) Boston U. American (1%) American
Southeastern Florida (53%) Florida Florida Florida
Southern Wofford (33%) Davidson (17%) Davidson Wofford
Southland Northwestern State (53%) Stephen F. Austin (6%) Stephen F. Austin Stephen F. Austin
Summit North Dakota State (67%) North Dakota State North Dakota State North Dakota State
Sun Belt Georgia State (23%) Georgia State Georgia State Louisiana-Lafayette (6%)
Southwestern Athletic Southern (68%) Southern Southern Texas Southern (20%)
West Coast Gonzaga (38%) Gonzaga Gonzaga Gonzaga
Western Athletic New Mexico State (80%) New Mexico State New Mexico State New Mexico State
# Correct Predicted: 14.2 17 19 11

The at-large odds are really just the odds of each team finishing in the national top 50  in the TAPE ratings. The expectation was that 34 or 35 would be correct. The preseason top 50 included 38 teams who finished the season in the top 50, and 34 teams who received 12-seeds or better in the NCAA Tournament.

All in all, a pretty successful season for the projections. Unlike the previous two seasons, no major overhauls are warranted. I've made a few tweaks here and there that I think will improve the forecasts a little bit. Plus, with another season's worth of player comps in the books, the individual player projections should be better as well. 

No comments: