Auditing The 2013-2014 Projections

A little bit later on today, the projections for the 2015 season will go live. Before the 2015 projections go up, though, I wanted to take a look back at how the 2014 version panned out.

First of all, I think it's important to point out what these projections are. When you look at the list of teams, what you're seeing is the system's best guess at how that team will perform in the season in question. It's not an absolute prediction, though; the numbers listed are merely the middle of a whole spectrum of possibilities for each team. 

Projection Accuracy

The first and best way to test the effectiveness of the system is to identify where each team's 2014 season fell on that spectrum. A team that blew its preseason projection out of the water will have a score approaching 1; a team whose season was a complete failure relative to expectations would be a 0; and a team whose season was right in line with the projection would be 0.5.

This histogram plots the results of this analysis. With 351 teams divided into 10 buckets, each one should have 35 teams in it. As you can see, the spread is pretty even, with a bit of skew to the far right. There were 13 more teams than expected who exceeded their 90th percentile projection, but the ranges that were short were distributed seemingly randomly across the rest of the bunch. It's something I'll keep an eye on--it's possible that I'm double-dipping on some downside risk that's already built into the player projections or something like that--but I'm not making any changes just yet.

Wins & Losses

TAPE ratings and the numbers behind the numbers are all well and good, but what's really important is how those numbers translate into the real world: how did the system do at projecting wins and losses.

Before the season begins it's impossible to make a prediction about teams' overall wins and losses because so many games and matchups are unknown. Only the first games of holiday tournaments are known before hand, while postseason tournaments obviously aren't seeded and set until after the regular season ends. Conference games are known, however, and we can use those as a barometer.

For the 2014 season, the average error of projected conference wins was 2.13 wins. Of the 350 teams who played conference schedules, 222 finished within 2 wins either way of their preseason projection.

Postseason Honors

The system also assigns the odds for teams winning their conference and for receiving NCAA Tournament bids. For the conference winners, the value assigned is left a little bit ambiguous (intentionally so), but it's really just the odds of each team finishing with the highest TAPE rating in their league. On that front, it did better than expected, correctly picking 19 of the 32 teams finishing with that distinction. 

Conference	Predicted Winner	Reg. Season Champ	Top TAPE Rank	NCAA Auto-Bid
Amerca East	Vermont (50%)	Vermont	Vermont	Albany (7%)
American Athletic	Louisville (52%)	Cincinnati (1%)	Louisville	Louisville
Atlantic 10	Virginia Commonwealth (39%)	Saint Louis (20%)	Virginia Commonwealth	St. Joseph's (1%)
Atlantic Coast	Duke (32%)	Virginia (28%)	Virginia	Virginia
Atlantic Sun	Florida Gulf Coast (46%)	Florida Gulf Coast	Mercer (35%)	Mercer
Big 12	Kansas (39%)	Kasas	Kansas	Iowa State (10%)
Big East	Marquette (31%)	Villanova (7%)	Villanova	Providence (3%)
Big Sky	Weber State (40%)	Weber State	Weber State	Weber State
Big South	Charleston Southern (49%)	High Point (25%)	Charleston Southern	Coastal Carolina (4%)
Big Ten	Michigan (31%)	Wisconsin (19%)	Michigan	Michigan State (21%)
Big West	UC Irvine (22%)	UC Irvine	UC Irvine	Cal Poly (20%)
Colonial	Northeastern (36%)	Delaware (3%)	Delaware	Delaware
Conference USA	Southern Miss (46%)	Louisiana Tech (37%)	Louisiana Tech	Tulsa (3%)
Horizon	Wright State (61%)	Green Bay (17%)	Green Bay	Milwaukee (<1%)
Ivy League	Harvard (75%)	Harvard	Harvard	Harvard
Metro-Atlantic	Iona (31%)	Iona	Manhattan (23%)	Manhattan
Mid-American	Buffalo (23%)	Western Michigan (21%)	Buffalo	Western Michigan
Mid-Eastern Athletic	Norfolk State (27%)	N.C. Central (25%)	N.C. Central	N.C. Central
Missouri Valley	Wichita State (69%)	Wichita State	Wichita State	Wichita State
Mountain West	New Mexico (40%)	San Diego State (16%)	San Diego State	New Mexico
Northeast	Wagner (33%)	Robert Morris (16%)	Robert Morris	Mount St. Mary's (10%)
Ohio Valley	Belmont (48%)	Belmont	Belmont	Eastern Kentucky (30%)
Pacific 12	Arizona (48%)	Arizona	Arizona	UCLA (16%)
Patriot	Boston U. (41%)	Boston U.	American (1%)	American
Southeastern	Florida (53%)	Florida	Florida	Florida
Southern	Wofford (33%)	Davidson (17%)	Davidson	Wofford
Southland	Northwestern State (53%)	Stephen F. Austin (6%)	Stephen F. Austin	Stephen F. Austin
Summit	North Dakota State (67%)	North Dakota State	North Dakota State	North Dakota State
Sun Belt	Georgia State (23%)	Georgia State	Georgia State	Louisiana-Lafayette (6%)
Southwestern Athletic	Southern (68%)	Southern	Southern	Texas Southern (20%)
West Coast	Gonzaga (38%)	Gonzaga	Gonzaga	Gonzaga
Western Athletic	New Mexico State (80%)	New Mexico State	New Mexico State	New Mexico State
# Correct	Predicted: 14.2	17	19	11

The at-large odds are really just the odds of each team finishing in the national top 50  in the TAPE ratings. The expectation was that 34 or 35 would be correct. The preseason top 50 included 38 teams who finished the season in the top 50, and 34 teams who received 12-seeds or better in the NCAA Tournament.

All in all, a pretty successful season for the projections. Unlike the previous two seasons, no major overhauls are warranted. I've made a few tweaks here and there that I think will improve the forecasts a little bit. Plus, with another season's worth of player comps in the books, the individual player projections should be better as well.