Preseason Prior Backtest: Team Elo With and Without the Player-Based Start
NCAAM team Elo now starts each season from a preseason prior instead of last season's rating alone. The prior adds what we know about the roster: who returns, who left, who transferred in, and the incoming freshmen. This report asks how much each of those parts helps, measured on seasons the prior never saw.
Short answer. Starting from the full prior cuts early-season log loss by about 5% (weeks 1-4: 0.520 to 0.493). Early accuracy rises by one point (73.7% to 74.7%), and the rank correlation of the start with the season's final strength rises from 0.72 to 0.83. Most of the gain comes from transfers in. Freshmen add a smaller but clear improvement. Returning and departing players add almost nothing beyond last season's rating.
Player models. Valuing players by last season's published player-model rating (RAPM) instead of box-score production makes the prior worse. Adding RAPM alongside box-score production makes it a little better: 4.5 milli-nats in weeks 1-4 on the two seasons we can test. Women's prior (added 2026-10-25). A roster-only NCAAW prior beats carryover in every held-out season: weeks 1-4 log loss 0.441 against 0.481.
The starts compared
Every variant runs the production NCAAM Elo (K 48, home advantage 100, margin-of-victory scaling) over the same games. Only the rating each team starts the season with differs.
- Base, carryover: last season's final Elo, regressed toward the mean the way Elo does between seasons. This is what the team models did before the prior.
- Base, as the prior fits it: the prior's last-season term alone, with roster and class held at league average. It is not a standalone model. On its own it regresses more than carryover does, because in the full model the roster terms carry part of last season's strength.
- + returning and departing players: adds returning minute share, the production of returning players (and its interaction with last season's rating), and the production of players who left.
- + transfers in: adds the production incoming transfers had at their previous school. This is the full roster part.
- + freshmen by 247 tier: adds counts of 247-ranked freshmen by tier (5-star, 4-star, 3-star, lower-ranked) and of other D1 debutants. This is the full prior, the one the site uses.
Each start is the prior's estimate of the season's SRS, converted at 26 Elo per SRS point. The parts add up, so each row adds one part to the row above.
Test 1: the three seasons after the fit (2023-24 to 2025-26)
The prior's coefficients were refit on the SportsRef seasons 2011-12 through 2022-23 only. The three ESPN seasons below were never used to fit it. Team membership in each season, and the production of returning players and transfers, come from ESPN box scores. The games scored are the ones the production team models read.
Log loss (lower is better). In parentheses: the change against carryover in thousandths (milli-nats), with a 95% paired-bootstrap interval. A negative change means the start beats carryover.
| Start | Weeks 1-2 (1,871 games) | Weeks 1-4 (3,804 games) | Weeks 1-6 (5,203 games) | Season (17,243 games) |
|---|---|---|---|---|
| Base: last season's Elo, regressed (carryover) | 0.4672 | 0.5203 | 0.5205 | 0.5562 |
| Base, as the prior fits it (average roster and class) | 0.4768 (+9.7 [+4.6, +15.0]) | 0.5221 (+1.8 [-1.9, +5.5]) | 0.5224 (+1.9 [-1.1, +5.0]) | 0.5566 (+0.4 [-0.7, +1.6]) |
| + returning and departing players | 0.4714 (+4.2 [-3.9, +12.5]) | 0.5191 (-1.2 [-7.1, +4.7]) | 0.5190 (-1.5 [-6.4, +3.2]) | 0.5546 (-1.6 [-3.3, -0.0]) |
| + transfers in (the full roster part) | 0.4418 (-25.4 [-33.5, -17.1]) | 0.4971 (-23.2 [-28.9, -17.3]) | 0.4998 (-20.7 [-25.5, -16.0]) | 0.5475 (-8.7 [-10.4, -7.1]) |
| + freshmen by 247 tier (the full prior) | 0.4312 (-35.9 [-45.6, -26.6]) | 0.4928 (-27.5 [-34.4, -20.9]) | 0.4948 (-25.7 [-31.3, -20.4]) | 0.5456 (-10.6 [-12.6, -8.6]) |
Accuracy (share of games the favorite won):
| Start | Weeks 1-2 | Weeks 1-4 | Weeks 1-6 | Season |
|---|---|---|---|---|
| Base: last season's Elo, regressed (carryover) | 77.8% | 73.7% | 73.3% | 70.6% |
| Base, as the prior fits it (average roster and class) | 77.8% | 73.9% | 73.4% | 70.6% |
| + returning and departing players | 77.9% | 74.2% | 73.9% | 71.0% |
| + transfers in (the full roster part) | 79.3% | 75.0% | 74.3% | 71.1% |
| + freshmen by 247 tier (the full prior) | 79.2% | 74.7% | 74.2% | 71.0% |
By season. The rank correlation compares each team's start with its SRS for that season (Spearman; the season's own end-of-season strength).
| Season | Start | Log loss, weeks 1-4 | Log loss, season | Accuracy, weeks 1-4 | Rank correlation with final SRS |
|---|---|---|---|---|---|
| 2023-24 | Carryover | 0.5490 | 0.5660 | 72.6% | 0.72 |
| 2023-24 | Base (prior's term) | 0.5491 | 0.5660 | 73.2% | 0.72 |
| 2023-24 | + returning and departing | 0.5425 | 0.5629 | 73.3% | 0.74 |
| 2023-24 | + transfers in (roster) | 0.5273 | 0.5569 | 73.4% | 0.82 |
| 2023-24 | + freshmen (full prior) | 0.5237 | 0.5555 | 73.4% | 0.84 |
| 2024-25 | Carryover | 0.5235 | 0.5482 | 72.3% | 0.71 |
| 2024-25 | Base (prior's term) | 0.5234 | 0.5488 | 72.3% | 0.71 |
| 2024-25 | + returning and departing | 0.5235 | 0.5475 | 73.1% | 0.71 |
| 2024-25 | + transfers in (roster) | 0.4996 | 0.5396 | 73.9% | 0.81 |
| 2024-25 | + freshmen (full prior) | 0.4979 | 0.5382 | 73.6% | 0.82 |
| 2025-26 | Carryover | 0.4873 | 0.5545 | 76.3% | 0.72 |
| 2025-26 | Base (prior's term) | 0.4927 | 0.5552 | 76.2% | 0.72 |
| 2025-26 | + returning and departing | 0.4902 | 0.5534 | 76.2% | 0.70 |
| 2025-26 | + transfers in (roster) | 0.4632 | 0.5461 | 77.8% | 0.81 |
| 2025-26 | + freshmen (full prior) | 0.4554 | 0.5431 | 77.1% | 0.82 |
The full prior beats carryover in every season and every window. The gain is largest in the first two weeks and shrinks as games replace the prior: Elo's own updates catch up by midseason. The gain still shows over the full season (-10.6 milli-nats).
Test 2: the earlier experiment (2017-18 to 2022-23)
The recruiting experiment (docs/recruiting_elo_results.md) fit the prior on 2011-12
through 2016-17 and tested it on 2017-18 through 2022-23, using SportsRef box scores. Its
roster variant bundles returning, departing and transfer production into one step, so it
cannot separate transfers. It does confirm the order on six more seasons:
| Start | Weeks 1-2 (3,580) | Weeks 1-4 (6,936) | Weeks 1-6 (9,398) | Season (28,982) | Accuracy, weeks 1-4 | Rank correlation with final SRS |
|---|---|---|---|---|---|---|
| Base: carryover | 0.4656 | 0.4948 | 0.4951 | 0.5443 | 74.6% | 0.78 |
| Base: fitted regression on last season only | 0.4659 | 0.4945 | 0.4947 | 0.5439 | 74.7% | 0.78 |
| + returning, departing and transfer production (roster) | 0.4476 | 0.4808 | 0.4838 | 0.5398 | 76.0% | 0.82 |
| + freshmen by 247 tier (full prior) | 0.4400 | 0.4740 | 0.4766 | 0.5366 | 76.3% | 0.84 |
Paired differences, weeks 1-4: roster against the base, -13.7 milli-nats [-18.0, -9.4]. Freshmen tiers against a flat count of debutants, -6.6 [-9.1, -4.2]. A flat count of debutants at replacement level adds nothing (-0.2, interval spans zero).
What this says
- Transfers carry the prior. In 2023-24 to 2025-26, adding incoming transfers' past production is worth about 22 milli-nats in weeks 1-4, on top of returning players. That is most of the total gain, and it lifts the rank correlation from about 0.72 to 0.81 each season. Returning and departing players alone barely move the start: last season's Elo already reflects most of what returners did.
- Freshmen help, mostly through calibration. Adding the 247 tiers lowers log loss in every window (weeks 1-2: -10.5 milli-nats beyond the roster). Accuracy does not move: freshmen change how confident the favorite is more often than who the favorite is.
- A fitted base is no better than Elo's own carryover (Test 2, a fitted regression on last season only, matches carryover). The prior helps only through what it knows about the roster.
Valuing players by our player models (RAPM)
The roster terms above value each player by box-score production. Here each player is instead valued by last season's published player-model rating: the live RAPM of the v2 player models, as 5 × minutes share × RAPM. That is the player's share of his team's net rating per 100 possessions. A player without a RAPM counts as average (0). Two more priors were fit, each on SportsRef 2011-12 to 2022-23 only, like the one above:
- RAPM instead: RAPM replaces Game Score in the returning, departing and transfer terms.
- Game Score + RAPM: three RAPM terms are added beside the Game Score ones.
Players are matched to their RAPM by name within a season for the SportsRef seasons (the name must be unique on both sides): 94-96% of minutes. For the ESPN seasons they are matched through the ESPN id's player hub, then by name: 97-99% of minutes for 2020-21 to 2023-24.
Only two seasons can be tested. 2025-26 starts from 2024-25, and 2024-25 has no v2 player-model fit yet (NCAA.com play-by-play covers 39% of its seconds; using Stats.NCAA is an open question for the owner). Its live RAPM is the legacy run on other player hubs, reachable for only 46% of minutes. So this comparison uses 2023-24 and 2024-25 (2,558 games in weeks 1-4, 11,490 in the season), every start including freshmen:
| Roster valued by | Weeks 1-2 | Weeks 1-4 | Weeks 1-6 | Season | Accuracy, weeks 1-4 | Rank correlation with final SRS (2023-24, 2024-25) |
|---|---|---|---|---|---|---|
| Carryover (no prior) | 0.4856 | 0.5364 | 0.5315 | 0.5571 | 72.4% | 0.72, 0.71 |
| Game Score (the site today) | 0.4544 | 0.5110 | 0.5076 | 0.5468 | 73.5% | 0.84, 0.83 |
| RAPM instead | 0.4600 | 0.5144 | 0.5120 | 0.5492 | 73.6% | 0.82, 0.82 |
| Game Score + RAPM | 0.4500 | 0.5065 | 0.5043 | 0.5456 | 74.0% | 0.85, 0.84 |
Change against Game Score, milli-nats (95% paired bootstrap):
| Weeks 1-2 | Weeks 1-4 | Weeks 1-6 | Season | |
|---|---|---|---|---|
| RAPM instead | +5.6 [-1.9, +13.6] | +3.5 [-2.0, +9.1] | +4.4 [+0.2, +8.7] | +2.4 [+0.8, +3.9] |
| Game Score + RAPM | -4.4 [-9.3, +0.5] | -4.5 [-7.8, -1.1] | -3.3 [-6.0, -0.6] | -1.2 [-2.2, -0.3] |
Without freshmen the picture is the same: RAPM instead of Game Score, +4.8 [-0.6, +10.5] in weeks 1-4.
- RAPM alone is a worse roster value than box-score production. One college season of RAPM is noisy (a typical standard error is 2-3 points per 100 possessions), while Game Score per 40 minutes is stable.
- RAPM beside Game Score helps a little. It is worth about a sixth of the prior's total gain in weeks 1-4, and it lifts the rank correlation by 0.01-0.02. Part of it is not roster information. In that fit, departing players' RAPM gets a positive weight, so the RAPM terms also act as a second measure of last season's team strength next to Elo (on-court RAPM adds up to an adjusted team rating).
- Before it can feed the live prior, one identity gap needs fixing. 680 players with 16% of 2025-26 minutes have their v2 RAPM on a player hub that their ESPN id does not reach. Bennett Stirtz is one: his Drake 2024-25 season and his ESPN id sit on one hub, his Iowa 2025-26 NCAA.com link on another. The 2026-27 prior reads ESPN rosters, so those players would count as average.
Is a women's prior worth building?
Before 2026-10-25 there was no NCAAW prior, so NCAAW's 2026-27 table started from carryover. A cheap check used the same features and method for both leagues on 2023-24 to 2025-26: a leave-one-season-out regression of each team's season SRS on last season's Elo, with and without the returning, departing and transfer terms (box-score production, no freshmen), scored on the held-out season:
| League | Last season's Elo only: R², rank correlation | + roster terms: R², rank correlation | Incoming transfers' share of minutes |
|---|---|---|---|
| NCAAW | 0.59, 0.75 | 0.73, 0.84 | 15%, 20%, 24% |
| NCAAM | 0.54, 0.72 | 0.68, 0.82 | 23%, 30%, 35% |
The roster terms explain about as much more of the season in the women's game as in the men's (+0.14 R² against +0.15). A women's roster prior is worth building. It would start without a freshmen part: we hold 216 women's recruits against 7,149 men's. This is a regression check, not an Elo backtest; the Elo backtest would follow the build.
The women's Elo backtest
The build: the same stage-1 regression without the debutant and freshman terms, fit on the four ESPN seasons with complete box scores before them (2022-23 to 2025-26; women's ESPN box scores are partial before 2021-22). Elo points per SRS point are estimated the same way (final Elo against SRS: 21.1 for the women, 26 for the men). Each season's prior is fit on the other three seasons only. Production NCAAW Elo (K 64, home advantage 150, carryover 0.85) then runs the held-out season from each start. Last season's Elo comes from an in-memory chain from 2016-17.
| Start | Weeks 1-2 | Weeks 1-4 | Weeks 1-6 | Full season | Favorite right, weeks 1-4 | Rank correlation with final SRS |
|---|---|---|---|---|---|---|
| Carryover | 0.460 | 0.481 | 0.474 | 0.502 | 75.8% | 0.72-0.78 |
| Roster prior | 0.408 | 0.441 | 0.439 | 0.488 | 78.1% | 0.84-0.86 |
Log loss over 5,026 games in weeks 1-4 (22,310 over full seasons). The prior is better in
every season and every window: -40 milli-nats in weeks 1-4 [-47, -33] (paired bootstrap)
and -14 over the full season [-16, -12]. That is a bigger gain than the men's, partly
because women's carryover starts further from the truth. The live spec is refit on all
four seasons (config/elo_priors.json, ncaaw-roster-v1), and the 2026-27 NCAAW table
starts from it for every team whose 2026-27 ESPN roster is published. Freshmen are listed
on each team's roster page, but they carry no Elo.
Limits
- Membership is hindsight. In both tests, a team's players are those in that season's box scores, the proxy the prior was fit on. A real preseason does not know about injuries, redshirts or midseason departures, so these numbers flatter the roster parts somewhat. The live prior reads ESPN's preseason rosters instead.
- Player value is box-score production (Game Score per 40 minutes, weighted by minutes) in the main tables. The RAPM comparison above covers only two seasons.
- Freshmen enter as 247 composite tier counts. About 80% of D1 debutants have no 247 row; they count only as debutants. A recruiting model that scores each freshman individually tied the tiers out of sample (weeks 1-4: 0.0 milli-nats [-0.8, +0.7], variant e in the experiment), so the prior keeps the tiers.
- Freshmen are placed by the published roster in the live prior only. Since 2026-10-24 a freshman counts for the team whose 2026-27 ESPN roster lists him, and not at all when his program's published roster does not. That covers early commitments the class rule dates a year early, and recruits who went elsewhere: Kevin Thomas counts for North Carolina, not LSU. The tested seasons have no historical rosters and keep the 247 program. A proxy check placed freshmen by each season's box-score newcomers instead: weeks 1-4 log loss +0.7 milli-nats, rank correlation 0.830 -> 0.825 on average, within noise. That proxy also drops redshirted freshmen, whom the roster rule keeps.
- The women's backtest has four seasons and no freshmen part, and in-sample Elo scaling (Elo points per SRS point is estimated on all four seasons).
- The live prior's coefficients are refit on every season through 2025-26
(
config/elo_priors.json), so 2026-27 starts use slightly different coefficients from the ones tested here.
How it was run
Read-only, from the recruiting experiment's data and production's games:
python -m modeling.backtest_elo_priors --data <recruiting data dir> --components components.json --rapm <rapm dir>
--rapm points at a read-only extract of the live rapm_ratings (ncaam), the ESPN id to
player hub links and ESPN player names.
component_backtest in modeling/backtest_elo_priors.py refits the prior on SportsRef
2011-12 to 2022-23. For each ESPN season it builds the season's team features from box
scores and starts Elo from each cumulative part of the prior. It runs the season and
scores every game by week window. The pooled intervals are paired bootstraps over games
against carryover. Test 2's numbers are elo_results.json from
modeling/recruiting/run_experiment.py.