Match Outcome Model

Summary

A multinomial logistic regression that predicts a probability for each of away win, draw, home win for every CPL regular-season match. The model is trained once from a frozen training split, exported as JSON coefficients, and applied to every historical match plus every new match as it lands.

The intent of v1 is a clean, reproducible training pipeline: a config file, fixed train/val/test split, two model families compared honestly, full metric dump, and a backtest report against the Elo-renormalised xPts baseline. Adding features (shot volume, set-piece volume, head-to-head, the Shot Quality Index) becomes a small change to the feature set followed by a retrain.

v1 does not beat the xPts baseline on Brier or log loss. It is shipped anyway: the pipeline itself is the deliverable, the ugly numbers are reported alongside the headline ones, and v1.1 has a clear path to improvement once shot features land.

Data

• Source: matches, team_elo_history, team_form_series from the local CPL database. One row per regular-season match, home perspective.
• Time range: 2019 through 2026 (in-season). 719 regular-season matches with both Elo entries and 5-match rolling form on both sides. Zero matches dropped at v1 because rolling-form gaps for early-season matches are filled with neutral priors (PPG 1.0, GD 0.0, rest 7 days) rather than dropped.
• Excluded: playoff matches.
• Splits:
  • Train: 2019, 2020, 2021, 2022, 2023 (482 matches)
  • Validation (model + hyperparameter selection): 2024 (117 matches)
  • Test (held out, reported once): 2025 (117 matches)
  • 2026 in-season: not used in training, only inference.

Choices

v1 feature set

Four features, deliberately small:

The intercept absorbs the league-wide home advantage, which is a deliberate divergence from the xPts baseline (which hard-codes +100 into the Elo expected-score). Letting the logit learn the home advantage from data is cleaner once the model has more features layered on.

The full candidate list (shot volume, box-shot share, set-piece volume, head-to-head, Shot Quality Index) is deferred to v1.1 for two reasons: (1) raw match_stats xG is identically zero across every season, so the most informative shot feature is unavailable; (2) on 720 matches, every additional feature buys overfitting risk that the small validation set struggles to detect. v1 is the calibration baseline that v1.1 has to clear.

Multinomial vs ordinal logit

Both are fit. Hyperparameters tuned on the validation set. Winner chosen by validation Brier. Multinomial wins on this dataset (val Brier 0.6371 vs ordinal 0.6393), which matches general findings in football modelling: proportional-odds rarely holds because draws are not "halfway between" wins and losses, they're a structurally different outcome.

Standardization

Z-scored on the training set only. Means and standard deviations are frozen into the artifact and applied identically at inference time, so the inference path can never see leakage from validation or test data.

Artifact format: JSON coefficients

Coefficients, intercepts, feature means, and feature stds are written to a plain JSON file. Inference is a hand-rolled pure-Python softmax with zero scikit-learn at runtime. This keeps the production container free of the ML dependency tree and removes any binary-format compatibility risk across redeploys. sklearn is only installed for training.

Frozen training, no automatic retrain

The nightly sync runs inference against the frozen artifact but does not retrain the model. Retraining is an explicit, version-bumped operation followed by a commit, so historical predictions don't silently shift underneath the user.

Per-match storage

Predictions are stored in a team_match_predictions table, two rows per match (one per team), keyed by (team_id, match_id, model_version). The xPts baseline's team_match_expectations table is left in place so the two metrics can be compared side-by-side and the simpler view does not regress when this model ships.

Validation

Headline numbers (model_v1, multinomial logit, C=0.03)

Read the test row honestly: v1 is 0.0019 worse on Brier and 0.0091 worse on log loss than the xPts baseline on the 2025 hold-out. It is 0.034 better on accuracy. Net: with the v1 feature set, the multinomial logit edges out the baseline on the "most-likely outcome" metric but is fractionally worse-calibrated.

v1 is the calibration baseline, not the headline result.

Calibration (test set, home-win column, 10 bins)

Three things stand out: (1) the model is under-confident in the 0.50-0.60 bin where it actually wins way more than it predicts, (2) the high-confidence tails are too small to read meaningfully on 117 matches, (3) the middle bins (0.30-0.50) are over-predicting home wins. Both issues are consistent with a feature set that under-uses home advantage and form, which is the v1.1 fix.

Inference parity

A regression test loads the JSON artifact, runs the pure-Python softmax, and asserts probability vectors sum to 1 within 1e-9. Any drift between sklearn's fit-time probabilities and the JSON-coefficient inference would crash the test suite.

v1.1 investigation: shot features, calibration, walk-forward

After v1 shipped, three follow-up experiments tried to close the small gap to the xPts baseline. None worked. The combined finding is worth more than any of the individual numbers: on this dataset, an Elo rating that already encodes home advantage and an empirical draw rate is at the noise floor of what 720 matches can support, and v1's extra features sit inside that noise floor. Elo with a fitted draw rate is genuinely a hard baseline for an 8-team league.

Shot feature ablation

Added rolling-5 means of shots, box-shot share (shots-at-goal-inside-box / shots), and corners to the v1 feature set. Test season is 2025. Multinomial picked by validation Brier on 2024.

Every shot feature combination is worse than v1 on every metric. Corners is the worst single addition. Plausible reason: rolling team-level shot totals are highly collinear with what Elo already encodes (good teams shoot more), so they add parameters faster than they add signal. The lift from "shot quality" requires shot location or shot xG, neither of which is available in the SDP feed today (expected-goals is identically zero across all seasons).

Calibration ablation

The v1 model loses to baseline by 0.0019 Brier on test. Three calibration methods, each fit on the 2024 validation set only, then applied to the 2025 test set:

Temperature scaling is the cleanest read: with one parameter and mathematically guaranteed not to hurt accuracy, it found T ≈ 1, which is the optimiser telling us there is no calibration headroom on this validation set. Isotonic and Platt both burn validation degrees of freedom and overfit immediately. The full calibrator infrastructure stays available for future phases, but v1 ships uncalibrated.

Walk-forward backtest

To escape the noise floor of a single 117-match test set, train a fresh v1 multinomial on every season strictly before season Y, then score on Y, for Y in 2021..2025. Hyperparameter C chosen in-fold by holding out the most recent training season. Compare per-season and pooled against the xPts baseline on identical matches.

Pooled (n=583):

Read this carefully:

• On Brier and log loss, v1 and the xPts baseline are statistically indistinguishable across 583 walk-forward matches. The sign of the per-season difference flips (v1 wins 2023 by a meaningful margin, loses 2021 and 2024 by smaller ones, ties 2022 and 2025).
• On accuracy, v1 wins by +2.2 percentage points. That is 13 more correct picks out of 583. Modest, but consistent across the seasons where it doesn't lose Brier outright.
• Translation: v1 makes slightly different calls than the Elo renormalisation, those calls happen to land on the most-likely outcome a bit more often, and the calibration is the same to within noise.

Why Elo is the hard baseline here

Three structural reasons specific to CPL:

1. The xPts baseline isn't "just Elo". It's Elo plus a hand-coded +100 home advantage plus an empirically-fitted league-wide draw rate. Three signals fused with strong priors before any model touches the numbers.
2. CPL is small. Eight teams, ~117 matches per season, every pair meets ~3-4 times. Elo converges fast on that. By matchday 5 the ratings already separate the league. Rolling-5 form features have little to add that the Elo update doesn't already capture.
3. The features that would actually beat Elo are unavailable. Shot quality (real xG by location), lineup quality (who is actually starting today), and live in-match state are the levers that move football models in the public literature. The Shot Quality Index targets the first of those. The rest are dataset extensions beyond this model's current scope.

The takeaway is a positive one: the v1 pipeline is correct, the backtest is rigorous, and the conclusion is "the right next move is the Shot Quality Index, not more feature hunting on this data."

Failure modes

• v1 does not beat the baseline on calibration metrics. Documented loudly above. Shipping the pipeline anyway.
• xG unavailable. The SDP match_stats blob has an expected-goals field but it is identically zero across all seasons. This is the single biggest gap between v1 and "a real match outcome model". The Shot Quality Index is the planned fix.
• Tiny validation and test sets. 117 matches each. A single bin in the calibration plot can swing meaningfully from one match. Don't read too much into individual bins.
• 2020 Island Games in train. 35 matches in a bubble with zero travel, no rest variance, and atypical home advantage. Currently treated as a normal training season. A future revision could exclude it or carry a season-fixed-effect.
• Neutral-prior fill for early-season form. First-of-season matches get PPG=1.0, GD=0.0, rest=7. The first three or four matchdays of every season are therefore predicted using slightly washed-out features, and the model's behavior on those matches is effectively "Elo only". Acceptable today, but worth tracking.
• Frozen home advantage. Baked into the model intercept at training time. If CPL home advantage shifts (rule changes, crowd patterns, expansion teams with weak home stadia), the model won't notice until the next retrain.
• No accounting for in-season strength swings. Rolling form is the only "what is this team doing right now" signal. Injuries, manager changes, and lineup quality are not features.

What would make it better

Reordered after the v1.1 investigation. The bottom four ideas are unchanged from v1; the top three reflect what the ablations actually showed.

1. Add the Shot Quality Index as a feature. The v1.1 shot ablation made it concrete: team-level shot totals do not help on this dataset, because Elo already encodes "good teams shoot more". Real shot quality (location-based xG) is the feature that Elo cannot see, and SQI is the closest proxy currently available.
2. Lineup quality features. Who is actually starting (and who is injured / suspended) is a signal Elo cannot capture and the v1 feature set does not even attempt. Requires lineup data ingestion.
3. Stop looking for v1.x improvements on the existing feature surface. The v1.1 walk-forward shows v1 and the xPts baseline are within sampling noise on Brier and log loss across 583 matches. There is no calibration headroom. Any further feature-engineering experiments on team-level totals are likely to replicate the v1.1 result.
4. Per-team home advantage. Today the intercept averages over the league. A team-level fixed effect is one extra parameter per club and might help calibration in the higher-confidence bins. Worth trying once an SQI feature exists, since the larger feature set will have more headroom for the team intercepts to interact with.
5. Set-piece volume from match_events corner / free kick counts. Worth trying alongside SQI, but on the v1.1 evidence it is unlikely to help on its own.
6. Head-to-head history with a small window (last 4 meetings) and a long-run prior shrink. Same caveat: deferred to a retest alongside SQI.
7. Bigger leagues / more matches. Not actionable from inside this project, but worth stating: most of v1's pain is sample size, not modelling choices. With 380 matches per season and 20 teams, the gap between an Elo baseline and a multi-feature model would be meaningfully larger.

Live diagnostics

The numbers above are frozen test-set results. The section rendered below this document is live: after every daily sync, the same metrics (Brier, log loss, accuracy, and the 10-bin home-win calibration table as a reliability diagram) are recomputed over the current season's scored predictions, with the Elo baseline evaluated over the same matches. Predictions are stored pre-match from point-in-time features, so the season-to-date numbers are honest forecasts, not hindsight. Early in a season the bins are thin — the section flags samples under 50 matches. Served by /api/model-diagnostics.