Shot Quality

Summary

Shot Quality (SQI) is not xG. It is a Poisson regression that maps a team's per-match shot-bucket counts to an expected goal total, fit on historical CPL matches. It lives at /analysis/shot-quality as a team-season over/underperformance table: actual goals vs SQI-expected goals, sorted by Δ. The per-match SQI values also feed SQI Expected Points, which converts each match's SQI pair into win/draw/loss probabilities — the principled successor to the threshold-based SQI standings on the same page.

The "not xG" distinction is the whole reason this metric exists. Canadian Premier League data does not include shot locations, body part, defender pressure, keeper position, assist type, or any situational context — only bucketed counts (shots inside the box, shots outside the box, big chances, shots on goal, penalty goals). We cannot build a real xG model on top of counts alone. What we can build is a calibrated "given this team's typical shot mix, how many goals would a league-average finisher convert" and ship it with loud disclaimers about what it is not.

This honesty is the point. Anywhere you see SQI in the UI or code, it should be clear what is being measured and what is not.

This is NOT xG

Real xG (Opta, StatsBomb, Wyscout) uses a per-shot model with features including:

• Shot coordinates — distance to goal, angle to the posts
• Body part — foot, head, other (materially different conversion rates)
• Assist type — cutback, through ball, cross, set piece, rebound
• Situation — open play, counter-attack, set piece, penalty
• Defender pressure — defenders between shooter and goal
• Keeper position — set / off-line / advancing
• Technique — first-time, volley, chest, deflection

SQI uses none of those. It only knows:

• How many shots you had from inside the box
• How many you had from outside the box
• How many "big chances" the scorer flagged
• How many of your shots were on target
• How many penalty goals you scored

This means a team with 10 shots from the edge of the six-yard box, all first-time volleys with defenders leaning the wrong way, gets the exact same box_shots = 10 as a team with 10 speculative toes from 17 yards surrounded by three defenders. SQI cannot tell the difference. Two teams with identical bucket counts get identical SQI, even if one was carving a weak defence open and the other was taking low-percentage shots to look busy.

The upgrade path that would make SQI a real xG model is a shot-level data source. CPL does not have one. The SDP match_stats expected-goals column exists but is identically zero across all seasons. Until that changes, SQI is what's possible.

Data

• Source: match_stats (shot buckets) joined to matches (actual goals), one row per (team, match).
• Time range: 2019-2025, regular season only. 1438 team-match rows after the playoff filter, zero dropped to missing features.
• Target: home_goals or away_goals from the matches table, per team. Includes own goals scored by the opponent against this team. With ~9-12 own goals per season (out of ~175 goals/season), this is small noise and including them simplifies the pipeline. The methods doc flags it loudly rather than silently excluding.
• Features (in canonical order):
  1. box_shots — shots-at-goal-inside-box
  2. out_box_shots — shots-at-goal-outside-box
  3. big_chances — big-chances (SDP's own "clear look" flag)
  4. penalty_goals — penalty-goals (see discussion below)
  5. shots_on_goal — shots-on-goal
• Excluded: shots total (linearly dependent on box + out_box), blocked-shots (redundant), own-goals-scored (the team's own attackers didn't score those), corners, free kicks.

Choices

Why Poisson regression

Goal counts are non-negative integers with a small mean (~1.27 goals per team per match, train set). Poisson is the textbook distribution for this. The log link naturally handles the "more shots → more goals, diminishing returns" curve. On 964 training rows, a linear model is the ceiling of what's defensible — anything fancier would overfit into noise.

Why penalty_goals is a feature, not a target adjustment

Penalties are rare (~25 per season) and convert at ~0.75 in CPL. Two options:

1. Subtract penalty goals from the target and drop the feature
2. Keep penalty goals in the target and include them as a feature

Option 2 is the honest one. A penalty goal is still a goal — if you want SQI to explain total goals, you need the model to account for them. Including penalty_goals as a feature lets the Poisson fit learn how much a penalty goal adds to log-expected-goals. The fitted coefficient is exp(0.198) = 1.22, which means each penalty goal marked in a match multiplies the expected goal total by ~1.22 on top of whatever the other features contribute. That is not "one extra expected goal per penalty" — Poisson log-link is multiplicative, not additive, and the cleaner interpretation is "matches with a penalty goal tend to be higher-scoring matches overall, beyond what the raw shot counts explain".

Negative coefficients on box_shots and out_box_shots

This is worth calling out because it looks wrong and isn't.

The fitted coefficients:

shots_on_goal is a subset of box_shots + out_box_shots. The regression reads that as: "given how many of your shots were on target, more off-target volume is marginally negative." That is internally consistent and calibrates well on the test set. The model is effectively saying: "on-target count carries the quality signal, and total shot volume after controlling for on-target counts is just noise / pressure shots that don't threaten."

The alternative parameterization would be box_shots_on_target / out_box_shots_on_target / box_shots_off_target / ..., which reduces the collinearity. We did not do that in v1 because the calibration is already good and the five-feature fit is interpretable enough. A v1.1 refactor could orthogonalize the shot buckets cleanly.

Alpha selection

sklearn.linear_model.PoissonRegressor with L2 (alpha) sweep over [0.0, 0.001, 0.01, 0.1, 1.0], picked by validation (2024) Poisson deviance. Chosen alpha: 0.1 — mild regularization that pulls the coefficients slightly toward zero without erasing them.

Frozen training

The model is trained once, exported as JSON coefficients, and applied to every match by the nightly compute step. The daily sync does not retrain. Retraining is an explicit, version-bumped operation followed by a commit, so historical SQI values don't silently shift underneath the user.

Validation

Headline numbers (model_v1, alpha=0.1)

The baseline is a constant: predict every team-match as the training set mean (1.273 goals per team per match). SQI crushes it on every metric: test deviance 0.7833 vs 1.2879, a 39% improvement. MAE drops from 1.02 to 0.76 goals per match.

The mean predicted vs mean actual gap on test (1.393 vs 1.504) is honest drift: 2025 CPL had slightly more goals per match than the train-era average, and a model fit on 2019-2023 does not know about that. Season-total drift is -26 goals across 234 team-match rows. Not worrying but worth reporting.

Walk-forward backtest

Same pattern as the match outcome model: for each test year in 2021..2025, train a fresh Poisson on every team-match strictly before the test year, pick alpha by in-fold leave-last-season-out, score on the test year, compare to the train-mean baseline.

Pooled (n=1166):

Every single year beats the baseline, and the margin is never smaller than 0.28 deviance. This is a real signal, not a statistical fluke of a single test season.

Coefficient sanity

As discussed under "Choices" — positive coefficients for big_chances, penalty_goals, shots_on_goal, small negative coefficients for raw box_shots and out_box_shots driven by collinearity with on-target. big_chances has a larger positive coefficient than box_shots, which is the right ordering (a "big chance" flag should be worth more than a generic box shot).

Total-goal calibration per season (test set)

Small underprediction driven by secular scoring-rate drift (2025 CPL scored at a higher per-shot rate than the train average).

Failure modes

• Not xG. The single biggest failure mode is conceptual: SQI cannot distinguish a chance-creation masterclass from speculative volume with the same bucket counts. Loud disclaimers live in the UI and the page header.
• Own goals in the target. A tiny (< 3%) noise source that slightly inflates actual goals relative to what the shot features can explain. Simpler than filtering them out and documented here.
• Collinearity between shots_on_goal and total shot buckets. The negative fitted coefficients on box/out-box shots are an artifact of that, and the interpretation is awkward even though the calibration is fine. A v1.1 that orthogonalizes into on-target vs off-target splits would clean this up.
• Secular scoring drift. CPL scoring rate is trending up slightly over time; a model frozen on 2019-2023 under-predicts 2025 totals by ~7% across the league. The per-team Δ column is still interpretable because the drift affects all teams, but aggregate season totals will drift low until the next retrain.
• Penalty_goals is a feature, not a per-shot rate. The fit learns a multiplicative effect, not "one goal per penalty". A purer formulation would be "scored vs missed penalties" as a two-bucket split, but the SDP data only exposes the scored count.
• Big-chance subjectivity. SDP's big-chances flag is hand-set by the data provider. We inherit their definition. If their rules change across seasons, our coefficient on big_chances is partially fitting to the taggers rather than reality.
• Small sample. 964 training rows, 234 test rows. Large enough for a 5-feature Poisson to be stable, nowhere near big enough for anything more exotic.
• In-match event data missing. We don't know which minute shots came in, who was on the pitch, or what the score was at the time. No way to model "chasing a deficit" shot quality vs "sitting on a lead".

What would make it better

1. A shot-level data source. The one upgrade that would make SQI into a real xG model. Requires a data partner beyond the current SDP feed. Out of reach for this project today.
2. Orthogonal bucket features. Refactor to on_target_box, off_target_box, on_target_out, off_target_out so coefficients don't drift negative via collinearity. Cleaner story, similar predictive power.
3. Per-team finishing efficiency as a side metric. We currently output a single "goals vs SQI" Δ per team-season. Splitting that into "finishing above/below SQI on big chances" vs "on ordinary shots" would expose whether a team is converting above the league rate everywhere or just getting hot in one bucket.
4. Retest as a match-outcome feature. The match-outcome v1.1 walk-forward showed raw rolling shot totals do not beat the Elo baseline. A calibrated SQI rolled forward is a different transformation of the same inputs; worth a separate backtest with sqi_diff_5 added to the feature set to see if the model-based feature moves the needle where raw rolling did not.
5. Own goals subtracted from target. Small effect but cleaner. A one-line loader change on the next retrain.

Live drift

The secular-drift failure mode above is watched live: the section rendered below this document shows season-to-date drift, defined as actual goals minus SQI-predicted goals (positive = the frozen model under-predicting, the documented direction). An informational flag trips when |drift| exceeds ±10% of predicted goals with at least 50 matches played — that threshold sits above the known ~8% drift on purpose, so it fires only if drift gets worse than what is already documented here. The model stays frozen either way; the flag never affects the sync. Served by /api/model-diagnostics.