Method & prediction ledger

# World Cup Squad Quality vs Performance — design notes

Base spec: squad-quality metrics frozen at kickoff for every WC squad 2002–2026,
model expected performance, compare expected vs actual, spotlight Australia.
Sources: Wikipedia squads/results, Transfermarkt (per-game JSON + market-value
history). See the original brief for cutoffs, top-5 competition codes, and
data-quality rules.

## Amendment: design v2

Predictor budget for the ordinal model is UNCHANGED. v2 items are either
replacements for an existing predictor, upgrades to the outcome variable, or
descriptive/diagnostic modules that never enter the regression RHS.

### Model changes
- **pedigree_adj** [math-only, REPLACEMENT]: top5_minutes weighted by an age
  curve peaking at 27 (age at cutoff). Replaces the raw appearance predictor
  (top5_apps). Benchmarked head-to-head against raw in the leave-one-tournament-
  out fit; whichever predicts better is the one kept — not both.
- **exit_elo** [OUTCOME UPGRADE]: Elo rating of the team that eliminated you.
  A luck-adjusted outcome variant sitting on the LHS alongside stage score /
  outlast_pct. Requires a national-team Elo series (source TBD: eloratings.net
  is the canonical historical source; Wikipedia has no clean series).

### Descriptive / diagnostic modules (not model predictors)
- **cohort_share** [math-only]: pedigree share held by the modal 4-year birth
  window of the squad.
- **succession_ratio** [math-only]: U25 pedigree minutes ÷ 30+ pedigree minutes.
- **dependency_index** [math-only]: top player's share of total squad value.
- **club_pairs** [parse cached pages, descriptive]: count of same-club teammate
  pairs at cutoff (club-at-cutoff from roster + TM).
- **league tiles** [parse-only, dashboard]: domestic-league share, league
  diversity index, retirement-league value share, by year.

### Outputs
- **live_2026** [output only]: model scores for surviving teams, refreshed per
  round through the 19 July 2026 final.

## Amendment: design v3 (descriptive / outcome / packaging only — no new predictors)

### Descriptive / diagnostic modules
- **diaspora_share** [parse cached TM profile headers]: % of squad born outside
  the federation's territory; dual-citizenship count. CAVEAT: birthplace is a
  noisy proxy for eligibility federation (foreign-born to national parents,
  naturalised players) — report as birthplace-based, not eligibility-based.
- **window_conversion** [math only]: per federation, peak expected-quality
  tournament vs peak actual result → "wasted generations" leaderboard.
- **pathway_speed** [math on cached career tables]: mean age at first top-5
  league minute, by federation.
- **pound_for_pound** [CSV download, no scraping]: over/under per capita and per
  GDP using tournament-year World Bank population + GDP values.

### Model change
- **value_momentum** [math on cached value histories, REPLACEMENT]: % of squad
  with rising market value over the 12 months pre-cutoff. Spec says test it as a
  replacement for `rust_index` — but rust_index is not defined anywhere in v1/v2
  (OPEN: see note below). Predictor budget stays fixed either way.

### Live upgrade (replaces v2 live_2026)
- Monte Carlo over the remaining bracket: Elo-driven, 10k sims, confidence
  intervals per surviving team. Manual bookmaker-odds column (user-supplied) →
  model-vs-market panel.

### Packaging
- Per-federation pages templated from the Australia spotlight; publish cleaned
  CSVs + a schema doc.

## Data-collection impact (v1 + v2 + v3)
- exit_elo → eloratings.net (team-level, ~240 rows, separable). CONFIRMED.
- pound_for_pound → World Bank population + GDP CSVs, one-time bulk download,
  keyed on country + tournament year. New but not scraping.
- diaspora_share → birthplace + citizenship parsed from TM profile pages, which
  are already cached in full; re-parseable without re-fetching. Profile parser
  currently extracts only DOB; extend it to capture birthplace/citizenship at
  build time (no re-scrape).
- Everything else is math on already-cached per-game JSON, value histories, and
  roster tables. No change to the overnight player scrape.

## Amendment: Champions League metric (2026-07-12)

Add cl_apps / cl_goals / cl_minutes as a SEPARATE descriptive metric — NOT folded
into top5 (which stays big-5 league-only for model comparability) and NOT a model
predictor (budget fixed six-wide; club prestige already enters via value_share).
- Source: TM competition code 'CL' (Champions League proper). Excludes 'CLQ'
  (qualifying) — qualifiers are played by weaker clubs, not the elite-exposure
  signal we want.
- Cutoff-aware, same freeze dates as everything else.
- Derived from the per-game JSON already cached (zero re-scrape) via
  scripts/enrich_cl.py, which ALTERs the columns and backfills from cache.
- RUN enrich_cl.py as a POST-SCRAPE step over the complete worldcup.db. Do not
  run it against the live DB mid-scrape (SQLite single-writer lock); tested on a
  copy. Rationale for league-only core metric: cup apps measure team progression
  + club prestige, confounding individual quality — see the cups discussion.

## Amendment: overperformance + prospect signals (modeling phase, zero scrape impact)

All cache-only derivations. NONE enter the fixed six-wide regression. Feasibility
verified: height is on cached TM profiles (itemprop="height", e.g. "1,84 m" —
note EU decimal comma); caps is on cached Wikipedia rosters (cell[4]).

Signals:
- elo_value_gap : Elo rank − value_share rank within tournament. [Elo (external,
  already planned) + value_share (have)]
- gk_quality    : MV of the top-valued GK; gk_share = that / squad value.
  [position + MV, both stored]
- height        : parse cached TM profiles; squad mean + defender mean (set-piece
  proxy). [re-parse cached profiles]
- prospect_share: squad MV share held by U23 players at cutoff. [DOB + MV]
- precocity     : per player, top-5 minutes before age 21. [per-game dates + DOB]
- caps_core     : share of squad caps held by the 5 most-capped players.
  [re-parse caps from cached rosters]

Placement (none join the six-wide RHS):
- (a) residual profiling POST-fit, labeled exploratory.
- (b) prospect_share and value_momentum are testable ONLY in the single
  challenger slot (vs rust_index) — never as a 7th predictor.
- (c) live 2026 Monte Carlo MAY carry gk_quality + height as variance adjusters.
- Single-match upsets = variance. Monte Carlo reports probabilities; build NO
  upset classifier.

Skip (explicitly out of scope):
- youth-international caps (needs per-player fetch)
- style / xG data (walled sources)

## Post-scrape enrichment queue (run in order, after scrape_all completes)

All cache-only (no re-scrape) EXCEPT build_perclub's ~300 one-time club-page
fetches. Run only after the main scrape finishes so they don't compete with its
throttled rate limit.

1. `enrich_cl.py`      — add cl_apps/cl_goals/cl_minutes (Champions League).
2. `build_perclub.py`  — resolve ~300 clubId->name (TM club-page h1), build
   `clubs` + `player_club_top5` (per-club/per-league apps/goals/assists/minutes,
   frozen at each cutoff). Join player_club_top5.club_id = clubs.club_id for
   readable names. Verified extraction: 985→Man Utd, 46→Inter, 583→PSG.

## Gate outcome (Wikipedia cross-check, 2026-07-12)

OUTCOME: **BAR FAIL** — 1/15 random players off by >1 league app = 6.7% > 5% bar.
Routed to launch via the fail-protocol PARSE-DELTA branch (not a pass): the sole
failure is a parse/definition delta, and zero wrong-player matches were found
(all 18 matched exact name+DOB; 14/15 randoms + 3/3 anchors agree with TM
exactly; TM independently confirmed against anchors).

- The failing delta: Gabriel Milito, La Liga TM=104 vs Wikipedia=102 (2 apps).
  PARKED as resolve-from-cache (see resolve_from_cache.md) — cross-check
  artifact, re-examine from cached pages post-scrape, not a launch blocker.
- Gaps (excluded per missing≠mismatch): Ryan (no parseable Wiki table),
  Makélélé Ligue 1 (Wikipedia labels pre-2002 French top flight "Division 1",
  unmapped — naming delta).
- First run showed 27.8% + 3 false wrong-player suspects; root cause was a
  rowspan bug in the Wiki parser (dropped continuation-season rows with a
  row-spanned competition cell), since fixed (expand_grid). Not a TM issue.

BAR CALIBRATION DEFECT (recalibrate before any future reuse): a 5% bar on n=15
permits ZERO misses — floor(0.05×15)=0, so 1/15=6.7% fails by construction. The
bar cannot express the intended "small tolerance." Fix before reuse: state it as
a count (e.g. ≤1 of 15) or size the sample to n≥20 so one miss can pass at 5%.

## Amendment: minutes data (restored) — resolves the model spec

- **top5_minutes** [have it]: career big-five first-division league minutes.
  PRIMARY pedigree metric; top5_apps demoted to display only (collinear — never
  both on the RHS). v2's age-curve (peak 27) applies to MINUTES, not apps.
- **season_minutes** [DERIVE FROM CACHE — see finding below]: league minutes in
  the season underway / most recently completed at cutoff, any club, any league.
- **2022 mid-season wrinkle**: cutoff 2022-11-20 falls mid-season; scale
  season_minutes by (full-season rounds ÷ rounds elapsed) via a per-league round
  lookup; set estimated=True flag.
- **rust_index** = share of squad with < 900 scaled league minutes at cutoff.
  The single form predictor.
- **value_momentum** = % of squad with rising MV over the 12 months pre-cutoff.
  rust_index's CHALLENGER: fit both, keep whichever predicts better, never both.

### Predictor budget — FIXED SIX-WIDE
1. elo_start (eloratings.net, rating at cutoff)          [NEW — needs Elo]
2. log(value_share)                                       [have]
3. pedigree minutes (top5_minutes, raw or age-adjusted; not both)  [have]
4. group_difficulty (mean opponent strength in group draw)  [NEW — draw + Elo]
5. host dummy                                             [have]
6. rust_index OR value_momentum (challenger test)         [derive from cache]

Note vs v1: G+A dropped from the model (display only); elo_start and
group_difficulty added; pedigree is minutes not apps.

## KEY FINDING: season_minutes needs ZERO extra fetches
Transfermarkt migrated player stats to a Svelte UI. Every stats view (season,
detailed, per-competition) is a CLIENT-SIDE re-aggregation of a single JSON
endpoint, /ceapi/performance-game/{id}, which we already fetch once per player.
Verified:
- Per-game JSON carries season league minutes for EVERY league class tested:
  top-5, 2.Bundesliga, Championship, Scottish, Danish, A-League, MLS, J-League,
  Cypriot.
- The plain season Leistungsdaten page embeds the same performance-game proxy
  and NO other endpoint — fetching it returns an HTML shell with no stats.
- Therefore the ~6,000 season-page fetches are REDUNDANT and are DROPPED from
  the plan. They could not even fix pre-2006 minute sparsity (season pages
  render from the same data), which is a TM data property → quality_flag it.
- club_pairs (v2) likewise derives from per-game clubId, not season pages.

## STANDING RULE — no unverified specifics (added 2026-07-14 after a hallucination)

A specific match, stat, ranking, or scoreline may be stated ONLY when the query
that produces it and its printed output appear in the same block. "The data shows
X" without an adjacent query is invalid — retract and re-run. Model prose is
untrusted until echoed by a query.

Site exposure: every editorial string containing a number, name, or scoreline
must be COMPUTED from the DB at generation time, never typed. Verified computed:
duel-card numbers, def-champion tags, the under-performer headline (count + names),
over/under hero context. Audited clean: all 25 shootout advancers vs known winners;
every knockout match vs history; team_results↔matches name join (fixed China PR /
Republic of Ireland / Czech Republic); ages floored at cutoff (Messi 2026 = 38).

## Amendment: league presence panel (v1.2 — supersedes both prior league options)

DO NOT build in the current pass; next fresh session builds from this doc.

- Data layer: per-squad apps + minutes in EVERY league, from the per-game cache.
  Complete, zero curation.
- Display: pinned editorial subset by default (Eredivisie, Primeira Liga, Belgian
  Pro League, Brasileirão, Argentine Primera, J-League, MLS, Championship),
  labeled "Selected pathway leagues — full list below", with a show-all expander
  sorted by apps. On squad pages + compare.
- Label rule: "selected", never "top". Descriptive only — no model.
- Build note: share the parse pass with cl_assists — one cache pass, two features.
  Confirmed queued elsewhere: height / prospect_share / precocity → "Squad traits";
  faded_star + MV sparkline → "Concentration".