Skip to content

Latest commit

 

History

History
275 lines (230 loc) · 18.8 KB

File metadata and controls

275 lines (230 loc) · 18.8 KB

Scoring Model Validation — Phase B

Generated by scripts/scoring_validation/. Dataset and methodology described in run_validation.py. Re-run after every meaningful scoring change.

1. Data scope

  • Sample size: 426,558 (stock, date) rows
  • Universe: 5,311 US stocks (≥400 daily bars)
  • Time window: 2024-08-19 ~ 2026-03-02 (81 weekly snapshots)
  • Sampling: first trading day of each ISO week
  • Forward horizons: 5 / 20 / 60 trading days (skipped final 60 days where no forward return is available)
  • Winsorization: forward returns clipped to 1%/99% percentiles per horizon to neutralise penny-stock split-adjustment errors (e.g. one 23,000× outlier on DXF 2024-09-16). IC is rank-based and already outlier-robust; clipping affects decile means only.

Data limitations (honest disclosure)

The stock_daily table covers only 2024-05-17 ~ 2026-05-28 (≈ 2 years), not the originally targeted 10 years. As a result:

  • Sample period spans one bull-leaning regime (post-2024 rally + early 2026 consolidation). Bear / sideways regimes are under-sampled.
  • 12-month return signals (ret_12m) only have valid values for stocks with full 12-month history at sample date — early dates drop these.
  • IC t-stats with n_dates < 30 are unreliable (single regime, high autocorrelation across weeks).

2. Models NOT validated (data gap)

Of the 18 quantcore scoring models, only the price-derived subset is validated below. The remaining 16 require external data not present at universe scale in our DB:

Model Required data DB rows available
Piotroski / Mohanram / Altman / Ohlson / Distress Full financial statements (BS + IS + CF) 43 in fundamental_snapshot
Valuation / Beneish / DuPont / EarningsQuality / MagicFormula / DividendSafety Finviz + OpenBB metrics 7 in finviz_forward_snapshot
Macro Fed rate / CPI / VIX / FearGreed not loaded
MarketRegime US index env (SPX / Nasdaq / DJI 20-day) partial (SPY only)
PeerRank Sector peer fundamentals not at scale
OptionsSentiment CBOE options chain not loaded
InsiderActivity Institutional ownership + insider trades not loaded
EarningsSurprise / ShortInterest Finviz EPS QoQ + short ratio 7 rows

Implication: the composite score's macro 25 + fundamental 20 + valuation 20 + distress 10 + sentiment 10 = 85 of 100 points is uncovered by this validation. Only the technical+momentum 15 bucket (and its underlying signals) is testable here.

3. Information Coefficient (IC)

IC = Spearman rank correlation between signal value and forward return, computed cross-sectionally per date and averaged over all dates. Higher absolute IC ⇒ stronger predictive power. t-stat = mean / (std / √n_dates); |t| ≥ 2 is the conventional significance bar for cross-sectional alpha factors.

Horizon: 5D

Signal IC mean IC std t-stat n_dates n_samples Verdict
vol_60 -0.0611 +0.1928 -2.85 81 426,558 moderate
golden_cross +0.0339 +0.1027 +2.38 52 426,558 moderate
ret_12m +0.0336 +0.1191 +1.81 41 213,912 weak
px_vs_ma200 +0.0304 +0.1190 +1.84 52 267,460 weak
heavy_up_10d -0.0243 +0.0532 -4.11 81 426,558 strong
momentum_blend +0.0242 +0.1084 +2.01 81 426,558 moderate
ret_6m +0.0218 +0.1229 +1.46 68 352,293 weak
ma50_slope +0.0197 +0.1097 +1.62 81 426,558 weak
macd_bar -0.0187 +0.0893 -1.88 81 426,558 weak
vol_ratio_5d +0.0134 +0.0373 +3.24 81 426,558 strong
rs_spy_20 -0.0127 +0.1096 -1.04 81 426,557 weak
sctr +0.0121 +0.1090 +0.80 52 267,460 noise
ret_1m -0.0110 +0.1094 -0.90 81 426,558 noise
ret_3m +0.0102 +0.1061 +0.87 81 421,410 noise
rsi_14 -0.0096 +0.0970 -0.90 81 426,428 noise
rs_spy_60 +0.0074 +0.1079 +0.62 81 426,556 noise
beta_60 -0.0061 +0.1922 -0.28 81 426,558 noise
sharpe_60 +0.0056 +0.1068 +0.47 81 426,558 noise
px_vs_ma50 -0.0029 +0.1122 -0.23 81 426,558 noise
heavy_down_10d +0.0000 +0.0407 +0.00 81 426,558 noise

Horizon: 20D

Signal IC mean IC std t-stat n_dates n_samples Verdict
vol_60 -0.0889 +0.1734 -4.62 81 426,558 strong
ret_12m +0.0551 +0.0930 +3.79 41 213,912 strong
golden_cross +0.0544 +0.0913 +4.30 52 426,558 strong
px_vs_ma200 +0.0533 +0.1094 +3.51 52 267,460 strong
ret_6m +0.0379 +0.0935 +3.34 68 352,293 strong
sctr +0.0328 +0.1124 +2.10 52 267,460 moderate
ma50_slope +0.0276 +0.1013 +2.45 81 426,558 moderate
momentum_blend +0.0261 +0.0953 +2.46 81 426,558 moderate
heavy_up_10d -0.0259 +0.0458 -5.09 81 426,558 very strong
macd_bar -0.0258 +0.0744 -3.12 81 426,558 strong
ret_3m +0.0222 +0.1010 +1.98 81 421,410 weak
rs_spy_60 +0.0206 +0.1037 +1.79 81 426,556 weak
vol_ratio_5d +0.0193 +0.0355 +4.88 81 426,558 strong
beta_60 -0.0184 +0.1659 -1.00 81 426,558 weak
px_vs_ma50 +0.0148 +0.1004 +1.33 81 426,558 weak
sharpe_60 +0.0138 +0.0964 +1.29 81 426,558 weak
ret_1m +0.0088 +0.0882 +0.89 81 426,558 noise
rs_spy_20 +0.0085 +0.0900 +0.85 81 426,557 noise
rsi_14 +0.0025 +0.0947 +0.24 81 426,428 noise
heavy_down_10d -0.0003 +0.0442 -0.06 81 426,558 noise

Horizon: 60D

Signal IC mean IC std t-stat n_dates n_samples Verdict
vol_60 -0.1237 +0.1819 -6.12 81 426,558 very strong
ret_12m +0.1081 +0.0679 +10.20 41 213,912 very strong
px_vs_ma200 +0.0892 +0.1164 +5.53 52 267,460 very strong
golden_cross +0.0861 +0.0927 +6.69 52 426,558 very strong
ret_6m +0.0788 +0.0869 +7.48 68 352,293 very strong
sctr +0.0686 +0.1140 +4.34 52 267,460 strong
ma50_slope +0.0512 +0.1064 +4.33 81 426,558 strong
momentum_blend +0.0506 +0.0975 +4.67 81 426,558 strong
ret_3m +0.0473 +0.1056 +4.04 81 421,410 strong
rs_spy_60 +0.0469 +0.1076 +3.92 81 426,556 strong
px_vs_ma50 +0.0409 +0.1051 +3.50 81 426,558 strong
sharpe_60 +0.0350 +0.0987 +3.19 81 426,558 strong
heavy_up_10d -0.0300 +0.0443 -6.10 81 426,558 very strong
rsi_14 +0.0274 +0.0907 +2.72 81 426,428 moderate
rs_spy_20 +0.0269 +0.0921 +2.63 81 426,557 moderate
beta_60 -0.0260 +0.1592 -1.47 81 426,558 weak
ret_1m +0.0256 +0.0917 +2.51 81 426,558 moderate
vol_ratio_5d +0.0211 +0.0345 +5.51 81 426,558 very strong
macd_bar -0.0192 +0.0662 -2.60 81 426,558 moderate
heavy_down_10d -0.0056 +0.0429 -1.17 81 426,558 weak

IC interpretation key

  • noise (|t| < 1): no signal
  • weak (1 ≤ |t| < 2): suggestive, not significant
  • moderate (2 ≤ |t| < 3): tradable in isolation
  • strong (3 ≤ |t| < 5): reliable factor
  • very strong (|t| ≥ 5): exceptional (suspect data leak — verify)

4. Decile spread (top-vs-bottom long-short)

At each date, sort stocks into deciles by signal value. D10 = top 10% (highest signal), D1 = bottom 10%. Spread (D10 − D1) is the raw return of an equal-weight long-short portfolio. Win% = fraction of (stock, date) pairs with positive forward return.

Horizon: 5D

Signal D1 mean D10 mean Spread (D10−D1) D10 win% D1 win%
ret_12m -0.37% +0.65% +1.02% +50.54% +44.24%
vol_60 +0.11% -0.70% -0.81% +42.55% +55.58%
px_vs_ma200 -0.13% +0.61% +0.73% +51.06% +44.95%
momentum_blend -0.39% +0.23% +0.62% +48.82% +44.00%
ret_6m -0.38% +0.22% +0.60% +48.84% +44.03%
ret_1m +0.30% -0.22% -0.52% +46.60% +47.20%
px_vs_ma50 +0.25% -0.22% -0.47% +47.21% +46.58%
rs_spy_20 +0.24% -0.24% -0.47% +46.61% +46.87%
vol_ratio_5d -0.33% +0.09% +0.42% +48.76% +45.77%
macd_bar +0.04% -0.36% -0.40% +48.83% +50.74%
sctr +0.23% +0.56% +0.34% +50.88% +47.31%
ma50_slope -0.17% +0.11% +0.28% +48.54% +44.95%
rsi_14 +0.27% +0.02% -0.24% +49.82% +49.97%
beta_60 -0.12% +0.12% +0.24% +47.44% +48.73%
ret_3m -0.09% +0.01% +0.10% +48.13% +45.19%
sharpe_60 +0.16% +0.19% +0.03% +51.41% +48.89%
rs_spy_60 -0.02% -0.05% -0.02% +48.12% +45.50%
heavy_up_10d +0.27% +50.75%
heavy_down_10d +0.19% +50.05%
golden_cross +0.43% +51.89%

Horizon: 20D

Signal D1 mean D10 mean Spread (D10−D1) D10 win% D1 win%
ret_12m -2.11% +1.73% +3.84% +49.36% +40.60%
px_vs_ma200 -1.13% +2.40% +3.53% +51.08% +42.54%
ret_6m -2.12% +0.74% +2.86% +47.09% +40.57%
vol_60 +0.30% -1.86% -2.17% +39.65% +57.47%
sctr +0.03% +2.07% +2.04% +51.23% +45.81%
momentum_blend -1.29% +0.67% +1.96% +47.18% +41.95%
ma50_slope -0.86% +0.59% +1.45% +47.31% +42.91%
macd_bar -0.12% -1.47% -1.36% +46.81% +49.92%
ret_3m -0.74% +0.52% +1.26% +46.82% +43.16%
vol_ratio_5d -0.65% +0.60% +1.25% +48.47% +45.54%
rs_spy_60 -0.66% +0.48% +1.14% +46.88% +43.39%
beta_60 -0.15% +0.56% +0.71% +45.96% +48.75%
sharpe_60 +0.03% +0.73% +0.70% +51.13% +47.96%
px_vs_ma50 -0.31% +0.16% +0.46% +46.25% +44.20%
rs_spy_20 -0.16% +0.14% +0.31% +45.85% +44.38%
ret_1m -0.13% +0.05% +0.18% +45.55% +44.46%
rsi_14 +0.37% +0.36% -0.01% +50.22% +49.00%
heavy_up_10d +0.72% +50.56%
heavy_down_10d +0.55% +49.81%
golden_cross +1.13% +51.86%

Horizon: 60D

Signal D1 mean D10 mean Spread (D10−D1) D10 win% D1 win%
ret_12m -6.01% +5.57% +11.59% +50.04% +35.17%
px_vs_ma200 -2.05% +8.41% +10.47% +54.08% +39.11%
ret_6m -5.49% +3.57% +9.06% +46.71% +36.40%
sctr -0.10% +8.11% +8.21% +54.81% +42.42%
momentum_blend -2.58% +3.17% +5.75% +47.17% +38.67%
ret_3m -2.09% +3.34% +5.43% +46.95% +39.51%
ma50_slope -1.91% +3.40% +5.31% +47.00% +39.62%
rs_spy_60 -1.68% +3.41% +5.09% +47.08% +39.87%
px_vs_ma50 -1.40% +2.76% +4.17% +46.25% +40.38%
vol_60 +0.98% -2.30% -3.28% +36.63% +57.26%
beta_60 +0.15% +3.32% +3.17% +44.29% +49.01%
rs_spy_20 -0.53% +2.59% +3.11% +45.55% +41.36%
sharpe_60 -0.04% +2.83% +2.87% +51.75% +45.65%
ret_1m -0.48% +2.35% +2.82% +45.46% +41.40%
vol_ratio_5d -0.52% +1.85% +2.37% +48.10% +44.23%
macd_bar -0.32% -2.61% -2.29% +45.71% +48.40%
rsi_14 +0.52% +2.31% +1.78% +50.56% +46.33%
heavy_up_10d +2.58% +50.70%
heavy_down_10d +2.21% +49.76%
golden_cross +4.87% +55.09%

5. Cross-signal correlation matrix (collinearity)

Pearson correlation between signals on the same (stock, date) row. Pairs with |corr| ≥ 0.7 are flagged as redundant — they carry essentially the same information and combining them in a composite score is double-counting.

Highly collinear pairs (|corr| ≥ 0.7)

Signal A Signal B Pearson corr
ret_3m rs_spy_60 +0.979
momentum_blend ret_12m +0.969
beta_60 vol_60 +0.846
px_vs_ma50 px_vs_ma200 +0.734
sctr sharpe_60 +0.718
sctr rsi_14 +0.717

Full correlation matrix

sctr momentum ret_1m ret_3m ret_6m ret_12m px_vs_ma px_vs_ma rsi_14 macd_bar ma50_slo vol_rati heavy_up heavy_do golden_c beta_60 sharpe_6 vol_60 rs_spy_2 rs_spy_6
sctr +1.00 +0.02 +0.03 +0.05 +0.11 +0.02 +0.70 +0.68 +0.72 -0.00 +0.55 +0.01 +0.36 -0.39 +0.45 +0.01 +0.72 +0.00 +0.04 +0.05
momentum_blend +0.02 +1.00 -0.08 +0.26 +0.18 +0.97 +0.01 +0.05 +0.00 -0.00 +0.03 -0.00 +0.01 +0.00 +0.02 +0.16 +0.02 +0.14 -0.05 +0.26
ret_1m +0.03 -0.08 +1.00 +0.22 +0.26 +0.04 +0.25 +0.39 +0.03 +0.00 +0.33 +0.02 +0.01 -0.00 +0.01 +0.16 +0.02 +0.31 +0.63 +0.20
ret_3m +0.05 +0.26 +0.22 +1.00 +0.43 +0.08 +0.12 +0.21 +0.02 -0.00 +0.19 +0.01 +0.01 -0.00 +0.03 +0.61 +0.05 +0.60 +0.15 +0.98
ret_6m +0.11 +0.18 +0.26 +0.43 +1.00 +0.13 +0.15 +0.31 +0.03 -0.00 +0.23 +0.01 +0.02 -0.01 +0.09 +0.29 +0.07 +0.34 +0.14 +0.43
ret_12m +0.02 +0.97 +0.04 +0.08 +0.13 +1.00 +0.03 +0.07 +0.00 -0.00 +0.04 +0.00 +0.01 -0.00 +0.02 +0.06 +0.01 +0.07 +0.03 +0.08
px_vs_ma50 +0.70 +0.01 +0.25 +0.12 +0.15 +0.03 +1.00 +0.73 +0.53 -0.00 +0.65 +0.05 +0.23 -0.20 +0.16 +0.08 +0.52 +0.17 +0.32 +0.11
px_vs_ma200 +0.68 +0.05 +0.39 +0.21 +0.31 +0.07 +0.73 +1.00 +0.32 -0.01 +0.70 +0.02 +0.12 -0.13 +0.46 +0.16 +0.52 +0.27 +0.45 +0.20
rsi_14 +0.72 +0.00 +0.03 +0.02 +0.03 +0.00 +0.53 +0.32 +1.00 +0.01 +0.26 +0.01 +0.31 -0.33 +0.11 -0.01 +0.50 -0.02 +0.04 +0.02
macd_bar -0.00 -0.00 +0.00 -0.00 -0.00 -0.00 -0.00 -0.01 +0.01 +1.00 -0.03 +0.00 +0.01 -0.01 -0.01 +0.00 -0.01 +0.00 +0.00 -0.00
ma50_slope +0.55 +0.03 +0.33 +0.19 +0.23 +0.04 +0.65 +0.70 +0.26 -0.03 +1.00 -0.00 +0.07 -0.11 +0.27 +0.15 +0.57 +0.30 +0.16 +0.18
vol_ratio_5d +0.01 -0.00 +0.02 +0.01 +0.01 +0.00 +0.05 +0.02 +0.01 +0.00 -0.00 +1.00 +0.02 +0.01 -0.01 +0.00 +0.00 +0.01 +0.03 +0.00
heavy_up_10d +0.36 +0.01 +0.01 +0.01 +0.02 +0.01 +0.23 +0.12 +0.31 +0.01 +0.07 +0.02 +1.00 -0.10 +0.01 -0.00 +0.14 +0.02 +0.02 +0.01
heavy_down_10d -0.39 +0.00 -0.00 -0.00 -0.01 -0.00 -0.20 -0.13 -0.33 -0.01 -0.11 +0.01 -0.10 +1.00 -0.05 +0.00 -0.21 +0.01 -0.01 -0.00
golden_cross +0.45 +0.02 +0.01 +0.03 +0.09 +0.02 +0.16 +0.46 +0.11 -0.01 +0.27 -0.01 +0.01 -0.05 +1.00 +0.01 +0.43 -0.01 +0.01 +0.03
beta_60 +0.01 +0.16 +0.16 +0.61 +0.29 +0.06 +0.08 +0.16 -0.01 +0.00 +0.15 +0.00 -0.00 +0.00 +0.01 +1.00 +0.01 +0.85 +0.13 +0.62
sharpe_60 +0.72 +0.02 +0.02 +0.05 +0.07 +0.01 +0.52 +0.52 +0.50 -0.01 +0.57 +0.00 +0.14 -0.21 +0.43 +0.01 +1.00 +0.01 +0.03 +0.05
vol_60 +0.00 +0.14 +0.31 +0.60 +0.34 +0.07 +0.17 +0.27 -0.02 +0.00 +0.30 +0.01 +0.02 +0.01 -0.01 +0.85 +0.01 +1.00 +0.24 +0.60
rs_spy_20 +0.04 -0.05 +0.63 +0.15 +0.14 +0.03 +0.32 +0.45 +0.04 +0.00 +0.16 +0.03 +0.02 -0.01 +0.01 +0.13 +0.03 +0.24 +1.00 +0.13
rs_spy_60 +0.05 +0.26 +0.20 +0.98 +0.43 +0.08 +0.11 +0.20 +0.02 -0.00 +0.18 +0.00 +0.01 -0.00 +0.03 +0.62 +0.05 +0.60 +0.13 +1.00

6. Conclusions and follow-ups

Key findings (TL;DR)

  1. Strongest 20-day predictors: heavy_up_10d (t=-5.1), vol_ratio_5d (t=+4.9), vol_60 (t=-4.6), golden_cross (t=+4.3), ret_12m (t=+3.8). These are the price-derived signals worth keeping in the technical+momentum bucket.
  2. Best 60-day predictor: ret_12m (IC mean = +0.1081, t-stat = +10.20, n_dates = 41). Long-horizon momentum / trend signals dominate, as expected.
  3. Is SCTR worth its complexity? sctr (t=+2.1) does NOT beat raw px_vs_ma200 (t=+3.5) at 20-day. The composite weighting is suspect — consider simplifying.
  4. Multicollinearity present: 6 signal pairs above |corr| ≥ 0.7 (worst: ret_3m vs rs_spy_60 at +0.98). The current composite score double-counts these. Suggest dropping one of each redundant pair before re-weighting.
  5. Volatility is a negative predictor (vol_60 IC = -0.0889, t = -4.6 at 20d). High-vol stocks underperform low-vol stocks going forward — confirms low-vol anomaly. The risk model should penalise volatility, not just measure it.
  6. Signal yield: 8 of 20 signals achieve |t-stat| ≥ 3 at 20-day with ≥ 30 dates. The remaining 12 are statistically indistinguishable from noise in this 2-year sample and should not anchor real decisions.

What this validation tells us

Look at the IC and decile-spread tables for the 20-day horizon (the natural sweet spot for swing-trade signals). The questions to ask:

  1. Which signals have |t-stat| ≥ 2 at 20d? Those are the factors that actually drive the technical+momentum bucket. Anything weaker should be either down-weighted or removed from the composite score.
  2. Are the high-IC signals collinear? If ret_3m and ret_6m both score well but correlate at 0.85, blending them adds noise rather than information. Pick one.
  3. Does sctr outperform its components? A composite is only worth its complexity if it dominates each individual input on a risk-adjusted basis. If raw px_vs_ma200 beats sctr, the SCTR weighting scheme is suspect.

What this validation cannot tell us

The 16 unvalidated models above hold 85 of the 100 composite-score points. Their effectiveness is completely unknown. This is the project's largest open question. Proposed next steps:

  1. Backfill fundamental_snapshot for at least the top-500 by market cap × 8 quarters of history. This unlocks Piotroski / Altman / Ohlson / DuPont / EarningsQuality validation.
  2. Backfill finviz_forward_snapshot weekly via the existing tickbridge.refresh.tasks.finviz job. Pipe runs unblock valuation / magic formula / dividend / earnings-surprise / short-interest validation.
  3. Load macro time series (tb_macro_snapshot is 0 rows): Fed funds, 2Y, CPI, VIX, FearGreed at daily granularity. Macro and MarketRegime are 25 of the 100 points and currently free-wheeling.
  4. Re-run this script monthly. Add the new IC tables to a docs/scoring_validation_history/ directory so we can track factor decay over time.

To regenerate this report: python scripts/scoring_validation/run_validation.py && python scripts/scoring_validation/analyze.py.