# H4 reliability-stratified effect audit

Verification Status: COMPUTED_PENDING_INDEPENDENT_VERIFICATION

Reliability cut points were written before model scores were loaded. The target is the inferential unit. Guide-pair analyses remain exploratory because only 133 targets are non-missing.

## Primary reliability strata

| stratum | family | metric | n_unique_targets | mean_effect | bootstrap_ci95_low | bootstrap_ci95_high | holm_p_within_stratifier_family | stratum_support |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| lower | rmse | rmse_vs_zero | 720 | 0.0138599 | 0.0128951 | 0.0148073 | 1 | NOT_SUPPORTED |
| lower | rmse | rmse_vs_train_mean | 720 | -0.000800775 | -0.00132091 | -0.000303152 | 0.00419992 | SUPPORTED |
| lower | directional | raw_cosine_vs_train_mean | 720 | 0.00185008 | 3.96916e-05 | 0.00371761 | 0.102158 | NOT_SUPPORTED |
| lower | directional | residual_cosine_vs_zero | 720 | -0.029374 | -0.0595083 | 0.000751818 | 0.97678 | NOT_SUPPORTED |
| middle | rmse | rmse_vs_zero | 720 | -0.00380138 | -0.0052965 | -0.00231494 | 0.000119998 | SUPPORTED |
| middle | rmse | rmse_vs_train_mean | 720 | 0.000249336 | -4.96708e-05 | 0.000551784 | 1 | NOT_SUPPORTED |
| middle | directional | raw_cosine_vs_train_mean | 720 | 0.00192349 | 0.000248755 | 0.00352704 | 0.0565989 | NOT_SUPPORTED |
| middle | directional | residual_cosine_vs_zero | 720 | 0.00035054 | -0.0222993 | 0.0232478 | 0.97678 | NOT_SUPPORTED |
| upper | rmse | rmse_vs_zero | 720 | -0.0289996 | -0.0303706 | -0.0276152 | 0.000119998 | SUPPORTED |
| upper | rmse | rmse_vs_train_mean | 720 | -0.000888766 | -0.0012323 | -0.000535208 | 0.000119998 | SUPPORTED |
| upper | directional | raw_cosine_vs_train_mean | 720 | 0.00113611 | -0.00150032 | 0.0035913 | 0.594408 | NOT_SUPPORTED |
| upper | directional | residual_cosine_vs_zero | 720 | 0.165351 | 0.139061 | 0.191363 | 0.000119998 | SUPPORTED |

## Reliability moderation gates

| stratifier | stratifier_role | lower_tertile_rmse_vs_train_mean | upper_tertile_rmse_vs_train_mean | upper_minus_lower_effect | favorable_negative_trend | heterogeneity_interpretation |
| --- | --- | --- | --- | --- | --- | --- |
| exploratory_reliability_weight | primary | -0.000800775 | -0.000888766 | -8.79909e-05 | NOT_DETECTED | no_adjusted_linear_reliability_moderation_detected |
| split_half_cosine_median | secondary | -0.000747045 | -0.0010725 | -0.00032545 | NOT_DETECTED | no_adjusted_linear_reliability_moderation_detected |
| batch_split_cosine | secondary | -0.00076383 | -0.00104699 | -0.000283159 | NOT_DETECTED | no_adjusted_linear_reliability_moderation_detected |
| guide_pair_cosine | exploratory_guide_pair_n133 | -0.00192945 | 0.00141007 | 0.00333953 | NOT_DETECTED | no_adjusted_linear_reliability_moderation_detected |