Prediction · Suspense Atlas

Target: LLM suspense (Qwen3-8B)

Grouped by story (5-fold)

model	R²	RMSE	ρ (95% CI)	within-story ρ
global mean (baseline)	-0.001	1.278	-0.04 [-0.07, -0.00]	–
bag-of-words · ridge	0.422	0.971	0.68 [0.66, 0.70]	0.56
hand features · ridge	0.365	1.018	0.60 [0.57, 0.62]	0.50
hand features · gradient boosting	0.361	1.021	0.59 [0.57, 0.62]	0.47
MiniLM embedding · ridge	-0.211	1.405	-0.02 [-0.05, 0.02]	0.00
features + embedding · boosting	0.340	1.037	0.57 [0.55, 0.60]	0.45

Grouped by author (5-fold)

model	R²	RMSE	ρ (95% CI)	within-story ρ
global mean (baseline)	-0.025	1.293	-0.20 [-0.23, -0.17]	–
bag-of-words · ridge	0.402	0.987	0.67 [0.65, 0.69]	0.56
hand features · ridge	0.352	1.028	0.58 [0.56, 0.61]	0.49
hand features · gradient boosting	0.344	1.034	0.58 [0.55, 0.60]	0.47
MiniLM embedding · ridge	-0.278	1.444	-0.06 [-0.09, -0.03]	-0.00
features + embedding · boosting	0.308	1.062	0.55 [0.52, 0.57]	0.43

Target: Composite lexical proxy

Grouped by story (5-fold)

model	R²	RMSE	ρ (95% CI)	within-story ρ
global mean (baseline)	0.000	0.573	-0.00 [-0.03, 0.03]	–
bag-of-words · ridge	0.364	0.457	0.69 [0.67, 0.70]	0.73
hand features · ridge	0.734	0.295	0.88 [0.88, 0.89]	0.96
hand features · gradient boosting	0.782	0.267	0.89 [0.89, 0.90]	0.95
MiniLM embedding · ridge	-0.148	0.614	-0.02 [-0.05, 0.01]	-0.03
features + embedding · boosting	0.771	0.274	0.89 [0.88, 0.89]	0.95

Grouped by author (5-fold)

model	R²	RMSE	ρ (95% CI)	within-story ρ
global mean (baseline)	0.000	0.573	0.01 [-0.03, 0.04]	–
bag-of-words · ridge	0.358	0.459	0.68 [0.66, 0.70]	0.72
hand features · ridge	0.741	0.291	0.88 [0.88, 0.89]	0.96
hand features · gradient boosting	0.784	0.266	0.90 [0.89, 0.90]	0.95
MiniLM embedding · ridge	-0.174	0.621	-0.03 [-0.06, -0.00]	-0.03
features + embedding · boosting	0.771	0.274	0.89 [0.88, 0.89]	0.95

Which features matter to the model

Three views on the gradient-boosting model: permutation importance on held-out folds (drop in R² when a feature is shuffled), mean |SHAP|, and standardized ridge coefficients. Predictive importance is not causal influence on readers; a feature can matter because it marks a register the LLM rater responds to.

Permutation importance (held-out, grouped by story)

sem_threat0.088

sem_arousal_lex0.044

lex_ttr0.043

lex_mean_sentence_len0.017

sem_valence_lex0.015

lex_pronoun_i_rate0.015

lex_temporal_rate0.013

nar_verb_rate0.011

lex_exclamation_rate0.010

nar_finite_verb_rate0.008

lex_present_tense_hint0.008

sem_perception0.007

nar_adj_rate0.007

sem_action0.006

nar_subject_shift_rate0.006

nar_person_density0.005

lex_dash_rate0.005

lex_past_tense_hint0.005

lex_sd_sentence_len0.005

sem_emo_disgust0.004

Mean |SHAP|

sem_threat0.247

sem_arousal_lex0.142

lex_ttr0.131

sem_valence_lex0.100

lex_pronoun_i_rate0.092

sem_action0.076

lex_mean_sentence_len0.072

lex_temporal_rate0.070

lex_exclamation_rate0.067

nar_finite_verb_rate0.062

nar_verb_rate0.060

lex_present_tense_hint0.055

lex_dash_rate0.054

sem_perception0.053

nar_adj_rate0.051

nar_noun_rate0.050

nar_subject_shift_rate0.049

nar_location_density0.042

nar_person_introductions0.042

sem_emo_surprise0.041

Ridge coefficients (standardized)

lex_ttr-0.430

sem_threat0.310

sem_action0.157

nar_person_density-0.128

lex_pronoun_i_rate0.117

lex_mean_word_len0.109

nar_verb_rate0.109

nar_person_distinct0.107

nar_person_introductions-0.107

lex_mean_sentence_len-0.100

lex_mattr500.093

nar_finite_verb_rate-0.092

lex_dash_rate0.091

nar_adj_rate-0.086

lex_short_sentence_rate0.079

lex_temporal_rate0.078

sem_emo_joy-0.074

lex_long_word_rate-0.068

lex_exclamation_rate0.067

lex_question_rate0.063

Family ablation (boosting, grouped by story)

feature family alone	R²	ρ	within-story ρ
lexical	0.141	0.40	0.26
semantic	0.277	0.52	0.39
narrative	0.107	0.35	0.24
model	0.016	0.20	0.06

How early can future suspense be predicted?

Story-level: features (and the rating itself) averaged over the first X% of a story predict the mean and maximum rating of the remainder, and the rise (future mean − early mean). The “early level only” model uses just the early rating; beating it means the text carries information about what is coming beyond how tense it already is.

Mean of the remaining story

model	first 10%	first 20%	first 30%	first 50%	first 70%
early level_only	ρ 0.50	ρ 0.64	ρ 0.69	ρ 0.75	ρ 0.69
ridge	ρ 0.32	ρ 0.57	ρ 0.71	ρ 0.71	ρ 0.42
boosting	ρ 0.45	ρ 0.60	ρ 0.70	ρ 0.75	ρ 0.57

Maximum of the remaining story

model	first 10%	first 20%	first 30%	first 50%	first 70%
early level_only	ρ 0.35	ρ 0.50	ρ 0.54	ρ 0.61	ρ 0.56
ridge	ρ 0.15	ρ 0.32	ρ 0.39	ρ 0.51	ρ 0.22
boosting	ρ 0.20	ρ 0.18	ρ 0.38	ρ 0.47	ρ 0.37

Rise: future mean − early mean

model	first 10%	first 20%	first 30%	first 50%	first 70%
early level_only	ρ 0.39	ρ 0.31	ρ 0.28	ρ 0.24	ρ 0.10
ridge	ρ 0.28	ρ 0.33	ρ 0.41	ρ 0.24	ρ 0.11
boosting	ρ 0.32	ρ 0.36	ρ 0.34	ρ 0.21	ρ 0.29

n = 92 stories per cell; 5-fold grouped by story. With fewer than a hundred stories these estimates have wide sampling error; read patterns across cutoffs, not single cells.