Methods

This page documents the protocol exactly as implemented in the repository (suspense_atlas/). Every number on the site is produced by python -m suspense_atlas.pipeline all, scripts/run_llm.py, scripts/analyze.py and python -m suspense_atlas.export, in that order, from the corpus manifest alone.

Corpus

Source and licence. All 92 texts are United States public domain and were fetched from Project Gutenberg by ebook number (corpus/manifest.yaml). Boilerplate is stripped at the *** START/END OF THE PROJECT GUTENBERG EBOOK markers; single stories are sliced out of collections with regular-expression markers on their headings (the manifest records every marker, and start_occurrence: -1 skips a table of contents). Texts are normalised (typographic quotes, Gutenberg's -- dash, _italic_ markers, hard line wraps) while preserving paragraph breaks; a SHA-256 of each clean text is stored in data/corpus_metadata.json. Nothing is served that is not already free to redistribute, and every story page links its source ebook.

Selection. Stories were chosen by hand to cover seven genre labels — horror, ghost, detective, adventure, speculative, literary, comic — across 1820–1928, with a length range from 1.2k to 43k words (three novellas are included deliberately so that the length axis is populated). Genre is a single hand-assigned dominant mode per story; it is a coarse, contestable label and is treated as such in the analyses (see Limitations).

Segmentation

Six schemes are computed for every story; all report character offsets into the clean text so the website can highlight exactly the span a curve point measures:

Narrative position is the word-midpoint of a segment divided by the story's word count (0→1). Any per-segment series can be re-binned onto a 40-bin grid (segment.rebin), which is how curves from different schemes are compared.

Features

Extracted per segment (suspense_atlas/features/):

• Lexical — sentence length (mean, SD, share of short sentences), word length, type–token ratio and MATTR-50, punctuation rates per 1,000 characters (!, ?, dashes, ellipses, semicolons, commas), negation rate, temporal-term rate, first/second-person pronoun rates, tense hints.
• Semantic (lexicon) — rates per 100 tokens for threat, uncertainty, action, conflict, perception, six basic emotions, and an arousal index (high-arousal minus low-arousal words). The lexicons are small, hand-curated seed lists committed with the code (features/lexicons.py), so every proxy score is traceable to a visible word list. If the user places NRC-VAD under data/lexicons/, VAD valence/arousal/dominance are added; the pipeline does not redistribute that resource.
• Narrative (spaCy en_core_web_sm) — dialogue ratio (quoted characters ÷ characters), entity density, distinct persons, character introductions (PERSON names not seen earlier in the story), location changes (GPE/LOC/FAC entities not present in the previous segment), finite-verb rate, motion- and speech-verb rates, subject shifts between consecutive sentences, POS rates.
• Model-based — 384-d MiniLM (all-MiniLM-L6-v2) embeddings; GPT-2 surprisal of the segment given up to 256 tokens of preceding story text (mean, SD, max, p90, in bits). Surprisal is the information-theoretic "surprise" operationalization; its limits are discussed below.

Operationalizing suspense

Suspense is latent. The pipeline computes several independent operationalizations and compares them instead of naming one the ground truth:

1. Human ratings — the browser study at /annotate (design below). At the time of writing no pooled human ratings exist; the site says so wherever a human number would otherwise appear.
2. LLM ratings — a 1–7 suspense rating, a 1–7 intensity rating, a 0–1 confidence and an eight-word cue per segment, obtained as structured JSON from a self-hosted Qwen3-8B endpoint through a provider-neutral adapter (llm/adapters.py; OpenAI-compatible, Anthropic SDK, Claude-CLI and mock adapters share one interface and one content-hashed disk cache). The main condition is: plain prompt, temperature 0, no position information.
3. Linguistic proxies — within-story z-scores of threat, uncertainty (+ question-mark rate), lexical arousal, action (+ motion verbs), negative emotion; a composite proxy is their mean.
4. Narrative uncertainty — the uncertainty proxy above (epistemic vocabulary and questions).
5. Event intensity — the action and threat proxies, and the LLM intensity rating.
6. Affect dynamics — the arousal proxy (and VAD arousal when available).
7. Information-theoretic surprise — contextual GPT-2 surprisal.

All measures are compared within stories (mean per-story Spearman ρ), pooled after within-story z-scoring, and at story level (ρ of story means).

LLM annotation validation

Robustness conditions are run on a fixed stratified subset (two stories per genre — the shortest and the median-length story — so all conditions share the same 14 stories): a defined prompt carrying a literary-theoretic definition (Brewer & Lichtenstein; Lehne & Koelsch), a contextual prompt with the tail of the preceding text as a recap, temperature 0.8 with two independent samples, a true position prompt ("this passage occurs about 72% of the way through the story"), and a shuffled position prompt in which the stated position is drawn from another segment of the same story. Position bias is estimated as the slope of (condition rating − main rating) on the stated position, and, in the shuffled condition, as the correlation of that difference with (stated − true position): a text-reading rater should show none. Model sensitivity is measured by repeating the main condition with a second model on the same subset.

Human annotation study

Items are 213 bins40 segments of 80–260 words, stratified by genre × position tercile, at most four per story. A session shows two calibration anchors (a manifestly calm and a manifestly tense passage, with suggested ratings), then 24 randomised items with two attention checks (whose instruction text specifies the expected answer) and one repeated item for within-session test–retest. The randomisation is a deterministic function of an anonymous rater id, so a session can be reconstructed. For every response we store rating, intensity, confidence, reading time (ms), item id, story position, order in session and a hash of the excerpt; nothing identifying. Raters pass quality control if both attention checks are correct, the repeat is within two points and the median reading time is ≥4 s. Reliability is reported as ICC(2,1); agreement with LLM and proxy measures as Spearman ρ over item means. Analysis of human ratings will use mixed-effects models with rater and story random effects (statsmodels MixedLM) once there are enough raters.

Curves and their comparison

A story's curve is its segment series re-binned to 40 equal-progress bins, z-scored within the story (so height and register differences are removed and only shape remains) and Gaussian-smoothed (σ = 1 bin). Group means (genre, era, length band, author with ≥3 stories) carry 95% bootstrap intervals over stories (2,000 resamples). Group differences are tested with (a) a permutation test on the between-group share of curve variance (2,000 label shuffles) and (b) Kruskal–Wallis tests on shape descriptors (peak position, late−early level, number of peaks, final level, trend slope). Curve shapes are clustered with k-means (k = 2…6); each silhouette is compared with the silhouette obtained on within-story shuffled curves, and no "archetype" is claimed unless it clears that null's 95th percentile.

Prediction

Targets: the LLM suspense rating (primary) and the composite proxy (robustness). Models: global-mean baseline, TF-IDF bag-of-words ridge, ridge on hand features, histogram gradient boosting on hand features, ridge on MiniLM embeddings, and boosting on features + embeddings. Cross-validation is 5-fold grouped by story (a test asserts that no story appears in both train and test) and, as a stricter generalisation test, grouped by author. Metrics: R², RMSE, pooled Spearman ρ with a bootstrap CI, and mean within-story Spearman ρ (shape recovery). Interpretability: permutation importance on held-out folds, mean |SHAP| (TreeExplainer), standardized ridge coefficients, and a per-family ablation. For RQ2 a story-level dataset averages features over the first 10/20/30/50/70% of each story and predicts the mean, maximum and rise of the remaining rating, against an "early level only" reference model.

Generative-AI experiment

Twenty stories (~1,400 words) were generated by the same Qwen3-8B endpoint at temperature 0.9 under five trajectory instructions (low, rising, early peak, late climax, repeated peaks) × four fixed premises, then processed by the unmodified pipeline. Agreement with the requested shape is the Spearman ρ between the measured 40-bin curve and an idealised target curve (for "low", the raw within-story SD is reported instead). Generated stories are stored under corpus/generated/ and never enter the human-corpus analyses.

Reproducibility

corpus/manifest.yaml → python -m suspense_atlas.pipeline all (ingest, segment, features) → scripts/run_llm.py (cached LLM ratings) → scripts/run_generate.py + the pipeline with --corpus generated → scripts/analyze.py → python -m suspense_atlas.export → web/ (Next.js). Seeds are fixed (SEED = 20260905); LLM calls are cached by content hash so re-runs never re-bill and never change; model weights are pinned by name. pytest covers ingestion, segmentation, metadata, curve normalisation, split leakage, annotation planning/QC and feature extraction.

Known limitations of the measures

• Lexicons are small, era-sensitive and sense-ambiguous ("still", "dead calm"); they measure vocabulary, not experience.
• GPT-2 surprisal mixes narrative surprise with rare names, archaic diction and dialect; it is reported, not trusted.
• LLM ratings are produced by one 8B open-weight model; the prior literature (Matlin et al., 2025) finds that language models distinguish suspenseful from non-suspenseful text but track within-text dynamics poorly. Our sensitivity conditions quantify prompt, temperature and position effects; they cannot substitute for human data.
• Genre is a single hand label, confounded with author and era in a 92-story corpus.