Comprehensive Speech Prosody Modeling: A Task-Agnostic Approach

Hybrid duration decoder. The duration head interpolates a regression decoder and a flow-matching decoder with weights λ_r and λ_f (λ_r + λ_f = 1). At the selected operating point (λ_r = 0.1, λ_f = 0.9) the model attains significantly lower consonant and vowel KS than the regression-only ablation on both corpora (1.6–2.4× lower, p < 0.05, Benjamini–Hochberg corrected) and is indistinguishable from the flow-only ablation, i.e., the hybrid keeps the distributional fidelity of the flow decoder while retaining a deterministic component.

Distributional match. Point-wise metrics such as MAE favor mean-regressing predictions and cannot detect whether a model preserves the spread and heavy tails of natural prosody, so the two-sample KS statistic is reported over the full empirical CDF. FluentSpeech and StyleTTS2 fail to capture the variance of these distributions. The proposed model produces distributions closer to the ground truth and achieves relative KS improvements of 61% and 68% for vowel and consonant durations, respectively, and 58% for F0, over the strongest baseline in each case.

In the downstream TTS setup, the proposed model yields 18.5% higher AutoPCP than the best baseline (and better SECS), indicating that it transfers prosody better. WER is not improved over the other TTS approaches, which may be because those models are trained with full spectrogram-based losses, whereas the proposed approach models only prosody during pre-training and may omit spectral cues needed for low WERs. The proposed model nevertheless has the advantage of being task-agnostic and is not fine-tuned specifically for TTS.

Two ABX surveys were run on Prolific, one on the GigaSpeech test set and one on the VoxPopuli-EN test set, each with 40 native English listeners. Listeners judged whether the proposed method or a baseline was more prosodically similar to the ground truth. The proposed model is significantly preferred over FluentSpeech and StyleTTS2 on both test sets. Against FluentSpeech-Up, which is trained on 10× the data with 3× the parameters, the proposed model is significantly preferred on VoxPopuli, while on GigaSpeech the preference rate does not differ significantly from chance. The proposed model therefore surpasses the baselines trained under comparable conditions and remains competitive with a substantially larger system.

On speaker- and text-disjoint emotion recognition (arousal, valence, dominance) on IEMOCAP, the proposed model—never trained on emotional speech or emotion labels—reaches 0.595 mean CCC, above both the raw prosody signals (0.501) and eGeMAPS (0.518), significantly on all three dimensions (BH-adjusted p ≤ 0.02).

Fusing the proposed prosody embeddings with a self-supervised backbone increases mean CCC for every backbone, and more than raw prosody or eGeMAPS features do. The gains are significant on valence for all seven backbones and on dominance for six, but on arousal for only two (e.g. +0.021/+0.020 on valence/dominance for WavLM-base against +0.007 on arousal), which suggests that arousal is largely already captured by SSLs unlike valence and dominance. Gains over task-supervised backbones are smaller on average but still significant, and give the two highest mean CCC values overall.

For 2-class word prominence, the frozen latent embeddings of the proposed model with a single learned linear layer reach 0.856 accuracy, above the raw prosody features that the model itself takes as input (0.843) and above BERT-base (0.802); both margins are significant under a paired speaker-clustered bootstrap (BH-adjusted p < 0.001). This indicates that the learned representation encodes more than the raw prosody values can capture, and that these labels are only partially recoverable from lexical content.

Late fusion improves every backbone, and the best system overall is a fusion (0.881 accuracy for Whisper-large-v3 + Proposed). The improvement is largest for the weakest backbone (wav2vec2-base, +0.021) and decreases to +0.005–0.009 for stronger backbones. As in emotion recognition, this indicates that SSL backbones—even with task supervision and more scaling—can still lack prosodic information that is useful for prosody-related downstream tasks.

Perceiver-style encoders represent variable-length inputs using a fixed set of learnable latents (queries). Because these latents are not inherently tied to specific time indices, their cross-attention to text and acoustic prosody features may be weakly structured or temporally inconsistent. To encourage temporally coherent alignments, Soft Alignment Loss (SAL) and summary rotary positional embeddings (S-ROPE) are applied to the model's cross-attention layers, including those involving the fixed-length prosody latents.

Temporal alignment is visualized using heatmaps of cross-attention weights for a sample from GigaSpeech-Dev. As shown in Figs. 3–6, the full model (SAL + S-ROPE) exhibits a more pronounced diagonal structure, consistent with improved temporal alignment between (i) text and prosody latents and (ii) acoustic prosody features and prosody latents.

F0/Energy Decoder

Fig. 3 shows the cross-attention between the length-regulated (LR) text sequence (with speaker embedding) and the fixed-length prosody embeddings. In this example, the full model displays a clearer diagonal trend than the ablations. Removing SAL produces a larger qualitative degradation than removing S-ROPE, which may be because SAL directly regularizes cross-attention toward a diagonal prior, whereas S-ROPE provides a positional inductive bias. Fig. 4 shows a similar trend for cross-attention between the acoustic prosody input and the fixed-length prosody embeddings in the F0/Energy decoder.

Duration Decoder

Fig. 5 shows cross-attention between the text (with speaker embedding) and the fixed-length prosody embeddings in the duration decoder. The full model again shows more structured attention. Interestingly, multiple diagonal bands are observed, which suggests that alignment remains broadly monotonic but is split into several locally monotonic regions (groups of prosody latents attend to successive text spans with small offsets) rather than forming one continuous diagonal as in the F0/Energy Decoder.

Prosody Encoder

Fig. 6 shows encoder cross-attention between the learnable prosody queries and the acoustic prosody input. In this example, the second half of the utterance/prosody input is masked; correspondingly, the diagonal trend is primarily visible in the unmasked region (most clearly in the log-scaled, contrast-enhanced visualization).

Correspondence Between Latent Index and Predicted F0 Frame

The cross-attention maps show where the model attends to; a masking probe additionally tests which part of the predicted F0 each prosody latent controls. One contiguous group of latents is masked at a time in the F0/Energy Decoder cross-attention (18 groups of about 14 latents, each covering about 330 ms in a six-second sentence), with text, speaker embedding, and ground-truth durations held fixed. The fixed-length prosody embeddings achieve ρ = 0.96 (ordering) and κ = 0.46 (localization), a clear diagonalization. Without SAL the ordering survives (ρ = 0.89) but the locality spreads across the utterance (κ = 0.21); without S-ROPE the ordering itself breaks down (ρ = 0.61). The two mechanisms are therefore complementary: S-ROPE ties each latent index to an output frame position, whereas SAL concentrates the influence of that latent around this position.