Sounding video generation: a brief introduction

发表时间

The ways to generate sounding-video

Trends: From multi-stage to end2end.

IO Pattern Approach Input Output Official demo Comment
Multi-stage
 Audio->Video
Most with lip-sync		 LatentSync
 Video/Image (Extend to video)
Audio		 Video
 Link
 Lip-edit in video
Pixel 512*512 only
Hallo2 Image
Audio
[opt.] Text: emotion control (calm, angry)		 Video Link
 Extremely long video generation
Not good performance
HuMo
 Text
[opt.] Audio
[opt.] Image		 Video Link
LongCat-Video-Avatar
 Text
Audio
[opt.] Image		 Video
 Link Long video generation
SoTA
Video->Audio
Only for ambient, no lip-sync		 MMAudio Video
[opt.] Text		 Audio Link
EchoFoley Video
[opt.] Text		 Audio Link
End2End Text -> Audio-Video
 UniVerse-1
 Image (As first frame)
Text		 Sounding video
 Link A valuable try
No pretraining
OVi
 Text
[opt.] Image		 Sounding video
 Link SoTA
JoVA
Closed-source		 Text
[opt.] Image		 Sounding video
 Link SoTA?

Preliminaries

Video: From raw to latent (Wan2.1)

  • Components: 3D-VAE, umT5, DiT.

  • Video dataflow

    • Raw video file: (3, T, H, W)

    • 3D-VAE: (3, T, H, W) -> (16, T/4, H/8, W/8)

image\.png

    - Upsample channel: \(3\-\>128, T, H, W\)

    - Upsample channel, downsample Temporal\+Spatial: \(128\-\>256, T\-\>T/2, H\-\>H/2, W\-\>W/2\)

    - Upsample channel, downsample Temporal\+Spatial: \(256\-\>512, T/2\-\>T/4, H/2\-\>H/4, W/2\-\>W/4\)

    - Downsample Spatial: \(512, T/4, H/4 \-\> H/8, W/4 \-\> W/8\)

    - Compress: \(32, T/4, H/8, W/8\)

    - Deterministic sample: Chunk on channel dim, use half vector as mu

- Patchify: Kernel size \(1,2,2\) \-\> \(16, T/4, H/8/2, W/8/2\)

    - Each patch contains 64 float numbers\. Use a MLP to map patch into real tensor\.

    > nn\.Conv3d\(in\_channels=16, out\_channels=D, kernel\_size=\(1,2,2\), stride=\(1,2,2\)\)
    > 
    > 

- Final shape: \(B, L, D\), where L=T/4 \* H/8/2 \* W/8/2

Compress ability: VAE 256x, Patchify 4x = 1024x

Transformer block:

image\.png

image\.png

Audio: From raw to latent (AceStep)

Learning from scratch

What is a sound?

The audio signal changes over time: air pressure. To capture this information digitally, we can sample air pressure over time.

image\.png

Sound freq: 1 / Period

image\.png

Most sounds in life do not follow a simple \& regular periodic pattern. Signals of different frequencies can be added together to form a composite signal.

image\.png

The rate can vary, but the most common is 44.1KHz(CD). Others can be 8KHz(phone), 11025Hz(AM radio), 22050(FM radio), 96KHz(DVD), etc.

How to represent a sound using a spectrum?

From time-axis to frequency-axis: the spectrum, another way to represent audio signal.

image\.png

image\.png

However, spectrum still needs a time-axis, because the voice changes from time to time, the spectrum can only represent a very short time.

How to generate spectrogram from signal?

  • Fourier theorem: Any complex periodic signal can be decomposed into a superposition of a series of sine waves (or complex exponential functions) with different frequencies, amplitudes, and phases.

  • Fast Fourier Transform: analyse signal components, split signal into different frequencies.

image\.png

image\.png

What is Mel spectrogram?

Research shows that humans cannot perceive frequencies within a linear range. We are better at detecting low-frequency differences than high-frequency differences. For example, we can easily distinguish the difference between 500 Hz and 1000 Hz, but even at the same distance, we have difficulty distinguishing the difference between 10,000 Hz and 10,500 Hz.

In 1937, Stevens, Volkmann, and Newmann proposed a unit of pitch so that equal pitch intervals sounded equal to the listener. This is called the mel scale. We perform mathematical operations on frequencies to convert them to the mel scale.

Audio dataflow

  • Raw audio: [2, sample_rate * seconds, eg. 16384*1]

  • STFT(Hop length=512)

    • [2, 1024, 32=16384/512]

  • scale to Mel Spectrogram: [2, 128, 32]

  • Audio VAE: [channel=8, 128/8, 32/8]

  • Patchify

Conditional on cross attention layer

  • Pre-train-capable: in one cross attention block

  • Post-train-only: insertion of another attention block

image\.png

image\.png

image\.png

image\.png

However, Waver is different.

image\.png

Multi-condition example:

image\.png

Train object: Flow Matching

Given a noise sample $x_0\sim p_0$ and a data sample $x_1\sim p_1$, a simple linear interpolation path is defined for time $t$:

$\begin{equation}
x_t=(1-t)x_0+tx_1
\end{equation}$

The target velocity vector along this path is constant: $u_t=x_1-x_0$. The model $v_{\theta}(x_t,t,c)$conditioned on $c$, is trained to predict this vector by minimizing the following L2 loss:

$\begin{equation}
L_{\text{FM}}=\mathbb{E}_{t\sim U(0,1),x_0\sim p_0,x_1\sim p_1}[||v_{\theta}((1-t)x_0+tx_1,t,c)-(x_1-x_0)||^2]
\end{equation}$

This objective directly trains the model to learn the vector field that maps noise to data, which leads to more stable and efficient training compared to traditional score-matching objectives.

End2End

Challenges: alignment

  • Time Alignment

    • Physical Distortion: The sound of a glass breaking does not match the visual transient.

    • Temporal Drift: The speaker’s lips close, but the voice continues (Lip-sync failure).

  • Semantic Alignment: The visual shows a heavy truck, but the sound is a light sedan.

Important aspects

  • Data curation

  • Fusion

    • Audio-video temporal alignment

    • In-model interaction

Joint Audio-Video Generation

Multi-modal CFG

image\.png

UniVerse-1

https://arxiv\.org/abs/2509\.06155

Main idea: Stitching of Experts

Data **Curation**

image\.png

image\.png

Data example:

image\.png

image\.png

Model architecture

  • Video model: Wan2.1-1.3B (30 layers)

  • Audio model: AceStep-3.5B (24 layers)

image\.png

image\.png

Training

$L=L_{\text{FM-video}}+L_{\text{FM-mel}}+\lambda_{\text{SSL}}\cdot L_{\text{SSL}}$

  1. Base model pre-training: flow matching loss

  2. Semantic alignment loss: $L_{\text{SSL}}=\frac{1}{2}(\cos\text{Sim}(h’_{audio}, h’_{MERT})+\cos\text{Sim}(h’_{audio},h’_{mHuBERT}))$

    1. For AudioSet \& VGGSound, employ a conditional loss scheme.

image\.png

Key points

  1. Audio-video Temporal alignment: 25fps video with 25.6KHz audio (Vary from the chosen of different base models)

  2. Layer interpolation: Strategically insert new blocks at uniform intervals into the shallower of the two models until depths align.

    1. The parameters for each new block are initialized by linearly interpolating the weights of its immediately adjacent (bracketing) layers. (Benefit from residual connection?)

  3. Independent noise sampling strategy

OVi [Character.AI]

https://arxiv\.org/abs/2510\.01284

Main idea: Stitching of Experts, but using Twin-backbone

Data Curation

Two corpus:

  • Audio-video corpus for learning modality alignment

  • Audio-only corpus for acoustic pretraining and fine-tuning: 12s \& 5s

Preprocess:

image\.png

image\.png

Data example:

image\.png

image\.png

Model architecture

Twin backbone from Wan2.2-5B. The video branch is initialized from Wan2.2 5B, and an identical audio branch is trained from scratch.

image\.png

fusion.py

Training

  1. Audio model training

    1. Audio dataflow: raw -> mel -> MMAudio VAE

    2. Loss: flow matching

    3. Stages

      1. Audio pretraining: 100k hours audio up to 12s.

      2. Audio fine-tuning: 5.04s audio

  2. Audio-video model training

    1. Initialization: load pretrained params, freeze all FFNs, initializing cross-modal attention from scratch.

    2. Loss: flow matching. $L_{total}=\lambda_vL_{FM}^v+\lambda_aL_{FM}^a, \quad \lambda_v=0.85,\;\lambda_a=0.15$

Key Points

  1. Audio-Video temporal alignment: Apply RoPE scale to the audio branch by 31/157=0.197

image\.png

  1. Two-stage training: Pretraining audio model, then fuse with video model by post-training.

JoVA [ByteDance]

https://arxiv\.org/abs/2512\.13677

Main idea: No additional fusion

Data Curation

  • Audio 2.4k hours + Audio-Video 1.3k hours + Speech-Video 0.7k hours

截屏2026\-01\-04 11\.33\.47\.png

image\.png

Model Structure

image\.png

Structure of Waver, helps to understand why JoVA uses “Joint Self-Attention”

image\.png

Structure of MMAudio, helps to understand what exactly “Joint-Attention” is

image\.png

Key points

  1. The proposal of the so-called “Join Self-Attention” actually aims to inherit the training methodology of Waver,

  2. Audio-Video temporal alignment: Apply RoPE scale to the audio branch by 31/157=0.197

  3. Two-stage training: Pretraining audio model, then fuse with video model by post-training.

  4. Mouth-aware supervision: Boost mouth region’s loss (need to align with video VAE latent space first).

image\.png

[TBD] Multi-stage

Audio-driven Video Generation

LongCat-Video-Avatar

Important inspiration: sometimes multi-task training settings serve the original task.

  1. HuMo supports generation with no audio

  2. LongCat supports video continuation task

Model structure

image\.png

  1. Initialize from a pretrained video diffusion model

  2. Insert audio cross-attention

    1. Directly adding this layer causes training instability and prevents the model from effectively aligning audio signals with corresponding mouth movements -> introduce an Adaptive Layer Normalization (adaLN) module before each audio cross-attention layer.

  3. Audio projector: to temporally compress the audio embeddings.

Training objective: $\epsilon=\epsilon_\theta(z_t,t,c_{\text{text}}, c_{\text{audio}})$

Key points

  • Disentangled unconditional guidance: model cannot distinguish between unconditional input and silent audio

image\.png

  • Reference skip attention

  • Cross-chunk Latent Stitching

Video-driven Audio Generation

MMAudio

Model architecture

image\.png