Everyone who makes comic dramas remembers the moment: two days of rendering, first playback in the editor — the character’s mouth has closed, and the voiceover is still talking. Weak visuals earn you an “AI-flavored” comment; desynced audio costs you the viewer in 10 seconds, and when the algorithm sees the bounce rate, your recommendation volume is halved. This part bundles the pipeline’s last two stages: motion (image-to-video) and voice (TTS). They share one article because they share one core problem — the timeline. Desync isn’t bad luck; it’s uncomputed arithmetic. And the arithmetic has formulas.
Where desync comes from: three layers of root cause
Locate the problem first; the fixes need somewhere to land. Audio-visual desync has three layers of causes:
- TTS pace vs the visual window. A line takes 2.8 seconds to speak; the character’s mouth-open window on screen is 2.1 seconds — the extra 0.7 seconds is the desync.
- Video models output no audio track. Most image-to-video models produce pure visuals; the character’s mouth flaps at random — whatever track you paste underneath, it “pretends” to say. (Exceptions are emerging: Kling 2.6+, Vidu Q3, and Seedance 2.0 now generate audio-video natively — more below.)
- Cumulative error across the edit. A 3-minute episode splices 30–50 clips; 100 ms of drift per clip is invisible, but 40 clips compound to 3–4 seconds — the back half of the episode is unwatchable. Cumulative drift is the silent killer, and it chooses the middle of your episode to surface.
The routing debate: voice-first
Once you see desync as “voice duration ≠ visual window,” the routing question is clear: whose timeline adapts to whom?
- Voice-first: lock the script → generate the voiceover → cut visuals to the audio waveform’s length. Sync is designed in, not rescued afterward. Radio-drama logic — the correct answer for dialogue-heavy episodes. Industry measurement: fixing audio first and scheduling shots to it compresses sync tuning from 2–3 hours to roughly 30 minutes.
- Picture-first: generate visuals → dub afterward → run lip-sync post-processing on close-ups. Fits action episodes with sparse dialogue — but sync is rescued, and every rescued second has a price.
- Rescue tooling also forks into two technical routes: audio-driven lip (Wav2Lip, 17k GitHub stars, the mature first choice) and lip-driven audio (timeline dubbing, the digital-human route — higher ceiling, higher barrier).
Field rule: long line, short window — never cram. Split the line, add a reaction shot, or nudge the speed; all three cost less than rework.
Motion: the four camera levers of image-to-video
Turning Part 4’s unbroken stills into shots, controllability comes from four levers:
First/last frames pin the start and end poses exactly (Kling’s first/last-frame control is the skeleton of narrative shots); camera words drive the movement — push, pull, pan, track, orbit — in the formula “subject + action + camera move + duration + style/light,” with Part 4’s identity sentence still riding verbatim on every prompt; motion amplitude tunes to content: low for facial close-ups (protects expressions), high for action; 2–4 seconds per shot (~20 shots a minute) — long takes are drift incubators.
A trend worth its own paragraph: native audio-video models are eating the lip-sync stage. Kling 2.6+ outputs Chinese/English dialogue with synced lip movement, Vidu Q3 does 16-second clips with native sound, and Seedance 2.0’s A/V sync has been called “a generational leap” in reviews. The correct posture for dialogue shots is becoming “generate a talking shot directly,” with post-hoc lip-sync demoted to patch duty — weight this capability higher when choosing tools.
Voice: emotion is TTS’s last weakness; desync gets the ladder
Part 2 solved TTS selection; this part solves TTS usage. First, the honest limit: current TTS emotion control is externally label-driven — it doesn’t understand subtext. It will read “oh, you’re so great” with genuine admiration even when the context is sarcasm, and above 1.2× speed, emotional expression decays audibly. Three countermeasures: annotate emotion per line (fine-grained instructions like “eyes reddening, holding back tears” go into the storyboard); pin one voice per character and record the voice ID (keep the voice table next to the Character Bible); swap in human dubbing or voice cloning (Moyin) surgically for the critical emotional lines.
Now the engineering of desync — the delta-tiered ladder. Measure δ = (voice duration − visual window) ÷ visual window, then pick a rung:
- δ ≤ 15%: adjust TTS speed. Safe range 0.8×–1.25×; beyond that, quality degrades audibly.
- 15% < δ < 35%: LLM constrained rewrite — keep the meaning, keep the emotional tone, give a target length range (a range, not an exact number), add nothing new; re-estimate the duration to verify.
- δ ≥ 35%: hybrid — rewrite + speed + trim inter-line silence (cut pauses by at most 50%; beyond that it sounds breathless).
Two lip-sync sweet-spot numbers to memorize: lip-sync tools track best at 220–240 characters per minute — past 280, mouths start lagging; a 3–5% audio stretch fixes offset inaudibly while visibly improving lip match.
Assembly and acceptance: the 30-minute six-step workflow
Bolt every part into a process, and syncing a 3-minute episode compresses to 30 minutes:
- Export every clip’s exact duration (millisecond precision);
- Estimate each line’s TTS duration; compute the deviation rate;
- Process by ladder tier; output the voiceover;
- Align the subtitle timeline to the audio waveform;
- Run lip-sync post-processing on every facial close-up;
- Full playback review — focus on the middle (where cumulative drift surfaces), kill the BGM and watch mouths alone, checking that sentence starts, stresses, and pauses land precisely.
One last acceptance rule: local rework beats wholesale regeneration. Redoing the few unnatural clips is always cheaper than scrapping the episode — true on the timeline and the budget axis alike.
Action items, then Part 6
- Measure the route difference: take one 3-second line and produce a 5-second shot each way — voice-first and picture-first — and compare calibration time. The data will settle the routing debate into intuition.
- Run the ladder once: deliberately write a line 30% too long for its window, walk the constrained-rewrite flow, and record semantic fidelity before and after — find the reliable range of your rewrite prompt.
- Build the voice table: pin one TTS voice for each protagonist and supporting role; record platform + voice ID + baseline speed; store it in the same document as Part 4’s Character Bible.
Part 6 leaves the studio and enters the battlefield: platforms and revenue share. Onboarding flows and review rules across the Douyin system, Tencent system, Bilibili, and Kuaishou, tested; how the revenue coefficient computes down to per-10k-plays; the matrix play of multi-platform distribution; and the portfolio strategy of how many baskets to hold your eggs in.