Seedance Can't Clone Your Face: The Real Blog-to-Short-Video Pipeline

Want short videos with your real face and voice? Seedance and Veo cannot do it — you need a lip-sync engine. Full 6-stage pipeline: Supabase → ElevenLabs → HeyGen → Remotion, ~$0.40 per video, fully automated.

If you want short videos with your real face and your real voice, do not use Seedance, Veo, or Runway. They cannot do it. Those models generate footage from a prompt — they will produce someone who resembles you, with mismatched lip movement, looking different on every render. For a personal brand, that is unusable. What you need is a talking-head lip-sync engine driven by a cloned voice.

Here is the full backend pipeline I would build to turn any published blog post into a 35-second vertical video, automatically, with zero human touch after publish.

Why can't Seedance or Veo make a talking-head video of me?

Because they solve a different problem. Seedance and Veo are generative video models: text or image goes in, novel footage comes out. They have no mechanism to preserve a specific human identity frame-to-frame, and no mechanism to align mouth shapes to a specific audio track.

Lip-sync avatar engines work the opposite way: you give them a reference recording of the real you plus an audio file, and they animate your actual face to that audio. Identity is preserved because it was never generated in the first place.

So the "motion video" layer in a generic AI-video pipeline gets swapped:

• Generic pipeline: script → TTS → Seedance/Veo → render
• Personal-brand pipeline: script → voice clone → avatar lip-sync → render

Everything else stays the same.

What does the pipeline look like end to end?

Six stages, triggered by a single database event:

[1] Supabase posts table
     |  trigger: status -> published
     v
[2] Hook Extractor      -> pull lead paragraph, append CTA
     v
[3] ElevenLabs TTS      -> your cloned voice
     |                     /with-timestamps -> audio.mp3 + alignment JSON
     v
[4] Avatar Lip-sync     -> HeyGen API (audio input, NOT their TTS)
     |                     -> avatar.mp4 (async, poll or webhook)
     v
[5] Remotion            -> 9:16 frame, word-level captions, logo, CTA endcard
     v
[6] Render + Upload     -> Supabase Storage / social publish

The critical detail is the handoff from stage 3 to stage 4. ElevenLabs generates the audio first; HeyGen only lip-syncs to that audio. If you let HeyGen speak the text with its own built-in TTS, you lose your real voice — which was the entire point.

Which tool for which stage?

What should the script actually say?

Take a real example — my Wizy article on using Reddit for AEO. Its lead paragraph is 75 words, roughly 28 seconds spoken. Add a fixed CTA and you land at ~35 seconds: exactly right for Shorts, Reels, and TikTok.

Have you noticed something lately? Customers don't just Google anymore. Many now ask ChatGPT or Gemini directly — "What's a good nail salon in Mississauga?" — and the AI answers on the spot, naming a few shops.

So the million-dollar question: how do you get your shop onto that list?

The short answer: get your shop's name mentioned on Reddit.

But the how is where it gets specific — I broke the whole playbook down in the article. Link's below.

The bolded closing line is a fixed template, reused across every video. Only the first three blocks come from the article.

Auto-extraction rule: take the first paragraph of content_en, strip the Markdown, cut at the last complete sentence before 130 words, then append the CTA template. If that first paragraph is under 40 words, pull in the second one too.

This is deliberately dumb logic, and that is the point. A good hook paragraph is already a good video script — you wrote it to stop a scroller. Do not send it through an LLM to be "improved"; you will get generic AI voice and lose the thing that made it work.

Show me the code

Stage 3 — ElevenLabs with timestamps:

const res = await fetch(
  `https://api.elevenlabs.io/v1/text-to-speech/${VOICE_ID}/with-timestamps`,
  {
    method: 'POST',
    headers: { 'xi-api-key': KEY, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      text: script,
      model_id: 'eleven_multilingual_v2',
      voice_settings: { stability: 0.5, similarity_boost: 0.85 },
    }),
  }
);
const { audio_base64, normalized_alignment } = await res.json();
// normalized_alignment -> build word-level captions for Remotion

Stage 4 — HeyGen receiving audio, not text:

const job = await fetch('https://api.heygen.com/v2/video/generate', {
  method: 'POST',
  headers: { 'X-Api-Key': HEYGEN_KEY, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    video_inputs: [{
      character: { type: 'avatar', avatar_id: MY_AVATAR_ID },
      voice: { type: 'audio', audio_url: publicAudioUrl },  // <- ElevenLabs voice
    }],
    dimension: { width: 1080, height: 1920 },
  }),
}).then(r => r.json());
// then poll /v1/video_status.get until status === 'completed'

Stage 5 — Remotion composition:

export const ShortVideo = ({ avatarUrl, captions, articleTitle }) => (
  <AbsoluteFill style={{ background: '#0B0B0F' }}>
    <OffthreadVideo src={avatarUrl} />
    <CaptionTrack captions={captions} />          {/* word-by-word highlight */}
    <BrandFrame title={articleTitle} />           {/* logo + article title */}
    <Sequence from={durationInFrames - 90}>
      <EndCard cta="Read the full playbook ->" url="wizy.ca/blog/..." />
    </Sequence>
  </AbsoluteFill>
);

How do you trigger it automatically?

The simplest setup that needs no new infrastructure:

1. Add a video_jobs table in Supabase: post_id, lang, status, audio_url, avatar_url, final_url.
2. A Supabase Database Webhook fires when posts.status flips to published, inserting two jobs — one Vietnamese, one English.
3. A GitHub Actions cron runs every 5 minutes, scans for pending jobs, and advances each one by a single state per run. Because stage 4 is async, the worker never blocks — it just checks whether HeyGen finished and moves on.

Every published article now produces two short videos with no manual step.

What does it cost per video?

For a ~35 second clip:

These are ballpark figures at common pricing tiers — verify current rates yourself, especially HeyGen, which is credit-based and changes often.

In what order should you build it?

1. Week 1 — do it manually, once. Clone your voice on ElevenLabs, record 3 minutes of footage for a HeyGen Instant Avatar, and hand-assemble a single video. The goal is one question: is the avatar and voice good enough that you would put it on your own profile?
2. Week 2 — script stages 2 to 4. Hook extractor, ElevenLabs, HeyGen. Output a raw MP4.
3. Week 3 — Remotion. Captions, brand frame, endcard.
4. Week 4 — automate. video_jobs table, webhook, cron.

Do not reorder this. The biggest risk in the whole project is not technical — it is that the avatar does not look enough like you to publish. That is a quality risk, and it is answerable in an afternoon. Answer it before you write a single line of pipeline code.

The mistake to avoid

Teams building this get seduced by the generation layer and spend weeks comparing video models. But in a talking-head pipeline, the model choice is nearly irrelevant — the avatar engine is doing all the work, and there are only two or three real options.

The leverage is in the boring parts: deterministic hook extraction, word-accurate captions from timestamps you already have, and a renderer that produces the identical video after a script edit. Get those right and you have a content engine. Get them wrong and you have an expensive random video generator.

Reference article used in the example: Reddit for AEO: How to Get Your Shop Recommended When Customers Ask ChatGPT & Google AI

FAQ

Can Seedance or Veo generate a video of my real face speaking? No. They are generative models with no identity-preservation or audio-driven lip-sync. Use a talking-head engine like HeyGen or D-ID instead.

How do I keep my real ElevenLabs voice when using HeyGen? Pass voice: { type: 'audio', audio_url: ... } in the HeyGen request instead of type: 'text'. HeyGen then only lip-syncs to your uploaded audio and never invokes its own TTS.

Do I need Whisper to generate subtitles? No. ElevenLabs' /with-timestamps endpoint returns character-level alignment alongside the audio, which is more accurate and one stage cheaper than re-transcribing.

Remotion or FFmpeg for the final render? Remotion if you want animated captions, a brand frame, and version-controlled video as React components. FFmpeg if you want raw speed with minimal dependencies and can live with static overlays.

How long should the video be? Aim for 30–40 seconds. A 75-word lead paragraph runs about 28 seconds; adding a fixed CTA line lands you around 35 — the sweet spot for Shorts, Reels, and TikTok.

#AIVideo #ElevenLabs #HeyGen #Remotion #Supabase #ContentAutomation #VoiceClone #ShortForm #AIAgents #DevTools

✍️ The Author: Do Ngoc Hoan Founder of CookConnects.ca & Wizy.ca. Bridging the gap between advanced algorithms and business execution. I write for technical founders looking to scale their impact with AI and robust engineering.

← Blog