We tested open-source lip-sync: MuseTalk vs EchoMimic for talking AI hosts

What we were trying to make

The test case was a short coffee-shop conversation between two invented hosts, Maya and Leo. Each host is a single still portrait. The voice track is generated first, then a lip-sync model repaints the face so the mouth follows the audio. The finished video cuts between the two hosts as they talk, with captions on top.

That rules out a lot of models straight away. Many talking-head systems need a big NVIDIA card, a cloud API, or a reference video of a real person. We wanted something that starts from one photo, runs on a Mac with Apple silicon, and has a licence that lets us ship it inside a paid app. MuseTalk (MIT licence) and EchoMimic V3 Flash both qualified, so they went head to head.

Everything below ran on one machine: an M1 Pro MacBook. No cloud GPU was used for this test.

How each model works

MuseTalk repaints only the lower half of the face. Whisper turns the speech into audio features, and MuseTalk redraws the mouth and jaw region frame by frame to match them. The rest of the face, the eyes and the head position stay exactly as they were in the photo. In TallyReel we find the face with YuNet, a 230 KB face detector that runs on ONNX Runtime, and line MuseTalk's crop up with its box and keypoints.

EchoMimic V3 Flash generates the whole frame. It is a video diffusion model driven by audio, so besides the mouth it can nod, tilt and shift the head. That is the reason we tested it: a host whose head never moves can look stiff over a long video. EchoMimic does not ship with Apple silicon support, so we ported it to run on the Mac's GPU through PyTorch's MPS backend.

The same line, side by side

Here is one line from Leo, rendered by each model from the same portrait and the same audio. Play them with sound on and watch the lips on the hard consonants.

EchoMimic's head motion is real and it is the one thing it does better. The mouth shapes, the teeth and the skin around the lips were weaker, and the face drifted away from the original portrait more than we liked. MuseTalk's version keeps the photo intact and only changes what needs to change. Our verdict after watching both several times was that MuseTalk was much better, and that one advantage of EchoMimic did not make up for the rest.

Render time on an M1 Pro

Speed matters as much as quality for us, because people render on their own computers and a host video can be several minutes long.

The last column is the render time divided by the length of the video. The two runs are not the same size of job: MuseTalk's time covers the whole finished video, EchoMimic's covers one short low-resolution clip, which makes the gap if anything larger than it looks.

The face you start from matters more than we expected

The most useful finding of the day was not about the models at all. Our first built-in host portraits were smiling with their teeth showing. A lip-sync model has to close that grin to say words like "map" or "bob", and a wide fixed smile fights it on every frame.

We drew fresh hosts with relaxed, closed mouths and ran the same line again through MuseTalk. The difference was obvious: the mouth opens and closes from a neutral position, so every sound has somewhere natural to go. A host drawn with a calm face and the mouth closed will lip-sync better than one with a big smile.

A full two-host conversation

This is the whole 43 second coffee talk made with MuseTalk, with a natural cloud voice for each host and the captions TallyReel adds. It is the output the 7.5 minute figure above refers to.

Voices turned out to be part of the lip-sync test too. With our fast local voice, the hosts sounded flat and the whole clip felt worse even where the mouths were right. With a more natural voice, the same lip-sync reads as a conversation. A good mouth on a robotic voice still looks fake.

What we did not pick, and why

• LTX-2.5 audio-to-video: we tried it on 1 October 2026 for talking animals. The mouth and head moved, but the result looked bad to us, and 3 seconds at 384x640 took 19 minutes on the same Mac. We removed it.
• MuseTalk on animals: it does find animal faces through YuNet, but it barely moved a camel's mouth and smeared a monkey's. MuseTalk is for human faces.
• LongCat-Video-Avatar 1.5: the strongest open model we found in our research, with an MIT licence, but it needs a large NVIDIA GPU. We have not tested it, so we make no claims about its quality.
• Paid avatar APIs: Kling Avatar 2.0, tried with free web credits, looked good. Its API price works out to about $0.056 per second, which is about $34 for a 10 minute podcast. That is far outside what a faceless video should cost, so we stayed with a local model.

What we changed in TallyReel

MuseTalk 1.5 stays as the lip-sync model for hosts. Instead of chasing another model, we put the work into what sits around it: hosts with relaxed closed mouths, more natural voices, and small idle motions like breathing, blinking and the occasional nod, which add life without the cost of a full video model.

If a future open model beats MuseTalk on a Mac at a similar speed, we will run the same test again and publish the result here.

Models and links

• MuseTalk on GitHub (TMElyralab)
• MuseTalk weights on Hugging Face
• EchoMimic V3 on GitHub (Ant Group)
• YuNet face detector (OpenCV model zoo)