Lip Syncing with Creature

Overview

This document describes the Lip Syncing functionality of Creature. This is a Pro version feature. Lip Syncing allows you to construct a set of animated mouth shapes via pre-drawn sprite frames and an input speech audio file of your choice. Lip syncing works in conjunction with the Sprite Frame Manager so be sure to read up on how to use the Sprite Frame Manager before proceeding with this document.

Step 1: Download Papagayo

Papagayo is a 3rd party open source tool that you will need in order to do any sort of Lip Syncing. This program allows you to import a .wav audio file, enter in some text that syncs up with the speech in your wav file, and then export it as a .dat Lip Sync file.

Papagayo: Download it here

Papagayo is open source and the full source code can be downloaded here

Step 2: Lip Sync Shapes

You will need to construct 10 Lip Sync Shapes representing the 10 different mouth shapes for any sort of speech pattern before you begin Lip Syncing. These 10 Lip Sync Shapes need to be saved into separate PNG images like the ones illustrated below:

For more reference examples, please refer to the Papagayo Source Code where additional mouth shape images are present in the mouths directories.

Step 3: Performing Lip Syncing

First, you should add in the Speech Audio .wav file as a guide. This will allow you to time your lip sync animation properly.

1. To do that, go under Animate->Sound Asset Manager to bring up the Sound Asset Manager.

2. Import your .wav speech file as a new Sound Asset

3. Now click on Animate->Add Sound to Animation Knots Timeline to add the sound track onto your Animation Knots Timeline.

4. Set a key at the appropriate time/frame where you want to start playing your .wav speech file.

Remember, this is just an audio guide to help you time your lip syncing animation.

5. Now select your mouth mesh region you want to swap the mouth shapes onto.

6. Enable Image Swap on that mesh region

7. Bring up the Sprite Frame Manager

8. From here, import in your 10 Mouth Shape PNG Images

9. Tag those images with the appropriate mouth shape by using the LipSync Tag functionality. Make sure you tag all 10 mouth shapes.

10. Now run the Lip Sync process by clicking the Run LipSync button.

11. Select the .dat Lip Sync file you generated from Papagayo, set your speed ( defaults to 1.0 )

12. You should now see a set of image swap knots set for you automatically in your timeline. Play the animation and you should have your Lip Sync results.

Video Tutorial

A full video tutorial on the entire Lip Sync process is also available:

Please watch the tutorial here.