Live Portrait vs. Hedra: AI Facial Animation Showdown & Tutorial

Bob Doyle Media
11 Jul 202420:29

TLDRThis video tutorial compares Live Portrait and Hedra, two AI facial animation tools. It demonstrates how to animate a singing portrait using Live Portrait, with a focus on audio and video synchronization. The creator shares tips for better results, such as adjusting eye and lip retargeting multipliers and avoiding camera distortion. The tutorial also explores creating animations with different vocal tracks and varying facial expressions, showcasing the potential of AI in facial animation.

Takeaways

  • 😀 The video discusses the creative uses of AI in facial animation, particularly focusing on the comparison between Live Portrait and Hedra.
  • 🕹️ Live Portrait is a throwback to technology from 3 years ago but offers better resolution, requiring a video to drive the facial animation rather than real-time processing.
  • 🎤 Hedra's animation is driven 100% by the audio fed into it, in contrast to Live Portrait which uses a video of the user's face for animation.
  • 🎵 The creator uses Adobe Audition to create music and vocal tracks for the animation, experimenting with voice conversion and different voices.
  • 🖥️ The process of creating animations involves several stages, including creating images of the singers, preparing video files, and using them to drive the facial animation.
  • 🎭 Live Portrait allows for the animation of various styles, including photographic, fantasy, 3D, illustrative, and anime, as long as a face is recognizable.
  • 🔧 The video provides a tutorial on how to use Live Portrait effectively, including tips on avoiding common issues like distortion and excessive movement.
  • 👩‍🎤 The creator emphasizes the importance of proper camera distance and lighting to prevent distortion and ensure natural-looking animations.
  • 🎬 Live Portrait can be used for more than just singing; it works for any facial expressions or movements, making it versatile for various applications.
  • 📲 The tutorial also covers the use of smartphone apps like Video Leap and iPhone's native editing tools to prepare video files for animation.

Q & A

  • What is the main focus of the video titled 'Live Portrait vs. Hedra: AI Facial Animation Showdown & Tutorial'?

    -The main focus of the video is to compare and demonstrate the AI facial animation technologies of Live Portrait and Hedra, providing a tutorial on how they work and how to use them effectively.

  • How does Live Portrait differ from Hedra in terms of facial animation?

    -Live Portrait requires a video that drives the facial animation, whereas Hedra is driven 100% by the audio fed into it. Live Portrait is not in real time and offers better resolution compared to Hedra, which is real-time but may not have as high resolution.

  • What is the process of creating an animation with Live Portrait as described in the video?

    -The process involves creating images of the singers, preparing video files recorded on a phone for animation, and using these videos to drive the animation of the created faces to sing. It also includes adjusting the retargeting multipliers for eyes and lips for a more natural look.

  • Why did the video creator choose to use Adobe Audition to create music for the animation?

    -The video creator used Adobe Audition to create music because it allows for the manipulation of vocal tracks, such as shifting semitones and converting voices, which is necessary for the animation project they are working on.

  • What role does the video editing software play in preparing the driving video for Live Portrait?

    -The video editing software is used to crop the recorded video into a square format and adjust the frame rate to ensure the driving video is compatible with Live Portrait's requirements for facial animation.

  • What is the significance of the 'eyre retargeting multiplier' and 'lips retargeting multiplier' in Live Portrait?

    -The 'eyre retargeting multiplier' and 'lips retargeting multiplier' in Live Portrait adjust how much influence the driving animation has on the eyes and lips of the animated face, allowing for a more natural or exaggerated expression depending on the setting.

  • How does the video creator address issues with hair tearing in the facial animation?

    -The video creator addresses hair tearing by ensuring the hair in the images used for animation is not too large or exaggerated, as this can cause the animation to distort. They also adjust the driving video to minimize excessive head movements.

  • What is the purpose of using different images and voices in the animation project?

    -The purpose of using different images and voices is to create a diverse and engaging animation that showcases the capabilities of Live Portrait and Hedra in animating various facial expressions and singing styles.

  • What advice does the video provide for those who are new to installing and using Live Portrait?

    -For those new to Live Portrait, the video suggests referring to another YouTube channel for a detailed installation guide, as the process can be complex and is beyond the scope of the tutorial provided in the video.

  • How does the video creator ensure that the facial animations look natural?

    -The video creator ensures natural-looking facial animations by adjusting the retargeting multipliers, selecting appropriate images with suitable hair and facial features, and recording driving videos with minimal distortion and exaggerated expressions.

Outlines

00:00

🎭 Introduction to Facial Animation with Live Portrait

The paragraph introduces the topic of facial animation using AI, specifically focusing on the Live Portrait technology. It discusses the evolution of facial animation since the presenter's earlier videos on avatarify theater and other apps like Mug Life and Expression. The presenter notes that Live Portrait differs from real-time apps like Hedra, as it requires a pre-recorded video to drive the facial animation. The comparison between Hedra and Live Portrait is highlighted, with Hedra being audio-driven and Live Portrait being video-driven. The paragraph also mentions the process of creating an animation with music, starting with composing tracks in Adobe Audition and converting voices using Otomy.

05:00

🎵 Voice Conversion and Image Generation for Animation

This paragraph delves into the process of voice conversion and image generation for the animation project. The presenter exports vocal tracks and uses Otomy to shift the pitch to create different voices. The goal is to find the best voice for the animation. The paragraph also discusses the use of Prome AI to generate images of a 1940s-style jazz female singer, with a focus on getting a direct camera-facing image for better animation results. The presenter experiments with image variations and lighting to achieve the desired look for the animation.

10:01

🖥️ Setting Up Hedra and Live Portrait Animations

The paragraph explains the process of setting up animations in Hedra and Live Portrait. The presenter uploads the generated images and audio tracks into Hedra to create animations. It mentions the importance of testing the audio tracks together before committing to the animation process. The paragraph also covers the technical aspects of installing Live Portrait and using it with Compy UI, suggesting viewers refer to another video for detailed installation instructions. The presenter shares a workflow for Live Portrait, including adjustments to the eye and lip retargeting multipliers for more natural animations.

15:01

🎥 Recording and Adjusting Driving Videos for Animation

This paragraph focuses on recording and adjusting the driving videos used to animate the facial expressions in Live Portrait. The presenter discusses the challenges of being too close to the camera and the resulting distortions, as well as issues with excessive head movement and hair that's too prominent. The solution involves re-recording the driving videos from a greater distance to reduce distortion and choosing images with less prominent hair. The paragraph also covers the process of cropping videos into a square format and adjusting frame rates for compatibility with Live Portrait.

20:01

🎤 Finalizing Live Portrait Animations and Testing Parameters

The paragraph describes the final steps in creating Live Portrait animations, including fine-tuning the parameters for eye and lip movement to achieve a more natural look. The presenter tests different multiplier values to reduce the exaggeration in the animations. The paragraph also covers the process of creating images with shorter hair to avoid issues with hair tearing in the animation. The presenter experiments with different images and driving videos, adjusting parameters to improve the quality of the animations. The paragraph concludes with a demonstration of the final animations, showcasing the improved results after making the necessary adjustments.

📢 Conclusion and Call to Action

In the final paragraph, the presenter wraps up the video by emphasizing the versatility of Live Portrait for various types of facial animations, not just for singing but also for speaking and any face-like imagery. The paragraph serves as a call to action for viewers to subscribe to the channel for more cutting-edge content on AI and technology. It humorously plays with the idea of the presenter looking for viewers if they don't subscribe, followed by a musical note to punctuate the call to action.

Mindmap

Keywords

💡Live Portrait

Live Portrait is a technology mentioned in the video that allows for facial animation. It is not in real-time and requires a video to drive the facial animation. The video's creator uses Live Portrait to animate images with audio tracks, aiming for higher resolution animations compared to what was possible three years prior. The process involves creating a video that will guide the facial movements of the animated image.

💡Facial Animation

Facial animation refers to the process of creating movement in the facial features of a character or image. In the context of the video, it involves using either an audio track or a video to drive the movements of a still image, making it appear as if the image is speaking or singing. The video discusses advancements in this technology over the past few years.

💡Hedra

Hedra is another AI facial animation tool compared to Live Portrait in the video. Hedra animations are driven 100% by the audio fed into it, which is different from Live Portrait, where a video of the face performing the desired actions is required. The video creator compares Hedra with Live Portrait to demonstrate the differences in how they animate facial features.

💡Adobe Audition

Adobe Audition is a digital audio workstation mentioned in the video, used for recording and editing audio tracks. The video creator uses Audition to lay down vocal tracks and perform voice conversion to create different singing voices for the animation, showcasing the creative process behind generating audio for facial animations.

💡Voice Conversion

Voice conversion is a process where the original voice of a recording is altered to sound like a different voice. In the script, the video creator uses voice conversion in Adobe Audition to change the pitch of the original vocal tracks to create harmonies, which will then be used to drive the facial animations in Live Portrait and Hedra.

💡Prome AI

Prome AI is an image generation tool referenced in the video. The creator uses it to generate images of a '40s style jazz female singer with specific characteristics like facing the camera. These images are then used as the basis for the facial animations in both Hedra and Live Portrait, demonstrating the tool's utility in creative projects.

💡Luminescence

Luminescence is described as a new feature in Prome AI that the creator uses to generate images with even lighting. This feature is important for facial animation as consistent lighting can improve the quality of the animation by ensuring that the facial features are clearly defined and easily recognizable by the animation software.

💡Video Editing

Video editing is a process mentioned in the video where the creator adjusts the aspect ratio of the recorded videos to fit a square format suitable for the animation process. The video creator uses a mobile app called Video Leap to crop the video into a square shape, which is necessary for the facial animation software to function correctly.

💡Compy UI

Compy UI seems to be a user interface or software environment where Live Portrait is run. The video creator mentions that installing Live Portrait can be complex, especially for newcomers, and suggests referring to another video for installation guidance. Compy UI is presumably where the creator interacts with Live Portrait to create the animations.

💡Retargeting Multiplier

Retargeting Multiplier is a term used in the video to describe a setting within Live Portrait that adjusts how much influence the driving video has on the animated image's eyes and lips. The creator experiments with these multipliers to achieve a more natural look in the facial animations, avoiding exaggerated movements.

💡Workflow

Workflow in the video refers to the series of steps the creator follows to create the facial animations using Live Portrait. The workflow includes arranging blocks, adjusting settings, and queuing up jobs. The creator shares insights about optimizing the workflow for better results, such as using a driving video with less exaggerated movements and adjusting the retargeting multipliers.

Highlights

Introduction to Live Portrait and its comparison with Hedra in AI facial animation.

Live Portrait's throwback to 3-year-old technology with improved resolution.

Hedra's animation is audio-driven, unlike Live Portrait which uses a video of facial movements.

Tutorial on how to create animations with Live Portrait and Adobe Audition.

Process of exporting vocal tracks and voice conversion using Adobe Audition.

Importance of choosing the right voice for the animation.

Steps to create animation files in multiple stages.

Creating images of singers to be animated with audio tracks.

Using phone-recorded videos to drive the facial animation process.

Comparison between Live Portrait and Hedra in terms of animation quality.

Creating a Hedra version of the animation using converted female voice files.

Importance of proper camera angle and lighting for creating initial images.

Using Prome AI to generate images of a 1940s style jazz female singer.

Tips for avoiding common mistakes in facial animation using Live Portrait.

Adjusting the eyes and lips retargeting multiplier for a more natural animation effect.

Creating a workflow in Live Portrait for efficient animation processing.

Troubleshooting and fixing issues with facial feature distortion in animations.

Techniques for re-recording driving videos to improve animation quality.

Finalizing the animation by adjusting the lip and eye movements for a more natural look.

Demonstration of how Live Portrait can animate various styles including photographic, fantasy, 3D, illustrative, and anime.