ComfyUI: Stable Video Diffusion (Workflow Tutorial)

Mali introduces the video tutorial on Stability AI's first model for stable video diffusion. The model allows for frame control and subtle animations on images, with six comfy graphs showcasing different image to video output fine-tuning techniques. Mali thanks new channel members and mentions the availability of resources like Json files and MP4 videos for YouTube channel members. The tutorial will cover the use of Comfy UI, custom nodes, and FFMpeg for video format conversion, focusing on the model's capabilities for video generation at 1024x576 resolution in both portrait and landscape orientations.

The paragraph details the initial steps in setting up the Comfy UI workflow for video generation using Stability AI's model. It includes installing necessary custom nodes like the W node suit, video helper suite, and image resize, as well as configuring the environment variables for FFMpeg. The workflow begins with the video model option and progresses through nodes for image to video conditioning, K-sampler, and VAE decode. Mali emphasizes the importance of maintaining image ratios and cropping for video output, using a candle image as an example to demonstrate motion control in the video.

This section delves into fine-tuning the video generation parameters, such as the CFG value and motion bucket ID, which determine camera motion and detail levels. Mali explains the relationship between these settings and their impact on the video output, using different values to illustrate the effects on motion and detail. The paragraph also covers the importance of the augmentation level, which adds noise and detail to the generation, and how it should be carefully adjusted to avoid poor motion details. The tutorial progresses with practical examples, including changing the frame rate and experimenting with different formats like GIF, webp, and MP4 for better quality.

Mali explores advanced techniques for animating specific parts of an image, such as making a woman's hand wave or creating a blinking effect in a portrait image. The tutorial covers the use of different samplers and schedulers to achieve these animations, as well as the importance of the motion bucket level in determining which parts of the image will move. The paragraph also introduces tricks for animating facial features, such as eyes, by using ancestral samplers and adjusting the motion bucket level, demonstrating how to achieve more natural and subtle animations.

The focus shifts to creating subtle animations like blinking eyes and lip movements using multi-image inputs. Mali demonstrates how providing the AI with a set of images in a specific order can influence the animation outcome, without the need for frame interpolation. The paragraph explains the process of setting up image loaders and batch nodes to feed the AI with a sequence of images for the desired animation. It also discusses the challenges of maintaining color consistency and offers solutions, such as adjusting the ratio of open to closed eye images to improve the animation quality.

This section discusses animating complex motions, such as forward movement, using the right parameters to avoid unwanted camera panning or background movement. Mali uses a motorbike example to illustrate the effect of the augmentation level on motion details and how sensitive this setting is. The tutorial shows how to adjust the motion bucket and CFG values to achieve a smooth forward motion, and how changing the sampler and scheduler can affect the animation outcome, with a special note on avoiding pedal movement animation due to model limitations.

Mali introduces a complex workflow for combining videos using a technique called noisy latent composition. The aim is to create a video with slow-moving waves and time-lapsed clouds in the sky from a DSLR photo. The paragraph explains the process of creating groups for each video, adjusting image sizes, and using nodes like image to video, image size to number, and number to text for precise resolution control. It also covers the importance of the order in which prompts and outputs are combined for the final video composition, using conditioning combine nodes and a latent composite node to layer the cloud animation over the main image.

The final paragraph outlines the process of finalizing the video output by integrating a text to image workflow. It describes setting up the text to image group with the appropriate aspect ratio and connecting it to the video processing group. The tutorial emphasizes adjusting the denoise value for the final output to ensure a close match to the previous video layers and using the feather value to blend the layers seamlessly. Mali concludes the tutorial by making the video output look beautiful and mentioning that all Json files will be available for YouTube members, hoping the tutorial was helpful.