SDXL ComfyUI Stability Workflow - What I use internally at Stability for my AI Art
TLDRIn this video, Scott Weather provides a detailed walkthrough on using SDXL within Comfy UI for creating AI art. He begins with a basic workflow that many artists use, emphasizing the importance of starting simple and then getting creative. Scott demonstrates how to set up the core graph for quality assurance, including loading checkpoints, using conditioners, and setting up positive and negative prompts. He also covers the use of an advanced sampler and discusses the relationship between the number of steps and the start and end steps. The video then delves into refining the image with a refiner, adjusting sampler steps, and experimenting with conditioning latent noise. Scott encourages viewers to play around with different settings and find their unique creative process. He concludes by showing how to save and reload a project using the save image feature and reminds viewers to protect their work when sharing images.
Takeaways
- ๐จ **Workflow Introduction**: Scott Weather discusses a core workflow for AI Art using SDXL within Comfy UI, which is a starting point for users to get creative with.
- ๐ **Basic Graph Explanation**: The video outlines how to create a basic Comfy UI graph for quality assurance, which can become quite complex but is essential for beginners to learn.
- ๐ต **Checkpoint Loading**: A checkpoint is loaded first, and the process involves adding nodes and dealing with a refiner later in the workflow.
- ๐ผ๏ธ **Image Conditioning**: The clip from the checkpoint is conditioned using SDXL's own conditioners, with width, height, and target width and height being set to specific values.
- โ **Positive Prompts**: Positive prompts are used to guide the AI, with an example given as 'a robot shopping at Walgreens', while negative prompts are used to exclude unwanted elements.
- ๐ฌ **Advanced Sampler**: An advanced sampler is introduced for sampling from the conditioned clip, with a focus on the number of steps and the start and end steps for sampling.
- ๐ ๏ธ **Refiner Usage**: The refiner is brought into the workflow to add more detail and perfection to the image, working on a different model trained on different content.
- ๐ **Passing Noise**: It's a best practice to pass any leftover noise from the base sampler to the refiner, adjusting steps to control the process.
- ๐ **Step Manipulation**: The video demonstrates how manipulating the steps at which the base sampler and the refiner operate can affect the final output.
- ๐ **Preview and Save**: The use of a preview node is shown to visualize the output without saving numerous files, which can help in managing storage and reviewing work.
- ๐ **Creative Exploration**: Scott encourages viewers to experiment with the workflow, play around with different settings, and find their unique creative approach.
Q & A
What is the core graph used for in the video?
-The core graph is used for quality assurance in AI art creation, providing a stable starting point for generating images with ComfyUI and SDXL.
How does Scott Weather recommend starting with the ComfyUI and SDXL workflow?
-Scott Weather suggests starting with the basic ComfyUI and SDXL workflow and then getting creative by adding complexity and personal touches.
What is the purpose of using a refiner in the workflow?
-The refiner is used to enhance the quality of the generated image by working on details such as faces and refining the overall aesthetic of the image.
How does the positive prompt function in the workflow?
-The positive prompt serves as a guide for the AI to generate images that align with the desired concept, such as 'a robot shopping at Walgreens' in the video example.
What is the role of the negative prompt in the AI art generation process?
-The negative prompt helps the AI to avoid generating unwanted elements in the image, by specifying what should not be included in the scene.
Why is the width and height of the image usually set to 4096 in the workflow?
-The width and height are set to 4096 to ensure high-quality images. Lower resolutions might result in quality issues, and this size is also divisible by 16 for best results.
What is the significance of using an advanced sampler in the workflow?
-An advanced sampler provides more control and capabilities for the sampling process, allowing for finer adjustments and potentially better image quality.
How does the refiner version of the clip conditioner differ from the regular one?
-The refiner version of the clip conditioner is used with the refiner model and includes an aesthetic score (A Score) which helps in guiding the refinement process for a better-looking image.
What is the recommended approach for managing the steps in the sampler?
-Scott Weather recommends leaving the end steps at 10,000 and focusing on the start step and the number of steps. It's also suggested to start with a smaller number of steps and then refine further.
Why is it important to have both a positive and negative prompt connected for the refiner to function properly?
-Both a positive and negative prompt are necessary for the refiner to understand what to include and what to exclude in the generated image, ensuring the output aligns with the desired concept.
What is the innovative concept introduced towards the end of the video?
-The innovative concept introduced is conditioning the latent noise before starting the main sampling process, which can lead to different and potentially more refined results.
How can one continue working on a project after saving the core graph?
-By saving the image generated from the core graph, one can simply drag and drop that image back into the system to reload the entire graph for further work.
Outlines
๐จ Introduction to SDXL in Comfy - The Basic Workflow
Scott Weather introduces the viewer to the process of using Stable Diffusion XL (SDXL) within Comfy, a tool for image generation. He emphasizes the importance of this workflow as a starting point and encourages viewers to get creative after understanding the basics. The video begins by loading a checkpoint and an SDXL model, then conditioning the clip with specific width, height, and target dimensions. Positive and negative prompts are discussed, with an example of a robot shopping at Walgreens as the positive prompt. Scott also covers the use of an advanced sampler and the importance of setting the right steps for the sampler to avoid unexpected results.
๐ Refining the Image with the Refiner and Sampler
The video continues with the use of a refiner in SDXL, which is loaded similarly to a checkpoint. Scott explains the process of conditioning the clip for the refiner and the importance of setting the right aesthetic score for both positive and negative prompts. He then demonstrates how to use a sampler with the refiner, adjusting the steps and starting points to control the level of detail in the generated image. The refiner is shown to be effective in enhancing details such as faces. Scott also discusses the potential of experimenting with different settings and encourages viewers to find their unique approach.
๐ Advanced Techniques: Conditioning Latent Noise and Multiple Refining Steps
Scott introduces an advanced technique involving an additional refining step that conditions the latent noise before the main sampling process begins. This is achieved by inserting a refiner step at the beginning of the workflow, which performs only a few steps to rough in the image. The base sampler then takes over, starting from where the refiner left off. This method is presented as an interesting concept for viewers to explore and experiment with, as it can lead to different and sometimes unexpected results in image generation.
๐พ Saving and Reusing the Comfy Graph
The final paragraph discusses how to save and reuse the Comfy graph for continued work on a project. Scott demonstrates that by saving the image to the hard drive, the entire graph can be reloaded by simply dragging and dropping the saved image back into Comfy. He also advises viewers to strip metadata from images before sharing them publicly to prevent accidental sharing of the entire graph. Scott concludes by inviting viewers to share their thoughts and workflow ideas in the comments and thanks them for watching the tutorial.
Mindmap
Keywords
๐กSDXL
๐กComfy UI
๐กCheckpoint
๐กRefiner
๐กConditioning
๐กPositive and Negative Prompts
๐กAdvanced Sampler
๐กLatent Noise
๐กVAE Decoder
๐กAesthetic Score
๐กMetadata
Highlights
Scott Weather shares Stability's official workflow for AI Art using SDXL in ComfyUI.
The workflow is a starting point for users to then get creative with their AI Art projects.
Scott demonstrates how to make a basic ComfyUI graph for SDXL and then build upon it.
The importance of using a checkpoint and refiner within the ComfyUI environment is discussed.
A detailed walkthrough of setting up the core graph for quality assurance is provided.
The use of positive and negative prompts in AI Art generation is explained.
Scott emphasizes the significance of the aesthetic score in clip conditioning.
An advanced sampler is introduced for sampling from the conditioned AI Art.
The concept of latent noise and its role in the sampling process is explored.
Scott discusses the relationship between the number of steps and the start and end steps in the sampler.
A unique approach to refining AI Art by conditioning the latent noise before sampling is introduced.
The video showcases how to add an additional refining step for more detailed AI Art results.
The practical application of the refiner model in perfecting AI Art images is demonstrated.
Tips for avoiding accidental sharing of the entire graph through metadata in images are given.
Scott encourages viewers to experiment and find their own unique workflow with AI Art.
The video concludes with an invitation for viewers to share their ideas and feedback in the comments.
An emphasis on the iterative and experimental nature of working with AI Art tools is highlighted.