SDXL ComfyUI Stability Workflow - What I use internally at Stability for my AI Art

Scott Detweiler
28 Jul 202316:44

TLDRIn this video, Scott Weather provides a detailed walkthrough on using SDXL within Comfy UI for creating AI art. He begins with a basic workflow that many artists use, emphasizing the importance of starting simple and then getting creative. Scott demonstrates how to set up the core graph for quality assurance, including loading checkpoints, using conditioners, and setting up positive and negative prompts. He also covers the use of an advanced sampler and discusses the relationship between the number of steps and the start and end steps. The video then delves into refining the image with a refiner, adjusting sampler steps, and experimenting with conditioning latent noise. Scott encourages viewers to play around with different settings and find their unique creative process. He concludes by showing how to save and reload a project using the save image feature and reminds viewers to protect their work when sharing images.


  • ๐ŸŽจ **Workflow Introduction**: Scott Weather discusses a core workflow for AI Art using SDXL within Comfy UI, which is a starting point for users to get creative with.
  • ๐Ÿ“ˆ **Basic Graph Explanation**: The video outlines how to create a basic Comfy UI graph for quality assurance, which can become quite complex but is essential for beginners to learn.
  • ๐Ÿ”ต **Checkpoint Loading**: A checkpoint is loaded first, and the process involves adding nodes and dealing with a refiner later in the workflow.
  • ๐Ÿ–ผ๏ธ **Image Conditioning**: The clip from the checkpoint is conditioned using SDXL's own conditioners, with width, height, and target width and height being set to specific values.
  • โœ… **Positive Prompts**: Positive prompts are used to guide the AI, with an example given as 'a robot shopping at Walgreens', while negative prompts are used to exclude unwanted elements.
  • ๐Ÿ”ฌ **Advanced Sampler**: An advanced sampler is introduced for sampling from the conditioned clip, with a focus on the number of steps and the start and end steps for sampling.
  • ๐Ÿ› ๏ธ **Refiner Usage**: The refiner is brought into the workflow to add more detail and perfection to the image, working on a different model trained on different content.
  • ๐Ÿ”„ **Passing Noise**: It's a best practice to pass any leftover noise from the base sampler to the refiner, adjusting steps to control the process.
  • ๐Ÿ“Š **Step Manipulation**: The video demonstrates how manipulating the steps at which the base sampler and the refiner operate can affect the final output.
  • ๐Ÿ” **Preview and Save**: The use of a preview node is shown to visualize the output without saving numerous files, which can help in managing storage and reviewing work.
  • ๐ŸŒŸ **Creative Exploration**: Scott encourages viewers to experiment with the workflow, play around with different settings, and find their unique creative approach.

Q & A

  • What is the core graph used for in the video?

    -The core graph is used for quality assurance in AI art creation, providing a stable starting point for generating images with ComfyUI and SDXL.

  • How does Scott Weather recommend starting with the ComfyUI and SDXL workflow?

    -Scott Weather suggests starting with the basic ComfyUI and SDXL workflow and then getting creative by adding complexity and personal touches.

  • What is the purpose of using a refiner in the workflow?

    -The refiner is used to enhance the quality of the generated image by working on details such as faces and refining the overall aesthetic of the image.

  • How does the positive prompt function in the workflow?

    -The positive prompt serves as a guide for the AI to generate images that align with the desired concept, such as 'a robot shopping at Walgreens' in the video example.

  • What is the role of the negative prompt in the AI art generation process?

    -The negative prompt helps the AI to avoid generating unwanted elements in the image, by specifying what should not be included in the scene.

  • Why is the width and height of the image usually set to 4096 in the workflow?

    -The width and height are set to 4096 to ensure high-quality images. Lower resolutions might result in quality issues, and this size is also divisible by 16 for best results.

  • What is the significance of using an advanced sampler in the workflow?

    -An advanced sampler provides more control and capabilities for the sampling process, allowing for finer adjustments and potentially better image quality.

  • How does the refiner version of the clip conditioner differ from the regular one?

    -The refiner version of the clip conditioner is used with the refiner model and includes an aesthetic score (A Score) which helps in guiding the refinement process for a better-looking image.

  • What is the recommended approach for managing the steps in the sampler?

    -Scott Weather recommends leaving the end steps at 10,000 and focusing on the start step and the number of steps. It's also suggested to start with a smaller number of steps and then refine further.

  • Why is it important to have both a positive and negative prompt connected for the refiner to function properly?

    -Both a positive and negative prompt are necessary for the refiner to understand what to include and what to exclude in the generated image, ensuring the output aligns with the desired concept.

  • What is the innovative concept introduced towards the end of the video?

    -The innovative concept introduced is conditioning the latent noise before starting the main sampling process, which can lead to different and potentially more refined results.

  • How can one continue working on a project after saving the core graph?

    -By saving the image generated from the core graph, one can simply drag and drop that image back into the system to reload the entire graph for further work.



๐ŸŽจ Introduction to SDXL in Comfy - The Basic Workflow

Scott Weather introduces the viewer to the process of using Stable Diffusion XL (SDXL) within Comfy, a tool for image generation. He emphasizes the importance of this workflow as a starting point and encourages viewers to get creative after understanding the basics. The video begins by loading a checkpoint and an SDXL model, then conditioning the clip with specific width, height, and target dimensions. Positive and negative prompts are discussed, with an example of a robot shopping at Walgreens as the positive prompt. Scott also covers the use of an advanced sampler and the importance of setting the right steps for the sampler to avoid unexpected results.


๐Ÿ” Refining the Image with the Refiner and Sampler

The video continues with the use of a refiner in SDXL, which is loaded similarly to a checkpoint. Scott explains the process of conditioning the clip for the refiner and the importance of setting the right aesthetic score for both positive and negative prompts. He then demonstrates how to use a sampler with the refiner, adjusting the steps and starting points to control the level of detail in the generated image. The refiner is shown to be effective in enhancing details such as faces. Scott also discusses the potential of experimenting with different settings and encourages viewers to find their unique approach.


๐ŸŒŸ Advanced Techniques: Conditioning Latent Noise and Multiple Refining Steps

Scott introduces an advanced technique involving an additional refining step that conditions the latent noise before the main sampling process begins. This is achieved by inserting a refiner step at the beginning of the workflow, which performs only a few steps to rough in the image. The base sampler then takes over, starting from where the refiner left off. This method is presented as an interesting concept for viewers to explore and experiment with, as it can lead to different and sometimes unexpected results in image generation.


๐Ÿ’พ Saving and Reusing the Comfy Graph

The final paragraph discusses how to save and reuse the Comfy graph for continued work on a project. Scott demonstrates that by saving the image to the hard drive, the entire graph can be reloaded by simply dragging and dropping the saved image back into Comfy. He also advises viewers to strip metadata from images before sharing them publicly to prevent accidental sharing of the entire graph. Scott concludes by inviting viewers to share their thoughts and workflow ideas in the comments and thanks them for watching the tutorial.




SDXL refers to a specific model or process used in AI Art generation. In the context of the video, it is the core model that the presenter uses for creating AI-generated images. The video discusses how to integrate and utilize SDXL within the Comfy UI for stability in AI Art creation.

๐Ÿ’กComfy UI

Comfy UI is a user interface or workflow environment used for AI Art. The video demonstrates how to navigate and use Comfy UI to create stable and high-quality AI Art, making it a central theme in the tutorial.


A checkpoint in the context of the video refers to a saved state or model within the AI Art generation process. The presenter discusses loading a checkpoint as a starting point for the AI Art creation workflow.


The refiner is a tool or process used to improve the quality of the AI-generated image. It is mentioned as a step following the base image generation, aiming to add more detail and perfection to the final output.


Conditioning in the video refers to the process of setting parameters or prompts that guide the AI in generating the desired image. It is a crucial part of directing the AI to create specific content, such as a robot shopping at Walgreens.

๐Ÿ’กPositive and Negative Prompts

These are specific instructions given to the AI to include (positive prompts) or exclude (negative prompts) certain elements in the generated image. They are used to steer the AI's creativity and ensure the output matches the desired theme.

๐Ÿ’กAdvanced Sampler

The advanced sampler is a tool within the AI Art generation process that helps to select and combine different elements to create the final image. The video explains how to use this tool to refine the AI Art output.

๐Ÿ’กLatent Noise

Latent noise refers to the initial random input or 'noise' used as a starting point for the AI's image generation process. The video discusses how conditioning this latent noise can influence the outcome of the AI Art.

๐Ÿ’กVAE Decoder

VAE stands for Variational Autoencoder, and the VAE Decoder is a component used to decode the encoded image data into a format that can be visualized. It is part of the process of generating and previewing AI Art in the video.

๐Ÿ’กAesthetic Score

The aesthetic score is a metric used within the refiner tool to evaluate the visual appeal of the generated image. The presenter mentions setting a score of 6 as a 'sweet spot' for balancing the detail and quality of the AI Art.


Metadata refers to the data that provides information about other data. In the context of the video, it is mentioned as a cautionary note to avoid sharing the original PNG files with embedded metadata, which could inadvertently share the entire AI Art generation graph.


Scott Weather shares Stability's official workflow for AI Art using SDXL in ComfyUI.

The workflow is a starting point for users to then get creative with their AI Art projects.

Scott demonstrates how to make a basic ComfyUI graph for SDXL and then build upon it.

The importance of using a checkpoint and refiner within the ComfyUI environment is discussed.

A detailed walkthrough of setting up the core graph for quality assurance is provided.

The use of positive and negative prompts in AI Art generation is explained.

Scott emphasizes the significance of the aesthetic score in clip conditioning.

An advanced sampler is introduced for sampling from the conditioned AI Art.

The concept of latent noise and its role in the sampling process is explored.

Scott discusses the relationship between the number of steps and the start and end steps in the sampler.

A unique approach to refining AI Art by conditioning the latent noise before sampling is introduced.

The video showcases how to add an additional refining step for more detailed AI Art results.

The practical application of the refiner model in perfecting AI Art images is demonstrated.

Tips for avoiding accidental sharing of the entire graph through metadata in images are given.

Scott encourages viewers to experiment and find their own unique workflow with AI Art.

The video concludes with an invitation for viewers to share their ideas and feedback in the comments.

An emphasis on the iterative and experimental nature of working with AI Art tools is highlighted.