Stable Diffusion 3 via API in comfyUI with Stability AI official nodes - SD Experimental

Andrea Baioni
19 Apr 202420:14

TLDRIn this video, Andrea Baioni guides viewers through using Stable Diffusion 3 (SD3) via API key in ComfyUI, a process that requires purchasing credits for image generation, costing about 6 cents per image. He demonstrates installing the necessary nodes from Stability AI's GitHub, inputting the API key for each node, and generating images using the SD3 and SD3 Turbo models. Baioni also explores additional features like upscaling, outpainting, inpainting, and search and replace, noting that the base model provides impressive results despite some minor issues. The video concludes with a slideshow of generated images and an invitation for viewers to suggest prompts for testing.

Takeaways

  • 📈 Stable Diffusion 3 (SD3) is available for use via API keys but not as a free checkpoint, requiring the purchase of credits for image generation.
  • 💵 The cost for generating an image with SD3 is around 6 cents USD per image, with the option to wait for a future free checkpoint release.
  • 🛠️ To use SD3 with ComfyUI, you need to install missing custom nodes from the Stability AI GitHub page and input your Stability AI API key into each node.
  • 🔑 API keys are found on the Stability AI account page, and additional credits can be purchased if the initial 25 free credits are insufficient.
  • 💡 The SD3 node offers options to select between SD3 and SD3 Turbo models, with the latter being slightly less costly in terms of credits.
  • 👗 A positive prompt for image generation was tested, resulting in a comparison between the Core and SD3 models, with SD3 producing more accurate results to the prompt.
  • 🎨 Additional nodes such as 'stability creative upscale' and 'stability outpainting' were explored, demonstrating impressive results with base models.
  • 🖼️ The 'stability inpainting' feature allowed for changes in the image, such as a quick change of clothes and model, albeit with some issues like hair transformation.
  • 🔍 The 'stability search and replace' node was tested, successfully replacing a person with a cat in an image, albeit with some unexpected perspective changes.
  • 🚫 The 'stability remove background' node was not functional in the demonstration, possibly due to a missing field for API key input.
  • 🤖 The entire process was conducted within ComfyUI, providing a seamless workflow for users familiar with the platform and offering the ability to test SD3 without relying on external interfaces.
  • 📸 The presenter, Andrea Baioni, offered to test additional prompts provided by the audience and shared a link to view the generated SD3 images.

Q & A

  • What is the current status of Stable Diffusion 3 (SD3) in terms of availability and usage?

    -Stable Diffusion 3 has been released for use with API keys but has not been released as a checkpoint. This means it can only be used with API calls and is not free. Users need to purchase credits to use it.

  • How much does it cost to generate an image using SD3 via API?

    -It costs around 6 cents of a US dollar for image generation using SD3 via API.

  • What is the process of setting up a workflow in ComfyUI to use SD3?

    -To set up a workflow in ComfyUI for SD3, you need to install missing custom nodes from the Stability AI GitHub page, restart ComfyUI, and then input your Stability AI API key into the API key override field for each node you want to use.

  • What are the different nodes available for use with SD3 in ComfyUI?

    -The available nodes include stability image core, stability SD3, preview image, stability remove background, stability creative upscale, stability outpainting, stability inpainting, and stability research and replace.

  • How can one obtain Stability AI API keys and credits?

    -You can obtain Stability AI API keys and credits by signing up or logging into your Stability AI account, navigating to the account page, and purchasing additional credits if needed. The initial 25 free credits are provided upon account creation.

  • What is the cost of an API call for different models in Stability AI?

    -The cost per API call varies by model: SD3 costs 6.5 credits, SD3 turbo costs 4 credits, and Core costs 3 credits.

  • How can one test SD3 without spending their own money?

    -New users can test SD3 with the initial 25 free credits provided by Stability AI upon account creation, which should cover at least a couple of generations.

  • What was the outcome of the first image generation test using SD3 and Core models?

    -The first image generation test resulted in two images: the Core model translated the environment settings into the clothing and general mood, while the SD3 model produced clothing more similar to Miu Miu and included a skylight as prompted.

  • What issues were encountered when trying to use the stability creative upscale and outpainting nodes?

    -The upscaled image was too large to be handled by the outpainting node, resulting in a payload error. Adjusting the outpainting settings to a smaller expansion size resolved the issue.

  • What was the result of the search and replace node when trying to replace a person with a cat?

    -The search and replace node successfully replaced the person with a giant cat, although the perspective and some details were altered in the process.

  • Why was the remove background node not functional in the given demonstration?

    -The remove background node was not functional because it kept asking for an API key, but there was no field provided to input it. A potential workaround was mentioned but not demonstrated.

  • How can users share their prompts for testing SD3 with the presenter?

    -Users can leave comments with their prompts, and the presenter may test them out and share the results on imgur for everyone to see.

Outlines

00:00

📝 Introduction to Stable Diffusion 3 with API Key in ComfyUI

The video begins with an introduction to Stable Diffusion 3 (SD3), a tool released by Stability AI for image generation via API keys. It's clarified that while SD3 is available for use, it isn't free and requires the purchase of credits for image generation, costing approximately 6 cents per image. The video promises a walkthrough on setting up the workflow in ComfyUI and using the API keys for image generation. The process includes installing missing nodes from the Stability AI GitHub page and using the provided workflow image to set up the ComfyUI workspace.

05:02

💳 Purchasing Credits and Setting Up API Key for Stability AI SD3

The speaker guides viewers on how to purchase credits and set up an API key for Stability AI's SD3. They explain the pricing structure for different models and how to buy additional credits if the initial 25 free credits are insufficient. The process includes navigating to the account page, revealing the API key, and copying it to the clipboard for use in ComfyUI. The video then demonstrates how to input the API key into each node within ComfyUI that requires it.

10:03

🖼️ Generating Images with Core and SD3 Models in ComfyUI

The video continues with a demonstration of generating images using the Core and SD3 models within ComfyUI. The speaker inputs a positive prompt for a fashion scene and compares the results from both models. They note that the SD3 model produces a more accurate representation of the Miu Miu fashion style and includes details like a skylight, which was absent in the Core model's output. The speaker also discusses the cost of credits used for these image generations.

15:05

🎨 Exploring Additional Nodes for Image Manipulation

The speaker explores additional nodes for creative image manipulation, including upscaling, outpainting, inpainting, and search and replace functionalities. They activate the nodes, input prompts, and discuss the results of each operation. The video highlights the impressive detail and quality of the base model's output, despite not having community fine-tuned models or additional tools like RealVIS. The speaker also humorously addresses an error encountered while trying to perform a search and replace operation, providing a solution to the issue.

🚀 Conclusion and Future Testing with SD3 in ComfyUI

The video concludes with the speaker's positive first impressions of the SD3 models and a summary of the nodes tested. They mention an issue with the remove background node but do not attempt a workaround. The speaker offers to test prompts provided by viewers and shares links to view the generated images. They sign off with contact information and a slideshow of SD3 images generated during the video.

Mindmap

Keywords

💡Stable Diffusion 3

Stable Diffusion 3 (SD3) is an advanced AI model developed by Stability AI for generating images from textual descriptions. It is a successor to previous models and is noted for its improved capabilities. In the video, the host discusses how to use SD3 via an API key, indicating its current state as a paid service requiring credits for image generation.

💡API key

An API key is a unique code used to authenticate a user with an application or service. In the context of the video, an API key is necessary to access and use the Stable Diffusion 3 model. The host explains the process of obtaining an API key from the Stability AI website and using it within the ComfyUI application to generate images.

💡ComfyUI

ComfyUI is a user interface that allows users to interact with and utilize AI models like Stable Diffusion 3 without needing to use external web interfaces. The video demonstrates how to set up a workflow in ComfyUI to use SD3, including installing necessary nodes and using API keys to generate images.

💡Image generation

Image generation refers to the process of creating visual content from textual descriptions using AI models like SD3. The video script details the steps to generate images, including setting up the API key, selecting models, and inputting prompts to create specific images as desired by the user.

💡Credits

In the context of using Stability AI's SD3, credits are a form of virtual currency used to pay for the service of image generation. The host mentions that each API call to generate an image costs a certain number of credits, with different models costing different amounts. The host also discusses purchasing additional credits.

💡Positive prompt

A positive prompt is a textual description that guides the AI model to include certain elements or characteristics in the generated image. In the video, the host uses a positive prompt to instruct SD3 to create an image of a young woman wearing Miu Miu haute couture in a baroque room.

💡Negative prompt

A negative prompt is a textual instruction used to specify what should be avoided or not included in the generated image. The host chooses to leave the negative prompt field empty in the example to see how the AI performs with only positive guidance.

💡Upscaling

Upscaling is the process of enhancing the resolution or detail of an image, often to make it appear larger without losing quality. The video demonstrates using the stability creative upscale node to improve the detail of an image generated by SD3.

💡Outpainting

Outpainting is an AI technique where the model generates and adds additional visual content to the edges of an existing image. The host experiments with outpainting by expanding the image to the sides and discusses the results and the perspective changes that occur.

💡Inpainting

Inpainting is the process of filling in missing or damaged parts of an image with AI-generated content that matches the surrounding areas. The host uses the stability inpainting node to change the clothing and model in an image while keeping the background intact.

💡Search and Replace

Search and Replace is a feature that allows users to identify a specific element in an image and replace it with something else. The host attempts to use this feature to replace a person in an image with a cat, resulting in a humorous and unexpected outcome.

Highlights

Stable Diffusion 3 (SD3) is accessible via API key in ComfyUI, but not yet available as a free checkpoint.

Using SD3 requires purchasing credits, costing approximately 6 US cents per image generation.

Stability AI's official nodes and workflow for ComfyUI are available on their GitHub page.

Missing nodes in ComfyUI can be installed through the manager and the application may need to be restarted.

Each Stability AI node features an API key override field for user-specific keys.

The Stability AI API key can be obtained and managed through the user's account on the Stability AI website.

Credits for image generation can be purchased in increments starting at $10 for 1000 credits.

The Core and SD3 models differ in their level of refinement and cost, with SD3 Turbo being a cost-effective alternative.

Positive and negative prompt fields are used to guide the image generation process in SD3.

The output format for SD3 should be set correctly to avoid errors in the generation process.

The generated images from the Core and SD3 models are of high quality, even without community fine-tuning.

Stability Creative Upscale node enhances image details and corrects anatomy with impressive results.

Outpainting node expands the image while maintaining perspective and ambience, though it has limitations with large payload sizes.

Inpainting node allows for changes in the image, such as altering clothing or models, with some minor issues.

Search and Replace node can substitute elements within an image, as demonstrated by replacing a person with a cat.

The Remove Background node was not functional in the demonstration, possibly due to a missing API key input field.

Users can utilize the 25 free credits provided by Stability AI to test out several generations of image generation.

ComfyUI provides a seamless workflow for using SD3 without relying on external web interfaces.