Kasucast #25 - Stable Diffusion 3 2B Medium Training with kohya and SimpleTuner (full finetune/LoRA)

The speaker begins by addressing the audience and discussing the recent developments in Stable Diffusion 3 (SD3). They clarify that despite being associated with Stability AI, they were not part of the SD3 fine-tuning team. The speaker intends to explore SD3 using publicly available tools, acknowledging the lack of credentials or a significant social media following as potential reasons for not being part of the core team. They also mention a community license update for Stability AI models and improvements to the existing SD 3.2b medium model. The speaker plans to use tools like Coo's SD scripts and Simple Tuner to train the model, sharing their settings and workflows on GitHub. They caution that training setups may change by the time the video is released and that they were unable to finalize the best training settings for Coo's SD scripts.

The speaker delves into the exploration of various tools available for training SD3, including diffusers by Hugging Face, Simple Tuner by B Gea, and Coo's SD scripts. They discuss the availability of these tools and their features, such as support for full fine-tuning and low training. The speaker chose Simple Tuner for its superior results. They also mention the challenges faced with Coo's SD scripts, particularly with the Prodigy Optimizer. The speaker guides the audience through setting up the environment for Coo's SD scripts, including cloning the repository, setting up a virtual environment, and installing dependencies. They emphasize the importance of using the correct version of Python and CUDA for compatibility.

The speaker discusses the configuration settings for training SD3, highlighting changes made to the training setup, such as switching to SDPA from Transformers to save memory. They also experiment with multi-resolution noise and time step settings to improve image functionality. The speaker provides detailed instructions on setting up the environment, including creating necessary directories and files, and modifying the configuration file. They emphasize the importance of creating a metacap JSON file for dataset metadata and provide a step-by-step guide on how to generate this file using the merge_captions_to_metadata script.

The speaker provides a detailed walkthrough of using Coo's SD scripts for fine-tuning SD3. They discuss the differences between SD3 and SDXL training pipelines, emphasizing the need for a local pre-trained model. The speaker also addresses various settings, such as text encoder batch size and the use of T5 XXL D type. They highlight the importance of caching options for training to commence and discuss the limitations of the current SD3 training branch, such as the lack of support for visualizing results. The speaker also provides insights into the learning rate settings and the use of the Prodigy Optimizer, sharing their experiences and challenges encountered during the training process.

The speaker analyzes the results of their training experiments with SD3, noting the undertraining issues despite using the Prodigy Optimizer. They discuss the potential reasons for this, such as an insufficient learning rate or issues with Coo's repository. The speaker then shifts to using the Adam Optimizer, adjusting the learning rate to 7.5 * 10^-6. They test the trained model using prompts from the dataset and observe the results, noting the lack of stylization and the need for further adjustments. The speaker also discusses the limitations of the current training setup and the challenges in achieving the desired results.

The speaker continues their experimentation with SD3 training, focusing on adjusting the learning rate and training strategies. They test different learning rates and observe the impact on the training results, noting the challenges in finding the optimal rate. The speaker also discusses the use of the Adam Optimizer and the need to warm up to a higher learning rate. They share their experiences with training using various settings and the results they obtained, highlighting the need for further experimentation and adjustments to achieve the desired level of stylization in the generated images.

Frustrated with the lack of success with Coo's experimental SD scripts, the speaker decides to abandon them and switch to Simple Tuner. They praise the well-documented nature of the Simple Tuner repository and guide the audience through the setup process, including cloning the repository, creating a virtual environment, and installing dependencies. The speaker also discusses the importance of setting up Weights and Biases and Hugging Face CLI login for tracking training progress and accessing models.

The speaker provides a detailed guide on setting up the configuration files for Simple Tuner, explaining the need to create a multi-backend JSON file and modify various settings. They discuss the importance of setting the correct paths for the dataset, cache directories, and model outputs. The speaker also addresses the need to adjust the learning rate, batch size, and other training parameters to match the desired training outcomes. They emphasize the importance of careful configuration to ensure successful training with Simple Tuner.

The speaker finalizes the training setup for Simple Tuner, discussing the need to adjust the learning rate, model name, and other parameters. They guide the audience through the process of starting the training and monitoring its progress. The speaker also addresses the challenges of converting the trained model into a format compatible with Comfy UI and shares a script to facilitate this process. They test the trained model using various prompts and observe the results, noting the need for further adjustments to achieve the desired style and quality in the generated images.

The speaker explores the impact of different prompts on the training outcomes of SD3, testing various prompts to see how the model responds. They discuss the challenges of achieving a full-body view and the limitations of the current training setup. The speaker also experiments with short prompts and observes the results, noting the need for further adjustments to improve the model's performance. They emphasize the importance of testing and experimentation in refining the training process and achieving the desired results.

The speaker tests different samplers, such as the Comfy UI OD sampler, to see how they affect the training outcomes. They discuss the need for further documentation on how to use these samplers effectively. The speaker also conducts qualitative experiments to determine the best learning rate and caption dropout settings for achieving the desired style in the generated images. They share their findings and discuss the challenges of balancing prompt adherence and visual style in the training process.

The speaker conducts an oblation study by removing certain layers and calculating new metrics to see the influence of removed layers on the training outcomes. They discuss the results and the potential overfitting of SD3 on prompts rather than visual information. The speaker also explores lower training with Simple Tuner, discussing the environment setup and the need to adjust the learning rate and other parameters. They provide a comparison between full fine-tuning and lower training, highlighting the differences in the results.

In conclusion, the speaker reflects on their experience with SD3 training, noting the challenges and the need for further refinement. They mention the good adherence to the dataset images but also the potential risks of overfitting. The speaker shares their plans to update their training settings on GitHub for reference and future use. They also discuss the upcoming 3.1 update to Stable Diffusion 3 and the potential impact on training. The speaker thanks the audience for watching and looks forward to future explorations in this area.