Wild AI Video Workflow with Viggle, Kaiber, Leonardo, Midjourney, Gen-2, and MORE!

Theoretically Media
2 Apr 202411:58

TLDRIn this video, the creator shares an innovative AI filmmaking workflow that spans from pre-production to generating short films. Inspired by the 2016 film 'Rogue One', the process involves using AI tools to create a hybrid storyboard animatic animation. The creator demonstrates the workflow using the 'Are you not entertained?' scene from 'Gladiator', incorporating elements from 'John Carter of Mars' and 'Warhammer 40K'. The video explores various AI platforms like Vigle, Midjourney, and Kyber, highlighting their strengths and limitations in generating characters, backgrounds, and editing for a cohesive cinematic output. The creator concludes that while the method may not be perfect for full-length features, it's promising for short films and pre-production planning.

Takeaways

  • 🎬 The speaker shares an AI filmmaking workflow that has potential from pre-production to generating short films.
  • 🚀 The inspiration comes from the 2016 film Rogue One, specifically an interview with the editor about creating a feature-length story reel before the script was finished.
  • 🧠 The idea is to use AI tools to create a hybrid storyboard, animatic, and animation, taking inspiration from the method used in Rogue One.
  • 🎥 The process involves clipping reference footage, using Vigle for initial video generation, and Midjourney for character design.
  • 🌟 The workflow includes using AI to augment footage, with examples like the 'Are you not entertained?' scene from Gladiator and elements from Warhammer 40K.
  • 📸 Vigle's 2.0 update is utilized for its improved capabilities, but it has limitations, especially with camera movement.
  • 🖼️ Leonardo is used to create character images in a 9:16 format, ensuring full-body shots for detailed reference.
  • 🎞️ The output from Vigle is refined using Kyber's motion 3.0 feature for a more cohesive and stylized result.
  • 🌐 Backgrounds are created with movement and life using Gen 2, and then combined with the Kyber-generated character for a unified look.
  • 🎨 Video editing software like Premiere or DaVinci is used for compositing the character and background, with adjustments for a more cinematic feel.
  • 🎙️ Audio elements like crowd chanting are generated using AI, and text-to-speech is utilized for dialogue, although finding the right model can be challenging.

Q & A

  • What is the main topic of the video?

    -The main topic of the video is an AI film making workflow that covers various stages from pre-production to generating short films, using a combination of different AI tools.

  • What film inspired the creation of this workflow?

    -The 2016 film Rogue One, directed by Gareth Edwards, inspired the creation of this workflow due to its historical significance and innovative use of AI in film production.

  • What specific scene from another movie is used as a reference in the video?

    -The 'Are you not entertained?' scene from the movie Gladiator is used as a reference, with elements from John Carter of Mars and Warhammer 40K added to create a unique AI-generated scene.

  • Which AI tool is used for clipping out reference footage?

    -Vigle is the AI tool used for clipping out reference footage.

  • What is the significance of the 2.0 update in Vigle?

    -The 2.0 update in Vigle improves the quality of the AI-generated content, including better handling of dancing and other movements, and provides a more refined output for the user's projects.

  • How does the video creator overcome the limitations of Vigle in handling camera movement?

    -The video creator suggests utilizing Leonardo for image-to-image references and Kyber for additional stylization and consistency in character appearance to overcome the limitations of Vigle's shaky and inconsistent camera movement handling.

  • What AI tool is used to add movement and life to the background?

    -Gen 2 is used to add movement and life to the background by applying simple commands like moving to the right, which helps create a dynamic and engaging visual effect.

  • How is the final composite of character and background achieved?

    -The final composite is achieved by bringing both the Kyber character and the Gen 2 background into a video editor like Premiere or DaVinci, using chroma key removal, and adjusting settings like choke, soften, and contrast to seamlessly blend the elements.

  • What tools are used for generating audio in the video?

    -AudioGen is used for generating crowd chanting as background sound, and typcast is used for text-to-speech conversion for dialogue, providing a complete audio-visual experience for the AI-generated film.

  • What is the creator's overall assessment of this AI film making workflow?

    -The creator believes that while the workflow is not perfect and may not be suitable for full feature films, it is quite effective for short films and pre-production stages, offering a more productive and innovative approach than traditional methods.

Outlines

00:00

🎬 AI Filmmaking Workflow Introduction

The speaker introduces an AI filmmaking workflow that has potential from pre-production to generating short films. They mention that the workflow is a kitbash of various tools and techniques, but the results are promising as seen in examples from friends of the channel. The inspiration comes from the 2016 film Rogue One, which featured the first major film with a fully deep faked character. The speaker plans to share their learnings and hopefully save time for those interested in trying out this workflow.

05:00

🌟 Utilizing AI for Hybrid Storyboard and Animation

The speaker discusses the idea of using AI to create a hybrid storyboard animatic animation. They reference a 2017 interview with the editor of Rogue One, Colin Ghoul, who talked about creating a feature-length story reel before the script was finished. The speaker then describes their attempt to recreate a scene from Gladiator using AI, incorporating elements from John Carter and Warhammer 40K. They explain the process of using reference footage, AI tools like Vigle for initial video generation, and Midjourney for character creation.

10:00

📸 Enhancing AI Generated Content with Additional Tools

The speaker details the process of refining the AI-generated content by using additional tools like Kyber for motion enhancement and Leonardo for background creation. They discuss the challenges of camera movement in AI videos and how using image references can improve the results. The speaker also talks about adding character depth and stylization, as well as the importance of background consistency in the final video output.

🎥 Post-Production and Audio Integration

In the final paragraph, the speaker talks about post-production processes such as compositing the character and background in a video editor like Premiere or DaVinci, using chroma key and color correction to integrate them seamlessly. They mention adding cinematic touches like black bars for a letterbox effect. For audio, the speaker describes using a free site called audiogen for crowd chanting and another free source called typcast for dialogue, using the Frankenstein model to generate a voiceover that fits their scene.

Mindmap

Keywords

💡AI film making workflow

The AI film making workflow refers to the process of using artificial intelligence tools to assist in the creation of films, from pre-production to the generation of short films. In the context of the video, this workflow is seen as a promising method with potential for innovation in the film industry, as it integrates various AI technologies to streamline and enhance the filmmaking process.

💡Pre-production

Pre-production is the initial phase of filmmaking that involves planning and preparation before the actual shooting begins. It is a critical stage where key creative and logistical decisions are made. In the video, the AI workflow is highlighted as particularly beneficial during pre-production for creating hybrid storyboard animatics and animations, which can help visualize the film before it is physically produced.

💡Deepfake

Deepfake refers to the use of artificial intelligence, specifically deep learning techniques, to create realistic but fake or altered images or videos of people, often used to manipulate or replace faces in existing footage. In the context of the video, the reference to the 2016 film Rogue One highlights the historical significance of deep fakes in major films, indicating a shift towards the use of AI in creating more realistic and complex characters or scenes.

💡Hybrid storyboard animatic

A hybrid storyboard animatic is a combination of traditional storyboarding and animated sequences that provide a more dynamic and fluid representation of a film's narrative. The use of AI in creating these animatics can enhance the pre-visualization process by generating animated scenes that give filmmakers a better sense of timing, pacing, and visual continuity.

💡Vigle

Vigle is an AI video editing tool mentioned in the video that assists in generating and editing video content. It is used for tasks such as dancing animations and other video manipulations. The 2.0 update of Vigle is highlighted as having improved capabilities, which are utilized in the AI film making workflow to create and refine the generated content.

💡Mid journey

Mid journey seems to be a reference to a process or tool used in the AI film making workflow for creating models for characters. It is used to generate images of characters in a specific format, such as full body or head to feet, which are then integrated into the AI-generated videos.

💡Leonardo

Leonardo is an AI tool or platform referenced in the video that is used for image manipulation and editing. It is utilized to enhance the quality of the AI-generated content, such as by using image-to-image references to improve the consistency and coherence of the characters and scenes.

💡Kyber

Kyber is an AI video generator mentioned in the video that offers unique capabilities for video creation. It is used to further refine the AI-generated footage by applying additional stylization and consistency to the character's appearance, as well as integrating the character with the background.

💡Chroma key

Chroma key, also known as green screen or blue screen technology, is a visual effects/post-production technique used to replace a specific color, typically a solid background, with another image or video. In the video, the speaker uses a chroma key remover in Premiere to composite the character onto the background, creating a seamless integration.

💡Audio generation

Audio generation refers to the process of creating or synthesizing sounds and music for a film or video. In the context of the video, the speaker uses AI tools to generate background sounds, such as crowd chanting, to enhance the realism and immersion of the AI-generated film.

💡Text-to-speech

Text-to-speech (TTS) is a technology that converts written text into spoken words, allowing for the creation of voiceovers or narrations without the need for a human voice actor. In the video, the speaker attempts to use TTS for dialogue but encounters challenges, ultimately finding an alternative source for the voiceover.

Highlights

The speaker shares an AI filmmaking workflow that covers pre-production to generating short films, showcasing its potential.

The inspiration for this workflow comes from the 2016 film Rogue One, which featured the first major film with a fully deep faked character.

Editor Colin Ghoul's 2017 interview discussed creating a feature-length story reel for Rogue One before the script was finished, using hundreds of movies.

The speaker's goal is to create a hybrid storyboard animatic animation using AI tools, taking the concept from Rogue One's pre-production.

The process begins by clipping out reference footage and using Vigle's 2.0 update for the initial generation of the video.

Vigle's 2.0 update includes dancing features, which can be used creatively, as demonstrated by a T800 T1000 dance-off.

The character model is created using Midjourney, focusing on a full-body image to ensure consistency.

Vigle's output can be improved by using Leonardo for image-to-image references, especially for fine-tuning character movements.

Kyber's new motion 3.0 feature is praised for its unique AI video generation capabilities.

The character's arm raise was fixed by using Leonardo with a low image strength and a specific prompt, resulting in a more coherent output.

Backgrounds are made dynamic and cohesive with the character by using Gen 2 and Kyber's stylization.

The final compositing is done in a video editor like Premiere, using chroma key and other adjustments for a polished result.

Crowd chanting audio was generated using the free site audiogen to enhance the short film's atmosphere.

Dialogue was created using typcast's Frankenstein model, which provided a more suitable result than other text-to-speech options.

The speaker concludes that while the method is not perfect for full feature films, it is useful for short films and pre-production work.

The workflow demonstrates the potential of kit bashing with AI tools to create engaging content quickly and efficiently.