NEW Open Source AI Video is BEST Yet! (Multiple Consistent Characters + More)
TLDRThe new open-source AI video model, Story Diffusion, is making waves in the world of AI-generated content. It has been hailed as a significant step forward in character consistency, producing videos up to 30 seconds long with a high level of detail and realism. Story Diffusion excels at maintaining character consistency not just in facial features, but also in clothing and body type, allowing for the creation of believable characters across different shots and scenes. This advancement opens up new possibilities for AI video and comic generation. The model has also demonstrated the ability to handle longer video clips with less computational power compared to its predecessors, such as Sora, which required 1250 times more compute power. Despite minor imperfections, such as occasional jitteriness and morphing, the tool shows great promise for the future of AI in animation and storytelling, offering a training-free method for generating consistent images that transition smoothly into fluid and natural-looking videos.
Takeaways
- ๐ฌ **Story Diffusion** is an open-source AI video model that produces high-quality, character-consistent videos up to 30 seconds long.
- ๐ It demonstrates significant advancements in character consistency, including facial features, clothing, and body type, which is crucial for believable character creation.
- ๐ Story Diffusion has made strides in reality adherence and physics, correcting issues like objects passing through each other, which were prevalent in previous models.
- ๐ The model uses **consistent self-attention** to ensure visual coherence in a sequence of images, maintaining character attributes across frames.
- ๐ It employs **story splitting**, breaking down narratives into text prompts that are processed simultaneously to generate a sequence of images that tell a story.
- ๐ค The AI uses a **motion prediction model** to animate images, creating fluid transitions that mimic real-life motion.
- ๐ Videos generated by Story Diffusion show a high level of expressiveness and emotion in characters, a notable improvement over previous models.
- ๐น The model has produced videos with less 'jitteriness' and more consistent character representation compared to its predecessors.
- ๐ฑ Despite being square format, the videos are of high resolution and could potentially be upscaled to 2K definition using AI technology.
- ๐ป It can handle diverse scenes, including realistic and animated content, with a high degree of natural movement and expression.
- ๐ While the model shows great promise, there are minor inconsistencies in animations, particularly when objects are occluded.
- ๐ป Story Diffusion was trained using significantly fewer computational resources than models like Sora, making it more accessible and cost-effective to use and develop.
Q & A
What is the name of the new open-source AI video model discussed in the transcript?
-The new open-source AI video model discussed is called 'Story Diffusion'.
What are the key features of Story Diffusion that make it stand out from other models?
-Story Diffusion stands out due to its ability to create videos up to 30 seconds long with high character consistency, adherence to reality and physics, and its capability to understand reality on a deeper level.
How does Story Diffusion handle character consistency in comparison to previous models?
-Story Diffusion handles character consistency by not only focusing on facial consistency but also on clothing and body type, allowing for believable characters that maintain perfect consistency between shots and scenes.
What is the significance of Story Diffusion's ability to generate AI Comics?
-The ability to generate AI Comics signifies a broader application of the model beyond video generation, allowing for the creation of a series of images that are consistent in terms of face and clothing, and then animated using the motion prediction model.
How does the length of the clips generated by Story Diffusion compare to previous models?
-The clips generated by Story Diffusion are significantly longer than those of previous models, with the ability to produce videos up to 23 seconds long while maintaining character consistency.
What are the limitations of Story Diffusion in terms of video resolution and interface?
-There is no information on the resolution of the videos generated by Story Diffusion, but previews are rendered at 832 pixels by 832. Additionally, it does not have a usable interface and requires self-downloading, installing, or running on a cloud server.
How does the computational power used to train Story Diffusion compare to Sora?
-Story Diffusion used eight GPUs for training, whereas Sora used 10,000, which is 1250 times more computational power.
What is the significance of Story Diffusion's consistent self-attention tool?
-Consistent self-attention enhances the consistency of different generated images by ensuring each one shares certain attributes or themes, making them visually coherent when viewed as a series.
How does Story Diffusion approach the creation of a narrative in its generated videos?
-Story Diffusion uses a method called 'story splitting', where a story is broken down into multiple text prompts, each describing a part of the story. These prompts are processed simultaneously to produce images that depict the narrative in sequence.
What are the potential applications of Story Diffusion's technology?
-Story Diffusion's technology can be applied to create realistic and cohesive AI-generated videos, animations, and comics, potentially enabling the creation of full films with AI for animated and anime-style genres.
What are some of the minor issues observed in the Story Diffusion's generated videos?
-Minor issues include slight jitteriness in the videos and the fact that all videos have been square. There are also moments where the hand animation does not appear natural, particularly when an object is obscured by something else.
Outlines
๐ฌ Introduction to Story Diffusion: AI Video Model
The first paragraph introduces 'Story Diffusion,' an open-source AI video model that outperforms others in character consistency and realism. It is capable of creating videos up to 30 seconds long with minimal morphing or disfigurement. The model is praised for its ability to maintain consistency in clothing, body type, and facial features across different shots and scenes. The paragraph also mentions the potential for generating AI comics using the model, showcasing a comic strip as an example. The video demo included demonstrates the model's ability to animate characters with lifelike movements and expressions.
๐ Story Diffusion's Technical Achievements
The second paragraph delves into the technical aspects of Story Diffusion, highlighting its efficiency in training with only eight GPUs compared to Sora's 10,000, which is 1250 times more compute power. The model is noted for its consistency in rendering multiple characters across scenes, a significant improvement over previous AI video generators. The paragraph also discusses the model's application in comic generation, showing how it can maintain character consistency from scene to scene. Despite some minor inconsistencies, the overall performance is considered impressive.
๐ค Story Diffusion's Approach to Consistency and Storytelling
The third paragraph explains the innovative techniques used by Story Diffusion to ensure consistency and coherence in its generated content. It discusses the use of consistent self-attention to maintain visual coherence across a series of images and story splitting, which breaks down a story into multiple text prompts processed simultaneously. The paragraph also describes how the model uses a motion predictor model to animate the generated images, creating fluid and natural-looking transitions between frames. The effectiveness of these techniques is demonstrated through various examples, including animations and realistic video scenes.
๐ Story Diffusion's Impact on AI Video and Future Prospects
The final paragraph wraps up the discussion on Story Diffusion, emphasizing its significant advancements in AI video generation. It invites viewers to explore another AI video model, Vidu, and encourages feedback on how Story Diffusion compares to existing models. The paragraph concludes with a hopeful note on the future of AI video, suggesting that high-quality, realistic videos are becoming increasingly accessible.
Mindmap
Keywords
๐กOpen Source AI Video Model
๐กCharacter Consistency
๐กReality and Physics Adherence
๐กAI Comics
๐กMotion Prediction Model
๐กResolution
๐กConsistent Self-Attention
๐กStory Splitting
๐กAnimation
๐กTraining-Free Generation
๐กAI Video Generation
Highlights
Story diffusion is an open-source AI video model that creates videos up to 30 seconds long with high character consistency and adherence to reality and physics.
The model has significantly improved character consistency, including facial, clothing, and body type, allowing for believable and consistent characters across shots and scenes.
Story diffusion can generate AI Comics by creating a series of consistent images for a sequence and animating them using a motion prediction model.
Videos generated by story diffusion feature minimal morphing or disfigurement, with anatomically correct characters and impressive expressiveness.
The model produces clips as long as 23 seconds, maintaining character consistency throughout the entire video.
Story diffusion has a slight jitteriness and all videos are currently square, but these are minor compared to the improvements in consistency and character clarity.
The model has been trained on only eight GPUs, compared to Sora's 10,000, making it significantly more efficient in terms of computational power.
Story diffusion is capable of including multiple characters consistently in scenes, overcoming a major challenge in AI video generation.
The model uses consistent self-attention to ensure visual coherence between generated images by sharing certain attributes or themes.
Story splitting is another technique used, breaking down a story into multiple text prompts processed simultaneously for sequenced image generation.
The motion predictor model is used to animate between two images, predicting the sequence of movements for a natural and fluid animation.
Story diffusion can create effective and usable anime-style animations, opening up possibilities for full films generated by AI.
The model handles a diverse range of scenes, including realistic camera shake and accurate animation of elements within the scene.
Story diffusion offers a novel method for generating consistent images in a training-free manner, suitable for storytelling and transitioning into videos.
The generated videos from story diffusion maintain continuity in appearance and motion, giving the impression of real-life captured scenes.
AI video generation is rapidly advancing, with story diffusion showing significant evolution in character consistency and the ability to create realistic and cohesive scenes.
The model's efficiency and open-source nature make it accessible for a wide range of applications, despite the current lack of a user-friendly interface.