How to Prompt for Text to Video | Runway Academy

Runway
5 Jan 202403:30

TLDRThis Runway Academy tutorial covers the fundamentals of text-to-video prompting in Gen 2. It guides viewers on crafting detailed prompts to control video output, including specific dog breeds, settings, and times of day. The video emphasizes the importance of focusing on 'what it looks like' and 'what it is', and demonstrates the use of style presets and prompt modifiers to refine the generation process, ultimately allowing for greater creative control and achieving the desired video style.

Takeaways

  • 🐾 Start with a simple prompt for text to video in Gen 2, like 'a dog'.
  • 🔍 Add specificity to your prompts by including details like dog breed, color, location, and time of day.
  • 🎨 As you add more detail, the images and videos generated become more reflective of those details.
  • 📸 Include details about the type of shot, such as 'extreme close-up' or 'shallow depth of field', for more control over output.
  • 🖼️ Understand the difference between 'what it looks like' and 'what it is' for clear prompting.
  • 🖌️ Experiment with style presets in Gen 2 to influence the style of your output.
  • 🔧 Use prompt modifiers to fine-tune control over the output, adding them as comma-separated values.
  • 🌆 Try combining prompts with modifiers for a stylized direction, such as 'a photo of a city skyline made out of paper' with modifiers like 'burnt texture'.
  • 🎭 Remember to focus prompts on appearance and essence, avoiding run-on sentences or contradictory modifiers.
  • 📚 Join the Runway Academy community on Discord for more resources and experimentation.
  • 🔄 Keep prompts focused and avoid unnecessary filler words or too many concepts in a single prompt.

Q & A

  • What is the main topic discussed in the Runway Academy video?

    -The main topic discussed in the video is the basics of prompting for text to video in Gen 2, including how to create specific and detailed prompts to control the output of images and videos.

  • How does the video guide viewers to start with text to video in Gen 2?

    -The video instructs viewers to navigate to Gen 2, click on 'text to video', and enter a simple prompt such as 'a dog', then click 'free preview' to see the starting images.

  • What is the purpose of adding specific details to the prompt?

    -Adding specific details to the prompt helps the images and resulting videos to reflect those details more meaningfully, giving the user more control over the outputs.

  • Can you give an example of a detailed prompt mentioned in the video?

    -An example of a detailed prompt given in the video is 'a light-colored French Bulldog laying on a couch during the day'.

  • What are the two main things to focus on when creating a prompt according to the video?

    -The two main things to focus on when creating a prompt are 'what it looks like' and 'what it is', which help in achieving a specific look and understanding the subject of the prompt.

  • What is the role of style presets in Gen 2 as mentioned in the video?

    -Style presets in Gen 2, found at the bottom of the prompt box, have a significant influence on the style of the output results and can be used even with the most simplified prompts.

  • How can prompt modifiers be used to gain further control over the output?

    -Prompt modifiers are descriptive words that can be added to the end of a prompt to help control the output further, and they can be used with style presets for even more control.

  • What is an example of a prompt modifier mentioned in the video?

    -Examples of prompt modifiers mentioned in the video include 'burnt texture', 'black and gray color palette', and 'flat composition'.

  • What advice does the video give for creating effective prompts?

    -The video advises to focus on what the output looks like and what it is, include relevant prompt modifiers, keep the prompt focused, and avoid run-on sentences, unnecessary filler words, too many concepts in a single prompt, or contradictory prompt modifiers.

  • How can viewers find more helpful resources and join the community for further learning?

    -Viewers can join the community on Discord for more information and experimentation using Runway, or they can find specific answers to their questions by using the 'help' button on their dashboard.

  • What is the final piece of advice given in the video for using Gen 2 effectively?

    -The final advice is to remember the formula and rules discussed in the video to take more control over Gen 2 and start creating in the specific style desired.

Outlines

00:00

📝 Introduction to Text-to-Video Prompting in Gen 2

The video script begins with an introduction to Runway Academy's tutorial on the basics of creating prompts for text-to-video in Gen 2. It guides viewers on how to navigate the platform, enter simple prompts, and utilize the 'free preview' feature to generate initial images. The script emphasizes the importance of specificity in prompts to refine the output, including details such as dog breed, location, and time of day. It also introduces the concept of adding descriptive details to achieve a particular look and style in the generated videos.

🔍 Enhancing Prompts for Detailed Video Outputs

This paragraph delves deeper into the art of prompting, illustrating how to enhance prompts with more specific details such as the type of shot, lighting conditions, and the setting to gain better control over the video output. The script uses the example of a light-colored French Bulldog on a couch to demonstrate how adding details like 'extreme close-up' and 'shallow depth of field' can influence the final result. It also touches on the importance of distinguishing between the visual appearance ('what it looks like') and the subject matter ('what it is') in crafting effective prompts.

🎨 Utilizing Style Presets and Prompt Modifiers

The script introduces the use of style presets and prompt modifiers in Gen 2 to achieve a desired style or to further control the output. Style presets are located at the bottom of the prompt box and can significantly influence the style of the generated content. Prompt modifiers are descriptive words that can be added to the end of a prompt to refine the output, such as 'burnt texture,' 'black and gray color palette,' and 'flat composition.' The paragraph provides an example of how to use these modifiers with a prompt to create a stylized city skyline made out of paper.

🛠️ Mastering Prompting Techniques for Creative Control

The final paragraph of the script offers guidance on mastering the prompting process. It advises viewers to focus on the visual appearance and the essence of the subject, to include relevant prompt modifiers, and to keep prompts focused and clear. The script warns against using run-on sentences, unnecessary filler words, too many concepts in a single prompt, or contradictory prompt modifiers. It concludes by encouraging viewers to join the community on Discord for further resources and experimentation, and to use the dashboard button for specific questions.

Mindmap

Keywords

💡Prompting

Prompting in the context of text to video refers to the process of providing specific instructions or cues to a system in order to generate content. In the video, it is the method used to guide Gen 2 in creating videos from textual descriptions. The script mentions 'prompting' as a way to control the output by specifying details like the type of dog, its color, and the setting.

💡Gen 2

Gen 2 is likely a reference to a second-generation system or software for creating videos from text prompts. The script describes navigating to Gen 2 and using its 'text to video' feature, indicating that it is a tool central to the video's theme of content creation.

💡Text to Video

Text to video is a technology that converts written text prompts into visual content. The video's transcript discusses the basics of prompting for this technology, showing how adding details to the text input can influence the resulting video output.

💡Free Preview

In the script, 'free preview' is a feature that allows users to see the starting images generated by the system based on their text prompt before deciding to create a video. It is an example of how users can assess and refine their prompts.

💡Specificity

Specificity in prompting refers to the level of detail included in the text prompts to guide the video generation process. The video emphasizes the importance of being specific, such as mentioning the breed and color of a dog, to achieve more accurate and controlled video outputs.

💡Type of Shot

The type of shot refers to the camera angle or perspective used in a video or photograph. The script gives an example of an 'extreme close-up shallow depth of field portrait photo' to illustrate how specifying the shot type can influence the video's visual style.

💡Style Presets

Style presets are pre-defined settings in Gen 2 that influence the visual style of the generated videos. The transcript mentions that these presets can significantly affect the output's style and can be experimented with to achieve a desired look.

💡Prompt Modifiers

Prompt modifiers are additional descriptive words that can be added to a text prompt to further control the output of the video generation. The script provides examples such as 'burnt texture,' 'black and gray color palette,' and 'flat composition' to demonstrate how these modifiers can guide the style and direction of the generated content.

💡Control

Control in this context means the ability to influence and direct the outcome of the video generation process through precise prompting. The video's theme revolves around gaining more control over outputs by using detailed prompts and modifiers.

💡Run-on Sentences

Run-on sentences are a grammatical error where two or more independent clauses are incorrectly joined without proper punctuation. The script advises against using them in prompts to maintain clarity and focus, which is crucial for effective video generation.

💡Discord Community

The Discord community mentioned in the script is a platform where users can join for more resources and experimentation with Runway or to find specific answers to their questions. It represents a supportive network for users engaging with the text-to-video technology.

Highlights

Introduction to the basics of prompting for text to video in Gen 2.

Navigate to Gen 2 and click on text to video to start.

Begin with a simple prompt like 'a dog' to generate initial images.

Use free preview to see starting images and select for video conversion.

Adding specificity to prompts improves image and video detail.

Include dog breed, color, location, and time of day for more detailed prompts.

Example prompt: 'a light-colored French Bulldog laying on a couch during the day'.

Details about shot type, such as extreme close-up or portrait, can be included.

The more information added, the more control over the output.

Focus on 'what it looks like' and 'what it is' for effective prompting.

Example of contrasting prompts: 'a painting' vs. 'a bowl of fruit'.

Adding detail to prompts can lead to a specific look, such as a textured historical oil painting.

Style presets in Gen 2 can significantly influence the style of output results.

Experiment with style presets even with simplified prompts.

Prompt modifiers can be used to gain further control over outputs.

Modifiers are descriptive words added to the end of a prompt.

Example prompt with modifiers: 'a photo of a city skyline made out of paper'.

Modifiers can be combined with style presets for more control.

When prompting, avoid run-on sentences and contradictory modifiers.

Join the Runway Academy community on Discord for more resources and support.