Building a Modern, Error-Resistant API from 0 to Production (2024)

Josh tried upstash
13 May 202464:35

TLDRThe video tutorial presents a modern approach to building a robust and error-resistant API using Next.js. The host illustrates common issues with APIs, such as the inefficiency of rerunning successful operations when a failure occurs and the limitations of serverless environments like Vercel, which may time out long-running functions. To address these, the video introduces a solution that leverages a workflow system, allowing for the division of tasks into separate HTTP functions that can be retried independently upon failure. This method not only optimizes efficiency by avoiding redundant calculations but also extends the runtime by breaking down the process into steps, each with its own timeout limit. The implementation is fully type-safe, utilizing TypeScript to ensure data integrity across steps. The workflow is orchestrated using an HTTP-based messaging and scheduling tool called Qstash, which handles retries and scheduling. The tutorial demonstrates how to set up this workflow, from defining API routes in Next.js to integrating with Qstash for automatic retries and robust error handling. The result is a scalable and resilient API design that simplifies complex operations and enhances software reliability.

Takeaways

  • 🚀 **Modular API Design**: The video introduces building an API that addresses two main issues: the inefficiency of rerunning successful operations after a failure, and time constraints in serverless environments like Vercel.
  • 🔁 **Automatic Retry Mechanism**: The solution involves breaking down the API into smaller steps, each of which can be retried independently upon failure without repeating successful prior steps.
  • ⏱️ **Overcoming Timeout Limits**: By splitting the workload into separate HTTP functions, the system can bypass the 10-second execution limit often found in serverless environments, significantly increasing the total runtime.
  • 📚 **Type Safety with TypeScript**: The API leverages TypeScript for type safety, ensuring that data returned from one step is correctly typed and accessible in subsequent steps.
  • 💡 **Workflow Abstraction**: The workflow is abstracted on top of Next.js Route handlers, providing a clean and efficient way to manage complex processes without worrying about timeouts.
  • 🧩 **Composability**: The API is designed to be composable, allowing developers to chain multiple steps together, each running independently and capable of retrying in case of failure.
  • 🔍 **Under-the-Hood Insights**: The presenter shares insights into the internal workings of the API, including the use of a separate core.ts file for the workflow logic and the integration with Qstash for task scheduling and retries.
  • 🔗 **Qstash Integration**: Qstash is used as an HTTP-based messaging and scheduling solution that handles background jobs, providing features like automatic retries and parallelism.
  • 🛠️ **Class-Based Approach**: The workflow engine is implemented using a class-based approach in TypeScript, which provides a structured and scalable way to manage the steps and their execution.
  • 📈 **Incremental Step Execution**: The API executes steps incrementally, only proceeding to the next step after the current one completes successfully, optimizing performance and resource usage.
  • 🌐 **Deployment to Vercel**: The video demonstrates deploying the API to Vercel, showcasing how the system works in a production environment and interacts with Qstash for robust error handling and execution.

Q & A

  • What are the two main problems the video aims to solve when working with APIs?

    -The video aims to solve the problem of inefficient retries when an API call fails after a successful previous calculation, and the issue of time limitations in serverless environments that can cause functions to fail if they exceed the maximum runtime.

  • How does splitting API tasks into separate HTTP functions help with efficiency?

    -Splitting API tasks into separate HTTP functions allows each function to run independently, which means that if one step fails, only that step needs to be rerun, not the entire process. This approach saves time and resources by avoiding the repetition of successful calculations.

  • What is the maximum runtime limit for functions in serverless environments like Vercel, and how does the proposed solution work around this?

    -In serverless environments like Vercel, there is a maximum runtime limit of 10 seconds for functions. The proposed solution works around this by splitting the workload into multiple steps, each of which can run for up to 10 seconds, effectively allowing for an almost infinite runtime by chaining steps together.

  • How is the workflow engine made robust and type-safe using TypeScript?

    -The workflow engine is made robust by automatically passing the results of each step to the next with retries and automatic rerunning of failed steps. It is made type-safe by using TypeScript's typing system to ensure that the data returned from one step is correctly typed and accessible in the subsequent step.

  • What is Qstash and how does it support the workflow?

    -Qstash is an HTTP-based messaging and scheduling solution designed for handling background jobs and schedules. It supports the workflow by managing separate HTTP functions as tasks, allowing for automatic retries and scheduling, which is crucial for steps that may exceed the serverless function timeout.

  • How does the video demonstrate the implementation of the robust API workflow?

    -The video demonstrates the implementation by outlining the creation of a workflow class in TypeScript, defining steps as separate functions, and showing how to use Qstash for scheduling and retrying steps. It also covers how to handle the workflow under the hood, including how to pass data between steps and ensure type safety.

  • What is the benefit of using a class-based approach for the workflow implementation?

    -The class-based approach provides a structured and object-oriented way to manage the workflow. It allows for encapsulation of the workflow logic and state, making it easier to maintain and scale. It also aligns well with the concept of creating reusable and maintainable software components.

  • How does the workflow engine handle the scenario where a step fails and needs to be retried?

    -The workflow engine uses Qstash to automatically retry the failed step with an exponential back-off strategy. This means that only the failed step is retried, not the entire workflow, and the results from the successful steps are preserved.

  • What is the significance of having a fully type-safe API workflow?

    -A fully type-safe API workflow ensures that the data passed between steps is correctly typed, reducing the chance of runtime errors and making the code more maintainable and understandable. It also allows developers to access the results from one step in the next step with confidence, knowing the data types are correct.

  • How does the video address the issue of API route handling in the context of the workflow?

    -The video addresses API route handling by showing how to define a POST method within the workflow class that will handle the incoming HTTP requests. It also demonstrates how to use the workflow engine to process these requests in a step-by-step manner, ensuring that each step's logic is executed in order.

  • What are the steps involved in implementing the workflow from scratch as shown in the video?

    -The steps include: defining the workflow class, creating a method to initialize the workflow, setting up a callback function for defining steps, implementing the steps with type safety using TypeScript generics, handling the runtime execution of each step, and integrating with Qstash for scheduling and retrying steps.

Outlines

00:00

🚀 Introduction to API Workflow Tool

The video introduces a tool designed to address two significant challenges when working with APIs: inefficient retry handling after partial failures and time limitations in serverless environments. The tool, which separates API tasks into individual steps, allows for each step to be retried independently upon failure without repeating successful ones. It also enables splitting tasks to run within the time constraints of serverless functions, such as the 10-second limit on Vercel's free plan.

05:02

📚 Understanding the API Workflow Concept

The script explains how the API workflow operates under the hood using Next.js API routes and a custom workflow class. It details how to export a POST method to handle requests and create a workflow with steps as separate HTTP functions. The workflow engine uses Qstash, an HTTP-based messaging and scheduling solution, to manage background jobs, retries, and ensure type safety across steps.

10:03

🛠️ Implementing the Workflow Class

The video outlines the process of implementing the workflow class, which includes defining a 'createWorkflow' function that expects a callback function outlining the steps. It also discusses the creation of an interface called 'step' to ensure type safety and the use of generics to define the input and output types for each step.

15:05

🔁 Handling Step Execution and Retries

The script describes how to execute the defined steps and handle retries. It explains the use of Qstash for automatic retries with exponential backoff when a step fails. The process involves publishing messages to Qstash, which then calls the appropriate step in the workflow. If a step fails, Qstash retries it while preserving the result of the successful steps.

20:06

🧩 Assembling the Workflow Steps

The video demonstrates how to chain the 'create' function to the initial step and create subsequent steps. It emphasizes the importance of type safety, where the return value of one step becomes the input for the next. The script also shows how to handle the HTTP route invocation to determine which step to execute based on the step parameter in the URL.

25:08

🌐 Deploying and Testing the Workflow

The script concludes with deploying the workflow to Vercel and testing its functionality. It shows how to set up a GitHub repository, push the project files, and use Vercel to deploy the application. The video also includes an example of how to simulate errors to demonstrate the automatic retry mechanism provided by Qstash.

Mindmap

Keywords

💡API

An API, or Application Programming Interface, is a set of rules and protocols that allows different software applications to communicate with each other. In the context of the video, the API is central to building a tool that performs calculations, handles database operations, and manages retries and timeouts, which are common challenges in serverless environments.

💡Serverless

Serverless computing is a cloud-computing execution model where the cloud provider dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. The video discusses how the API tool can overcome limitations like maximum function run times, which are common in serverless environments like AWS Lambda or Vercel.

💡Workflow

A workflow in the context of the video refers to a sequence of steps or processes that must be completed to achieve a certain outcome. The tool being built in the video automates these steps, allowing for complex operations to be broken down into smaller, more manageable pieces that can be executed sequentially.

💡TypeScript

TypeScript is an open-source programming language developed and maintained by Microsoft. It is a strict syntactical superset of JavaScript, and adds static typing to the language. The video emphasizes the use of TypeScript to ensure type safety in the API, which helps in catching errors early in the development process and provides better tooling support.

💡Retries

Retries are attempts to execute a function or operation again after it has failed, in order to achieve a successful outcome. The video discusses implementing a retry strategy for steps in the API workflow that may fail, which is crucial for creating a robust and reliable system.

💡Exponential Backoff

Exponential backoff is a technique used in API requests where the wait time between retries increases exponentially. This approach is beneficial in the video's context to avoid overwhelming the system with rapid retries and to gradually increase the wait time between each retry attempt.

💡HTTP Functions

HTTP functions refer to individual units of functionality within an API that respond to HTTP requests. The video script describes each step of the workflow as a separate HTTP function, allowing each part of the process to run independently and potentially in parallel, which can improve efficiency and manage long-running tasks.

💡Vercel

Vercel is a cloud platform that enables developers to host and deploy web services and applications. It is mentioned in the video as an example of a serverless environment where the API tool can be deployed, taking advantage of Vercel's scalability and integration with Next.js.

💡Next.js

Next.js is an open-source JavaScript framework for building user interfaces and APIs. It is built on top of Node.js and is used in the video to create the API routes and handle server-side logic. Next.js is known for its ease of use and ability to build server-rendered React applications.

💡Qovery

Qovery is a tool mentioned in the video for managing database schemas, migrations, and secrets. It is used to handle database operations within the API workflow, ensuring that the results of calculations are saved reliably, even when the workflow involves multiple steps that can fail and require retries.

💡Environment Variables

Environment variables are a set of dynamic values that can be used by an application to adjust its behavior without the need to modify the application's code. In the video, environment variables such as the Qovery token are used to securely manage and use sensitive information like API keys and authentication tokens.

Highlights

Building a modern, error-resistant API from scratch to production in 2024.

Addressing two key problems when working with APIs: efficient error handling and serverless environment time limits.

Demonstrating how to avoid repeating successful calculations when a subsequent step fails.

Introducing a solution that splits workload into separate HTTP functions, overcoming the 10-second limit in serverless functions.

Automating retries for failed steps with an exponential back-off strategy.

Implementing a fully type-safe workflow using TypeScript for robust API calls.

Creating a workflow engine on top of Next.js Route handlers for efficiency and timeout avoidance.

Utilizing Qstash, an HTTP-based messaging and scheduling solution, for task management.

Designing a system where only failed steps are rerun, optimizing performance and resource usage.

Exploring the internal workings of the API, including the creation of a workflow class and step interface.

Ensuring type safety by defining generics for inputs and outputs across steps.

Developing an API route that chains multiple steps together, allowing for complex operations to be executed in stages.

Incorporating Qstash for automatic retries and handling of background jobs within the workflow.

Implementing a strategy to handle long-running tasks by breaking them into smaller, manageable steps.

Deploying the API to Vercel and leveraging its environment variables for token-based authentication with Qstash.

Creating a local development setup that mimics production behavior using the Fetch API.

Logging and monitoring the progress of each step in the workflow for debugging and oversight purposes.

Successfully executing a multi-step API workflow that exceeds the standard serverless function timeout limits.