I Built a CoPilot+ AI PC (without Windows)

Jeff Geerling
4 Jun 202412:49

TLDRThe video showcases the creation of a custom CoPilot AI PC using a Raspberry Pi and various neural processing units (NPUs). The CoPilot PC, equipped with a Hailo NPU offering 13 TOPS of compute, is designed for real-time video processing tasks such as object detection and pose estimation. The creator explores the potential of combining multiple AI accelerators to surpass the computational power of Microsoft's Copilot, Apple's M4, and Snapdragon X, demonstrating the capabilities and limitations of the Raspberry Pi AI kit in machine vision and robotics applications.

Takeaways

  • 🤖 The CoPilot PC is a custom-built AI PC using a Raspberry Pi, not to be confused with Microsoft's Copilot.
  • 🚀 The CoPilot PC aims to achieve over 40 TOPS of neural compute, which is more than Apple's M4 or Snapdragon X, starting with 13 TOPS and eventually reaching 51 TOPS.
  • 💻 A $70 Raspberry Pi AI kit is used, which includes an M.2 HAT and a 13 TOPS Hailo NPU to enhance the Pi's capabilities.
  • 🔍 The system is designed for practical machine learning applications such as object detection, pose estimation, and image segmentation.
  • 🌟 Raspberry Pi is focusing on real-time video applications, which can be beneficial for various real-world uses like monitoring and quality assurance.
  • 🛠️ The AI kit is a PCI Express device that can be used to accelerate machine learning tasks on the Pi and potentially other computers.
  • 🔬 The video demonstrates the performance difference between running an AI model on the Pi's CPU versus using the Hailo NPU, with the latter being significantly faster and more efficient.
  • 📈 The script discusses the trade-offs between using custom NPUs, GPUs, and CPUs for AI tasks, highlighting the suitability of the Raspberry Pi setup for low-power edge devices.
  • 🔌 The AI kit has limitations, such as not offering much RAM, which restricts the complexity of the models that can be run and the impracticality for training models on the Pi.
  • 🛡️ There are alternative approaches to integrating AI capabilities, such as baking NPUs into main chips or using AI extensions on the CPU, each with its benefits.
  • 🔬 The video concludes with an ambitious attempt to connect multiple AI accelerators to the Raspberry Pi, exploring the limits of the system's capabilities.

Q & A

  • What is the main purpose of the CoPilot PC built in the video?

    -The CoPilot PC is a custom Raspberry Pi AI PC designed for machine learning tasks, particularly for machine vision applications like object detection, pose estimation, and image segmentation.

  • Why is the CoPilot PC different from Microsoft's Copilot?

    -The CoPilot PC is a Raspberry Pi AI PC, not related to Microsoft's Copilot. The creator built it because they had reservations about Microsoft's marketing approach to their product.

  • What is the neural compute requirement for Microsoft's Copilot?

    -Microsoft states that you need at least 40 TOPS of neural compute for Copilot.

  • What is included in the $70 Raspberry Pi AI kit?

    -The $70 Raspberry Pi AI kit includes an M.2 HAT and a 13 TOPS Hailo NPU, which functions as a machine learning accelerator for the Raspberry Pi.

  • How does the Hailo-8L NPU compare to the Coral TPU in terms of performance and efficiency?

    -The Hailo-8L NPU operates at 13 TOPS with 8 TOPS per watt efficiency, making it a more powerful and efficient machine learning accelerator compared to the Coral TPU, which offers 2 TOPS at 2 TOPS per watt.

  • What are some practical applications of the Raspberry Pi AI kit mentioned in the video?

    -The Raspberry Pi AI kit can be used for real-time video processing in applications such as security surveillance, traffic planning, factory production line monitoring, and agricultural spoilage detection.

  • Why might Raspberry Pi not integrate an NPU into their main chip like some other companies?

    -Raspberry Pi does not build their own main chips; they rely on Broadcom. The decision to integrate an NPU would be up to Broadcom or other chip manufacturers.

  • What are some limitations of using the Raspberry Pi AI kit for AI tasks?

    -The AI kit has constraints such as limited RAM, making it unsuitable for running large models or training AI models, which can be very power-intensive and time-consuming.

  • How does the performance of the Hailo NPU compare when running YOLOv5 on the Raspberry Pi without and with the AI kit?

    -Running YOLOv5 on the Hailo NPU with the AI kit is significantly faster and more efficient than running it on the Raspberry Pi's CPU alone, with real-time object detection and minimal CPU usage.

  • What is the purpose of the HatBrick! Commander and HatDrive! Top in the video?

    -The HatBrick! Commander and HatDrive! Top are Pineboards products used to expand the Raspberry Pi's PCI Express capabilities, allowing for the connection of multiple AI accelerators to increase the neural compute power.

  • What is the final neural compute capability achieved with the additional PCI Express devices in the video?

    -The final neural compute capability achieved is 47 TOPS [actually 51], although it was not fully usable due to potential power and configuration issues.

Outlines

00:00

🤖 Custom CoPilot PC with Raspberry Pi AI Kit

The script introduces a custom CoPilot PC, a Raspberry Pi AI kit that includes an M.2 HAT and a 13 TOPS Hailo NPU, which functions as a machine learning accelerator. The creator expresses skepticism about Microsoft's marketing of AI but is excited about the practical applications of AI in machine vision. The kit is compared to other technologies like Apple's M4 and Snapdragon X, and its efficiency and potential uses in real-world problems such as traffic planning and factory monitoring are highlighted. The video also discusses the limitations of integrating AI into the Raspberry Pi's main chip and explores different approaches to AI hardware, including the use of GPUs versus NPUs.

05:04

🔍 Exploring AI Capabilities and Limitations with Raspberry Pi

This paragraph delves into the capabilities of the Raspberry Pi when equipped with the AI kit, showcasing its ability to perform tasks like real-time object detection, pose estimation, and image segmentation. It discusses the potential for gesture-based applications, behavior prediction in cars, and iPhone-like portrait modes. The paragraph also touches on the limitations of the AI kit, such as its constrained RAM and the impracticality for tasks like image generation or running large language models. The script mentions the 'model zoo' provided by Hailo for pre-built models and the first-party support from Raspberry Pi, indicating the potential for future integrations in various projects.

10:09

🚀 Pushing the Limits with Multiple AI Coprocessors

The script describes an ambitious attempt to surpass Microsoft's neural compute requirements by connecting multiple AI coprocessors to the Raspberry Pi, including the Hailo 8L and various Coral TPUs. The process involves using Pineboard's HatBrick! Commander and other PCI Express devices to maximize the neural compute power. However, the experiment faces challenges due to power limitations and potential bugs with multiple PCI Express switches. Despite the unsuccessful attempt to boot all the devices, the script reflects on the potential of the AI kit and the evolving landscape of AI hardware, hinting at future developments in AI and robotics.

Mindmap

Keywords

💡CoPilot PC

CoPilot PC refers to a custom-built personal computer that is designed to function similarly to Microsoft's Copilot, but with a unique twist. In the video, the creator has built a Co-Pi-lot, a play on words using 'Pi' to refer to the Raspberry Pi, a popular single-board computer. This custom PC is not associated with Microsoft's AI software but is instead a Raspberry Pi AI kit with an attached neural processing unit (NPU) for machine learning tasks.

💡TOPS

TOPS stands for trillion operations per second and is a unit of measurement for the performance of neural processing units (NPUs) or GPUs in processing machine learning tasks. In the script, the creator mentions that Microsoft suggests at least 40 TOPS for their Copilot, while the custom CoPilot PC is capable of 13 TOPS, and later attempts to increase this to 47 TOPS [or 51], which is a significant computational power for AI tasks.

💡Raspberry Pi

Raspberry Pi is a series of small single-board computers that have gained popularity for their versatility and affordability. In the context of the video, the creator uses a Raspberry Pi as the base for their custom CoPilot PC, highlighting its capability to be augmented with additional hardware like the M.2 HAT and Hailo NPU for enhanced AI capabilities.

💡M.2 HAT

M.2 HAT refers to an add-on board for the Raspberry Pi that provides an M.2 slot, allowing for the connection of devices such as SSDs or, in this case, the Hailo NPU. The script mentions the $70 Raspberry Pi AI kit, which includes an M.2 HAT, enabling the Raspberry Pi to have dedicated AI processing power.

💡Hailo NPU

Hailo NPU is a neural processing unit designed for accelerating machine learning tasks. The script describes the Hailo-8L NPU as a 'little machine learning jetpack' for the Raspberry Pi, capable of 13 TOPS and 8 TOPS per watt efficiency, significantly boosting the AI capabilities of the custom CoPilot PC.

💡Coral TPU

Coral TPU is a tensor processing unit developed by Google, designed to accelerate machine learning workloads. In the video, the creator has previously used a Coral TPU for their Frigate server, highlighting its use in detecting objects like cars and people, and comparing its performance to that of the Hailo NPU.

💡Machine Learning

Machine learning is a subset of artificial intelligence that enables computers to learn and improve from experience without being explicitly programmed. The video's theme revolves around using machine learning for various practical applications, such as object detection and pose estimation, by leveraging the computational power of the custom CoPilot PC.

💡Object Detection

Object detection is a machine learning task that involves identifying and locating objects in images or video frames. The video script discusses the use of the Raspberry Pi AI kit for object detection, such as in surveillance or quality assurance in manufacturing settings.

💡Pose Estimation

Pose estimation is the process of determining the position and orientation of a body or body parts within an image or video. The script mentions the potential of the CoPilot PC to perform real-time pose estimation, which could be used for applications like gesture-based interfaces or predicting pedestrian behavior for autonomous vehicles.

💡Image Segmentation

Image segmentation is the process of partitioning an image into segments or regions to simplify analysis or enhance processing. The video describes how the custom CoPilot PC can perform image segmentation for tasks such as creating portrait modes on smartphones or applying unique lighting effects in real-time.

💡AI Coprocessor

An AI coprocessor is a specialized hardware component that offloads AI-related tasks from the main CPU, allowing for more efficient processing of machine learning models. The video script discusses the benefits of having a dedicated AI coprocessor like the Hailo NPU in the CoPilot PC for tasks that require real-time processing with lower power consumption.

Highlights

Introduction of the custom CoPilot PC, built with a Raspberry Pi and an AI kit instead of using Microsoft's Copilot.

Explanation of the Hailo-8L NPU, which provides 13 TOPS and is a part of the $70 Raspberry Pi AI kit.

Comparison of different AI hardware, including the Coral TPU and the Hailo-8L, with a focus on performance and power efficiency.

Discussion on the practical applications of AI, such as machine vision, object detection, and pose estimation.

Demonstration of YOLOv5 object identification model running on the Raspberry Pi with and without the AI kit, highlighting performance differences.

Detailed look at setting up multiple NPUs on a Raspberry Pi to achieve higher TOPS for neural compute, despite power limitations.

Exploration of the AI kit’s potential use cases in real-world scenarios like traffic monitoring, factory QA, and agricultural monitoring.

Mention of Raspberry Pi's approach compared to other companies like Apple, Qualcomm, and Rockchip in integrating NPUs.

Discussion on the constraints of the AI kit, such as limited RAM and the impracticality of training models on the Raspberry Pi.

Introduction of various PCI Express devices and configurations to enhance the neural compute power of the Raspberry Pi.

Explanation of the limitations encountered when trying to power multiple NPUs simultaneously due to power constraints.

Mention of other products and companies pushing AI hardware, like ArduCam’s PiNSIGHT camera and Useful Sensor for presence detection.

Insight into the future potential and evolution of AI hardware, and the importance of having multiple options like the Coral and Hailo.

Final thoughts on the niche but valuable use cases for the AI kit, especially for those in machine vision and robotics.

Encouragement for continued exploration and integration of AI hardware into practical and innovative projects.