Rigel : flexible multi-rate image processing hardware
Abstract/Contents
- Abstract
- Computational photography and computer vision have a number of exciting new emerging applications, such as scene understanding, tracking, and lightfield photography. Unfortunately, running applications like these on a battery-powered mobile device in real time on full resolution images is impossible using CPUs and GPUs. Traditional processors are simply not power efficient enough to support the computational throughput required for these applications. Historically, real-time mobile image processing has been implemented using custom hardware called camera image signal processors (camera ISPs). Camera ISPs exploit the structure of image processing pipelines to minimize memory bandwidth using an architectural pattern called line-buffering, where all intermediate data between each stage is stored in small on-chip buffers, yielding energy efficiency orders-of-magnitude better than CPUs and GPUs. Unfortunately, the languages and tools currently used to generate custom hardware like camera ISPs are tedious to use, error-prone, and not widely known in the graphics and vision community. This makes it very difficult for developers and researchers to have access to the benefits of custom hardware. This thesis presents Rigel, a new domain-specific system for generating fixed-function image processing hardware. Rigel is based around an abstract hardware representation called the line-buffered pipeline, which corresponds to the architecture of camera ISPs. Rigel takes as input a new language for stencil image processing called Darkroom. Our compiler lowers stencil code to line-buffered pipelines with optimal buffering using a simple Integer Linear Programming formulation. The line-buffered pipeline is then lowered to a concrete hardware implementation for FPGA and custom hardware. Next, this thesis presents a novel generalization of line-buffered pipelines, which expands their capabilities to support multi-rate modules. Adding multi-rate capabilities to line-buffered pipelines allows them to support three key features: space-time tradeoffs, image pyramids, and sparse computations. The extra flexibility enabled with multi-rate modules allows Rigel to support a wide range of future vision applications. Crucially, multi-rate pipelines also allow more flexibility in how algorithms are mapped to hardware, which enables Rigel to successfully compile complex vision applications to FPGA, which are more restricted than custom hardware. This thesis evaluates the design of Rigel by synthesizing FPGA and custom hardware implementations for a number of key image processing and computer vision algorithms. For stencil pipelines, we show how our system can map a camera pipeline, edge detection, corner detection, deblurring and more to custom hardware. Our system delivers 1 gigapixel/sec. performance in under 0.5mm2 of ASIC silicon at 250 mW (simulated on a 45nm foundry process), which is competitive with the performance of commercial ISPs. By adding multi-rate modules, we then demonstrate how advanced vision pipelines like depth from stereo, Lucas-Kanade optical flow, and Gaussian pyramids can run efficiently on two FPGA boards. Rigel can synthesize hardware for these pipelines on FPGA with up to 436 megapixels/second throughput, and up to 297X faster runtime than a tablet-class ARM CPU.
Description
| Type of resource | text |
|---|---|
| Form | electronic; electronic resource; remote |
| Extent | 1 online resource. |
| Publication date | 2017 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Associated with | Hegarty, James |
|---|---|
| Associated with | Stanford University, Computer Science Department. |
| Primary advisor | Hanrahan, P. M. (Patrick Matthew) |
| Thesis advisor | Hanrahan, P. M. (Patrick Matthew) |
| Thesis advisor | Aiken, Alexander |
| Thesis advisor | Horowitz, Mark |
| Advisor | Aiken, Alexander |
| Advisor | Horowitz, Mark |
Subjects
| Genre | Theses |
|---|
Bibliographic information
| Statement of responsibility | James Hegarty. |
|---|---|
| Note | Submitted to the Department of Computer Science. |
| Thesis | Thesis (Ph.D.)--Stanford University, 2017. |
| Location | electronic resource |
Access conditions
- Copyright
- © 2017 by James Hegarty
- License
- This work is licensed under a Creative Commons Attribution Non Commercial No Derivatives 3.0 Unported license (CC BY-NC-ND).
Also listed in
Loading usage metrics...