F4graph : an API for camera scheduling and heterogeneous image processing on mobile devices
Abstract/Contents
- Abstract
- Mobile device photography is in the midst of a transformation, with a proliferation of new hardware for both image capture and processing. However, the camera APIs for mobile operating systems rarely expose the full capabilities of the hardware, because these depend on low-level code or platform-specific details. This work introduces F4graph, an application model and software interface which opens up many more capabilities of the imaging hardware to userspace applications. Within F4graph, captures on multiple image sensors or other hardware can be linked together to produce complex sequences of actions using a straightforward constraint-based scheduling model. The system provides strong guarantees by checking as many constraints as possible before committing actions to hardware and by only breaking constraints at well-defined points. Additionally, F4graph allows the application to define the processing pipeline along with the capture sequence so image data can be streamed directly to a reconfigurable image processor or other hardware. This includes the ability to stream continuously, to process multiple images from a stream, and to create feedback cycles for metering and similar applications. This thesis also describes how F4graph can be implemented in a real system, using a Xilinx Zynq Ultrascale board as a prototyping platform. Our demonstration system uses the Zynq's FPGA fabric as a reconfigurable image processing accelerator and its real-time core to orchestrate timing-critical events while the application processor cores run a normal multitasking Linux OS. Using the F4graph API on our prototype, userspace applications are able to schedule and execute actions with a temporal resolution under 20 microseconds. The complexity of this heterogeneous hardware system is managed by a build system which generates the entire hardware/software stack from the application code and a short human-readable system description file. This makes it possible to develop complete applications that use precise capture sequencing and hardware-accelerated image processing without the need to write platform-specific or real-time code.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2022; ©2022 |
| Publication date | 2022; 2022 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Bell, Steven Emberton |
|---|---|
| Degree supervisor | Horowitz, Mark (Mark Alan) |
| Thesis advisor | Horowitz, Mark (Mark Alan) |
| Thesis advisor | Hanrahan, P. M. (Patrick Matthew) |
| Thesis advisor | Levoy, Marc |
| Degree committee member | Hanrahan, P. M. (Patrick Matthew) |
| Degree committee member | Levoy, Marc |
| Associated with | Stanford University, Department of Electrical Engineering |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Steven Bell. |
|---|---|
| Note | Submitted to the Department of Electrical Engineering. |
| Thesis | Thesis Ph.D. Stanford University 2022. |
| Location | https://purl.stanford.edu/jy210vq1769 |
Access conditions
- Copyright
- © 2022 by Steven Emberton Bell
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...