Camera Sensor Driver Development And Integration Introduction Camera enables multimedia on phones. It is going to be an important human machine interface, adding to augmented reality possibilities on embedded platforms. There are three major camera interfaces to be taken into account during sensor driver implementation and integration into the system. Pathpartner has demonstrated experiences in all these interfaces. Specific case studies are listed in this Camera sensor interfaces paper illustrating our experiences in camera sensor integration and driver development opportunities with device makers. Case studies for camera sensor driver development and integration into Android devices, based on Freescale s imx53, Samsung s Exynos 3, Broadcom s BCM2835, application processors, and Aptina s MT9P111 & Omnivision s OV7675 sensors are presented here. Camera enables multimedia on phones. It is going to be an important HMI interface Camera Control interface (CPI) CCIR 601/565: 8 or 10-bit parallel data lines are used in this case. Data synchronization is obtained through dedicated pins including some or all of HSYNC, VSYNC, and FRAME fields. Pixel clock is provided by the sensor module. Depending on the raw image type the synchronization signals might be embedded in the data. Maximum pixel clock supported is usually 96 Mhz. 1 Camera sensor driver integration www.pathpartnertech.com
Camera serial interface CSI2-MIPI: This is a low pin-count interface targeted at handheld device space. Data and synchronization information is sent over differential serial lines. There could be up-to 4 data lane pairs plus 1 clock pair. The specification conforms to a standard set up by MIPI sensor module. We have implemented CSI2 interface for connecting OV5650 sensor to the Raspberry Pi development board. Camera Control Interface CCI: This is usually a two or three-wire interface used to control the sensor module. Though named differently by different vendors (e,g. Serial Camera Control Bus, SCCB by Omni Vision), it usually confirms to the I2C standards (defined by Philips) We used the Linux I2C bus driver to implement this layer and to get and set parameters from the module. This includes the reset sequence, changing of the picture resolution, frame rate, special effects etc Integrating camera sensor to Samsung S5PC110 Camera sensor [Aptina s MT9P111] is integrated to Samsung s S5PC110 processor with Android ICS OS. Sensor driver integration is layered and first it needs to be done at the Linux kernel. The camera sensor driver plugs into the V4L2 (Video for Linux ver 2) driver layer of the Linux as a sub-dev (sub-device) of type video capture interface. The application sees the interface as a device node in the filesystem tree - /dev/video0 and needs to call the standard V4L2 APIs to capture a frame. V4L2 Abstraction on application side: The application calls various operations on the video device node. Each call translates into a call into the v4l2 core layer in Linux (located in drivers/media/video/, all files with v4l2 prefix). Application calls are routed to the v4l2 device registered in the system. The v4l2 device is usually one or more instances of the image/video capture pipelines provided in the application processor SoC. 2 Camera sensor driver integration www.pathpartnertech.com
In Samsung s5pc110 this is called FIMC (Fully Interactive Mobile Camera), so the FIMC driver (provided by Samsung) calls the following functions [v4l2_device_register & video_register_device] to register with the v4l2 framework. It also provides the video framegrabber capability to the system. FIMC driver provides implementation of the v4l2_ioctl_ops IOCTL operations for the device. These are the operations which get called from user-space. The sensor driver in turn plugs into (as a slave) to the FIMC. In the v4l2 framework, the driver needs to register itself as a subdev [v4l2_i2c_subdev_init]. More often than not sensor drivers are connected and detected using the i2c interface. The i2c slave address, bus number and other info ( the power up sequence, pixel format, default resolution supported, the Hsync/Vsync polarity etc. ) is provided in the board file (arch/arm/mach-s5pv210/machsmdkc110.c in this case). Refer to structures [i2c_board_info & s3c_platform_camera].cam_power structure element points to the function which gets called to power up/down the sensor module. This function gets written by the integrator after observing GPIO pins and other schematic details. The sensor platform information is provided to the FIMC layer (e,g. Which FIMC port this sensor connects to? etc.) by populating the structure refer to [s3c_platform_fimc]. This structure needs to be populated with.camera structure element for further accessing sensor s core and video ops. It is important to populate and map sensor driver related functions into this structure accurately. For example, In our case FIMC drivers calls the sensor's mt9p111_querymenu and provides the list of controls supported by the sensor. These could be white balance controls, saturation, sharpness and contrast controls etc. The ranges of values supported are also provided. These could be further used by the application to set app values. Invoking V4L2 APIs at HAL layer Application, in this case the ICS HAL for camera app follows the following procedure to grab a frame. open( /dev/video0, O_RDWR); This would call the FIMC open operation to perform FIMC related initialization. Sensor still remains powered down though. Actual function of camera sensor starts by issuing a series of IOCTL commands. These commands are typically issued in following order: VIDIOC_QUERYCAP: queries the capability of v4l2 device (FIMC) 3 Camera sensor driver integration www.pathpartnertech.com
VIDIOC_ENUMINPUT: enumerates front & back camera inputs. VIDIOC_S_INPUT: sanity check for camera initialization parameters (like i2c slave address, clock etc.) i2c probing and device power up does not happen yet!! There is no sensor driver function that gets called here. VIDIOC_ENUM_FMT: Repeated query to get pixel formats supported by the sensor. We added support for these formats only: YCbCr 4:2:0 tiled semi-planar (64x32 pixel blocks). (12 bits per pixel) YCbCr 4:2:0 linear semi-planar linear (12 bpp) Camera sensor provided 8-bit parallel Y/Cb/Cr 4:2:2 (which translates to 2 bits per pixel) interleaved data which FIMC converts to 4:2:0 semi planar data on the fly. VIDIOC_S_FMT: Change image format (width, height, color space, color format etc.) VIDIOC_REQBUFS/VIDIOC_QUERYBUF: requests and queries buffers for frame capture, no hooks from the sensor driver. streaming. Enhancements at Android HAL and Stagefright Video recording was made configurable between dual port and single video recording. Dual port video recording Two different FIMC ports are used one (FIMC0) for preview and other (FIMC2) for video recording. Preview (from FIMC0) is either rendered back to the display driver (software loop-back) or encoded and saved to JPEG format (still image capture). Frames captured through FIMC2 are directly passed through to the MFC (multiformat codec) where MPEG-4 encoding is done by the MFC block in hardware and the resultant compressed frames are provided to the videorecorder library for writing to file. Single port video recording FIMC0 port alone is used and the preview frames themselves are being MPEG4 (or H264) encoded in software (OMXCodec) before being presented to the video-recorder library for writing to file. Physical orientation of the sensor on the device needs to be accounted for as well. Accordingly software configuration of orientation (flip and mirroring) can be changed. Rotation can also be enabled before hardware encoding of frames. VIDIOC_G_CTRL/VIDIOC_S_CTRL: Get & set values for controls like brightness, saturation, contrast, rotation, horizontal and vertical flip etc. VIDIOC_G_PARM/VIDIOC_S_PARM : Get and set stream parameters (like capture capabilities, time between successive frames etc.) In our case the only parameter that is configurable is the time per frame attribute. VIDIOC_QBUF/VIDIOC_DBUF: These are used to queue and de queue buffers from the capture queue. No hooks from sensor driver. VIDIOC_STREAMON: Here the actual sensor hardware is turned on. Calls sensor driver hooks in particular order to turn it on and start video Image from sensor post integration VGA 640X 480 4 Camera sensor driver integration www.pathpartnertech.com
Integrating camera sensor to BCM 2835 (Raspberry Pi) Broadcom s application processor BCM2835 based low cost computer Raspberry Pi has generated large interest among developer and off-the block designer communities. This development board has most of the peripheral interfaces brought out on connecter sockets. We took up interfacing, development and integration of Omnivision s OV5640 camera sensor to this board for increasing use cases with this low-cost computer. Raspberry Pi has its camera interface brought out on board through a 15-pin connector socket. Camera sensor needs to be interfaced using CSI2- MIPI interface. For interfacing the camera sensor we have used: Two pair of differential data lines is used One pair of differential clock line is used. Two wire I2C control lines are used. With appropriate ground lines this set constitutes the 15-lines from Raspberry Pi. Due to difference in voltage levels between OV5650 (needs 2.8V) and Raspberry Pi board output (gives at 3.3V) a voltage shifter using Max3001E circuit is designed. Sensor platform data in the board file is entered through the structure ov5640_platform_camera Voltage Shifter Raspberry Pi Omnivision sensor Pathpartner Technology Consulting Pvt Ltd., # 16, PSS Plaza, 1st and 2nd Floor, New Thippasandra Main Road, H.A.L. III Stage, Bangalore, Karnataka 560075. www.pathpartnertech.com, email: sales@pathpartnertech.com 5 Camera sensor driver integration www.pathpartnertech.com