SoMA. Automated testing system of camera algorithms. Sofica Ltd

Transcription

1 SoMA Automated testing system of camera algorithms Sofica Ltd February 2012

2 2 Table of Contents Automated Testing for Camera Algorithms 3 Camera Algorithms 3 Automated Test 4 Testing 6 API Testing 6 Functional Testing of the Algorithms 8 Performance Testing 8 Stress Testing 9 Quality Testing 10

3 3 Automated Testing of Camera Algorithms embraces three key features of modern digital camera development. The sheer pace with which camera sensors are being integrated to various devices has highlighted the importance of the camera algorithms: The image quality is improved significantly and different image effects are generated using algorithms. Moreover, the testing is an important part of the camera development process and test automation has proven benefits. Camera Algorithms are an essential feature of the camera system operation. Captured images are significantly modified before being presented to the user. The algorithms can be divided roughly into quality related, sensor and camera module specific and special effects as following figure describes. Very often, there is a strong interdependency between algorithms and therefore, changes in one algorithm changes the characteristics of another. The combinations of different algorithm parameters are numerous and the testing of each case is very demanding.

4 4 A good example of a challenging algorithm to test is the auto white balance. Various algorithms and camera settings influence the color balancing of the image and the final result may be as in following image. The leftmost image has the correct colors while the right has serious color defects. Other common issues of digital camera modules can include, exposure and focus problems, noise artifacts and lens specific errors, like lens shading and distortion. Even if some camera algorithm defects can be detected by human eye, objective testing and repetitive measurement is very demanding. Furthermore, increasing the number of camera algorithms and their interdependencies increases the test load significantly. SoMA provides a solution, which analyzes mathematically the captured images and compares the results to reference images. This brings strict objectivity to the testing. When mathematical analysis is executed in the automated test environment of SoMA, testing is very efficient and reproducible. Automated test environment is the key point. Generally, the automated testing offers very valuable features like efficient repetitive testing, objectivity, repeatability and it also enables new ways to test. Sofica offers a twofold automated testing environment where the environment enables to use various reference images and the captured images are measured automatically, without human interactions.

5 5 Even though the SoMA offers a ready test environment, it can also be integrated to an existing test system and used part by part. The SoMA DUT tests can be replaced by existing tests and the interface with the verification service is made using the easily integrated SoMA API. Beyond this, the SoMA database can be replaced using existing database as can the user interface of SoMA. SoMA can be used as a stand-alone test system, containing all the required parts of the automated test environment or it can be integrated to the existing test system, when the existing tests can be used. In this case, SoMA increases the existing test coverage and also automatically verifies the real output of the camera: captured images and recorded video.

6 6 Testing contains several, separately usable, testing levels which are ensuring efficient testing at key phases of development. Different test levels support the development. In the first phase, the compatibility of the camera API is ensured. Onwards, when the features of the camera are implemented, the functional tests are effective. At the same time, performance tests can be started to detect the bottlenecks as early as possible. Stress and quality testing finalizes the testing entity by ensuring the robustness of the system and the final quality of the captured images. Even if tests are divided according to the development phases, all test levels can be used thorough the whole camera R&D process to avoid the regression. In addition, test reports and measurement data collected through the testing process is a valuable asset for later analysis and comparison of devices. API testing with SoMA is more than a traditional API testing. Normally, the API testing concentrates on verifying the compatibility of the API methods (or functions) and testing the parameter ranges of each method. As the following table describes, API testing of SoMA verifies also the

7 7 dependencies between methods and also verifies the robustness by testing illegal method scenarios. Three steps of the API testing: 1. Parameter testing: Compatibility verification of the methods by testing the smallest and largest input parameter values, as well as random values between them. Furthermore, negative testing is done by providing illegal input parameter values to the method interface and verifying that the correct error code is returned. 2. Correlation between inputs: The API testing verifies the correlation between methods. Normally, there are two kinds of dependencies between methods: callbacks and set-get method pairs. Testing verifies that correct GET values are returned for each SET input. Also the correct response of illegal input values is verified. In addition, the callback logic also is checked by detecting the response (callback) to each call. 3. Method scenarios: The API testing verifies the method sequences and the stability of the system when illegal method sequences are executed. This has revealed to be a very effective and stressful test to the camera systems and several robustness improvements are done after the sequence tests. The API testing also offers the possibility to compare the API compatibility and robustness between different devices. When the existing API tests were executed against commercial devices, the pass rates were as low as 57% and severe crash issues were identified. The API testing can be implemented against every API in several platforms. Currently, the SoMA API testing has large reference implementation against Android CameraHardwareInterface. The API testing contains about 200 different tests against this interface. CameraHardwareInterface is the device driver interface of Android camera system and it offers C++ API. However, the same kind of tests can be done, for example, against Java-API. SoMA also has the option to execute external tests, such as, Android

8 8 Compatibility Test Suite (CTS) tests. When SoMA API testing verifies the device driver interface, CTS ensures the compatibility of the application layer. Together these features create a very comprehensive testing entity of the whole camera system. Functional testing of the algorithms requires the capture and verification of the real images. The functionality of different camera algorithms and parameters can be tested by measuring their influence on the captured image. SoMA uses mathematical analysis to detect the differences between reference and captured images. The mathematical analysis of the automatic test environment enables efficient testing of both individual functionalities and the combination of several features. The main purpose of the functional testing is to verify the correct functionality and generate repeatable measurements between test executions and across devices. However, functional testing alone is not sufficient to determine the absolute quality of the camera system. SoMA quality testing provides these additional measurements capabilities. Performance testing reveals bottlenecks in the camera system by measuring the execution times of different camera functions and combinations of functions. Very high resolution camera sensors can cause critical performance issues when megabyte sized images are processed in a very short time. Performance testing creates timestamps when the camera API calls are made and when the corresponding response is received or callback method is executed. The execution time is simply the difference between these timestamp values. SoMA also measures and records the execution time of each camera function. For example, the compression time of the encoder can be measured as can the shutter speed with different auto focus settings.

9 9 Such measurements are quite straightforward however, different combinations of features may change the performance values significantly. With SoMA it is possible to measure different combinations and identify the critical ones. In general, performance tests do not require reference images. However, there are some features which affect the performance and the image capture circumstances should be static to ensure objective test results. As an example, the execution time of the auto focus and auto exposure depends on the environment. SoMA also enables the study of performance trends between test executions. This facility is valuable for early detection of development trends to detect the improvement of the development or the regression of some feature. Performance testing also enables benchmarking and comparison between different devices. When tests were executed on a sample of commercial devices, significant differences were found. For example, the shutter speed varied from 50 to 225 ms. Stress testing, as the name suggests, aims to identify the limits of the camera system, by testing the robustness and stability of the system. Furthermore, it reveals cumulative defects of the system and analyzes the memory usage. SoMA simplifies the creation of large stress test entities, consisting of API, functional, performance and quality tests. This enables, for example, the generation of performance statistics from long duration tests. Random testing is one way to search for unexpected defects. Different tests are randomly executed against camera system and the corresponding test execution schemes and logs are saved. If defects are detected, the saved test flow can be executed again and the defects can be analyzed more closely.

10 10 Stress testing, like all SoMA testing, automatically saves data in the event of a crash, which helps to resolve the root cause. The crash data files comprise process information, file handlers, system log, memory information and tombstone log. Quality testing of the camera algorithms helps to fine tune the captured images. Testing against standardized test charts gives absolute and comparable information of the quality of the whole camera system. Standardized test charts like Macbeth and noise charts together with automated algorithm testing evaluate the corresponding features of the camera. Quality of the processed video stream is the most common problem when developing new multimedia products. Video processing algorithms can introduce distortion in the video signal, so testing video quality in every R&D phase is important. SoMA's video processing algorithm testing helps to find anomalous behaviour of encoders and decoders like Audio-Video Synchronization problems from processed video stream.