Browser Testing Framework for LHG Presented by Trevor Woerner, Will Chen Date February, 2015
Outline Overview of the test suite Test category Run the test On Linux On Android Verified platforms Test output Output file Browser comparison
Overview of the test suite
Overview of the test suite Linaro Web Browser Test Suite is a benchmark tool to evaluate the performance of web browsing. It consists of Bash script, Python script, html, and Javascript files.
Overview of the test suite Test suite contain six categories (Sites, Layout, Image, Rendering, Javascript, HTML5). The testing result is reported in milliseconds, which is a measurement of the test suite processing time.
Overview of the test suite Features: Automatic Can be used on Linux and Android platform Easy to add test cases for future extension Browser options can be configured Test case/category/iteration number can be configured
Overview of the test suite Compared to other test suite: High testing coverage - Combine most of the existing browser test suite on the internet. Easy to compare - One score to represent the performance. Has recent popular test cases - Add web album, 4k image, and web game test.
Test category
Test category Sites: This benchmark is an offline copy of the front page of some real and commonly used web sites.
Test category Layout: this benchmark performs mostly static HTML/CSS or javascript controlled dynamic HTML layout of objects on the page. This is supposed to benchmark sites that are mainly static and with little computation, few images and little user interaction.
Test category Image: this benchmark performs mainly image rendering and layout. This is supposed to benchmark sites with lots of images. For instance, it should demonstrate the laying out of the images in an online album.
Test category Rendering: this benchmark mainly consists of rendering animated vector graphics into a canvas. It is controlled by Javascript, but most of the time is spent in graphics libraries. This is supposed to benchmark sites that are graphic intensive.
Test category Javascript: this benchmark consists of Javascript tests, mainly focusing on the Javascript engine performance. It is supposed to reflect performance of rich internet application (RIA) sites. For instance it should demonstrate the usage of an online mail agent.
Test category HTML5: this benchmark performs image/canvas drawing, graphic manipulation, and 2D shape rendering specific for HTML5. For instance it should demonstrate the usage of the web game.
Run the test On Linux
Run the test on Linux We need a browser, bash script, and Python to run the test suite on Linux. Put test suite in the storage device and issue command, then the test suite will finish all the tests and generate testing result automatically.
Run the test on Linux
Run the test On Android
Run the test on Android We need to establish ADB connection between testing device and host computer to run the test. The test suite can install itself in the testing device automatically. The installed path is configured in script file.
Run the test on Android Host computer ADB connection Target device
Run the test Verified platforms
Run the test on Android
Test output Output file
Test output suite_scores.out: Record Real Time and Process Time of each testing category. Also record the summing time of all the categories except html5. The total Process Time is the final benchmarking score (the unit is ms).
Test output all_scores.out: Similar with suite_scores.out, but it will append each testing result in the file. So, we can check the previous result in this file.
Test output run. category.log: The log file when run the test for specific category.
Test output run. category.out: Record the Suite Score of each test case in the specific category. This file will record each case s score and sum of the score. It will also contain the real time, user time, and system time while running this category (the unit is second).
Test output test_result_ category.html: Similar with run. category.out file, but it is more easy to read. It shows the name and Suite Score of each case by table with html format.
Test output Browser comparison
Test output Comparison between chromium and firefox browser: (Platform is Intel i5 CPU + Ubuntu on Laptop)
Test output