JSMeter: Characterizing the Behavior of JavaScript Web Applications Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research, Redmond in collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock 1
Why Measure JavaScript? Standardized, de facto language for the web Support in every browser, much existing code Browser and JavaScript performance is important Are current JavaScript benchmarks representative? Limited understanding of JavaScript behavior in real sites Who cares? Users, web application developers, JavaScript engine developers 3
ZDNet 29 May 2008 Browser Wars! ghacks.net Dec. 2008 4
Artificial Benchmarks versus Real World Sites 7 V8 programs: richards deltablue crypto raytrace 8 SunSpider programs: 3-draytrace access-nbody bitops-nsieve controlflow JSMeter Goals of JSMeter Project 11 real sites: earleyboyer crypto-aes Instrument date-xparb JavaScript execution and measure behavior regexp Compare behavior math-cordic of JavaScript benchmarks against real sites splay string-tagcloud Consider how benchmarks can mislead design decisions Maps Maps 5
How We Measured JavaScript \ie\jscript\*.cpp Source-level instrumentation custom jscript.dll website visits custom trace files custom trace files Offline analyzers 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Constant Other Str Ops Concat Op 6
Visiting the Real Sites Getting past page load performance Attempted to use each site in normal way: amazon Search a book, add to shopping cart, sign in, and sign out bing Type in a search query and also look for images and news bingmap Search for a direction from one city to another cnn Read front page news ebay Search for a notebook, bid, sing in, and sign out economist Read front page news, view comments facebook Log in, visit a friend pages, browse through photos and comments gmail Sign in, check inbox, delete a mail, and sign out google Type in a search query and also look for images and news googlemap Search for a direction from one city to another hotmail Sign in, check inbox, delete a mail, and sign out 7
Understanding JavaScript Behavior Code JavaScript Events Objects 8
Code Behavior Function size Instructions/call Code Code locality Instruction mix JavaScript Events Objects 9
amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Source size (kilobytes) Total Bytes of JavaScript Source 2500 Code Objects Events 2000 1500 1000 500 0 Real Sites V8 SunSpider 10
amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Unique Functions Executed Code Objects Events Static Unique Functions Executed 4,000 3,500 3,000 2,500 2,000 1,500 1,000 500 0 Real Sites V8 SunSpider 12
amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Bytecodes / Call Code Objects Events Bytecodes / Call 300.00 250.00 200.00 150.00 function(a,b) { var i=0,elem,pos=a.length; if(d.browser.msie) { while(elem=b[i++]) if(elem.nodetype!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a } 100.00 50.00 0.00 Real Sites V8 SunSpider 15
amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Fraction of Code Executed Fraction of Code Executed Code Objects Events 100.00% 90.00% 80.00% 70.00% 60.00% Most code not executed 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Real Sites V8 SunSpider 16
Object Allocation Behavior Allocation by types Code Live heap composition Lifetime distribution JavaScript Events Objects 17
amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Total heap data (kilobytes) Code Objects Events 45000 Total Bytes Allocated 40000 35000 30000 25000 20000 15000 10000 5000 0 Real Sites V8 SunSpider 18
amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay V8 aggragate 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud aggregate Heap Data by Type 1 Heap Data by Type Code Objects Events Few benchmarks allocate much data 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Objects Enumarator RegExp Math Error Date Native Func String Arrays Script Func Many functions Rest are strings Real Sites V8 SunSpider 19
Code Objects Events Live Heap Over Time (gmail) Functions grow steadily GC reduces size of heap Objects grow steadily too 20
Code Objects Events Live Heap over Time (ebay) Heap contains mostly functions Heaps repeatedly created, discarded Heap drops to 0 on page load 21
Code Objects Events 2 Search Websites, 2 Architectures You stay on the same page during your entire visit Code loaded once Heap is bigger Every transition loads a new page Code loaded repeatedly Heap is smaller Bing Google 22
Event Handlers in JavaScript Number of events Sizes of handlers Code JavaScript Events Objects 26
Code Objects Events Event-driven Programming Model Single-threaded, non-preemptive event handlers Example handlers: onabort, onclick, etc. Very different from batch processing of benchmarks Handler responsiveness critical to user experience 27
Total Events Handled Total Events Handled Code Objects Events 7,000 6,000 5,000 4,000 3,000 2,000 Almost no events 1,000 0 Real Sites V8 28
Median Bytecodes / Event Handled Median Bytecodes / Event Handled 500 506 2137 Code Objects Events 450 400 350 300 250 200 150 100 50 0 30
Sure, this is all good, but Everyone knows benchmarks are unrepresentative How much difference does it make, anyway? Wouldn t any benchmarks have similar issues? 33
Observation Cold-code Experiment Real web apps have lots of code (much of it cold) Benchmarks do not Question: What happens if the benchmarks have more code? We added extra, unused to code to 7 SunSpider benchmarks We measured the impact on the benchmark performance 34
Time (msec) Time (msec) Performance Impact of Cold Code 900 900 800 800 700 700 600 600 500 400 0K 200K 400K 500 400 0K 200K 400K 300 800K 300 800K 200 100 1M 2M 200 100 1M 2M 0 0 Chrome 3.0.195.38 Cold code makes SunSpider on Chrome up to 4.5x slower IE 8 8.0.601.18865 Cold code has non-uniform impact on execution time 35
Impact of Benchmarks What gets emphasis Making tight loops fast Optimizing small amounts of code Important issues ignored Garbage collection (especially of strings) Managing large amounts of code Optimizing event handling Considering JavaScript context between page loads 36
Conclusions JSMeter is an instrumentation framework Used to measure and compare JavaScript applications High-level views of behavior promote understanding Benchmarks differ significantly from real sites Misleads designers, skews implementations Next steps Develop and promote better benchmarks Design and evaluate better JavaScript runtimes Promote better performance tools for JavaScript developers 37
Additional Resources Project: http://research.microsoft.com/en-us/projects/jsmeter/ Video: Project JSMeter: JavaScript Performance Analysis in the Real World" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn Paper: JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps 10), June 2010. Additional measurements: "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009. 38
Additional Slides 39
Hot Function Distribution Code Objects Events 80% of time in 100+ functions 80% of time in < 10 functions 80% 80% Real Sites V8 Benchmarks 40
Opcode Distribution Code Objects Events Outliers Real Apps Green = SunSpider Blue= Real Web Apps Red = V8 41
Object Type Distribution Code Objects Events Real Apps economist is an outlier (arrays) Green = SunSpider Blue= Real Web Apps Red = V8 42
Code Objects Events Distribution of Time in Handlers 43
JavaScript Related Work Richards, Lebresne, Burg, and Vitek (PLDI 10) Draw similar conclusions Java Doufour et al. (OOPSLA 03), Dieckmann and U. Hölzle (ECOOP 99) Other languages C++: Calder et al. (JPL 95) Interpreted languages: Romer et al. (ASPLOS 96) 44