JSMeter: Characterizing the Behavior of JavaScript Web Applications

Size: px

Start display at page:

Download "JSMeter: Characterizing the Behavior of JavaScript Web Applications"

Cathleen Hodges
7 years ago
Views:

1 JSMeter: Characterizing the Behavior of JavaScript Web Applications Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research, Redmond in collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock 1

2 Why Measure JavaScript? Standardized, de facto language for the web Support in every browser, much existing code Browser and JavaScript performance is important Are current JavaScript benchmarks representative? Limited understanding of JavaScript behavior in real sites Who cares? Users, web application developers, JavaScript engine developers 3

3 ZDNet 29 May 2008 Browser Wars! ghacks.net Dec

crypto-aes Instrument date-xparb JavaScript execution and measure behavior regexp Compare behavior math-cordic of

4 Artificial Benchmarks versus Real World Sites 7 V8 programs: richards deltablue crypto raytrace 8 SunSpider programs: 3-draytrace access-nbody bitops-nsieve controlflow JSMeter Goals of JSMeter Project 11 real sites: earleyboyer crypto-aes Instrument date-xparb JavaScript execution and measure behavior regexp Compare behavior math-cordic of JavaScript benchmarks against real sites splay string-tagcloud Consider how benchmarks can mislead design decisions Maps Maps 5

$How We Measured JavaScript \ie\jscript\*.$ dll website visits custom trace files custom trace files

5 How We Measured JavaScript \ie\jscript\*.cpp Source-level instrumentation custom jscript.dll website visits custom trace files custom trace files Offline analyzers 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Constant Other Str Ops Concat Op 6

6 Visiting the Real Sites Getting past page load performance Attempted to use each site in normal way: amazon Search a book, add to shopping cart, sign in, and sign out bing Type in a search query and also look for images and news bingmap Search for a direction from one city to another cnn Read front page news ebay Search for a notebook, bid, sing in, and sign out economist Read front page news, view comments facebook Log in, visit a friend pages, browse through photos and comments gmail Sign in, check inbox, delete a mail, and sign out google Type in a search query and also look for images and news googlemap Search for a direction from one city to another hotmail Sign in, check inbox, delete a mail, and sign out 7

7 Understanding JavaScript Behavior Code JavaScript Events Objects 8

8 Code Behavior Function size Instructions/call Code Code locality Instruction mix JavaScript Events Objects 9

9 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Source size (kilobytes) Total Bytes of JavaScript Source 2500 Code Objects Events Real Sites V8 SunSpider 10

10 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Unique Functions Executed Code Objects Events Static Unique Functions Executed 4,000 3,500 3,000 2,500 2,000 1,500 1, Real Sites V8 SunSpider 12

amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes

11 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Bytecodes / Call Code Objects Events Bytecodes / Call function(a,b) { var i=0,elem,pos=a.length; if(d.browser.msie) { while(elem=b[i++]) if(elem.nodetype!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a } Real Sites V8 SunSpider 15

12 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Fraction of Code Executed Fraction of Code Executed Code Objects Events % 90.00% 80.00% 70.00% 60.00% Most code not executed 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Real Sites V8 SunSpider 16

13 Object Allocation Behavior Allocation by types Code Live heap composition Lifetime distribution JavaScript Events Objects 17

14 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Total heap data (kilobytes) Code Objects Events Total Bytes Allocated Real Sites V8 SunSpider 18

15 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay V8 aggragate 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud aggregate Heap Data by Type 1 Heap Data by Type Code Objects Events Few benchmarks allocate much data Objects Enumarator RegExp Math Error Date Native Func String Arrays Script Func Many functions Rest are strings Real Sites V8 SunSpider 19

16 Code Objects Events Live Heap Over Time (gmail) Functions grow steadily GC reduces size of heap Objects grow steadily too 20

17 Code Objects Events Live Heap over Time (ebay) Heap contains mostly functions Heaps repeatedly created, discarded Heap drops to 0 on page load 21

18 Code Objects Events 2 Search Websites, 2 Architectures You stay on the same page during your entire visit Code loaded once Heap is bigger Every transition loads a new page Code loaded repeatedly Heap is smaller Bing Google 22

19 Event Handlers in JavaScript Number of events Sizes of handlers Code JavaScript Events Objects 26

20 Code Objects Events Event-driven Programming Model Single-threaded, non-preemptive event handlers Example handlers: onabort, onclick, etc. Very different from batch processing of benchmarks Handler responsiveness critical to user experience 27

21 Total Events Handled Total Events Handled Code Objects Events 7,000 6,000 5,000 4,000 3,000 2,000 Almost no events 1,000 0 Real Sites V8 28

22 Median Bytecodes / Event Handled Median Bytecodes / Event Handled Code Objects Events

23 Sure, this is all good, but Everyone knows benchmarks are unrepresentative How much difference does it make, anyway? Wouldn t any benchmarks have similar issues? 33

24 Observation Cold-code Experiment Real web apps have lots of code (much of it cold) Benchmarks do not Question: What happens if the benchmarks have more code? We added extra, unused to code to 7 SunSpider benchmarks We measured the impact on the benchmark performance 34

25 Time (msec) Time (msec) Performance Impact of Cold Code K 200K 400K K 200K 400K K K M 2M M 2M 0 0 Chrome Cold code makes SunSpider on Chrome up to 4.5x slower IE Cold code has non-uniform impact on execution time 35

26 Impact of Benchmarks What gets emphasis Making tight loops fast Optimizing small amounts of code Important issues ignored Garbage collection (especially of strings) Managing large amounts of code Optimizing event handling Considering JavaScript context between page loads 36

27 Conclusions JSMeter is an instrumentation framework Used to measure and compare JavaScript applications High-level views of behavior promote understanding Benchmarks differ significantly from real sites Misleads designers, skews implementations Next steps Develop and promote better benchmarks Design and evaluate better JavaScript runtimes Promote better performance tools for JavaScript developers 37

28 Additional Resources Project: Video: Project JSMeter: JavaScript Performance Analysis in the Real World" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn Paper: JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps 10), June Additional measurements: "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR , December 2009 (49 pages), November

29 Additional Slides 39

30 Hot Function Distribution Code Objects Events 80% of time in 100+ functions 80% of time in < 10 functions 80% 80% Real Sites V8 Benchmarks 40

31 Opcode Distribution Code Objects Events Outliers Real Apps Green = SunSpider Blue= Real Web Apps Red = V8 41

32 Object Type Distribution Code Objects Events Real Apps economist is an outlier (arrays) Green = SunSpider Blue= Real Web Apps Red = V8 42

33 Code Objects Events Distribution of Time in Handlers 43

34 JavaScript Related Work Richards, Lebresne, Burg, and Vitek (PLDI 10) Draw similar conclusions Java Doufour et al. (OOPSLA 03), Dieckmann and U. Hölzle (ECOOP 99) Other languages C++: Calder et al. (JPL 95) Interpreted languages: Romer et al. (ASPLOS 96) 44

JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications

JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications Paruj Ratanaworabhan Kasetsart University paruj.r@ku.ac.th Benjamin Livshits and Benjamin G. Zorn Microsoft Research