who supply the system vectors for their JVM products. 1 HBench:Java will work best with support from JVM vendors

Transcription

1 Appeaed in the ACM Java Gande 2000 Confeence, San Fancisco, Califonia, June 3-5, 2000 HBench:Java: An Application-Specific Benchmaking Famewok fo Java Vitual Machines Xiaolan Zhang Mago Seltze Division of Engineeing and Applied Sciences Havad Univesity 33 Oxfod Steet Cambidge, MA 02138, USA , {cxzhang, ABSTRACT Java applications epesent a boad class of pogams, anging fom pogams unning on embedded poducts to highpefomance seve applications. Standad Java benchmaks ignoe this fact and assume a fixed wokload. When an actual application s beha vio diffes fom that included in a standad benchmak, the benchmak esults ae useless, if not misleading. In this pape, we pesent HBench:Java, an application-specific benchmaking famewok, based on the concept that a system's pefomance must be measued in the context of the application of inteest. HBench:Java employs a methodology that uses vectos to chaacteize the application and the undelying and caefully combines the two vectos to fom a single metic that eflects a specific application s pefomance on a paticula such that the pefomance of multiple s can be ealistically compaed. Ou pefomance esults demonstate HBench:Java s supeioity ove taditional benchmaking appoaches in pedicting eal application pefomance and its ability to pinpoint pefomance poblems. Keywods Java pefomance, benchmaking. 1. INTRODUCTION In ecent yeas, the Java pogamming language has enjoyed inceasing populaity and thee has been a polifeation of Java Vitual Machine () implementations. This poses a question fo end uses: which should they choose to un thei applications? Thee have been many attempts to evaluate diffeent implementations. Unfotunately, these appoaches shae a common dawback: they assume a fixed set of wokloads and ignoe the application s pe fomance concens. Java applications epesent a divese set of pogams, anging fom those unning on embedded poducts such as PDAs, to applets unning in bowse envionments, to scientific computing applications, and ecently to seve applications, which have taditionally been the stonghold of system languages such as C and C++. Often the actual application unde test diffes enough fom any standad benchmak that the esults fom taditional benchmaks ae useless and sometimes even misleading. Moeove, since the wokloads ae fixed, taditional benchmaks encouage vendos to ove-optimize thei implementations to achieve good esults on the benchmaks. This may potentially hut the pefomance of eal applications. Such incidents have aleady been epoted in the aea of OS benchmaking, whee gaphics cad vendos employ a hack, which can seveely hampe the pefomance of othe devices, to impove thei esults in standad benchmaks [8]. We believe that the goals of benchmaking in geneal should be theefold: 1. To compae the pefomance of systems and to eason about why applications un faste on one system than on anothe. Not only should benchmaks poduce meaningful esults, they should also povide a easonable explanation fo the pefomance diffeences. 2. To guide pefomance optimizations. Benchmaks should eveal pefomance bottlenecks o limitations of the undelying system in the context of a paticula application, and thus help system implementes impove the system in a way that will benefit the application of inteest. 3. To pedict an application s pe fomance on non-existent platfoms. Benchmaks should help answe what if questions and povide uses with a easonable estimate of the application s pefomance when some components of the undelying system change, o when the behavio of the application changes. In this pape, we pesent HBench:Java, pat of a moe geneal application-specific benchmaking famewok called HBench designed to ealize the above goals. The est of the pape is oganized as follows. Section 2 gives an oveview of some of the most popula standad Java benchmaks. Section 3 descibes the design of HBench:Java, and Section 4 descibes ou pototype implementation of HBench:Java in detail. Section 5 pesents expeimental esults. Section 6 descibes some elated woks. Section 7 discusses some unesolved issues and Section 8 concludes.

2 2. JAVA BENCHMARKS Taditional Java benchmaks can be classified into the following thee categoies: 1. Micobenchmaks. CaffeineMak [3] is a typical example, in which a set of pimitive opeations such as method invocation, aithmetic and gaphics opeations, and shot sequences of code (kenels) that solve small and welldefined poblems, ae measued, and the mean (typically geometic mean) of the individual times (o scoes as a function of the time) is epoted. Micobenchmaks ae useful in compaing the low-level opeations of s, but it is difficult to elate them to actual application pefomance in a quantitative way. 2. Macobenchmaks that contain one o moe medium-scale to lage-scale Java applications. Examples include the SPEC98 suite [15], which includes a set of pogams simila to those found in the SPECCPU suite, and VolanoMak fom Volano LLC, which is based on the company s VolanoChat seve. VolanoMak focuses on a s ability to handle long-lasting netwok connections and theads [16]. 3. Combinations of the above. The JavaGande benchmak [2][10] is an example of this type. Designed to compae the ability of diffeent Java Vitual Machines to un lage-scale scientific applications, the JavaGande benchmak suite contains thee sections. The fist section consists of micobenchmaks such as aithmetic opeations, mathematical functions, and exception handling. The second section consists of kenels, each of which contains a type of computation likely to appea in lage scientific pogams. The final section includes ealistic applications, such as a financial simulation based on Monte Calo techniques. This hybid appoach of combining micobenchmaking and macobenchmaking povides the ability to eason about pefomance dispaities between Java Vitual Machines and is paticulaly useful in pinpointing pefomance anomalies in immatue Java Vitual Machine implementations. The common dawback with the above appoaches is that Java applications ae so divese that it is difficult, if not impossible, to find a set of wokloads that ae epesentative of the applications in which end uses ae inteested, even within a sub-field. If the behavio of the benchmak s wokloads does not match that of the intended application, then the benchmak might give misleading infomation egading which is the best fo the application of inteest. In compaison, HBench:Java is a geneal benchmaking famewok that can be applied to any specific wokload. 3. HBENCH:JAVA DESIGN 3.1 Oveview HBench:Java is based on the vecto-based methodology of the HBench famewok [14]. The pinciple behind the vecto-based methodology is the obsevation that a system s pefomance is detemined by the pefomance of the individual pimitive opeations that it suppots, and that an application s pefomance is detemined by how much it utilizes the pimitive opeations of the undelying system. As the name vecto-based indicates, we use a vecto V s = ( v1, v2,..., vn ), to epesent the pefomance chaacteistics of a, with each enty v i epesenting the pefomance of a pimitive opeation of the. We call this vecto V s a system vecto, and it is obtained by unning a set of micobenchmaks. A key featue of HBench:Java is that it incopoates chaacteistics of the application into the benchmaking pocess. This is achieved using an application vecto, V A = ( u1, u2,..., un ), with each element u i epesenting the numbe of times that the coesponding ith pimitive opeation was pefomed. Intuitively, the application vecto indicates how much demand the application places on the undelying and is obtained though pofiling. The dot poduct of the two vectos poduces the pedicted unning time of the application on a given. The basic stategy behind HBench has been to use the simplest model possible without sacificing accuacy. To that end, we use a simple linea model, until we find that it is no longe able to povide the pedictive and explanatoy powe we seek. In some cases, athe than going to a moe complex model, we etain the simplicity of a linea model by adding multiple data points fo a single pimitive. Fo example, on some systems, TCP connect times gow non-linealy with the numbe of connections. Rathe than modeling the non-lineaity explicitly, we povide thee o fou points in the system vecto that coespond to diffeing odes of magnitude fo the numbe of connections. HBench:Java addesses the benchmaking goals outlined in Section 1 in the following ways: 1. The system vecto and the application vecto povide an effective way to study and explain pefomance diffeences between diffeent s. 2. The application vecto indicates which pimitive opeations ae impotant, and the system vecto eveals which pimitive opeations ae pefomance bottlenecks. System implementes can use this infomation to impove pimitive opeations that ae significant fo the application. At the same time, application pogammes can use this infomation to optimize the application by educing the numbe of calls to expensive pimitive opeations. 3. One can pedict the pefomance of the application on a given without actually unning the application on it, as long as the system vecto is available 1. One might also answe what if questions such as What if this pimitive takes twice as long? by modifying the appopiate system and application vecto enties. 3.2 Identifying Pimitive Opeations A is a complicated piece of softwae. Figue 1 shows a schematic view of a implementation. Much of a s functionality is suppoted via the system classes (also called built-in classes o bootstap classes). A includes a memoy management system that automatically manages the heap fo the application. The execution engine is esponsible fo bytecode intepetation, class loading, exception handling, thead 1 HBench:Java will wok best with suppot fom vendos who supply the system vectos fo thei poducts.

3 Use App. System Classes JIT // empty loop fo (int i = 0; i < numiteations; i++) { ; } Memoy System Execution Engine Figue 1. Schematic view of a. scheduling and context switches, the native method inteface, and synchonization. The implementation is futhe complicated by the JIT (Just In Time) component, which compiles Java bytecode on the fly into native machine code. In ode to ceate a system vecto fo a, we need to decompose this complexity into a set of pimitive opeations. One set of candidates is the s assembly instuctions, i.e., bytecodes. This appoach, howeve, poved inadequate pimaily due to the pesence of the JIT. Once bytecodes ae compiled into native machine code, optimizations at the hadwae level such as out-of-ode execution, paallel issue and cache effects can lead to a unning time that is significantly diffeent fom the sum of the execution times of the individual instuctions executed alone. Fo example, Figue 2(a) shows two Java code sequences: an empty loop and a loop containing an intege addition opeation. The coesponding native code poduced by the JIT is shown in Figue 2(b). On a Pentium III pocesso, both loop iteations take 2 cycles to execute, due to paallel instuction issues. This leads one to conclude that the addition opeation is fee, which is clealy not tue. A highe level of abstaction that is immune o less sensitive to hadwae optimization is theefoe needed. We identified the following fou types of high-level components of a system vecto: system classes, with method invocations to the system classes being pimitive opeations; memoy management, whee pimitive opeations could include object allocation, live-object identification, liveobject elocation (fo copying gabage collectos) and deadobject eclamation; execution engine, whee pimitive opeations include bytecode intepetation, exception handling, context switching, synchonization opeations, etc.; JIT, which can be measued by two metics: ovehead and quality of code geneated. JIT ovehead can be appoximated as a function of bytecode size, in which case the pimitive opeation is the time it takes to JIT one bytecode instuction. The poduct of this pe-bytecode ovehead and the numbe of JITted bytecodes yields the oveall ovehead. Note that the numbe of JITted bytecodes cannot be diectly obtained fom the application, as it is dependent. Rathe, it is obtained by applying a // loop containing intege addition fo (int i = 0; i < numiteations; i++) { sum += i; } Figue 2(a). Java code sequences //empty loop loop_stat: inc ecx ;; i++ cmp ecx, [esi+04h] ;; i<numiteations jnge loop_stat // loop containing intege addition loop_stat: add edi,ecx ;; sum += i inc ecx ;; i++ cmp ecx, [esi+04h] ;; i<numiteations jnge loop_stat Figue 2(b). Coesponding native code sequences dependant function J to the base application vecto N, and S, whee each enty in N and S epesent each method s invocation count and bytecode size, espectively. Fo example, if a compiles a method the fist time it is invoked, then J ( N, S) = s i, i whee s i is the ith element of S. The quality of JITted-code is hade to quantify, and is a subject of ongoing eseach. The system classes component povides a convenient abstaction laye and is a good stating point fo ou pototype implementation, which cuently includes only this component, as highlighted by the cicle in Figue 1. Ou expeience shows that applications tend to spend a significant amount of time in system classes. Theefoe we believe that this simplistic system vecto, albeit cude, can be indicative of application pefomance. Ou esults demonstate that HBench:Java aleady povides bette pedictive powe than existing benchmaks. 4. HBENCH:JAVA IMPLEMENTATION The implementation of HBench:Java consists of two independent pats: a pofile that taces an application s inteactions with the to poduce an application vecto and a set of micobenchmaks that measues the pefomance of the to poduce a system vecto. The following two sub-sections descibe these pats in moe detail.

4 4.1 Pofile The pofile is based on JDK s Java Vitual Machine Pofiling Inteface (PI) [7]. Once attached to the, a pofile can intecept events in the such as method invocation and object ceation. The Java SDK1.2.2 kit fom Sun comes with a default pofiling agent called hpof that povides extensive pofiling functionality [9]. We use this default pofile to obtain statistics of method invocations fom which we deive an application vecto. As a fist step, ou application vecto (and accodingly ou system vecto) only contains method invocations to system classes. A moe complete custom pofile that incopoates the gabage collecto (GC) and the execution engine and that is able to diectly poduce an application vecto is cuently unde development. A dawback of PI is that it does not povide callbacks to etieve aguments of method calls. To emedy this poblem, we implemented a second pofile that is able to ecod method aguments; it is based on JDK s Java Vitual Machine Debugge Inteface (DI) [6]. Since DI can only be enabled with JIT tuned off (fo the classic vesion of JDK), we keep both pofiles fo obvious pefomance easons, with the fist pofile esponsible fo extensive pofiling and the second pofile esponsible fo the much simple task of call tacing. 4.2 Micobenchmaks The cuent set of micobenchmaks consists of appoximately thity methods including fequently invoked methods and methods that take a elatively long time to complete, based on taces fom sample applications. Even though these methods epesent only a tiny potion of the entie Java coe API, we found them quite effective in pedicting application pefomance, as shown late in Section 5. The micobenchmak suite is implemented using an abstact Benchmak class. To add a micobenchmak to the suite, one implements a class that extends the Benchmak class. Specifically, this means implementing the untial() abstact method. A utility pogam facilitates this pocess by automatically geneating the coesponding souce Java pogam fom a template file and a file that specifies key infomation about the paticula micobenchmak. Typically, the untial() method invokes the method to be measued in a loop fo some numbe of iteations. A nice featue of ou micobenchmaks is that the numbe of iteations is not fixed, but athe dynamically detemined based on the time esolution of the System.cuentTimeMillis() function of the specific. A micobenchmak is un long enough that the total unning time is at least n times the time esolution (to allow fo accuate measuement), and less than 2n times the time esolution (so that the benchmak doesn t un fo an unnecessaily long time). Fo the expeiments epoted in this pape, we used a value of 10 fo n. Fo methods whose unning time also depends on paametes, such as the BuffeedReade.ead() method that eads an aay of bytes fom an input steam, we measue the pe-byte eading cost and the coesponding enty in the application vecto includes the total numbe of bytes instead of the numbe of times the ead() method is called. Ou cuent pototype implementation suppots this simple case of linea dependency on a single agument, and we found it sufficient fo the sample applications we tested. Fo moe complicated agument types, the system vecto enty would consist of a list of (n+1)-tuples, (t, a 1, a 2,, a n), whee a i is the value of the ith agument, and t is the time it takes to invoke the method with the given aguments. We then measue seveal data points in this n-dimension space, and extapolate the unning time based on the actual paametes included in the coesponding application vecto enty. Figue 3 shows some sample micobenchmak esults fo JDK1.2.2 (Windows NT). The time fo the ead() method of BuffeedReade is the pe-byte ead cost, and the Class.foName() method loads an empty class. 4.3 Suppot fo Pofiling and Micobenchmaking Fo some pimitive opeations such as class loading, the fisttime invocation cost is the tue cost and subsequent invocations just etun a cached value. As a esult we cannot simply measue the cost by epeatedly calling the method with the same aguments in a loop and dividing the total time by the numbe of iteations. In the case of class loading, it means we need to load a diffeent class evey iteation. With the time esolution of cuent implementations, to achieve easonable accuacy, the numbe of iteations equied is on the ode of hundeds and inceases as pocesso speed inceases. We could automatically ceate these dummy classes befoe stating the loop. Howeve, not only does this appoach not scale well, ceating a lage numbe of class files also petubs the esults since the numbe of classes within a diectoy is usually not that lage. A bette solution is to have the povide a high-esolution time API. This appoach has the added advantage of educed benchmak unning time (ecall that the numbe of loop iteations is invesely popotional to the time esolution). Most moden CPUs povide cycle countes that ae accessible in use mode, and many popula opeating systems such as Solais and Windows NT aleady povide high-esolution time APIs. Method Name Method Signatue Time(us) java.lang.chaacte.tosting ()Ljava/lang/Sting; java.lang.sting.chaat (I)C java.io.buffeedreade.ead ([CII)I java.lang.class.foname (Ljava/lang/Sting;)Ljava/lang/Class ; java.net.socket.<init> (Ljava/net/InetAddess;I)V Figue 3. Sample micobenchmak esults.

5 Table 1. Java Vitual Machines tested. CPU Memoy (MB) Opeating System Vesion Vendo JDK1.2.2_NT_PRO Pentium Po Classic Sun Micosystems 128 SDK3.2_NT_PRO 200MHz Micosoft Windows NT 4.0 JDK1.2.2_NT_II Pentium II Classic Sun Micosystems 64 SDK3.2_NT_II 266MHz Micosoft JDK1.2.2_SunOS_Classic UltaSpac IIi Classic Sun Micosystems 128 Solais 7 JDK1.2.1_SunOS_Pod 333 MHz 1.2.1_O3 Poduction Sun Micosystems One of the difficulties of micobenchmaking is that sometimes a good JIT will ecognize the micobenchmak code as dead code and optimize it out. We have to inset code to fool the JIT into believing that the vaiables used in the micobenchmak loop ae still live afte the loop, and subsequently not optimized out of the loop. Howeve, thee is a limit as to how much this wokaound can do. A bette solution would be fo the JIT to include command-line options that allow uses to specify optimization levels, simila to those pesent in C/C++ compiles. Advanced JIT techniques such as the adaptive compilation used in HotSpot [5] pose some difficulties measuing JIT ovehead, which cannot be ovecome without help fom implementes. An adaptive compile compiles methods based on thei usage. Methods might be intepeted initially. As time pogesses, some ae compiled into native code with a lightweight compile (with little optimization). Fequently executed methods might be e-compiled with a moe poweful backend compile that pefoms extensive optimization. The poblem lies in how to model the dependent function J which, given the numbe of method invocations and method bytecode sizes, yields the numbe of bytecodes compiled/optimized. We think the following enhancement to would be useful: A PI event should be geneated at the beginning and end of the compilation of a method, so that we can model and evaluate J. To measue the pe-bytecode compile/optimize ovehead, the java.lang.compile class should be augmented with APIs fo compiling and optimizing methods. 5. EXPERIMENTAL RESULTS 5.1 Expeimental Setup We an ou expeiments on a vaiety of Java Vitual Machines. Table 1 shows the list of s tested and thei configuations. Thee non-tivial Java applications (Table 2) wee used to evaluate HBench:Java. Fist, we an the applications with pofiling tuned on and deived application vectos fom the collected pofiles. Fo Mecato, which is a web cawling application, we an the poxy seve and the web cawle on two diffeent machines connected with a 100Mb Ethenet switch, isolated fom the outside netwok. The machine that hosted the poxy seve was at least as fast as the machine that hosted the client, to insue that the poxy seve was not the bottleneck. Next we an the HBench:Java micobenchmaks on the s listed in Table 1 and obtained thei system vectos. The dot poducts of the system and application vectos gave the estimated unning time fo each application on each, which was then compaed with the actual unning time to evaluate the effectiveness of HBench:Java. Since ou initial goal is to coectly pedict the atios of execution times of the applications on diffeent platfoms, we use nomalized speed in epoting expeimental esults. This also allows us to compae HBench:Java with conventional benchmaking appoaches such as SPEC98 that epot esults in the fom of atios. Table 2. Java applications used in the expeiments. Application Desciption Input Data WebL Cloudscape Mecato A scipting language designed specifically fo pocessing documents etieved fom the web [17]. A Java- and SQL-based ORDBMS (object-elational database management system). The embedded vesion is used, i.e., the database is unning in the same as the use pogam [4]. A multi-theaded web cawle [11]. A WebL scipt that counts the numbe of images contained in a sample html file. The JBMSTous sample application included in the Cloudscape distibution kit. Only the BuildATou pogam, which simulates the task of booking flights and hotels, is used. The synthetic poxy povided by the Mecato kit that geneates web documents on the fly instead of etieving them fom the Intenet.

6 Nomalized Speed Spec98 Actual hbench Pedicted JDK1.2.2_NT_II JDK1.2.2_NT_Po SDK3.2_NT_II JDK1.2.1_SunOS_Pod JDK1.2.2_SunOS-Classic SDK3.2_NT_Po Figue 4. Nomalized unning speeds fo WebL. Nomalized Speed Spec98 Actual hbench Pedicted JDK1.2.2_NT_Po JDK1.2.2_NT_II SDK3.2_NT_Po JDK1.2.1_SunOS_Pod SDK3.2_NT_II Figue 5. Nomalized unning speeds fo Cloudscape. Nomalized Speed Spec98 Actual Hbench Pedicted 0.00 JDK1.2.2_NT_Po SDK3.2_NT_Po JDK1.2.2_SunOS_Classic JDK1.2.1_SunOS_Pod Figue 6. Nomalized unning speeds fo Mecato.

7 Table 3. Impotant pimitive opeations fo WebL. Time (µs) Class.foName() ClassLoade.loadClass() BuffeedReade.ead() JDK1.2.2_NT_PRO SDK3.2_NT_PRO JDK1.2.2_NT_II SDK3.2_NT_II JDK1.2.2_SunOS_Classic JDK1.2.1_SunOS_Pod Table 4. Impotant pimitive opeations fo Mecato. Time (µs) Socket.<init>() SocketInputSteam.ead() JDK1.2.2_NT_PRO SDK3.2_NT_PRO JDK1.2.2_SunOS_Classic JDK1.2.1_SunOS_Pod Results Figue 4 shows the esults fo the scipting language WebL. In this expeiment, thee pimitive opeations account fo the majoity of the unning time, shown in Table 3. Also shown in Table 3 ae the thei measued pefomance on the five Java Vitual Machine tested. The coesponding application vecto is (80, 121, 32768). It s inteesting to note that the SPEC98 scoe of JDK1.2.2 on the PentiumPo NT machine is highe than that on the SpacStation. Howeve, WebL uns close to thee times as fast on the SpacStation. HBench:Java s system vecto eveals the poblem. Class loading is twice as fast fo the SpacStation JDK, and the BuffeedReade.ead() method executes almost 35 times faste. It tuns out that fo some eason, the NT JDK1.2.2 s JIT didn t compile the method sun.io.bytetochasinglebyte.convet(), an expensive method called many times by java.io.buffeedreade.ead(). The diffeences esult in supeio pefomance on the SpacStation. Besides explaining pefomance diffeences, the pedicted atios of execution speeds ae within a small magin of the eal execution speed atios. Figue 5 shows the esults fo Cloudscape, a database management system. We did not epot the esult fo the Sun JDK1.2.2 classic vesion on the SpacStation because Cloudscape wasn t able to un on it. Similaly to what we obseved fo the WebL esults, not only does HBench:Java coectly pedict the ode of the unning speed on the diffeent platfoms, the pedicted atios of the execution speeds closely match the actual atios. On the othe hand, SPEC98 does not pedict the ode coectly, and its pedicted speed atios ae off by a lage magin in most cases. Also simila to the case of WebL, Cloudscape spends lage amount of time in class loading. Figue 6 shows the esults fo Mecato, the web cawle. We only collected esults fo a limited numbe of s due to the difficulty of setting up the machines in an isolated netwok 2. The esults, howeve, ae quite encouaging. Even though HBench:Java pedicted the ode fo JDK1.2.2_NT_Po and SDK3.2_NT_Po incoectly, the pedicted atio still matches the actual atio quite closely. As a matte of fact, the actual atio is so close to one, it is difficult to tell which one is faste. SPEC98 again pedicted the wong ode fo Sun JDK In this case, two pimitive opeations, the constucto of java.net.socket and java.net.socketinputsteam.ead(), account fo the majoity of the unning time. Table 4 lists the cost of these two pimitives fo the fou Java Vitual Machines tested. The pe-byte socket ead time is quite simila fo the fou s. The socket initialization time, which includes the cost of ceating a TCP connection, vaies a lot among the fou s. The coesponding application vecto enty is (19525, ). To undestand why SPEC pefoms pooly, we examined the time beakdown fo use vesus system classes. Tables 5 and 6 show the pecentage of time spent in system classes fo SPEC pogams and the thee sample applications we tested, espectively. These numbes wee obtained using the sampling facility of the hpof agent included in Sun s JDK As the data show, the SPEC pogams spend most of the time in use classes. Theefoe, they ae poo pedictos fo applications that spend a lot of time in system classes. Notice that even though a lage pecentage of time goes to use classes fo the Cloudscape 2 We have an ageement with Compaq that equies expeiments concening Mecato to be un in an isolated (disconnected) netwok envionment.

8 Table 5. Time beakdown fo SPEC pogams. Pogam System Time (%) Use Time (%) _201_compess _202_jess _209_db _213_javac _222_mpegaudio _227_mtt _228_jack Aveage Table 6. Time beakdown fo sample applications. Pogam System Time (%) Use Time (%) WebL Cloudscape Mecato case, HBench:Java was still able to pedict the atios quite accuately. We suspect that this is because pefomance of use classes is lagely detemined by JIT quality. System classes ae also compiled by the same JIT, thus pefomance of a collection of system classes in some way eflects the JIT quality, which applies to use classes as well. In theoy we can use HBench:Java to pedict the unning time of SPEC pogams. Howeve, since SPEC pogams spend little time in system classes, the few system classes seve as poo sample data fo measuing JIT quality, esulting in lage eo ate. Theefoe, fo SPEC-like applications, moe sophisticated techniques to measue JIT quality ae needed. In summay, the thee examples pesented demonstate HBench:Java s ability to pedict eal applications pefomance. The esults ae especially encouaging since the system vecto contains only a small set of system class methods. We expect the accuacy of HBench:Java to impove as the system vecto is completed. 6. RELATED WORK The HBench:Java appoach is simila to the abstact machine model [12], whee the undelying system is viewed as an abstact Fotan machine, and each pogam is decomposed into a collection of Fotan abstact opeations called AbOps. The machine chaacteize obtains a machine pefomance vecto, wheeas the pogam analyze poduces an application vecto. The linea combination of the two vectos gives the pedicted unning time. This appoach equies extensive compile suppot fo obtaining the accuate numbe of AbOps and is limited to pogamming languages with extemely egula syntax. It is also highly sensitive to compile optimization and hadwae achitectue [13]. As hadwae becomes moe sophisticated, the accuacy achievable with this technique tends to decease. This is the key eason we did not use bytecodes as pimitive opeations. Bown and Seltze [1] used the vecto-based appoach of HBench to evaluate opeating systems. They demonstated that it effectively pedicts the pefomance of the Apache web seve on diffeent platfoms. The pimitive opeations in this case ae system calls, and the application vecto is essentially the system call tace. 7. DISCUSSION AND FUTURE WORK HBench:Java is still in the ealy stages of its development. Hee we identify a few unesolved issues and descibe how we plan to addess them. The fist issue is the lage numbe of API method calls. We plan to attack this poblem by identifying a set of coe methods, including methods executed fequently by most applications (such as those in the Sting class), and methods upon which many othe methods ae built (such as those in the FileInputSteam class). We then plan to analyze method intedependencies and deive unning time estimates of non-coe methods fom the unning times of the coe methods. Fo instance, a length() method typically takes the same time as a size() method. We believe that it is acceptable if the estimates of non-coe classes ae not 100% accuate, since we expect these methods to be infequently invoked. Ou goal is to keep the numbe of micobenchmaks fo the system class method calls unde 200. Anothe issue is that JIT compiles could alte an application enough that no single application vecto could be used acoss all platfoms. Ou expeience so fa indicates that this is not yet a poblem. Howeve, we will closely follow this issue as JIT technologies become moe advanced. Ou shot-tem goal is to implement a complete set of system class micobenchmaks fo HBench:Java and to test it on moe vaieties and commecial applications. In the long un, we will implement othe pats of the system vecto, including components epesenting the memoy system and the execution engine. 8. CONCLUSION HBench:Java is a vecto-based, application-specific benchmaking famewok fo s. Ou pefomance esults demonstate HBench:Java s supeioity ove taditional benchmaking methods in pedicting the pefomance of eal applications and in pinpointing pefomance poblems. By taking the natue of taget applications into account and offeing fine-gained pefomance chaacteizations HBench:Java can povide meaningful metics to both consumes and developes of s and Java applications. 9. ACKNOWLEDGMENTS We wish to thank Allan Heydon and Mac Najok at Compaq Systems Reseach Cente fo poviding us with the Mecato kit, and Suiling Ku fo he assistance with the Cloudscape softwae. We also thank the ceatos of the WebL scipting language. Special thanks go to Lance Bec fom Compaq Systems Reseach Cente fo his suggestions of Java applications fo measuement. We ae gateful to the anonymous eviewes fo thei insightful

9 comments and suggestions on impoving this pape, and to David Sullivan, Keith Smith and Kostas Magoutis fo poofeading the daft. David Sullivan implemented the fist vesion of the HBench:Java micobenchmaks. 10. REFERENCES [1] Bown, A. B. A Decompositional Appoach to Compute System Pefomance Evaluation. Technical Repot TR-03-97, Cente fo Reseach in Computing Technology, Havad Univesity, [2] Bull, J. M., Smith, L. A., Westhead, M. D., Henty, D. S., and Davey, R. A. A Methodology fo Benchmaking Java Gande Applications. In Poceedings of the ACM 1999 Confeence on Java Gande, pages 81-88, Palo Alto, CA, June 12-14, [3] CaffeineMak. [4] Cloudscape. [5] HotSpot. [6] DI, Java Vitual Machine Debugge Inteface. [7] PI, Java Vitual Machine Pofiling Inteface. [8] Jones, M., and Regeh, J. The Poblems You e Having May Not Be the Poblems You Think You e Having: Results fom a Latency Study of Windows NT. In Poceedings of the 1999 Wokshop on Hot Topics in Opeating Systems (HotOS VII), pages , Rio Rico, AZ, Mach 29-30, [9] Liang, S., and Viswanathan, D. Compehensive Pofiling Suppot in the Java Vitual Machine. In 5th USENIX Confeence on Object-Oiented Technologies and Systems (COOTS 99), pages , San Diego, CA, May 3-7, [10] Mathew, J. A., Coddington, P. D., and Hawick, K. A. Analysis and Development of Java Gande Benchmaks. In Poceedings of the ACM 1999 Confeence on Java Gande, pages 72-80, Palo Alto, CA, June 12-14, [11] Mecato. [12] Saaveda-Baea, R. H., Smith, A. J., and Miya, E. Machine Chaacteization Based on an Abstact High-Level Language Machine. IEEE Tansactions on Compute, 38(12), Decembe 1989, [13] Saaveda-Baea, R. H., Smith, A. J., Analysis of Benchmak Chaacteistics and Benchmak Pefomance Pediction. ACM Tansactions on Compute Systems, 14(4), Novembe 1996, [14] Seltze, M., Kinsky, D., Smith, K., and Zhang X. The Case fo Application-Specific Benchmaking. In Poceedings of the 1999 Wokshop on Hot Topics in Opeating Systems (HotOS VII), pages , Rio Rico, AZ, Mach 29-30, [15] SPEC98. [16] VolanoMak. [17] WebL.