"No Power Struggles! Co-or d i n a t e d M u l t i -l e v e l P ow e r M a n a g e m e n t f or t h e D a t a Ce n t e r Ramya Rag h av e n d r a, P ar t h a Ran g an at h an, V an i s h T al w ar, Z h i k u i W an g, i ao yu n Z h u 2 0 0 6 H e w l e t t -P a c k a r d D e v e l o p m e n t C o m p a n y, L. P. T h e i n f o r m a t i o n c o n t a i n e d h e r e i n i s s u b j e c t t o c h a n g e w i t h o u t n o t i c e

Motivation P ow e r & c ool ing a b u r ning is s u e f or f u tu r e d ata c e nte r s 5 0 % 2 5 % $ 3 2 b illion World wide cost of purchasing and operating servers environmental impact, heat& density, reliability

H P L ab s S m ar tp ow e r P r oj e c t T he pow er lif ecycle S ystems approach to redu cing pow er A d d r e s s i n g t h e s o u r c e m e a n s c u m u l a t i v e b e n e f i t s A t t h e i n t e r s e c t i o n o f I T a n d f a c i l i t i e s d o m a i n s K ey research streams P o w e r -a w a r e s y s t e m a n d s o l u t i o n a r c h i t e c t u r e s M o d e l s a n d m e t r i c s f o r p o w e r c h a r a c t e r i z a t i o n Ref. Patel, C. and Shah, A., Co s t M o del fo r Planni ng, D ev elo p m ent and O p er ati o n o f a D ata Center, H PL -2 0 0 5-1 0 7 ( r.1 )

S om e R e c e nt W or k (citations refer to bibliography at end of slid e d eck ) P ow er-aw are systems & solu tions architectu re B l a d e p o w e r c a p p i n g, m u l t i -l e v e l p o w e r m a n a g e m e n t, p r o f i t -a w a r e s c h e d u l i n g, t e m p e r a t u r e -a w a r e s c h e d u l i n g [ 3 ] [ 4 ] [ 7 ] [ 1 0 ] [ 1 3 ] [ 1 4 ] M odels and metrolog y f or pow er & heat J o u l e S o r t, Z e s t i, B l a d e S i m, C o n S i l, W e a t h e r m a n, S p l i c e [ 1 ] [ 2 ] [ 5 ] [ 6 ] [ 8 ] [ 9 ] [ 1 5 ] O ther pow er manag ement [ 1 1 ] [ 1 2 ] M ore details in H otc hips tu torial or S martp ow er w eb site T oday s talk M u l t i -l e v e l p o w e r m a n a g e m e n t

O O T h e P ow e r S tr u g g l e P r ob l e m heterogeneity V M R ack E nclosure I A M S erver il O L S F S I M V m S -gwlm V M S -wlm otion C P U -res. all L ocal optim a CHAOS!! objectives, actuators, time-cn sts, f ormul ation s A verage power P eak therm al power glob al optim a P eak electrical power

C ontr ib u tions F irst u nif ied architectu re f or data center pow er manag ement M i n i m a l i n t e r f a c e s & i n f o r m a t i o n e x c h a n g e b e t w e e n l o o p s F e e d b a c k c o n t r o l t h e o r y f o r m a t h e m a t i c a l r i g o r E v a l u a t i o n o n r e a l -w o r l d t r a c e s : c o r r e c t n e s s, s t a b i l i t y, e f f i c i e n c y I nsig hts on desig n trade-of f s A r c h i t e c t u r a l a l t e r n a t i v e s f o r v a r i o u s o b j e c t i v e f u n c t i o n s I m p l e m e n t a t i o n a l t e r n a t i v e s ( t i m e c o n s t a n t s a n d h w / s w ) M e c h a n i s m s ( p -s t a t e s, V M s ) & p o l i c i e s ( p r e -e m p t i v e, f a i r -s h a r e, )

U nif ie d and E x te ns ib l e A r c h ite c tu r e V M C o nt ro l l er (V MC ) W o r k l o a d c o n s o l i d a t i o n P o w e r m i n i m i z a t i o n P o w e r / c a p a c i t y c o n s t r a i n t s G ro u p Manager (G M) R a c k o r Da t a C e n t e r l e v e l p o w e r t h r o t t l i n g a n d b u d g e t d i s t r i b u t i o n E nc l o s u re Manager (E M) E n c l o s u r e -l e v e l p o w e r c a p p i n g Server Manager (SM) S e r v e r -l e v e l p o w e r c a p p i n g E f f i c i enc y C o nt ro l l er (E C ) P o w e r a d a p t a t i o n t o d e m a n d

B ac k g r ou nd F eedback control theory F o r m a l t h e o r e t i c a l g u a r a n t e e s o f s t a b i l i t y a n d p e r f o r m a n c e C a n a c c o u n t f o r i n a c c u r a c i e s i n m o d e l A d a p t s t o t h e w o r k l o a d o n t h e f l y T ar g et e + Co ntr o ller Allo c ati o n Ac tu ato r De m a n d Sy s tem M eas u r em ent - Standard feedback control loop

E f f ic ie nc y + c ap p ing at s ing l e s e r ve r A d a p t p o w e r c o n s u m t o t h e l o c a l b u d g e t p t i o n A d a p t t h e c a p a c i t y a l l o c a t i o n t o t h e r e s o u r c e d e m a n d P o w e r C a p p i n g P o w e r e f f i c i e n c y E f f i c i e n c y : t r a c k i n g p r o b l e m N o t i o n o f c o nt ai ner f ro m A C T S w o rk o n ad ap t i ve res o u rc e c o nt ro l A l l o w s p o w er c o ns u m p t i o n t o t rac k w o rk l o ad d em and C a p p i n g : t h r o t t l i n g p r o b l e m C h ange rref t o ad ap t t o p o w er c ap

G M M M Mu l ti-l e ve l P ow e r C ap p ing E n c l o s u r e M a n a g e r ( E M ) E nc l o s u re-l evel t h ro t t l i ng S e r v e r M a n a g e r s ( S M + E C ) E f f i c i enc y + c ap p i ng L oca l b ud g et i n S M r re f 1 EC 1 1 D 1 B L A D E P1 G r o u p M a n a g e r ( G M ) P o l i c y -b as ed al l o c at i o n Enclosure i n b ud g et - + P e n c l E M S M L oca l b ud g et m D m m r re f m M a x EC B L A D E Pm s u m D N-1 GM M i n S M N-1 r re f N-1 EC N-1 S e r v e r PN-1 roup b ud g et + i n S M N r re f N EC N D N S e r v e r PN - P grp s u m

G V ir tu al Mac h ine W or k l oad D is tr ib u tor D 1 V MC SM E C BLADE r 1 V i r t u a l M a c h i n e C o n t r o l l e r EM SM E C D m BLADE r m W o rk l o ad p l ac em ent P o w er m i ni m i z at i o n P o w er b u d get c o ns t rai nt s r o u p b u d g e t GM SM E C D N-1 S ER V ER r N-1 0-1 I nt eger P ro gram m i ng G reed y h eu ri s t i c D N SM E C S ER V ER r N C a p p i n g + e f f i c i e n c y W o r k l o a d c o n s o l i d a t i o n & m i g r a t i o n

U nif ie d and E x te ns ib l e A r c h ite c tu r e V M C o nt ro l l er (V MC ) W o r k l o a d c o n s o l i d a t i o n P o w e r m i n i m i z a t i o n P o w e r / c a p a c i t y c o n s t r a i n t s G ro u p Manager (G M) R a c k o r Da t a C e n t e r l e v e l p o w e r t h r o t t l i n g a n d b u d g e t d i s t r i b u t i o n E nc l o s u re Manager (E M) E n c l o s u r e -l e v e l p o w e r c a p p i n g Server Manager (SM) S e r v e r -l e v e l p o w e r c a p p i n g E f f i c i enc y C o nt ro l l er (E C ) P o w e r a d a p t a t i o n t o d e m a n d

E val u ation - Me th od ol og y C halleng es w ith f u ll-scale deployment stu dy A n y o n e h a v e a l i v e d a t a c e n t e r I c a n b o r r o w? E v a l u a t i n g n e x t -g e n h a r d w a r e w i t h c u r r e n t s y s t e m s? P r o t o t y p i n g s c a l e a n d w o r k -b e n e f i t t r a d e o f f s O u r approach: mu lti-prong ed S i m u l a t i o n -b a s e d e v a l u a t i o n o f d e s i g n s p a c e ( B l a d e S i m ) R e a l w o r l d t r a c e s f r o m 9 e n t e r p r i s e s ( 1 8 0 s e r v e r s ) P o w e r / p e r f m o d e l s f r o m p r o t o t y p e c a l i b r a t i o n S m a l l e r -s c a l e p r o t o t y p e e v a l u a t i o n o f s p e c i f i c s o l u t i o n s M o d e l -b a s e d a n a l y s i s f o r s t a b i l i t y

M W M E val u ation D e s ig n C h oic e s C ombinatorial ex plosion in desig n space C o n t r o l l e r s, h a r d w a r e, f r e q u e n c i e s, o v e r h e a d s, p o l i c i e s, l e v e l s, O u r approach: representative su bset s i m p l i f y i n g b u t n o t s i m p l i s t i c a s s u m p t i o n s e tr ic e s and k nob s Notation B as e v al u e ( o de l s ) : po w pe rf = g p ( r ) = c p r + d p, = h p ( r ) = a p r, p = p = 0, 1, 2,... 0, 1, 2,... ( E C ) : f ( k ) = f ( k 1) λf Q ( k 1) r ( k 1) / rref ( rref r ( k 1)). ( S M ): r ( kˆ) = r ( kˆ 1) ( c a p _ l o c p o w ( kˆ r e f r e f β loc 1)). p o w _ l o c ( E M ) : c a p _ l o c = m i n ( C A P _ L O C, c a p _ e n c p o w _ e n c ( GMs ) : c a p ( V MC s ) _ e nc c a p _ lo c (1) = mi n ( C A P _ E N C = mi n ( C A P _ L O C m p o w i = 1 m n i + α M i= 1 j = 1 n (2) s. t. ij r j (1 + α V ) r j = 1 (3) (4) (5) (6) (7) mi n, p o w p o w p o w, p o w _ e nc _ g rp _ lo c _ g rp 0 ij ij ). C A P _ GRP ). C A P _ GRP ). p o w c C A L O C p o w e C A E N C p o w g C A i (1 b _ lo ) P _ i, i = 1, 2,..., m m M iq i (1 b _ nc ) P _ q, q = 1, 2,..., l i= 1 m i (1 b _ rp ) P _ GRP i= 1 m ij = 1, j = 1, 2,..., n i= 1 ij = {0,1}, i = 1, 2,..., m, j = 1, 2,..., n S erv er static pow er budget CAP_LOC 10 % of f serv er m ax dy n am ic pow er budget c a p _l o c tun ed by E M or G M pow er con sum ption p o w m easured f or S M / E M / G M target util iz ation r_re f tun ed by S M m easured util iz ation r m easured f or E C P - states p 0, p 1, p0,, p4, tun ed by E C f f H z [, 8 3 3 M, 7 0 0 M, 6 0 0 M, q f f 3 M ] H z p e N f m E n n c a p n c G M desired cl ock req uen cy uan tiz ed req uen cy _Q 1G 53 perf orm an ce rf work don e static pow er budget CAP_E C 15% of en cl osure ax cl osure dy am ic pow er budget _e tun ed by G roup V irtual M ach in e ork l oad S y stem P roperty C on trol I n terv al C on trol l er G ain pow er con sum ption p o w _e n c m easured f or E M an d G M pow er budget CAP_G R P 2 0 % of f group m ax pow er con sum ption p o w _g rp m easured f or G M v irtual iz ation ov erh ead α_ V 10 % of V M util iz ation m igration ov erh ead α_μ 10 % of V M util iz ation con strain ts buf f ers b _l o c, b _e n c, b _g rp tun ed based on budget v iol ation s n um ber of work l oads n 18 0 en terprise traces dem an d f or capacity D in util iz ation pl acem en t on serv ers m atrix w ith 0 / 1 el em en ts n um ber of serv ers m 18 0 n l 2 0 m 0 / & f ( E C ) T c um ber of en cl osures rel ation sh ip betw een serv ers en cl osures M atrix w ith 1 el em en ts ef icien cy con trol _e 1 serv er m an ager ( S M ) T _s m 5 en cl osure m an ager ( E M ) T _e m 2 5 group m an ager ( G M ) T _g rp 50 V M C on trol l er ( V M C ) T _v m c 50 0 ef f icien cy con trol ( E C ) λ 0. 8 serv er m an ager ( S M ) β _l o c 1

R e s u l ts : I t w or k s! W o r k l o a d -a w a r e a d a p t a t i o n & c o r r e c t n e s s

R e s u l ts : I t w or k s w e l l! A v erag e Pow er 6 5 % saving s ( O pe x ) Peak Pow er 2 0 % saving s ( C ape x ) Performance S imilar perf ormance 0 2 0 4 0 6 0 8 0 1 00 U n i f i e d B a s e l i n e

D e s ig n I ns ig h ts : A n e x am p l e A v erag e Pow er U n i f i e d VMC on l y No VMC Peak Pow er Performance 0 2 0 4 0 6 0 8 0 1 00 Unified V M C O nl y N o V M C B a s el ine

O th e r ins ig h ts on d e s ig n tr ad e of f s A rchitectu ral choices V M v s E C, c o o r d i n a t i o n a l t e r n a t i v e s S ystem desig n choices P o w e r b u d g e t s, P -s t a t e s I mplementation aspects K n o b s, o v e r h e a d s, t i m e -c o n s t a n t s, p o l i c i e s C oordination enables f lex ibility and simplicity S e e p a p e r f or m or e d e t a i l s

O ng oing and F u tu r e W or k Detailed validation and prototyping D eploy on H P L abs S mart D ata C enter E x tens ion to oth er c ontrollers P erf ormance, cooling, S L A

S u m m ar y P ow er an important challeng e f or f u tu re data centers D a t a c e n t e r p o w e r h o t a r e a $ 3 0 B m a r k e t i n 2 0 0 7, c o m p a c t i o n, e n v i r o n m e n t a l b e n e f i t s U n c o o r d i n a t e d s o l u t i o n s i n e f f i c i e n t & u n s t a b l e V a r y i n g t i m e c o n s t a n t s, o v e r l a p p i n g o b j e c t i v e f u n c t i o n s & a c t u a t o r s, F irst u nif ied architectu re f or data center pow er manag ement U n i f i e d a r c h i t e c t u r e b a s e d o n f o r m a l i s m A d a p t i v e c o n t r o l l e r c o r e + w e l l -d e f i n e d i n t e r f a c e s a n d p o l i c i e s c o r r e c t n e s s + s t a b i l i t y + p e r f o r m a n c e 6 5 % e l e c t r i c i t y r e d u c t i o n, 2 0 % p r o v i s i o n i n g r e d u c t i o n, s i m i l a r p e r f I n s i g h t s o n d e s i g n t r a d e o f f s A r c h i t e c t u r a l a n d i m p l e m e n t a t i o n t r a d e o f f s ; m e c h a n i s m s a n d p o l i c i e s

Q u e s tions?

B ib l iog r ap h y 1. J o u l e S o r t : A B a l a n c e d E n e r g y -E f f i c i e n c y B e n c h m a r k, S u z a n n e R i v o i r e, M e h u l S h a h, C h r i s t o s K o z y r a k i s, P a r t h a s a r a t h y R a n g a n a t h a n, N o v e m b e r 2 0 0 6 [p d f ] 2. Z e s t i : F u l l -S y s t e m P o w e r M o d e l i n g a n d E s t i m a t i o n, D i m i t r i s E c o n o m o u s, S u z a n n e R i v o i r e, C h r i s t o s K o z y r a k i s, a n d P a r t h a s a r a t h y R a n g a n a t h a n, J u n e 2 0 0 6 [p d f ] 3. C o s t -a w a r e S c h e d u l i n g f o r H e t e r o g e n e o u s E n t e r p r i s e M a c h i n e s ( C A S H E M ), J e n n i f e r B u r g e, P a r t h a s a r a t h y R a n g a n a t h a n, J a n e t L. W i e n e r, S e p t e m b e r 2 0 0 6 [p d f ] 4. N o P o w e r S t r u g g l e s : A U n i f i e d P o w e r M a n a g e m e n t A r c h i t e c t u r e f o r t h e D a t a C e n t e r, R a m y a R a g h a v e n d r a, P a r t h a s a r a t h y R a n g a n a t h a n, V a n i s h T a l w a r, i a o y u n Z h u, a n d Z h i k u i W a n g, ASPLOS, March 2008 [h p -p d f ] 5. " S i m u l a t i n g C o m p l e x E n t e r p r i s e W o r k l o a d s u s i n g U t i l i z a t i o n T r a c e s, " P a r t h a s a r a t h y R a n g a n a t h a n a n d P h i l i p L e e c h, T e n t h W o rk s ho p o n C o m p u t e r Archi t e ct u re E v al u at i o n u s i n g C o m m e rci al W o rk l o ad s ( C AE C W ), h e l d w i t h H P C A -1 3, F e b r u a r y 2 0 0 7 [p d f ] 6. F u l l -s y s t e m P o w e r A n a l y s i s a n d M o d e l i n g f o r S e r v e r E n v i r o n m e n t s, D i m i t r i s E c o n o m o u s, S u z a n n e R i v o i r e, C h r i s t o s K o z y r a k i s, a n d P a r t h a s a r a t h y R a n g a n a t h a n, W o rk s ho p o n Mo d e l i n g, B e n chm ark i n g, an d Si m u l at i o n ( Mo B S), J u n e 2 0 0 6 [p d f ] 7. E n s e m b l e -l e v e l P o w e r M a n a g e m e n t f o r D e n s e B l a d e S e r v e r s, P a r t h a s a r a t h y R a n g a n a t h a n, P h i l L e e c h, D a v i d I r w i n, a n d J e f f C h a s e, Pro ce e d i n g s o f t he I n t e rn at i o n al Sy m p o s i u m o n C o m p u t e r Archi t e ct u re ( I SC A), J u n e 2 0 0 6 [p d f ] A n e a r l i e r v e r s i o n a p p e a r e d i n T e c h c o n 2 0 0 5. 8. C o n S i l : L o w -c o s t T h e r m a l M a p p i n g o f D a t a C e n t e r s, J u s t i n M o o r e, J e f f C h a s e, a n d P a r t h a s a r a t h y R a n g a n a t h a n. F i rs t W o rk s ho p o n T ack l i n g C o m p u t e r Sy s t e m s Pro b l e m s w i t h Machi n e Le arn i n g T e chn i q u e s ( Sy s ML), J u n e, 2 0 0 6 [p d f ] 9. W e a t h e r m a n : A u t o m a t e d, O n l i n e, a n d P r e d i c t i v e T h e r m a l M a p p i n g a n d M a n a g e m e n t f o r D a t a C e n t e r s, J u s t i n M o o r e, J e f f C h a s e, a n d P a r t h a s a r a t h y R a n g a n a t h a n, Pro ce e d i n g s o f t he T hi rd I n t e rn at i o n al C o n f e re n ce o n Au t o n o m i c C o m p u t i n g ( I C AC ), J u n e 2 0 0 6 [p d f ] 10. P o w e r B a l a n c i n g f o r F u t u r e -g e n e r a t i o n B l a d e s, H e r n a n L a f f i t t e, P h i l L e e c h, P a r t h a s a r a t h y R a n g a n a t h a n, C h a r l i e S h a v e r, K h a l d o u n A l z i e n, Pro ce e d i n g s o f H P T e chco n, A p r i l 2 0 0 6 [h p -p d f ] 11. E n e r g y -a w a r e u s e r i n t e r f a c e s a n d e n e r g y -a d a p t i v e d i s p l a y s, P a r t h a s a r a t h y R a n g a n a t h a n, E r i k G e e l h o e d, M e e r a M a n a h a n, a n d K e n N i c h o l a s, I E E E C o m p u t e r, M a r c h 2 0 0 6 (c o v e r f e a t u r e ) [p d f ] 12. H e t e r o g e n e o u s c h i p m u l t i p r o c e s s o r s, R a k e s h K u m a r, D e a n T u l l s e n, N o r m a n J o u p p i, P a r t h a s a r a t h y R a n g a n a t h a n, I E E E C o m p u t e r, N o v e m b e r 2 0 0 5 (c o v e r f e a t u r e ) [p d f ] 13. D e n s e a n d s m a r t : H a r d w a r e -s o f t w a r e c o -o r d i n a t i o n f o r b l a d e s e r v e r p o w e r r e d u c t i o n, P a r t h a s a r a t h y R a n g a n a t h a n e t a l, Pro ce e d i n g s o f H P T e chc o n, M a r c h 2 0 0 5. [h p -p d f ] 14. M a k i n g S c h e d u l i n g C o o l : T e m p e r a t u r e -a w a r e R e s o u r c e S c h e d u l i n g, J u s t i n M o o r e, J e f f C h a s e, P a r t h a s a r a t h y R a n g a n a t h a n, R a t n e s h S h a r m a. P r o c e e d i n g s o f t he 2005 An n u al U s e n i x C o n f e re n ce, A p r i l 2 0 0 5. [p d f ] A s h o r t e r v e r s i o n a p p e a r s a s a p o s t e r i n H P T e c h C o n, M a r c h 2 0 0 5 [h p -p d f ] 15. D a t a C e n t e r W o r k l o a d M o n i t o r i n g, A n a l y s i s, a n d E m u l a t i o n, J u s t i n M o o r e, J e f f C h a s e, K e i t h F a r k a s, a n d P a r t h a s a r a t h y R a n g a n a t h a n. I n t h e E i g ht h W o rk s ho p o n C o m p u t e r Archi t e ct u re E v al u at i o n u s i n g C o m m e rci al W o rk l o ad s ( C AE C W ), F e b r u a r y, 2 0 0 5. ( I n v i t e d p a p e r ) [p d f ]

B onu s S l id e s

G u ar ante e ing S tab il ity ( A p p e nd ix A ) Control theory provides formal analysis and synthesis for desirab le properties of the system u nder c ontrol, e. g., stab ility and z ero trac k ing error. S tab ility g u arantees predic tab le b ehavior w hen the system ex perienc es c hang es, e. g., w ork load c hang es, different b u dg et c onfig u ration. A q u antitative proof for b oth stab ility and z ero trac k ing error in one ex ample sc enario is sk etc hed. T he E C c ontroller c an mak e the CP U u tiliz ation trac k a spec ified u tiliz ation targ et b y dynamic ally tu ning the c loc k freq u enc y, in spite of c hang es in the w ork load demand; T he S M c ontroller c an mak e the server pow er c onsu mption trac k a g iven loc al pow er c ap, possib ly set b y the u pper layer c ontrollers su c h as the E M or the G M, b y dynamic ally tu ning the u tiliz ation targ et fed into the E C c ontroller. T he hierarc hic al arc hitec tu re desig n and u se of different time sc ales in different c ontrol loops mak e it possib le to provide at least q u alitative arg u ments for stab ility. ( Models: ) fq( k 1) r( k 1) ( EC) f ( k) = f ( k 1) λ ( rref r( k 1)). rref ( SM) r ( kˆ) = r ( kˆ 1) β ( cap_ loc pow( kˆ 1)). ref pow= g perf = h ( r) = a r, ref p ( r) = c r + d p p p loc p, p = 0,1,2,... p = 0,1,2,... I: For a given u tiliz ation target r ref (kˆ), the C P U u tiliz ation of the server, r( k ), converges glob ally and asy m ptotically, u sing the ef f iciency controller ( E C ), u d er the cond ition 0 < λ < 1/ r. ref II: For a given local power cap cap_ loc, the server power consu m ption pow(kˆ) converges glob ally, u sing the server m anager ( E C ), u d er the cond ition 0 < λ < 1/ r ref and 0 < β < 2 / cmax.

Coordination Federation v s c entral iz ation M u l t i -v e n d o r, i s o l a t i o n, a b s t r a c t i o n, l o c a l i n f o r m a t i o n S ol v ing one as p ec t does n t w ork F i g u r e 9 i n p a p e r E x p l p l V M C + g r o u p c a p p e r v i c i o u s c y c l e am e ap ic ation E x tens ions C o m p o n e n t / p l a t f o r m c o o r d E l e c t r i c a l p o w e r c a p p e r M u l t i p l e a c t u a t o r s a t l e v e l V M -p l a t f o r m c o o r d H e t e r o g e n e i t y E n e r g y -d e l a y I m p l e m e n t a t i o n s i n h a r d w a r e / s w System under control P ow er v i ola ti ons G M E M SM B la de A p erf loss p w r sa v e C oordi na ted 0 0-5 - 3 6 4 U ncoordi na ted 0 0-8 -12 7 2 C oordi na ted, a p p r uti l 0 0-3 - 2 5 6 C oordi na ted, no f eedb a ck 0 0-7 - 4 6 9 C oordi na ted, no b udg et li mi ts 0-5 -23-8 7 6 U ncoordi na ted, mi n P sta tes 0 0 0-13 7 1 Serv er B C oordi na ted 0 0-7 - 6 5 7 U ncoordi na ted 0 0-1 -19 6 3 C oordi na ted, a p p r uti l 0 0-3 - 3 4 4 C oordi na ted, no f eedb a ck 0 0-13 - 7 6 6 C oordi na ted, no b udg et li mi ts 0-15 -18-12 7 2 U ncoordi na ted, mi n P sta tes 0 0 0-19 5 0 Level EC C h a n g es t o en a b le c o o r d i n a t i o n Ex p o s e A P I t o S M t o c h a n g e r _ r e f S M p o s e A P I t o a n d G M t o c h a n g e p o w e r b u d g e t p o s e A P I t o G M t o c h a n g e p o w e r b u d g e t p o s e p o w e r b u d g e t v i o l a t i o n s t o V M G M p o s e p o w e r b u d g e t v i o l a t i o n s t o V M U s e " r e a l u t i l i z a t i o n " ; u s e p o w e r b u d g e t s a s c o n s t r a i n t s ; V M e x p l i c i t f e e d b a c k t o v i o l a t i o n s Ex EM Ex EM Ex C Ex C C I nterf ac es A P I D M T F, C I M

M ode l s and s e ns itiv ity % pwr savings 8 0 6 0 4 0 2 0 C o o rd in a te d N o V M C V M C O n l y 0 1 8 0 6 0 L 6 0 M B l a d e A 6 0 H 6 0 H H 6 0 H H H 1 8 0 6 0L 6 0 M S e r v e r B 6 0 H 6 0 H H 6 0 H H H Power Consumption 1 00 9 0 8 0 7 0 6 0 p0, p1 p2 p3 p4 P o w e r M o d e l o f B l a d e A Performance 1 00 8 0 6 0 4 0 20 Performance Model of Blade A p 0, p 1 p 2 p 3 p 4 Power 1 00 9 0 8 0 7 0 6 0 p 0, p 1 p 2 p 3 p 4 p 5 Power model of Server B Performance 1 00 8 0 6 0 4 0 20 Performance model of Server B p 0, p 1 p 2 p 3 p 4 p 5 5 0 0 2 0 4 0 6 0 8 0 1 00 U t i l i z a t i o n ( % ) 0 20 4 0 6 0 8 0 1 00 U t i li z at i on ( % ) 5 0 0 0 2 0 4 0 6 0 8 0 1 00 U t i li z a t i on ( % ) 20 4 0 6 0 8 0 1 00 U t i li z at i on ( % )

Simulation methodology