Highly Reliable Two-Dimensional RAID Arrays for Archival Storage

Transcription

1 Highly Reliable Two-Dimeioal RAID Array for Archival Sorage Jeha-Fraçoi Pâri Comuer Sciece De. Uiveriy of Houo Houo, TX 77- Thoma Schwarz, S. J. Deo. de Iformáica y Ciecia de la Comuació U. Caólica del Uruguay 6 Moevideo, Uruguay chwarz@ucu.edu.uy Ahmed Amer Comuer Egieerig De. Saa Clara Uiveriy Saa Clara, CA 9 a.amer@acm.org Darrell D. E. Log Comuer Sciece De. Uiveriy of Califoria Saa Cruz, CA 96 darrell@c.ucc.edu Abrac We ree a wo-dimeioal RAID archiecure ha i ecifically ailored o he eed of archival orage yem. Our rooal ar wih a fairly coveioal wodimeioal RAID archiecure where each dik belog o exacly oe horizoal ad oe verical RAID level rie. Oce he array ha bee oulaed, we add a uerariy device ha coai he excluive OR of all he coe of all horizoal or verical ariy dik. The ew orgaizaio olerae all rile dik failure ad early all quadrule ad quiule dik failure. A a reul, i rovide mea ime o daa lo MTTDL more ha a hudred ime beer ha hoe of e of RAID level 6 rie wih equal caaciy ad imilar ariy overhead. Keyword- dik array, RAID array, faul-olerace, orage yem reliabiliy. D D D P D D D P D D D P Q Q Q Fig.. A wo-dimeioal RAID array wih 9 daa ad 6 ariy dik. D D D P I. ITRODUCTIO Archival orage yem coiue a ew cla of orage yem ha were ecifically deiged o reerve fairly able daa over log eriod of ime. A a reul, hee yem differ from coveioal orage yem i everal imora oi. Fir, he daa hey coai have o remai available over ime eriod ha ca a decade. Hece archival orage yem mu be exceioally reliable. Secod, hee daa will remai largely umodified oce hey are ored. For hi reao, wrie rae are a much le imora iue ha i coveioal orage yem. Fially, archival daa are o likely o be acceed a frequely a acive daa. We ca hu exec ha may of he dik of a archival yem will be owered dow mo of he ime. We rooe here a low-redudacy orage oluio ha ake advaage of hee acce aer o achieve much higher mea ime daa lo ha RAID level 6 archiecure. While he archival ore i oulaed, i oerae a a coveioal wo-dimeioal RAID array uch a he oe decribed o Fig. : each dik belog o exacly oe horizoal ad oe verical RAID level rie. A we will how laer, hi orgaizaio roec daa agai all double dik failure ad mo rile ad quadrule dik failure. Oce he array i filled wih daa, we add a uerariy dik [WS9]. Thi dik will coai he excluive OR Suored i ar by Gra CCF-96, by he Dearme of Eergy uder Award umber DE-FC-ER67/DE-SC7 ad by he idurial member of he Sorage Syem Reearch Ceer. D D D P D D D P Q Q Q S Fig.. The ame array wih a uerariy dik S. XOR of all he coe of all horizoal or verical ariy dik. A we will ee, he ew orgaizaio olerae all rile dik failure ad mo quadrule ad ad quiule dik failure. Should a igifica fracio of he ored daa eed o be udaed, we will dicoec he uerariy dik ad rever o he origial dik orgaizaio uil he udae ha comleed. The mai advaage of our ew orgaizaio i i excelle reliabiliy: wo-dimeioal RAID array wih uerariy rovide mea ime o daa lo MTTDL more ha a hudred ime beer ha hoe of e of RAID level 6 rie wih equal caaciy ad imilar ariy overhead. I mai limiaio i i oor wrie hroughu, which reric i alicaio o he orage of archival ad oher immuable daa. The remaider of hi aer i orgaized a follow. Secio II review reviou work. Secio III iroduce our archiecure ad dicue i vulerabiliy o quadrule ad quiule failure. Secio IV evaluae i reliabiliy ad comare i o ha of RAID level array

2 D D D P D D X P D X D X D X D X D D D P D D D P Q X Q Q X Q Fig.. A rile failure reulig i a daa lo. wih he ame ace overhead. Secio V dicue oible imlemeaio ad Secio VI ha our cocluio. II. PREVIOUS WORK RAID array were he fir dik array orgaizaio o uilize eraure codig i order o roec daa agai dik failure [PGK88, G9, SB9, CL9]. While RAID level, ad oly olerae igle dik failure, RAID level 6 orgaizaio ue -ou-of- code o roec daa agai double dik failure [BM9]. EveOdd, Row- Diagoal Pariy ad he Liberaio Code are hree imlemeaio of RAID level ha ue oly XOR oeraio o coruc heir ariy iformaio [BB9, CE, GX8, P8a, P8b]. Huag ad Xu rooed a codig cheme correcig rile failure [HX]. Two-dimeioal RAID array, or d-pariy array, were iveigaed by Schwarz [S9] ad Hellerei e al. [HG9] who oed ha hee array oleraed all double dik failure bu did o iveigae how hey reaced o rile or quadrule dik failure. More recely, Lee aeed a wo-dimeioal dik array orgaizaio wih rom ariy udae i oe dimeio ad delayed ariy udae i he ecod dimeio [L]. Pâri e al. [PSL7] iveigaed wo-dimeioal RAID array ha reorgaized hemelve afer a dik failure ad oed ha all wo-dimeioal RAID array oleraed mo rile failure. Suerariy device were iroduced by Wildai e al. [WS9] i order o icreae he reliabiliy of archival orage yem for very lile co. Their rooal ariio each dik io fixed-ize dikle, which are ued o form coveioal RAID rie. Thee rie are he groued io larger ui, called uergrou, ad add o each uergrou oe or more uerariy device. Becaue uerariy dikle exeriece a high ud ae load, hey are imlemeed uig a faer, bu more exeive echology uch a flah drive. Whe here are failure, he yem rie o recover wihou uig he uerariy. If hi i imoible, he he yem hu dow ormal acce ad recover uig uerariy. The reao for he hu-dow i he high read load ha i eceary ice uergrou are quie large ad he yem i o execed o uor boh workload. Calculaio how he effecivee of iroducig he uergrou, bu alo ha he eed o have recoure o i rarely arie. Fig.. A Tye quadrule failure reulig i a daa lo. D D D P D X X P D D D P Q X X Fig.. A Tye quadrule failure reulig i a daa lo. D D D P D X X P D X X P Q Q Q Fig. 6. A Tye quadrule failure reulig i a daa lo. III. OUR PROPOSAL While wo-dimeioal RAID array roec daa agai all igle ad all double dik failure, ome rile dik failure will reul i a daa lo. A ee i Fig., hee faal rile failure coi of he failure of oe daa dik ad i wo ariy dik. Coider ow a wo-dimeioal array comriig daa dik ad ariy dik. Ou of he oible rile failure he array ca

3 D D D P D D D P D X D X D X D X D D D P D D D P Q X Q S Q X Q X Fig. 7. Recoverig from a reviouly faal rile failure. exeriece, exacly will be faal. A i ize grow, he raio bewee he umber of faal rile failure ad he oal umber of rile failure will quickly decreae: i become le ha erce for ad le ha. erce for 8. Le u ow coider which ye of quadrule failure will be faal. Thee ye will iclude:. The failure of a daa dik, i wo ariy dik ad ay oher dik: Fig. how of hee Tye failure.. The failure of wo daa dik i he ame row or colum ad heir ariy dik i he oher dimeio: Fig. how oe of hee Tye failure;. The failure of four daa dik formig a quare: Fig. 6 how oe of hee Tye failure. Ou of he oible quadrule failure he array ca exeriece, we ca eumerae:. Tye failure;. Tye failure;. Tye failure. A we oberved for rile failure, he erceage of faal quadrule failure alo decreae wih he ize of he array: i become le ha erce for ad le ha. erce for 8. Oe obviou way o icreae he reliabiliy of wodimeioal array would be o elimiae all faal rile failure ad reduce a much a oible he umber of faal quadrule failure. Le u how how i ca be doe a a fairly low co. Aume ha we add o he array a addiioal uerariy device coaiig he ariy of all daa dik. Oce he archive i able, hi ariy could be eaily comued by comuig he ariie of eiher Fig. 8. A Tye quadrule failure reulig i a daa lo.. All horizoal ariy dik, ha i, dik P o P i Fig. o 6, or. All verical ariy dik, ha i, dik Q o Q i Fig. o 6. I oher word we would have S P P P Q Q Q oe ha he uerariy device will oly be ued o recover he coe of failed ariy dik. While Wildai e al. meioed ha heir orgaizaio would have o read all daa dik i a uer-grou whe uig i uer-ariy device [WS9], our wo-dimeioal array will erform all i recovery oeraio o a row or a colum bai. Le u coider how he ew array would hadle he failure of a daa dik ad i wo ariy dik. A Fig. 7 how, he remaiig ariy dik ca ow recoiue he coe of he wo failed ariy dik wih he hel of he uerariy dik S by doig P P S P Q Q S Q While all faal rile failure would hu be elimiaed, ome quadrule failure would remai faal. Thee failure iclude:. The failure of a daa dik, i wo ariy dik ad he uerariy dik: Fig. 8 how oe of hee Tye failure;. The failure of wo daa dik i he ame row or colum ad heir ariy dik i he oher dimeio: hee failure are ideical o he Tye failure ha dicued before ;. The failure of four daa dik formig a quare: hee failure are ideical o he Tye failure ha we dicued before. Ou of he oible quadrule failure he array ca exeriece, we ca hu eumerae:. Tye faal failure,

4 . Tye faal failure,. Tye faal failure, for a oal of faal quadrule failure. A we oberved before, he fracio of faal quadrule failure alo decreae wih he ize of he array: i become le ha erce for ad le ha. erce for 8. Thi i four ime le ha he erceage of faal failure we oberved o he origial array wihou a uerariy device. Thi reducio i largely due o he lower umber of Tye faal quadrule failure, which were for he origial array bu oly for he array wih a uerariy device. Le u ow ur our aeio o quiule failure ad eumerae which oe will reul i a daa lo. Thee faal failure will coi of all quadrule faal failure lu ay oher dik. Ou of he oible quadrule failure he array ca exeriece, we ca hu eumerae faal failure. I hould come a o urrie ha he erceage of faal quiule failure alo decreae wih he ize of he array: i become le ha erce for ad le ha. erce for 8. IV. RELIABILITY AALYSIS Eimaig he reliabiliy of a orage yem mea eimaig he robabiliy R ha he yem will oerae correcly over he ime ierval [, ] give ha i oeraed correcly a ime. Comuig ha fucio require olvig a yem of liear differeial equaio, a ak ha become quickly iracable a he comlexiy of he yem grow. A imler oio i o ue iead he mea ime o daa lo MTTDL of he orage yem, which i he aroach we will ake here. A our aim i o meaure he imac of he uerariy device o wo-dimeioal array reliabiliy, we will ar by evaluaig he reliabiliy of wo-dimeioal array wihou uerariy device. A. Our Model Our yem model coi of a array of dik wih ideede failure mode. Wheever a dik fail, a reair roce i immediaely iiiaed for ha dik. Should everal dik fail, he reair roce will be erformed i arallel o hoe dik. We aume ha dik failure are ideede eve ad are exoeially diribued wih mea. I addiio, we require reair o be exoeially diribued wih mea. Boh hyohee are eceary o reree our yem by a Markov roce wih a fiie umber of ae. - -α- -β- α- β- Fig. 9. Sae raiio robabiliy diagram for a wo-dimeioal RAID array wih dik. Daa Lo - B. Array wihou uerariy device Fig. 9 dilay he ae raiio robabiliy diagram for a wo-dimeioal RAID array wih daa dik ad ariy dik for a oal of dik. Sae <> i he origial ae where all dik are oeraioal. Should oe of he dik fail, he yem would move o ae <> wih a aggregae failure rae. A ecod failure would brig he yem o ae <>. A hird failure would could eiher brig he yem o ae <> or reul i a daa lo. Give ha of he oible rile failure will reul i a daa lo, he wo failure raiio from ae <> are:. A raiio o he failure ae wih rae α where α. A raiio o ae <> wih rae α. Similarly, he wo failure raiio from ae <> will be:. A raiio o he failure ae wih rae β where β. A raiio o ae <> wih rae β. A we did o ake io accou he oibiliy ha he array could urvive a quiule failure, here i a igle failure raiio leavig ae <>. Recovery raiio are more raighforward: hey brig he array from ae <> o ae <>, he from ae <> o ae <> ad o o uil he yem reur o i origial ae <>.

5 - Daa Lo -δ- - -γ- - γ- δ- Fig.. Sae raiio robabiliy diagram for a wo-dimeioal RAID array wih dik. The Kolmogorov yem of differeial equaio decribig he behavior of he array i d d d d d d d d d d β α where i i he robabiliy ha he yem i i ae <i> wih he iiial codiio ad i for i. The Lalace raform of hee equaio are α β Obervig ha he mea ime o daa lo MTTDL of he array i give by i i MTTDL, we olve he yem of Lalace raform for ad a fixed value of he ue hi reul o comue he MTTDL of our yem. C. Addig a uerariy device Fig. dilay he ae raiio robably diagram for a wo-dimeioal RAID array wih daa dik, ariy dik ad a uerariy dik for a oal of dik. A we ca ee, here i a addiioal ae <> ad o failure raiio leavig ae <>. The wo failure raiio from ae <> are:. A raiio o he failure ae wih rae γ where γ. A raiio o ae <> wih rae γ. Similarly, he wo failure raiio from ae <> will be:. A raiio o he failure ae wih rae δ where δ. A raiio o ae <> wih rae δ. The Kolmogorov yem of differeial equaio decribig he behavior of he array i d d d d d d d d d d d d δ γ Uig he ame echique a before, we obai a algebraic exreio of he yem for a fixed value of. D. Comarig he wo orgaizaio Fig. dilay o a logarihmic cale he MTTDL achieved by wo-dimeioal RAID array wih 6 daa dik for average reair ime varyig bewee half a day ad eve day. The lower curve refer o a array wih 6 ariy block while he uer curve refer o he ame array wih a addiioal uerariy dik. We eleced hi array ize becaue i correoded o a ace overhead of erce for he array wih 6 daa ad 6 ariy dik, a overhead ha we aumed o be reaoable.

6 .E.E9.E8 Square 6 daa 7 ariy dik Square 6 daa 6 ariy dik.e7 MTTDL year.e6.e.e.e.e.e.e 6 7 Reair ime day Fig.. MTTDL of wo-dimeioal RAID array wih ad wihou a uerariy dik 7 v. 6 ariy dik reecively..e.e9.e8.e7 Square 6 daa 7 ariy dik Square 6 daa 6 ariy dik Sixee RAID-6 wih 6 dik each Eigh RAID-6 wih dik each MTTDL year.e6.e.e.e.e.e.e 6 7 Reair ime day Fig.. Comarig he MTTDL achieved by wo-dimeioal RAID array ad hoe achieved by e of ideede RAID level 6 rig.

7 TABLE I. RELATIVE MTTFS OF THE TWO SQUARE ORGAIZATIOS WITH REGARDS TO THOSE OF A SET OF 8 RAID LEVEL 6 ARRAYS WITH DISKS EACH. Reair Time Square 6D 7 P Square 6D 6P 8 RAID 6 Half a day Oe day.8.89 Two day Half a week Oe week We aumed a dik failure rae of oe failure every oe hudred houad hour, which i lighly le ha oe failure every eleve year. Thi rae i a he high ed of he failure rae oberved by Piheiro e al. [PWB7] a well a Schroeder ad Gibo [SG7]. MTTDL are exreed i year ad reair ime i day. A we ca ee, addig a uerariy dik o he origial array reul i dramaic icreae i i MTTDL. Thi effec i mo roouced for hor reair ime. Addig a uerariy dik o he array mulilie i MTTDL by a facor of a lea whe he reair ime remai below hree day ad half bu oly by a facor of 8 o whe he reair ake oe week o comlee. Thee imroveme are ruly remarkable a hey were achieved by addig a igle ariy dik o a array already couig 6 daa ad 6 ariy dik. E. Comarig he wo orgaizaio wih coveioal RAID array Fially, we comared our wo array orgaizaio wih e of RAID level 6 rig wih imilar orage caaciie ad comarable ace overhead. Thee wo orgaizaio are:. Eigh RAID level 6 array wih e dik each;. Sixee RAID level 6 array wih ix dik each. Their ace overhead are erce for he fir orgaizaio ad erce for he ecod. Obervig ha he MTTDL of a RAID level 6 wih dik i 6 MTTDL, we obai he MTTDL of a orgaizaio comriig m uch array uig he formula MTTDL MTTDL m; m Fig. ummarize our reul. A we ca ee, boh wo-dimeioal RAID array achieve much beer MTTDL ha he wo e of RAID level 6 rig eve hough oe of hem ha a much higher ace overhead ha our wo-dimeioal array. A before, we oberve ha our orgaizaio erform be a hor reair ime, ha i, a low / raio. The raio bewee he MTTDL afforded by our orgaizaio ad ha of ixee RAID level 6 array wih ix dik each varie bewee, whe he reair roce ake ju half a day ad 7 whe he roce ake a whole week. Thi i a very imreive achieveme whe we realize ha he wo dimeioal array wih a uerariy device ha a ace overhead of erce while he ace overhead of he RAID orgaizaio i. erce. The ga i eve wider whe we comare he erformace of our orgaizaio wih ha of he RAID orgaizaio comriig eigh RAID level 6 rie wih e dik each. A Table I how, he raio bewee he MTTDL afforded by he wo orgaizaio varie bewee,88 whe he reair roce ake ju half a day ad 69 whe i ake a whole week. V. IMPLEMETATIO ISSUES A we meioed earlier, he mai limiaio of our ew cheme i i oor wrie hroughu: ice he uerariy device coai he ariy of all daa dik, i mu be udaed each ime ay daa dik i udaed. Hece he wrie hroughu of he whole array i equal o he hroughu of i uerariy device. We already rooed oe way o miigae hi roblem: i coi of diablig uerariy device udae while he archive i oulaed a well a every ime a igifica fracio of he ored daa mu be udaed. The uerariy device would he lay he role of a lock ecurig he archived daa. We would ulock he archive wheever i i beig udaed ju a we oe a coveioal file cabie whe we wa o acce i coe. Oce he udae i comleed, we would udae he ariy device ad reur he archive o i locked ae. Eve he, he daa would remai roeced agai all oible double-dik failure ad mo rile failure, which i ill beer ha ay RAID level oluio. A a reul, he rocedure would have a miimal imac o he MTTDL of he array a log a i remai roeced by i ariy device mo of he ime. Oher oio are oible. Fir, we could ue orage cla memory for he ariy device. Thee ew device are execed o have acce ime of he order of ad acce rae bewee ad, megabye er ecod [7]. Their uerior erformace would grealy reduce he eed for diablig uerariy device udae. Uforuaely, he higher reliabiliy of hee device will o have a direc imac o he MTTDL of he array becaue mo faal quadrule failure are failure of Tye ad ha do o ivolve he uerariy device. We would ill oberve a mall icreae i he MTTDL of he array becaue orage cla memorie are immue o irrecoverable read error. A ecod oluio alie o all archive requirig more ha oe wo-dimeioal RAID array. We could form grou of k array uch ha he uerariy device of each array would coai /k of he uerariy daa of each array. Thi would mulily by k he maximum wrie hroughu of each array bu o he maximum wrie hroughu of he whole archive.

8 VI. COCLUSIO We have reeed a wo-dimeioal RAID archiecure ha i ecifically ailored o he eed of archival orage yem. Our rooal ar wih a fairly coveioal wo-dimeioal RAID archiecure where each dik belog o exacly oe horizoal ad oe verical RAID level rie. Oce he array ha bee oulaed, we add a uerariy device ha coai he excluive OR of all he coe of all horizoal or verical ariy dik. The ew orgaizaio olerae all rile dik failure ad early all quadrule ad quiule dik failure. A a reul, i rovide MTTDL A lea hiry ime beer ha hoe o a wodimeioal array wihou uerariy device; A lea 69 ime beer ha hoe of a e of RAID level 6 rie wih equal caaciy ad comarable ariy overhead. Our ex ak will be o cofirm our reul hrough dicree imulaio. Thi would allow u o coider imedeede failure rae ad arbirary reair ime diribuio ad would rovide u wih cofidece ierval for he robabiliy ha a give wo-dimeioal RAID array will oerae durig i ueful lifeime wihou exeriecig a daa lo. REFERECES [BG7] L. Bairavaudaram, G. Goodo, S. Pauahy, J. Schidler, A aalyi of lae ecor error i dik drive, Proc. ACM SIGMETRICS 7 I.Cof o Meaureme ad Modelig of Comuer Syem, 7. [BB9] M. Blaum, J. Brady, J. Bruck, ad J. Meo, EveOdd: A efficie cheme for oleraig double dik failure i RAID archiecure, IEEE Tra. Comuer :9, 99. [BM9] W. A. Burkhard ad J. Meo. Dik array orage yem reliabiliy. Proc. rd Ieraioal Sym. o Faul-Tolera Comuig,., Jue 99. [CE] P. Corbe, B. Eglih, A. Goel, T. Grcaac, S. Kleima, J. Leog, ad S. Sakar, Row-diagoal ariy for double dik failure correcio, Proc. USEIX Cof. o File ad Sorage Techologie,.,. [CL9] P. M. Che, E. K. Lee, G. A. Gibo, R. Kaz ad D. A. Paero. RAID, High-erformace, reliable ecodary orage, ACM Comuig Survey 6: 8, 99. [GX8] W. Gag, L. Xiaoguag, L. Sheg, X. Guagju, ad L. Jig, Geeralizig RDP code uig he combiaorial mehod, Proc. 7h IEEE I. Sym. o ework Comuig ad Alicaio,. 9, July 8. [HG9] L. Hellerei, G. Gibo, R. M. Kar, R. H. Kaz, ad D.A. Paero. Codig echique for hadlig failure i large dik array. Algorihmica, -:8-8, Jue 99 [HX] C. Huag ad L. Xu, STAR: a efficie codig cheme for correcig rile orage ode failure, Proc. h USEIX Cof. o File ad Sorage Techologie,. 97, Dec.. [L] W. S. Lee, Two-dimeioal orage array wih rom ariy i oe dimeio ad delayed ariy i a ecod dimeio, US Pae #6678 B, [7] S. araya, Sorage cla memory a diruive echology, Diruive Techologie Pael: Memory Syem of SC '7, Reo, V, ov. 7. [PGK88] D.A. Paero, G. Gibo, ad R.H. Kaz, A cae for [PSL7] [PA9] [P8a] [P8b] [PWB7] [SG7] reduda array of iexeive dik RAID. Proc. SIGMOD I. Cof. o Daa Maageme,. 9 6, Jue 988. J.-F. Pâri, T. J. Schwarz ad D. D. E. Log. Selfadaive archival orage yem. Proc. 6 h I. Performace of Comuer ad Commuicaio Cof.,. 6, Ar. 7. J.-F. Pâri ad A. Amer. Uig Shared Pariy Dik o Imrove he Reliabiliy of RAID Array, Proc. 8 h Ieraioal Performace of Comuer ad Commuicaio Cof., age 9 6, Dec. 9. J.S. Plak, A ew miimum deiy RAID-6 code wih a word ize of eigh, Proc. 7 h IEEE Ieraioal Sym. o ework Comuig ad Alicaio.,. 8 9, July 9. J. S. Plak, The RAID-6 liberaio code, Proc. 6h USEIX Cof. o File ad Sorage Techologie,., Feb. 8. E. Piheiro, W.-D. Weber ad L. A. Barroo, Failure red i a large dik drive oulaio, Proc. h USEIX Cof. o File ad Sorage Techologie,. 7 8, Feb. 7. B. Schroeder ad G. A. Gibo, Dik failure i he real world: wha doe a MTTF of,, hour mea o you? Proc. h USEIX Cof. o File ad Sorage Techologie,. 6, Feb. 7. [SB9] T. J. E. Schwarz ad W. A. Burkhard. RAID orgaizaio ad erformace. Proc. h Ieraioal Coferece o Diribued Comuig Syem ICDCS 9, Yokohama, Jaa,. 8 Jue 99. [SX] T. J. E. Schwarz, Q. Xi, E. L. Miller, D. D. E. Log, A. Hoodor ad S. g, Dik crubbig i large archival orage yem, Proc. h IEEE I. Sym. o Modelig, Aalyi, ad Simulaio of Comuer ad Telecommuicaio Syem,. 9-8, Oc.. [WS9] A. Wildai, T. J. E. Schwarz, E. L. Miller ad D. D. E. Log, Proecig agai rare eve failure i archival yem, Proc. 7 h IEEE I. Sym. o Modelig, Aalyi, ad Simulaio of Comuer ad Telecommuicaio Syem,. 6 6, Se. 9.