US008612715B2 (12) United States Patent Mine et al. (10) Patent N0.: (45) Date of Patent: US 8,612,715 B2 *Dec. 17, 2013 (54) (75) (73) (21) (22) (65) (63) (51) (52) (58) STORAGE SYSTEM AND UTILIZATION MANAGEMENT METHOD FOR STORAGE SYSTEM Inventors: Hiroshi Mine, Yokohama (JP); Hitoshi Kamei, Sagamihara (JP); Takahiro Nakano, Yokohama (JP) Assignee: Hitachi, Ltd., Tokyo (JP) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 USC 154(b) by 0 days. This patent is subject to a terminal dis claimer. Appl. N0.: 13/529,626 Filed: Jun. 21, 2012 Prior Publication Data US 2012/0265961A1 Oct. 18,2012 Related US. Application Data Continuation of application No. 12/526,779,?led as application No. PCT/JP2009/002074 on May 13, 2009, now Pat. No. 8,225,066. Int. Cl. G06F 12/02 US. Cl. (2006.01) USPC..... 711/171; 711/E12.002 Field of Classi?cation Search None See application?le for complete search history. (56) References Cited 2006/0069665 A1 2006/0155950 A1 2008/0155214 A1 2010/0095164 A1 U.S. PATENT DOCUMENTS 3/2006 Yamakawa et al. 7/2006 Smith 6/2008 Shitomi 4/2010 Kamei et al. FOREIGN PATENT DOCUMENTS EP 1 770 499 Al 4/2007 JP 2006-092322 4/2006 JP 2006-195960 7/2006 JP 2008-159027 7/2008 WO WO-00/4l5l0 7/2000 OTHER PUBLICATIONS PCT International Search Report on International Application No. PCT/JP2009/002074 dated Dec. 7, 2009; 3 pages. Primary Examiner * Kevin Verbrugge (74) Attorney, Agent, or Firm * Foley & Lardner LLP (57) ABSTRACT A storage system 1000 providing a storage area to an external device 1500 comprises a plurality of storage apparatuses 1100, 1200 and 1300 of different performance capabilities, and a storage controller 2600. The storage controller 2600 holds storage cost coe?icients 3202 of the respective storage apparatuses 1100, 1200 and 1300, user cost allocation infor mation 3302 for eachuserusing the storage system 1000 from the external device 1500, and user cost distribution informa tion 3402 that de?nes, for each user, proportions of an avail able space amount distributed to the storage apparatuses 1100, 1200 and 1300. The storage controller 2600 calculates an ideal utilization 3502 for each user, from the storage cost coef?cients 3202 and the user cost distribution information 3402, and allocates, for each user, the ideal utilization 3502 to each ofthe storage apparatuses 1100, 1200 and 1300, in the order of the performance capabilities from highest to lowest. 18 Claims, 21 Drawing Sheets 1 1500 1500 1500 / CLIENT DEVICE CLIENT DEVICE CLIENT DEVICE 1400 NETWORK / HIERARCHICAL STORAGE C0NTR0LLER I r2000 HIERARCHICAL STORAGE SYSTEM
US. Patent Dec. 17, 2013 Sheet 2 0121 US 8,612,715 B2 STORAGE APPARATUS /200 COMMUNICATION UP /211,210 212 K230 / K214 CPU r213 CACHE MEMORY SHARED MEMORY DISK CONTROLLER 215 216 DISK VF / MANAGEMENT / l/f DISK DEv1OE I420 / 221 221 221 FIG. 2
US. Patent Dec. 17, 2013 Sheet 3 0f 21 US 8,612,715 B2 11 12 13 COMPUTER N W\ CENTRAL PROCESSOR INPUT DEVICE /W 14 \ NAIN STORACE DEVICE OUTPUT DEVICE /W 15 \ AUXILIARY STORAGE DEVICE COMMUNICATION INTERFACE /W 16 10 FIG. 3
US. Patent Dec. 17, 2013 Sheet 4 0121 US 8,612,715 B2 300 / I-IIERARGI-IIGAL f 2000 STORAGE OONTROLLER NETWORK r 2400 INTERFAOE MEMORY / / 3000 {2100 HIERARCHIOAL STORAGE,/, UTILIZATION MANAGEMENT TABLE PROCESSOR fz?oo I-IIERAROI-IIOAL STORAGE,/ 2300 CONTROL PROGRAM / J/ 2500 DISK INTERFAGE BASIC PROGRAM I FIG. 4A
US. Patent Dec. 17, 2013 Sheet 7 0121 US 8,612,715 B2 INITIAL DISTRIBUTION f 84000 PROCESSING l IDEAL UTILIZATION CALCU LATION PROCESSING I INITIAL CAPACITY CHECK PROCESSING 1 END fssooo fsbooo FIG. 6
US. Patent Dec. 17, 2013 Sheet 8 0121 US 8,612,715 B2 IDEAL UTILIZA f 85000 CALCULATION CESSING 1 P 1: REPEAT PROCE G FROM ER LOWER-LEVEL STORAGE AP ATU SET ID UTILIZATION BY fs5100 DISTRI ON VALUE >< COST COEFFICIENT l LOOP 1 END FIG. 7
US. Patent Dec. 17, 2013 Sheet 10 0121 US 8,612,715 B2 DISTRIBUTION CHANGE PROCESSING f $7000 IDEAL UTILIZATION CALCULATION PROCESSING f55000 CAPACITY CHANGE f$8000 CHECK PROCESSING FILE MIGRATION PROCESSING END FIG. 9
US. Patent Dec. 17, 2013 Sheet 12 0121 US 8,612,715 B2 # # 00261.. Qzm :.UE 8 \ WE mm;
US. Patent Dec. 17, 2013 Sheet 13 0121 US 8,612,715 B2 FILE MIGRATION PROCESSING l fsmooo / LOOP 4; \ REPEAT PROCESSING FROM UPPER- TO LOWER-LEVEL STORAGE APPARATUSES I MIGRATE MIGRATION TARGET FILE IN UPPER-LEVEL STORAGE TO TARGET STORAGE I SUBTRACT SIZE OF MIGRATION FILE FROM ACTUAL UTILIZATION OF UPPER-LEVEL STORAGE I ADD SIZE OF MIGRATION FILE TO ACTUAL UTILIZATION OF TARGET STORAGE I LOOP 4 fsmmo fsiozoo fsmoo END FIG. 12
US. Patent Dec. 17, 2013 Sheet 15 0121 US 8,612,715 B2 # % 3.0E comm Pm. mm; % Oz
US. Patent Dec. 17, 2013 Sheet 16 0121 US 8,612,715 B2 coon Fm 00% Pm. M Qzm 1,. 85w $> \ BE 65% cog Fm czammooma
US. Patent Dec. 17, 2013 Sheet 17 0121 US 8,612,715 B2 CAPACITY ADDITION PROCESSING f $14000 " f $14100 ADD FREE SPACE AMOUNT $14200 NO ACTUAL UTILIZATION < IDEAL UTILIZATION? SPACE AMOUNT NOT IN USE I 814300 E IDEAL UTILIzATIoN - ACTUAL UTILIZATION $14400 SPACE AMOUNT NOT IN USE ; $14500 FREE SPACE AMOUNT? NO I / space ANICUAT NOT IN USE <- FREE SPACE AMOUNT YES EXECUTE FILE-MIGRATION-UPON-DELETION I 813000 PROCESSING ON LOWER-LEVEL STORAGE I END FIG. 16
US. Patent Dec. 17, 2013 Sheet 18 0121 US 8,612,715 B2 :.UE com? \
US. Patent Dec. 17, 2013 Sheet 19 0f 21 US 8,612,715 B2 DEDUPLIOATION-CAPABLE HIERARCHICAL STORAGE GONTROLLER f10000 f 2200 Ii'l'TTEvgg/i ge 2400 MEMORY /r HIERARCHICAL STORAGE // 3000 2100 UTILIZATION - 1 MANAGEMENT TABLE CPU /16100 DEDUPLICATION-CAPABLE / 2300 HIERARCHICAL STORAGE I OONTROL PROGRAM /2500 DISK INTERFACE BASIC PROGRAM // FIG. 18