Time ECE7660 Memory.2 Fall 2005

Transcription

1 ECE7660 Advanced Computer Architecture Basics of Memory Hierarchy ECE7660 Memory.1 Fall 2005 Who Cares About the Memory Hierarchy? Processor-DRAM Memory Gap (latency) Performance Moore s Law CPU µproc 60%/yr. (2X/1.5yr) Processor-Memory Performance Gap (grows 50% / year) DRAM DRAM 9%/yr. (2X/10 yrs) Time ECE7660 Memory.2 Fall 2005

2 Memory Hierarchy of a Modern Computer System Processor Datapath Control Registers On-Chip Second Level (SRAM) Main Memory (DRAM) Secondary Storage (Disk) Level/Name Typical size Implementation technology Access time (ns) 1/RF < 1 KB Custom memory w. multiple ports, CMOS /C < 16 MB On-chip or off-chip CMOS SRAM /MM < 16 GB CMOS DRAM /DM > 100GB Magnetic disk 5,000,000 Bandwidth 20, ,000 (MB/s) ,000 (MB/s) (MB/s) (MB/s) Managed by Compiler Hardware Operating system OS/operator Backed by Main memory Disk CD or tape ECE7660 Memory.3 Fall 2005 Memory Hierarchy Principles of Operation At any given time, data is copied between only 2 adjacent levels Upper Level the one closer to the processor - Smaller, faster, and uses more expensive technology Lower Level the one further away from the processor - Bigger, slower, and uses less expensive technology Block The minimum unit of information that can either be present or not present in the two level hierarchy To Processor From Processor Upper Level Memory Blk X Lower Level Memory Blk Y ECE7660 Memory.4 Fall 2005

3 Memory Hierarchy Terminology Hit data appears in some block in the upper level (example Block X) Hit Rate the fraction of memory access found in the upper level Hit Time Time to access the upper level which consists of RAM access time + Time to determine hit/miss Miss data needs to be retrieve from a block in the lower level (Block Y) Miss Rate = 1 - (Hit Rate) Miss Penalty Time to replace a block in the upper level + Time to deliver the block to the processor Hit Time << Miss Penalty To Processor From Processor Upper Level Memory Block X Lower Level Memory Block Y ECE7660 Memory.5 Fall 2005 Memory Hierarchy How Does it Work? Principle of Locality Program access a relatively small portion of the address space at any instant of time. Example 90% of time in 10% of the code Temporal Locality (Locality in Time) If an item is referenced, it will tend to be referenced again soon. Keep more recently accessed data items closer to the processor Spatial Locality (Locality in Space) If an item is referenced, items whose addresses are close by tend to be referenced soon. Move blocks consists of contiguous words to the upper levels To Processor From Processor Upper Level Memory Blk X Lower Level Memory Blk Y ECE7660 Memory.6 Fall 2005

4 Memory Hierarchy Technology Random Access Random is good access time is the same for all locations DRAM Dynamic Random Access Memory - High density, low power, cheap, slow - Dynamic need to be refreshed regularly SRAM Static Random Access Memory - Low density, high power, expensive, fast - Static content will last forever Non-so-random Access Technology Access time varies from location to location and from time to time Examples Disk, tape drive, CDROM The lectures will concentrate on random access technology The Main Memory DRAMs s SRAMs ECE7660 Memory.7 Fall 2005 Memory Hierarchy System Virtual Memory System ECE7660 Memory.8 Fall 2005

5 What is a cache? Small, fast storage used to improve average access time to slow memory. Exploits spacial and temporal locality In computer architecture, almost everything is a cache! Registers a cache on variables First-level cache a cache on second-level cache Second-level cache a cache on memory Memory a cache on disk (virtual memory) TLB a cache on page table Branch-prediction a cache on prediction information? Proc/Regs Bigger L1- L2- Memory Faster Disk, Tape, etc. ECE7660 Memory.9 Fall 2005 The Simplest Direct Mapped Memory Address A B C D E F Memory 4 Byte Direct Mapped Index Location 0 can be occupied by data from Memory location 0, 4, 8,... etc. In general any memory location whose 2 LSBs of the address are 0s Address<10> => cache index (Block Addr) module (#cache blocks) How can we tell which one is in the cache? Which one should we place in the cache? ECE7660 Memory.10 Fall 2005

6 Example Consider an eight-word direct mapped cache. Show the contents of the cache as it responds to a series of requests (decimal addresses) 22, 26, 22, 26, 16, 3, 16, miss miss hit hit miss miss hit miss ECE7660 Memory.11 Fall 2005 Organization Tag and Index Assume a 32-bit memory (byte ) address N A 2 bytes direct mapped cache - Index The lower N bits of the memory address - Tag The upper (32 - N) bits of the memory address 31 N 0 Tag Example 0x50 Index Ex 0x03 Valid Bit Stored as part of the cache state 0x50 2 N Bytes Direct Mapped Byte 0 Byte 1 Byte 2 Byte 3 N Byte N 2-1 ECE7660 Memory.12 Fall 2005

7 Access Example Start Up V Tag Data Access (miss) Access (HIT) 000 M [00001] 010 M [01010] Miss Handling Load Data Write Tag & Set V Access (miss) 000 M [00001] Access (HIT) 000 M [00001] 010 M [01010] Load Data Write Tag & Set V 000 M [00001] 010 M [01010] Sad Fact of Life A lot of misses at start up Compulsory Misses - (Cold start misses) ECE7660 Memory.13 Fall 2005 Definition of a Block Block the cache data that has in its own cache tag Our previous extreme example (see slide 10) 4-byte Direct Mapped cache Block Size = 1 Byte Take advantage of Temporal Locality If a byte is referenced, it will tend to be referenced soon. Did not take advantage of Spatial Locality If a byte is referenced, its adjacent bytes will be referenced soon. In order to take advantage of Spatial Locality increase the block size Valid Tag Direct Mapped Data Byte 0 Byte 1 Byte 2 Byte 3 Question Can we say the larger, the better for the block size? ECE7660 Memory.14 Fall 2005

8 Example 1 KB Direct Mapped with 32 B Blocks For a 2 ** N byte cache The uppermost (32 - N) bits are always the Tag The lowest M bits are the Byte Select (Block Size = 2 ** M) 31 Tag Example 0x50 Stored as part of the cache state 9 Index Ex 0x01 4 Byte Select Ex 0x00 0 Valid Bit Tag 0x50 Data Byte 31 Byte 63 Byte 1 Byte 33 Byte 0 Byte Byte 1023 Byte ECE7660 Memory.15 Fall 2005 Mapping an Address to a Multiword Block Address Tag is divided into three fields tag, index, offset ttttttttttttttttt iiiiiiiiii oooo tag index byte to check to offset if have select within correct block block block Example Consider a cache with 64 Blocks and a block size of 16 bytes. What block number does byte address 1200 map to? [1200/16] module 64 = 11 ECE7660 Memory.16 Fall 2005

9 DM Example One Suppose we have a 16KB of data in a direct-mapped cache with 4 word blocks Determine the size of the tag, index and offset fields if we re using a 32-bit architecture # rows/cache =# blocks/cache (since there s one block/row) = 2 10 rows/cache need 10 bits to specify this many rows Tag use remaining bits as tag tag length = mem addr length - offset - index = bits = 18 bits so tag is leftmost 18 bits of memory address Why not full 32 bit address as tag? All bytes within block need same address (-4b) Index must be same for every address within a block, so its redundant in tag check, thus can leave off to save memory (- 10 bits in this example) ECE7660 Memory.17 Fall 2005 DM Example Two (Bits in a cache) How many total bits are required for a directed mapped cache with 64 KB of data and one-word blocks, assuming a 32-bit address? 2^14 (32 + ( )+1) = 2^14*49 = 784 Kbits ECE7660 Memory.18 Fall 2005

10 Block Size Tradeoff In general, larger block size take advantage of spatial locality BUT Larger block size means larger miss penalty - Takes longer time to fill up the block If block size is too big relative to cache size, miss rate will go up Average Access Time = Hit Time x (1 - Miss Rate) + Miss Penalty x Miss Rate Miss Penalty Miss Rate Exploits Spatial Locality Fewer blocks compromises temporal locality Average Access Time Increased Miss Penalty & Miss Rate Block Size Block Size Block Size ECE7660 Memory.19 Fall 2005 Another Extreme Example Valid Bit Tag Data Byte 3 Byte 2 Byte 1 Byte 0 0 Size = 4 bytes Only ONE entry in the cache Block Size = 4 bytes True If an item is accessed, likely that it will be accessed again soon But it is unlikely that it will be accessed again immediately!!! The next access will likely to be a miss again - Continually loading data into the cache but discard (force out) them before they are used again - Worst nightmare of a cache designer Ping Pong Effect Conflict Misses are misses caused by Different memory locations mapped to the same cache index - Solution 1 make the cache size bigger - Solution 2 Multiple entries for the same Index ECE7660 Memory.20 Fall 2005

11 How Do you Design a? Set of Operations that must be supported read data <= Mem[Physical Address] write Mem[Physical Address] <= Data Physical Address Read/Write Data Memory Black Box Inside it has Tag-Data Storage, Muxes, Comparators,... Determine the internal register transfers Design the Datapath Design the Controller Address Data In Data Out DataPath Control Points Signals Controller ECE7660 Memory.21 Fall 2005 R/W Active wait Impact on time I - PC CPI = Ideal CPI + memory stall cycles CPU time = (CPU execution clock cycles + Memory stall cycles) x clock cycle time miss invalid IR IRex IRm A R B Memory stall clock cycles = (Reads x Read miss rate x Read miss penalty + Writes x Write miss rate x Write miss penalty) IRwb D T Miss Memory stall clock cycles = Memory accesses x Miss rate x Miss penalty Average Memory Access time (AMAT) = Hit Time + (Miss Rate x Miss Penalty) Memory hit time is included in execution cycles. ECE7660 Memory.22 Fall 2005

12 Example 1 Assume an instruction cache miss rate for gcc of 2% and a data cache miss rate of 4%. If a machine has a CPI of 2 without any memory stalls and the miss penalty is 40 cycles for all misses, determine how much faster a machine would run with a perfect cache that never missed. (It is known that the frequency of all loads and stores in gcc is 36%, unused) Answer Instruction miss cycles = I * 2% * 40 = 0.80I Data miss cycles = I * 36% * 4% * 40 = 0.56I The CPI with memory stalls is = 3.36 The performance with the perfect cache is better by 1.68 ECE7660 Memory.23 Fall 2005 Example 2 Suppose a processor executes at Clock Rate = 200 MHz (5 ns per cycle), Ideal (no misses) CPI = % arith/logic, 30% ld/st, 20% control Suppose that 10% of memory operations get 50 cycle miss penalty Suppose that 1% of instructions get same miss penalty Questions what is CPI? What s the percentage of proc time stalled waiting for memory? What is average memory access time? CPI = ideal CPI + average stalls per instruction 1.1(cycles/ins) + [ 0.30 (DataMops/ins) x 0.10 (miss/datamop) x 50 (cycle/miss)] + [ 1 (InstMop/ins) x 0.01 (miss/instmop) x 50 (cycle/miss)] = ( ) cycle/ins = 3.1 Percentage of the memory stall time = ( )/3.1 = 64.5% AMAT=(1/1.3)x[1+0.01x50]+(0.3/1.3)x[1+0.1x50]=2.54 cycles/access ECE7660 Memory.24 Fall 2005

13 Impact on Time Vector Product Example float dot_prod(float x[1024], y[1024]) { float sum = 0.0; int i; for (i = 0; i < 1024; i++) sum += x[i]*y[i]; return sum; } Machine Processor with 64KB direct-mapped cache, 16 B line size Performance Good case 24 cycles / element Bad case 66 cycles / element ECE7660 Memory.25 Fall 2005 Thrashing Example x[0] x[1] x[2] x[3] Line y[0] y[1] y[2] y[3] Line Line Line x[1020] x[1021] x[1022] x[1023] Line y[1020] y[1021] y[1022] y[1023] Line Access one element from each array per iteration ECE7660 Memory.26 Fall 2005

14 Thrashing Example Good Case x[0] x[1] x[2] x[3] y[0] y[1] y[2] y[3] Line Access Sequence Read x[0] - x[0], x[1], x[2], x[3] loaded Read y[0] - y[0], y[1], y[2], y[3] loaded Read x[1] - Hit Read y[1] - Hit 2 misses / 8 reads Analysis x[i] and y[i] map to different cache lines Miss rate = 25% - Two memory accesses / iteration - On every 4th iteration have two misses Timing 10 cycle loop time 28 cycles / cache miss Average time / iteration = * 2 * 28 ECE7660 Memory.27 Fall 2005 Thrashing Example Bad Case x[0] x[1] x[2] x[3] y[0] y[1] y[2] y[3] Line Access Pattern Read x[0] - x[0], x[1], x[2], x[3] loaded Read y[0] - y[0], y[1], y[2], y[3] loaded Read x[1] - x[0], x[1], x[2], x[3] loaded Read y[1] - y[0], y[1], y[2], y[3] loaded 8 misses / 8 reads Analysis x[i] and y[i] map to same cache lines Miss rate = 100% - Two memory accesses / iteration - On every iteration have two misses Timing 10 cycle loop time 28 cycles / cache miss Average time / iteration = * 2 * 28 ECE7660 Memory.28 Fall 2005

15 General Approaches to Improve Perf. Average Memory Access time (AMAT) = Hit Time + (Miss Rate x Miss Penalty) 1. Reduce the miss rate, I - PC 2. Reduce the miss penalty, or miss IR 3. Reduce the time to hit in the cache. invalid IRex A B IRm IRwb D Miss R T ECE7660 Memory.29 Fall 2005 Design to Reduce Miss Rate Four Key Questions Q1 Where can a block be placed in the upper level? (Block placement) Fully Associative, Set Associative, Direct Mapped Q2 How is a block found if it is in the upper level? (Block identification) Tag/Block Q3 Which block should be replaced on a miss? (Block replacement) Random, LRU Q4 What happens on a write? (Write strategy) Write Back or Write Through (with Write Buffer) ECE7660 Memory.30 Fall 2005

16 Where can a block be place? Block 12 placed in 8 block cache Fully associative, direct mapped, 2-way set associative S.A. Mapping = Block Number Modulo Number Sets Block no. Fully associative block 12 can go anywhere Block no. Direct mapped block 12 can go only into block 4 (12 mod 8) Block no. Set associative block 12 can go anywhere in set 0 (12 mod 4) Block-frame address Set 0 Set 1 Set Set 2 3 Block no ECE7660 Memory.31 Fall 2005 And yet Another Extreme Example Fully Associative Fully Associative Forget about the Index Compare the Tags of all cache entries in parallel Example Block Size = 32 B blocks, we need N 27-bit comparators By definition Conflict Miss = 0 for a fully associative cache 31 Tag (27 bits long) 4 Byte Select Ex 0x01 0 X X X X Tag Valid Bit Data Byte 31 Byte 63 Byte 1 Byte 33 Byte 0 Byte 32 X ECE7660 Memory.32 Fall 2005

17 A Two-way Set Associative N-way set associative N entries for each Index N direct mapped caches operates in parallel Valid Example Two-way set associative cache Index selects a set from the cache The two tags in the set are compared in parallel Data is selected based on the tag result Tag Data Block 0 Index Data Block 0 Tag Valid Adr Tag Compare Sel1 1 Mux 0 Sel0 Compare OR Hit Block ECE7660 Memory.33 Fall 2005 Disadvantage of Set Associative N-way Set Associative versus Direct Mapped N comparators vs. 1 Extra MUX delay for the data Data comes AFTER Hit/Miss In a direct mapped cache, Block is available BEFORE Hit/Miss Possible to assume a hit and continue. Recover later if miss. Valid Tag Data Block 0 Index Data Block 0 Tag Valid Adr Tag Compare Sel1 1 Mux 0 Sel0 Compare OR Block Hit ECE7660 Memory.34 Fall 2005

18 Sources of Misses Compulsory (cold start, first reference) first access to a block Cold fact of life not a whole lot you can do about it Conflict (collision) Multiple memory locations mapped to the same cache location Solution 1 increase cache size Solution 2 increase associativity Capacity cannot contain all blocks access by the program Solution increase cache size Invalidation other process (e.g., I/O, processors) updates memory ECE7660 Memory.35 Fall 2005 The Need to Make a Decision! Direct Mapped Each memory location can only mapped to 1 cache location No need to make any decision -) - Current item replaced the previous item in that cache location N-way Set Associative Each memory location have a choice of N cache locations Fully Associative Each memory location can be placed in ANY cache location miss in a N-way Set Associative or Fully Associative Bring in new block from memory Throw out a cache block to make room for the new block ====> We need to make a decision on which block to throw out! ECE7660 Memory.36 Fall 2005

19 Block Replacement Policy Random Replacement Hardware randomly selects a cache item and throw it out Least Recently Used Hardware keeps track of the access history Replace the entry that has not been used for the longest time First-in, first out (FIFO) replace the oldest block Example of a Simple Pseudo Least Recently Used Implementation Assume 64 Fully Associative Entries Hardware replacement pointer points to one cache entry Whenever an access is made to the entry the pointer points to - Move the pointer to the next entry Otherwise do not move the pointer Replacement Pointer Entry 0 Entry 1 ECE7660 Memory.37 Fall 2005 Entry 63 Write Policy Write Through versus Write Back read is much easier to handle than cache write Instruction cache is much easier to design than data cache write How do we keep data in the cache and memory consistent? Two options (decision time again -) Write Back write to cache only. Write the cache block to memory when that cache block is being replaced on a cache miss. - Need a dirty bit for each cache block, so as to the frequency of writing back blocks - Greatly reduce the memory bandwidth requirement - Control can be complex Write Through write to cache and memory at the same time. - The cache is always clean - What!!! How can this be? Isn t memory too slow for this? ECE7660 Memory.38 Fall 2005

20 Write Buffer for Write Through Processor DRAM Write Buffer Write stall the time when cpu s waiting for writes to complete A Write Buffer is needed between the and Memory Processor writes data into the cache and the write buffer Memory controller write contents of the buffer to memory Write buffer is just a FIFO Typical number of entries 4 Works fine if Store frequency (w.r.t. time) << 1 / DRAM write cycle Memory system designer s nightmare Store frequency (w.r.t. time) > 1 / DRAM write cycle Write buffer saturation ECE7660 Memory.39 Fall 2005 Write Buffer Saturation Processor DRAM Write Buffer Store frequency (w.r.t. time) > 1 / DRAM write cycle If this condition exist for a long period of time (CPU cycle time too quick and/or too many store instructions in a row) - Store buffer will overflow no matter how big you make it - The CPU Cycle Time <= DRAM Write Cycle Time Solution for write buffer saturation Use a write back cache Install a second level (L2) cache Processor L2 DRAM Write Buffer ECE7660 Memory.40 Fall 2005

21 Write Allocate versus Not Allocate Assume a 16-bit write to memory location 0x0 and causes a Write Miss Do we read in the rest of the block (Byte 2, 3,... 31)? Yes Write Allocate No Write Not Allocate 31 Tag Example 0x00 9 Index Ex 0x00 4 Byte Select Ex 0x00 0 Valid Bit Tag 0x00 Data Byte 31 Byte 63 Byte 1 Byte 33 Byte 0 Byte Byte 1023 Byte ECE7660 Memory.41 Fall 2005 How important are caches? Floorplan Register files in middle of execution units 64k instr cache 64k data cache s take up a large fraction of the die (Figure from Jim Keller, Compaq Corp.) ECE7660 Memory.42 Fall 2005

22 The Processor Itself ECE7660 Memory.43 Fall 2005 Summary of Basics The Principle of Locality Temporal Locality vs Spatial Locality Four Questions For Any Where to place in the cache How to locate a block in the cache Replacement Write policy Write through vs Write back - Write miss Three Major Categories of Misses Compulsory Misses sad facts of life. Example cold start misses. Conflict Misses increase cache size and/or associativity. Nightmare Scenario ping pong effect! Capacity Misses increase cache size ECE7660 Memory.44 Fall 2005