1 / 45

Caches

Caches. Vincent H. Berk October 21, 2005 Reading for Wednesday: Sections 5.1 – 5.4 Reading for Friday: Sections 5.5 – 5.8 (Jouppi article). CPU-DRAM Gap. Who Cares about the Memory Hierarchy?. So far, have discussed only processor CPU Cost/Performance, ISA, Pipelined Execution, ILP

rajah-decker
Download Presentation

Caches

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ENGS 116 Lecture 12 Caches Vincent H. Berk October 21, 2005 Reading for Wednesday: Sections 5.1 – 5.4 Reading for Friday: Sections 5.5 – 5.8 (Jouppi article)

  2. CPU-DRAM Gap ENGS 116 Lecture 12 Who Cares about the Memory Hierarchy? • So far, have discussed only processor • CPU Cost/Performance, ISA, Pipelined Execution, ILP • 1980: no cache in microprocessors • 1995: 2-level cache, 60% transistors on Alpha 21164 • 2002: IBM experimenting with Main Memory on die.

  3. MainMemory Processor Cache ENGS 116 Lecture 12 The Motivation for Caches Memory System • Motivation: • Large memories (DRAM) are slow • Small memories (SRAM) are fast • Make the average access time small by servicing most accesses from a small, fast memory • Reduce the bandwidth required of the large memory

  4. ENGS 116 Lecture 12 Principle of Locality of Reference • Programs do not access their data or code all at once or with equal probability • Rule of thumb: Program spends 90% of its execution time in only 10% of the code • Programs access a small portion of the address space at any one time • Programs tend to reuse data and instructions that they have recently used • Implication of locality: Can predict with reasonable accuracy what instructions and data a program will use in the near future based on its accesses in the recent past

  5. ENGS 116 Lecture 12 Memory System Illusion Reality Processor Processor Memory Memory Memory Memory

  6. ENGS 116 Lecture 12 General Principles • Locality • Temporal Locality: referenced again soon • Spatial Locality: nearby items referenced soon • Locality + smaller HW is faster  memory hierarchy • Levels: each smaller, faster, more expensive/byte than level below • Inclusive: data found in top also found in lower levels • Definitions • Upper is closer to processor • Block: minimum, address aligned unit that fits in cache • Address = Block frame address + block offset address • Hit time: time to access upper level, including hit determination

  7. ENGS 116 Lecture 12 Cache Measures • Hit rate: fraction of accesses found in that level • So high that we usually talk about the miss rate • Miss rate fallacy: miss rate induces miss penalty, determines average memory performance • Average memory-access time (AMAT) = Hit time + Miss rate  Miss penalty (ns or clocks) • Miss penalty: time to replace a block from lower level, including time to copy to and restart CPU – access time: time to lower level = ƒ(lower level latency) – transfer time: time to transfer block = ƒ(BW upper & lower, block size)

  8. ENGS 116 Lecture 12 Block Size vs. Cache Measures • Increasing block size generally increases the miss penalty Miss Penalty Miss Rate Avg. Memory Access Time  => Miss penalty Transfer time Miss rate Average access time Access time Block Size Block Size Block Size

  9. ENGS 116 Lecture 12 Key Points of Memory Hierarchy • Need methods to give illusion of large, fast memory • Programs exhibit both temporal locality and spatial locality • Keep more recently accessed data closer to the processor • Keep multiple contiguous words together in memory blocks • Use smaller, faster memory close to processor – hits are processed quickly; misses require access to larger, slower memory • If hit rate is high, memory hierarchy has access time close to that of highest (fastest) level and size equal to that of lowest (largest) level

  10. ENGS 116 Lecture 12 Implications for CPU • Fast hit check since every memory access needs this check • Hit is the common case • Unpredictable memory access time • 10s of clock cycles: wait • 1000s of clock cycles: (Operating System) » Interrupt & switch & do something else » Lightweight: multithreaded execution

  11. ENGS 116 Lecture 12 Four Memory Hierarchy Questions • Q1: Where can a block be placed in the upper level? (Block placement) • Q2: How is a block found if it is in the upper level? (Block identification) • Q3: Which block should be replaced on a miss? (Block replacement) • Q4: What happens on a write? (Write strategy)

  12. 0 0 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 1 1 1 1 1 1 1 1 1 1 3 3 2 2 2 2 2 2 2 2 2 2 1 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 0 0 1 2 3 4 5 6 7 8 9 ENGS 116 Lecture 12 Q1: Where can a block be placed in the cache? Fully associative: block 12 can go anywhere Direct mapped: block 12 can go only into block 4 (12 mod 8) Set associative: block 12 can go anywhere in set 0 (12 mod 4) • Block 12 placed in 8 block cache: • Fully associative, direct mapped, 2-way set associative • S.A. Mapping = Block number modulo number sets Block no. Block no. Block no. Cache Set 0 Set 3 Set 1 Set 2 Block frame address Block no. Memory

  13. 000 001 010 011 100 101 110 111 00000 00100 01000 01100 10000 10100 11000 11100 ENGS 116 Lecture 12 Direct Mapped Cache • Each memory location is mapped to exactly one location in the cache • Cache location assigned based on address of word in memory • Mapping: (address of block) mod (# of blocks in cache)

  14. ENGS 116 Lecture 12 Associative Caches • Fully Associative: block can go anywhere in the cache • N-way Set Associative: block can go in one of N locations in the set

  15. Block Address Block Offset Index Tag ENGS 116 Lecture 12 Q2: How is a block found if it is in the cache? • Tag on each block • No need to check index or block offset • Increasing associativity shrinks index, expands tag Fully Associative: No index Direct Mapped: Large index

  16. ENGS 116 Lecture 12 Examples • 512-byte cache, 4-way set associative, 16-byte blocks, byte addressable • 8-KB cache, 2-way set associative, 32-byte blocks, byte addressable

  17. ENGS 116 Lecture 12 Q3: Which block should be replaced on a miss? • Easy for direct mapped • Set associative or fully associative: • Random (large associativities) • LRU (smaller associativities) • FIFO (large associativities) Associativity: 2-way 4-way Size LRU Random FIFO LRU Random FIFO 16 KB 114.1 117.3 115.5 111.7 115.1 113.3 64 KB 103.4 104.3 103.9 102.4 102.3 103.1 256 KB 92.2 92.1 92.5 92.1 92.1 92.5

  18. ENGS 116 Lecture 12 Q4: What Happens on a Write? • Write through: The information is written to both the block in the cache and to the block in the lower-level memory. • Write back: The information is written only to the block in the cache. The modified cache block is written to main memory only when it is replaced. • Is block clean or dirty? • Pros and Cons of each: • WT: read misses cannot result in writes (because of replacements) • WB: no writes of repeated writes • WT always combined with write buffers so that we don’t wait for lower level memory • WB write buffer, giving a read-miss precedence

  19. Block offset Block address <21> <8> <5> Index Tag CPU address Data Data in out Valid<1> Tag <21> Data <256> 3 =? 1 4 (256 Blocks) 4:1 MUX 2 • • • • • • Write buffer Lower Level Memory ENGS 116 Lecture 12 Example: 21064 Data Cache Direct Mapped • Index = 8 bits: 256 blocks = 8192/(32  1)

  20. Block offset Block address <22> <7> <5> Data <64> Index Tag CPU address Data Data in out Valid<1> Tag <21> =? =? 2:1 MU X • • • • • • • • • • • • Write buffer Lower Level Memory ENGS 116 Lecture 12 2-way Set Associative,Address to Select Word Two sets of address tags and data RAM 2:1 mux selects data Use address bits to select correct RAM

  21. ENGS 116 Lecture 12 Structural Hazard: Instruction and Data? Size Instruction Cache Data Cache Unified Cache 8 KB 8.16 44.0 63.0 16 KB 3.82 40.9 51.0 32 KB 1.36 38.4 43.3 64 KB 0.61 36.9 39.4 128 KB 0.30 35.3 36.2 256 KB 0.02 32.6 32.9 Misses per 1000 instructions Mix: instructions 74%, data 26%

  22. ENGS 116 Lecture 12 Cache Performance includes hit time CPU time = (CPU execution clock cycles + Memory-stall clock cycles)  Clock cycle time Memory-stall clock cycles = Read-stall cycles + Write-stall cycles = =

  23. ENGS 116 Lecture 12 Cache Performance CPU time = IC  (CPIexecution + Mem accesses per instruction  Miss rate  Miss penalty)  Clock cycle time Misses per instruction = Memory accesses per instruction  Miss rate CPU time = IC  (CPIexecution + Misses per instruction  Miss penalty)  Clock cycle time

  24. ENGS 116 Lecture 12 Summary of Cache Basics • Associativity • Block size (cache line size) • Write Back/Write Through, write buffers, dirty bits • AMAT as a basic performance measure • Larger block size decreases miss rate but can increase miss penalty • Can increase bandwidth of main memory to transfer cache blocks more efficiently • Memory system can have significant impact on program execution time, memory stalls can be over 100 cycles • Faster processors  memory stalls more costly

  25. ENGS 116 Lecture 12 Improving Cache Performance • Average memory-access time (AMAT) = Hit time + Miss rate  Miss penalty (ns or clocks) • Improve performance by: 1. Reducing the miss penalty (5.4) 2. Reducing the miss rate (5.5) 3. Reducing through parallelism (5.6) 4. Reducing the time to hit in the cache (5.7)

  26. ENGS 116 Lecture 12 Reducing Miss Penalty • Multilevel Caches • Critical Word First and Early Restart • Read Misses over Writes • Merging Write Buffer • Victim Caches • Subblock Placement

  27. ENGS 116 Lecture 12 1. Reduce Miss Penalty: L2 Caches • L2 Equations AMAT = Hit TimeL1 + Miss RateL1 Miss PenaltyL1 Miss PenaltyL1 = Hit TimeL2 + Miss RateL2 Miss PenaltyL2 AMAT = Hit TimeL1 + Miss RateL1 (Hit TimeL2 + Miss RateL2 Miss PenaltyL2) • Definitions: • Local miss rate — misses in this cache divided by the total number of memory accesses to this cache (Miss rateL2) • Global miss rate — misses in the cache divided by the total number of memory accesses generated by the CPU(Miss RateL1 Miss RateL2) • Global miss rate is what matters —indicates what fraction of memory accesses from CPU go all the way to main memory

  28. Linear Cache Size Log Cache Size ENGS 116 Lecture 12 Comparing Local and Global Miss Rates • 32 KByte 1st level cache;Increasing 2nd level cache • Global miss rate close to single level cache rate provided L2 >> L1 • Don’t use local miss rate • L2 not tied to CPU clock cycle! • Cost & A.M.A.T. • Generally fast hit times and fewer misses • Since hits are few, target miss reduction

  29. ENGS 116 Lecture 12 L2 cache block size & A.M.A.T. • 32KB L1, 8-byte pathto memory

  30. block ENGS 116 Lecture 12 2 . Reduce Miss Penalty: Early Restart and Critical Word First • Don’t wait for full block to be loaded before restarting CPU • Early restart — As soon as the requested word of the block arrives, send it to the CPU and let the CPU continue execution • Critical Word First — Request the missed word first from memory and send it to the CPU as soon as it arrives; let the CPU continue execution while filling the rest of the words in the block. Also called wrapped fetch and requested word first. • Generally useful only in large blocks, • Spatial locality a problem; tend to want next sequential word, so not clear if benefit by early restart

  31. ENGS 116 Lecture 12 3. Reduce Miss Penalty: Read Priority over Write on Miss • Write through with write buffers offer RAW conflicts with main memory reads on cache misses • If simply wait for write buffer to empty, might increase read miss penalty (old MIPS 1000 by 50%) • Check write buffer contents before read; if no conflicts, let the memory access continue • Write Back? • Read miss replacing dirty block • Normal: Write dirty block to memory, and then do the read • Instead copy the dirty block to a write buffer, then do the read, and then do the write • CPU stalls less frequently since restarts as soon as read finished

  32. Write Address V V V V 1 0 0 0 100 100 1 0 0 0 104 1 0 0 0 108 1 0 0 0 112 Write Address V V V V 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 ENGS 116 Lecture 12 4. Reduce Miss Penalty by Merging Write Buffer • Write merging in write buffer 4 entry, 4 word 16 sequential writes in a row

  33. CPU address Data Data in out =? Tag Victim cache Data =? Write buffer Lower Level Memory ENGS 116 Lecture 12 5. Reduce Miss Penalty via a “Victim Cache” • How to combine fast hit time of direct mapped yet still avoid conflict misses? • Add buffer to place data discarded from cache • Jouppi [1990]: 4-entry victim cache removed 20% to 95% of conflicts for a 4-KB direct-mapped data cache • Used in Alpha, HP machines

  34. 100 300 200 204 ENGS 116 Lecture 12 6. Reduce Miss Penalty: Subblock Placement • Don’t have to load full block on a miss • Have valid bits per subblock to indicate valid • (Originally invented to reduce tag storage) 1 1 1 1 1 1 0 0 0 1 1 0 0 0 0 0 Valid Bits Subblocks

  35. ENGS 116 Lecture 12 Reducing Miss Rate • Larger Block Size • Larger Caches • Higher Associativity • Way Prediction and Pseudoassociative Caches • Compiler Optimizations: • Merging Arrays • Loop Interchange • Loop Fusion • Blocking

  36. ENGS 116 Lecture 12 Classifying Misses: 3 Cs • Compulsory: The first access to a block is not in the cache, so the block must be brought into the cache. Also called cold start misses or first reference misses. (Misses even in an infinite cache) • Capacity: If the cache cannot contain all the blocks needed during execution of a program, capacity misses will occur due to blocks being discarded and later retrieved. (Misses in fully associative, size X cache) • Conflict:If block-placement strategy is set associative or direct mapped, conflict misses (in addition to compulsory & capacity misses) will occur because a block can be discarded and later retrieved if too many blocks map to its set. Also called collision misses or interference misses. (Misses in N-way set associative, size X cache)

  37. 0.14 1-way 0.12 2-way Conflict 0.1 4-way 0.08 8-way 0.06 Miss Rate per Type Capacity 0.04 0.02 0 1 2 4 8 16 32 64 128 Compulsory vanishingly small Cache Size (KB) Compulsory ENGS 116 Lecture 12 3Cs Absolute Miss Rate (SPEC92)

  38. 0.14 1-way Conflict 0.12 2-way 0.1 4-way 0.08 Miss Rate per Type 8-way 0.06 Capacity 0.04 0.02 0 1 2 4 8 16 32 64 128 Compulsory Cache Size (KB) ENGS 116 Lecture 12 2:1 Cache Rule miss rate 1-way associative cache size X = miss rate 2-way associative cache size X/2

  39. 100% 1-way Conflict 80% 2-way 4-way 60% 8-way Miss Rate per Type 40% Capacity 20% 0% 1 2 4 8 16 32 64 128 Flaws: for fixed block size Good: insight Compulsory Cache Size (KB) ENGS 116 Lecture 12 3Cs Relative Miss Rate

  40. ENGS 116 Lecture 12 How Can We Reduce Misses? • 3 Cs: Compulsory, Capacity, Conflict • In all cases, assume total cache size not changed • What happens if we: 1) Change Block Size: Which of 3Cs is obviously affected? 2) Change Associativity: Which of 3Cs is obviously affected? 3) Change Compiler: Which of 3Cs is obviously affected?

  41. ENGS 116 Lecture 12 1. Reduce Misses via Larger Block Size

  42. ENGS 116 Lecture 12 2. Reduce Misses: Larger Cache Size • Obvious improvement but: • Longer hit time • Higher cost • Each cache size favors a block-size, based on memory bandwidth

  43. ENGS 116 Lecture 12 3. Reduce Misses via Higher Associativity • 2:1 Cache Rule: • Miss Rate DM cache size N ≈ Miss Rate 2-way SA cache size N/2 • Beware: Execution time is final measure! • Will clock cycle time increase? • 8-Way is almost fully associative

  44. ENGS 116 Lecture 12 Example: Avg. Memory Access Time vs. Miss Rate • Example: assume CCT = 1.10 for 2-way, 1.12 for 4-way, 1.14 for 8-way vs. CCT direct mapped Cache Size Associativity (KB) 1-way 2-way 4-way 8-way 1 2.33 2.15 2.07 2.01 2 1.98 1.86 1.76 1.68 4 1.72 1.67 1.61 1.53 8 1.46 1.48 1.47 1.43 16 1.29 1.32 1.32 1.32 32 1.20 1.24 1.25 1.27 641.14 1.20 1.21 1.23 1281.10 1.17 1.18 1.20 (Red means A.M.A.T. not improved by more associativity)

  45. Hit Time Miss Penalty Pseudo Hit Time Time ENGS 116 Lecture 12 Reducing Misses via “Pseudo-Associativity” or way prediction • How to combine fast hit time of Direct Mapped and have the lower conflict misses of 2-way SA cache? • Divide cache: on a miss, check other half of cache to see if there, if so have a pseudo-hit (slow hit) • Way Prediction: keep prediction bits to decide what comparison is made first • Drawback: CPU pipeline is hard if hit takes 1 or 2 cycles • Better for caches not tied directly to processor (L2) • Used in MIPS R1000 L2 cache, similar in UltraSPARC

More Related