1 / 59

ECE 4100/6100 Advanced Computer Architecture Lecture 9 Memory Hierarchy Design (I)

ECE 4100/6100 Advanced Computer Architecture Lecture 9 Memory Hierarchy Design (I). Prof. Hsien-Hsin Sean Lee School of Electrical and Computer Engineering Georgia Institute of Technology. Why Care About Memory Hierarchy?. Processor-DRAM Performance Gap grows 50% / year. Processor 60%/year

jerica
Download Presentation

ECE 4100/6100 Advanced Computer Architecture Lecture 9 Memory Hierarchy Design (I)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ECE 4100/6100Advanced Computer ArchitectureLecture 9 Memory Hierarchy Design (I) Prof. Hsien-Hsin Sean Lee School of Electrical and Computer Engineering Georgia Institute of Technology

  2. Why Care About Memory Hierarchy? Processor-DRAM Performance Gap grows 50% / year Processor 60%/year (2X/1.5 years) 1000 “Moore’s Law” 100 CPU Performance 10 DRAM 9%/year (2X/10 years) DRAM 1 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 Time

  3. I/O, Memory, Cache CPU An Unbalanced System Source: Bob Colwell keynote ISCA’29 2002

  4. Memory Issues • Latency • Time to move through the longest circuit path (from the start of request to the response) • Bandwidth • Number of bits transported at one time • Capacity • Size of memory • Energy • Cost of accessing memory (to read and write)

  5. SRAM DRAM DISK Main Memory Reg File L1 Data cache L2 Cache L1 Inst cache Model of Memory Hierarchy

  6. Levels of the Memory Hierarchy Capacity Access Time Cost Upper Level Staging Transfer Unit faster CPU Registers 100s Bytes <10 ns Registers Compiler 1-8 bytes Instr. Operands Cache K Bytes 10-100 ns 1-0.1 cents/bit Cache Cache controller 8-128 bytes This Lecture Cache Lines Main Memory M Bytes 200ns- 500ns $.0001-.00001 cents /bit Memory Operating system 512-4K bytes Pages Disk G Bytes, 10 ms (10,000,000 ns) 10 - 10 cents/bit Disk -6 -5 User Mbytes Files Larger Tape infinite sec-min 10 Tape Lower Level -8

  7. Topics covered • Why do caches work • Principle of program locality • Cache hierarchy • Average memory access time (AMAT) • Types of caches • Direct mapped • Set-associative • Fully associative • Cache policies • Write back vs. write through • Write allocate vs. No write allocate

  8. Principle of Locality • Programs access a relatively small portion of address space at any instant of time. • Two Types of Locality: • Temporal Locality(Locality in Time): If an address is referenced, it tends to be referenced again • e.g., loops, reuse • Spatial Locality(Locality in Space): If an address is referenced, neighboring addresses tend to be referenced • e.g., straightline code, array access • Traditionally, HW has relied on locality for speed Locality is a program property that is exploited in machine design.

  9. D C[99] C[98] C[97] C[96] . . . . . . . . . . . . . . C[7] C[6] C[5] C[4] C[3] C[2] C[1] C[0] . . . . . . . . . . . . . . B[11] B[10] B[9] B[8] B[7] B[6] B[5] B[4] B[3] B[2] B[1] B[0] A[99] A[98] A[97] A[96] . . . . . . . . . . . . . . A[7] A[6] A[5] A[4] A[3] A[2] A[1] A[0] Example of Locality int A[100], B[100],C[100],D; for (i=0; i<100; i++) { C[i] = A[i] * B[i] + D; } A Cache Line (One fetch)

  10. Modern Memory Hierarchy • By taking advantage of the principle of locality: • Present the user with as much memory as is available in the cheapest technology. • Provide access at the speed offered by the fastest technology. Processor Control Secondary Storage (Disk) Third Level Cache (SRAM) Tertiary Storage (Disk/Tape) Main Memory (DRAM) Second Level Cache (SRAM) L1 I Cache Datapath Registers L1 D Cache

  11. DL1 DL1 Core0 Core1 IL1 IL1 L2 Cache Example: Intel Core2 Duo Source: http://www.sandpile.org

  12. Example : Intel Itanium 2 3MB Version 180nm 421 mm2 6MB Version 130nm 374 mm2

  13. Core 0 Core 0 3MB 3MB 3MB 3MB Core 1 3MB 3MB 3MB 3MB Intel Nehalem 24MB L3

  14. Example : STI Cell Processor Local Storage SPE = 21M transistors (14M array; 7M logic)

  15. Cell Synergistic Processing Element Each SPE contains 128 x128 bit registers, 256KB, 1-port, ECC-protected local SRAM (Not cache)

  16. Cache Terminology • Hit: data appears in some block • Hit Rate: the fraction of memory accesses found in the level • Hit Time: Time to access the level (consists of RAM access time + Time to determine hit) • Miss: data needs to be retrieved from a block in the lower level (e.g., Block Y) • Miss Rate= 1 - (Hit Rate) • Miss Penalty: Time to replace a block in the upper level + Time to deliver the block to the processor • Hit Time << Miss Penalty Lower Level Memory Upper Level Memory From Processor Blk X Blk Y To Processor

  17. Average Memory Access Time • Average memory-access time = Hit time + Miss rate x Miss penalty • Miss penalty: time to fetch a block from lower memory level • access time: function of latency • transfer time: function of bandwidth b/w levels • Transfer one “cache line/block” at a time • Transfer at the size of the memory-bus width

  18. Main Memory (DRAM) First-level Cache Memory Hierarchy Performance 1 clk 300 clks Miss % * Miss penalty Hit Time • Average Memory Access Time (AMAT) = Hit Time + Miss rate * Miss Penalty = Thit(L1) + Miss%(L1) * T(memory) • Example: • Cache Hit = 1 cycle • Miss rate = 10% = 0.1 • Miss penalty = 300 cycles • AMAT = 1 + 0.1 * 300 = 31 cycles • Can we improve it?

  19. Third Level Cache Second Level Cache Main Memory (DRAM) First-level Cache Reducing Penalty: Multi-Level Cache 1 clk 10 clks 20 clks 300 clks L1 L2 L3 On-die Average Memory Access Time (AMAT) = Thit(L1) + Miss%(L1)* (Thit(L2) + Miss%(L2)* (Thit(L3) + Miss%(L3)*T(memory) ) )

  20. AMAT of multi-level memory = Thit(L1) + Miss%(L1)* Tmiss(L1) = Thit(L1) + Miss%(L1)* { Thit(L2) + Miss%(L2)* (Tmiss(L2) } = Thit(L1) + Miss%(L1)* { Thit(L2) + Miss%(L2)*(Tmiss(L2) } = Thit(L1) + Miss%(L1)* { Thit(L2) + Miss%(L2) * [ Thit(L3) + Miss%(L3) * T(memory) ] }

  21. AMAT Example = Thit(L1) + Miss%(L1)* (Thit(L2) + Miss%(L2)* (Thit(L3) + Miss%(L3)*T(memory) ) ) • Example: • Miss rate L1=10%, Thit(L1) = 1 cycle • Miss rate L2=5%, Thit(L2) = 10 cycles • Miss rate L3=1%, Thit(L3) = 20 cycles • T(memory) = 300 cycles • AMAT = ? • 2.115 (compare to 31 with no multi-levels) 14.7x speed-up!

  22. Types of Caches • DM and FA can be thought as special cases of SA • DM  1-way SA • FA  All-way SA

  23. 0x0F 00000 00000 0x55 11111 0xAA 0xF0 11111 Direct Mapping Index Tag Data 0 00000 0 0x55 0x0F 1 00000 00000 1 00001 0 • Direct mapping: • A memory value can only be placed at a single corresponding location in the cache 11111 0 0xAA 0xF0 11111 11111 1

  24. Set Associative Mapping (2-Way) Index Data Tag Way 0 Way 1 0 0000 0 0x55 0000 0 0 0x55 0x0F 0x0F 1 0000 1 1 0000 1 0 • Set-associative mapping: • A memory value can be placed in any location of a set in the cache 1111 0 0 0xAA 1111 0 0xAA 0xF0 0xF0 1111 1 1 1111 1

  25. 0x0F 0x0F 0000 0000 0000 0000 0x55 0x55 1111 1111 0xAA 0xAA 0xF0 0xF0 1111 1111 000000 0x55 0x0F 000001 000110 111110 0xAA 0xF0 111111 Fully Associative Mapping Tag Data 000000 0x55 0x0F 000001 000110 • Fully-associative mapping: • A memory value can be placed anywhere in the cache 111110 0xAA 0xF0 111111

  26. Direct Mapped Cache Memory • Cache location 0 is occupied by data from: • Memory locations 0, 4, 8, and C • Which one should we place in the cache? • How can we tell which one is in the cache? Address DM Cache 0 1 Cache Index 2 0 3 1 4 2 5 3 6 7 A Cache Line (or Block) 8 9 A B C D E F

  27. Three (or Four) Cs (Cache Miss Terms) • Compulsory Misses: • cold start misses (Caches do not have valid data at the start of the program) • Capacity Misses: • Increase cache size • Conflict Misses: • Increase cache size and/or associativity. • Associative caches reduce conflict misses • Coherence Misses: • In multiprocessor systems (later lectures…) 0x1234 Processor Cache 0x1234 0x5678 0x91B1 0x1111 Processor Cache 0x1234 0x5678 0x91B1 0x1111 Processor Cache

  28. Example: 1KB DM Cache, 32-byte Lines • The lowest M bits are the Offset (Line Size = 2M) • Index = log2 (# of sets) Address 31 9 4 0 Tag Index Offset Ex: 0x01 Ex: 0x00 Valid Bit Cache Tag Cache Data : Byte 31 Byte 1 Byte 0 0 : Byte 63 Byte 33 Byte 32 1 2 3 # of set : : : : Byte 1023 Byte 992 31

  29. Example of Caches • Given a 2MB, direct-mapped physical caches, line size=64bytes • Support up to 52-bit physical address • Tag size? • Now change it to 16-way, Tag size? • How about if it’s fully associative, Tag size?

  30. 27 26 25 24 Example: 1KB DM Cache, 32-byte Lines • lw from 0x77FF1C68 Tag Index Offset 77FF1C68 = 0111 0111 1111 1111 0001 1100 0101 1000 Tag array Data array 2 DM Cache

  31. DM Cache Speed Advantage • Tag and data access happen in parallel • Faster cache access! Offset Tag Index Tag array Data array Index

  32. Associative Caches Reduce Conflict Misses • Set associative (SA) cache • multiple possible locations in a set • Fully associative (FA) cache • any location in the cache • Hardware and speed overhead • Comparators • Multiplexors • Data selection only after Hit/Miss determination (i.e., after tag comparison)

  33. Cache Data Cache Tag Valid Cache Line 0 : : : Compare Set Associative Cache (2-way) • Cache index selects a “set” from the cache • The two tags in the set are compared in parallel • Data is selected based on the tag result • Additional circuitry as compared to DM caches • Makes SA caches slower to access than DM of comparable size Cache Index Valid Cache Tag Cache Data Cache Line 0 : : : Adr Tag Compare 1 0 Mux Sel1 Sel0 OR Cache Line Hit

  34. Set-Associative Cache (2-way) • 32 bit address • lw from 0x77FF1C78 Tag Index offset Tag array0 Data aray0 Data array1 Tag array1

  35. = = = = Fully Associative Cache tag offset Data Tag Associative Search Multiplexor Rotate and Mask

  36. Fully Associative Cache Tag offset Write Data Address Tag Data Tag Data Tag Data Tag Data compare compare compare compare Additional circuitry as compared to DM caches More extensive than SA caches Makes FA caches slower to access than either DM or SA of comparable size Read Data

  37. Cache Write Policy • Write through -The value is written to both the cache line and to the lower-level memory. • Write back - The value is written only to the cache line. The modified cache line is written to main memory only when it has to be replaced. • Is the cache line clean (holds the same value as memory) or dirty (holds a different value than memory)?

  38. Write-through Policy 0x1234 0x1234 0x1234 0x5678 0x5678 0x1234 Processor Cache Memory

  39. Write Buffer • Processor: writes data into the cache and the write buffer • Memory controller: writes contents of the buffer to memory • Write buffer is a FIFO structure: • Typically 4 to 8 entries • Desirable: Occurrence of Writes << DRAM write cycles • Memory system designer’s nightmare: • Write buffer saturation (i.e., Writes  DRAM write cycles) Cache Processor DRAM Write Buffer

  40. Writeback Policy 0x1234 0x1234 0x1234 ????? 0x5678 0x9ABC 0x5678 0x1234 0x5678 Processor Cache Memory Write miss

  41. On Write Miss • Write allocate • The line is allocated on a write miss, followed by the write hit actions above. • Write misses first act like read misses • No write allocate • Write misses do not interfere cache • Line is only modified in the lower level memory • Mostly use with write-through cache

  42. Quick recap • Processor-memory performance gap • Memory hierarchy exploits program locality to reduce AMAT • Types of Caches • Direct mapped • Set associative • Fully associative • Cache policies • Write through vs. Write back • Write allocate vs. No write allocate

  43. Cache Replacement Policy • Random • Replace a randomly chosen line • FIFO • Replace the oldest line • LRU (Least Recently Used) • Replace the least recently used line • NRU (Not Recently Used) • Replace one of the lines that is not recently used • In Itanium2 L1 Dcache, L2 and L3 caches

  44. E G D C C A C E D C C B A D E A B D A D LRU Policy MRU-1 LRU+1 LRU MRU A B C D Access C Access D Access E MISS, replacement needed Access C MISS, replacement needed Access G

  45. LRU From Hardware Perspective LRU Way3 Way2 Way1 Way0 State machine Access update Access D A B C D LRU policy increases cache access times Additional hardware bits needed for LRU state machine

  46. LRU Algorithms • True LRU • Expensive in terms of speed and hardware • Need to remember the order in which all N lines were last accessed • N! scenarios –O(log N!) O(N log N) LRU bits • 2-ways  AB BA = 2 = 2! • 3-ways  ABC ACB BAC BCA CAB CBA = 6 = 3! • Pseudo LRU: O(N) • Approximates LRU policy with a binary tree

  47. Pseudo LRU Algorithm (4-way SA) • Tree-based • O(N): 3 bits for 4-way • Cache ways are the leaves of the tree • Combine ways as we proceed towards the root of the tree AB/CD bit (L0) A/B bit (L1) C/D bit (L2) Way A Way B Way C Way D Way3 Way2 Way1 Way0 A B C D

  48. AB/CD bit (L0) A/B bit (L1) C/D bit (L2) Way A Way B Way C Way D Pseudo LRU Algorithm • Less hardware than LRU • Faster than LRU • L2L1L0 = 001, • a way needs to be replaced, which way would be chosen? • L2L1L0 = 000, • there is a hit in Way B, what is the new updated L2L1L0? Replacement Decision LRU update algorithm CD AB AB/CD CD AB AB/CD

  49. Not Recently Used (NRU) • Use R(eferenced) and M(odified) bits • 0 (not referenced or not modified) • 1 (referenced or modified) • Classify lines into • C0: R=0, M=0 • C1: R=0, M=1 • C2: R=1, M=0 • C3: R=1, M=1 • Chose the victim from the lowest class • (C3 > C2 > C1 > C0) • Periodically clear R and M bits

  50. 4 0 % 3 5 % 3 0 % 2 5 % e t a r 2 0 % s s i M 1 5 % 1 0 % 5 % 0 % 4 1 6 6 4 2 5 6 B l o c k s i z e ( b y t e s ) 1 K B 8 K B 1 6 K B 6 4 K B 2 5 6 K B Reducing Miss Rate • Enlarge Cache • If cache size is fixed • Increase associativity • Increase line size • Does this always work? Increasing cache pollution

More Related