CS162 Computer Architecture Lecture 16: Multiprocessor 2: Directory Protocol, Interconnection Networks

1. CS162 Computer ArchitectureLecture 16: Multiprocessor 2: Directory Protocol, Interconnection Networks

2. Larger MPs • Separate Memory per Processor • Local or Remote access via memory controller • 1 Cache Coherency solution: non-cached pages • Alternative: directory per cache that tracks state of every block in every cache • Which caches have a copies of block, dirty vs. clean, ... • Info per memory block vs. per cache block? • PLUS: In memory => simpler protocol (centralized/one location) • MINUS: In memory => directory is ƒ(memory size) vs. ƒ(cache size) • Prevent directory as bottleneck? distribute directory entries with memory, each keeping track of which Procs have copies of their blocks

3. Distributed Directory MPs

4. Network Examples • Bi-directional Ring – EX: HP V Class • 2-D Mesh and Hypercube – SGI Origin and Cray T3E • Crossbar and Omega Network – SMPs, IBM SP3, and IP Routers • Clusters using ethernet, Gigabit ethernet, Myrinet, etc. Properties of various networks will be discussed later

5. CC-NUMA Multiprocessor: Directory Protocol • What is Cache Coherent Non-Uniform Memory Access (CC-NUMA)? • Similar to Snoopy Protocol: Three states • Shared: ≥ 1 processors have data, memory up-to-date • Uncached (no processor hasit; not valid in any cache) • Exclusive: 1 processor (owner) has data; memory out-of-date • In addition to cache state, must track which processors have data when in the shared state (usually bit vector, 1 if processor has copy) • Directory Size: Big => Limited Directory Schemes (Not to be discussed)

6. Directory Protocol • No bus and don’t want to broadcast: • interconnect no longer single arbitration point • all messages have explicit responses • Terms: typically 3 processors involved • Local node where a request originates • Home node where the memory location of an address resides • Remote node has a copy of a cache block, whether exclusive or shared • Example messages on next slide: P = processor number, A = address

7. Example Directory Protocol • Message sent to directory causes two actions: • Update the directory • More messages to satisfy request • Block is in Uncached state: the copy in memory is the current value; only possible requests for that block are: • Read miss: requesting processor sent data from memory &requestor made only sharing node; state of block made Shared. • Write miss: requesting processor is sent the value & becomes the Sharing node. The block is made Exclusive to indicate that the only valid copy is cached. Sharers indicates the identity of the owner. • Block is Shared => the memory value is up-to-date: • Read miss: requesting processor is sent back the data from memory & requesting processor is added to the sharing set. • Write miss: requesting processor is sent the value. All processors in the set Sharers are sent invalidate messages, & Sharers is set to identity of requesting processor. The state of the block is made Exclusive.

8. Example Directory Protocol • Block is Exclusive: current value of the block is held in the cache of the processor identified by the set Sharers (the owner) => three possible directory requests: • Read miss: owner processor sent data fetch message, causing state of block in owner’s cache to transition to Shared and causes owner to send data to directory, where it is written to memory & sent back to requesting processor. Identity of requesting processor is added to set Sharers, which still contains the identity of the processor that was the owner (since it still has a readable copy). State is shared. • Data write-back: owner processor is replacing the block and hence must write it back, making memory copy up-to-date (the home directory essentially becomes the owner), the block is now Uncached, and the Sharer set is empty. • Write miss: block has a new owner. A message is sent to old owner causing the cache to send the value of the block to the directory from which it is sent to the requesting processor, which becomes the new owner. Sharers is set to identity of new owner, and state of block is made Exclusive.

10. Interconnection Network Routing, Topology Design Trade-offs

11. Interconnection Topologies • Class networks scaling with N • Logical Properties: • distance, degree • Physical properties • length, width • Static vs. Dynamic Networks • Fully connected network • diameter = 1 • degree = N • cost? • bus => O(N), but BW is O(1) - actually worse • crossbar => O(N2) for BW O(N) • VLSI technology determines switch degree

12. What characterizes a network? • Topology (what) • physical interconnection structure of the network graph • direct: node connected to every switch • indirect: nodes connected to specific subset of switches • Routing Algorithm (which) • restricts the set of paths that msgs may follow • many algorithms with different properties • gridlock avoidance? • Switching Strategy (how) • how data in a msg traverses a route • circuit switching vs. packet switching • Flow Control Mechanism (when) • when a msg or portions of it traverse a route • what happens when traffic is encountered?

13. Flow Control • What do you do when push comes to shove? • Ethernet: collision detection and retry after delay • FDDI, token ring: arbitration token • TCP/WAN: buffer, drop, adjust rate • any solution must adjust to output rate • Link-level flow control

14. Topological Properties • Routing Distance - number of links on route • Diameter - maximum routing distance between any two nodes in the network • Average Distance – Sum of distances between nodes/number of nodes • Degree of a Node – Number of links connected to a node => Cost high if degree is high • A network is partitioned by a set of links if their removal disconnects the graph • Fault-tolerance – Number of alternate paths between two nodes in a network

15. Sender Overhead Transmission time (size ÷ bandwidth) Review: Performance Metrics Sender (processor busy) Time of Flight Transmission time (size ÷ bandwidth) Receiver Overhead Receiver (processor busy) Transport Latency Total Latency Total Latency = Sender Overhead + Time of Flight+ Message Size ÷ BW +Receiver Overhead Includes header/trailer in BW calculation?

16. Example Static Network: 2-D Mesh Architecture

17. More Static Networks: Linear Arrays and Rings • Linear Array • Diameter? • Average Distance? • Bisection bandwidth? • Route A -> B given by relative address R = B-A • Torus? • Examples: FDDI, SCI, FiberChannel Arbitrated Loop, KSR1

18. Multidimensional Meshes and Tori • d-dimensional array • n = kd-1 X ...X kO nodes • described by d-vector of coordinates (id-1, ..., iO) • d-dimensional k-ary mesh: N = kd • k = dÖN • described by d-vector of radix k coordinate • d-dimensional k-ary torus (or k-ary d-cube)? Ex: Intel Paragon (2D), SGI Origin (Hypercube), Cray T3E (3DMesh) 3D Cube 2D Grid

19. Hypercubes • Also called binary n-cubes. # of nodes = N = 2n. • O(logN) Hops • Good bisection BW • Complexity • Out degree is n = logN correct dimensions in order • with random comm. 2 ports per processor 0-D 1-D 2-D 3-D 4-D 5-D !

20. Origin Network • Each router has six pairs of 1.56MB/s unidirectional links • Two to nodes, four to other routers • latency: 41ns pin to pin across a router • Flexible cables up to 3 ft long • Four “virtual channels”: request, reply, other two for priority or I/O

21. Case Study: Cray T3D • Build up info in ‘shell’ • Remote memory operations encoded in address

22. Trees • Diameter and ave distance logarithmic • k-ary tree, height d = logk N • address specified d-vector of radix k coordinates describing path down from root • Fixed degree • Route up to common ancestor and down • R = B xor A • let i be position of most significant 1 in R, route up i+1 levels • down in direction given by low i+1 bits of B • H-tree space is O(N) with O(ÖN) long wires • Bisection BW?

23. Real Machines • Wide links, smaller routing delay • Tremendous variation