Parallel garbage collection with a block-structured heap

1. Parallel garbage collection with a block-structured heap Simon Marlow (Microsoft Research) Simon Peyton Jones (Microsoft Research) Roshan James (U. Indiana) Tim Harris (Microsoft Research)

2. What’s parallel GC? A parallel Haskell program Mutator GC GC GC Time The GC is single-threaded, so Amdahl’s Law limits the amount of speedup we can expect. GC is the bottleneck for parallel execution. A single-threaded Haskell program (e.g. GHC itself) Mutator Mutator

3. Setting the scene… • NB. parallel, not concurrent • “Parallel” means we stop all the mutator threads and perform GC using multiple GC threads. • “Concurrent” means running the GC at the same time as the mutator • concurrent GC is more difficult, and can impose some overhead on the mutator, since the mutator must synchronise with the GC. • Target: commodity multi-cores • for now we aim at <=8 cores, which may mean we sometimes choose low overhead over scalability. Later we may revisit these decisions.

4. High-level structure • Key design decision: our storage manager is divided into two layers. • The block allocator • requests memory from the OS • provides blocks of memory to the rest of the RTS • manages a pool of free blocks. • The GC allocates memory from the block layer only. GC Block Allocator malloc() / mmap()

5. Blocks of memory • Memory is managed in units of the fixed block size (currently 4k). • Blocks can be allocated singly, or in contiguous groupsto accomodate larger objects. • Blocks can be linked together into lists to form areas

6. Why divide memory into blocks? • Flexibility, Portability • The storage manager needs to track multiple regions(e.g. generations) which may grow and shrink over time. • contiguous memory would be problematic: how much space do we allocate to each one, so that they don’t grow into each other? • Address space is limited. • With linked lists of blocks, we can grow or shrink a region whenever we like by adding/removing blocks. • managing large objects is easier: each gets its own block group, and we can link large objects onto lists, so no need to copy the object to move it. • some wastage due to slop (<1%)

7. More advantages of blocks • Portable: all we require from the OS is a way to allocate memory, there are no dependencies on address-space layout. • Memory can be recycled quickly: cache-friendly • Sometimes we need to allocate when it is inconvenient to GC: we can always grab another block. • A “useful granularity” • useful for performing occasional checks during execution (e.g. context switching) • for dividing up the work in a parallel GC…

8. How do blocks work? • Each block has a fixed table of data: the “block descriptor”. 1st free byte in the block struct bdescr { void *start, *free, *link; int blocks; … } bdescr *Bdescr (void *); bdescr *allocBlocks (int blocks); void freeBlocks (bdescr *); Bdescr() maps address to block descriptor in a few instructions, no memory accesses chains blocks together (or links to head of group) Start of the block Number of blocks in group (0 if not the head)

9. Where do block descriptors live? • Choice 1: at the start of a block. •  Bdescr(p) is one instruction: p & 0xfffff000 •  Bad for cache & TLB. • We often traverse block descriptors: if they are scattered all over memory this thrashes the TLB. •  Contiguous multi-block groups can only have a descriptor for the first block. •  A block contains “4k minus a bit” space for data (awkward)

10. Choice 2 • Block descriptors are grouped together, and can be found by an address calculation. •  Bdescr(p) is ~6 instructions (next slide…) •  We can keep block descriptors together, better for cache/TLB •  Contiguous block groups are easier: all blocks in the group have descriptors. •  Blocks contain a full 4k of data

11. Megablocks 2m bytes Block 1 Block 2 Block N 2k bytes 2m bytes aligned The block allocator requests memory from the operating system in units of a Megablock, which is divided into N blocks and block descriptors.

12. Empty Block desc. 1 Block desc. 2 Block desc. N Megablocks 2m bytes Block 1 Block 2 Block N 2k bytes Bdescr(p) = ((p & 2m-1) >> k) << d) | (p & ~2m-1) 2m bytes aligned 2d

13. Parallel GC • First we consider how to parallelise 2-space copying GC, and then extend this to generational GC.

14. Roots point to live objects Copy live objects to to-space Complete when scan pointer catches up with allocation pointer. Scan live objects for more roots Background: Cheney copying collection Allocation area To-space

15. How can we parallelise copying GC? • Basically, we want each CPU to copy and scan different objects. • The main problem is finding an effective way to partition the problem, so we can keep all CPUs busy all the time (load balancing). • Static partitioning (eg. partition the heap by address) is not good: • live data might not be evenly distributed, leading to poor load balancing • Need synchronisation when pointers cross partition boundaries

16. Load balancing • So we need dynamic load-balancing for GC. • the pending set is the set of objects copied but not yet scanned. • Each CPU: • where • (Need synchronisation to prevent two threads copying the same object – later) while (pending set non-empty) { remove an object p from the pending set scan(p) } scan(p) { for each object q pointed to by p { if q has not been copied { copy q add q to the pending set } } }

17. The Pending Set • Now the problem is reduced to finding a good representation for the pending set. • Operations on the pending set are in the inner loop, so heavyweight synchronisation would be very expensive. • In standard copying GC the pending set is represented implicitly (and cheaply!) by to-space. Hence any explicit representation will incur an overhead compared to single-threaded GC, and eat into any parallel speedup.

18. Previous solutions • per-CPU work-stealing queues (Flood et. al. (2001)). • good load balancing, but • some administrative overhead (quantity unknown) • needs clever lock-free data structures • needs some strategy for overflow (GC can’t use arbitrary extra memory!) • Dividing the pending set into chunks (Imai/Tick (1993), others). • coarser granularity reduces synchronisation overhead • less effective load-balancing, especially if the chunk size is too high.

19. How blocks help • Since to-space is already a list of blocks, it is a natural representation for a chunked pending set! • No need for a separate pending set representation, no extra admin overhead relative to single threaed GC. • Larger blocks => lower synchronisation overhead • Smaller blocks => better load balancing

20. But what if… • … the pending set isn’t large enough to fill a block? E.g. If the heap consists of a single linked list of integers, then the scan pointer will always be close to the allocation pointer, we will never generate a full block of work. • There may be little available parallelism in the heap structure anyway. • But with e.g. 2 linked lists, we would still not be able to parallelise on two cores, because the scan pointer will only be 2 words behind the allocation pointer.

21. Available parallelism • There should be enough parallelism, at least in old-gen collections.

22. GC data structures Pending set = free memory = not scanned GC thread 1 Workspace GC thread 2 Workspace done list = scanned

23. = free memory Scan block Alloc block = not scanned = scanned Alloc pointer Scan pointer Inside a workspace… • Objects are copied into the Alloc block (thread-private allocation!) • Loop: • Grab a block to be scanned from the pending set • Scan it • Push it back to the “done” list • When an Alloc block becomes full, move it to the pending set, grab an empty block

24. Inside a workspace… = free memory Scan block Alloc block • When the pending set is empty • Make the Scan block = the Alloc block • Scan until complete • Look for more full blocks… • Note that we may now have scanned part of the Alloc block, so we need to remember what portion has been scanned. (full details of the algorithm are in the paper). = not scanned = scanned Alloc pointer Scan pointer

25. Termination • Keep a global counter of running threads • When a thread finds no work to do, it decrements the count of running threads • If it finds the count = 0, all the work is done; stop. • Poll for work: if there is work to do, increment the counter and continue (don’t remove the work from the queue yet).

26. When work is scarce • We found that often the pending set is small or empty (despite earlier measurements), leading to low parallelism. • The only solution is to export smaller chunks of work to the pending set. • We use a dynamic chunk size: when the pending set is low, we export smaller chunks. • smaller chunks leads to a fragmentation problem: we want to fill up the rest of the block later, so we have to keep track of these partially-full blocks in per-thread lists.

27. FWD Header Header Payload Payload Object is copied Into to-space Overwrite with a forwarding pointer Forwarding pointers and synchronisation • Must synchronise if two threads attempt to copy the same object, otherwise the object is duplicated. • Use CAS to install the forwarding pointer; if another thread installs the pointer first, return it (don’t copy the object). One CAS per object! • If the object is immutable, then we don’t mind coying it twice, and in this case we could omit the CAS (but note that the forwarding pointer must not overwrite the payload).

28. Overhead due to atomic forwarding

29. Parallel Generational-copying GC • Fairly straightforward generalisation of parallel copying GC • Three complications: • maintaining the remembered sets • deciding which pending set to take work from • tenuring policy • see paper for the details

30. Speedup results

31. Measuring load balancing • Ctot is the total copied by all threads • Cmaxis the maximum copied by any thread • Work balance factor • Perfect balance = N, perfect imbalance = 1. • balance factor = maximum possible speedup given the distribution of work across CPUs (speedup might be lower for other reasons).

32. Load balancing results

33. Status • Already checked in, you can try it out right now in GHC HEAD. • Tested on a GHC build – saved about ~6% • Will be available in GHC 6.10 (autumn 2008) • Multi-threaded GC will usually be a win on 2+ cores, although requires increasing the heap size to get the most benefit: parallelising small GCs doesn’t work so well.

34. A war story… • This GC was first implemented by Roshan James, in the summer of 2006. • measurements showed negative speedup • Re-implemented by SimonM in 2007 • also achieved negative speedup • despite having good load-balancing. • The cause of the bottleneck: • after copying an object, a pointer in the block descriptor was updated. Adjacent block descriptors sometimes share a cache line, so multiple threads were writing to the same cache line => Very Bad. • It took multiple man-weeks and 3 profiling tools to find the problem. • Solution: cache the pointer in thread-local storage.

35. Further work • Investigate/improve load-balancing • Avoid locking for immutable objects • Contention is very low • We might get a tiny amount of duplication per GC • Independent minor GCs. • Hard to parallelise minor GC: too quick, not enough parallelism • Stopping the world for minor GC is a severe bottleneck in a program running on multiple CPUs. • So do per-CPU independent minor GCs. • Main techincal problem: either track or prevent inter-minor-generation pointers. (eg. Doligez/Leroy(1993) for ML, Steensgaard(2001)). • Concurrent marking, with simple sweep: blocks with no live objects can be freed immediately, compact or copy occasionally to recover fragmentation. done! partly done!

36. Optimisations… • There is a long list of tweaks and optimisations that we tried, some of which helped, some didn’t. • Move block descriptors to the beginning of the block: bad cache/TLB effects. • Prefetching: no joy, too fragile, and recent CPUs do automatic prefetching • Should the pending block set be FIFO or LIFO? or something else? • Some objects don’t need to be scanned, copy them to a separate non-scanned area (not worthwhile)