1 / 44

SMiLE Shared Memory Programming

SMiLE Shared Memory Programming. Wolfgang Karl & Martin Schulz Lehrstuhl für Rechnertechnik und Rechnerorganisation Technische Universität München 1st SCI Summer School October 2nd – 4th, 2000, Trinity College Dublin. SMiLE software layers. True Shared Memory Programming.

kenny
Download Presentation

SMiLE Shared Memory Programming

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SMiLE Shared Memory Programming Wolfgang Karl & Martin SchulzLehrstuhl für Rechnertechnik und RechnerorganisationTechnische Universität München 1st SCI Summer SchoolOctober 2nd – 4th, 2000, Trinity College Dublin

  2. SMiLE software layers True Shared MemoryProgramming Target applications / Test suites Target applications / Test suites High-levelSMiLE High-levelSMiLE NTprot.stack NTprot.stack SISALon MuSE SISALon MuSE SISCI PVM SISCI PVM SPMD style model SPMD style model TreadMarkscompatible API TreadMarkscompatible API Low-levelSMiLE Low-levelSMiLE AM 2.0 AM 2.0 SS-lib SS-lib CML CML SCI-VM lib SCI-VM lib SCI-Messaging SCI-Messaging Raw SCI Programming User/Kernelboundary User/Kernelboundary NDISdriver NDISdriver SCI Drivers &SISCI API SCI Drivers & SISCI API SCI-VM SCI-VM SCI-Hardware: SMiLE & Dolphin adapter, HW-Monitor SCI-Hardware: SMiLE & Dolphin adapter, HW-Monitor Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  3. Overview • SCI hardware DSM principle • Using raw SCI via SISCI • SCI-VM: A global virtual memory for SCI • Shared Memory Programming Models • A sample model: SPMD • Outlook on the Lab-Session Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  4. SMiLE Shared Memory Programming  SCI hardware DSM principle Using raw SCI via SISCI SCI-VM: A global virtual memory for SCI Shared Memory Programming Models A sample model: SPMD Outlook on the Lab-Session

  5. SCI-based PC clusters PCs with PCI-SCI adapter SCI interconnect Global address space • Bus-like services • Read-, write-, and synchronization transactions • Hardware-supported DSM • Global address space to access any physical memory Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  6. User-level communication Read/write to mapped segment NUMA characteristics High performance No protocol overhead No OS influence Latency: < 2.0 ms Bandwidth: > 80 MB/s Setup of communication Export Map SCIphysical addressspace • Export of physical memory • SCI Physical address space • Mapping into virtual memory SCI Ringlet Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  7. SCI remote memory mappings Processon A Processon B Virtual address space on A Virtual address space on B Physical mem. on A Physical mem. on B PCI addr. on A PCI addr. on B SCI physical address space CPU MMU CPU MMU Node A SCI Bridge ATTs Node B Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  8. Address Translation Table (ATT) • Present on each SCI adapter • Controls outbound communication • 4096 entries available • Each entry controls between 4KB and 512KB • 256 MB total memory  64 KB per entry • Many parameters per ATT entry, e.g. • Prefetching & Buffering • Enable atomic fetch & inc • Control ordering Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  9. Some more details • Inbound communication • No remapping • Offset part of SCI physical addr. = physical addr. • Basic protection mechanism using access window • Outstanding transactions • Up to 16 write streams • Gather write operations to consequent addresses • Read/Write Ordering • Complete write transactions before read transaction Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  10. SMiLE Shared Memory Programming SCI hardware DSM principle  Using raw SCI via SISCI SCI-VM: A global virtual memory for SCI Shared Memory Programming Models A sample model: SPMD Outlook on the Lab-Session

  11. The SISCI API • Standard low-level API for SCI-based systems • Developed within the SISCI Project • Standard Software Infrastructure for SCI • Implemented by Dolphin ICS (for many platforms) • Goal: Provide the full SCI functionality • Secure abstraction of user-level communication • Based on the IRM (Interconnect Resource Mng.) • Global resource management • Fault tolerance & configuration mechanisms • Comprehensive user-level API Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  12. SISCI in the overall infrastructure Target applications / Test suites Target applications / Test suites High-levelSMiLE High-levelSMiLE NTprot.stack NTprot.stack SISALon MuSE SISALon MuSE SISCI PVM SISCI PVM SPMD style model SPMD style model TreadMarkscompatible API TreadMarkscompatible API Low-levelSMiLE Low-levelSMiLE SISCI API / Library AM 2.0 AM 2.0 SS-lib SS-lib CML CML SCI-VM lib SCI-VM lib SCI-Messaging SCI-Messaging User/Kernelboundary User/Kernelboundary NDISdriver NDISdriver SCI Drivers &SISCI API SCI Drivers & SISCI API SCI-VM SCI-VM SCI-Hardware: SMiLE & Dolphin adapter, HW-Monitor SCI-Hardware: SMiLE & Dolphin adapter, HW-Monitor Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  13. Use of SISCI API • Provides raw shared memory functionalities • Based on individual shared memory segments • No task and/or connection management • High complexity due to low-level character • Application area 1: Middleware • E.g. base for efficient message passing layers • Hide the low-level character of SISCI • Application area 2: Special purpose applications • Directly use the raw performance of SCI Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  14. SISCI availability • SISCI API directly available from Dolphin • http://www.dolphinics.no/ • Multiplatform support • Windows NT 4.0 / 2000 • Linux (currently for 2.2.x kernels) • Solaris 2.5.1/7/8 (Sparc) & 2.6/7 (X86) • Lynx OS 3.01 (x86 & PPC), VxWorks • True64 (first version available) • Documentation also On-line • Complete user-library specification • Sample codes Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  15. Basic SISCI functionality • Main functionality:Basic shared segment management • Creation of segments • Sharing/Identification of segments • Mapping of segments • Support for Memory transfer • Sequence control for data integrity • Optimized SCI memcpy version • Direct abstraction of the SCI principle Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  16. Advanced SISCI functionality • Other functionality • DMA transfers • SCI-based interrupts • Store-barriers • Implicit resource management • Additional library in SISCI kit • sisci_demolib • Itself based on SISCI • Easier Node/Adapter identification Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  17. SCICreateSegment Allocating and Mapping of Segments Data Transfer Virt. Addr. Phys. Mem. PCI Addr. Virt Addr. SCIPrepareSegment SCIMapLocalSegment SCISetSegmentAvailable SCIConnectSegment SCIMapRemoteSegment SCIUnMapSegment SCIDisconnectSegment SCIUnMapSegment SCIRemoveSegment Node A Node B Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  18. SISCI Deficits • SISCI based on individual segments • Individual segments with own address space • Placed on a single node • No global virtual memory • No task/thread control, minimal synchronization • No full programming environment • Has to rebuilt over and over again • No cluster global abstraction • Configuration and ID management missing Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  19. SMiLE Shared Memory Programming SCI hardware DSM principle Using raw SCI via SISCI  SCI-VM: Global virtual memory for SCI Shared Memory Programming Models A sample model: SPMD Outlook on the Lab-Session

  20. SCI support for shared memory • SCI provides Inter-process shared memory • Available through SISCI • Shared memory support not sufficient fortrue shared memory programming • No support for virtual memory • Each segment placed on a single node only • Intra-process shared memory required • Based on remote memory operations • Software DSM support necessary Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  21. Shared memory for clusters • Goal: SCI Virtual Memory • Flexible, general global memory abstraction • One global virtual address space across nodes • Total transparency for the user • Direct utilization of SCI HW-DSM • Hybrid DSM based system • HW-DSM for implicit communication • SW-DSM component for system management • Platform: Linux & Windows NT Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  22. SCI-VM design • Global process abstraction • Team on each node host threads • Each team needs identical virtual memory view • Virtual memory distributed at page granularity • Map local pages conventionally via MMU • Map remote pages with the help of SCI • Extension of the virtual memory concept • Use memory resources across nodes transparently • Modified VM-Manager necessary Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  23. Design of the SCI-VM Team on A Team on B Thread Thread Thread Thread Thread Thread Virtual address space on A Virtual address space on B 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 Physical mem. on A Physical mem. on B PCI addr. on A PCI addr. on B 4 6 1 1 4 6 4 6 1 SCI physical address space Global process abstraction with SCI Virtual Memory Node A Node B Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  24. Implementation challenges • Integration into virtual memory management • Virtual memory mappings at page granularity • Utilization of paged memory • Integration into SCI driver stack • Enable fast, fine grain mappings of remote pages • Caching of remote memory • SCI over PCI does not support cache consistency • Caching necessary for read performance • Solution: Caching & Relaxing consistency • Very similar to LRC mechanisms Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  25. Implementation design • Modular implementation • Extensibility and Flexibility • VMM extension • Manage MMU mappings directly • Kernel driver required • Synchronization operations • Based on SCI atomic transactions • Consistency enforcing mechanisms • Often in coordination with synchronization points • Flush buffers and caches Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  26. PCI/SCI & OS Problems • Missing transparency of the hardware • SCI adapter and PCI host bridge create errors • Shmem model expects full correctness • Integration • SCI Integration done by IRM extension • Missing OS Integration leads to instabilities • Real OS integration for Linux in the works • Locality problems • Often the cause for bad performance • Easy possibility for optimizations necessary Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  27. Implementation status • SCI-VM prototype running • Windows NT & Linux • Static version (no remapping) • Mostly user level based • VMM extensions for both OSs • Current work • Experiments with more applications • Includes comparison with SW-DSM systems • Adding dynamic version of SCI-VM • Real extension of virtual memory management Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  28. SMiLE Shared Memory Programming SCI hardware DSM principle Using raw SCI via SISCI SCI-VM: A global virtual memory for SCI  Shared Memory Programming Models A sample model: SPMD Outlook on the Lab-Session

  29. Multi-Programming Model support • Abundance of shared memory models • In contrast to Message Passing (MPI & PVM) • Few standardization approaches • OpenMP, HPF, Posix threads • Many DSM systems with own API (historically) • Negative impact on portability • Frequent retargeting necessary • Required: Multi-Programming Model support • One system with many faces • Minimize work for support of additional models Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  30. Flexibility of HAMSTER • HAMSTER: Hybrid-dsm based Adaptive and Modular Shared memory archiTEctuRe • Goal: • Support for arbitrary Shmem models • Standard models -> portability of applications • Domain specific models • SCI-VM as efficient DSM core for clusters • Export of various shared memory services • Strongly modularized • Many parameters for easy tuning Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  31. HAMSTER in the infrastructure Target applications / Test suites Target applications / Test suites HAMSTER Hybrid DSM based Adaptive and Modular Shared Memory archiTEctuRe High-levelSMiLE High-levelSMiLE NTprot.stack NTprot.stack SISALon MuSE SISALon MuSE SISCI PVM SISCI PVM SPMD style model SPMD style model TreadMarkscompatible API TreadMarkscompatible API Low-levelSMiLE Low-levelSMiLE AM 2.0 AM 2.0 SS-lib SS-lib CML CML SCI-VM lib SCI-VM lib SCI-Messaging SCI-Messaging User/Kernelboundary User/Kernelboundary NDISdriver NDISdriver SCI Drivers &SISCI API SCI Drivers & SISCI API SCI-VM SCI-VM SCI-Hardware: SMiLE & Dolphin adapter, HW-Monitor SCI-Hardware: SMiLE & Dolphin adapter, HW-Monitor Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  32. HAMSTER framework Task Mgmt. Sync. Mgmt. SCI-VM: Hybrid DSM for SCI clusters SAN with HW-DSM Cluster built ofcommodity PC hardware Shared Memory application Arbitrary Shared Memory model VI-like comm. access through HW- DSM Clus.Ctrl. Cons. Mgmt. Mem. Mgmt. Standalone OSLinux & WinNT NIC driver Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  33. HAMSTER status • Prototype running • Based on the current SCI-VM • Most modules more or less completed • Available programming models • SPMD (incl. SMP version), Jia-Jia • TreadMarks ™, ANL macros (for SPLASH 2) • POSIX thread API (Linux), Win32 threads (WinNT) • Current work • Further programming models (for comparison) • Evaluation using various applications Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  34. SMiLE Shared Memory Programming SCI hardware DSM principle Using raw SCI via SISCI SCI-VM: A global virtual memory for SCI Shared Memory Programming Models  A sample model: SPMD Outlook on the Lab-Session

  35. What is SPMD ? • Single Program Multiple Data • The same code is executed on each node • Each node works on separate data • Characteristics of SPMD • Static task model (no dynamic spawn) • Fixed assignment of node IDs • Main synchronization construct: Barriers • All data stored in a globally accessible memory • Release Consistency Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  36. Task / Node management spmd_getNodeNumber spmd_getNodeCount Memory management spmd_alloc spmd_allocOpt Barrier spmd_allocBarrier spmd_barrier Locks spmd_allocLock spmd_lock spmd_unlock Synchronization routines spmd_sync Atomic counters Misc spmd_getGlobalMem spmd_distInitialData SMPD model Core routines used today • Task / Node management • spmd_getNodeNumber • spmd_getNodeCount • Memory management • spmd_alloc • spmd_allocOpt • Barrier • spmd_allocBarrier • spmd_barrier Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  37. Sample SPMD application • Successive Over-Relaxation (SOR) • Iterative method for solving PDEs • Advantage: Straightforward parallelization • Main concepts of SOR • Dense matrix • For each iteration • Go through whole matrix • Update points Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  38. Task distribution • Dense Matrix • Boundary • Split Matrixe.g. 2 nodes • Local boundaries • Overlap area with communication • Possibility for locality optim. Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  39. SPMD code for SOR application intcount = spmd_getNodeCount();intnumber = spmd_getNodeNumber(); start_row = (N / count) * number;end_row = (N / count) * (number+1); intbarrier = spmd_allocBarrier();float* matr = (float*) spmd_alloc(size); Doiter iterations Work on Matrix start_row – end_row;spmd_barrier(barrier); If (number == 0)Print the result; Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  40. Configuration & Startup • Startup • Install SCI-VM driver and IRM extension • Application execution • Shell on each node to run on • Execute program with identical parameters • Output on the individual shells • Configuration file • Placed in home dir. • SMP description(for certain models) • Configuration file • Placed in home dir. • SMP description(for certain models) ~/.hamster # Name ID CPUs smile1 4 2 smile2 68 2 Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  41. SMiLE Shared Memory Programming SCI hardware DSM principle Using raw SCI via SISCI SCI-VM: A global virtual memory for SCI Shared Memory Programming Models A sample model: SPMD  Outlook on the Lab-Session

  42. A warning for the Lab-Session • SCI-VM is still a prototype • First time out of the lab  • Remaining problems • Transparency of PCI bridges • OS integration • Hang-Ups may occur – In this case: • Remain calm and do not panic • Follow the instructions of the crew • Emergency exits on both sides • Most of the times: A restart/reboot will do the trick Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  43. Outline • Lab Setup • Experiments with the SISCI API • SISCI sample code • Add segment management code • Experiments with the SPMD model • SCI-VM / HAMSTER installation • Given: seq. SOR application • Goal: Parallelization Martin Schulz, SCI Summer School, Oct 2-4, Dublin

  44. Before the Lab-Session Any Questions ? http://smile.in.tum.de/ http://hamster.in.tum.de/ Martin Schulz, SCI Summer School, Oct 2-4, Dublin

More Related