Sensitivity of Cluster File System Access to I/O Server Selection

1. Sensitivity of Cluster File System Access to I/O Server Selection A. Apon, P. Wolinski, and G. Amerson University of Arkansas

2. Overview • Benchmarking study • Parallel Virtual File System (PVFS) • Network File System (NFS) • Testing parameters include • Pentium-based cluster node hardware • Myrinet interconnect • Varying number and configuration of I/O servers and client request patterns

3. Outline • File system architectures • Performance study design • Experimental results • Conclusions and future work

4. NFS Architecture • Client/server system • Single server for files NFS Server Node 1 Each cluster node has dual-processor Pentium Linux, HD, lots of memory Node 0 Node 2 Network Switch DATA FILE Node N

5. Node 0 Node 2 Node 1 PVFS Architecture • Also a client/server system • Many servers for each file • Fixed sized stripes in round-robin fashion DATA FILE Network Switch Each cluster node still has dual-processor Pentium Linux, HD, lots of memory

6. PVFS Architecture • One node is a manager node • Maintains metadata information for files • Configuration and usage options include: • Size of stripe • Number of I/O servers • Which nodes serve as I/O servers • Native PVFS API vs. UNIX/POSIX API

7. Native PVFS API example #include <pvfs.h> int main() { int fd, bytes; fd=pvfs_open(fn,O_RDONLY,0,NULL,NULL); ... pvfs_lseek(fd, offset, SEEK_SET); ... bytes_read = pvfs_read(fd, buf_ptr, bytes); ... pvfs_close(fd); }

8. Performance Study Design • Goals • Investigate the effect on cluster I/O when using the NFS server or the PVFS I/O servers also as clients • Compare PVFS with NFS

9. Performance Study Design • Experimental cluster • Seven dual-processor Pentium III 1GHz, 1GB memory computers • Dual EIDE disk RAID 0 subsystem in all nodes, measured throughput about 50MBps • Myrinet switches, 250MBps theoretical bandwidth

10. Performance Study Design • Two extreme client workloads • Local whole file (LWF) • Takes advantage of caching on server side • One process per node, each process reads the entire file from beginning to end Node 1 Node 2 Node N

11. Performance Study Design • Two extreme client workloads • Global whole file (GWF) • Minimal help from caching on the server side • One process per node, each process reads a different portion of the file, balanced workload Node 1 Node 2 Node N

12. NFS Parameters • Mount on Node 0 is a local mount • Optimization for NFS • NFS server can participate or not as a client in the workload

13. PVFS Parameters • A preliminary study was performed to determine the “best” stripe size and request size for the LWF and GWF workloads • Stripe size of 16KB • Request size of 16MB • File size of 1GB • All I/O servers for a given file participate in all requests for that file

14. System Software • RedHat Linux version 7.1 • Linux kernel version 2.4.17-rc2 • NFS protocol version 3 • PVFS version 1.5.3 • PVFS kernel version 1.5.3 • Myrinet network drivers gm-1.5-pre3b • MPICH version 1.2.1

15. Experimental Pseudocode For all nodes Open the test file Barrier synchronize with all clients Get start time Loop to read/write my portion Barrier synchronize with all clients Get end time Report bytes processed and time For Node 0 Receive bytes processed, report aggregate throughput

16. Clearcache • Clear NFS client and server-side caches • Unmount NFS directory, shutdown NFS • Restart NFS, remount NFS directories • Clear server-side PVFS cache • Unmount PVFS directories on all nodes • Shutdown PVFS I/O daemons, manager • Unmount pvfs-data directory on slaves • Restart PVFS manager, I/O daemons • Remount PVFS directories, all nodes

17. Experimental Parameters • Number of participating clients • Number of PVFS I/O servers • PVFS native API vs. UNIX/POSIX API • I/O servers (NFS as well as PVFS) may or may not also participate as clients

18. Experimental Results • NFS • PVFS native API vs UNIX/POSIX API • GWF, varying server configurations • LWF, varying server configurations

19. NFS, LWF and GWF with and without server reading

20. PVFS, LWF and GWFnative PVFS API vs. UNIX/POSIX API

21. PVFS UNIX/POSIX API compared to NFS

22. PVFS, GWF using native API servers added from Node 6 down

23. PVFS and NFS, GWF, 1 and 2 clients with/without server participating

24. PVFS, LWF using native API servers added from Node 6 down

25. PVFS and NFS, LWF, 1, 2, 3 clients with/without servers participating

26. PVFS, LWF and GWF, separate clients and servers, seven nodes

27. Conclusions • NFS can take advantage of a local mount • NFS performance is limited by contention at the single server • Limited to the disk throughput or the network throughput from the server, whichever has the most contention

28. Conclusions • PVFS performance generally improves (does not decrease) as the number of clients increases • More improvement seen with LWF workload than with the GWF workload • PVFS performance improves when the workload can take advantage of server-side caching

29. Conclusions • PVFS is better than NFS for all types of workloads where more than one I/O server can be used • PVFS UNIX/POSIX API performance is much less than the performance using the PVFS native API • May be improved by a new release of the Linux kernel

30. Conclusions • For a given number of servers, PVFS I/O throughput decreases when the servers also act as clients • For the workloads tested, PVFS system throughput increases to the maximum possible for the cluster when all nodes participate as both clients and servers

31. Observation • The drivers and libraries have been in constant upgrade during these studies. However, our recent experiences indicate that they are now stable and interoperate well together.

32. Future Work • Benchmarking with cluster workloads that include both computation and file access • Expand the benchmarking to a cluster with a higher number of PVFS clients and PVFS servers