Object-based Storage

1. Object-based Storage Long Liu 2010-10-23

2. Outline • Why do we need object based storage? • What is object based storage? • How to take advantage of it? • What's the status of object based storage? • What can we do about it?

3. Outline • Why do we need object based storage? • What is object based storage? • How to take advantage of it? • What's the status of object based storage? • What can we do about it?

4. Background • Existing enterprise storage infrastructures are feeling the strain • the volume of data generated by many network-based applications continues to escalate

5. Background Two Technologies: • Files: • Flexible data sharing • Secure • Blocks: • High performance • Scalable Files Blocks

6. Comparison Objects can be regarded as the convergence of two technologies: files and blocks Operations: Create object Delete object Read object Write object Addressing: [object, byte range] Operations: Read block Write block Addressing: Block range Object Based Disk Block Based Disk

7. Comparison Traditional Storage Object-based Storage Applications Applications File System File System Object Interface Storage component Logical Block Interface Hard Drive Object-based Storage Device (OSD)

8. Comparison CPU Applications System Call Interface File System User Component File System Storage Component Block I/O Manager Storage Device (a) Traditional model (b) Object storage model • Two changes : • Object-based storage offloads the storage component to the storage device • The device interface changes from blocks to objects • CPU • Applications • System Call Interface • File System • User Component • Object Interface Block Interface File System Storage Component Block I/O Manager Storage Device

9. Motivation Improved device and data sharing – Platform-dependent metadata moved to device • • Improved scalability & security – Devices directly handle client requests – Object security • Improved performance – Data types can be differentiated at the device • Improved storage management – Self-managed, policy-driven storage – Storage devices become more autonomous Objects

10. Outline • Why do we need object based storage? • What is object based storage? • How to take advantage of it? • What's the status of object based storage? • What can we do about it?

11. Object-based Storage • Object • OSD(Object-based Storage Device) • MDS(Metadata Server)

12. Object-based Storage • Object • OSD(Object-based Storage Device) • MDS(Metadata Server)

13. Object • An object is a logical unit of storage • ID (Identification) • Application data • Metadata which includes block allocation and length • Attributes that is accessible by users • Objects have file-like methods • open, close, read, write

14. Object • The root object -- The OSD itself • User object -- Created by SCSI commands from the application or client • Collection object -- A group of user objects, such as all .mp3 • Partition object -- Containers that share common security and space managementcharacteristics

15. Object P4 P3 P2 OSD P1 Root Object (one per device) Partition Objects U1 User Data Collection Objects Metadata Attributes • User Objects(for user data) Object ID

16. Object-based Storage • Object • OSD(Object-based Storage Device) • MDS(Metadata Server)

17. Object Storage Devices

18. Object Storage Devices CPU Applications System Call Interface File System User Component File System Storage Component Block I/O Manager Storage Device (a) Traditional model (b) Object storage model • Two changes : • Object-based storage offloads the storage component to the storage device • The device interface changes from blocks to objects • CPU • Applications • System Call Interface • File System • User Component • Object Interface Interface File System Storage Component Block I/O Manager Storage Device

19. Object Storage Devices Expect wide variety of Object Storage Devices Disk array subsystem 2 SATA disks – 240/500 GB Highly integrated, single disk 4 Gbps per shelf to cluster Orchestrates system activity Balances objects across OSDs Stores up to 5 TBs per shelf

20. Object-based Storage • Object • OSD(Object-based Storage Device) • MDS(Metadata Server)

21. Metadata Server

22. Client Storage component Metadata Servers (MDS) Object Storage Devices (OSDs) Object-based Storage Device (OSD) Dataflow of Metadata Applications Client File System Metadata Manager

23. Outline • Why do we need object based storage? • What is object based storage? • How to take advantage of it? • What's the status of object based storage? • What can we do about it?

24. Interface OSD Client OSD Interface T10 OSD Commands(face) CREATE/REMOVE GET/SET ATTR READ/WRITE Opaque attributes (stored only) Shared attributes (stored & processed) Attribute pages OSD Target 10001110101 10000001110 11001110111 10001111000 ..… User Data Object

25. OSD Commands(Interface) • Security – Authorization – Integrity – SET KEY – SET MASTER KEY • Groups – CREATE COLLECTION – REMOVE COLLECTION – LIST COLLECTION • Management – CREATE PARTITION – REMOVE PARTITION – FLUSH PARTITION – PERFORM SCSI COMMAND – PERFORM TASK MGMT • Basic Protocol – READ – WRITE – CREATE – REMOVE – GET ATTR – SET ATTR • Specialized – APPEND – CREATE & WRITE – FLUSH – LIST very basic shared secrets space mgmt attributes • opaque • internal • shared

26. Storage Technology Today • Direct attached storage (DAS) • Fabric Attached Storage (FAS) -Network Attached Storage (NAS) -Storage Area Networks (SAN)

27. Direct Attached Storage LAN Windows UNIX Windows A traditional Direct Attached Storage model RAID BACKUP SERVER RAID RAID RAID TAPE

28. Fabric Attached Storage C/S Fabric Attached Storage SERVER DATA CLIENTS

29. Network Attached Storage This figure illustrates NAS being used to share files among a number of clients. Clients File I/O IP network Storage area network Block I/O The files themselves may be stored on a fast SAN File server Block storage

30. Storage Area Networks Clients This figure illustrates a SAN file system The files themselves are stored on a fast storage Area to which the clients are also attached. Servers Metadata Storage area network Data Management Block-based storage devices

31. Security CLIENTS METADATA SERVERS Attribute Capability NETWORK Attribute Capability Data Management OBJECT-BASED STORAGE DEVICE

32. Outline • Why do we need object based storage? • What is object based storage? • How to take advantage of it? • What's the status of object based storage? • What can we do about it?

33. Status • Industrial • Lustre • Panasas • Academic

34. Status • Industrial • Lustre • Panasas • Academic

35. Lustre • First open sourced system with object storage • High-performance parallel file system • Consist of clients, MDS and OST(Object Storage Targets)

36. Lustre Metadata Metadata METADATA SERVER CLUSTER Data Management NETWORK CLIENTS Management Data OBJECT STORAGE TARGETS(OST)

37. Panasas • High-performance file system • Consist of OSD, Panasas File System, MDS

38. Panasas KeyObjectStorageFeatures Intelligent space management in storage layer ßMedia geometry aware placement ßData aware prefetching, caching & recovery Encapsulation of data and attributes ßNative object interface, good programming model ßStorage interpreted attributes for per file properties KeyObjectStorageAdvantages ßRobust, shared access by many clients ßScalable performance via an offloaded data path ßStrong fine-grained end-to-end security

39. Panasas • Clients are from Energy, Government, Finance, Manufacturing and Higher Education

40. Status • Industrial • Lustre • Panasas • Academic

41. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Motivation: • Metadata server cluster maybe the bottleneck • Frequent metadata access and movement • Terrible load balance management

42. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Application Server Cluster VoD Server Web Server E-mail Server File Server Database Server Metadata Data Storage Network (Fibre Channel) MDS Cluster Security Object-based Storage Device Cluster Object-based Storage System Architecture

43. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Application Servers Application Hashing Partition: File Hashing Manager Mapping Manager Metadata Server Cluster • A total solution for • File hashing • Metadata partitioning • Metadata storage Hashing Partition Logical Partition Manager Metadata Server Backend Common Storage Space Figure 3. Hashing Partition

44. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Pathname: /Dir1/Dir2/filename 1 4 ①.Filename hashing ②.Selecting MDS through Mapping Manager ③ .Accessing metadata by pathname hashing result ④.Returning metadata to application server Pathname Hashing Result (i) Pathname Mapping Manager Metadata & etc 2 Pathname Hashing Result (i+1) Pathname Metadata Server Cluster Metadata & etc 3 Logical Partitions Figure 4. Metadata Access Pattern

45. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Hashing Partition Mapping Manager 2 1 3 Metadata Server Cluster Logical Partitions 4 Common Storage Space Figure 5. MDS cluster failover procedure

46. Outline • Why do we need object based storage? • What is object based storage? • How to take advantage of it? • What's the status of object based storage? • What can we do about it?

47. From Our Perspective

48. The End Thank you