WP4: Cloud Hosting Chapter Object Storage Generic Enabler

Transcription

1 WP4: Cloud Hosting Chapter Object Storage Generic Enabler Webinar John Kennedy, Thijs Intel

2 Outline 1

3 Overview of the Cloud Hosting Work Package 2

4 Functionality Trust and Security Operations FI-WARE Core Platform Overview Service delivery Cloud Hosting Internet of Things Support Services Developer tools I2ND = Interface to the Network and devices 3

5 Cloud Hosting Overview Lead: Partners: Cloud Hosting is the fundamental layer which provides the computation, storage and network resources, upon which applications are provisioned and managed Topic for today 4

6 Architecture Generic Enablers Architecture: Self-Service Interfaces User Portal Admin Portal User API Admin API Admin Toolkit DMTF IaaS API PaaS Management Topic for today Monitoring IaaS Service Management OCCI IaaS Resource Management IaaS Data Center Resource Management Cloud Edge Res. Mgmt CDMI Object Storage Metering & Accounting Security (WP8) Cloud Edge (WP7) Data Warehouse Business Framework (WP3) 5

7 Generic Enablers make it work R SM GE R Identity Management GE R DCRM GE Monitoring GE R Object Storage GE Core GEs Ecosystem GEs Cloud Proxy GE 6

8 Object Storage Generic Enabler 7

9 Object Storage Distributed High Availability Secure Fast Supports the storage of data in the cloud Open-Source Scalable Standard based! 8

10 Implementation OpenStack is the core of this Generic Enabler Originally developed by Nasa and Rackspace OpenStack Foundation: over 7000 individual members and 850 different organizations Licensed under Apache 2.0 OpenStack Swift is the Object Storage part of OpenStack 9

11 Swift Architecture The ring Object Swift Proxy Zone 1 Proxy Provides interface Rings One ring each for accounts / containers / objects Controls mapping of items to physical locations All contents replicated 3 times by default Object Server / Container Server / Account Server Stores raw content Zone 2 Zone 3 Zone 4 10

12 Swift Architecture (2) Replication Ensures enough replicas, and all are in sync. Manages deletions via special tombstone files. Updaters Handle (typically rare) occasions when updates to a ring cannot be immediate. E.g. under heavy load For example, adding a new object to a container requires updates to both the object ring and the container ring. If the container ring was unusually busy its update may be delayed. Thus for a short time a query of the container may not reveal the existence of the newly stored object. Hence swift is classed as being eventually consistent Auditors Check integrity of local server Quarantines and replaces any corrupted files Logs any issues 11

13 The Interface and API for Object Storage 12

14 Interface OpenStack Swift implements it s own API See BUT: Standard Based Interface is preferable Storage Networking Industry Association 400 member companies April 2010 released CDMI V1.0 Sept 2011 released CDMI V1.0.1 June 2012 released CDMI V1.0.2 CDMI: Cloud Data Management Interface Recently designated by ISO as an international standard See 13

15 CDMI: 10,000 foot view 14

16 Terminology Concept Capabilities Objects and Containers Metadata Security HTTP Support Description features of the storage implementation Objects: Piece of data Container: An abstraction for storage space (Container can have 0..* Objects) Data about data The usual suspects for a modern API 15

17 Capabilities Capabilities are features of the storage implementation Storage consumers can query the storage implementation for capabilities it supports, and decide whether appropriate to use it or not Example capabilities: cdmi_create_container cdmi_delete_container cdmi_read_metadata cdmi_modify_metadata cdmi_create_dataobject cdmi_delete_dataobject cdmi_read_value cdmi_modify_value 16

18 Objects and Containers Data Object Opaque piece of data Can have associated metadata Data can be text (e.g. utf-8 encoding) or binary ( base64 encoding) Container An abstraction for storage space Can have associated metadata Can contain zero or more data objects Although CDMI supports containers that contain other containers, for the first release of the Object Storage GE only top level Containers are recommended 17

19 Metadata Metadata is data about data. Both HTTP metadata and CDMI specific metadata can be found in the Object Storage GE HTTP Metadata HTTP specific metadata e.g. content-length CDMI specific metadata: Name-value pairs Vendor specific names begin with reverse domain name of vendor: e.g. eu.fi-ware or com.intel or com.ibm Three classes of CDMI specific metadata Data system metadata data requirements of data services. Defined by client, for use by the data services User metadata arbitrary JSON strings. Defined by the user, for use by the user Storage system metadata data about the stored data. Defined by the storage service for use by the user. E.g. creation-time, size, etc. 18

20 Security CDMI demands mandatory functionality supported by the Object Storage GE: Transport Security (HTTPS support) Security Capabilities describe security capabilities available The following security functionality is optional in CDMI: user and entity authentication authorization and access controls data integrity data and media sanitization data retention malware protection data at-rest encryption 19

21 HTTP Support Reserved characters in RFC3986 shall be escaped in all text strings in URIs E.g. A client retrieving a metadata item named "@user" from a container object with the name of "@MyContainer" would perform the following request: GET /%40MyContainer/?objectName;metadata:%40user HTTP/1.1 Host: cloud.example.com Accept: application/cdmi-container X-CDMI-Specification-Version: And receive the following response: HTTP/ OK Content-Type: application/cdmi-container X-CDMI-Specification-Version: { "objectname": "@MyContainer", "metadata": { "@user": "test" } } 20

22 Time As per CDMI standard, time will typically be in ISO 8601:2004 extended representation: YYYY-MM-DDThh:mm:ss.ssssssZ In particular: The full precision shall be specified the sub-second separator shall be a. the Z UTC zone indicator shall be included all timestamps shall be in UTC time zone The YYYY-MM-DDT24:00: Z hour shall not be used, and instead, it shall be represented as YYYY-MM-DDT00:00: Z. 21

23 Using CDMI with Swift 22

24 ObjectStorage GE API The ObjectStorage GE implements a CDMI compatible API on top of OpenStack Swift Originally developed by IBM, enhanced by Intel Provides a Restful API: Stateless HTTP commands Self documenting Can be manipulated by any client that can communicate in HTTP Network firewall issues minimised 23

25 CDMI - Walkthrough 24

26 Example API Walk-through Summarised from CDMI documentation. For complete details see: 25

27 Authentication $ curl -d '{"auth": {"project": "admin", "passwordcredentials": {"username":"admin", "password": "},"tenantid":"d418851c6d294381bbe6e d6"}}' - H"Content-type: application/json" {"access": {"token": {"expires": " T13:39:12Z", "id": "ea379b97b79c4f1bbaaedfa58ad48e82", [ ] {"endpoints": [ {"adminurl": " "region": "RegionOne", "internalurl": " 9686d6", "publicurl": " 9686d6"}], [ ] 26

28 Capability discovery $ curl -v -X GET -H 'X-Auth-Token: b518d061ce2844ac80d1e116c9d857ed' - H 'Accept: application/cdmi-capability' -H 'X-CDMI-Specification- Version: 1.0.1' d6/ < HTTP/ OK < Content-Type: application/cdmi-capability < X-Cdmi-Specification-Version: < Content-Length: 200 < X-Trans-Id: txdfa4fd5a2c8f4c7d91076b4d4a6b7f00 < Date: Tue, 06 Nov :47:23 GMT < { "capabilities": { "cdmi_list_children": true, "cdmi_create_container": true }, "objectname": "AUTH_d418851c6d294381bbe6e d6", "objecttype": "application/cdmi-capability" } 27

29 Create container $ curl -v -X PUT -H 'X-Auth-Token: 8197f bd3181ca79b0b884' - H 'Content-Type: application/directory'-h 'Content-Length: 0' foo/ < HTTP/ Created < Content-Length: 18 < Content-Type: text/html; charset=utf-8 < X-Trans-Id: tx63ffae10c01542ca8c8b07b130d0387c < Date: Tue, 06 Nov :59:27 GMT < 201 Created 28

30 Capability discovery (2) $ curl -v -X GET -H 'X-Auth-Token: b518d061ce2844ac80d1e116c9d857ed' - H 'Accept: application/cdmi-capability' -H 'X-CDMI-Specification- Version: 1.0.1' d6/foo/ [ ] { "parenturi": "cdmi_capabilities/[ ]", "objectname": "foo/", "capabilities": { "cdmi_create_dataobject": true, "cdmi_read_metadata": true, "cdmi_modify_metadata": true, "cdmi_list_children": true, "cdmi_create_container": true, "cdmi_copy_dataobject": true, "cdmi_delete_container": true }, "objecttype": "application/cdmi-capability } 29

31 Create object $ curl -v -X PUT -d '{"mimetype":"text/plain", "metadata":{}, "value":"bar"}' -H 'X-Auth-Token: 8197f bd3181ca79b0b884' -H 'Accept: application/cdmi-object' -H 'Content-Type: application/cdmiobject' -H 'X-CDMI-Specification-Version: 1.0.1' < HTTP/ Created < Content-Type: text/html; charset=utf-8 < Etag: 37b51d194a7513e45b56f6524f2d51f2 < Last-Modified: Tue, 06 Nov :00:17 GMT < Content-Length: 155 < X-Trans-Id: tx03b9cb b118e9fd7d927d1cf4c < Date: Tue, 06 Nov :00:17 GMT < { "parenturi": "AUTH_d418851c6d294381bbe6e d6/foo/", "metadata": {}, "objectname": "text_doc", "objecttype": "application/cdmi-object" } 30

32 List container contents $ curl -v -X GET -H 'X-Auth-Token: 8197f bd3181ca79b0b884' -H 'Content-Type: application/directory oo/ [ ] { "completionstatus": "Complete", "objectname": "foo/", "capabilitiesuri": "cdmi_capabilities/auth_d418851c6d294381bbe6e d6/foo/", "parenturi": "AUTH_d418851c6d294381bbe6e d6/", "objecttype": "application/cdmi-container", "children": [ "test", "text_doc" ], "metadata": {}} 31

33 Retrieve object $ curl -v -X GET -H 'X-Auth-Token: 8197f bd3181ca79b0b884' - H 'Accept: application/cdmi-object' -H 'X-CDMI-Specification-Version: 1.0.1' foo/text_doc [ ] { "completionstatus": "Complete", "mimetype": "text/plain", "valuetransferencoding": "utf-8", "objectname": "text_doc", "capabilitiesuri": "cdmi_capabilities/auth_d418851c6d294381bbe6e d6/foo/text_doc/ ", "parenturi": "AUTH_d418851c6d294381bbe6e d6/", "value": "bar", "valuerange": "0-3", "objecttype": "application/cdmi-object", "metadata": {} } 32

34 Retrieve object (2) $ curl -v -X GET -H 'X-Auth-Token: 8197f bd3181ca79b0b884' - H Accept: text/plain' foo/text_doc < HTTP/ OK < X-Object-Meta-Valuetransferencoding: utf-8 < Last-Modified: Tue, 06 Nov :00:17 GMT < Etag: 37b51d194a7513e45b56f6524f2d51f2 < Accept-Ranges: bytes < Content-Length: 3 < Content-Type: text/plain < X-Trans-Id: tx9d41b7989b0d4d ce9b700aa01 < Date: Tue, 06 Nov :05:12 GMT < bar 33

35 Delete object $ curl -v -X DELETE -H 'X-Auth-Token: 8197f bd3181ca79b0b884' foo/text_doc < HTTP/ No Content < Content-Length: 0 < Content-Type: text/html; charset=utf-8 < X-Trans-Id: tx6378ad47919b427a8da6fc9fe214f96e < Date: Tue, 06 Nov :11:16 GMT < 34

36 Delete container $ curl -v -X DELETE -H 'X-Auth-Token: 8197f bd3181ca79b0b884' foo < HTTP/ No Content < Content-Length: 0 < Content-Type: text/html; charset=utf-8 < X-Trans-Id: txc1df346456fa4729af51c92fc613ffcd < Date: Tue, 06 Nov :11:53 GMT < 35

37 Swift & Horizon FI-WARE Testbed: 36

38 In Summary Supporting the storage of arbitrary data in the cloud Implementation - open-source based on OpenStack Swift highly-available distributed eventually-consistent multi-tenant secure Interface - standards-based CDMI Restful Stateless 37

39 Resources FI-WARE: FI-WARE Wiki: Object Storage Generic Enabler OpenStack Swift CDMI CDMI for OpenStack 38

40 Thank you! 39