WP4: Cloud Hosting Chapter Object Storage Generic Enabler Webinar John Kennedy, Thijs Metsch@ Intel
Outline 1
Overview of the Cloud Hosting Work Package 2
Functionality Trust and Security Operations FI-WARE Core Platform Overview Service delivery Cloud Hosting Internet of Things Support Services Developer tools I2ND = Interface to the Network and devices 3
Cloud Hosting Overview Lead: Partners: Cloud Hosting is the fundamental layer which provides the computation, storage and network resources, upon which applications are provisioned and managed Topic for today 4
Architecture Generic Enablers Architecture: Self-Service Interfaces User Portal Admin Portal User API Admin API Admin Toolkit DMTF IaaS API PaaS Management Topic for today Monitoring IaaS Service Management OCCI IaaS Resource Management IaaS Data Center Resource Management Cloud Edge Res. Mgmt CDMI Object Storage Metering & Accounting Security (WP8) Cloud Edge (WP7) Data Warehouse Business Framework (WP3) 5
Generic Enablers make it work R SM GE R Identity Management GE R DCRM GE Monitoring GE R Object Storage GE Core GEs Ecosystem GEs Cloud Proxy GE 6
Object Storage Generic Enabler 7
Object Storage Distributed High Availability Secure Fast Supports the storage of data in the cloud Open-Source Scalable Standard based! 8
Implementation OpenStack is the core of this Generic Enabler Originally developed by Nasa and Rackspace OpenStack Foundation: over 7000 individual members and 850 different organizations Licensed under Apache 2.0 OpenStack Swift is the Object Storage part of OpenStack http://www.openstack.org/software/openstackstorage/ http://swift.openstack.org 9
Swift Architecture The ring Object Swift Proxy Zone 1 Proxy Provides interface Rings One ring each for accounts / containers / objects Controls mapping of items to physical locations All contents replicated 3 times by default Object Server / Container Server / Account Server Stores raw content Zone 2 Zone 3 Zone 4 10
Swift Architecture (2) Replication Ensures enough replicas, and all are in sync. Manages deletions via special tombstone files. Updaters Handle (typically rare) occasions when updates to a ring cannot be immediate. E.g. under heavy load For example, adding a new object to a container requires updates to both the object ring and the container ring. If the container ring was unusually busy its update may be delayed. Thus for a short time a query of the container may not reveal the existence of the newly stored object. Hence swift is classed as being eventually consistent Auditors Check integrity of local server Quarantines and replaces any corrupted files Logs any issues 11
The Interface and API for Object Storage 12
Interface OpenStack Swift implements it s own API See http://docs.openstack.org/api/openstack-object-storage/1.0/content/ BUT: Standard Based Interface is preferable Storage Networking Industry Association 400 member companies April 2010 released CDMI V1.0 Sept 2011 released CDMI V1.0.1 June 2012 released CDMI V1.0.2 CDMI: Cloud Data Management Interface Recently designated by ISO as an international standard See http://cdmi.sniacloud.org/ 13
http://cdmi.sniacloud.org/cdmi_spec/5-overview_of_cloud_storage/cloudstoragerefmodelnoshadow.jpg CDMI: 10,000 foot view 14
Terminology Concept Capabilities Objects and Containers Metadata Security HTTP Support Description features of the storage implementation Objects: Piece of data Container: An abstraction for storage space (Container can have 0..* Objects) Data about data The usual suspects for a modern API 15
Capabilities Capabilities are features of the storage implementation Storage consumers can query the storage implementation for capabilities it supports, and decide whether appropriate to use it or not Example capabilities: cdmi_create_container cdmi_delete_container cdmi_read_metadata cdmi_modify_metadata cdmi_create_dataobject cdmi_delete_dataobject cdmi_read_value cdmi_modify_value 16
Objects and Containers Data Object Opaque piece of data Can have associated metadata Data can be text (e.g. utf-8 encoding) or binary ( base64 encoding) Container An abstraction for storage space Can have associated metadata Can contain zero or more data objects Although CDMI supports containers that contain other containers, for the first release of the Object Storage GE only top level Containers are recommended 17
Metadata Metadata is data about data. Both HTTP metadata and CDMI specific metadata can be found in the Object Storage GE HTTP Metadata HTTP specific metadata e.g. content-length CDMI specific metadata: Name-value pairs Vendor specific names begin with reverse domain name of vendor: e.g. eu.fi-ware or com.intel or com.ibm Three classes of CDMI specific metadata Data system metadata data requirements of data services. Defined by client, for use by the data services User metadata arbitrary JSON strings. Defined by the user, for use by the user Storage system metadata data about the stored data. Defined by the storage service for use by the user. E.g. creation-time, size, etc. 18
Security CDMI demands mandatory functionality supported by the Object Storage GE: Transport Security (HTTPS support) Security Capabilities describe security capabilities available The following security functionality is optional in CDMI: user and entity authentication authorization and access controls data integrity data and media sanitization data retention malware protection data at-rest encryption 19
HTTP Support Reserved characters in RFC3986 shall be escaped in all text strings in URIs E.g. A client retrieving a metadata item named "@user" from a container object with the name of "@MyContainer" would perform the following request: GET /%40MyContainer/?objectName;metadata:%40user HTTP/1.1 Host: cloud.example.com Accept: application/cdmi-container X-CDMI-Specification-Version: 1.0.1 And receive the following response: HTTP/1.1 200 OK Content-Type: application/cdmi-container X-CDMI-Specification-Version: 1.0.1 { "objectname": "@MyContainer", "metadata": { "@user": "test" } } 20
Time As per CDMI standard, time will typically be in ISO 8601:2004 extended representation: YYYY-MM-DDThh:mm:ss.ssssssZ In particular: The full precision shall be specified the sub-second separator shall be a. the Z UTC zone indicator shall be included all timestamps shall be in UTC time zone The YYYY-MM-DDT24:00:00.000000Z hour shall not be used, and instead, it shall be represented as YYYY-MM-DDT00:00:00.000000Z. 21
Using CDMI with Swift 22
ObjectStorage GE API The ObjectStorage GE implements a CDMI compatible API on top of OpenStack Swift Originally developed by IBM, enhanced by Intel https://github.com/osaddon/cdmi Provides a Restful API: Stateless HTTP commands Self documenting Can be manipulated by any client that can communicate in HTTP Network firewall issues minimised 23
CDMI - Walkthrough 24
Example API Walk-through Summarised from CDMI documentation. For complete details see: http://cdmi.sniacloud.org 25
Authentication $ curl -d '{"auth": {"project": "admin", "passwordcredentials": {"username":"admin", "password": "},"tenantid":"d418851c6d294381bbe6e082849686d6"}}' - H"Content-type: application/json" http://130.206.80.100:5000/v2.0/tokens {"access": {"token": {"expires": "2012-11-07T13:39:12Z", "id": "ea379b97b79c4f1bbaaedfa58ad48e82", [ ] {"endpoints": [ {"adminurl": "http://130.206.80.102:8080/v1", "region": "RegionOne", "internalurl": "http://130.206.80.102:8080/v1/auth_d418851c6d294381bbe6e08284 9686d6", "publicurl": "http://130.206.80.102:8080/v1/auth_d418851c6d294381bbe6e08284 9686d6"}], [ ] 26
Capability discovery $ curl -v -X GET -H 'X-Auth-Token: b518d061ce2844ac80d1e116c9d857ed' - H 'Accept: application/cdmi-capability' -H 'X-CDMI-Specification- Version: 1.0.1' http://130.206.80.102:8080/cdmi_capabilites/auth_d418851c6d294381bbe6e 082849686d6/ < HTTP/1.1 200 OK < Content-Type: application/cdmi-capability < X-Cdmi-Specification-Version: 1.0.1 < Content-Length: 200 < X-Trans-Id: txdfa4fd5a2c8f4c7d91076b4d4a6b7f00 < Date: Tue, 06 Nov 2012 13:47:23 GMT < { "capabilities": { "cdmi_list_children": true, "cdmi_create_container": true }, "objectname": "AUTH_d418851c6d294381bbe6e082849686d6", "objecttype": "application/cdmi-capability" } 27
Create container $ curl -v -X PUT -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' - H 'Content-Type: application/directory'-h 'Content-Length: 0' http://130.206.80.102:8080/cdmi/auth_d418851c6d294381bbe6e082849686d6/ foo/ < HTTP/1.1 201 Created < Content-Length: 18 < Content-Type: text/html; charset=utf-8 < X-Trans-Id: tx63ffae10c01542ca8c8b07b130d0387c < Date: Tue, 06 Nov 2012 13:59:27 GMT < 201 Created 28
Capability discovery (2) $ curl -v -X GET -H 'X-Auth-Token: b518d061ce2844ac80d1e116c9d857ed' - H 'Accept: application/cdmi-capability' -H 'X-CDMI-Specification- Version: 1.0.1' http://130.206.80.102:8080/cdmi_capabilites/auth_d418851c6d294381bbe6e 082849686d6/foo/ [ ] { "parenturi": "cdmi_capabilities/[ ]", "objectname": "foo/", "capabilities": { "cdmi_create_dataobject": true, "cdmi_read_metadata": true, "cdmi_modify_metadata": true, "cdmi_list_children": true, "cdmi_create_container": true, "cdmi_copy_dataobject": true, "cdmi_delete_container": true }, "objecttype": "application/cdmi-capability } 29
Create object $ curl -v -X PUT -d '{"mimetype":"text/plain", "metadata":{}, "value":"bar"}' -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' -H 'Accept: application/cdmi-object' -H 'Content-Type: application/cdmiobject' -H 'X-CDMI-Specification-Version: 1.0.1' http://130.206.80.102:8080/[...]/foo/text_doc < HTTP/1.1 201 Created < Content-Type: text/html; charset=utf-8 < Etag: 37b51d194a7513e45b56f6524f2d51f2 < Last-Modified: Tue, 06 Nov 2012 14:00:17 GMT < Content-Length: 155 < X-Trans-Id: tx03b9cb0122754b118e9fd7d927d1cf4c < Date: Tue, 06 Nov 2012 14:00:17 GMT < { "parenturi": "AUTH_d418851c6d294381bbe6e082849686d6/foo/", "metadata": {}, "objectname": "text_doc", "objecttype": "application/cdmi-object" } 30
List container contents $ curl -v -X GET -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' -H 'Content-Type: application/directory http://130.206.80.102:8080/cdmi/auth_d418851c6d94381bbe6e082849686d6/f oo/ [ ] { "completionstatus": "Complete", "objectname": "foo/", "capabilitiesuri": "cdmi_capabilities/auth_d418851c6d294381bbe6e082849686d6/foo/", "parenturi": "AUTH_d418851c6d294381bbe6e082849686d6/", "objecttype": "application/cdmi-container", "children": [ "test", "text_doc" ], "metadata": {}} 31
Retrieve object $ curl -v -X GET -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' - H 'Accept: application/cdmi-object' -H 'X-CDMI-Specification-Version: 1.0.1' http://130.206.80.102:8080/cdmi/auth_d418851c6d294381bbe6e082849686d6/ foo/text_doc [ ] { "completionstatus": "Complete", "mimetype": "text/plain", "valuetransferencoding": "utf-8", "objectname": "text_doc", "capabilitiesuri": "cdmi_capabilities/auth_d418851c6d294381bbe6e082849686d6/foo/text_doc/ ", "parenturi": "AUTH_d418851c6d294381bbe6e082849686d6/", "value": "bar", "valuerange": "0-3", "objecttype": "application/cdmi-object", "metadata": {} } 32
Retrieve object (2) $ curl -v -X GET -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' - H Accept: text/plain' http://130.206.80.102:8080/cdmi/auth_d418851c6d294381bbe6e082849686d6/ foo/text_doc < HTTP/1.1 200 OK < X-Object-Meta-Valuetransferencoding: utf-8 < Last-Modified: Tue, 06 Nov 2012 14:00:17 GMT < Etag: 37b51d194a7513e45b56f6524f2d51f2 < Accept-Ranges: bytes < Content-Length: 3 < Content-Type: text/plain < X-Trans-Id: tx9d41b7989b0d4d2493265ce9b700aa01 < Date: Tue, 06 Nov 2012 14:05:12 GMT < bar 33
Delete object $ curl -v -X DELETE -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' http://130.206.80.102:8080/cdmi/auth_d418851c6d294381bbe6e082849686d6/ foo/text_doc < HTTP/1.1 204 No Content < Content-Length: 0 < Content-Type: text/html; charset=utf-8 < X-Trans-Id: tx6378ad47919b427a8da6fc9fe214f96e < Date: Tue, 06 Nov 2012 14:11:16 GMT < 34
Delete container $ curl -v -X DELETE -H 'X-Auth-Token: 8197f75188304431bd3181ca79b0b884' http://130.206.80.102:8080/cdmi/auth_d418851c6d294381bbe6e082849686d6/ foo < HTTP/1.1 204 No Content < Content-Length: 0 < Content-Type: text/html; charset=utf-8 < X-Trans-Id: txc1df346456fa4729af51c92fc613ffcd < Date: Tue, 06 Nov 2012 14:11:53 GMT < 35
Swift & Horizon FI-WARE Testbed: http://portal.lab.fi-ware.eu 36
In Summary Supporting the storage of arbitrary data in the cloud Implementation - open-source based on OpenStack Swift highly-available distributed eventually-consistent multi-tenant secure Interface - standards-based CDMI Restful Stateless 37
Resources FI-WARE: www.fi-ware.eu FI-WARE Wiki: https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php Object Storage Generic Enabler http://catalogue.fi-ware.eu/enablers/object-storage-ge-fi-wareimplementation OpenStack Swift http://swift.openstack.org/ CDMI http://www.snia.org/cdmi CDMI for OpenStack https://github.com/osaddon/cdmi 38
Thank you! http://catalogue.fi-ware.eu/enablers/object-storage-ge-fi-ware-implementation 39