HTTP REST API for OpenStack Object Storage Metadata Search (OSMS)

Transcription

1 HTTP REST API for OpenStack Object Storage Metadata Search (OSMS) Revision History Version Date Author Changes Mar-2013 Lincoln Thomas Initial draft Jul-2013 Lincoln Thomas Cumulative changes from comments and work by OSMS team so far (mostly Tomer Meshorer and Kimberly Keeton). Better alignment with OpenStack Object Storage API Aug-2013 Lincoln Thomas Updates from OSMS team review meetings and further investigation Aug-2013 Lincoln Thomas Updates from OSMS team review meetings and further investigation. More explanations and details Sep-2013 Lincoln Thomas Updates during design and implementation phases. Some simplifications, more explanations, and example queries added Oct-2013 Lincoln Thomas Minor wording changes Oct-2013 Lincoln Thomas Minor clarifications per comments from Perry Merritt Nov-2013 Lincoln Thomas Further updates per comments from Perry and team review. Introduction This document specifies a set of REST API additions to OpenStack's Object Storage API to support a new metadata search feature, abbreviated OSMS, for Object Storage Metadata Search. The OSMS API extends the set of features described in the base OpenStack Object Storage API, which can be found at The OSMS feature maintains metadata for Object Storage accounts, containers, and objects. OSMS provides an HTTP REST API with a rich set of metadata search parameters for users and applications to search the stored metadata. We use the term "item" to refer to an account, container, or object, when the distinction is unnecessary. The HTTP REST request defines the base URI to search, the search criteria, the list of metadata fields to return for the items matching the criteria, and how to filter and format the resulting output. 6 November 2013 Page 1 of 33 Hewlett-Packard Company

2 System and custom metadata Two types of metadata are supported for searches, and both can be referenced in the same search: System metadata applies to all items (accounts, containers, and objects). Each item stored includes a fixed set of attributes comprising its system metadata. System metadata attributes cannot be deleted by the user through the API. Custom metadata applies only to items where the user assigns them using the "X- [Account Container Object]-Meta-<key>: <value>" HTTP header. Custom metadata names are user-defined, with value strings also defined by the user. Custom metadata can be added, replaced, or deleted by the user as specified in the OpenStack API. Custom metadata attributes are distinguished from system metadata attributes by the [account container object]_meta_ prefix. Metadata attributes available The following table describes the system and custom metadata attributes available for searches using the OSMS API. For descriptions of the Types column, see the section "Data types." Live and deleted items With the exception of the *delete_time attributes, all of the system metadata attributes listed in this table are valid for live (not-yet-deleted) items. For deleted items, only the following attributes are valid: *uri, *name, *delete_time, and *last_activity_time. By default, only results for live items will be returned. To include deleted items in the results, include the appropriate *delete_time system attribute(s), either as part of the attribute list to be returned, or as a query criterion. For example, including delete_time or account_delete_time will cause account results to include live and deleted items that meet the query criteria to be returned. System attribute Type Description Example value account_uri string An account in URI format, /ABSmith which is a slash character followed by the account_name, with no trailing slash. account_name string The name of an account, ABSmith as defined by the authentication mechanism. Matches the account name provided in HTTP requests. account_tenant_id string The name of the OpenStack Keystone AlbertBSmith 6 November 2013 Page 2 of 33 Hewlett-Packard Company

3 account_first_use_tim e date tenant that maps to this account. If Keystone is not used as the authentication mechanism, this attribute will be empty. The time at which the account was first referenced by a base Object Storage API request T19:02: Z account_last_modified _time account_last_changed_ time date date An account is never explicitly added by an API request. It is added when an API request first references the account. For example, creating the first container in an account causes a new account record to be added to the object store. Other examples include listing containers or adding custom metadata to an account, if the account has never been referenced before. The most recent time at which the list of containers changed. It does not include system nor custom metadata changes on the account. The most recent time at which any metadata about the account changed, or the list of containers changed. account_delete_time date The time at which the account was removed from the object store T19:02: Z T19:02: Z T19:02: Z account_last_activity _time date For live accounts, equivalent to account_last_changed_ti me T19:02: Z For deleted accounts, equivalent to 6 November 2013 Page 3 of 33 Hewlett-Packard Company

4 account_delete_time. The attribute provides a single source for the time of the last action upon an account, regardless of whether it is live or deleted. It can be used to query for all changes made to this account (including deletions) since a specified point in time. account_container_cou nt numeric Deleted accounts are only queried or returned according to the rules in the section "Live and deleted items". The number of containers in this account. 12 account_object_count numeric The number of objects in all containers in this account. account_bytes_used numeric The number of bytes used by all objects in all containers in this account. Containers themselves do not contribute to the bytes used, only their objects. account_meta_<name> string The value of the account custom metadata attribute <name> Example attribute name: account_meta_usergro up Matches the "X- Account-Meta- <name>:<value>" header in HTTP requests. container_uri string A container in URI format, which is an account_uri, then a slash, then the container_name, with no trailing slash. container_name string The name of a container. Matches the container name provided in HTTP requests. Example value: Administrators /ABSmith/container1 container1 container_account_nam string The name of the account ABSmith 6 November 2013 Page 4 of 33 Hewlett-Packard Company

5 e that includes this container. container_create_time date The time at which the container was created. container_last_modifi ed_time container_last_change d_time date date The most recent time at which the list of objects changed. It does not include system nor custom metadata changes on the container. The most recent time at which any metadata about the container changed, or the list of objects changed. container_delete_time date The time at which the container was deleted T19:02: Z T19:02: Z T19:02: Z T19:02: Z container_last_activi ty_time date For live containers, equivalent to container_last_changed_ti me T19:02: Z For deleted containers, equivalent to container_delete_time. container_read_permis sions container_write_permi ssions Refer to the account_last_activity_tim e attribute and the section "Live and deleted items" for details. string The value of the X- Container-Read header, obtained from the HTTP request when the container is created or modified. Empty if no such header is ever supplied. string The value of the X- Container-Write header, obtained from the HTTP request when the container is created or modified..r:*,.r:-.thief.com,bobs_acco unt,sues_account:sue,.rlistings bobs_account,sues_ac count:sue 6 November 2013 Page 5 of 33 Hewlett-Packard Company

6 Empty if no such header is ever supplied. container_sync_to string The container URL to mirror this container's contents to, for container sync. Includes the full HTTP URL with cluster name and API version followed by the containerlevel URI. BSmith/container2 Matches the X- Container-Sync-To header in HTTP requests when container sync is configured. Empty if no such header is ever supplied. container_sync_key string A secret key string defined during container sync to allow this container's contents to be mirrored to another container that accepts this key. mysecretkey Matches the X- Container-Sync-Key header in HTTP requests when container sync is configured. Empty if no such header is ever supplied. container_versions_lo cation string The container URI containing previous versions of objects in this container, if object versioning is configured. /ABSmith/container1o ldversions Matches the X- Versions-Location header in HTTP requests when container sync is configured. Empty if no such header is ever supplied. 6 November 2013 Page 6 of 33 Hewlett-Packard Company

7 container_object_coun t numeric The number of objects in this container. 768 container_bytes_used numeric The number of bytes used by all objects in this container. Containers themselves do not contribute to the bytes used, only their objects. container_meta_<name> string The value of the container custom metadata attribute <name>. Matches the "X- Container-Meta- <name>:<value>" header in HTTP requests. object_uri string An object in URI format, which is a container_uri, then a slash character, then the object_name, with no trailing slash Example attribute name: container_meta_depar tment Example value: sales /ABSmith/container1/ objectdir1/subdir1/p hoto.jpg In this example, the object_name is objectdir1/subdir1/p hoto.jpg See the object_name description below. object_name string The name of an object, which must be a string that is unique within a container. Matches the object name provided in HTTP requests. objectdir1/subdir1/p hoto.jpg Note that the object_name may contain a "pseudo path" of directory names and a file name, as in this example, or it may contain any arbitrary unique string such as "myobject". When pseudo-paths are used, the path is only meaningful to the client. No such directory 6 November 2013 Page 7 of 33 Hewlett-Packard Company

8 hierarachy exists in the object store, and the path is unrelated to where the object is stored on the object store's internal file system. object_account_name string The name of the account that includes this object. object_container_name string The name of the container that includes this object. object_location string The URL of the location of the object on the storage system. The URL may be accessible from the client or may be accessible only from the storage system. ABSmith container1 file:///fs1/objectst oreroot/node /objects/576/0c9 /240dcb5fac9bb402f06 b43a3b1ae50c9/ data object_uri_create_tim e date This attribute is intended to be useful to administrators that manage an object store. If an implementation chooses not to expose this attribute, for security or other reasons, it can omit this attribute from the attr_list of the services request. If omitted, any client use of this attribute will return in the HTTP error code 405 (Method Not Allowed) in the response. See the section "Services request" for details. The time at which the first object with this URI was created. In this example, the URL is a file location accessible only to users logged in to one of the server nodes of the storage system T19:02: Z If an object with this URI does not already exist when the object is uploaded with a PUT, then this attribute is assigned the same value as object_last_modified_tim e. 6 November 2013 Page 8 of 33 Hewlett-Packard Company

9 object_last_modified_ time object_last_changed_t ime date date If an object with this URI already exists when the object is uploaded, then this attribute is not changed. The most recent time at which the object contents changed. It does not include system nor custom metadata changes on the object. The most recent time at which any metadata about the object changed T19:02: Z T19:02: Z object_delete_time date The time at which the object was deleted T19:02: Z object_last_activity_ time date For live objects, equivalent to object_last_changed_time T19:02: Z For deleted objects, equivalent to object_delete_time. Refer to the account_last_activity_tim e attribute and the section "Live and deleted items" for details. object_etag_hash string The MD5 checksum of the object contents (or of the concatenated string of the segments' MD5 checksums, for a manifest object), as generated by the Object Storage code in response to the base API's Create/Update Object Request. The response to that request includes the ETag HTTP header containing this checksum value. d9f5eb4bba4e2f2f046e 54611bc8196b In the base API's List 6 November 2013 Page 9 of 33 Hewlett-Packard Company

10 Objects response, this checksum is shown as the "hash" attribute in the JSON or XML output. object_content_type string The MIME type of the object, as supplied by the client in the Content-Type header in the HTTP request. object_content_length numeric The size of the object in bytes. object_content_encodi ng object_content_dispos ition object_content_langua ge string string string If the object is a manifest, the size will be the sum of the sizes of all of the object's segments. The compression method, if supplied by the client in the Content- Encoding header in the HTTP request. Defines the browser's download behavior, if supplied by the client in the Content- Disposition header in the HTTP request. Defines the natural language(s) of the intended audience for the object, if supplied in the Content-Language header in the HTTP request. object_cache_control string Defines caching mechanisms to be obeyed by all caching mechanisms along the request / response chain, if supplied in the Cache-Control header in the HTTP request. object_delete_at date The time at which the object should be deleted. Converted from the UNIX epoch time in the X- Delete-At header in the HTTP request. text/plain; charset=utf gzip attachment; filename=platmap.tif mi, en max-age=2844, notransform T19:02: Z 6 November 2013 Page 10 of 33 Hewlett-Packard Company

11 Converted from the X- Delete-After header of the HTTP request, which defines the integer number of seconds from now when the object should be deleted. For either conversion, the resulting date is the same time format as the other times (ISO 8601:2004 format). If both X-Delete-At and X-Delete-After appear in the same HTTP request, the one that appears later in the request is the one that will be stored. object_manifest_type numeric If 0 (zero), this object is not a manifest. 2 If 1, this object is a dynamic large object (DLO). If 2, this object is a static large object (SLO). object_manifest string For DLOs uploaded in segments, the value is the X-Object-Manifest header supplied by the client in the HTTP request. The string is a container name followed by a slash character, then the common object names prefix for all object segments. For SLOs, the value is the entire JSON manifest generated by Swift based on the body of the uploaded SLO manifest object. For a DLO: container1/objectdir 1/subdir/photo1segme nts For an SLO: ["path": "/container1/object1 ", "etag": " d9f5eb4bba4e2f2f046e 54611bc8196b", "size_bytes": ] object_access_control string Cross-Object Resource See the w3.org page. 6 November 2013 Page 11 of 33 Hewlett-Packard Company

12 _allow_origin object_access_control _allow_credentials object_access_control _expose_headers object_access_control _max_age object_access_control _allow_methods object_access_control _allow_headers object_origin object_access_control _request_method object_access_control _request_headers Sharing (CORS) attributes, as described in cess-control/#accesscontrol-allow-originresponse-header object_meta_<name> string The value of the object custom metadata attribute <name>. Matches the "X-Object- Meta- <name>:<value>" header in HTTP requests. Example attribute name: object_meta_departme nt Example value: sales Superset metadata attributes available The OSMS API provides the following special attribute names that can be requested in the &attributes parameter list, to return a superset of certain system and/or custom attributes. These attributes cannot be used as query criteria or as sort parameters. The same rules about live and deleted files that apply to system attributes apply to these attributes as well. Superset attribute all_attrs all_system_attrs Description Returns all system and custom metadata attributes for the items matching the query. Equivalent to all_system_attrs,all_meta_attrs. Returns all system metadata attributes except *delete_time, for the items matching the query. It does not include any custom metadata attributes. Equivalent to all_account_system_attrs, all_container_system_attrs, all_object_system_attrs. The system attributes returned will be appropriate to the item. If the item is an account, only account system metadata will be returned. Containers and objects likewise return only the system metadata appropriate to those items. 6 November 2013 Page 12 of 33 Hewlett-Packard Company

13 all_meta_attrs Returns all custom (user-defined) metadata attributes of the form "[account container object]_meta_<name>" for the items matching the query. It does not include any system metadata attributes. all_account_attrs Returns all system and custom metadata attributes for the accounts matching the query (but not for containers nor objects). Equivalent to all_account_system_attrs,all_account_meta_at trs. all_account_system_attrs Returns all system metadata attributes except *delete_time, for the accounts matching the query. It does not include any custom metadata attributes. all_account_meta_attrs Returns all custom (user-defined) metadata attributes of the form "account_meta_<name>" for the accounts matching the query. It does not include any system metadata attributes. all_container_attrs Same as all_account_attrs but for containers. all_container_system_att Same as all_account_system_attrs but for containers. rs all_container_meta_attrs Same as all_account_meta_attrs but for containers. all_object_attrs Same as all_account_attrs but for objects. all_object_system_attrs Same as all_account_system_attrs but for objects. all_object_meta_attrs Same as all_account_meta_attrs but for objects. HTTP syntax The HTTP request format for obtaining the list of services for this implementation is the following: GET /services HTTP/1.1 The HTTP request for performing metadata searches has the following syntax, as one request but shown on multiple lines for readability: GET /<Object Storage API version> [/<account>[/<container>[/<object>]]]?<osms API version> [&attributes=<attr1>[,<attr2>][, ]] [&query=[(]<query expr1>[%20[and OR]%20<query expr2>][)][%20[and OR]%20 ]] [&sorted[=<attr1>[,<attr2>, ]]] [&limit=<max records>] [&all_results] [&marker='<marker string>'] [&end_marker='<end marker string>'] [&offset=<start record>] [&prefix='<prefix string>'] [&path='<object directory path>'] [&delimiter='<delimiter character>'] [&format=[json xml]] HTTP/1.1 6 November 2013 Page 13 of 33 Hewlett-Packard Company

14 The equivalent curl command formats for the two request types are the following, each as one command line: curl g "http[s]://<ip address hostname>[:<port>] /services curl g "http[s]://<ip address hostname>[:<port>] /<Object Storage API version> [/<account>[/<container>[/<object>]]]?<osms API version> [&attributes=<attr1>[,<attr2>][, ]] [&query=[(]<query expr1>[ [AND OR] <query expr2>][)][ [AND OR] ]] [&sorted[=<attr1>[,<attr2>, ]]] [&limit=<max records>] [&all_results] [&marker='<marker string>'] [&end_marker='<end marker string>'] [&offset=<start record>] [&prefix='<prefix string>'] [&path='<object directory path>'] [&delimiter='<delimiter character>'] [&format=[json xml]]" Syntax notes Only the HTTP GET verb is used for metadata searches. Optional parameters are shown in square brackets [ and ]. Everything enclosed in the brackets can be omitted from the request. Do not include the square brackets in the request. For example, the query parameter is optional and is not required to be in the HTTP request. Parameters are shown in angle brackets < and >. Replace the parameter with the actual value, without the angle brackets. Other characters shown in the syntax (such as =,?, &, and /) must also be entered as-is in the request, and sometimes must be URL-encoded. Quoting Numeric values for parameters must not be quoted. Date values for parameters must be enclosed in single quotes, e.g.: ' Z' Reserved strings, including API versions, attribute names, and format types, must not be quoted, e.g.: &attributes=objectlastmodifiedtime,objectcontenttype User-defined string values for parameters must be enclosed in single quotes, e.g.: prefix='employees/' Any single quotes that are part of a quoted string value must be escaped with a second single quote. For example: 'Dave''s book' In the curl syntax shown above, the g option and the double quotes around the entire URL ensure that the URL contents will be parsed correctly by curl. Any double quotes that are part 6 November 2013 Page 14 of 33 Hewlett-Packard Company

15 URL encoding of a quoted string value in a double quoted curl command must be escaped with a backslash: the \"right\" way HTTP request strings are URL-decoded by the API code. API clients must encode special characters, such as greater-than character (>), by replacing them with their hexadecimal equivalent values as shown by the examples in this section. The API s URL decoder interprets certain special characters properly without being URL encoded. Before the question mark character (?) in any HTTP request URL, the following characters are safe and do not need to be URL encoded: / : - _. # After the question mark character (?), the following characters are safe and do not need to be URL encoded: = & # All other characters must be URL encoded as their hexadecimal value as described in the ISO (ISO-Latin) standard. For example, the plus character (+) must be encoded as %2B, and the greater than character (>) must be encoded as %3E. Spaces can be encoded as either %20 or as the plus character (+), such as "my%20file.txt" or "my+file.txt" for the file "my file.txt". The plus character (+) is converted to a space when the URL is decoded by the API code. To include a plus character (+) in the URL, encode it as %2B, such as "A%2B" instead of "A+". Data types The type of each attribute will be one of: string numeric date A series of characters meeting the given <charset> in the Content-Type HTTP header. If <charset> or Content-Type is not present in the HTTP header, UTF-8 will be assumed. No null terminators or null characters will be present. An empty string is represented by no characters. Zero or a positive integer value less than 2^64 (18,446,744,073,709,551,616). In HTTP requests and responses, the value is represented as a string as described above. See the "Date formats" section below. Date formats All date/time values accepted by the API in HTTP requests, must be in ISO 8601:2004 format. See or for details on the format. 6 November 2013 Page 15 of 33 Hewlett-Packard Company

16 Examples of values accepted in HTTP requests: ' ' (9-Jun-2013, at time 00:00:00Z, where Z = Zulu a.k.a. GMT a.k.a. UTC time zone) ' T09:02:26Z' (9-Jun-2013 at 9:02:22am in Zulu/ GMT/UTC time zone) ' T090226Z' (The same date/time without separators) ' T02:02: ' (9-Jun-2013 at 2:02:22am in time zone = 7 hours behind UTC, which could be e.g. Pacific Daylight Time, or Mountain Standard Time. Equivalent to T09:02:22Z) Example value not accepted in HTTP requests: ' T09:02:26' (No timezone indicator, will return an error) 'Mon, 17 Oct :31:11 GMT' (RFC 5322 format, not ISO 8601 format, will return an error) ' ' (UNIX epoch format, not ISO 8601 format, will return an error) All date/time values returned in HTTP responses also will be in ISO 8601:2004 format, and will always be in the UTC time zone. The granularity of the metadata time values stored and returned in HTTP responses may vary depending on how the value is stored and obtained. The granularity of a given time value may be in nanoseconds, microseconds, or in seconds only. The fractional seconds shown in the response, however, will always be in nanoseconds. Example values returned include: " T19:02: Z" (seconds granularity) " T19:02: Z" (microseconds granularity) " T19:02: Z" (nanoseconds granularity) URIs and metadata search scope A URI uniquely defines an account, container, or object. In an HTTP request, the URI to the left of the '?' character defines the scope of the metadata search. The URI can specify any of these combinations to the left of the '?': 6 November 2013 Page 16 of 33 Hewlett-Packard Company

17 1. An account, container, and object, known as an object-level URI. Searches will operate only upon that object. 2. An account and container only, known as a container-level URI. Searches will be limited to the container and the objects within it. 3. An account only, known as an account-level URI. Searches will be limited to the account, and all containers and objects within it. 4. None of these three parameters. Searches will operate on all accounts, containers, and objects in the object store. The search behavior depends on the parameters specified in the request. See the table below for details on each of these parameters. HTTP request parameters Parameter IP_address hostname port services Object Storage API version account Description The dotted decimal IP address of the Object Storage proxy server that will handle this HTTP request and return an HTTP response, e.g The host name of the Object Storage proxy server that will handle this HTTP request and return an HTTP response, e.g. proxy1 or proxy1.mycompany.com. The listener IP port of the Object Storage proxy server. If the proxy is listening on the default HHTP port of 80, or the default HTTPS port 443, this parameter is unnecessary. Requests the list of services provided by the server's implementation of the base Object Storage API and the Metadata Search API. See the section "Services request" below this table for further information. Indicates the version of the OpenStack Object Storage API that this OSMS API is extending. This element is specified in the Object Storage API. As of this writing, the only version that exists is "v1". The name of the account to search. If present, the search scope will be limited to that account. If absent, all accounts will be searched. The following access rules apply. See the section "Authorized searchers" below this table for further information on the authorized searcher feature. If absent: If the searching user is a member of authorized searchers, then access will be provided to all metadata for all accounts and all of their containers and objects. If the searching user is not an authorized searcher, then access to each account's metadata will depend on the OpenStack authentication mechanism approving access to the account in the URI for the given X- Auth-Token. For each account that the user can access, she has access to the metadata of that account and all containers and objects in that account. For accounts inaccessible to the user, she can only access object metadata in those containers that provide.rlistings access in their 6 November 2013 Page 17 of 33 Hewlett-Packard Company

18 If present: X-Container-Read permission, and no access to any container or account metadata for those accounts. If the searching user is a member of authorized searchers, then access will be provided to all metadata for that account and all of its containers and objects. If the searching user is not an authorized searcher, then access to that account's metadata will depend on the OpenStack authentication mechanism approving access to the account in the URI for the given X- Auth-Token. If the user has access to the account, she has access to the metadata of the account and all containers and objects in that account. If the user does not have access to the account, she can only access object metadata in those containers that have.rlistings permission in their X-Container-Read ACLs, and no access to any container or account metadata. container The name of the container to search. If present, the search scope will be limited to that container. If absent, see the description of the <account> parameter. The following access rules apply: If the searching user is a member of authorized searchers, then access will be provided to all metadata for that container, its parent account, and all objects in that container. Account metadata does not include its container list. object If the searching user is not an authorized searcher, then metadata access will depend on the OpenStack authentication mechanism approving access to the <container> in the URI for the given X-Auth-Token. If the user has access to the container's account, he will be able to access the metadata for the <account>, for the <container>, and for all objects in that container. If the user does not have access to the account, but the container has.rlistings permission in the X-Container-Read ACL that includes the searching user, he will be able to access the metadata for all objects in that container, but not the container's nor the account's metadata. The name of the object to search. If present, the search scope will be limited to that container. If absent, see the description of the <container> parameter. The following access rules apply: If the searching user is a member of authorized searchers, then access will be provided to the metadata for the <account>, for the <container>, and for the <object>. Account metadata does not include its container list, and container 6 November 2013 Page 18 of 33 Hewlett-Packard Company

19 metadata does not include its object list. If the searching user is not an authorized searcher, then metadata access to this object will depend on the authentication mechanism approving access to the <container> in the URI for the given X-Auth-Token. If the user has access to the container's account, she will be able to access the metadata for the <account>, for the <container>, and for the <object>. If the user does not have access to the account but has.rlistings permission in the X-Container- Read ACL for the container, she will be able to access the metadata for the <object>, but not the container's nor the account's metadata. OSMS API version attributes= <attr1>[,<att r2>][, ] Indicates the version of the OSMS API. As of this writing, always "v1". Separating the OSMS API version from the OpenStack Object Storage API version allows the OSMS API to change independently of the Object Storage API that it extends. Retaining the version element of the Object Storage API maintains compatibility with that API's URI specification. If present, a comma-separated list of system, custom, and/or superset attribute names to be returned in the response for each item matching the query criteria. The URI of every item returned will always be shown in the item's attribute list, whether or not its attribute is listed, even if <attributes> is absent. The URI is the unique identification of every item returned. Besides defining which attributes to return for the items in the result set, <attributes> can also filter the result set: If any account_* attributes are listed, or any superset attributes that include account attributes are listed, then any account items in the result set will be returned as entries in the response. Otherwise, no account items will be returned. Similar filtering is done for container_* and object_* attributes. For example, if the URI /acc1/ctr2 is given, and only object <attr>s are listed, then only the entries for objects from the result set will be returned. If only account and container <attr>s are listed, then only two entries at most will be returned: one for acc1 (if it exists in the result set), and one for ctr2 (if it exists in the result set). If absent, then every item in the result set will be returned, but only the *_uri attribute will be shown. query expr The entries from the result set that will be returned in the response may be further limited by other request parameters such as <limit>. Defines the result set of items, even if it is absent. The result set begins with the scope defined by the URI in the request, before applying a query. See the section "URIs and metadata search scope". The items in the scope's pre-query result set include the URI item, its parent items, and all 6 November 2013 Page 19 of 33 Hewlett-Packard Company

20 items in lower-level URIs. For example, the pre-query result set for a containerlevel URI includes all objects in the container, the container itself, and its parent account. If any <query expr>s are present: The result set is further limited to the items meeting all <query expr> expressions. A <query expr> compares an attribute against a given value. The format of a query expression is: <query attr><operator><query value> See the next table rows below for details on these parameters. Query expressions can be combined using one or more AND and OR Boolean operators (case-insensitive), as well as parentheses to override default order of operations rules. There must be one space character (%20) before and after each AND or OR. By default, all AND expressions will be processed (left to right) before all OR expressions. Then the OR expressions will be processed (left to right). See the "Example queries" section for examples. Other Boolean operators such as NOT, NAND, and XOR are not supported. The NOT operator is not needed because each expression can be converted to its opposite to achieve the same effect. Other operators can be achieved using combinations of AND, OR, and parentheses. An implementation can choose to support a simpler subset of Boolean operator support. See the Services section for details. If <attributes> is absent, then the result set will be filtered further, based on the <query attr>s in the <query expr>s: If any account_* attributes are specified in any <query attr>, then any accounts shown in any URIs in the result set will be included as items in the filtered result set. Otherwise, no accounts will be included in the filtered result set. If any container_* attributes are specified in any <query attr>, then any containers shown in any URIs in the result set will be included as items in the filtered result set. Otherwise, no containers will be included in the filtered result set. If any object_* attributes are specified in any <query attr>, then all objects in the result set will be included as items in the filtered result set. Otherwise, no objects will be included in the filtered result set. If no <query expr>s are present, the result set is defined by the URI scope. For example, if the URI /acc1/ctr2 is given, then the result set includes an entry for 6 November 2013 Page 20 of 33 Hewlett-Packard Company

21 acc1, an entry for ctr2, and an entry for each object in ctr2. query attr (element of <query expr>) operator (element of <query expr>) The result set may be further filtered by other request parameters such as <attributes> or <limit>. A system and/or custom metadata attribute to be compared against the <query value> as a query criterion. The query operation to perform against the <query attr> and <query value>, one of: = (equals exactly)!= (does not equal) < (less than) <= (less than or equal to) > (greater than) >= (greater than or equal to) Only for custom attributes and string-valued system attributes (for example, object_container_name): ~ (regular expression match)!~ (does not match regular expression) The arguments to the regular expression operators (~ and!~) are POSIX regular expressions, as described in POSIX at section 9, Regular Expressions. query value (element of <query expr>) sorted[= <attr1>[,<att r2>, ]] The value to compare against the <query attr> using the <operator>. The value is either a numeric decimal value without quotes, or a string enclosed in single quotes. See the "Quoting" section above for details. If this parameter is present with no value, the list of items returned is sorted lexicographically by URI. Accounts, containers, and items will be arranged hierarchically, such as: /a1 /a1/c1 /a1/c1/o1 /a1/c1/o2 /a1/c2 /a2 etc.} The use of <sorted> on large result sets might negatively affect the performance of the query due to the sort processing required. If this parameter is present with one or more named system or custom metadata attributes, the list of items returned is sorted by that list of attributes, first by attr1, then by attr2, and so forth. String attributes are sorted lexicographically. Numeric attributes are sorted in increasing numeric value. Date attributes are sorted by increasing timestamp. If <sorted> is absent, items will be returned by item levels. First, all accounts 6 November 2013 Page 21 of 33 Hewlett-Packard Company

22 meeting the query criteria are returned. Next, all containers meeting the criteria are returned. Next, all objects meeting the criteria are returned. Each level s results are returned in an undefined order, but the order is repeatable if none of the items in the result set are added or deleted between two identical queries. limit= <max records> If this parameter is present, at most <max records> items will be returned in the response. The value is a numeric data type, see the "Data types" section. The result set is sorted first (or created in the default order), then the limit and other pagination attributes are applied. If absent, at most 10,000 items will be returned, by default. If other results are desired, use one or more of the parameters: <all_results>, <marker>, <end_marker>, <offset>. all_results If this parameter is present, then all results will be returned, overriding the default limit of 10,000 items. If <limit> is also present, then the <all_results> parameter is ignored. marker= '<marker string>' A pagination attribute. For all pagination attributes, note that a separate query will be issued for each HTTP request. If this parameter is present, then <marker string> is in the URI format: '/<account>[/<container>[/<object>]]' Items that appear after the given URI in the result set will be returned in the response (not including the URI). The behavior is independent of the <sorted> parameter. The result set may be sorted or not, but the order of the results is repeatable if none of the items in the result set are added or deleted between two identical queries. There is no state saved between queries. If objects are added or deleted during the time that the client is issuing queries to get each page, the results from the set of all queries may not represent the complete set at any particular point in time. If no objects are added nor deleted during this time, then the results will include all items that match the query during that time. If the marker does not match any URI in the result set, and <sorted> is not present, an error 404 Not Found is returned, because it's unclear where the marker would have appeared in the result set. end_marker= '<end marker string>' If <offset> is present, then <marker> is ignored. A pagination attribute. If this parameter is present, then only items that appear before the given <end marker string> in the result set will be returned in the response. If <offset> is present, then <end_marker> is ignored. 6 November 2013 Page 22 of 33 Hewlett-Packard Company

23 offset= <start record> See the <marker> parameter for further details. A pagination attribute. If this parameter is present, then results will be returned starting with <start record>. The first (<start record>-1) items from the result set are not returned. For example, if the <start record> is 100, then the first item returned will be the 100 th item, and the prior 99 items in the result set will not be returned. The result set may be sorted or not, but the order of the results is repeatable if none of the items in the result set are added or deleted between two queries of different offsets. prefix= '<prefix string>' An object name filter attribute. If present and <delimiter> is absent, then the query will return only the objects in the result set where the <object_name> begins with the given <prefix string>. If <delimiter> is absent, the same functionality can be accomplished by specifying: &query=object_name~'<prefix string>.*' (without the angle brackets). If <delimiter> is also present, see the <delimiter> description for details. If <path> is present, <prefix> and <delimiter> are ignored. delimiter= '<delimiter character>' An object name filter attribute. The <delimiter> parameter is used to limit the results of a hierarchy of <object_name>s to a certain level of nesting, as described in the base Object Storage API's two "Pseudo-Hierarchical Folders/Directories" sections. However, whereas the base API may return a partial object name that represents multiple objects, as a single result item, metadata search returns metadata for actual objects. Therefore, the output using <delimiter> differs from the List Objects output in the base API, but its use case is the same: All entries at the given hierarchy level will be returned. But if an entry contains objects at lower levels, metadata from one object from the lower levels will be returned, instead of just the partial object name returned by the base API. If present, and <prefix> is not present, then the query will return one object from each set of <object_name>s. The objects of a set begin with the same string up to the <delimiter character>. If no delimiter exists in the <object_name>, that object will be returned. If present, and <prefix> is also present, then the query will return one object fromeach set of <object_name>s. The objects of a set begin with <prefix string> and continue with the same string after the prefix, up to the <delimiter character>. Every object with no delimiter after the <prefix string> in the <object_name> will be returned. If <sorted> is present, then the object from each set will be the first object of that 6 November 2013 Page 23 of 33 Hewlett-Packard Company

24 set sorted lexicographically by URI. If <sorted> is not present, the object from each set is undetermined but repeatable if none of the items in the result set are added or deleted between two identical queries. If <path> is present, <prefix> and <delimiter> are ignored. The combination of <prefix> and <delimiter>, or the use of <path> without <prefix> or <delimiter>, allows queries of a specified depth of pseudohierarchical directories used within object names. For example, suppose the following unsorted list of objects exists in a container: employees/eve employees/hourly/ted employees/hourly/deb employees/former/ employees2/bob contractors/joe A query containing: &sorted&prefix='employees/'&delimiter='/' and no additional <query expr> criteria will return three objects sorted lexicographically: employees/eve employees/former/ employees/hourly/deb (Note that a <prefix> of employees without the trailing '/' would also return employees2/bob.) In the base API, the same number of items would be returned, but the item names would be: employees/eve employees/former/ employees/hourly/ Metadata search cannot return employees/hourly/ because it does not represent an actual object. Instead, it returns the metadata of one object that begins with that prefix, in this case employees/hourly/deb. The same effect can be achieved using <path> instead of <prefix> and <delimiter>: &sorted&path='employees' Note the absence of the trailing '/' in the <path>. This equivalent usage is only possible because the <prefix> ends in a '/' and the <delimiter> is '/'. If <sorted> is not present in the above example, then the query will return objects in an undefined order, and the query will include either 6 November 2013 Page 24 of 33 Hewlett-Packard Company

25 'employees/hourly/ted' or '/employees/hourly/deb', since the internal order is undefined. Because the order is repeatable, however, either Ted or Deb will always be returned if none of the items in the result set are added or deleted between two identical queries. path= '<object directory path>' format= [json xml] The <prefix> description describes which objects in which containers and accounts will be queried. An object name filter attribute. If present, a '/' will be appended to the given <object directory path> parameter value string, unless it already contains a trailing '/'. Then the query will be processed exactly as if the <prefix> parameter contained the same <object directory path> with a '/' character at the end, and a <delimiter> of '/' was present. If present, <prefix> and <delimiter> are ignored. If present, must be one of the string literals json or xml. Results are returned in JSON or XML format. If absent, plain text is returned. Authorized searchers The concept of authorized searchers provides a way to maximize query speed by bypassing the need to check the searching user's access to each account, for a set of users defined by the storage administrator. These users must authenticate via the OpenStack authentication mechanism to be able to perform any query, just like all users. Once authenticated, however, the authorized searcher can access any metadata for all accounts, containers, and objects via the API. For users that are not listed as authorized searchers, the engine will check the access to each account and container to be queried, and will only return results for accounts and containers to which the user has access. This access checking can significantly impact the performance of the query. A storage administrator defines this list of users that are authorized searchers using a management console or using CLI commands. The administrator may distribute different user credentials to different users, for example to allow each user to manage their own password, or for audit tracking. The authorized searchers feature is optional. If not used, access to metadata will be limited for all users, according to the user's access to accounts and containers. Output formats The default output format for the query results is plain text, if no "format" parameter is supplied. Plain Text format The plain text output format is human readable, with indent levels of 4 spaces per indent. The top level has no indent, and always represents the URI of an item matching the query criteria. The requested attributes for that item are listed under the URI, indented. If <sorted> or <sorted=uri> is specified, then the results will be in lexicographic order by URI. Example: 6 November 2013 Page 25 of 33 Hewlett-Packard Company

26 /account1 account_container_count:15 /account1/container1 container_last_modified_time: t13:17: z /account1/container1/objectdir1/subdir1/photo.jpg object_last_changed_time: t00:53: z object_content_length: /account1/container2 container_last_modified_time: t13:17: z /account1/container2/anotherobject object_last_changed_time: t00:53: z object_content_length: If <sorted> or <sorted=uri> is not specified, then the results will not be sorted, or will be sorted by the attribute defined in <sorted=attr>. Example of unsorted output: /account1 account_container_count:15 /account1/container1 container_last_modified_time: t13:17: z /account1/container2 container_last_modified_time: t13:17: z /account1/container1/objectdir1/subdir1/photo.jpg object_last_changed_time: t00:53: z object_content_length: /account1/container2/anotherobject object_last_changed_time: t00:53: z object_content_length: Each line of output is terminated by a single UNIX-style end-of-line character, UTF-8 value 10 (0x0a). JSON format The JSON output format for query responses conforms to standard and well-formed JSON. The first level always represents the URI of an item matching the query criteria. The requested attributes for that item are listed as the second level after the URI. See the description of sorted vs. unsorted output in the "Plain text" section above. Example for sorted output: [ "/account1" : "account_container_count" : "15" } "/account1/container1" : 6 November 2013 Page 26 of 33 Hewlett-Packard Company

27 "container_last_modified_time" : " T13:17: Z" } "/account1/container1/objectdir1/subdir1/photo.jpg" : "object_last_changed_time" : " T00:53: Z", "object_content_length" : } "/account1/container2" : "container_last_modified_time" : " T13:17: Z" } "/account1/container2/anotherobject" : "object_last_changed_time" : " T00:53: Z", "object_content_length" : } } ] XML format The XML output format for query responses conforms to standard and well-formed XML. The output is flat, not hierarchical. Objects are not nested in containers, and containers are not nested in accounts. Each item is provided separately, which allows for unsorted and sorted outputs based on arbitrary sorting criteria. See the description of sorted vs. unsorted output in the "Plain text" section above. Example for sorted output: <?xml version="1.0" encoding="utf-8"?> <account uri="/account1"> <account_container_count>15</account_container_count> </account> <container uri="/account1/container1"> <container_last_modified_time> T13:17: Z</container_last_modified_time> </container> <object uri="/account1/container1/objectdir1/subdir1/photo.jpg"> <object_last_changed_time> T00:53: Z</object_last_changed_time> 6 November 2013 Page 27 of 33 Hewlett-Packard Company

28 <object_content_length>194532</object_content_length> </object> <container uri="/account1/container2"> <container_last_modified_time> T13:17: Z</container_last_modified_time> </container> <object uri="/account1/container2/anotherobject"> <object_last_changed_time> T00:53: Z</object_last_changed_time> <object_content_length>194532</object_content_length> </object> Example searches Get all metadata for all accounts, containers, and objects curl -g " The search returns an entry for every account, every container, and every object. Each entry contains all system attributes, and any custom attributes. The output format is plain text (see the section "Plain Text format"). Note that unless the user issuing the search is an authorized searcher, the search must authenticate the user against each account, and each container in non-authorized accounts. The results will contain metadata only from authorized accounts, and containers from non-authorized accounts that provide read permission to the user in the container read ACL. Thus, searches by non-authorized searchers across multiple accounts may be significantly slower than searches by authorized searchers. Get selected metadata for an object, its container, and its account curl -g " _count,account_object_count,account_meta_billing_method,all_container_ system_attrs,all_object_meta_attrs&format=json" The search returns first an entry for account acc1 with the three named attributes (two system attributes account_container_count and account_object_count, and one custom attribute account_meta_billing_method). The next entry is for the container ctr2, returning all system attributes (all_container_system_attrs) but no custom attributes. If the object obj3 has any custom metadata attributes, the next entry is for obj3 with its custom metadata but no system attributes (all_object_meta_attrs). If obj3 has no custom attributes, then no results are returned for obj3. The output format is JSON (see the section "JSON format"). Get metadata for an account, and objects meeting a set of criteria curl -g " 6 November 2013 Page 28 of 33 Hewlett-Packard Company

29 _last_changed_time&query=container_create_time> AND object_manifest_type=1 and object_manifest~'segctr1/.*'" The search returns first the object count for the acc1 account. Next, it returns the last changed time for all objects in the account where the object's container was created after 1-Aug-2013 at 00:00:00 UTC and the object is a manifest for a DLO object that has segments in the container segctr1. This example also shows how the result set is limited by the attributes list. Although a container attribute is present in the query, no container entries exist in the final result set because the attribute list contains no container attributes. But because the URI of each item is always returned, the account and container for each object will be shown. Note that the 'AND' operator is case-insensitive. Identify accounts and containers meeting a set of criteria curl -g " AND container_meta_customer_name~'.*'" The search returns only the URIs for all accounts and containers (but not objects), where the account's object count is less than 100, and containers have the custom attribute "customer_name" regardless of its value. This example also shows how the result set is limited by the item types in the query expressions, in the absence of a list of <attributes> to be returned. Since no object attributes exist in the query, no object items are returned. Get metadata for all results, sorted by URI curl -g " t,object_content_type&sorted&all_results" The search first returns the account container count for acc1. Next, it returns the content type of each object in ctr2, sorted by the object URI. The all_results parameter overrides the default limit of 10,000 items in the response. All objects in ctr2 will be returned, regardless of the number of objects in it. The client accepting the HTTP response must be able to receive an arbitrarily large amount of data in this case. Get the 3rd page of 100 results, sorted by multiple attributes curl -g " ct_content_length> &sorted=container_object_count,object_uri_cre ate_time&limit=100&offset=201" The search returns results 201 through 300 of the sorted result set defined by the scope and query. The results in total are sorted first by container object count, then by the object URI's first creation time. 6 November 2013 Page 29 of 33 Hewlett-Packard Company

30 Note that because the superset attribute all_system_attrs in the attributes list applies to all item levels, an entry for acc1 will be returned, as well as entries for all containers and objects satisfying the query. For example, suppose two containers exist, with 300 and 200 objects respectively. First, the system attributes for the acc1 account is returned. Next, the second container is returned, followed by its objects that are greater than 1,000,000 bytes (of the 200 that exist), in order of the object URIs' creation times. Next, the first container and all of its objects satisfying the query (of the 300 that exist) are returned, sorted the same way. Get a portion of the sorted results, between the given URIs curl -g " ker='acc1/ctr13/obj_x'&end_marker='acc1/ctr17/obj_y'" The search returns the account, containers, and objects in the acc1 account, between the URIs 'acc1/ctr13/obj_x' and 'acc1/ctr17/obj_y' (non-inclusive) in the sorted (by URI) list of results. Services request The <services> parameter requests the API versions and list of services provided by the server's implementation of the given Object Storage API version and Metadata Search API version. Following are the elements of the response. This request was created with the intention of generalizing the OSMS API to become a standard OpenStack metadata search API. That API would have a reference implementation provided by OpenStack, but the API could also be implemented by other vendors supplying a search provider with different levels of API support. A search provider may not choose to implement all of the API, and/or may add elements to the API. This request defines the elements provided with the implementation. It allows clients to access elements of the API appropriate to different search providers on different OpenStack instances. Element min_base_api_version max_base_api_version search_provider search_enabled Description The minimum version of the Object Storage API accepted by the search provider's implementation, e.g. "v1". The maximum version of the Object Storage API accepted by the search provider's implementation, e.g. "v2". As of this writing, only the "v1" version of the API exists. The string identifier of the entity providing the metadata search service on the server, e.g. "HP". If "true" (case-sensitive), then the metadata search feature is supported by this implementation, and the following fields are meaningful. If not "true", the string should be interpreted as false. In this case, no metadata search features are available at all, and any search request that is not understood by the base API will return an HTTP error response 405, "Method Not Allowed". All fields after this one in 6 November 2013 Page 30 of 33 Hewlett-Packard Company

31 the Services request are meaningless and should not be used. min_search_api_version The minimum version of the metadata search API for the search provider's implementation, e.g. "v1". Note that e.g. "v2" for one provider may list a different set of service elements than "v2" from another provider. max_search_api_version The maximum version of the metadata search API for the search provider's implementation, e.g. "v2". freshness_complete Note that e.g. "v2" for one provider may list a different set of service elements than "v2" from another provider. If "true" (case-sensitive), then each HTTP response to a metadata search request will contain the following HTTP header in the response, in ISO 8601:2004 format. For example: X-Freshness-Complete-Date: T18:56: Z The X-Freshness-Complete-Date indicates the time at which all operations on the object store have been stored in the metadata search database and are queryable. If not "true", the string should be interpreted as false. freshness_partial If "true" (case-sensitive), then each HTTP response to a metadata search request will contain the following HTTP header in the response, in ISO 8601:2004 format. For example: X-Freshness-Partial-Date: T19:02: Z The X-Freshness-Partial-Date indicates the latest time when any operations have been stored. Operations between the two timestamps may or may not be stored in the database and returned in query results. No operations after the partial date timestamp have been stored and will not be returned in query results. If not "true", the string should be interpreted as false. complex_boolean_expr attr_list If "true" (case-sensitive), then ANDs, ORs, and parentheses (nested expressions) are supported as query operators by this implementation. This is the full PI support for Boolean expressions defined by the API. If not "true", the string should be interpreted as false. In this case, only ANDs are supported, which makes all queries simpler to implement. Multiple ANDs in one expression must be supported. The list of system, and superset attributes that the server can return in an attribute list and can query on. If an attribute is not listed, then 6 November 2013 Page 31 of 33 Hewlett-Packard Company

32 the implementation does not support it and will return and error if referenced in an HTTP request. If a listed attribute is not one of the set defined on this API specification, further details should be provided in the search provider's documentation. Each attribute contains the following sub-elements. attr_name data_type sortable The string name of the attribute. The type of the attribute. See the section "Data types" for details. If "true" (case-sensitive), the attribute can be requested as the <sort> attribute value. If not "true", the string should be interpreted as false. Responses follow the following JSON format example. No other formats such as XML or plain text are supported. [ "min_base_api_version" : "v1" "max_base_api_version" : "v1" "search_provider" : "HP" "search enabled" : "true" "min_search_api_version" : "v2.1" "max_search_api_version" : "v2.1" "freshness_complete" : "true" "freshness_partial" : "" "complex_boolean_expr" : "false" "attributes" : 6 November 2013 Page 32 of 33 Hewlett-Packard Company

33 "attr_name" : "account_uri" "data_type" : "string" "sortable" : "true" } more attributes "attr_name" : "size" "data_type" : "numeric" "sortable" : "false" } } ] 6 November 2013 Page 33 of 33 Hewlett-Packard Company