Advanced Information Management

Anwendersoftware a Advanced Information Management Chapter 7: XML and Databases Holger Schwarz Universität Stuttgart Sommersemester 2009

Overview Introduction SQL/XML data type XML XML functions mappings Example: DB2 native XML DBMS 2

XML and Databases X Path XQ Query Query Optimization and Execution describe schema: XML schema query languages: XQuery storage technologies indexing technologies DBMS Operators Files and Access Methods Buffer Management Disk Space Management DB native XML-DBMS hybrid systems: relational + xml SQL/XML store XML data as columns of type XML Build XML from SQL data retrieve SQL data from XML content 3

Overview Introduction SQL/XML data type XML XML functions mappings Example: DB2 native XML DBMS 4

Data Type XML Predefined SQL data type Primary type modifiers DOCUMENT: XML document CONTENT: XQuery document node SEQUENCE: XQuery sequence Secondary type modifiers UNTYPED: no type information provided XMLSCHEMA: schema provided ANY: XMLDataType ::= XML ( "(" ( DOCUMENT CONTENT SEQUENCE ) ( "(" ( ANY UNTYPED XMLSCHEMA Schema ) ")" )? ")" )? Schema ::= ( URI NamespaceURI ( LOCATION SchemaLocationURI )? NO NAMESPACE ( LOCATION SchemaLocationURI )? ID RegisteredSchema ) ( NAMESPACE NamespaceURI ELEMENT ElementName )? 5

Data Type XML XML(SEQUENCE) every XML value, i.e., null value or XQuery sequence untyped elements and attributes XML(CONTENT(ANY)) null value or XQuery document child nodes are valid according to schema XML(CONTENT(UNTYPED)) XML(CONTENT(XMLSCHEMA)) XML(DOCUMENT(UNTYPED)) XML(DOCUMENT(ANY)) well-formed well-formed i.e., a single element node as well-formed root node XML(DOCUMENT(XMLSCHEMA)) marks additional restrictions child nodes are valid according to schema 6

SQL/XML and XML Schema A registered XML Schema is an XML Schema that has been made known to the SQL-server. The means by which an XML Schema is registered is implementation- defined. Pre-defined schemas: xs: http://www.w3.org/2001/xmlschema xsi: http://www.w3.org/2001/xmlschemainstance sqlxml: http://standards.iso.org/iso/9075/2003/sqlxml 7

Overview Introduction SQL/XML data type XML XML functions mappings Example: DB2 native XML DBMS 8

Functions for XML Data XMLELEMENT, XMLFOREST, XMLCOMMENT, XMLTEXT, XMLPI, XMLPARSE, XMLAGG, XMLQUERY SQL data XML data XMLDOCUMENT, XMLCONCAT, XMLVALIDATE, XMLQUERY XMLTABLE, XMLITERATE, XMLSERIALIZE 9

Expressions Some expressions used in subsequent definitions: XMLValueExpression ::= XMLValueFunction XMLValueFunction ::= XMLComment XMLConcatenation XMLDocument XMLElement XMLForest XMLParse XMLPI XMLQuery XMLQuery XMLText XMLValidate StringValueExpression ::= <<<Expression that evaluates to a String>>> CharacterValueExpression ::= <<< Expression that evaluates to characters >>> ValueExpressions ::= <<< Expression that evaluates to a general value, e.g., derived from a column value >>> 10

XMLELEMENT Construct XML elements from SQL data Students ID Name Term 101 Smith 5 202 Berger 7 SELECT XMLELEMENT(NAME "Student", XMLATTRIBUTES(Term), Name) AS Person FROM Students Person <Student Term="5">Smith</Student> <Student Term="7">Berger</Student> XMLElement ::= XMLELEMENT "(" NAME ElementName ( "," XMLATTRIBUTES "(" AttributeList ")" )? ( ",", XMLElementContent)* ( OPTION XMLContentOption )? ( XMLReturningClause )? ")" XMLAttributeList ::= XMLAttribute ( "," XMLAttribute )? XMLAttribute ::= XMLAttributeValue ( AS XMLAttributeName )? XMLElementContent ::= ValueExpression ( "," ValueExpression ])? XMLContentOption ::= NULL ON NULL EMPTY ON NULL ABSENT ON NULL NIL ON NULL NIL ON NO CONTENT XMLReturningClause ::= RETURNING ( CONTENT SEQUENCE ) 11

XMLFOREST Construct sequence of XML elements from SQL data Students ID Name Term 101 Smith 5 202 Berger 7 SELECT XMLFOREST(Name, XMLELEMENT(NAME "Term", Term) AS Study) AS Person FROM Students Person <Name>Smith</Name> <Study><Term>5</Term></Study> <Name>Berger</Name> <Study><Term>7</Term></Study> XMLForest ::= XMLFOREST "(" ( XMLNamespaceDeclaration "," )? ForestElement t ( ","" ForestElement t )* ( OPTION XMLContentOption )? ( XMLReturningClause )? ")" ForestElement ::= ForestElementValue ( AS ForestElementName )? 12

XMLAGG Aggregates g XML Elements of a group Students ID Name Grade 101 Smith A 202 Berger A 202 Berger C SELECT Name, XMLAGG( XMLELEMENT(NAME "Grade", Grade)) AS Results t GROUP BY Name Name Smith Berger Results <Grade>A</Grade> <Grade>A</Grade> <Grade>C</Grade> XMLAggregate ::= XMLAGG "("( XMLValueExpression ( ORDER BY SortSpecificationList )? ( XMLReturningClause )? ")" 13

Other Functions XMLCOMMENT: Provides an XML comment node XMLPI: Provides a processing instruction node XMLTEXT: Provides a text node XMLPARSE: parses the argument as an XML document or an XML element and returns an XML value XMLComment XMLPI XMLText ::= XMLCOMMENT "(" CharacterValueExpression ( XMLReturningClause )? ")" ::= XMLPI "(" NAME Identifier ( "," CharacterValueExpression ( XMLReturningClause )? ")" ::= XMLTEXT "(" CharacterValueExpression ( XMLReturningClause )? ")" XMLParse ::= XMLPARSE "("( ( DOCUMENT CONTENT ) StringValueExpression ( STRIP RESERVE ) WHITESPACE ")" 14

XMLDOCUMENT Returns an XML value with a single document node with zero or more children nodes. XMLDocument ::= XMLDOCUMENT "(" XMLValueExpression ( XMLReturningClause )? ")" 15

XMLCONCAT Concatenate XML elements to a sequence Students ID Name Term 101 Smith 5 202 Berger 7 SELECT XMLCONCAT( XMLELEMENT(NAME "Name", Name), XMLELEMENT(NAME "Term", " Term)) AS Person FROM Students Person <Name>Smith</Name> <Term>5</Term> <Name>Berger</Name> <Term>7</Term> XMLConcatenation ::= XMLCONCAT "(" XMLValueExpression ( "," XMLValueExpression )* ( XMLReturningClause )? ")" 16

XMLVALIDATE Returns a copy of the input XML value augmented with information obtained from XML schema validation, including default values and type annotations. XMLValidate ::= XMLVALIDATE "(" ( DOCUMENT CONTENT SEQUENCE ) XMLValueExpression ( ACCORDING TO XMLSCHEMA ( URI NamespaceURI ( LOCATION SchemaLocationURI )? NO NAMESPACE ( LOCATION SchemaLocationURI )? ID RegisteredSchema ) ( ELEMENT ElementName NAMESPACE NamespaceURI ELEMENT ElementName NO NAMESPACE ELEMENT ElementName )? ")" 17

XMLQUERY Returns an XML value from the evaluation of an XQuery expression possibly using specified input arguments as XQuery variables. Allows to define XML views on XML data XMLQuery ::= XMLQUERY "(" XQueryExpressionStringConstant ( PASSING ( BY REF BY VALUE ) ValueExpression AS Identifier ( BY REF BY VALUE )? ("," ValueExpression AS Identifier ( BY REF BY VALUE )? )? )? ( XMLReturningClause )? ( NULL ON EMPTY EMPTY ON EMPTY ) ")" 18

XMLQUERY Example: Students ID Name Term Age 101 Smith 5 22 202 Berger 7 21 SELECT XMLQUERY('<data> <Person age="{$age}"> <Name>{$name}</Name> <Term>{$term}</Term> </Person> </data>' PASSING Name AS $name, Term AS $term, Age AS $age EMPTY ON EMPTY) AS Person FROM Students Person <data> <Person age="22"> <Name>Smith</Name> <Term>5</Term> </Person> </data> <data> <Person age="21"> <Name>Berger</Name> <Term>7</Term> </Person> </data> 19

XMLTABLE Returns a result table from the evaluation of XQuery expressions, possibly using specified input arguments as XQuery variables. Each sequence item in the result sequence of the row XQuery expression represents a row of the result table. Allows to define SQL views on XML data XMLTable ::= XMLTABLE "(" XQueryExpressionStringConstant ( PASSING ( BY REF BY VALUE ) ValueExpression AS Identifier ( BY REF BY VALUE )? ("," ValueExpression AS Identifier ( BY REF BY VALUE )? )? )? COLUMNS ( ColumnName FOR ORDINALITY ColumnName DataType ( BY REF BY VALUE )? ( DefaultClause )? PATH PathStringConstant )+ ")" 20

XMLTABLE Example: Students ID Person 101 <data> <Person age="22"> <Name>Smith</Name> <Term>5</Term> </Person> </data> 202 <data> <Person age="21"> <Name>Berger</Name> <Term>7</Term> </Person> </data> SELECT s.id, t.* FROM Students s, XMLTABLE('for $x in $person_data where $x/person/@age >20 return $x/person' PASSING s.person AS "person_data" COLUMNS Name VARCHAR(50) PATH 'Person/Name' Term INT PATH 'Person/Term' ) AS t ID Name Term 101 Smith 5 202 Berger 7 XMLITERATE takes an XML sequence and returns it as the column of a table, i.e., each sequence element gets a row 21

XMLSERIALIZE Returns a serialized XML value of the specified data type generated from the expression argument. Possible data types: e.g. VARCHAR, CHAR, CLOB, BLOB, Serialized value may include the XML declaration of a certain version XMLSerialize ::= XMLSERIALIZE "(" ( DOCUMENT CONTENT )? XMLValueExpression AS DataType ( ENCODING EncodingSpecification )? ( VERSION StringLiteral )? ( INCLUDING XMLDECLARATION EXCLUDING XMLDECLARATION )? ")") 22

Overview Introduction SQL/XML data type XML XML functions mappings Example: DB2 native XML DBMS 23

Mapping between SQL and XML The mappings from SQL to XML include: Mapping SQL character sets to Unicode. Mapping SQL identifiers to XML Names. Mapping SQL data types (as used in SQL-schemas to define SQL-schema objects such as columns) to XML Schema data types. Mapping values of SQL data types to values of XML Schema data types. Mapping an SQL table to an XML document and an XML Schema document. Mapping an SQL schema to an XML document and an XML Schema document. Mapping an SQL catalog to an XML document and an XML Schema document. The mappings from XML to SQL include: Mapping Unicode to SQL character sets. Mapping XML Names to SQL identifiers. 24

Mapping Identifiers Since not every SQL identifier is an acceptable XML Name, it is necessary to define a mapping of SQL identifiers to XML Names and vice veras. The basic idea of this mapping is that characters that are not valid in XML Names are converted to a sequence of hexadecimal digits derived from the Unicode encoding of the character. partially escaped: escape all relevant characters but not colons and 'xml' at the beginning i of the identifier fully escaped: escape colons and 'xml' at the beginning of the identifier as well Examples: Income:FY2009 -> Income_x003A_FY2009 Work@home -> Work_x0040_home 25

Mapping Data Types SQL/XML namespace defines: Types of SQL data types (basic, distinct type array type, ) SQL basic data types SQL schema objects (schemas, tables, views, ) Sample Mappings: CHAR (10) CHARACTER SET LATIN1 COLLATION DEUTSCH <xsd:simpletype> <xsd:restriction base="xsd:string"> <xsd:length value="10"/> <xsd:annotation> <sqlxml:sqltype q name="char" length="10" charactersetname="latin1" collation="deutsch"/> </xsd:annotation> </xsd:restriction> </xsd:simpletype> CHAR (10) CHARACTER SET LATIN1 COLLATION DEUTSCH <xsd:simpletype> <xsd:restriction base="xsd:string"> <xsd:length value="10"/> </xsd:restriction> </xsd:simpletype> INTEGER <xsd:simpletype> <xsd:restriction base="xsd:integer"> <xsd:maxinclusive value="2157483647"/> <xsd:mininclusive value="-2157483648"/> <xsd:annotation> <sqlxml:sqltype name="integer"/> </xsd:annotation> </xsd:restriction> </xsd:simpletype> 26

SQL/XML Namespace 27

Mappings: Overview SQL Type XML Schema Type Restriction BOOLEAN xsd:boolean CHAR xsd:string xsd:length VARCHAR, CLOB xsd:string xsd:maxlength BLOB xsd:hexbinary xsd:maxlength SMALLINT, INT xsd:integer xsd:mininclusive xsd:maxinclusive NUMERIC, DECIMAL xsd:decimal xsd:totaldigits xsd:fractiondigits REAL, FLOAT xsd:float dfl DOUBLE PRECISION xsd:double DATE xsd:date xsd:pattern TIME xsd:time xsd:pattern TIMESTAMP xsd:datetime xsd:pattern 28

Mapping SQL Objects Options for mapping SQL tables to XML: map the table to a sequence of XML elements where the name of each toplevel element is derived from the table name and represents a row in the table, or map the table to an XML document with a single root element whose name is derived from the table name and to map each row to an element named <row>. Options for NULL values: missing elements elements marked as xsi:nil="true" " SQL/XML provides mapping to XML document XML schema document both Target namespace may be specified 29

Overview Introduction SQL/XML data type XML XML functions mappings Example: DB2 native XML DBMS 30

XML Support in DB2 Access XML data XQuery SQL and SQL/XML functions XML schema repository XML support for data import and export several programming languages functions and procedures Performance XML indexes query optimizer treats SQL, SQL/XML and XQuery explain 31

Processing XML Data in DB2 Insert Update Delete Insert Update Delete SQL SQL/XML XQuery XML dat ta Import Load tablename INT CHAR XML 101 Smith 202 Berger Export I1 I2 Create XML columns Create XML indexes 32

Storing XML Data Create table with XML column CREATE TABLE Customer (Cid BIGINT NOT NULL PRIMARY KEY, Info XML) No type modifiers (as defined by SQL/XML) supported 33

Using SQL to Query XML Data Read complete document SELECT Cid, Info FROM Customer Select parts of an XML document SELECT XMLQUERY ( 'declare default element namespace "http://posample.org"; for $d in $doc/customerinfo return <out>{$d/name}</out>' passing INFO as "doc") FROM Customer as c WHERE XMLEXISTS (declare ('declare default element namespace "http://posample.org org"; $i/customerinfo/addr[city="toronto"]' passing c.info as "i") XMLEXISTS (SQL/XML) is used to identify the row to be updated tests whether an XQuery expression returns a sequence of one or more items 34

Using XQuery to Query XML Data Read complete document using an XQuery function XQUERY db2-fn:xmlcolumn ('CUSTOMER.INFO') keyword to designate XQuery expressions Read complete document using a fullselect XQUERY db2-fn:sqlquery ('SELECT Info FROM Customer') XQuery function xmlcolumn access XML data in a table 'schema.table.column' schema is optional XQuery function sqlquery access XML data in a table query has to result in a single XML column result is provided d as XQuery sequence 35

Using XQuery to Query XML Data Function xmlcolumn to access data Restrictions by XQuery XQUERY declare default element namespace "http://posample.org"; for $d in db2-fn:xmlcolumn('customer.info INFO')/customerinfo where $d/addr/city="toronto" return <out>{$d/name}</out> Function sqlquery to access data Restrictions by fullselect and XQuery XQUERY declare default element namespace "http://posample.org org"; for $d in db2-fn:sqlquery( 'SELECT INFO FROM CUSTOMER WHERE Cid < 2000')/customerinfo where $d/addr/city="toronto" return <out>{$d/name}</out> 36

Using XQuery to Query XML Data Function sqlquery qq to access data Restrictions by full-select and XQuery possible VALUES XMLQUERY ( 'declare default element namespace "http://posample.org"; for $d in db2-fn:sqlquery( ''SELECT INFO FROM CUSTOMER WHERE Cid = parameter(1)'', $testval)/customerinfo return <out>{$d/name}</out>' passing 1000 as "testval" ) PASSING clause is used to pass value 1000 to the XQuery expression evaluated by XMLQUERY Function sqlquery may have several parameters Fullselect refers to these parameters by parameter(1), parameter(2), 37

Indexing XML Data Create indexes on XML columns CREATE UNIQUE INDEX cust_cid_xmlidx ON Customer(Info) GENERATE KEY USING XMLPATTERN 'declare default element namespace "http://posample.org"; /customerinfo/@cid' AS SQL DOUBLE No combined indexes (several XML columns) possible Several indexes for one column possible XML pattern defines the element or attribute to be indexed AS clause defines datatype that is used to store keys 38

Indexing XML Data XMLIndexSpecification ::= GENERATE KEY USING XMLPATTERN XMLPatternClause XMLTypeClause XMLPatternClause ::= ( NamespaceDeclaration )? PatternExpression NamespaceDeclaration::= ( DECLARE NAMESPACE NamespacePrefix "=" NamespaceURI ";" DECLARE DEFAULT ELEMENT NAMESPACE NamespaceURI ";" )+ PatternExpression ::= ( ( "/" "//" ) ForwardAxis ( XMLNameTest XMLKindTest ) ForwardAxis ::= ( child:: attribute:: @ descendant:: self:: descendant-or-self:: ) XMLKindTest ::= ( node() text() comment() processing instruction() ) XMLTypeClause ::= AS DataType (IGNORE INVALID VALUES REJECT INVALID VALUES )? DataType ::= SQL ( VARCHAR DOUBLE DATE TIMESTAMP ) 39

Insert XML Data Options: Use SQL/XML functions to provide XML data, e.g., XMLPARSE Provide XML data as String in INSERT, UPDATE and DELETE statements INSERT INTO Customer (Cid, Info) VALUES (1000, ' <customerinfo xmlns="http://posample.org" Cid="1000"> <name>kathy Smith</name> <addr country="canada"> <street>5 Rosewood</street> <city>toronto</city> <prov-state>ontario</prov-state> <pcode-zip>m6w 1E6</pcode-zip> </addr> <phone type="work">416-555-1358</phone> </customerinfo>') 40

Update XML Data Replace entire XML value UPDATE customer SET info = ' <customerinfo xmlns="http://posample.org" Cid="1002"> <name>jim Noodle</name> <addr country="canada"> <street>1150 Maple Drive</street> <city>newtown</city> <prov-state>ontario</prov-state> <pcode-zip>z9z 2P2</pcode-zip> </addr> <phone type="work">905-555-7258</phone> </customerinfo>' WHERE XMLEXISTS ( 'declare default element namespace "http://posample.org org"; $doc/customerinfo[@cid = 1002]' PASSING info AS "doc") 41

Update XML Data Modify XML value using an XQuery update expression UPDATE customer SET info = XMLQUERY( 'declare default element namespace "http://posample.org"; transform copy $mycust := $cust modify do replace $mycust/customerinfo/addr with <addr country="canada"> <street>25 EastCreek</street> <city>markham</city> <prov-state>ontario</prov-state> <pcode-zip>n9c 3T6</pcode-zip> </addr> return $mycust' passing INFO as "cust") WHERE CID = 1002 $cust refers to the XML values in column info COPY: provide copy of the XML value in $mycust MODIFY: replace element node RETURN: provide new content of $mycust as result 42

Update XML Data Using multiple MODIFY clauses xquery declare default element namespace "http://posample.org"; let $email := <email>jnoodle@my-email.com</email> let $status := <status>current</status> return transform copy $mycust := db2-fn:sqlquery('select info from customer where cid = 1002') modify ( do replace $mycust/customerinfo/phone with $email, do insert $status after $mycust/customerinfo/phone[@type = "work"] ) return $mycust 43

Delete Data Use XML data to identify rows to be deleted DELETE FROM Customer WHERE XMLEXISTS ( 'declare default element namespace "http://posample.org"; $doc/customerinfo[@cid = 1003]' passing INFO as "doc") 44

Delete Data Delete XML data partially UPDATE Customer SET info = XMLQUERY( 'declare default element namespace "http://posample.org"; transform copy $newinfo := $info modify do delete ($newinfo/customerinfo/phone) return $newinfo' passing info as "info") WHERE cid = 1003 $info refers to the XML values in column info COPY: provide copy of the XML value in $newinfo MODIFY: delete element node RETURN: provide new content of $newinfo as result 45

XQuery Syntax of Updates XQueryUpdate ::= transform CopyClause ModifyClause ReturnClause CopyClause ::= copy "$" $ VariableName ":=" XQueryExpression 1 ("," "$" VariableName ":=" XQueryExpression 1 )* ModifyClause ::= modify ( Renaming Delete Insert Replace ) ( "," (Renaming Delete Insert Replace ) )* Renaming ::= do rename XQueryExpression 1 as QNameExpression Delete ::= do delete XQueryExpression 2 Insert ::= do insert XQueryExpression 2 ( before after as first into as last into into ) XQueryExpression 1 Replace ::= do replace ( value of )? XQueryExpression 1 ReturnClause with XQueryExpression 2 ::= return XQueryExpression 1 Some XQueryExpression have to result in a single result node possibly with subnodes 2 Some XQueryExpressions may result in sequences The XQuery expressions must not contain updates 46

Register XML Schema Register a single schema document REGISTER XMLSCHEMA 'http://posample.org' FROM 'file:///c:/sqllib/samples/xml/customer.xsd' AS posample.customer COMPLETE How to register multi-document schemas? 1. Option: ADD clause - ADD clause of the REGISTER statement allows to add several documents 2. Option: several REGISTER statements - Register documents in separate REGISTER statements - Complete registration process by COMPLETE XMLSCHEMA command 47

Using XML Schemas Example: INSERT INTO Customer(Cid, Info) VALUES (1003, XMLVALIDATE ( XMLPARSE (DOCUMENT ' <customerinfo xmlns="http://posample.org" Cid="1003"> <name>robert Shoemaker</name> <addr country="canada"> <street>1596 Baseline</street> <city>aurora</city> <prov-state>ontario</prov-state> <pcode-zip>n8x 7F8</pcode-zip> </addr> <phone type="work">905-555-7258</phone> <phone type="home">416-555-2937</phone> <phone type="cell">905-555-8743</phone> <phone type="cottage">613 >613-555-3278</phone> 3278</phone> </customerinfo>' PRESERVE WHITESPACE ) ACCORDING TO XMLSCHEMA ID posample.customer )) XMLPARSE provides XML data form string XMLVALIDATE uses the registered schema to validate XML data 48

Overview Introduction SQL/XML data type XML XML functions mappings Example: DB2 native XML DBMS 49

XML Enabled Databases A database that has an added XML mapping layer provided either by the database vendor or a third party. This mapping layer manages the storage and retrieval of XML data. Data that is mapped into the database is mapped into application specific formats and the original XML meta-data and structure may be lost. Data retrieved as XML is NOT guaranteed to have originated in XML form. Data manipulation may occur via either XML specific technologies (e.g. XPath, XSL-T, DOM or SAX) or other database technologies(e.g. SQL). The fundamental unit of storage in an XEDB is implementation dependent. Source: [Sch03, XML:DB Initiative for XML Databases] 50

Native XML Databases Defines a (logical) model for an XML document -- as opposed to the data in that document -- and stores and retrieves documents according to that model. At a minimum, the model must include elements, attributes, PCDATA, and document order. Examples of such models are the XPath data model, the XML Infoset, and the models implied by the DOM and the events in SAX 1.0. Has an XML document as its fundamental unit of (logical) storage, just as a relational database has a row in a table as its fundamental unit of (logical) storage. Is not required to have any particular underlying physical storage model. For example, it can be built on a relational, hierarchical, or object- oriented database, or use a proprietary storage format such as indexed, compressed files. Source: [Sch03, XML:DB Initiative for XML Databases] 51

Hybrid XML Databases A database that can be treated as either a Native XML Database or as an XML Enabled Database depending on the requirements of the application. Source: [Sch03, XML:DB Initiative for XML Databases] 52

Sample XML Databases System Provider Remarks MonetDB Research Institutions [monetdb.cwi.nl] open-source Tamino Software AG commercial, native Xindice [apache.org] open-source, native exist-db [sourceforge.net] open-source, native SQL Server Microsoft commercial DB2 IBM commercial Oracle Oracle commercial Oracle Berkeley DB XML Oracle open-source 53

Tamino 54

Literature & Information [Tur03] Can Türker: SQL:1999 & SQL:2003. dpunkt.verlag, 2003. [Sch03] Harald Schöning: XML und Datenbanken. Hanser, 2003. 55