Consuming JSON Data with SQL

Consuming JSON Data with SQL Birgitta Hauser bha@toolmaker.de / Hauser@SSS-Software.de TOOLMAKER Advanced Efficiency GmbH Tel. (+49) 08191 968-0 www.toolmaker.de Landsberg am Lech - Bayertor Seite 2 1

Agenda Consuming JSON Data JSON Support JSON Basics JSON_TABLE Table Function JSON Path Expression JSON to SQL Data Type Conversion Column Definition Regular Definition Nested Path Definition Ordinality Columns Returning JSON Formatted Data Consuming Web-Services returning JSON Data Seite 3 JSON - Support Seite 4 2

What is JSON? JSON = JavaScript Object Notation Is based on a subset of the JavaScript Programming Language, (Standard ECMA-262 3rd Edition - December 1999) Is a lightweight data-interchange format Is a completely language independent text format JSON text is a sequence of Unicode code points that conforms to the JSON value grammar Includes: six structural tokens { } [ ] :, necessary for defining (Nested) Objects and Arrays strings numbers three literal name tokens true false null Seite 5 JSON Support in DB2 for i TR2 - March 2017 IBM i 7.2 & 7.3 TR2: JSON Predicates Is JSON / Is Not JSON JSON_EXISTS Source: What s New with DB2 for i Doug Mack - IBM DB2 for i Analytics Consultant Common Fall Conference 2016 Seite 6 3

JSON Support in DB2 for i JSON Publishing Functions are planned Source: What s New with DB2 for i Scott Forstie - IBM DB2 for i Business Architect Common Annual Conference 2017 Seite 7 JSON - Basics Seite 8 4

JSON Values A JSON value can be an Object, Array, Number, String, true, false, null Seite 9 JSON String Seite 10 5

JSON String Sequence of Unicode code points Wrapped with quotation marks ( ) All characters can be placed within the quotation marks except for the characters that must be escaped: \" quotation mark character (U0022) \\ reverse solidus/backslash character (U005C) \/ solidus/slash character (U002F) \b backspace character (U0008) \f form feed character (U000C) \n line feed character (U000A) \r carriage return character (U000D) \t character tabulation character (U0009) Unicode Hex Value: \u005c (= Back slash) Seite 11 JSON Numbers JSON Numbers Consists of digits 0-9 (without leading zeros) Negative Numbers have a leading minus sign ( - ) As Decimal Separator a period (. ) must be used May have an exponent of ten, prefixed by e/e Seite 12 6

JSON Objects Object Structure Consists of a pair of curly brackets ( { } ) surrounding zero or more name/value pairs A single colon ( : ) follows each name separating the name from the value number, string, true, false, null, object, array A single comma (, ) separates (multiple) name/value pairs Object consisting of FirstName and LastName Seite 13 JSON Arrays Array Structure Consists of a pair of square brackets ( [ ] ) Surrounding zero or more values Values are separated by commas (, ) Order of the values is significant, Array of Objects Seite 14 7

JSON Document # = Numeric Value s = String RDi JSON Editor Object Name FirstName, LastName Array Phone Multiple Objects with CountryCode, AreaCode, PhoneNumber Seite 15 JSON_TABLE Seite 16 8

JSON_TABLE Table Function Parsing and Converting JSON Data JSON Parser Reads JSON data located In an Character Column in a database table In the IFS (Integrated File System) in composition with the GET_CLOB_FROM_FILE or GET_BLOB_FROM_FILE scalar functions (with http-functions in SYSTOOLS) returned by a RESTful WebService Hierarchical JSON information is converted into a relational format Returns a table Must be specified within the FROM clause of a SELECT statement JSON_TABLE Table Function can be used like any table or view {"id" : 901, "name": {"first":"birgitta", "last" :"Hauser" }, "department": "Programming"}... additional employees id first last department 901 Birgitta Hauser Programming Seite 17 SQL/JSON Path Expressions Seite 18 9

SQL/JSON Path Expressions SQL/JSON Path Expressions are used for locating the requested Information within the JSON Text SQL/JSON Path Expression defines the starting point SQL/JSON Column Path Expression Constant is used to find the column value relative to the starting point Not specified: Default = '$.' Current Level NESTED Paths are used for locating sub-ordinated Objects/Arrays Empty Sequences and Errors On Empty Clause: Provides the default value for an empty element Not specified NULL value On Error Clause: Defines the default value to be returned if an error occurs On Error On Empty Returns an error for an empty element Seite 19 JSON_TABLE Table Function Syntax JSON Text JSON_TABLE ( JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (JSON_Table-Column Definition,... ) EMPTY/ERROR ON ERROR) JSON_TEXT Column in a database Table, in a Common-Table-Expression (CTE) or nested Sub-Select LOB-Locator on an IFS-File returned by one of the following functions GET_BLOB_FROM_FILE Returns a BLOB-Locator GET_CLOB_FROM_FILE Returns a CLOB-Locator GET_DBCLOB_FROM_FILE Returns a DBCLOB-Locator LOB-Locator returned by one of the http-functions located within the SYSTOOLS schema Consuming Web-Services Seite 20 10

JSON_TABLE Table Function Syntax Format JSON/BSON JSON_TABLE ( JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (JSON_Table-Column Definition,... ) EMPTY/ERROR ON ERROR) FORMAT JSON or FORMAT BSON FORMAT JSON Text/Document contains JSON data FORMAT BSON Text/Document contains data in binary JSON Text/Document must be in an binary Format Default: FORMAT JSON Text/Document is a character or graphic string FORMAT BSON Text/Document is in binary format Seite 21 JSON_TABLE Table Function Syntax SQL/JSON Path Expression JSON_TABLE ( JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (JSON_Table-Column Definition,... ) EMPTY/ERROR ON ERROR) SQL/JSON Path Expression Starting Point '$' = parse the complete document '$.Obj1.Obj2.Obj3' or '$.Obj1.[*] Attention: Path Expressions are case senitive Empty Sequences and Errors On Empty Clause: Provides the default value for an empty element Not specified NULL value On Error Clause: Defines the default value to be returned if an error occurs On Error On Empty Returns an error for an empty element Seite 22 11

JSON_TABLE Table Function Syntax Column Definition JSON_TABLE ( JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (JSON_Table-Column Definition,... ) EMPTY/ERROR ON ERROR) Different JSON_Table-Definition Types Regular Column Definition Nested Column Definition Standard Definition Necessary for nested arrays Ordinality Column Definition adds columns with running numbers for sequencing array elements Formatted Column Definition returns object and array data in JSON Format Seite 23 JSON_TABLE Table Function Syntax Regular Column Definition JSON_TABLE (JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (ColumnName DataType PATH Column-Path-Expression-Constant NULL/Error/Default on Empty/Error) EMPTY/ERROR ON ERROR) Regular Column Definition ColumnName DataType Unique Column Name in the result table cannot be qualified DataType and Length of the column in the result table Seite 24 12

JSON to SQL Data Type Conversion Seite 25 JSON to SQL Data Type Conversion Numeric Values JSON Data Type to SQL Data Type Conversion JSON Type Number SQL Type SmallInt Integer BigInt Decimal Numeric Float Double Real DecFloat Comments Decimal Positions are truncated conversion occurs to the specified precision and scale If the source contains more significant digits than the precision of the target data type, the source vlaue is rounded to the precision of the target data type. If an overflow occurs an error is returned Seite 26 13

JSON to SQL Data Type Conversion Numeric or String Values JSON Data Type to SQL Data Type Conversion JSON Type String or Number SQL Type Char VarChar CLOB Graphic VarGraphic DBCLOB Comments Conversion to the CCSID of the target data type. Truncation occurs if the specified length limit is smaller than the text. A warning occurs if non blank characters are truncated Seite 27 JSON to SQL Data Type Conversion Date and Time Values JSON Data Type to SQL Data Type Conversion JSON Type String SQL Type Date Time Timestamp Comments The JSON String is converted into a date, if the string is in one of the following formats: ISO or JIS = YYYY-MM-DD USA = MM/DD/YYYY EUR = DD.MM.YYYY The JSON String is converted into a time, if the string is in one of the following formats: ISO or EUR = HH.MM.SS JIS or HMS = HH:MM:SS The JSON String is converted into a timestamp, if the string is in one of the following formats: ISO = YYYY-MM-DD HH:MM:SS.NNNNNN IBMSQL = YYYY-MM-DD-HH.MM.SS.NNNNNN ISO-8601 = YYYY-MM-DDTHH:MM:SS.NNNNN<+/- Offset> Seite 28 14

JSON to SQL Data Type Conversion Miscellaneous JSON Data Type to SQL Data Type Conversion JSON Type Array or Object null Bololean (true/false) SQL Type Char VarChar CLOB Graphic VarGraphic DBCLOB Any Char VarChar CLOB Graphic VarGraphic DBCLOB Comments JSON Arrays or objects are returned as JSON formatted character strings Converted to an SQL NULL value A JSON boolean value is converted to either a true or false String and returned as String Seite 29 JSON Document # = Numeric Value s = String Object Name FirstName, LastName RDi JSON Editor Array Phone Multiple Objects with CountryCode, AreaCode, PhoneNumber Seite 30 15

Access JSON Document - Example Retrieve First/Last Name Document located in MYJSON Table JSONTEXT Column Starts with the Name object Returns FirstName and LastName from Name Result column names and JSON names identical Path specification not required Seite 31 JSON_TABLE Column Definition Seite 32 16

JSON_TABLE Table Function Syntax Regular Column Definition JSON_TABLE (JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (ColumnName DataType PATH Column-Path-Expression-Constant NULL/Error/Default on Empty/Error) EMPTY/ERROR ON ERROR) PATH Part of the syntax must be specified Column-Path-Expression-Constant Format 'lax/strict $.Obj1.Obj2.Name' $. = Current Level = (starting) Object/Array specified within the SQL/JSON Path Expression lax / strict Optional Default = lax Attention: lax/strict in lowercase lax strict Structural problems are tolerated Arrays are automatically unnested Structural problems result in an error Arrays are not automatically unnested Seite 33 JSON_TABLE Table Function Syntax Regular Column Definition JSON_TABLE (JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (ColumnName DataType PATH Column-Path-Expression-Constant NULL/Error/Default on Empty/Error) EMPTY/ERROR ON ERROR) NULL / ERROR / Default Value for an EMPTY Information NULL ON EMPTY ERROR ON EMPTY Default Returns a NULL value Returns an Error instead of the empty information Default ON EMPTY Returns the specified Default value NULL / Default Value if an Error occurs NULL ON ERROR Returns a NULL Value if an error occurs Default ON ERROR Returns the specified Defaut value if an error occurs Seite 34 17

Access JSON Document - Example Retrieve First/Last Name Document located in MYJSON Table JSONTEXT Column Result Column names and JSON names different Path Specification required lax/strict not specified Seite 36 Access JSON Document - Example Retrieve First/Last Name Document located in the IFS /home/hauser/json01-namephone.json Starts with the Name object Returns FirstName and LastName from Name lax/strict not specified Path Constant includes lax specification Seite 37 18

JSON_TABLE Table Function Syntax Regular Column Definition JSON_TABLE (JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (ColumnName DataType PATH Column-Path-Expression-Constant NULL/Error/Default on Empty/Error) EMPTY/ERROR ON ERROR) Array Specification [ ] $.Path[*] $.Path[number] All array elements Specific array element Attention: Array Index starts with 0 1st Element Index = 0 $.Path[last] Last array element $.Path[last number] Determining the element based on the last element and going back the specified number of elements $.Path[number to number] Range of elements Seite 38 Access JSON Document Example Arrays All / Specific Elements Returns the complete phone number array Returns the 3rd [2]! phone number within the array Seite 39 19

Access JSON Document Example Arrays Last/Previous Elements Returns the last phone number within in the array Returns the 3rd last phone number within in the array Seite 40 Access JSON Document Example Arrays Range of Elements Returns the 2nd to 4th phone number within in the array Seite 41 20

Access JSON Document Example Array Elements side by side Returns the first 3 phone numbers side by side Array Element is explicitly specified within the column path expression Seite 42 JSON_TABLE Column Nested Path Definition Seite 43 21

JSON_TABLE Tabellen-Funktion Syntax Nested Path Expression JSON_TABLE (JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (Regular Column Definition, NESTED Nested-Path-Expression) EMPTY/ERROR ON ERROR) Nested Path Expression Allows retrieving multiple rows from objects embedded within array elements Indicates which elements of an array should be used to produce rows Starts with NESTED Followed by COLUMNS ( JSON-Table-Regular-Column-Defininition, JSON-Table-Formatted-Column-Definition, JSON-Table-Ordinality-Column-Definition, JSON-Table-Nested-Column-Definition) Seite 44 Access JSON Document Example Object Infos nested in Arrays Id, FirstName, LastName Single occurences not embedded in an array CountryCode, AreaCode, PhoneNumber Embedded within the Phone Array NESTED path specification required Seite 45 22

JSON_TABLE Ordinality Column Definition Seite 50 JSON_TABLE Table Function Syntax Ordinality Column Definition JSON_TABLE(JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (Regular Column Definition, ColumnName FOR ORDINALITY) EMPTY/ERROR ON ERROR) Ordinality Column Ordinality column is used to number rows within a nesting level An Ordinality Column can be defined on each level Multiple Ordinality columns within the same JSON_TABLE function Defined through a ColumnName followed by FOR ORDINALITY Seite 51 23

Access JSON Document Example Add Ordinality Colums Ordinality Column on Id/FirstName/LastName Level Ordinality Column on the Phone Array Level Seite 52 JSON_TABLE Return JSON-formatted Data Seite 53 24

JSON_TABLE Table Function Syntax JSON Formatted Column Definition JSON_TABLE( JSON_Text FORMAT JSON/BSON, SQL/JSON Path Expression AS PathName COLUMNS (Regular Column Definition, ColumnName DataType FORMAT JSON Column-Path-Expression-Constant) EMPTY/ERROR ON ERROR) Array and Object Information are returned in JSON Format Syntax is similar to the JSON-Regular-Column-Definition except FORMAT JSON must be specified For returning JSON-formatted data ColumnName DataType FORMAT JSON Unique Column Name within the result table Character Data Type for receiving the JSON-text data Required Part of the syntax Column-Path-Expression-Constant Location of the Object/Array Seite 54 Access JSON Document Example Returning JSON-formatted Data Name Object returned as JSON-formatted data Phone Array returned as JSON-formatted data Seite 55 25

More complex Examples Seite 56 Multiple Nested Objects and Arrays Example Opened with the RDi JSON Editor Seite 57 26

Multiple Nested Objects and Arrays Structure - Example Staff Object Department Array Department Array Employee Array, DeptName Employee Array Name Object, Privacy Object, CostCenter, EmployeeId Name Object FirstName, LastName Privacy Object email, Phone Array, Birthday, Salary Object Phone Array Objects with CountryCode, AreaCode and Phone Number Salary Object Currency, Salary Seite 58 Multiple Nested Objects and Arrays Nested Arrays Example 1 All elements of an array in an array Default values for date and numeric columns Seite 59 27

Multiple Nested Objects and Arrays Nested Arrays Example 2 Department Name on a higer level must be added Start with $.Staff.Department[*] Employee Information must be nested Seite 60 Multiple Nested Objects and Arrays Double Nested Arrays Example 3 Phone is an array within the Employee array double nested Ordinality Colums for all arrays Department, Employee, Phone Seite 61 28

Query the Web with SQL Seite 62 http Functions for DB2 for i for RESTFul Web-Services Prerequisites: 5770-SS1 DB2 PTF Group SF99701 Level 23 Java 1.6 or later (5761-JV1 Option 11, 12, 14, or 15) New http Functions http Functions added to the SYSTOOLS schema SYSTOOLS schema is NOT part of the system path/library list must be (manually) added to the SQL path/library list http Functions they are NOT subject to IBM service and support IBM maintenance for SYSTOOLS is to delete existing objects followed by the creation of the newer versions (without considering any modifications) Can be used as soon as DB2 PTF Group Level 23 is installed Java source is provided in /QIBM/ProdData/OS/SQLLIB/bin/systools_java_source.jar Seite 63 29

http Functions for DB2 for i Scalar and Table Functions (names starting with http) Located in schema SYSTOOLS Function Names http (Method) (DataType) (Verbose) Method: Data type: GET, PUT, POST, DELETE, HEAD Without method Method passed as parameter value of the request AND response messages CLOB (Character Large Object) / BLOB (Binary Large Object) Verbose: with Verbose: Table function with 2 columns 1st column: http header information 2nd column: response message without Verbose: Scalar Function response message only Examples: httpgetclob(), httpputblobverbose(), httpclob() Seite 64 IBM i Navigator http Functions in Schema SYSTOOLS Seite 65 30

Consuming Webservices returning JSON Seite 69 Web Services Weather Forecast Free (upto 5000 calls/month) Weather API/Web Services At https://www.apixu.com Seite 70 31

Web Services Weather Forecast Call Seite 71 Web Services Weather Forecast Call Result Current Weather Forecast for Kaufering Browser: Call for Kaufering &q=kaufering httpgetclob SQL Function: Call for Kaufering &q=kaufering Seite 72 32

Web Services Weather Forecast Result Displayed with RDi Current Weather Forecast for Kaufering Result displayed with the RDi JSON-Editor Seite 73 Web Services Weather Forecast Result JSON_TABLE Access using the httpgetclob function Define Output Columns Default values for errorneous numeric vales Seite 74 33

Global Variables Seite 75 Global Variable What is a SQL Global Variable? It is a permanent object (*SRVPGM) located in a schema Is created with the SQL command CREATE VARIABLE Almost any SQL data type is allowed (Except ROWID and DATALINK) Cannot be created in the QTEMP library/schema Even though being global it is managed at job/connection level The same variable may have different values for different jobs/connections Initialization occurs with the first use in a job/connection Initialization can be based on a constant value a special register an other (SQL) global variable a scalar function or a User Defined Function (UDF) a SQL statement that returns a single value (row/column) Can be used in triggers and in view definitions Seite 76 34

Global Variable SQL Command CREATE VARIABLE Variable Name (upto 128 characters) must be specified Almost any data type allowed (except ROWID and DATALINK) Default Values can be specified (Default = NULL) Seite 77 Web Services Weather Forecast View including Global Variable Global Variable for City View Accessing WebService URL includes global Variable Seite 82 35

Web Services Weather Forecast Access through View and Global Variable Initialize the global variable with Kaufering Access the view in an SELECT-Statement Initialize the Variable with New York global variable is integrated in the URL URL encoding is required URLENCODE scalar function within the SYSTOOLS schema Access the view in an SELECT-Statement Seite 83 Any Questions? Seite 84 36

Kaufering Kaufering Schloss Igling (Castle) Seite 85 Special Thanks to Deon (Kit) von Blerk For proof-reading/correcting the English translation of these presentations www.changefit.com Holger Scherer For providing an AS/400 (IBM i-system) enabling the creation of the samples/code used in these presentations http://www.rzkh.de/start_en.html Your data is save! in the bunker Seite 86 37

References IBM i Information Center DB2 for i SQL reference http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzajp/rzajppdf.pdf?lang=en Embedded SQL programming http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_73/db2/rbafzpdf.pdf?lang=en RPG Reference http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzasd/sc092508.pdf?lang=en IBM Redbooks Who Knew You Could Do That with RPG IV? A Sorcerer's Guide to System Access and More http://www.redbooks.ibm.com/abstracts/sg245402.html?open Modernizing IBM eserver iseries Application Data Access A Roadmap Cornerstone http://www.redbooks.ibm.com/abstracts/sg246393.html?open Modernizing IBM i Applications from the Database up to the User Interface and Everything in Between http://www.redbooks.ibm.com/abstracts/sg248185.html?open Seite 87 Speaker s Biography Birgitta Hauser has been a Software and Database Engineer since 2008, focusing on RPG, SQL and Web development on IBM i at Toolmaker Advanced Efficiency GmbH in Germany. She graduated with a business economics diploma, and started programming on the AS/400 in 1992. She also works in consulting and education as a trainer for RPG and SQL developers. Since 2002 she has frequently spoken at the COMMON User Groups and other IBM i and Power Conferences in Germany, other European Countries, USA and Canada. In addition, she is co-author of two IBM Redbooks and also the author of several articles and papers focusing on RPG and SQL for a German publisher, ipro Developer and IBM DeveloperWorks. In 2015 she received the John Earl Speaker Scholarship Award. Seite 88 38

Consume JSON with SQL! i love it! Thank you! Birgitta Hauser Hauser@sss-software.de bha@toolmaker.de 11.06.2017 Seite 89 CEC 2017 - June, 18-21 2017 - Consuming JSON Data with SQL - Birgitta Hauser Toolmaker Advanced Efficiency GmbH Westendstraße 34 86916 Kaufering www.toolmaker.de Tel. (+49) 08191 968-0 39