Configuration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Transcription

1 PUBLIC SAP HANA Platform SPS 12 Document Version: Configuration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data

2 Content 1 Getting Started Open a Support Connection Overview of Developer Tasks Remote and Virtual Objects Search for an Object in a Remote Source Creating Virtual Tables Creating Virtual Tables from Remote Objects Creating Virtual Tables from a New Remote Source Create a Virtual Function Partitioning Virtual Table Data for Better Performance Enterprise Semantic Services Basic Search Query Syntax Search String Examples Search String Attribute Type and Content Type Names Define Term Mappings for Search Search for Remote Objects Using the SQL View Attribute Filter Expressions Choosing the Run-time Behavior AFL Function Aggregation Case Cleanse Cleanse Configuration in Web-based Development Workbench About Cleansing Cleanse Input Columns Cleanse Output Columns Data Mask Change Default Data Mask Settings Mask Type Date Variance Type Numeric Variance Type Pattern Variance Type Data Sink SAP SE or an SAP affiliate company. All rights reserved. Content

3 Data Sink Options Using Virtual Tables Data Source Data Source Options Reading from Virtual Tables Date Generation Filter Filter Options Geocode Geocode Configuration in Web-based Development Workbench About Geocoding Geocode Input Columns Geocode Output Columns Hierarchical Configure a Hierarchical node History Preserving Input Type Join Join Options Lookup Map Operation Match Match Options Match Input Columns Match Output Columns Output Type Pivot Procedure Procedure options R-Script Row Generation Sort Sort Options Table Comparison Template File Template File Options Union Union Options UnPivot Add a Variable to the Container Node Load Behavior Options for Targets in Flowgraphs Content 2016 SAP SE or an SAP affiliate company. All rights reserved. 3

4 5.31 Importing an ADP Flowgraph Activate and Execute a Flowgraph Reserved Words Nodes Available for Real-time Processing Use the Expression Editor Profiling Data Semantic Profiling Distribution Profiling Metadata Profiling Replicating Data Create a Replication Task Add a Target Column Edit a Target Column Delete a Target Column Load Behavior Options for Targets in Replication Tasks Activate and Execute a Replication Task SAP HANA DDL Propagation Scan Interval SAP SE or an SAP affiliate company. All rights reserved. Content

5 1 Getting Started The describes concepts necessary for replicating, enhancing, cleansing, and transforming your data to make it more accurate and useful in less time using SAP HANA. There are two main concepts regarding this product: smart data integration and smart data quality. This Configuration Guide provides conceptual, procedural, and reference information for a subset of the available features. The information in this guide is mainly for users in a developer role but may contain some helpful information for administrators as well. See the Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data for installation, security, and administration tasks. See the Adapter SDK Guide for SAP HANA Smart Data Integration for creating, configuring, and deploying custom adapters. SAP HANA smart data integration is a set of functionality provided by several components that you can use to retrieve data from an external system, transform it, and persist it in SAP HANA database tables. SAP HANA smart data integration features and tools addressed in this guide: The Replication Editor in the SAP HANA Web-based Development Workbench is for creating real time or batch replication scenarios for moving data into SAP HANA studio. SAP HANA smart data integration transformation nodes available in the application function modeler of SAP HANA studio can be used for pivoting tables, capturing changed data, comparing tables, and so on. SAP HANA smart data quality is a set of functionality provided by several components that you can use to cleanse and enrich data before it is persisted in the SAP HANA database. SAP HANA smart data quality features described in this guide include batch and real-time data cleansing using the Cleanse node and enriching geospatial data with the Geocode node. In combination, smart data integration and smart data quality support the following use cases: Federation: Sending a query to the SAP HANA database for data that exists in an external system. Replication: Copying tables and their contents from an external system into the SAP HANA database with minimal processing. Transformation, Cleanse, and Enrich: Extracting data from an external system, transforming, cleansing, enriching, and persisting the results in the SAP HANA database. Use the information in this guide to configure the application to support these use cases. Getting Started 2016 SAP SE or an SAP affiliate company. All rights reserved. 5

6 1.1 Open a Support Connection In some support situations, it may be necessary to allow an SAP support engineer to log into your system to analyze the situation. Procedure 1. To enable a support user to log on to your system, complete the following tasks: a. Install the SAProuter as described on SAP Support Portal. b. Set up a support connection as described in SAP (SAP HANA database service connections). c. Configure a Telnet connection as described in SAP (Telnet link to customer systems) d. Configure an SAP HANA database connection as described in SAP (SAP HANA studio service connection). e. Configure a TREX/BIA/HANA service connection as described in SAP (TREX/BIA/HANA service connection to customer systems). 2. Create a database user and grant the MONITORING role. The MONITORING role allows a database user to open the SAP HANA Administration Console perspective with read-only access to the system, system views, statistics views, trace files, and so on. However, this role does not provide any privileges for accessing application data. With the MONITORING role, it is also not possible to change the configuration of or start and stop a system. You can grant the MONITORING role to a support engineer if SAP support needs to connect to the system. Depending on the issue to be analyzed, further privileges may be needed to allow sufficient analysis (for example, to access application data or data models). Related Information SAP SAP SAP SAP SAProuter SAP SE or an SAP affiliate company. All rights reserved. Getting Started

7 2 Overview of Developer Tasks Developer tasks described in this guide consist of designing processes that replicate data and processes that transform, cleanse, and enrich data. The administrator should have already installed the Data Provisioning Agents, deployed and registered the adapters, and created the remote sources. See the Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data. Here are the tasks typically performed by a developer. Design data replication processes. Design data transformation processes, which can include cleansing and enrichment. See the Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data for details. Related Information Replicating Data [page 190] Overview of Developer Tasks 2016 SAP SE or an SAP affiliate company. All rights reserved. 7

8 3 Remote and Virtual Objects This section provides an overview of how to use Data Provisioning adapters with remote sources, virtual tables, virtual functions, and virtual procedures with SAP HANA. Administrators add remote sources to the SAP HANA interface to make a connection to the data. Then developers access the data by creating a virtual table from a table in the remote source. A virtual table is an object that is registered with an open SAP HANA database connection with data that exists on the external source. In SAP HANA, a virtual table looks like any other table. You can also create virtual functions, which allow access to remote sources like web services, for example. Virtual procedures expand on virtual functions by letting you have large objects and tables as input arguments and can also return multiple tables. 3.1 Search for an Object in a Remote Source You can search remote sources to find objects in the SAP HANA Web-based Development Workbench Catalog and create virtual tables. Prerequisites Searching a remote source requires the following privilege on the remote source: GRANT ALTER ON REMOTE SOURCE <remote_source_name> TO <user>. Granting this privilege allows the creation of a dictionary, which can then be searched. Additionally, if you are using an SAP ECC adapter, be sure that you have a SELECT privilege on the following tables: DM41S DM26L DD02VV DM40T DD02L DD16S DD02T DD03T DD03L SAP SE or an SAP affiliate company. All rights reserved. Remote and Virtual Objects

9 Context In the SAP HANA Web-based Development Workbench Catalog, expand Provisioning Remote Sources. Procedure 1. Right-click the remote source to search in and select Find Table. 2. In the Find Remote Object window, click Create Dictionary to build a searchable dictionary of objects from the source. 3. To search, enter filter criteria for Display Name, Unique Name, or Object that Contains, Equals, Starts with, or Ends with characters you enter. For example, to filter by name, enter the first few characters of the object name to display the objects that begin with those characters. The Case sensitive restriction is optional. To add additional criteria to further filter the list, click the plus sign and enter the additional parameter(s). 4. (Optional) The bottom of the window includes a time stamp for when the dictionary was last updated. You can refresh or clear the dictionary here. 5. Click Create Virtual Table. 6. Enter a Table Name. 7. Select a target Schema. 8. Click OK. 9. Close the Find Remote Object window. 3.2 Creating Virtual Tables To read and write data from sources external to SAP HANA, create virtual tables within SAP HANA to represent that data. You can create virtual tables that point to remote tables in different data sources. You can then write SQL queries in SAP HANA that can operate on virtual tables. The SAP HANA query processor optimizes these queries, and then executes the relevant part of the query in the target database, returns the results of the query to SAP HANA, and completes the operation. Within SAP HANA studio, you can create virtual tables in SAP HANA by searching for the exact name of the table in the remote system or by browsing for the remote object. Within the SAP HANA Web-based Development Workbench, you create virtual tables in SAP HANA by browsing for the remote object. (Right-click the remote source and select New virtual table.) Remote and Virtual Objects 2016 SAP SE or an SAP affiliate company. All rights reserved. 9

10 Related Information SAP HANA Administration Guide (HTML) SAP HANA Administration Guide (PDF) Creating Virtual Tables from Remote Objects How to create virtual tables from remote objects Context You have already created a remote source. It appears in the Systems view, Provisioning Remote Sources. Procedure 1. In the Systems view, expand the Remote Sources. Select the correct user and correct remote object on which you want to create your virtual table. 2. Right click the remote object and select Add as Virtual Table Enter the Table Name and select the Schema from the drop-down list. 4. Click Create, Result: An information box appears stating that the virtual table has been added. Results The new virtual table will appear in the Systems view, Catalog <schema> Tables Creating Virtual Tables from a New Remote Source How to create a virtual table from a new remote source Context Before proceeding, decide from which remote source you want to create a virtual table SAP SE or an SAP affiliate company. All rights reserved. Remote and Virtual Objects

11 Procedure 1. In the Systems view Catalog <Schema_Name> Tables, right click and select New Virtual Table Enter a Table Name, then click Browse... A dialog box appears. Select the remote source, database, and table. 3. Click OK. Result: The source name and the remote object are filled in automatically by the names of the components chosen in the previous step. 4. Click the Save the Editor icon in the upper right hand corner of the screen. Results This creates a virtual table. The new virtual table will appear in the tree view Catalog <schema> Tables. 3.3 Create a Virtual Function You can browse remote functions on remote sources using Data Provisioning adapters, then import them as virtual functions in the same way you browse remote tables and import them as virtual tables. Prerequisites The remote function has been created and is available in the SAP HANA Web-based Development Workbench: Catalog. Procedure 1. In the SAP HANA Web-based Development Workbench: Catalog, expand Provisioning Remote Sources. 2. Expand the remote source where you want to add the new virtual function. 3. Right-click the remote function and select New Virtual Function. 4. In the Create Virtual Function dialog box, enter a Function Name and select a Schema from the drop-down list. 5. Click OK. Results The new virtual table appears in the SAP HANA Web-based Development Workbench: Catalog ><Schema> > Functions. Remote and Virtual Objects 2016 SAP SE or an SAP affiliate company. All rights reserved. 11

12 Example If you use the SQL Console to create a function, the following example illustrates how to create a function that returns the sum of two numbers. First run the built-in procedure GET_REMOTE_SOURCE_FUNCTION_DEFINITION: CALL "PUBLIC"."GET_REMOTE_SOURCE_FUNCTION_DEFINITION" ('testadapter','sum',?,?,?); Copy the output of the configuration and paste it in the CONFIGURATION section: CREATE VIRTUAL FUNCTION SUM_TEST(A INT, B INT) RETURNS TABLE (SUM_VALUE INT) CONFIGURATION '{" DP_UNIQUE_NAME ":"sum"}' AT "testadapter"; For more information about using the SQL Console, see the SAP HANA Administration Guide. For syntax details for CREATE VIRTUAL FUNCTION, refer to the SAP HANA SQL and System Views Reference. Related Information SAP HANA Administration Guide (HTML) SAP HANA Administration Guide (PDF) SAP HANA SQL and System Views Reference (HTML) SAP HANA SQL and System Views Reference (PDF) 3.4 Partitioning Virtual Table Data for Better Performance For better performance, you can partition virtual table input sources and have SAP HANA read those input sources in parallel. You can specify within a replication task or flowgraph (within the Data Source node) that the system perform multiple reads in parallel from a virtual table input source in order to improve the reading throughput for some records of data provisioning type sources to be loaded in parallel. To verify that a certain data provisioning adapter supports partitioning, see the Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data for information on each adapter. By default, partitioning of data is not enabled. You can enable this feature to improve performance by selecting one of the two partition types: range partitions and list partitions. Range partitions may specify only a single value. List partitions may specify either a single value or a comma delimited list of values. For either type of partition, the value for each partition, representing which data goes into which parallel thread, should be entered in single quotation marks, as follows: 'example' SAP SE or an SAP affiliate company. All rights reserved. Remote and Virtual Objects

13 Partitioning of columns requires that the Not null attribute of the column is set to TRUE. Related Information Data Source Options [page 99] Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data (PDF) Remote and Virtual Objects 2016 SAP SE or an SAP affiliate company. All rights reserved. 13

14 4 Enterprise Semantic Services Enterprise Semantic Services provides an API to enable searching for artifacts and run-time objects based on their metadata and contents. Related Information Reference Information 4.1 Basic Search Query Syntax Basic search query syntax supported by Enterprise Semantic Services. query ::= [ scope-spec ] ( qualified-expression )+ scope-spec ::= (' category ' ' appscope ' ) ' : ' IDENTIFIER ( scope-spec )? qualified-expression ::= [ ' + ' ' - ' ] term-expression term-expression ::= attribute-type-expression attribute-filter-expression term attribute-type-expression ::= (attribute-type-name ' :: ' ( disjunctive-term-expression conjunctive-termexpression term )) ' date ' ' :: ' (disjunctive-date-expression conjunctive-date-expression date) attribute-filter-expression ::= attribute-filter-name ' : ' ( disjunctive-term-expression conjunctive-termexpression term ) disjunctive-term-expression ::= ' ( ' term ( ' OR ' term )* ' ) ' conjunctive-term-expression ::= ' ( ' term ( ' AND ' term )* ' ) ' disjunctive-date-expression ::= ' ( ' date ( ' OR ' date )* ' ) ' conjunctive-date-expression ::= ' ( ' date ( ' AND ' date )* ' ) ' term ::= WORD PHRASE attribute-name ::= IDENTIFIER attribute-type-name ::= 'AddressLine' 'FullAddress' 'BuildingName' 'StreetName' 'SecondaryAddress' 'Country' 'City' 'Postcode' 'Region' 'Firm' 'Person' 'FirstName' 'LastName' 'HonoraryPostname' 'MaturityPostname' 'Prename' 'PersonOrFirm' 'Title' 'Phone' 'SSN' 'NameInitial' ' ' /* attribute type names are case-insensitive */ attribute-filter-name ::= 'DesignObjectName' 'DesignObjectType' 'DesignObjectPath' 'DesignObjectFullName' 'EntitySetName' 'EntitySetType' 'EntitySetPath' 'EntitySetFullName' 'EntitySetLocation' /* attribute filter names are case-insensitive */ SAP SE or an SAP affiliate company. All rights reserved. Enterprise Semantic Services

15 WORD ::= ( [A-Za-z0-9] WILDCARD ) + /* A word containing wildcard characters is also called pattern */ PHRASE ::= ' " ' ( [/u0020-/u0021 /u0023-/uffff] WILDCARD) + ' " ' /* A phrase containing wildcard characters is also called pattern */ WILDCARD = ' * ' date::= [0-9] [0-9] [0-9] [0-9] ' - ' [0-9] [0-9] ' - ' [0-9] [0-9] /* YYYY-MM-DD */ [0-9] [0-9] [0-9] [0-9] [0-9] [0-9] [0-9] [0-9] /* YYYYMMDD */ [0-9] [0-9] [0-9] [0-9] ' - ' [0-9] [0-9] /* YYYY-MM */ [0-9] [0-9] [0-9] [0-9] /* YYYY */ IDENTIFIER ::= [A-Za-z]+ [A-Za-z0-9._$] Search String Examples Examples of search string elements and their descriptions. Element in Search String Search String Examples of Search Results Words Contracts by sales units Dataset names that include the words contracts, by, sales, or units Words and phrase number of contracts by customer region age year Dataset names that include number or contracts or customer region or age or year Words and pattern Revenue by prod* region Dataset names that include revenue or a word that starts with prod or the word region. Possible results include revenue production or products region. Pattern within phrase insur* cont* Dataset names that include a word starting with insur followed by a word starting with cont. Possible results include insurance contracts" or "insured contacts" Words and attribute type expression Number reversals date::(2010 OR 2011 OR 2012) Dataset names that includes the word number or reversals or the contents of the dataset include a date column that contains at least one of the values 2010, 2012, or 2012 Words and qualified expression Loss and Expense +"Outstanding Reserves" Loss and Expense -insurance Dataset names that optionally include the word loss or expense, but must contain "outstanding reserves" Dataset names that optionally include loss or expense, but must not contain "insurance" Enterprise Semantic Services 2016 SAP SE or an SAP affiliate company. All rights reserved. 15

16 Element in Search String Search String Examples of Search Results Date values Date::2000 Date::(2000 AND AND ) Date::(2000 OR OR ) Dataset that contains a date column with the value Dataset that contains a date column with the value 2000 and and Dataset that contains a date column with the value 2000 or or Attribute type expressions with geography values Attribute type expressions with pattern Attribute filter expressions city::("new York" OR Paris) country::(usa AND Canada) region::"ile de France" City::Washington* LastName::Panla* EntitySetLocation:local foodmart EntitySetFullName:(foodmart OR adventureworks) EntitySetType:table DesignObjectType:"hana * view" Dataset that contains a city column with either the value New York or Paris Dataset that contains a country column with both the value USA and Canada Dataset that contains a region column with the value Ile de France Dataset that contains a city column with a value that starts with Washington Dataset that contains a last name column with a value that starts with Panla Entitysets matching foodmart in local SAP HANA instance Entitysets with their name matching either foodmart or adventureworks Entitysets of sql table or hana virtual table type Entitysets with design objects of hana calculation view, hana attribute view or hana analytic view type See Attribute Filter Expressions [page 20]. Stopwords are either ignored or considered optional in a phrase. Stopwords are any pronoun, preposition, conjunction, particle, determiner, and auxiliary. For example, number of contracts will include the search results number contracts and number of contracts. Special characters are ignored. Special characters include \/;,.:-_()[]<>!?*@+{}="&. For example, "contract_number" will be handled as "contract number". 4.3 Search String Attribute Type and Content Type Names The search string can contain an attribute type name that corresponds to a content type name. The search results will return data set names that contain the content type and specified value SAP SE or an SAP affiliate company. All rights reserved. Enterprise Semantic Services

17 Attribute type names are not case sensitive in search strings. Attribute Type in Search String AddressLine FullAddress BuildingName StreetName SecondaryAddress Country City Postcode Region Firm Person FirstName LastName HonoraryPostname MaturityPostname Prename PersonOrFirm Title Date Phone SSN NameInitial Content Type Name Address Line Full Address Building Name Street Name Secondary Address Country City Postcode Region Firm Person First Name Last Name Honorary Postname Maturity Postname Prename Person Or Firm Title Date Phone SSN Name Initial Enterprise Semantic Services 2016 SAP SE or an SAP affiliate company. All rights reserved. 17

18 4.4 Define Term Mappings for Search Administrators define term mappings to provide multiple explicit interpretations of hypernyms, hyponyms, synonyms, acronyms, and abbreviations in a semantic search query. Context Term mappings provide explicit interpretations of a keyword in a semantic search query. A keyword can be interpreted as a hypernym, hyponym, or synonym in a given language, or as an acronym or abbreviation in a given business domain. Keyword Interpretation Term Mapping Example of Search Results Hypernym Find hyponyms (subcategories) of the search term. (Car, VW Golf) Search for car will match VW Golf" in the Entity Grid contents. Hyponym Find hypernyms (superordinates) of the search term. (VW Golf, Car) Search for VW Golf will match car in the Entity Grid. Synonym Find synonyms of the search term. (client, customer) and (customer, client) A search for client will match customer (and vice versa)) in the Entity Grid.. Acronym or Abbreviation Find acronyms or abbreviations of the search term. (Ltd, Limited) and (Limited, Ltd) A search for ltd will match limited (and vice versa) in the Entity Grid. (contract, contr) Plurals must be explicitly defined: (contracts, contrs) A search for contract will match contr in the Entity Grid. To define term mappings, do the following: Procedure 1. Log in to SAP HANA studio with a user or any user who has the Enterprise Semantic Search Administrator role. 2. For each term you want to map, insert a row into the term mapping table SAP_HANA_IM_ESS"."sap.hana.im.ess.services.search::Mapping, which has the following columns: Column Name MAPPING_ID Unique identifier SAP SE or an SAP affiliate company. All rights reserved. Enterprise Semantic Services

19 Column Name LIST_ID LANGUAGE_CODE TERM_1 TERM_2 A list_id value can be passed in the search.request parameter of the search API. Currently, only the following value is possible: en Term in the search query. Matching term in the Entity Grid. WEIGHT Always use 1. The following sample SQL statement maps the abbreviation Insur to insurance. insert into "SAP_HANA_IM_ESS"."sap.hana.im.ess.services.search::Mapping" values ('20','1','en','Insur','insurance',1); 4.5 Search for Remote Objects Using the SQL View You can view the metadata of all remote objects published in the Enterprise Semantic Services (ESS) Entity Grid using a SQL view. All remote objects published in ESS Entity Grid can be queried through the public SQL view "SAP_HANA_IM_ESS"."sap.hana.im.ess.services.views::REMOTE_OBJECTS". Users only see the remote objects for which they have access privileges. Grant the privilege CREATE VIRTUAL TABLE on the remote sources for which the user should have access. This view displays metadata information for each remote object. The description of each column of the view can be displayed in the view definition: Column REMOTE_SOURCE UNIQUE_NAME DISPLAY_NAME UNIQUE_PARENT_NAME DISPLAY_CONTAINER_PATH DATABASE OWNER Name of the remote source containing the remote object Unique identifier of the remote object within the remote source Display name of the remote object in the browsing hierarchy of the remote source Unique identifier of the parent node of the remote object in the browsing hierarchy of the remote source Display name of the container path of the remote object in the browsing hierarchy of the remote source Database name for the remote source. Can be null. Database owner name for the remote source. Can be null. Enterprise Semantic Services 2016 SAP SE or an SAP affiliate company. All rights reserved. 19

20 Column OBJECT_TYPE Type of the remote object (table or view) 4.6 Attribute Filter Expressions Enterprise Semantic Services attribute filter expression descriptions and examples. Attribute filters belong to two categories: Object filters apply on an individual object (for example, a design object or an entity set) Class filters apply to a group of objects. A class filter must be used in conjunction with at least one object filter or a keyword; otherwise, the query does not return any objects because the filter is considered to be too broad (can return too many objects). Attribute Filter Category Example Matching example DesignObjectName object filter Applies on the name of a designtime object from which runtime objects are created; for example, an SAP HANA view. DesignObjectName: (inventory OR ECC) This filter can match an SAP HANA view with name INVENTORY or ECC. RemoteSourceName object filter Applies on the name of a remote source. RemoteSourceName: ("DB2_ECC" OR "ORACLE ECC") RemoteSourceName: ("*ECC*" OR "*ECC*") RemoteSourceName: ("DB2" AND "ECC") This filter can match the remote sources: DB2_ECC_REMOTE_S OURCE ORACLE_ECC_REMOT E_SOURCE DesignObjectType class filter Applies on the type of a designtime object that was used to create a runtime object. Possible values of types of design-time objects are: DesignObjectType: "hana * view" This filter can match any SAP HANA view. SAP HANA calculation view SAP HANA analytic view SAP HANA attribute view DesignObjectPath object filter Applies on the path of the fully qualified name of a design-time object that was used to create a runtime object. For an SAP HANA view, the path represents the path of packages containing DesignObjectPath: "foodmart" DesignObjectPath: "hba.fscx604" DesignObjectPath:"sap * fscx604" The first filter can match any design object whose container path contains the string "foodmart" SAP SE or an SAP affiliate company. All rights reserved. Enterprise Semantic Services

21 Attribute Filter Category Example Matching example the view. There is no path for a remote source because it is the same as its full name. The second filter can match any design object whose container path matches the phrases "hba.fscx604" or "sap * fscx604". DesignObjectFullName object filter Applies on the fully qualified name of a design-time object. For an SAP HANA view, the fully qualified name includes the container path and the name. DesignObjectFullName : (foodmart OR "DB2 ECC") DesignObjectFullName :"foodmart/calculat* views" DesignObjectFullName :"foodmart calculat*views" DesignObjectFullName :"foodmart calculationviews" DesignObjectFullName :"hba.fscx604.calculati onviews" EntitySetName object filter Applies on the name of an entity set, which represents any object that can be returned in a search result. An entity set can represent: An SAP HANA catalog object A remote object EntitySetName: inventory EntitySetName: "business partner" The first filter matches any entity set that contains "inventory" in its name. The second filter matches any entity set that contains "business partner" in its name. EntitySetType class filter Applies on the type of an entity set. Possible values are: SQL table SQL view SAP HANA column view SAP HANA virtual table that remote objects are either of type SQL table or SQL view. EntitySetType: ("column view" OR "SQL table)" This filter matches any entity set of type "column view" or "SQL table". Enterprise Semantic Services 2016 SAP SE or an SAP affiliate company. All rights reserved. 21

22 Attribute Filter Category Example Matching example EntitySetPath object filter Applies on the path of the container of an object represented by an entity set. The path can be: A schema name for an SAP HANA catalog object A database.owner name for a remote object in a database system A path of folders for a remote object in an external application (for example ECC). EntitySetPath: "_SYS_BIC" EntitySetPath: "SAP_ANW" EntitySetPath:"SAP_C A - Cross Application Models ORG-EINH - Organizational units ORGE_A - Organizational Units Finance ORGE_ Dunning Area" +EntitySetPath: "finance" +EntitySetPath: "SAP" is equivalent to: +EntitySetPath: ("finance" AND "SAP") The first filter matches any entity set in schema _SYS_BIC. The second filter matches any entity set in the folder path matching the phrases. EntitySetFullName object filter Applies on the fully qualified name of an entity set. The fully qualified name includes the container path and the name of the object represented by the entity set. EntitySetFullName: (inventory OR T407M) EntitySetFullName: "DB2_ECC_REMOTE_S OURCE" EntitySetFullName: "DB2_ECC_REMOTE_S OURCE * * T047M" EntitySetFullName: "DB2_ECC_REMOTE_S OURCE null null T047M" EntitySetFullName: ("DB2_ECC_REMOTE_ SOURCE" AND "T047M") The first example matches any entity set whose qualified name contains one of the two strings "inventory" or "T407M". The second example matches any entity set whose qualified name contains the phrase DB2_ECC_REMOTE_S OURCE. The last three filters match the entity set: DB2_ECC_REMOTE_S OURCE.<NULL>.<NUL L>.T047M EntitySetLocation class filter Applies on the location of the object represented by an entity set. Possible values for location are: local means local SAP HANA instance, implicitly EntitySetLocation: local EntitySetLocation: remote Matches any SAP HANA catalog object Matches any remote object SAP SE or an SAP affiliate company. All rights reserved. Enterprise Semantic Services

23 Attribute Filter Category Example Matching example qualifying an SAP HANA catalog object remote means a remote object Enterprise Semantic Services 2016 SAP SE or an SAP affiliate company. All rights reserved. 23

24 5 Use the application function modeler in SAP HANA Studio or the flowgraph editor in Web-based Development Workbench to create flowgraphs to transform your data. Before using the application function modeler or the flowgraph editor, you must have the proper rights assigned. See the "Assign Roles and Privileges for tasks" topic in the SAP HANA Smart Data Integration and SAP HANA Smart Data Administration Guide. In application function modeler and Web-based Development Workbench, the data flows are stored as flowgraph objects with an extension of.hdbflowgraph. When activated, the data flows generate a stored procedure or a task plan. They can consume: database tables, views, and links to external resources relational operators such as filter, join, and union custom procedures written in SQL script functions from optional components such as the Application Function Library (AFL) or Business Function Library (BFL) smart data quality nodes such as Cleanse, Geocode, and Match (Web-based Development Workbench only) smart data integration nodes such as History Preserving, Lookup, Pivot, and Case See the SAP HANA Developer Guide for SAP HANA Studio for more information about the application function modeler such as creating flowgraphs, connections, adding/editing nodes, and templates. See the SAP HANA Developer Guide for SAP HANA Web Workbench for more information about Web-based Development Workbench. Configuring the flowgraph This section describes the options available for processing the flowgraph. In SAP HANA Web-based Development Workbench: 1. Click the Properties icon. 2. Select the target schema. This is where you can find the available input and output tables. 3. Select the Runtime behavior type. For details, see "Choosing a Runtime Behavior." Option Procedure Batch Task Realtime Task Processes data with a stored procedure. It cannot be run in realtime. A stored procedure is created after running a flowgraph. Only a portion of the nodes are available to use in the flowgraph (no Data Provisioning nodes). Processes data as a batch or initial load. It cannot be run in realtime. A stored procedure and a task is created after running a flowgraph. All nodes are available in the flowgraph. Processes data in realtime. A stored procedure and two tasks are created after running a flowgraph. The first task is a batch or initial load of the input data. The second task is run in realtime for any updates that occur to the input data SAP SE or an SAP affiliate company. All rights reserved.

25 Option Transactional Task Processes data in realtime. A single task is created after running a flowgraph that is run in realtime for any updates that occur to the input data. 4. Select Data Type Conversion (For Loader only) if you want to automatically convert the data type when there is a conflict. If a loader (target) data type does not match the upstream data type, an activation failure occurs. When you select this option, a conversion function is inserted to change the upstream data type to match the loader data type. For example, if you have selected this option and the loader data type for Column1 is NVARCHAR and is mapped to ColumnA that has a data type of CHAR, then a conversion function of to_nvarchar is inserted so that the flowgraph can be activated. However, if the input and output data types do not match, and this option is not enabled, then the flowgraph will not be activated. Upstream data type Conversion function Loader data type Flowgraph activation CHAR to_nvarchar NVARCHAR Activated CHAR n/a NVARCHAR Not activated 5. Click the plus icon to add any variables or scalar parameters that you want to execute during run time. 6. Click OK. Configuring nodes This section of the guide describes the input, output and configurable properties of smart data integration and the smart data quality nodes. In SAP HANA Web-based Development Workbench: 1. Select a node and drag it onto the canvas, and double-click to open. 2. To change the name of the node, enter a unique name in the Node Name option. 3. To perform just-in-time data preview, select the JIT Data Preview option. Just-in-time data preview is an option where you can process data from the beginning of the flowgraph up until the point of this node. After configuring this node, go back to the Flowgraph Editor, and click Save. Click the Data Preview icon to the right of this node to verify what your output will look like before running the entire flowgraph. The data is a temporary preview and is not written to any downstream output targets. Any changes to upstream nodes will result in changes to the data preview when the flowgraph is saved again. 4. To copy any columns from the input source to the output file, drag them from the Input pane to the Output pane. 5. Continue configuring the node in the Node Details pane. See the details of configuring each node later in this document. 6. Click Back to return to the Flowgraph Editor. In SAP HANA Studio Application Function Modeler: 1. Select a node and drag it onto the canvas SAP SE or an SAP affiliate company. All rights reserved. 25

26 2. To change the name of the node, click on the name. The name field becomes active for editing. The name of a node may only contain letters, underscores, and digits. It must be unique within the flowgraph. 3. In the General tab of the Properties view, configure the nodes. Related Information Choosing a Task Plan or a Stored Procedure [page 26] Nodes Available for Real-time Processing [page 172] Add a Variable to the Container Node [page 165] 5.1 Choosing the Run-time Behavior When creating a flowgraph, you need to consider how you want it processed. consider these main differences when selecting the runtime behavior: Whether to create a stored procedure, task plan, or both Which nodes you might use in the flowgraph Whether you want to process in realtime or batch mode When choosing between running a flowgraph as a task or a stored procedure is the availability of the nodes, and whether the flowgraph creates a stored procedure, a task plan or both. Procedure When selecting a stored procedure, you won't see the Data Provisioning palette, which contains a number of nodes used to transform data such as Cleanse, Match and Geocode. After activating the flowgraph, you will have created a stored procedure. Unlike task plans, stored procedures cannot be run in realtime mode. Instead, a stored procedure always runs in batch mode, that is, on the complete procedure input. Tasks The following general information applies to batch, realtime and transactional tasks SAP SE or an SAP affiliate company. All rights reserved.

27 Choose a task plan when you want to use any Data Provisioning nodes. These nodes are available in the Data Provisioning palette. The nodes in the General palette as well as those in the application function library and R script palettes can be used in the task plan. However, you cannot use the Data Sink (Template Table) node. You can use the standard Data Sink node. For this, the Data Sink table has to exist in the catalog. When you select a task plan, a Variable tab is enabled on the container node. There you can create variables to be used as part of function calls. Variables are created and initialized when the task is processed. You can explicitly specify the arguments to variables in the start command, or you will be prompted for initial values. For example, if you want to run the flowgraph for different regions in the US, then you can create variables such as "Southwest" or "Northeast" or "Midwest" in the container node. You'll set up a filter using the Filter node that can run only those records that match the filter. Then you can call the correct variable when calling Start Task, and only the data for those regions are processed. A connection can represent only one-to-one table mappings. The only exception is if the target anchor of the connection includes one of the following nodes: AFL function node Data Sink node Procedure node R script node Tip You can always represent the table mapping of a connection by adding a Filter node between the source and target of the connection, and then editing the table mapping in the Mapping Editor of the Filter node. You can learn more about these nodes in the SAP HANA Developer Guide. When creating a task plan, ensure that the column names for the input source and output target do not include any of the reserved words listed in the Reserved Words topic. For Realtime and Transactional tasks, when you are loading data from a virtual table, you must enable realtime processing on the source data in the flowgraph, depending on which data provisioning adapter you are using. Click the table and check the Real-time option in the properties. Batch Task In batch tasks, the initial load is updated only when the process is started or is scheduled to run. Unlike the procedure, you have access to all of the nodes. After running the flowgraph, a stored procedure and a task plan to run the batch is created. A batch task cannot be run in realtime. Realtime Task In realtime tasks, transactions are updated continuously. When the source is updated with a new or modified record, that record is immediately processed. After running a flowgraph, an initialization procedure and two tasks are generated. The first task is for the initial load, and the second task processes any new or updated data in the data source SAP SE or an SAP affiliate company. All rights reserved. 27

28 Transactional Task In transactional tasks, transactions are updated continuously in realtime. When the source is updated with a new or modified record, that record is immediately processed. Unlike the realtime task, only a single task is generated to process any new or updated data in the data source. You cannot use a virtual table for transactional tasks. Use Table-Type as the input source. Related Information Reserved Words [page 171] Add a Variable to the Container Node [page 165] Activate and Execute a Flowgraph [page 170] Administration Guide for SAP HANA smart data integration and SAP HANA Smart Data (PDF) 5.2 AFL Function Access functions of the Application Function Library. Prerequisites You have added an AFL Function node to the flowgraph. Context Use this node to model functions of the Application Function Library (AFL) that are registered with the system. AFL functions are grouped by function areas. You can retrieve the list of all AFL areas and functions registered in a HANA system by viewing the content of the views SYS. AFL_AREAS and SYS. AFL_FUNCTIONS. Many AFL areas are optional components for HANA. For some of these optional components the SAP HANA Application Function Modeler (AFM) provides preconfigured node templates. In this case, the AFM automatically displays a separate compartment for this area in the Node Palette. You can refresh the Node Palette by choosing Refresh in its context-sensitive menu SAP SE or an SAP affiliate company. All rights reserved.

29 The AFL Function node is not available for real-time processing. Procedure 1. Select the AFL Function node. 2. In the General tab in the Properties view, select the drop-down menus for Area and the Function. The AFM changes the inputs and outputs of the node according to the existing meta-data for the function on the server. For some AFL areas there exists a preconfigured Node Palette compartment. You cannot change the Area or the Function of a node added from one of these compartments. 3. If applicable, change the Category of the function. 4. Specify the inputs and the outputs of the function by editing the signature and the fixed content of its anchors. For some AFL areas there exists a preconfigured node template for this function. In this case, the fixed content of the inputs that define parameters is preconfigured. Related Information SAP HANA Business Function Library (BFL) SAP HANA Predictive Analysis Library (PAL) 5.3 Aggregation An Aggregation node represents a relational group-by and aggregation operation. Prerequisites You have added an Aggregation node to the flowgraph SAP SE or an SAP affiliate company. All rights reserved. 29

30 The Aggregation node is available for realtime processing. Procedure 1. Select the Aggregation node. 2. Map the input columns and output columns by dragging them to the output pane. You can add, delete, rename, and reorder the output columns, as needed. To multi-select and delete multiple columns use CTRL/ Shift keys, and then click Delete. 3. In the Aggregations tab, specify the columns that you want to have the aggregate or group-by actions taken upon. Drag the input fields and then select the action from the drop-down list. 4. (Optional) Select the Having tab to run a filter on an aggregation function. Enter the expression. You can drag and drop the input and output columns from the Elements pane, then drag an aggregation function from the Functions pane. Click or type the appropriate operators. For example, if you want to find the transactions that are over $75,000 based on the average sales in the 1st quarter, your expression might look like this: AVG("Aggregation1_Input"."SALES") > Option Avg Count Group-by Max Min Sum Calculates the average of a given set of column values. Returns the number of values in a table column. Use for specifying a list of columns for which you want to combine output. For example, you might want to group sales orders by date to find the total sales ordered on a particular date. Returns the maximum value from a list. Returns the minimum value from a list. Calculates the sum of a given set of values. 5. (Optional) Select the Filter Node tab to compare the column name against a constant value. Enter the expression by dragging the column names, the function, and entering the operators from the pane at the bottom of the node. For example, if you want to the number of sales that are greater than 10000, your expression might look like this: "Aggregation1_input"."SALES" > See the "SQL Functions" topic in the SAP HANA SQL and System Views Reference for more information about each function. 5.4 Case Specifies multiple paths in a single node (the rows are separated and processed in different ways). Route input records from a single source to one or more output paths. You can simplify branch logic in data flows by consolidating case or decision making logic in one node. Paths are defined in an expression table SAP SE or an SAP affiliate company. All rights reserved.

31 The Case node is available for real-time processing. General properties Table 1: General options Option Name Display name The name for the output target. This can be named the same as the input source. The name shown in the Palette pane. This option can only be changed when creating a template. It cannot be changed when using the node outside of a template. Produce default output Row can be true for one case only (Optional.) Provides a comment about the operation. For example, "Splitting the information into North America and Europe." Adds a default output target, such as a table. On the Default tab, specify the fields that should be included in the output. There can be one default output target only. If the record does not match any of the other output cases, it goes to the default output. Specifies whether a row can be included in only one or in many output targets. For example, you might have a partial address that does not include a country name such as 455 Rue de la Marine. It is possible that this row could be output to the tables named Canada_Customer, France_Customer, and Other_Customer. Select this option to output the record into the first output table whose expression returns TRUE. Leaving this option blank would put the record in all three tables. Expression name Expression Specify the name of the expression you are creating. Create multiple expressions to route the records the next node. For example, you might have an expression for "Marketing", "Finance", "Development" and the default expression might be for "Others". The default expression is used when all other Case expressions evaluate to false. The expression used for the Case node. Double click the cell in the table to open the Expression Editor. If you have multiple expressions, you can move them up or down in the list. The expressions are processed in order from top to bottom. See the "Using the Expression Editor" topic in the SAP HANA Developer Guide. Mappings The mappings tab shows how the input column names are mapped to output column names. If you have a large table, you can use Filter pattern to search for specific columns. See "Using the Mapping Editor" topic in the SAP HANA Developer Guide SAP SE or an SAP affiliate company. All rights reserved. 31

32 Input data Select the input data General tab by clicking Input_<n>. Table 2: General Option Name Kind The name of the input source. You can rename this source. Identifies the type of input source. For example, table, column, scalar. Table 3: Signature Option Name Type Length Scale Nullable The column name in the input source. This can be named the same as the output from the previous node. The type of data contained in the column, for example, Nvarchar, Decimal, Date, and so on. The number of characters allowed in the column. The number of digits to the right of the decimal point. This is used when the data type is a decimal. Indicates whether the column can be null. Use the Add, Remove, Up and Down buttons to edit the input fields accordingly. Table 4: Fixed Content Option Fixed Content Enable to have the input table of the node saved with the flowgraph file. Otherwise, it is placed in a separate table connected to the node. For more information, see the SAP HANA Developer Guide topic "Flowgraphs". Output data The Case node can output to one or more targets. Select the output data General tab by clicking Output_<n>. Table 5: General Option Name Kind The name of the output source. You can rename this source. Identifies the type of output source. Table 6: Signature Option Name The column name in the input source. This can be named the same as the output from the previous node SAP SE or an SAP affiliate company. All rights reserved.