SnapLogic Tutorials Document Release: October 2013 SnapLogic, Inc. 2 West 5th Ave, Fourth Floor San Mateo, California U.S.A.

Transcription

1 Document Release: October 2013 SnapLogic, Inc. 2 West 5th Ave, Fourth Floor San Mateo, California U.S.A.

2 Table of Contents SnapLogic Tutorials 1 Table of Contents 2 SnapLogic Overview 4 Glossary 4 The Tutorial Environment 4 About the Tutorial 5 Exercise 1: Working with Components 6 Reviewing the Sample Leads to Prospects Pipeline 6 Hands-on Exercise: Creating Components 8 Creating a Writer Component 11 Building the Read-Write Pipeline 12 Exercise 2: Filtering Data 16 Reviewing the California Prospects Pipeline 16 Hands-on Exercise: Creating a Filter Component 17 Duplicating Components 19 Building the Read-Write-Filter Pipeline 20 Exercise 3: Data Services 22 Reviewing the Census Feed and Qualified California Pipelines 22 Hands-On Exercise: Data Services 25 Creating a Join Component 27 Duplicating a Writer Component 28 Building a Data Service Pipeline 29 Exercise 4: Data Sorting 32 Reviewing the Filtered Qualifying California Prospects Pipeline 32 Hands-on Exercise: Sorting Data 34 Exercise 5: Utilizing Pass-Through 38 Reviewing the California Prospects Pipeline, Exercise 5 38 Hands-on Exercise: Implementing Pass-Through

3 - 3 -

4 1 SnapLogic Overview SnapLogic is the only integration platform built on modern web standards and "containerized" Snaps, allowing you to easily connect any combination of Cloud, SaaS or On-premise applications and data sources. Like the web itself, SnapLogic standardizes the access protocols, access methods and data structure for every interaction. These RESTful1 interactions allow data to be exchanged between servers in an organized way that makes it easy for the data to be processed and interpreted. This creates a resource-oriented, data-services layer that provides transformation and integration services for all kinds of integrations. Glossary Components: Components provide basic integration capabilities such as database queries, file read and write, aggregate, sort, filter, join, and others. Components are configured with specific properties. Pipelines: Pipelines can be assembled into hierarchies that can implement complex logic and transformations on data for sophisticated integrations. Repository: Resource definitions are stored locally in a SnapLogic Server s Repository. The Tutorial Environment To illustrate how to build a data services infrastructure layer with SnapLogic, this tutorial presents four simple exercises. Each exercise builds upon the previous one to demonstrate how existing Resources can be reused as well as introduces new SnapLogic Components and capabilities. Because exercises build on the previous exercise, they should be completed in order as they depend on the successful completion of earlier work. Once you complete this tutorial you will know how to use the SnapLogic Designer interface to: Configure Resources for data sources and targets Configure transformation Resources Link Components to form Pipelines Specify field mappings between Components Execute Pipelines from the Designer Reuse Components in other Pipelines This tutorial is only an introduction to the power of SnapLogic. There are more Components available that can be referenced in the Documentation section located at

5 to help you understand how each Component functions. Note: The tutorial is based on the version of the software at the time of the release date on the front page. If you are using a different version, your experience may vary. About the Tutorial This tutorial is based on a hypothetical example that requires transforming a generic list of sales leads into a list of qualified sales prospects. We begin by processing an input file consisting of leads that might be generated or purchased from some list provider (leads.csv). The file includes first name, last name, phone (home/work/cell) and address (street, city, state, and zip/postal code). The objective is to identify which of these leads are sales prospects by filtering them based on location and household income level. This requires selecting only those leads within a specific territory (California) and matching them against household income data for the lead's zip code. Household income information is available at and provides for each zip code the percentage of households that have income less than $10K, $10-$15K, $15-25K, $25K-50K and greater than $50K. For this tutorial, data for the state of California is accessed by reading a local file (CAIncomeByZip.csv). The individual tutorials consist of: Exercise 1 begins by reading the leads.csv file and writing it out to a file where the output file omits the home and cell phone numbers from the leads.csv file. This is done by creating a Pipeline with two Resources: A file Reader and a file Writer. Exercise 2 uses the Resources from Exercise 1 and creates another Pipeline that filters the leads to include only the leads that live in California. The Writer Resource is reused from Exercise 1 and a runtime parameter is set to use a new filename to preserve the previous results. (Parameters are values that can be specified at runtime, and are typically used for specifying things like file names at runtime, so Resources don't have to be edited.) Exercise 3 reads the household income data from the file CAIncomeByZip.csv and joins (looks up) that data with the filtered data from Exercise 2. It then writes to a new output file. Exercise 4 filters and sorts the joined data based on zip codes where the percentage of households that have an income greater than $50K is above a specified threshold. Exercise 5 introduces you to the concept of pass-through, where fields not needed for a specific Component are passed through for use downstream in the Pipeline. Note: The Tutorial Exercise Pipelines generate sample output files like ex_n_prospects.csv, where `n` represents the exercise number (for example, ex_1_prospects.csv is generated in Exercise 1). By default, these output files will be overwritten each time the Pipeline is executed

6 2 Exercise 1: Working with Components Exercise 1 reads in the leads.csv file and writes out a file which reorders the First Name and Last Name fields and leaves out the home and mobile phone numbers. To accomplish this, Exercise 1 constructs a Pipeline with two Components: A file Reader and a file Writer. The output file will be named: ex_1_prospects.csv. Note: The Tutorial Pipelines generate the sample output files ex_n_prospects.csv, (for example, ex_1_prospects.csv, ex_2_prospects.csv). When you run your exercises, you will overwrite these files. To avoid this, you can rename your exercise output files to be unique. Or you can simply re-run the sample pipelines to recreate the output files if you need to examine them. Your version of SnapLogic includes an example for Exercise 1 that illustrates all the steps you need to take to create the necessary Pipeline. First let s look at the Exercise 1 Components, then we ll create it ourselves. Reviewing the Sample Leads to Prospects Pipeline After installing SnapLogic successfully, connect to SnapLogic Designer using the hostname and port number selected during the installation. For example: Double-click on the Ex1_Leads_to_Prospects Pipeline within the Library. Pipelines and Components are indicated by different icons, which also indicate their status. These icons, as well as the other icons used in the client, are described in the following table. Icon Type This is a server icon and it appears on the Server List. This is a folder icon and is used to organize Pipelines and Components on the main screen. This is a Component icon. This is a Pipeline icon. Click on the Component named Leads to view its configuration. The Properties tab contains file delimiter and output file variables for this Component. Note: In this sample exercise, the delimiter and filename are specified as $?{SOM- ENAME}. The syntax (dollar sign, question mark, and braces) indicates to the Component that the values for these properties are parameters that will be set at runtime. For an example of this usage, see Hands-on Exercise 1: Creating Components

7 The Output tab defines the input.csv file fields, data types, and descriptions. The Parameters tab assigns default values to the delimiter and input file name variables. The Preview tab allows you to see the data produced by the Component. Only Components which are "Producers", those which have no input views and only output views, will have this tab present. Click on the Component named Prospects to view its configuration. The Properties tab defines the output.csv file to be created (written out) by specifying if it has a header row, what the delimiter is, and what the filename and path is. This tab also provides optional descriptive information for this Writer Component. The Input tab specifies the fields (name, datatype, and description) that are to be read from the leads.csv file. The Parameters tab assigns values to the delimiter and input file name variables. Click on the Pipeline Settings button in the upper left hand corner of the Pipeline. Click the following tabs: The General Info tab provides optional descriptive information for this Pipeline as well as buttons to access the Related Pipelines information and the Scheduler. The Parameters tab shows what parameters are used in the Pipeline, and what the default values should be. Any parameter which does not have a default value is considered a required parameter, which must be provided when executing the Pipeline. This tab presents only the information that a Pipeline user would need. The Run tab contains the Run button which starts the Pipeline processing. There is also a parameters table that lets you specify parameters at runtime that might be different from the default values indicated on the Parameters tab. Note: The purple and green blocks on the Leads and Prospects Components are output/input views. In this case, the purple block on the Leads Component is an output view for the data that the Component has read in. The green block on the Prospects Component is an input view for this Writer Component to obtain the data from the Reader. The link between Components is created by clicking on a purple output box at the bottom of one Component and dragging it to a green input box at the top of another. Click anywhere on the link to display the Field linking page. The Field Linker lets you map fields from the source to the destination. Each input field must be linked before the Pipeline can be executed. Not every output field must be used and you can use an output field more than once. Here, the input for the Writer Prospects is being linked to the output from the Reader Leads. There are two ways to make the connections. For each input field you can select an output field from the list of values. You can also click Suggest to have fields with the same names and datatypes automatically linked

8 Exercise 1: Working with Components When you click the Run button, the Pipeline processes the Reader output, maps it into the Writer input, and creates the ex_1_prospects.csv file. To confirm that the operation was successful, open the ex_1_prospects.csv file and see that the fields you mapped are the field contained in the file. Hands-on Exercise: Creating Components In this exercise, you will start to recreate the Leads to Prospects Pipeline that is used through several of the exercises 1. Double-click on the CSV Reader Component in the Foundry pane. 2. In the URI field enter: SnapLogic/User/Exercise_1/Leads where User is the username you wish to use. In this example, and for all of these exercises, we will use the name User. Substitute your name for User everywhere it is specified in these exercises if desired. Note: You can click on the Component name in the General tab to access the online help for that Component. 3. On the Properties tab, leave the properties unchanged except for the following. (For details about these properties you can look at the CSV Read Component Reference documentation.) Specify the delimiter as the parameter $?{DELIMITER}. We will specify the actual delimiter character at runtime. Specify the skip_lines value as 1. This setting allows us to skip the specified number of lines when reading an input file that might contain column headers. Since leads.csv contains a header row, we want to skip this row. Specify the input filename as $?{INPUTFILE}. We will specify the actual name of the file at runtime. 4. Click Save. Saving is optional until all the edits are made. All changes are stored in the client browser until they are saved to the Data Server. If you have made changes to a Component, but have not saved them, you can discard the changes by simply closing the Pipeline s tab. The Leads Component tab opens. Note: The URI that you specified has generated a folder hierarchy in the Data Server. For each subsequent Component, be sure to use the same path throughout these exercises so the Components show up in the same folders. Your screen should look like the following figure

9 To verify what type of Component you are working with, the Component type is always displayed in the General Info tab and in the Properties tab as shown in the preceding figure. 5. Select the General Info tab. This section contains fields for setting the Author and Description of the Component. These properties are informational only and are optional. You will not need to set any of these properties for the Tutorial exercises. 6. Select the Parameters tab. You should see the two paramaters DELIMITER and INPUT- FILE added. If not, click Add Row twice and name them. If you need help configuring the Component, open the version of the Leads Component in the "Ex1_Leads_to_Prospects" Pipeline. 7. For INPUTFILE,, specify the Default Value column as the input path and file name file://tutorial/data/leads.csv. This points to a file in the /snaplogic/data/tutorial/data directory on the SnapLogicServer. 8. For the DELIMITER parameter, enter, (comma) as the default value. 9. Click Save. Your screen should look like the following figure. At this point we have specified enough information to allow the Suggest feature to derive the remaining information. If a Component supports auto-fill, then the Suggest button will be enabled at the bottom of the Component tab. The CSV Reader Component can, given a filename, derive the number of output fields. It may also be able to - 9 -

10 Exercise 1: Working with Components recommend the correct delimiter, quote character, and even field names for the output view. 10. Click Suggest. A dialog opens with the suggested output view and a suggested quote character. 11. Accept the suggestions by clicking on the Apply All Changes button. Click the Output tab. You should see the view Output1 with all the field names correctly derived. If you choose not to use Suggest, you can create the output view manually by clicking the Add Output button and click Finish to accept the default view name (Output1). 12. The output view of the Leads Reader Component should match the fields in the leads.csv file. Click each name Field001, Field002,and so on, enter meaningful names, and specify their data type as string. All of the fields of the Leads Component should be defined as strings. Keep in mind that the fields in leads.csv contain the following types of information: 1. First Name 2. Last Name 3. Home Phone 4. Cell Phone 5. Work Phone 6. Street Address 7. City 8. State 9. Zip Code 13. A sample output definitions screen is shown in the following figure. Note that field names cannot contain spaces. Use underscore characters if you want to concatenate words such as Net Profit becomes Net_Profit

11 14. Click Validate. Validation checks that your Component is well-formed and performs checks such as all required properties are set and the necessary input/output views are defined. 15. Click Save. The Reader is now defined. Creating a Writer Component 1. Navigate to the SnapLogic > User > Exercise_1 folder in the Library pane. Note: By navigating to the intended location for a new Component, the New Component dialog will automatically include this path in your URI. 2. Double-click the CSV Writer Component in the Component Library pane. 3. In the URI field, add Prospects to the end so it reads: SnapLogic/User/Exercise_ 1/Prospects. 4. Click OK. The Prospects Writer Component icon displays, and the Prospects Component opens for editing. 5. Select the Properties tab. 6. For Header Present keep the default false (as we do not need a header row in the output file). 7. Specify the Delimiter as $?{DELIMITER}. 8. Specify the File name as the parameter $?{OUTPUTFILE}. We will specify the actual name of the file at runtime. 9. Click Save. 10. Select the General Info tab. Add your name as the author, and provide a description. These fields are optional. Click Save. 11. Select the Input tab and click the Add Input button. a. Accept the default name and click Next. b. Select Create a new (blank) view and click Finish to display the Input1 view. Now we need to tell the Writer to only write out the last name, first name, street address, city, state, zip code, and work phone number from the leads.csv file. 12. Click Add Row and enter the field name, data type, and description for Field001 as we defined it in the Reader Output field definitions. 13. Repeat this for last name, address, city, state, zip, and work phone. Here are several things you should notice: In our output result, we do not want home phone or cell phone, so those fields are not added to the input view definition

12 Exercise 1: Working with Components 14. Click Save. We have specified last name first. The order of the input fields defines the order in which they will be written out. This allows us to reorder the fields from upstream Components. Some of the field names do not match exactly with those used by the Leads Component (for example, Zip vs. Zip_Code). This is intentional and will be used to illustrate that exact matches are not necessary when linking Components together. Note: Every field you define in an input view has to be mapped from upstream views, so do not define fields you do not want. Also, for these exercises, all fields have to be specified using string as the datatype. 15. Click the Parameters tab and click Add Row. 16. In the Name column, change PARAM_001 to be the output file parameter name we introduced in the Properties tab, OUTPUTFILE. 17. In the Default Value column, specify the output file path and name, file:/- /tutorial/data/ex_1_prospects.csv. 18. Click Add Row and repeat for the DELIMITER parameter. 19. Click Save. The Writer Component is now defined. Building the Read-Write Pipeline Next we need to build a Pipeline with the Reader and Writer Components to perform a data integration. 1. Click the New Pipeline button. 2. In the URI field, add the name Leads_to_Prospects to the path Snap- Logic/User/Exercise_1/. 3. Click OK. 4. Click on the canvas. The Leads_to_Prospects Pipeline is created and is opened for editing. 5. Click the General Info tab. 6. Enter a creator and description for this Component. These values are optional. 7. Click the Pipeline tab within the Pipeline Properties. 8. Drag the Leads and Prospects icons onto the grid field (also known as the Pipeline canvas). Note: While naming the Components as we have done is optional, it is important to match the names used here with the ones used in the Parameter Map

13 9. Click the Leads output view box (purple box) and drag to link to the Prospects input view box (green box). By default, the Field Linker dialog displays automatically whenever you connect two Components. 10. Click Apply. 11. Map the fields. For each input view field, select one of the available output view fields. You can also click the B button, which will link any fields whose names and data types match. By default, Smart Link will be invoked automatically when you connect two Components together. Again, this behavior can be controlled in the Pipeline Options. As some field names do not match, link the remaining unlinked fields. 12. Click Save. 13. Click Pipeline Properties on the canvas toolbar. 14. Click the Parameters tab. We are now going to define the Pipeline parameters and how they map to the Pipeline's Components. When you drag a Component onto the Pipeline canvas, the Component's parameters are automatically mapped as Pipeline parameters. Double-click on a parameter to edit it's name or value. If you do not wish to expose a Component's parameter, you can delete it by selecting the parameter and clicking the Remove button. To add a parameter, you can click the Add button to enter a new parameter. To add a parameter, specify a Pipeline parameter name and select the Component parameter to which it maps, and a default value if any. 15. Edit each parameter and change the default parameter names. By defining the DELIM- ITER parameters as INPUT_DELIMITER and OUTPUT_DELIMITER, rather than simply DELIMITER as was defined for both Components, we can now use different delimiters if we need to. In this case, we will leave both delimiters as a comma. You can also using a single pipeline parameter and map it to multiple Component parameters. For example, if you know that all input files use comma as the delimiter, you could map a single parameter FILE_DELIMITER to both Component DELIMITER parameters. 16. In the Default Value field, enter the value to be used when running the Pipeline. 17. Click Save. 18. Click the Run tab. The Run tab contains three sub-tabs, Run, Preview Data and Runtime Information. The Parameters table in the Run sub-tab is populated with the parameters to be used in this Pipeline. Notice that the default values appear in the table. 19. If we wanted to, we could change the values right here on the Run tab and run it. 20. Click the Run button. The Pipeline will read in the data specified in the Reader Component and write it to the file specified in the Writer Component (ex_1_prospects.csv). The status of the Pipe

14 Exercise 1: Working with Components line's execution is displayed in the Runtime Information sub-tab. You can access detailed information by clicking on the row to access the Show Log menu. 21. Verify that the Pipeline performed this task by opening the output file and verifying that the First, Last,... City fields display properly. Notice that we have achieved the goal of creating a prospects output file from the original leads input file where the First Name and Last Name columns have been reordered and which contains just the work phone numbers for all the leads. Also notice that the input file contains 1,000 records. The output file also contains 1,000 records. Note: If the ex_1_prospects.csv file does not appear as expected, open the Show Log menu for the run to see if there was an error

15 - 15 -

16 3 Exercise 2: Filtering Data Exercise 2 duplicates the Components from Exercise 1 and creates another Pipeline that filters the leads to include only those in California. To accomplish this, the FilterLeads Component examines the State column in the leads.csv file for the value equal to CA. The filtered output is written to the file named ex_2_prospects.csv. Note: The Tutorial Pipelines generate the sample output files ex_n_prospects.csv, (for example, ex_1_prospects.csv, ex_2_prospects.csv). When you run your exercises, you will overwrite these files. To avoid this, you can rename your exercise output files to be unique. Or you can simply re-run the sample pipelines to recreate the output files if you need to examine them. Your version of SnapLogic includes an example for Exercise 2 that illustrates the steps to create the necessary Components and link them with the Pipeline. This exercise will show you how to create these two Components, how to duplicate and modify the Leads and Prospects Components from Exercise 1, and combine the FilterLeads Component with the Leads and Prospects Components in a new Pipeline that will filter out all the non-california leads. First let s look at the Exercise 2 components, then we ll go create it ourselves. Reviewing the California Prospects Pipeline Double-click the pipeline Ex2_CA_Prospects and it will open up in the Canvas. The Components used in this exercise will be the same ones used for Exercise 1. We will reuse the Leads Component exactly as it was configured in Exercise 1, but we will specify a runtime parameter for the Prospects Component so that it can write to a different file. Click the Pipeline Properties button for the Ex2_CA_Prospects Pipeline to see what the Pipeline includes. Click the following tabs: The General tab provides descriptive information for this Component. This data is optional. The Input tab defines the structure of the data that is provided by the upstream Components into the Pipeline. In this exercise, the Pipeline gets its data from a CSV Reader Component that reads it from a local file. That is different from data that enters a Pipeline from one of its input views. Since there is no upstream Component feeding the Pipeline, there are no input views. The Output tab defines data exiting the Pipeline. In this exercise, the Pipeline has a CSV Writer Component that writes the data to a local file. Since no data is exiting the Pipeline directly, it has no output views

17 The Parameters tab shows what parameters are used in the Pipeline, and what the default values are if any. The Run tab contains the Run button which starts the Pipeline processing. There is also a parameters table that lets you specify parameters at runtime that might be different from the default values indicated on the Parameters tab. The purple block on the Leads Component is an output view and the green block on the FilterLeads Component is an input view. The link between them is created by clicking on a Component that can generate output and dragging to another Component. In this case, you click Leads to draw a link to FilterLeads. You cannot link Prospects to FilterLeads because the Writer Component does not provide output. Now double-click the FilterLeads Component: The Properties tab defines the field to be used to filter data, its value, and the condition to be met. In this example, to find all the California leads, we need to specify that the Column be State, the value be CA, and the condition be ==. The Input tab defines the structure of the data that is provided by the upstream Components into the Filter. The Output tab defines the structure of the data from the FilterLeads Component. In this exercise, Filter output view should match its input view. Click the link between the Leads Component and the FilterLeads Component to display the Field Linker. The fields coming in from the Reader Leads are mapped here to fields in the FilterLeads input view by clicking on the fields in the Incoming fields (available to map) panel and dragging them to the From panel. Then these are matched to the FilterLeads fields by dragging Unmapped outgoing fields to the corresponding To panel such that From fields are on the same line as the fields there are going To. Click anywhere on the link between the FilterLeads Component to the Prospects Component to display the Link menu and select Field Linking to display the Field Linking page. The fields coming in from the FilterLeads are mapped here to fields in the Writer Prospects by clicking on the fields in the Incoming fields (available to map) panel and dragging them to the From panel. Then these are matched to the Writer fields by dragging Unmapped outgoing fields to the corresponding To panel such that From fields are on the same line as the fields there are going To. When you click Run, the Pipeline processes the Reader output, passes it through the Filter and into the Writer input, then creates the ex_2_prospects.csv file. You should see that the output result contains 675 records. Now let s create the Components in Exercise 2. Hands-on Exercise: Creating a Filter Component 1. Double-click the Filter Component in the Foundry pane. 2. Specify the URI as SnapLogic/User/Exercise_2/FilterLeads

18 Exercise 2: Filtering Data 3. Click OK. 4. Select the Input tab, click the Add Input button, and click Next on the View Name pop-up to start the view creation wizard. Here you will be able to choose between creating a new view, just like we did in Exercise 1, or copying an existing view from another Component. 5. Select the Copy an Existing view option and then click Next. 6. Navigate to the Exercise 1 Leads Component you created in the first tutorial. When you select a Component, the wizard will display a list of any input or output views it has. When you place your mouse over a view, you will get a brief summary of the fields in that view. Select the output view Output1 and click Finish. The output view definition will be copied from your Leads Component. 7. We do not require the Phone_h and Phone_m fields in this Component so click the "X" beside those rows to delete the fields from this view. 8. Click Save. 9. Click the Output tab. 10. Repeat the steps you performed to define the input view Input1 to create the output view Output1, but instead of copying the view from the Leads Component, just copy the input view of your Filter Component. 11. Click Save. 12. Click the Properties tab. 13. Specify the properties to be: Field: State Operator: == Value: CA View: Output1 Any Component which defines a property as being an input view field name will populate the list of values for that property with values derived from any input view it finds at the time. The list of values will be empty if no input view has been defined yet. You must define the input view first before attempting to enter a value for the property. This restriction applies to properties which reference input or output view names or field names. 14. Click Close. This creates a FilterLeads Component that will search the State column of the incoming data stream and select all records that have CA as the state name. 15. Click the General Info tab and enter an author name, and description then click Save. These values are optional

19 16. Skip the Parameters tab. Since we have not defined any parameters for this Component, do not make any changes to this tab. The Filter is now defined. Duplicating Components Duplicating a Reader Component The Leads Component created in your Exercise 1 folder is the same as what we need for a Reader in this exercise. So let s duplicate that version of the Reader and use it here. This step diverges from the sample Tutorial version of Exercise 2 in that there is no Leads Component duplicated in the sample version. In the sample version of Exercise 2, you simply drag the Leads icon from the Exercise 1 folder and drop it on the Exercise 2 Pipeline canvas. Both methods are valid. However, the instructions in this Hands-on section assume that you duplicate the Leads Component. Duplicating a Component allows you to modify a Component without affecting the pipelines that reference the original Component. You can also CTRL+Drag items within the tree to create duplicates, or use the "Copy" command from the "Edit" menu. 1. Double-click the Leads Component icon from Tutorial Exercise 1 to open it. 2. Select Save As from the Save menu button. 3. From the Save Component As... pop-up, specify the URI SnapLogic/User/Exercise_ 2/Leads. 4. Click OK. The duplicated Leads Component appears in the Library. Since the Leads Reader will function exactly as it did in Exercise 1 (that is, reading in the fields from the leads.csv file) we can leave its Properties, Basic Properties, Output, and Parameters unchanged. Duplicating a Writer Component The Prospects Component created in your Exercise 1 folder can be used as the Writer Component in this exercise. So let s copy that Writer Component. 1. Click the Prospects Component icon from Tutorial Exercise Select Save As from the Save menu button. 3. From the Save As... pop-up, specify the URI SnapLogic/User/Exercise_2/Prospects. 4. Click OK. Since the Prospects Writer will function as it did in Exercise 1, and because we have used parameters for the output file (which we can change at runtime from ex_1_prospects.csv to ex_2_prospects.csv) we can leave its Properties, Basic Properties, Output, and Parameters unchanged

20 Exercise 2: Filtering Data Building the Read-Write-Filter Pipeline Next we need to link the Leads, Prospects, and FilterLeads Components in a new Pipeline to process the data. 1. Click the New Pipeline button. 2. Specify the URI as SnapLogic/User/Exercise_2/CA_Prospects. 3. Click OK. 4. Drag the FilterLeads Component onto the Pipeline canvas. 5. Drag your Leads and Prospects Components onto the Pipeline canvas. 6. Link the Leads Reader to the FilterLeads Component. By default, the Field Linker dialog appears, and all of the fields will be mapped automatically because a matching field name was found for each To field. 7. Click OK when finished. 8. Link the FilterLeads Component to the Prospects Writer. 9. When the Link dialog appears, some of the To fields will remain unmapped because the From field names do not match exactly. Map the From fields to the remaining unmapped To fields. 10. Click OK when finished. 11. From the Pipeline, click the Parameters tab. 12. Edit the parameters and change the default names and values as follows. Specify the Reader input file, the Writer output file, and the delimiters (commas) for both Components. Pipeline Parameter Mapped to PARAM_000 Leads_1.DELIMITER, Default Value PARAM_001 Leads_1.INPUTFILE file://tutorial/data/leads.csv PARAM_002 Prospects_1.outPUTFILE file://tutorial/data/ex_2_prospects.csv PARAM_003 Prospects_1.DELIMITER, 13. Click Save. 14. Click the Run tab. The Pipeline Parameters table is now populated with the parameters to be used in this Pipeline. Notice that the default values appear in the table. If we wanted to, we could change the values right here on the Run tab and run it. 15. Click Run Pipeline Now. The Pipeline will read in the data specified in the Leads Reader Component, apply the filtering criteria to pass on the records with CA as the State value and write them to the file specified in the Prospects Writer Component (ex_2_prospects.csv)

21 16. Open ex_2_prospects.csv to verify the filtered results. You should see only the 675 leads that have California addresses

22 4 Exercise 3: Data Services Exercise 3 starts with all of the California-based leads and adds information about income levels by zip code. This exercise introduces the concept of a Data Service Pipeline. A Data Service Pipeline is one that provides a feed of data that can be used by other Pipelines or other applications. Data from these pipelines can be presented in different Representations when used by other applications. We will create a simple Data Service Pipeline that provides a feed of income data taken from the file CAIncomeByZip.csv. Next, we will introduce a Join Component that reads the income data from the Data Service feed and joins it (looks up) with the filtered data from the leads Leads and FilterLeads Components. It then writes the output to a file named ex_3_prospects.csv. This file still includes all of the records from ex_2_prospects.csv, but this time, income information is added. Note: The Tutorial Pipelines generate the sample output files ex_n_prospects.csv, (for example, ex_1_prospects.csv, ex_2_prospects.csv). When you run your exercises, you will overwrite these files. To avoid this, you can rename your exercise output files to be unique. Or you can simply re-run the sample pipelines to recreate the output files if you need to examine them. Your version of SnapLogic includes an example for Exercise 3 that illustrates all the steps you need to take to create the necessary Components and link them with the Pipeline. First let s look at the Exercise 3 components, then we ll go create them ourselves. Reviewing the Census Feed and Qualified California Pipelines From the Data Server icon, you should see the Ex3_CensusFeed and Ex3_Qual_CA_Prospects Pipelines. Double-click Census and the following tabs are available: The Properties tab defines the file delimiter and output file name. It also provides descriptive information for this Reader Component. This data is optional. The Output Views tab defines the CSV file fields, data types, and descriptions. The Parameters tab defines the values of the file delimiter and output file name. Double-click Prospects and the following tabs are available: The Properties tab defines the output CSV file to be created (written out) by specifying if it has a header row, what the delimiter is, and what the filename and path is. This tab also provides descriptive information for this Writer Component. This data is optional. The delimiter and filename are specified as $?{SOMENAME}. The dollar sign, question

23 mark, and braces indicate to the Component that the values for these properties are parameters that will be set at runtime (that is, when the Pipeline processes associated Components). The Input tab specifies the fields (name, datatype, and description) that are to be written to the output file. This matches the fields defined in the Reader Output tab. The Parameters tab defines the values of the file delimiter and output file name. Double-click Join and then click: The Properties tab defines the join expressions and the field aliasing properties. It also provides optional descriptive information for this Writer Component. The Join Expressions dialog contains the list of expressions used to join the input views. If there is more than one such expression, there is an implicit AND between them. A join expression takes the following logical form: (Input1.name == Input2.name). The Aliases dialog is used to disambiguate the output view field names. Since the input views that are being joined often have fields of the same name, in order to distinguish these fields, any ambiguous output fields must be aliased. A fully-qualified input field is given an alias, which is how this field will be represented in the output. For example, consider a Join Component with two input views: Input1: name, zip and Input2: name, company. These will be joined by name, (Input1.name = Input2.name) And an output: name, company, zip. Join needs to know which of these names should be used for the output. That is what Aliases specify. The Input tabs specify the fields (name, datatype, and description) that are to be read from the leads.csv and CAIncomeByZip.csv file. The Output tab specifies the fields (name, datatype, and description) that are to be output from the Join Component. The Parameters tab defines any parameters used by the Component. In this example the Join Component does not use any parameters. Double-click the CensusFeed Pipeline to see what the Pipeline includes. Click the following tabs: The General Info tab provides descriptive information for the Pipeline. This data is optional. The Input tab defines the structure of the data that upstream Components provide to the Pipeline. In this exercise, the Pipeline gets its data from a CSV Reader Component that reads it from a local file. That is different from data that enters a Pipeline from one of its input views. Since there is no upstream Component feeding this Pipeline, there are no input views

24 Exercise 3: Data Services The Output tab defines data exiting the Pipeline. The CensusFeed Pipeline is directing the output from the Census Component's output view Output001 to the Pipeline's output view Output1. The Parameters tab allows you to define the Pipeline parameters and map the parameters to the Pipeline's Components. The Pipeline tab shows the single Component. The Run tab contains the Run button which starts the Pipeline processing. There is also a parameters table that lets you specify parameters at runtime that might be different from the default values indicated on the Parameters tab. Double-click the Qual_CA_Prospects Pipeline to see what the Pipeline includes. Click the following tabs: The General Info tab provides descriptive information for the Pipeline. This data is optional. The Input tab, like the CensusFeed Pipeline above, defines no input views. The Output tab defines data exiting the Pipeline. In this exercise, this Pipeline has a CSV Writer Component that writes the data to a local file. Since no data is exiting the Pipeline directly, it has no output views. The Parameters tab allows you to define the Pipeline parameters and map the parameters to the Pipeline's Components. The Pipeline tab shows how the Components are linked. The Run tab contains the Run button which starts the Pipeline processing. There is also a parameters table that lets you specify parameters at runtime that might be different from the default values indicated on the Parameters tab. Click the link from Leads to FilterLeads to display the Link menu and select Field Linking to display the Field Linking page. The fields coming in from the Leads are mapped here to fields in the FilterLeads by clicking on the names in the Incoming fields (available to map) panel and dragging them to the From panel. Then these are matched to the FilterLeads fields by dragging Unmapped outgoing fields to the corresponding To panel such that From fields are on the same line as the corresponding To field names. Click the link from CensusFeed to Join and select Field Linking to display the Field Linking page. The fields coming in from CensusFeed (the Data Service Pipeline) are mapped here to fields in Join. When you click the link from FilterLeads (the Filter Component) to Join (the Join Component) and select Field Linking to display the Field Linking page, you see how the filtered fields from FilterLeads are mapped to the Join Component with the fields in Census. To understand how Join combines the FilterLeads and Census fields, left click the link from Join to Prospects. Select Field Linking to display the Field Linking page

25 Notice that for this exercise, the input view of the Prospects Component has been changed to include the new field: MoreThan50K. When you click Run, the Pipeline processes the data from the Leads and Census Components, and sends the data to the Prospects Component, which creates the ex_3_prospects.csv file. Now let s create the Components in Exercise 3. Hands-On Exercise: Data Services This exercise uses two Reader Components to read data from two different source files. We reuse the filtered data from Exercise 2, create a new Reader Component to read in data from another source file, then join the two data sources to write out the desired file. The Writer Component in this exercise is very similar to the Writer Component we created in Exercise 2, so we can duplicate it to save time. Creating a Data Service Component 1. Double-click the CSV Reader Component in the Component Library pane. 2. Name the URI SnapLogic/User/Exercise_3/Census. 3. Click OK. 4. Click the Properties tab. 5. Set the Delimiter to $?{DELIMITER}. 6. Set the File name to $?{INPUTFILE}. Rather than explicitly defining the delimiter and the input file name, we use parameters that can be defined at runtime. 7. Set Skip lines to 1. The datafile contains a header row that we want to skip. Leave the other property settings unchanged. 8. Click Save. 9. Click the Parameters tab and confirm that the DELIMITER and INPUTFILE parameters are listed. 10. Click Save. 11. Select the Output tab, click the Add Output. 12. Accept the default name and create the view based on a blank view. The output view for the Census Reader Component requires that you define the five columns in CAIncomebyZip.csv and assign meaningful names. The fields are: Zip LessThan10K MoreThan10K MoreThan25K MoreThan50K

26 Exercise 3: Data Services 13. Click Add Row five times and assign field names and data types as follows. Field Name Type Description Zip string Zip Code LessThan10K number Income less than $10K per year MoreThan10K number Income between $10K and $25K MoreThan25K number Income between $25K and $50K MoreThan50K number Income greater than $50K For the purposes of this example, the data type for Zip must be string for this exercise to work correctly. The remaining fields must be number types. 14. Click Save. The Census Reader is now defined. Next we will create the Data Service Pipeline incorporating the Census Component. 15. Click the New Pipeline button. 16. In the URI field, enter SnapLogic/User/Exercise_3/CensusFeed. 17. Click OK. The CensusFeed Pipeline tab should open. 18. Drag the Census Component onto the Pipeline canvas, naming it Census. 19. Click Save. 20. Click Pipeline Properties. 21. In the Output tab, click the Add button. 22. Name the Pipeline output view Output001 and map it to Census.Output1, then click Finish. 23. Click Save. 24. Click the Parameters tab and edit the parameter definitions to match the following: 25. Click Save. Pipeline Parameter: INPUT_DELIMITER Mapped To: Census.DELIMITER Default Value:, Pipeline Parameter: CENSUS Mapped To: Census.INPUTFILE Default Value: file://tutorial/data/caincomebyzip.csv 26. Click the Run tab. In the Run sub-tab, you can see that the Pipeline has an output view available to read from

27 27. Click the Preview button. This will run the pipeline and send the Pipeline's output to the Preview sub-tab. 28. Click the Preview Data sub-tab to see the data output by the data service. Another way to demonstrate the utility of the Data Service Pipeline is to view it directly from another browser window. Open a new browser window and enter the URI as follows: where servername:8088 is replaced with the name and port of your SnapLogic Data Server. The /feed prefix is necessary to indicate to the server that a GET to this URI will trigger Pipeline execution. The view name suffix is how you select which output is being requested. The browser's HTTP request will start the CensusFeed Pipeline and return the output view data from Output001 in a simple HTML representation. For more information on available representations, see the section entitled "Output Data Representation Formats" in the SnapLogic User Guide. You can also switch to the General tab and click on the Component hyperlink to access a list of available feeds. For your convenience, the Designer will launch these in a new window when clicked. Creating a Join Component 1. Double-click the Join Component in the Foundry pane. 2. Name the URI SnapLogic/User/Exercise_3/Join and click OK. 3. Select the Input tab and click the Add Input button. 4. Click Next to use the View Creation Wizard to copy the output view (Output1) from your FilterLeads Component from Exercise 2. You should have the fields First, Last, Phone_w, Address, City, State, and Zip defined. 5. Repeat these steps to create an Input2, copying the output view from your CensusFeed Pipeline. Input2 matches the output view of the Census Reader Resource, Output001. You should have defined Zip, LessThan10K, MoreThan10K, MoreThan25K, and MoreThan50K fields. 6. Click Save. 7. Select the Output tab and copy your Input1 input view and manually add the More- Than50K field after the view has been copied. 8. Select the Properties tab. 9. Click View/Edit to set the Aliases settings. 10. Click Add row

28 Exercise 3: Data Services 11. Enter the following values: In the Fully Qualified Field column, select Input1.Zip. In the Output Field column, select Zip. 12. Click Close. 13. Click View/Edit to set the Join Expressions settings. 14. Click Add row. 15. Enter the following values: In the Join Field 1 column, select Input1.Zip. In the Join Field 2 column, select Input2.Zip. 16. Click Close. 17. Click Save. 18. Select the General Info tab. Fill in the author and description fields as desired and click Save. These fields are optional. There are no parameters defined for this Component, so you can leave it unchanged. 19. Click Save. The Join Component is now defined. Duplicating a Writer Component The Prospects Component created in your Exercise 1 folder is close to what we need for a Writer in this exercise, so let s make a copy of that version of the Writer and modify it here. 1. Double-click on the Prospects Component from Tutorial Exercise Select Save As from the Save menu button. 3. From the Save Resource As... dialog, select the URI SnapLogic/User/Exercise_3/Prospects. 4. Click OK. 5. Select the Input tab. Because we copied this Component, we need to add a row to add the MoreThan50K field. The field type for this field must be number. Do not use the Delete button for editing as this deletes the entire Component. If you wish to delete a row from the output view, click the '''X''' button to the left of the row you wish to remove. 6. Select the Parameters tab. Again, since we copied this Component, the values here are almost correct for our needs. We just want to change the default value for OUT- PUTFILE to be file://tutorial/data/ex_3_prospects.csv. Your Parameters tab should contain the following information

29 Parameter Name DELIMITER, OUTPUTFILE Default Value file://tutorial/data/ex_3_prospects.csv 7. Click Save. The Prospects Component is now defined. Building a Data Service Pipeline Next we need to build a Pipeline with the two Reader Components linked to the Join Components, which is in turn linked to the Writer Component to perform a join. 1. Click the New Pipeline button. 2. In the URI field, enter SnapLogic/User/Exercise_3/EX3_Qual_CA_Prospects. 3. Click OK. 4. Drag the Leads Component from Exercise 1, the FilterLeads Component from Exercise 2, and the Join and Prospects Components and the CensusFeed Pipeline created in this exercise onto the Pipeline canvas. Name the Components as you drop them onto the canvas to match their URI names. 5. Click on the purple output view of Leads and drag to link to FilterLeads. The Field Linker dialog appears and all fields should be automatically linked. 6. Click OK. 7. Link FilterLeads to Join. 8. Select the Input1 input view, then click OK. Again all fields should be automatically linked in the Field Linker dialog. 9. Link CensusFeed also to Join. 10. Select the Input2 input view and click OK. All fields should be linked in the Field Linker dialog. 11. Link Join to Prospects. Auto linking was not able to match all fields, so manually map any remaining unlinked fields by selecting the appropriate field names from the list of values to the left of each field. For example, for the Work_Phone field, manually select Phone_w from the list. When complete the fields should be linked. 12. From the Pipeline, select the Parameters tab

30 Exercise 3: Data Services 13. Edit the parameter definitions as follows. Pipeline Parameter Mapped to Default Value LEADS Leads.INPUTFILE file://tutorial/data/leads.csv CENSUS CensusFeed.CENSUS file://tutorial/data/caincomebyzip.csv PROSPECTS Prospects.OUTPUTFILE file://tutorial/data/ex_3_prospects.csv INPUT1_DELIM- Leads.DELIMITER, ITER INPUT2_DELIM- CensusFeed.INPUT_ ITER DELIMITER OUTPUT_ DELIMITER Prospects.DELIMITER,, 14. Note that we have distinguished the input delimiters as INPUT1_DELIMITER, INPUT2_ DELIMITER, and OUTPUT_DELIMITER to give us the option of defining them uniquely should the need arise. For this exercise, all the delimiters are commas. 15. Click Save. 16. Select the Run tab. 17. Select the Run sub-tab. The Pipeline Parameters table is now populated with the parameters to be used in this Pipeline. Notice that the default values appear in the table. If necessary, we could change the values here. 18. Click Run button. Confirmation that the Pipeline is starting is provided on the status line in the Runtime Information sub-tab. Verify that the Pipeline performed this task by opening the output file ex_3_prospects.csv and verifying that the MoreThan50K field has been added. We have accomplished the goal of creating a Data Service Pipeline and using it as part of another Pipeline, which takes the data in the original leads input file, refining it to yield just the work phone numbers for all the leads and a new column has been populated with census data for each Zip code. (This new column shows the percentage of people in a Zip Code who make more than $50,000 a year.)

31 - 31 -

32 5 Exercise 4: Data Sorting Exercise 4 filters and sorts joined data based on zip codes where 25% or more of households have an income greater than $50,000. The results are written to a file named ex_4_prospects.csv. Reviewing the Filtered Qualifying California Prospects Pipeline In the Library, you will see the Ex4_Filtered_Qual_CA_Prospects Pipeline. Let s take a look at the Components used in this Pipeline, then we ll go create it ourselves. Double-click the Ex4_Filtered_Qual_CA_Prospects Pipeline in the Library. This will open the Pipeline in a new Canvas tab. In addition to Components used in previous exercises, you will see the FilterProspects and SortProspects Components. Double-click FilterProspects and then click: The General Info tab provides descriptive information for this Component. These fields are optional. The Properties tab is where you define the filter criteria used by this Component. In this example, we filter for zip codes where at least 25% of households have incomes greater than $50,000. To do this, we define a filter expression to check when the value of the MoreThan50K is greater than or equal to In these exercises, zip code is defined as a string datatype. The Input tab defines the fields consumed by this Component. The Output tab specifies the fields output by this Component. The Parameters tab shows parameters used by this Component as well as their optional default values. Double-click SortProspects. The General Info tab provides descriptive information for this Component. These fields are optional. The Properties tab is where you define the sorting criteria used by this Component. In this example, we will be sorting based on the following fields: MoreThan50K (primary key), Zip_Code (secondary key). The Input tab defines the fields consumed by this Component. The Output tab specifies the fields output by this Component

33 The Parameters tab shows parameters used by this Component as well as their optional default values. Click the Pipeline Properties button. The General Info tab provides descriptive information for this Pipeline. These fields are optional. The Inputtab defines the fields consumed by this Pipeline. No Input Views are used by this Pipeline. The Outputtab specifies the fields output by this Pipeline. No Output Views are used by this Pipeline. The Parameters tab shows parameters used by this Component as well as their optional default values. The Census, Leads, FilterLeads, and Join Components and used in this Pipeline and their field mappings are identical to how they are defined in Exercise 3. The following information focuses on what has been introduced in this exercise. Click the link between the Join Component and FilterProspects Component. This displays the following field mappings: Now click the link between the FilterProspects Component and SortProspects Component. You will see the following field mappings: Lastly, click the link between the SortProspects Component and Prospects Component. This will show the following field mappings: Running this Pipeline results in list of 112 records written to a file named ex_4_prospects.csv. These records exactly satisfy the criteria that the qualifying leads must live in Cal

34 Exercise 4: Data Sorting ifornia and reside in zip codes with at least 25% of households having an income greater than $50,000. Now that you ve had an opportunity to review the Components used in this exercise, let s create this Pipeline from scratch. Hands-on Exercise: Sorting Data This section explains how to build the Component and generate the output contained in the Tutorial/Exercise_4 folders. If you need additional information at any time during this hands-on portion of this exercise, feel free to open and examine each of the Tutorial versions of these Components. Duplicating a Writer Component 1. Open the Ex3_Qual_QA_Prospects Pipeline. 2. Double-click the Prospects Component. 3. Duplicate (Save As) the Component as SnapLogic/User/Exercise_4/Prospects. 4. In the new Prospects Component, click the Parameters tab. 5. Change the Default Value for the OUTPUTFILE parameter to ex_4_prospects.csv. 6. Click Save. The Prospects Component is now defined for Exercise 4. Creating a Filter Component 1. In the Foundry, type filter into the search text box. 2. Double-click the Filter Component Template. 3. Specify the relative URI as SnapLogic/User/Exercise_4/FilterProspects. 4. Click OK. 5. Click the Input tab. 6. Click Add Input to trigger the Add Input View Wizard. 7. Click Next. 8. Make sure that the Copy an existing view radio button is selected. Click Next. 9. Select the Output view of the Join Component created in Exercise 3. Click Finish. 10. Click Save. 11. Click the Output tab. 12. Click the Add Output button and copy the input view of the FilterProspects Component. 13. Click the Properties tab and click the View/Edit link for Field Filters. 14. Click Add Row

35 15. Specify the Field Filters properties to be: Field: MoreThan50K Value: 0.25 Comparison: >= View: Output1 16. Click Close. 17. Click Save. Creating a Sort Component 1. In the Foundry, type sort into the search text box. 2. Double-click the Sort Component Template and create a new Component with the relative URI to SnapLogic/User/Exercise_4/SortProspects. 3. Select the Input tab and copy the input view of the Prospects Component. 4. Select the Output tab and copy the input view of the SortProspects Component. 5. Click the Properties tab and click the View/Edit link for Sort Specifications. 6. Click Add Row. 7. Specify the Sort specs properties to be: Sort field: MoreThan50K Sort order: desc 8. Click Add Row and specify the Sort spec properties to be: Sort field: Zip_Cde Sort order: asc 9. Click Close. 10. Click Save. Building the Pipeline Next you can now link these Components together. 1. Click the New Pipeline button. 2. Specify the URI as SnapLogic/User/Exercise_4/Ex4_Filtered_Qual_CA_Prospects. 3. Click OK. 4. Drag the required Components onto the Pipeline Canvas and link the Components as follows: Census, connect to Join Leads, connect to FilterLeads

36 Exercise 4: Data Sorting FilterLeads, connect to Join Join, connect to FilterProspects FilterProspects, connect to SortProspects SortProspects, connect to Prospects Prospects 5. Define field mappings as described in "Reviewing the Filtered Qualifying California Prospects Pipeline". 6. Click the Parameters tab and modify the Pipeline Parameter and Default fields follows: Pipeline Parameter Mapped to Default Value LEADS Leads.INPUTFILE file://tutorial/data/leads.csv CENSUS Census.INPUTFILE file:/- /tutorial/data/caincomebyzip.csv PROSPECTS Prospects.OUTPUTFILE INPUT1_DELIMITER Leads.DELIMITER, INPUT2_DELIMITER Census.DELIMITER, OUTPUT_DELIMITER Prospects.DELIMITER, file://tutorial/data/ex_4_prospects.csv We have distinguished the input delimiters as `INPUT1_DELIMITER`, `INPUT2_DELIM- ITER`, and `OUTPUT_DELIMITER` to give us the option of defining them uniquely should the need arise. For this exercise, all the delimiters are commas. 7. Click Save. 8. Click the Run tab. The Pipeline Parameters table is populated with the parameters to be used by this Pipeline. Notice that default values appear in the table. If necessary, these values could be changed right here before running the Pipeline. 9. Click Run. The Pipeline will read the data specified in the Reader Census and Leads Components and write it to the file specified in the Prospects Component (ex_4_prospects.csv). You can verify the results of this exercise by examining that 112 records contained in the ex_4_prospects.csv file all have an percentage greater than 25% in the last column

37 - 37 -