PROC SQL. Beyond the Basics Using SAS Second Edition. Kirk Paul Lafler

Transcription

1 PROC SQL Beyond the Basics Using SAS Second Edition Kirk Paul Lafler

2 From PROC SQL, Second Edition. Full book available for purchase here. Contents About This Book...xv About The Author... xix Acknowledgments... xxi Chapter 1: Designing Database Tables... 1 Introduction... 2 Database Design... 2 Conceptual View... 2 Table Definitions... 3 Redundant Information... 3 Normalization... 4 Normalization Strategies... 5 Column Names and Reserved Words... 7 ANSI SQL Reserved Words... 8 SQL Code... 8 Data Integrity... 8 Referential Integrity... 9 Database Tables Used in This Book... 9 CUSTOMERS Table... 9 INVENTORY Table INVOICE Table MANUFACTURERS Table PRODUCTS Table PURCHASES Table Table Contents The Database Structure Sample Database Tables Summary... 21

3 viii Contents Chapter 2: Working with Data in PROC SQL Introduction Overview of Data Types Numeric Data Date and Time Column Definitions Character Data Missing Values and NULL Arithmetic and Missing Data SQL Keywords SQL Operators and Functions Comparison Operators Logical Operators Arithmetic Operators Character String Operators and Functions Summarizing Data Predicates Dictionary Tables Dictionary Tables and Metadata Displaying Dictionary Table Definitions Dictionary Table Column Names Accessing a Dictionary Table s Contents Summary Chapter 3: Formatting Output Introduction Formatting Output Writing a Blank Line between Each Row Displaying Row Numbers Using the FORMAT= Column Modifier to Format Output Concatenating Character Strings Inserting Text and Constants between Columns Using Scalar Expressions with Selected Columns Ordering Output by Columns Grouping Data with Summary Functions Grouping Data and Sorting

4 Contents ix Subsetting Groups with the HAVING Clause Formatting Output with the Output Delivery System ODS and Output Formats Sending Output to a SAS Data Set Converting Output to Rich Text Format Exporting Data and Output to Excel Delivering Results to the Web Summary Chapter 4: Coding PROC SQL Logic Introduction Conditional Logic SQL Code SQL Code SQL Code SQL Code CASE Expressions Simple Case Expression Searched CASE Expression Case Logic versus COALESCE Expression Assigning Labels and Grouping Data Logic and Nulls Interfacing PROC SQL with the Macro Language Exploring Macro Variables and Values Creating Multiple Macro Variables Using Automatic Macro Variables to Control Processing Building Macro Tools and Applications Creating Simple Macro Tools Cross-Referencing Columns Determining the Number of Rows in a Table Identifying Duplicate Rows in a Table Summary Chapter 5: Creating, Populating, and Deleting Tables Introduction Creating Tables

5 x Contents Creating a Table Using Column-Definition Lists Creating a Table Using the LIKE Clause Deriving a Table and Data from an Existing Table Populating Tables Adding Data to a Table with a SET Clause Adding Data to All of the Columns in a Row Adding Data to Some of the Columns in a Row Adding Data with a SELECT Query Bulk Loading Data from Microsoft Excel Integrity Constraints Defining Integrity Constraints Types of Integrity Constraints Preventing Null Values with a NOT NULL Constraint Enforcing Unique Values with a UNIQUE Constraint Validating Column Values with a CHECK Constraint Referential Integrity Constraints Establishing a Primary Key Establishing a Foreign Key Displaying Integrity Constraints Deleting Rows in a Table Deleting a Single Row in a Table Deleting More Than One Row in a Table Deleting All Rows in a Table Deleting Tables Deleting a Single Table Deleting Multiple Tables Deleting Tables That Contain Integrity Constraints Summary Chapter 6: Modifying and Updating Tables and Indexes Introduction Modifying Tables Adding New Columns Controlling the Position of Columns in a Table Changing a Column s Length Changing a Column s Format

6 Contents xi Changing a Column s Label Renaming a Column Renaming a Table Indexes Designing Indexes Cardinality Index Selectivity Defining Indexes Creating a Simple Index Creating a Composite Index Preventing Duplicate Values in an Index Modifying Columns Containing Indexes Deleting (Dropping) Indexes Updating Data in a Table Summary Chapter 7: Coding Complex Queries Introduction Introducing Complex Queries Joins Why Joins Are Important Information Retrieval Based on Relationships DATA Step Merges versus PROC SQL Joins Types of Complex Queries Demystifying Join Algorithms Influencing Joins with a Little Magic Cartesian Product Joins Inner Joins Equijoins Non-Equijoins Reflexive or Self Joins Using Table Aliases in Joins Performing Computations in Joins Joins with Three Tables Joins with More Than Three Tables

7 xii Contents Outer Joins Left Outer Joins Right Outer Joins Full Outer Joins Subqueries Alternate Approaches to Subqueries Passing a Single Value with a Subquery Passing More Than One Row with a Subquery Comparing a Set of Values Correlated Subqueries Set Operations Rules for Set Operators Set Operators and Precedence Accessing Rows from the Intersection of Two Queries Accessing Rows from the Combination of Two Queries Concatenating Rows from Two Queries Comparing Rows from Two Queries Complex Query Applications One-to-One, One-to-Many, Many-to-One, and Many-to-Many Relationships Processing First, Last, and Between Rows for BY-and Groups Determining the Number of Rows in an Input Table Identifying Tables with the Most Indexes Summary Chapter 8: Working with Views Introduction Views Windows to Your Data What Views Aren t Types of Views Creating Views Displaying a View s Contents Describing View Definitions Creating and Using Views in SAS Views and SAS Procedures Views and DATA Steps Eliminating Redundancy

8 Contents xiii Restricting Data Access Security Hiding Logic Complexities Nesting Views Updatable Views Inserting New Rows of Data Updating Existing Rows of Data Deleting Rows of Data Deleting Views Summary Chapter 9: Troubleshooting and Debugging Introduction The World of Bugs The Debugging Process Types of Problems Troubleshooting and Debugging Techniques Validating Queries with the VALIDATE Statement Documented PROC SQL Options and Statement Undocumented PROC SQL Options Macro Variables Troubleshooting and Debugging Examples Summary Chapter 10: Tuning for Performance and Efficiency Introduction Understanding Performance Tuning Sorting and Performance User-Specified Sorting (SORTPGM= System Options) Automatic Sorting Grouping and Performance Splitting Tables Indexes and Performance Reviewing CONTENTS Output and System Messages Optimizing WHERE Clause Processing with Indexes Constructing Efficient Logic Conditions Avoiding UNIONs

9 xiv Contents Summary Index From PROC SQL: Beyond the Basics Using SAS, Second Edition by Kirk Paul Lafler. Copyright 2013, SAS Institute Inc., Cary, North Carolina, USA. ALL RIGHTS RESERVED.

10 From PROC SQL, Second Edition. Full book available for purchase here. Chapter 1: Designing Database Tables Introduction... 2 Database Design... 2 Conceptual View... 2 Table Definitions... 3 Redundant Information... 3 Normalization... 4 Normalization Strategies... 5 Column Names and Reserved Words... 7 ANSI SQL Reserved Words... 8 SQL Code... 8 Data Integrity... 8 Referential Integrity... 9 Database Tables Used in This Book... 9 CUSTOMERS Table... 9 INVENTORY Table INVOICE Table MANUFACTURERS Table PRODUCTS Table PURCHASES Table Table Contents The Database Structure Sample Database Tables Summary... 21

11 2 PROC SQL: Beyond the Basics Using SAS, Second Edition Introduction The area of database design is very important in relational processes. Much has been written on this subject, including entire textbooks and thousands of technical papers. No pretenses are made about the thoroughness of this very important subject in these pages. Rather, an attempt is made to provide a quick-start introduction for those readers who are unfamiliar with the issues and techniques of basic design principles. Readers needing more information are referred to the references listed in the back of this book. As you read this chapter, the following points should be kept in mind. Database Design Activities related to good database design require the identification of end-user requirements and involve defining the structure of data values on a physical level. Database design begins with a conceptual view of what is needed. The next step, called logical design, consists of developing a formal description of database entities and relationships to satisfy user requirements. Seldom does a database consist of a single table. Consequently, tables of interrelated information are created to enable more complex and powerful operations on data. The final step, referred to as physical design, represents the process of achieving optimal performance and storage requirements of the logical database. Conceptual View The health and well-being of a database depends on its database design. A database must be in balance with all of its components (or optimized) to avoid performance and operation bottlenecks. Database design doesn t just happen and is not a process that occurs by chance. It involves planning, modeling, creating, monitoring, and adjusting to satisfy the endless assortment of user requirements without impeding resource requirements. Of central importance to database design is the process of planning. Planning is a valuable component that, when absent, causes a database to fall prey to a host of problems including poor performance and difficulty in operation. Database design consists of three distinct phases, as illustrated in Figure 1.1.

12 Chapter 1: Designing Database Tables 3 Figure 1.1: Three Distinct Phases of Database Design Table Definitions PROC SQL uses a model of data that is conceptually stored as multisets rather than as physical files. A physical file consists of one or more records ordered sequentially or some other way. Programming languages such as COBOL and FORTRAN evolved to process files of this type by performing operations one record at a time. These languages were generally designed and used to mimic the way people process paper forms. PROC SQL was designed to work with multisets of data. Multisets have no order, and members of a multiset are of the same type using a data structure known as a table. For classification purposes, a table is a base table consisting of zero or more rows and one or more columns, or a table is a virtual table (called a view), which can be used the same way that a table can be used (see Chapter 8, Working with Views ). Redundant Information One of the rules of good database design requires that data not be redundant or duplicated in the same database. The rationale for this conclusion originates from the belief that if data appears more than once in a database, then there is reason to believe that one of the pieces of data is likely to be in error. Furthermore, redundancy often leads to the following: Inconsistencies, because errors are more likely to result when facts are repeated. Update anomalies where the insertion, modification, or deletion of data may result in inconsistencies. Another thing to watch for is the appearance of too many columns containing NULL values. When this occurs, the database is probably not designed properly. To alleviate potential table design

13 4 PROC SQL: Beyond the Basics Using SAS, Second Edition issues, a process referred to as normalizing is performed. When properly done, this ensures the complete absence of redundant information in a table. Normalization The development of an optimal database design is an important element in the life cycle of a database. Not only is it critical for achieving maximum performance and flexibility while working with tables and data, it is essential to the organization of data by reducing or minimizing redundancy in one or more database tables. The process of table design is frequently referred to by database developers and administrators as normalization. The normalization process is used for reducing redundancy in a database by converting complex data structures into simple data structures. It is carried out for the following reasons: To organize the data to save space and to eliminate any duplication or repetition of data. To enable simple retrieval of data to satisfy query and report requests. To simplify data manipulation requests such as data insertions, updates, and deletions. To reduce the impact associated with reorganizing or restructuring data as new application requirements arise. The normalization process attempts to simplify the relationship between columns in a database by splitting larger multicolumn tables into two or more smaller tables containing fewer columns. The rationale for doing this is contained in a set of data design guidelines called normal forms. The guidelines provide designers with a set of rules for converting one or two large database tables containing numerous columns into a normalized database consisting of multiple tables and only those columns that should be included in each table. The normalization process consists of multiple steps with each succeeding step subscribing to the rules of the previous steps. Normalization helps to ensure that a database does not contain redundant information in two or more of its tables. In an application, normalization prevents the destruction of data or the creation of incorrect data in a database. What this means is that information of fact is represented only once in a database, and any possibility of it appearing more than once is not, or should not be, allowed. As database designers and analysts proceed through the normalization process, many are not satisfied unless a database design is carried out to at least third normal form (3NF). Joe Celko in his popular book SQL for Smarties: Advanced SQL Programming (Morgan Kaufman, 1999), describes 3NF this way: Databases are considered to be in 3NF when a column is dependent on the key, the whole key, and nothing but the key. While the normalization guidelines are extremely useful, some database purists actually go to great lengths to remove any and all table redundancies even at the expense of performance. This is in direct contrast to other database experts who follow the guidelines less rigidly in an attempt to improve the performance of a database by only going as far as third normal form (or 3NF). Whatever your preference, you should keep this thought in mind as you normalize database tables. A fully normalized database often requires a greater number of joins and can adversely affect the

14 Chapter 1: Designing Database Tables 5 speed of queries. Celko mentions that the process of joining multiple tables in a fully normalized database is costly, specifically affecting processing time and computer resources. Normalization Strategies After transforming entities and attributes from the conceptual design into a logical design, the tables and columns are created. This is when a process known as normalization occurs. Normalization refers to the process of making your database tables subscribe to certain rules. Many, if not most, database designers are satisfied when third normal form (3NF) is achieved and, for the objectives of this book, I will stop at 3NF, too. To help explain the various normalization steps, an example scenario follows. First Normal Form (1NF) First normal form (1NF) involves the elimination of data redundancy or repeating information from a table. A table is considered to be in first normal form when all of its columns describe the table completely and when each column in a row has only one value. A table satisfies 1NF when each column in a row has a single value and no repeating group information. Essentially, every table meets 1NF as long as an array, list, or other structure has not been defined. The following table illustrates a table satisfying the 1NF rule because it has only one value at each row-and-column intersection. The table is in ascending order by CUSTNUM and consists of customers and the purchases they made at an office supply store. Table 1.1: First Normal Form (1NF) Table CUSTNUM CUSTNAME CUSTCITY ITEM UNITS UNITCOST MANUCITY 1 Smith San Diego Chair 1 $ San Diego 1 Smith San Diego Pens 12 $0.89 Los Angeles 1 Smith San Diego Paper 4 $6.95 Washington 1 Smithe San Diego Stapler 1 $8.95 Los Angeles 7 Lafler Spring Valley Mouse Pad 1 $11.79 San Diego 7 Loffler Spring Valley Pens 24 $1.59 Los Angeles 13 Thompson Miami Markers. $0.99 Los Angeles Second Normal Form (2NF) Second normal form (2NF) addresses the relationships between sets of data. A table is said to be in second normal form when all the requirements of 1NF are met and a foreign key is used to link any data in one table which has relevance to another table. The very nature of leaving a table in first normal form (1NF) may present problems due to the repetition of some information in the table. One noticeable problem is that Table 1.1 has repetitive information in it. Another problem is that there are misspellings in the customer name. Although repeating information may be permissible with hierarchical file structures and other legacy type file structures, it does pose a potential data consistency problem as it relates to relational data. To describe how data consistency problems can occur, let s say that a customer takes a new job and moves to a new city. In changing the customer s city to the new location, it would be very easy to miss one or more occurrences of the customer s city resulting in a customer residing incorrectly in

15 6 PROC SQL: Beyond the Basics Using SAS, Second Edition two different cities. Assuming that our table is only meant to track one unique customer per city, this would definitely be a data consistency problem. Essentially, second normal form (2NF) is important because it says that every non-key column must depend on the entire primary key. Tables that subscribe to 2NF prevent the need to make changes in more than one place. What this means in normalization terms is that tables in 2NF have no partial key dependencies. As a result, our database that consists of a single table that satisfies 1NF will need to be split into two separate tables in order to subscribe to the 2NF rule. Each table would contain the CUSTNUM column to connect the two tables. Unlike the single table in 1NF, the tables in 2NF allow a customer s city to be easily changed whenever they move to another city because the CUSTCITY column only appears once. The tables in 2NF would be constructed as follows. Table 1.2: CUSTOMERS Table CUSTNUM CUSTNAME CUSTCITY 1 Smith San Diego 1 Smithe San Diego 7 Lafler Spring Valley 13 Thompson Miami Table 1.3: PURCHASES Table CUSTNUM ITEM UNITS UNITCOST MANUCITY 1 Chair 1 $ San Diego 1 Pens 12 $0.89 Los Angeles 1 Paper 4 $6.95 Washington 1 Stapler 1 $8.95 Los Angeles 7 Mouse Pad 1 $11.79 San Diego 7 Pens 24 $1.59 Los Angeles 13 Markers. $0.99 Los Angeles Third Normal Form (3NF) Referring to the two tables constructed according to the rules of 2NF, you may have noticed that the PURCHASES table contains a column called MANUCITY. The MANUCITY column stores the city where the product manufacturer is headquartered. Keeping this column in the PURCHASES table violates the third normal form (3NF) because MANUCITY does not provide factual information about the primary key column (CUSTNUM) in the PURCHASES table. Consequently, tables are considered to be in third normal form (3NF) when each column is dependent on the key, the whole key, and nothing but the key. The tables in 3NF are constructed so the MANUCITY column would be in a table of its own as follows.

16 Chapter 1: Designing Database Tables 7 Table 1.4: CUSTOMERS Table CUSTNUM CUSTNAME CUSTCITY 1 Smith San Diego 1 Smithe San Diego 7 Lafler Spring Valley 13 Thompson Miami Table 1.5: PURCHASES Table CUSTNUM ITEM UNITS UNITCOST 1 Chair 1 $ Pens 12 $ Paper 4 $ Stapler 1 $ Mouse Pad 1 $ Pens 24 $ Markers. $0.99 Table 1.6: MANUFACTURERS Table MANUNUM MANUCITY 101 San Diego 112 San Diego 210 Los Angeles 212 Los Angeles 213 Los Angeles 214 Los Angeles 401 Washington Beyond Third Normal Form In general, database designers are satisfied when their database tables subscribe to the rules in 3NF. But, it is not uncommon for others to normalize their database tables to fourth normal form (4NF) where independent one-to-many relationships between primary key and non-key columns are forbidden. Some database purists will even normalize to fifth normal form (5NF) where tables are split into the smallest pieces of information in an attempt to eliminate any and all table redundancies. Although constructing tables in 5NF may provide the greatest level of database integrity, it is neither practical nor desired by most database practitioners. There is no absolute right or wrong reason for database designers to normalize beyond 3NF as long as they have considered all the performance issues that may arise by doing so. A common problem that occurs when database tables are normalized beyond 3NF is that a large number of small tables are generated. In these cases, an increase in time and computer resources frequently occurs because small tables must first be joined before a query, report, or statistic can be produced. Column Names and Reserved Words According to the American National Standards Institute (ANSI), SQL is the standard language used with relational database management systems. The ANSI Standard reserves a number of SQL keywords from being used as column names. The SAS SQL implementation is not as rigid, but

17 8 PROC SQL: Beyond the Basics Using SAS, Second Edition users should be aware of what reserved words exist to prevent unexpected and unintended results during SQL processing. Column names should conform to proper SAS naming conventions (as described in the SAS Language Reference), and they should not conflict with certain reserved words found in the SQL language. The following list identifies the reserved words found in the ANSI SQL standard. ANSI SQL Reserved Words AS INNER OUTER CASE INTERSECT RIGHT EXCEPT JOIN UNION FROM LEFT UPPER FULL LOWER USER GROUP ON WHEN HAVING ORDER WHERE You probably will not encounter too many conflicts between a column name and an SQL reserved word, but when you do you will need to follow a few simple rules to prevent processing errors from occurring. As was stated earlier, although PROC SQL s naming conventions are not as rigid as other vendor s implementations, care should still be exercised, in particular when PROC SQL code is transferred to other database environments expecting it to run error free. If a column name in an existing table conflicts with a reserved word, you have three options at your disposal: 1. Physically rename the column name in the table, as well as any references to the column. 2. Use the RENAME= data set option to rename the desired column in the current query. 3. Specify the PROC SQL option DQUOTE=ANSI, and surround the column name (reserved word) in double quotes, as illustrated below. SQL Code PROC SQL DQUOTE=ANSI; SELECT * FROM RESERVED_WORDS WHERE "WHERE"= EXAMPLE ; QUIT; Data Integrity Webster s New World Dictionary defines integrity as the quality or state of being complete; perfect condition; reliable; soundness. Data integrity is a critical element that every organization must promote and strive for. It is imperative that the data tables in a database environment be reliable, free of errors, and sound in every conceivable way. The existence of data errors, missing information, broken links, and other related problems in one or more tables can impact decisionmaking and information reporting activities resulting in a loss of confidence among users.

18 Chapter 1: Designing Database Tables 9 Applying a set of rules to the database structure and content can ensure the integrity of data resources. These rules consist of table and column constraints, and will be discussed in detail in Chapter 5, Creating, Populating, and Deleting Tables. Referential Integrity Referential integrity refers to the way in which database tables handle update and delete requests. Database tables frequently have a primary key where one or more columns have a unique value by which rows in a table can be identified and selected. Other tables may have one or more columns called a foreign key that are used to connect to some other table through its value. Database designers frequently apply rules to database tables to control what happens when a primary key value changes and its effect on one or more foreign key values in other tables. These referential integrity rules apply restrictions on the data that may be updated or deleted in tables. Referential integrity ensures that rows in one table have corresponding rows in another table. This prevents lost linkages between data elements in one table and those of another enabling the integrity of data to always be maintained. Using the 3NF tables defined earlier, a foreign key called CUSTNUM can be defined in the PURCHASES table that corresponds to the primary key CUSTNUM column in the CUSTOMERS table. Users are referred to Chapter 5, Creating, Populating, and Deleting Tables for more details on assigning referential integrity constraints. Database Tables Used in This Book This section describes a database or library of tables that is used by an imaginary computer hardware and software wholesaler. The library consists of six tables: Customers, Inventory, Invoice, Manufacturers, Products, and Purchases. The examples used throughout this book are based on this library (database) of tables and are described and displayed below. An alphabetical description of each table used throughout this book appears below. CUSTOMERS Table The CUSTOMERS table contains customers that have purchased computer hardware and software products from a manufacturer. Each customer is uniquely identified with a customer number. A description of each column in the Customers table follows. Table 1.7: Description of Columns in the Customers Table CUSTNUM CUSTNAME CUSTCITY Unique number identifying the customer. Name of customer. City where customer is located.

19 10 PROC SQL: Beyond the Basics Using SAS, Second Edition INVENTORY Table The INVENTORY table contains customer inventory information consisting of computer hardware and software products. The Inventory table contains no historical data. As inventories are replenished, the old quantity is overwritten with the new quantity. A description of each column in the Inventory table follows. Table 1.8: Description of Columns in the Inventory Table PRODNUM MANUNUM INVENQTY ORDDATE INVENCST Unique number identifying product. Unique number identifying the manufacturer. Number of units of product in stock. Date product was last ordered. Cost of inventory in customer s stock room. INVOICE Table The INVOICE table contains information about customers who purchased products. Each invoice is uniquely identified with an invoice number. A description of each column in the Invoice table follows. Table 1.9: Description of Columns in the Invoice Table INVNUM MANUNUM CUSTNUM PRODNUM INVQTY INVPRICE Unique number identifying the invoice. Unique number identifying the manufacturer. Customer number. Product number. Number of units sold. Unit price. MANUFACTURERS Table The MANUFACTURERS table contains companies who make computer hardware and software products. Two companies cannot have the same name. No historical data is kept in this table. If a company is sold or stops making computer hardware or software, then the manufacturer is dropped from the table. In the event that a manufacturer has an address change, the old address is overwritten with the new address. A description of each column in the Manufacturers table follows.

20 Chapter 1: Designing Database Tables 11 Table 1.10: Description of Columns in the Manufacturers Table MANUNUM MANUNAME MANUCITY MANUSTAT Unique number identifying the manufacturer. Name of manufacturer. City where manufacturer is located. State where manufacturer is located. PRODUCTS Table The PRODUCTS table contains computer hardware and software products offered for sale by the manufacturer. Each product is uniquely identified with a product number. A description of each column in the Products table follows. Table 1.11: Description of Columns in the Products Table PRODNUM PRODNAME MANUNUM PRODTYPE PRODCOST Unique number identifying the product. Name of product. Unique number identifying the manufacturer. Type of product. Cost of product. PURCHASES Table The PURCHASES table contains computer hardware and software products purchased by customers. Each product is uniquely identified with a product number. A description of each column in the Purchases table follows. Table 1.12: Description of Columns in the Purchases Table CUSTNUM ITEM UNITS UNITCOST Unique number identifying the customer. Name of product. Number of items purchased by customer. Cost of product.

21 12 PROC SQL: Beyond the Basics Using SAS, Second Edition Table Contents An alphabetical list of tables, variables, and attributes for each table is displayed below. Output 1.1: Customers CONTENTS Output Output 1.2: Inventory CONTENTS Output Output 1.3: Invoice CONTENTS Output

22 Chapter 1: Designing Database Tables 13 Output 1.4: Manufacturers CONTENTS Output Output 1.5: Products CONTENTS Output Output 1.6: Purchases CONTENTS Output

23 14 PROC SQL: Beyond the Basics Using SAS, Second Edition The Database Structure The logical relationship between each table, and the columns common to each, appear below. Figure 1.2. Logical Database Structure Sample Database Tables The following tables: Customers, Inventory, Manufacturers, Products, Invoice, and Purchases represent a relational database that will be illustrated in the examples in this book. These tables are small enough to follow easily, but complex enough to illustrate the power of SQL. The data contained in each table appears below.

24 Table 1.13: CUSTOMERS Table Chapter 1: Designing Database Tables 15

25 16 PROC SQL: Beyond the Basics Using SAS, Second Edition Table 1.14: INVENTORY Table Table 1.15: INVOICE Table

26 Chapter 1: Designing Database Tables 17 Table 1.16: MANUFACTURERS Table Table 1.17: PRODUCTS Table

27 18 PROC SQL: Beyond the Basics Using SAS, Second Edition Table 1.18: PURCHASES Table

28 Chapter 1: Designing Database Tables 19

29 20 PROC SQL: Beyond the Basics Using SAS, Second Edition

30 Chapter 1: Designing Database Tables 21 Summary 1. Good database design often improves the relative ease by which tables can be created and populated in a relational database and can be implemented into any database (see the Conceptual View section). 2. SQL was designed to work with sets of data and accesses a data structure known as a table or a virtual table, known as a view (see the Table Definitions section). 3. Achieving optimal design of a database means that the database contains little or no redundant information in two or more of its tables. This means that good database design calls for little or no replication of data (see the Redundant Information section). 4. Good database design avoids data redundancy, update anomalies, costly or inefficient processing, coding complexities, complex logical relationships, long application development times, and/or excessive storage requirements (see the Normalization section). 5. Design decisions made in one phase may involve making one or more tradeoffs in another phase (see the Normalization section). 6. A database in third normal form (3NF) is defined as a column that is dependent on the key, the whole key, and nothing but the key (see the Normalization section). From PROC SQL: Beyond the Basics Using SAS, Second Edition by Kirk Paul Lafler. Copyright 2013, SAS Institute Inc., Cary, North Carolina, USA. ALL RIGHTS RESERVED. From PROC SQL, Second Edition. Full book available for purchase here.

31 Index A ADD clause addition (+) operator ad-hoc queries 280 _AGGR option 342 aggregate functions creating macro variables with specifying ALL keyword 268 ALTER TABLE statement , 207, , , Ambiguous reference, column error 347 AND operator ANSI (American National Standards Institute) 7 8 APPEND procedure 276 arithmetic data arithmetic operators AS keyword _ASGN option 342 asterisk (*) wildcard 59 automatic macro variables, controlling processing with automatic sorting 353 AVG function 59 B BEST option 353 BETWEEN predicate B-tree 223 bugs 324 bulk loading data from Microsoft Excel Burlew, Michele M. Output Delivery System: The Basics and Beyond 114 SAS Macro Programming Guide Made Easy, Second Edition 148 BY statement 144 C calculated column view 298 calculated columns 299 cardinality 225 Carpenter, Art Carpenter's Complete Guide to the SAS Macro Language, Second Edition 148 Carpenter's Complete Guide to the SAS Macro Language, Second Edition (Carpenter) 148 Cartesian product joins 242 Cartesian Product query 237 CASE expressions about assigning labels and grouping data case logic versus COALESCE expression logic and nulls searched simple case logic, COALESCE expression versus CAT function 41 CATALOGS dictionary table 73, 79 CATS function 41 CATX function 100 Celko, Joe SQL for Smarties: Advanced SQL Programming 4, 47, 48, 298 change control, preserving 202 character data 28 character strings concatenating operators and functions characters, aligning 43 44

32 368 Index CHECK constraint CHECK_CONSTRAINTS dictionary table 73 COALESCE expression, case logic versus COALESCE function 49, Code Complete: A Practical Handbook of Software Construction, Second Edition (McConnell) 134 coding See complex queries, coding coding logic about 123 CASE expressions conditional logic interfacing PROC SQL with Macro language column aliases, creating column constraints 192 column names 7 8, column-definition lists, creating tables using columns adding data to in rows adding to tables calculated 299 changing format of changing label 219 changing length of controlling position of in tables cross-referencing derived 299 inserting text and constants between modifying columns containing indexes 229 ordering output by renaming using scalar expressions with selected COLUMNS dictionary table 73, Comma Separated Value (CSV) file 116 comparison operators The Complete Guide to Using SAS Indexes (Raithel) 224 complex comparisons, with searched CASE expressions complex queries, coding about Cartesian product joins 242 complex query applications inner joins 237, joins outer joins 238, set operations 238, subqueries 238, types of complex query applications about 280 determining number of rows in input tables identifying tables with most indexes many-to-many relationships 237, many-to-one relationships 237, one-to-many relationships 237, one-to-one relationships 237, processing first, last, and between rows for BY-and groups composite indexes computations, performing in joins concatenating character strings rows from two queries concatenating strings concatenation character string operator ( ) conceptual database design 2 3 conditional logic constants, inserting between columns CONSTRAINT_COLUMN_USAGE dictionary table 73 CONSTRAINT_TABLE_USAGE dictionary table 73 CONTENTS procedure , 218, , correlated subqueries 238, 261, COUNT function 59, 60,

33 Index 369 CREATE TABLE statement 74 75, , 164, , , , , CREATE VIEW statement , 297, Cross Join query 237 cross-referencing columns CSS function 59 CSV (Comma Separated Value) file 116 custom queries 280 CUSTOMERS table 9, 12, 15 customized lists creating with searched CASE expressions creating with simple CASE expressions CV function 59 D DASD (Direct Access Storage Device) 352 data about 24 accessing dictionary table's contents adding to columns in rows adding to tables with SET clause adding with SELECT query arithmetic arithmetic operators bulk loading from Microsoft Excel character 28 character string operators and functions comparison operators date and time column definitions deleting rows of 320 dictionary table column names dictionary tables displaying dictionary table definitions exporting to Microsoft Excel grouping , grouping with summary functions logical operators metadata and dictionary tables missing missing values 28 29, NULL values 3, 28 29, numeric predicates sorting SQL keywords SQL operators and functions summarizing types 24 updating in tables updating rows of Data Definition Language (DDL) statements 210 data integrity 8 9 data problems 326 data security DATA step 25, 42 43, 113, 216, , 236, database design about 2 column names 7 8 conceptual view of 2 3 data integrity 8 9 database tables used in this book 9 20 normalization 4 7 redundant information 3 4 reserved words 7 8 table definitions 3 database structure 14 database tables See tables database-enforced constraints See integrity constraints DATAITEMS dictionary table 73 DATASETS procedure date column definitions DDL (Data Definition Language) statements 210

34 370 Index debugging See also troubleshooting about 323 bugs 324 examples with macro variables process of techniques for defining indexes DELETE statement deleting indexes rows in tables rows of data 320 tables views 321 derived columns 299 DESCRIBE TABLE statement 74 75, DESCRIBE VIEW statement DESTINATIONS dictionary table 73 _DFR option 342 DICTIONARIES dictionary table 73, dictionary tables about 72 accessing content column names displaying definitions metadata and Direct Access Storage Device (DASD) 352 DISTINCT keyword division by zero, preventing with simple CASE expressions division (/) operator DROP INDEX statement DROP TABLE statement , 222 DROP VIEW statement 321 dropping indexes duplicate values, finding DUPS E Effective Methods for Software Testing (Perry) 324 ENGINES dictionary table 73 equals (=) operator 38 40, 243, 245, equijoins 237, ERRORSTOP/NOERRORSTOP option EXCEPT operator 238, EXEC/NOEXEC option EXISTS predicate 71 exponent (**) operator exporting data and output to Microsoft Excel EXTFILES dictionary table 73, 82 F feature creep problems 327 FEEDBACK option fifth normal form (5NF) 7 FILENAME statement 82 FILTERS dictionary table 73 first normal form (1NF) 5 foreign key 9, FORMAT= column modifier FORMAT procedure FORMATS dictionary table 73 formatting output about 92 concatenating character strings converting output to rich text format delivering results to Web displaying row numbers exporting data and output to Excel FORMAT= column modifier grouping data and sorting grouping data with summary functions inserting text and constants between columns

35 Index 371 ordering output by columns with Output Delivery System (ODS) scalar expressions sending output to SAS data sets subsetting groups with HAVING clause writing blank lines between rows fourth normal form (4NF) 7 FREQ function 59 FROM clause 78, full outer joins functions See SQL operators and functions FUNCTIONS dictionary table 73 G GOPTIONS dictionary table 73 greater than operator (>) 35 36, 246 GROUP BY clause 58, , , 354 grouped view 298 grouping , , 354 groups, subsetting with HAVING clause Gupta, Sunil Kumar Quick Results with the Output Delivery System 114 H hash join algorithm 239 HAVING clause 58, , , , , Haworth, Lauren Output Delivery System: The Basics and Beyond 114 HOST option 353 Hsieh, Yuan The Science of Debugging 324 HTML statement hybrid view 298 HyperText Markup Language (HTML) 117 I IMPORT procedure 186, 190 IN predicate INDEX function 45 index join algorithm 239 indexes about 210, composite creating defining deleting designing identifying tables with most modifying columns containing 229 optimizing WHERE clause processing with performance and preventing duplicate values in 229 selectivity in INDEXES dictionary table 73, 83 INFOMAPS dictionary table 73 information collection based on relationships as step in debugging process 325 inner joins about 237, equijoins 237, with more than three tables non-equijoins 237, performing computations in reflexive joins 237, self joins 237, with three tables using table aliases in INOBS= option INSERT INTO statement , , integrity 9 integrity constraints about 192

36 372 Index defining 192 deleting tables containing displaying preventing null values with NOT NULL constraint referential types of 192 INTERSECT operator 238, INTO clause , , , INVENTORY table 10, 12, 16 INVOICE table 10, 12, 16 IS MISSING predicate IS NOT NULL predicate 67 IS NULL predicate J JOIN construct joined view 298 joins about 235 algorithms 239 importance of influencing with more than three tables performing computations in with three tables using table aliases in L LABEL= option labels, assigning LEFT function left outer joins LENGTH function 42 LENGTH= modifier LENGTH statement 25, 166, 168, 216, %LET, creating macro variables with LIBNAME statement 186 LIBNAMES dictionary table 73 LIKE clause, creating tables using LIKE predicate logic conditional nulls and logic complexities, hiding logic problems 326 logical design 2 3 logical operators LOOPS= option LOWCASE function M macro applications, building Macro language, interfacing PROC SQL with macro tools, building macro variables and values about 149 controlling processing with controlling selection and population of with WHERE clause creating from table row columns creating lists of values in creating multiple creating with aggregate functions creating with %LET troubleshooting and debugging with MACROS dictionary table 73, MAGIC option MANUFACTURERS table 10 11, 13, 17 many-to-many relationships 237, many-to-one relationships 237, MAX function 59 McConnell, Steve Code Complete: A Practical Handbook of Software Construction, Second Edition 134 MEAN function 59 MEMBERS dictionary table 73, metadata, dictionary tables and 72 74

37 Index 373 _METHOD option Microsoft Excel bulk loading from exporting data and output to MIN function 59, 60 missing data missing values 28 29, MODIFY clause , MONOTONIC() function MSGLEVEL= multiplication (*) operator N N function 59 nested loop join algorithm 239 nested view 298 nesting , NMISS function 59 NOBS NOFEEDBACK option nonconsecutive values, selecting non-equijoins 237, normalization 4 7 NOT IS NULL predicate 67 NOT NULL constraint NOT operator 36 38, 68 NULL values 3, 28 29, nulls, logic and NUMBER option numeric data O ODS (Output Delivery System) ODS statement : Statement is not valid or it is used out of proper order error one-to-many relationships 237, one-to-one relationships 237, operators, combining with functions See also SQL operators and functions OPTIONS dictionary table 73, OR operator 36 38, 64 ORDER BY clause , , outer joins about 238, 255 full outer joins left outer joins right outer joins OUTER UNION operator 238, OUTOBS= option 341 output, formatting See formatting output Output Delivery System (ODS) Output Delivery System: The Basics and Beyond (Haworth, Zender, and Burlew) 114 OUTPUT statement P patterns finding in strings finding occurrences of with INDEX function 45 percent sign (%) performance See tuning process period (.) 29 Perry, William E. Effective Methods for Software Testing 324 phonetic matching physical design 2 3 _PJD option 342 populating tables precedence, set operators and 272 predicates about finding patterns in strings selecting nonconsecutive values selecting ranges of values testing for existence of values 71 testing for NULL or MISSING values primary key 9,

38 374 Index PRINT procedure 52, 92, 93, problem assessment and classification, as step in debugging process 325 problem identification, as step in debugging process 325 problem resolution, as step in debugging process 325 PROC step 113 PRODTYPE macro variable 149 production-oriented queries 280 PRODUCTS table 11, 13, 17 PROMPT option PROMPTS dictionary table 74 PROMPTSXML dictionary table 74 propagation of nulls 29 PRT function 59 PURCHASES table 11, 13, %PUT statement 330, Q queries See also complex queries, coding accessing rows from combination of two accessing rows from intersection of two ad-hoc 280 comparing rows from two concatenating rows from two Cross Join 237 custom 280 production-oriented 280 validating with VALIDATE statement Quick Results with the Output Delivery System (Gupta) 114 QUIT statement 93 R Raithel, Michael The Complete Guide to Using SAS Indexes 224 RANGE function 59 read-only view 298 redundancy, eliminating 307 redundant information, in database design 3 4 referential integrity 9, REFERENTIAL_CONSTRAINTS dictionary table 74 reflexive joins 237, relationships, information retrieval based on REMEMBER dictionary table 74 renaming columns tables requirements problems 327 reserved words 7 8 RESET statement 93, rich text format, converting output to RIGHT function right outer joins rows accessing from combination of two queries accessing from intersection of two queries adding data to columns in comparing from two queries concatenating from two queries deleting in tables deleting rows of data 320 determining number of in tables displaying numbers identifying duplicates in tables passing more than one row with subqueries producing numbers updating rows of data , writing blank lines between _RSLV option 343

39 Index 375 S samples of database tables SAS data sets, sending output to SAS Language Reference: Dictionary 8, 27 SAS Macro Language: Reference (SAS Institute Inc.) 148 SAS Macro Programming Guide Made Easy, Second Edition (Burlew) 148 SAS Procedures Guide 75 SASHELP views scalar expressions, using with selected columns SCAN function 43 The Science of Debugging (Telles and Hsieh) 324 searched CASE expression about 137 complex comparisons with creating customized lists with in SELECT clause second normal form (2NF) 5 6 2NF (second normal form) 5 6 security, data SELECT clause CREATE TABLE statement , searched CASE expressions in simple CASE expressions in , SELECT query , , 222 SELECT statement FROM clause 78 creating macro variables with aggregate functions creating views , finding first nonmissing value 49 grouping data with summary functions with joins 235 MONOTONIC() function SQLOOPS macro variable 344 SQLRC macro variable 345 summarizing data 58 updating rows of data using scalar expressions with selected columns validating queries wildcard characters in 279 selectivity, of indexes self joins 237, SET clause , set operation view 298 set operations about 238, 271 accessing rows from combination of two queries accessing rows from intersection of two queries comparing rows from two queries concatenating rows from two queries precedence and 272 rules for set operators : Expecting an AS error , 349 single-table view 298 solution complexity problems 327 sorting automatic 353 data performance and user-specified 353 sort-merge join algorithm 239 SORTPGM= system option 353 SOUNDEX function sounds-like operator (=*) splitting tables 354 SQL for Smarties: Advanced SQL Programming (Celko) 4, 47, 48, 298 SQL keywords SQL language 7 8 SQL operators and functions about 35 aggregate functions , arithmetic operators character string combining functions with operators 42 43