DB2 Web Query Tips and Techniques

Transcription

1 DB2 Web Query Tips and Techniques Presented by Jarek Miszczyk ISV Solutions Enablement IBM Rochester, MN 8 Copyright IBM Corporation, All Rights Reserved. This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein. Agenda General Usage Tips Security Tips Performance Tips 2 1

2 General Usage Tips Starting DB2 Web Query Automatically In CL program QSTRUP add the following line SBMJOB CMD(QWEBQRY76/STRWEBQRY) USER(QWEBQRYADM) Notes TCP must be active before the STRWEBQRY command will execute successfully Since QSTRUP runs with user profile QPGMR you must give QPGMR object *USE authority to QWEBQRYADM user profile with the following command EDTOBJAUT OBJ(QWEBQRYADM) OBJTYPE(*USRPRF) 3 General Usage Tips Create Domains with exactly eight characters Security is assigned by 8 character HREF which relates to the domain If duplicate, random HREF is generated If less than 8, HREF is rightpadded with random characters 8 character domain names give you more control over naming schemes for group profiles 4 4 2

3 General Usage Tips Use Folders to organize reports Within each domain are folders Folders can go two levels deep Authority is the same to all folders within a domain based on the level of access to the domain Allows for logical segments of reports within the security (domain) group Month End Inventory CFO Reports 5 General Usage Tips Use a Prefix or a Suffix as you define the metadata Assists in quickly locating your files in the metadata list Make Prefix = library name to quickly determine where file is located Make Suffix = object type (for example, _QRYDFN for imported Query/400 objects) 6 6 3

4 General Usage Tips Library lists step 1 (DB2 CLI adapter Only) Select One-part name during metadata creation process -- Removes hard coding of library 7 General Usage Tips Library lists step 2 Set library list System wide - QSYSLIBL and QUSRLIBL Per user profile INLLIBL parm in user profile s Job Description Program controlled - Requires creation of exit program and use of CLI connect exit point. Technique to use user profile s initial program use with caution!! PGM PARM(&USER) DCL VAR(&USER) TYPE(*CHAR) LEN(10) DCL VAR(&INLPGM) TYPE(*CHAR) LEN(10) DCL VAR(&INLPGMLIB) TYPE(*CHAR) LEN(10) RTVUSRPRF USRPRF(&USER) INLPGM(&INLPGM) INLPGMLIB(&INLPGMLIB) IF COND(&INLPGM *NE *NONE) THEN(DO) CALL &INLPGMLIB/&INLPGM ENDDO ENDPGM PGM ADDLIBLE LIB(QGPL) ADDLIBLE LIB(LIBRARY1) ADDLIBLE LIB(LIBRARY2) ENDPGM 8 4

5 General Usage Tips Library lists using QIBM_QSQ_CLI_CONNECT exit point Details of Exit Point Creation 1 Create a new source file member, enter the source code, and save it (for example, one named CLI_EXIT) 2 - Compile this new program into QGPL. So you should have program object named QGPL/CLI_EXIT 3- Issue the following CL command to add a new exit point. ADDEXITPGM EXITPNT(QIBM_QSQ_CLI_CONNECT) FORMAT(CLIC0100) PGMNBR(1) PGM(QGPL/CLI_EXIT) The new program will be called every time a CLI connect event occurs (this is the database access mechanism used by the DB2 CLI adapter). So every time you run a report, the exit program is called in the QSQSRVR job that is running the SQL request. Before the database request happens, the exit program calls the user's initial program to set up the lib list. 9 General Usage Tips Backup and recovery Use i5/os save and restore commands SAVLIB and RSTLIB commands for library SAV and RST for IFS directories and files Objects to back up QWEBQRY76 product library /QIBM/PRODDATA/WEBQUERY/IBI/* install directories /QIBM/USERDATA/WEBQUERY/IBI/* your environment directories 10 5

6 General Usage Tips Advantages of Internet Explorer over Firefox Ability to drag and drop fields in development tools (Report Assistant, Graph Assistant) Provides list of fields in Report Headings tab Ability to save Active Reports to hard drive (requires ActiveX controls) 11 Security Tips End users still need i5/os object level authority to all objects that are used in a report This includes: Tables / Physical Files SQL Views Stored Procedures IFS Files Query/400 Definitions Just because a user has access to a report does not mean they will be able to run it 12 6

7 Security Tips Row and column level security Report Requirements: Show only those employees that are in the same division as user running the report. Note: This requires row-level security. Sort and group the employees by division and department name. Include other columns (employee name [first and last], job title and salary). If the user running report is a manger, display salaries only for those employees who directly report to the manager (in the same department as the manager running report). Otherwise, show blanks for this column. Note: This requires column-level security. 13 Security Tips i5/os object level security is ALWAYS enforced!! Use SQL views and UDF s to enhance security Can provide row and column level security Reduces report creation/maintenance! Row level security Column level security 14 7

8 Security Tips Row and column level security Technique described in white paper: Using SQL views and stored procedures with DB2 Web Query 15 Security Tips Two options for row level security 1 - SQL Views 2- DB2 Web Query filters 16 8

9 Performance Tips Keys to good DB2 Web Query performance Hardware and system resources Newer models have more processing power & better throughput Fair share of memory allocation DB2 for i database engine handles most of the work (not DB2 Web Query Reporting Server) DB2 much more efficient at handling: Selection of rows Joining of tables Grouping (aggregation) Sorting Use of DB2 Symmetric MultiProcessing (SMP) Use of SQL Query Engine (SQE) More efficient, more features than Classic Query Engine (CQE) Good Indexing Strategy White paper - Indexing and statistics strategies for DB2 for i5/os 17 Performance Tips Let DB2 for i handle the heavy lifting Why? Access to indexes and statistics Access to plan cache Ability to rewrite query Selection Joining Grouping Ordering Ability to cache results SMP (if installed) Advanced database primitives and data access techniques Materialized Query Tables Optimizer constraint awareness Many more reasons All of this means queries that run FASTER! 18 9

10 Performance Tips Push as much logic as possible down to DB2 Use Run w/sql trace option or edit source and add trace statements Look for database optimization disablers Determine if a single SQL statement is being generated Multiple SQL statements mean that the DB2 Web Query reporting server is merging the result sets Use SQL views to overcome database optimization disablers Trace statements -*-- Deactivate SQL tracing SET TRACEOFF = ALL -*-- Show SQL statements SET TRACEON = STMTRACE//CLIENT -*-- Show Optimization information SET TRACEON = SQLAGGR//CLIENT -*-- Activate SQL tracing SET TRACEUSER = ON -*-- Do not retrieve results SET XRETRIEVAL = OFF Performance Tips Push as much logic as possible down to the database Consider implementation of Referential Integrity at database level Use With Foreign Keys setting in metadata Can replace synonym and report defined joins - simplifying report development and possibly resulting in better performance All database interfaces can take advantage of RI Data-centric model can extend the life of your applications

11 Performance Tips Joining: Option 1 - Implement Referential Integrity To add constraints you can use IBM i command ADDPFCST ADDPFCST FILE(mylib/CUSTOMER) TYPE(*PRIKEY) KEY(CUSNBR) CST(customer_key) ADDPFCST FILE(mylib/ORDERHDR) TYPE(*REFCST) KEY(CUSNBR) CST(orderhdr_cnbr) PRNFILE(mylib/CUSTOMER) PRNKEY(CUSNBR) DLTRULE(*RESTRICT) UPDRULE(*RESTRICT) SQL statement CREATE TABLE or ALTER TABLE System i Navigator ALTER TABLE mylib/customer ADD CONSTRAINT customer_key PRIMARY KEY (CUSNBR) ALTER TABLE mylib/orderhdr ADD CONSTRAINT orderhdr_cnbr FOREIGN KEY (CUSNBR) REFERENCES mylib/customer (CUSNBR) ON DELETE RESTRICT ON UPDATE RESTRICT RI implementation impacts Cleanse data Applications must handle RI constraint violation errors Journaling and commitment control may be required 21 Performance Tips Joining: Option 1 - Implement Referential Integrity 22 11

12 Performance Tips Joining: Option 2 - Use SQL Views Create joins in SQL views (then create metadata against the views) Technique for implementing row and column level security Encourages database optimization ( pushes logic down to DB2) Take advantage of additional join types, unions, intersects, excepts, Common Table Expressions Data-centric Accessible by any SQL interface Embedded SQL JDBC ODBC... others... CREATE VIEW ORDERSVIEW (ORDERNUMBER FOR COLUMN ORDER_NUM, PRODUCTNUMBER FOR COLUMN PROD_NUM,...) AS SELECT a.*, b.*, c.*, d.* FROM orders a INNER JOIN stores b ON a.storecode = b.storecode INNER JOIN inventory c ON a.prod_num = c.prod_num INNER JOIN plant d ON a.plantcode = d.plantcode 23 Performance Tips Joining: Option 3 - Define joins in synonym DB2 Web Query join syntax stored in metadata translated to SQL statement Requires Developer s Workbench Can be accessed by all DB2 Web Query reports and graphs Performance could suffer if logic is not pushed down to database (due to datebase optimization disablers) Not accessible from other SQL interfaces 24 12

13 Performance Tips Joining: Option 4 - Define joins in report development tools (Report Assistant, Graph Assistant) DB2 Web Query join syntax in Report procedure is translated to SQL statement Developer s Workbench not required Performance could suffer if logic is not pushed down to database (due to datebase optimization disablers) Can be accessed ONLY by that DB2 Web Query report/graph 25 Performance Tips Maximize SQE usage Goal is to have SQE process every query Enhanced optimization techniques Better/faster database primitives Features MQTs Maintained Temporary Indexes (MTIs) SQE plan cache Ability to cache results Many more

14 Performance Tips Maximize SQE usage - Avoid SQE Inhibitors Avoid creating metadata for DDS logical files Only tables/physical files, SQL views, or MQTs Create SQL View for Logical File Equivalent Watch out for Select-Omit logical files against physical files (not as important in V6R1) Both the Heritage File and Query/400 adapter use CQE Use the DB2 CLI adapter if possible Recreate Query/400 reports as new reports that use synonyms based on DB2 CLI adapter No way to avoid it for multi-format files (must use Heritage File adapter) Use SQL Aliases to access multiple members TechTip: Maximize SQL Query Engine (SQE) Usage of Your DB2 Web Query Reports Performance Tips Multi-member file support DB2 CLI adapter Sends SQL statements to database, SQL can only read data from the first member in a file, so you must first create an SQL ALIAS for the member on which you want to define the metadata CREATE ALIAS MYLIB/MYALIAS FOR MYLIB/MULTI_MBR_FILE (MBR_NUM_2) Knowledge Base doc. Number Accessing Files with Multiple Members Using SQL DB Heritage Files adapter Sends OPNQRYF CL commands to database, so you can directly specify the member name creating metadata SQE will NOT be used! 28 14

15 Performance Tips A case for not using the Query/400 adapter If you use the Query/400 adapter, there are several limitations: Since it uses RUNQRY, query is processed by CQE Do not get the advantages of SQE We are bound by the data in the result set Cannot add additional fields or rows We are dependent on the QRYDFN object If someone changes (or deletes) it, all based on reports change Amount of time to rewrite is low Use the query definition printout to recreate quickly Cannot use DRDA to query remote DB2 for i databases CLI connect exit point 29 Performance Tips Date conversion alternative Date conversion table Push as much logic as possible down to the database Use pre-populated table instead of DB2 Web Query built-in functions for date conversion Can be implemented via Referential Integrity constraints SQL views DB2 Web Query synonym defined joins DB2 Web Query report defined joins Can result in significant performance improvements over built-in date conversion functions 30 15

16 Performance Tips Date conversion alternative Date conversion table ORDERS legacy file Date conversion table Join (RI constraints, SQL view, synonym join, report join) 31 Performance Tips Use of Materialized Query Tables (MQTs) DB2 for i Enhancement (V5R3 ) Also known as "materialized views" and "automatic summary tables" User maintained only MQTs contain: Query definition Results of the query SQE optimizer can rewrite the user's query to use the MQT Can significantly increase query performance Can significantly decrease resource utilization Default behavior is: do not use MQTs QAQQINI file options to enable/disable support White paper - Creating and using materialized query tables (MQT) in IBM DB2 for i

17 Performance Tips Use of Materialized Query Tables (MQTs) Before... SELECT year, quarter, month, SUM(revenue_w_tax) AS srevenue_w_tax, SUM(revenue_wo_tax) AS srevenue_wo_tax, SUM(profit_w_tax) AS sprofit_w_tax, SUM(profit_wo_tax) AS sprofit_wo_tax, SUM(quantity) AS squantity, COUNT(*) as number_items_per_group FROM ITEM_FACT GROUP BY year, quarter, month; After... Without MQT... Scan and aggregate 6,000,000 rows Base table(s) replaced by MQT With MQT... Scan 36 rows 33 Performance Tips Use of Materialized Query Tables (MQTs) Where do you use MQTs Static Data that is massaged/ consolidated, summarized MQTs especially benefit queries that use complex joins and aggregations BI and DW environments where data is loaded periodically are particularly good candidates. Daily, weekly and monthly processing provides a natural opportunity to refresh or update the MQTs

18 Performance Tips Avoid runaway queries! Consider use of QUERY_TIME_LIMIT Specifies a time limit for database queries allowed to be started based on the estimated number of elapsed seconds that the query requires to process. WRKSYSVAL QQRYTIMLMT QAQQINI options file (QUERY_TIME_LIMIT) Query Governor Exit Point (QIBM_QQQ_QUERY_GOVR) Provides granularity - Reject long running query based on specific users, day of week, time of day, and so forth Only Only applies to to single SQL SQL statements being optimized by by the the database engine!! Performance Tips Other techniques Use the On-demand paging option Loads one page of data at a time Use the Format HTML Output set at Paged Breaks up HTML tables into page size chunks Use the Use database optimization option Forces items such as joins to DB2 instead of doing them in Web Query If accessing data on other partitions, set up Virtual LAN connections 36 18

19 Performance Tips Case Study for Performance Analysis Converted Query/400 that does join over 3 files and reads over 6 Million Rows Ran 28 Minutes on Query/400 Converted using DB2 CLI adapter and ran in 8 Minutes Performed Performance Analysis to optimize query (created indexes) - ran in less than 90 seconds on first run Sub-second response time in subsequent runs (utilized SQE s ability to cache results) 37 Performance Tips Use On Demand Performance Center tools Performance analysis Index advice SQE inhibitors MQT usage Caching results Before and after comparisons

20 Query Optimization Feedback Indexes Advised SQE Plan Cache SQE Plan Cache Snapshots Visual Explain SQL request Query Optimization Print SQL Information Messages Detailed DB Monitor Data Summarized DB Monitor Data Debug Job Log Messages 39 More information Getting Started with DB2 Web Query Redbook Tutorial Driven Download Sample Database E-Learning Modules Self paced, downloadable end user training

21 Building a DB2 Web Query Community New DB2 Web Query Site Registration Recommended KnowledgeBase Forums Links to Additional Information Expert BLOGs WIKI Links to other pertinent Information Supplements ww.ibm.com/systems/i/db2/webquery 41 Building an SQL Optimization Education Plan OL45 1 day DB2 Fundamentals OL37 2 days Accessing Data Using SQL OD40 4 days SQL/Query Performance Workshop 4.5 Day Course in Rochester ibm.com/servers/eserver/iseries/service/igs/db2performance.html OD050 3 days DB2 Web Query for Advanced Users, Administrators, and Developers SQL Optimization Assessment Available for use with Services Vouchers NOT a performance analysis only an assessment Database Analysis Tools from Centerfield Technologyhttp:// Note: Traditional Performance Analysis addresses RESOURCE utilization, but does not address database optimization

22 43 Special notices This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area. Information in this document concerning non-ibm products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY USA. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied. All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice. IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generallyavailable systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment. Revised September 26,

23 Special notices (cont.) IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/os, i5/os (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pseries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xseries, z/os, zseries, AIX 5L, Chiphopper, Chipkill, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Purpose File System,, GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iseries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, System i, System p, System p5, System Storage, System z, Tivoli Enterprise, TME 10, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org. UNIX is a registered trademark of The Open Group in the United States, other countries or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries or both. Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both. Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both. AMD Opteron is a trademark of Advanced Micro Devices, Inc. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC). SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC). NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both. AltiVec is a trademark of Freescale Semiconductor, Inc. Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc. InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association. Other company, product and service names may be trademarks or service marks of others. Revised April 24, Trademarks The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both. Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States. For a complete list of IBM Trademarks, see *, AS/400, e business(logo), DBE, ESCO, eserver, FICON, IBM, IBM (logo), iseries, MVS, OS/390, pseries, RS/6000, S/30, VM/ESA, VSE/ESA, WebSphere, xseries, z/os, zseries, z/vm, System i, System i5, System p, System p5, System x, System z, System z9, BladeCenter The following are trademarks or registered trademarks of other companies. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-ibm products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography