SQL Considerations for a Big Data 22 Billion row data warehouse By Dave Beulke

Transcription

1 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke D a v d a v e b e u l k e. c o m ember of the inaugural B DB nformation hampions One of 4 B DB old onsultant Worldwide President of D-NR, Past President of nternational DB Users roup - DU Best speaker at conference & former TDW instructor Former o-uthor of certification tests DB DB ertification tests B Business ntelligence certification test Former olumnist for B Data anagement agazine Weekly Performance Tips: onsulting PU Demand Reduction uaranteed! DB Performance Review DW & Database Design Review ecurity udit & ompliance DB igration ssistance DB Performance B White Paper & Redbook Teaching Educational eminars DB Version Transition DB Performance for Java Developers Data Warehousing Designs for Performance How to Do a Performance Review Data tudio and purequery Extensive experience in VDB databases, DW design and performance Working with DB on z/o since V. Working with DB on UW since O/ Extended Edition Designed/implemented first data warehouse in 988 for E.F. Hutton Working with Java for yspedia since yspedia - Find, understand and integrate your data faster! heck out the DB performance blog at w w w. d a v e b e u l k e. c o m

2 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke Big Data Era bends the Q Rules o The bigger the database the more attention needed to: 3+ tandard Q Rules Database Design Database onditions Database ndexes DB Q being used Page - 3 tandard Q Tips. Only EET the columns needed because it optimizes the /O between DB and your program.. Only EET rows needed for the process or transaction and use FETH FRT x ROW ONY whenever possible. 3. ode WHERE clauses with indexed columns that have unique or good indexes. 4. Reference only tables that are absolutely necessary for getting the data. 5. ode as many WHERE column clauses as possible because the DB engine filters data faster than any application code. Page - 4 heck out the DB performance blog at w w w. d a v e b e u l k e. c o m

3 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke tandard Q Tips 6. Prioritize the WHERE column clauses to maximize their effectiveness. First code the WHERE column clauses that reference indexed keys, than the WHERE column clauses that limit the most data, than WHERE clauses on all columns that can filter the data further. 7. ode Q JONs instead of singular Q access whenever possible. single Q JON is always faster than two Q statements within an application program comparing and filtering the result set data. 8. ode Q tables JONs when there is a common indexed column or columns shared by both or all of the JONed tables. f a common column is not indexed talk to a DB and make a new index if possible. 9. When coding JONs make sure to code the most restrictive JON table first and provide as many indexed and restrictive WHERE clauses as possible to limit the amount of data that needs to be JONed to the second or subsequent tables.. Use DB OP DB functions to optimize your DB Q answer result sets. Using these Q DB OP functions is usually much faster than application code summing, totaling and calculations over your data. Page - 5 tandard Q Tips. everage DB functions appropriately. nalyze the Q to determine the need for the various DB functions. Report writers generate Q with multiple, duplicate or unnecessary DB functions. V, U etc... Examine the data columns and tables EETed in the Q. any of these new big data report writers have nice U interfaces that allow the users to point and click on anything. Dup of #? 3. Verify the DB table joins for improving your DB Q performance. evaluate table indexes uniqueness, cardinality, relationship to the table partitioning or column frequencies. 4. Eliminate data translations. ometime the report writer generated Q uses many T, UBTR, E statements, constants, or formulas in the Q. The generated Q may also have math formulas compared to DB table columns within the Q. heck the ones that are not index-able and clean them for performance. 5. Understand the ORDER of JON and ccess. By making sure the JON of the smaller table drives the data EETed from the big data DB table, your DB Q should eliminate/minimize as much data as possible in the first JON. lso minimize Q such as ORDER BY, ROUP BY, DTNT, UNON, or JON predicates that do not leverage the clustering order of the table, because they will require a DB sort. Page - 6 heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 3

4 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke tandard Q Tips 6. Understand all Q phrases that results in a ORT. DB Q workloads orts are one of the most expensive operations especially with Big Data. i.e. -Distinct, Order by, UNON etc. 7. Verify calar function is index-able. n DB many scalar function became index-able not all of them like DTE(timestamp-column). 8. Remove all math from the WHERE lause if possible. ath in WHERE clauses sometime disrupts the data type comparisons and precision. Pre-calculate math before Q statement and always verify index-ability of any that are questionable. 9. Understand the number of trips to the database. Use multi-row EET and NERT operations whenever possible to access or to the add the database.. inimize and optimize any WHERE OR clauses. ake sure that the OR does not make the entire WHERE clause become a table scan. Optimize all OR phases and turn them into a correlated sub-query if possible. Page - 7 tandard Q Tips. EET from NERT, UPDTE whenever possible. voids multiple trips to the database and is great for identity, sequence and any other database column keys. Use ERE to reference the database and make sure the data is there and updated. void complex program logic to perform the data checking and then update. 3. reate indexes on all WHERE clause expressions that are not already index-able. ompanies have unique situations and data types. reate ndexes on Expressions that are used in your daily Q processing. 4. Use ommon Table Expression (TEs) whenever possible because they can get the data so that the result set can be used repeatedly and efficiently. 5. Use DB TEP tables for all data that is referenced more than three times in the flow of daily business. t is more efficient to get the TEP result set working instead of repeated redundant data access. Remember they can also be set up as NOT OED. Page - 8 heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 4

5 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke tandard Q Tips 6. Use DB Virtual ndexes to model your potential ndexes odel NUDE olumn, ndex on Expression, Uniqueness and other new index definition possibilities. 7. Use Optimizer Filter Factor Hints to give full information. JON kew/orrelation, accurate statistics, complex predicates and host variable and special register information. 8. nalyze the YTTFEEDBK table contents to understand your Q ccess Path results better. an be used to generate better RUNTT input. 9. everage EXUDE NU when creating indexes to reduce index size. Verify that the NU-ability is not within the application WHERE clauses. 3. Overriding predicate selectivity with the DNPREDT% tables. Populate these tables with the details of the selectivity of the tables uniqueness to improve optimizer knowledge for better access paths and improved performance. Page - 9 Big Data Database Design o Big data physical objects /O bound operations Partition the database tables to minimize the data required for daily Q Use the thousands of partitions available for a design o How many parallel processes are your applications running today? o eparate old data from new data urrent Year, Quarter, Week, Day Temporal tables with the HTORY tables o omplicates the Q also can make a lot of data quickly aterialized Query Table YTD sales figures o Or composite tables to separate via TE axis Year, Quarter, Week, Day o Or composite tables to separate via ales territory axis ountry, Region, tate, ity, Zip code Page heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 5

6 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke Database Design for parallelism o Partitioning to leverage??? more concurrent processes ame partitioning limit keys across multiple table spaces o Via ustomer number across those related tables o Via Product KU number across all the product related data o Partitioning design leverages time properly The active partitions are only a segment of the entire table o oncentrates the /O into the right sized portion of the database o urrent history available - ncient history is not in database/archived easily o ndexes (NPs or DPs) are appropriately designed Partitioned for parallelism and recovery Table clustered for Q efficiencies zp zp zp zp Page - zp zp zp Big Data Flow odesign encourages concurrent loads and queries Either through bulk, trickle or custom loads utomatically aggregates data within hierarchy assively parallel processing of all workload aspects everage daily processing aggregates Fact-Daily Fact-Week Fact-onth Fact-Q Fact-Yearly heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 6

7 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke aterialized Query Table- Example o Daily sales figures feed s reate additive s s created for all analysis comparison points Daily ales Weekly Totals o s can be built from other s Define Quarterly ales from onthly ales o ombine s through Views onthly Totals Views over region, territory, store id etc Quarterly Totals View Y-T-D Totals OTP Detail Trans from today opyright Dave Beulke & ssociates dave@davebeulke.com Page 3 to times improvement! o5b rows per year per 4k page= ½B pages o aggregates save large amounts of everything oreate aggregates for every possibility On Demand information ales by department ales by zip code Fact-Yearly Y-T-D View Fact-Q ales by time period day/week/month/quarter/p ll reporting and analysis areas Trace usage to create/eliminate aggregates ototal by month ½B /Os versus /Os Fact-onth Fact-Week Fact-Daily Page - 4 heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 7

8 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke Database onditions oruntts ample your database 5% Table o Page size Optimization Do you have the correct DB Page ize? olumn ardinality o mportant olumns ndex columns JON columns ll columns used within any WHERE clauses ndex o ardinality ake sure to get detailed statistics o NUDE olumns Page - 5 Database onditions ohow long does it take until your ndex goes bad? o You can collect distribution statistics when you collect inline statistics when you run the following utilities: OD REOR TBEPE o You can collect histogram statistics when you collect inline statistics when you run the following utilities: OD REBUD NDEX REOR NDEX REOR TBEPE Page - 6 heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 8

9 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke Database onditions odentify your VOTE TBEs Targeted for P ustomers and the P luster Tables Favors using ndex access whenever possible o voids list prefetch o an be a problem for OR predicates or UPDTEs at risk of loop ooptimize Page level options PE ROW Ratio needs to be optimized for storage and Update frequency PTFREE FOR UPDTE in DB o Help keep clustered longer, reduces the use of overflow records and indirect references Page - 7 tatement evel Hints teps. Zparm required - Enable OPTHNT. Populate DNUERQUERYTBE with Q text From YPKTT or tatement ache 3. Populate PNTBE with corresponding hints ake sure to match QUERYNO in both tables - DNUERQUERYTBE & PNTBE 4. BND QUERY to put Hint into the repository Next dynamic prepare should use the Hint FREE QUERY to remove it Page - 8 heck out the DB performance blog at w w w. d a v e b e u l k e. c o m 9

10 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke Database Design ouse correct database column definitions o all the DB Online nalytic Processing-OP functions can be used ounderstand the new tage predicates processed value BETWEEN col ND col value BETWEEN column-expression ND column-expression UBTR(OX,, n) = value **NOTE only from first position** DTE(timestamp-column) = value YER(date-column) = value E WHEN THEN EE END = value Page - 9 teps to resolve Q problems. tart with all EXPN tables information. Research the database objects, Tables, ndexes 3. Research and understand the statistics a. Understand where the problems are: PU, /O, #-orts, #-tables, wrong indexes 4. Using all WHERE clauses possible most restrictive first 5. How many times accessing the same table 6. hange the Q table order? orts 7. Filter Factors, uneven data distribution, range predicates a. re tats current, RT Usage & an you change the objects statistics to improve? 8. an you use Hints to improve the optimizer access path choice 9. Buy a B DB nalytic ccelerator - D Page - heck out the DB performance blog at w w w. d a v e b e u l k e. c o m

11 Q onsiderations for a Big Data Billion row data warehouse By Dave Beulke Questions? Thank You! heck out all the articles at www. D a v e B e u l k e. c o m D a v D a v e B e u l k e. o m heck out the DB performance blog at w w w. d a v e b e u l k e. c o m