Web Intelligence Report Validation Tips and Techniques

Transcription

1 September 9 11, 2013 Anaheim, California Web Intelligence Report Validation Tips and Techniques Michael Welter

2 Learning Points First, become familiar with what is right Carefully observe how your report divergences from right Categorize the divergence Systematically fix the problem 2

3 Return on Investment Time saved in validating and fixing reports Consumer confidence in reliable BI reports Proper business decisions based on correct analysis Reduced wear and tear on report developer(s) 3

4 Breakout Abstract Web Intelligence Report Validation Tips and Techniques Have you ever had unexpected results when developing a complex Web Intelligence report? This session is for Web Intelligence report designers and IT staff who support them, looking for some time saving pointers and frustration alleviating tools for troubleshooting and validating reports. See how to identify problems in the source data, in the universe, and in the report itself. Learn techniques for verifying accuracy, and explore the mysteries of calculation contexts in Web Intelligence. See these methods in action with live examples. 4

5 Agenda 1. Why are we talking about validation? 2. What s Right? 3. What s Wrong? 4. Conclusion 5. Q&A 5

6 Why Are We Talking About Validation? 1. Why are we talking about validation? 2. What s Right? 3. What s Wrong? 4. Conclusion 5. Q&A A fancy, beautiful report that s wrong is useless (or worse) 6

7 BusinessObjects Gives You the Power of Perspective Turn stored data into decisions Decision making is a maze We have a ground level perspective Need an aerial perspective Business decisions require access to information Flexible Dependable Accurate 7

8 A Big Job That Looks Easy Querying a database is not a trivial matter SQL can be complex Database designs are often complicated BusinessObjects Web Intelligence is a big lever Drag and drop to build complex queries Navigates the labyrinth for you Knows the rules of connecting database tables Knows when to separate statements for you You don t have to understand what happens behind the scenes Consistently correct results amazingly often! 8

9 Detect and Fix Errors Still plenty of room for error! One day to build; Two weeks to debug Errors can be elusive Often takes much longer to find and fix than original development Generally frustrating We need a way to find and fix these errors A systematic approach to detecting problems A map for when to fix and when to get help A set of tools for when it s time to dive in 9

10 Assumptions You have basic to intermediate Web Intelligence skills You are using Web Intelligence to report from data that Is believed to be correct Has some benchmarks (perhaps a previous report) You (or a colleague) know the data and the business it represents The database you are reporting from is either The same as the original work, or A data warehouse/mart based on the original database (and is valid) You have IT support (or skills) You have extract/transform/load support (if warehouse) You have universe support (or skills) You have some idea how to read SQL 10

11 What s Right? 1. Why are we talking about validation? 2. What s Right? 3. What s Wrong? 4. Conclusion 5. Q&A What target are you trying to hit? 11

12 Start With What is Correct Define benchmarks first You ll save time if you know the answer If you don t know what correct is, you won t know when you hit or miss it Then analyze your report s deviation from the benchmark The following is a strategy for defining benchmarks 12

13 Collect Data Points From Business Expert(s) Collect benchmark data points from business experts Perhaps the person who was responsible for the previous version of a report you re replacing Someone who knows the subject and the data Someone who knows how it was done Collect data points at the lowest grain Most specific Example: Sales_Qty, Sales_Amt of Product 1059 on 8/1/2007 at Store 21 by Employee 99 and at various levels of aggregation Increasingly more summarized Across all dimensions involved in the report Example: Sales_Qty, Sales_Amt of all Products on 8/1/2007 at Store 21 by all employees 13

14 Discover How These Data Points Were Produced What elements were used? What calculations were used? How was time related? Any outside data sources? Is any data source required outside the database from which we re reporting? If so, how was that data related to the rest? Get the SQL If possible, get a copy of the SQL used Soft copy is best 14

15 Discovery Methods Interview There s no substitute for talking Do this in person if possible Observation Example 1: reports currently produced from Access Note what database, queries, macros (if any) they used Get a copy of the SQL used Note the processing flow: What steps and in what order? Note any temporary tables used Example 2: reports currently coming from ERP system Note the processing flow: What utilities/commands, in what order? Ask IT to explain how that app gets its report Ask IT for copy of SQL the app generates if possible 15

16 Verify Results Verify you or IT is able to duplicate results of benchmark SQL At all levels of aggregation provided in benchmark data points In same system as original report came from In data warehouse with analogous SQL Verify required SQL is present in universe Ask universe designer for help here Fragments similar to benchmark examples, or Analogous fragments built to work against data warehouse Duplicate benchmark numbers with simple reports Make bare bones Web Intelligence report Check SQL against benchmark SQL Get IT/designer help as necessary 16

17 What s Wrong? 1. Why are we talking about validation? 2. What s Right? 3. What s Wrong? 4. Conclusion 5. Q&A How does your report deviate from the benchmarks? 17

18 Now You Can Start to See What s Wrong Canadian bank training technique Plumb line example Now we can start to see what s wrong We have some known good data points The more you know the source data, the more intuitive you ll be at spotting deviation 18

19 Your Business Objects Team Database problems go to IT Universe problems go to your universe designer A sermon/word of caution regarding your team When a problem is in another s area of expertise It s out of your hands (relief) Why didn t they catch that problem (frustration) By the way, it s, um, out of your hands (more frustration) You need the whole team Data problems are a reality in young data warehouses Avoid blame Resentment and defensiveness degrade teamwork quickly You work half speed with a sword in one hand 19

20 The Tools The cure is often easy once you know what s wrong The tools The following strategy helps focus your efforts The following techniques give you leverage The approach Group possible issues by symptom Work through the possibilities 20

21 Number Values Consistently Too High or Too Low Report or block filter There s a filter in place that isn t obvious Demo Too low condition only Query condition isn t right You may have too many or too few where statements You may be filtering on the wrong elements Wrong universe measure (units instead of dollars?) Measure is defined incorrectly or imprecisely Ex: booked orders vs. revised bookings Semantic problem needs to be addressed in the business and in the universe 21

22 Check Filters Show filter pane View structure 22

23 Check Filters Select filters on maps tab 23

24 Check Filters Click through report elements to find filters... Aha... found one! 24

25 Check Query Conditions Look for unintended conditions 25

26 Wrong Universe Measure? Should have $ instead of units 26

27 Object Defined Incorrectly Check for what correct is... 27

28 Object Defined Incorrectly Check the SQL your object is generating 28

29 Number Values Consistently Too High or Too Low Wrong aggregation type on a universe measure Unusual, but possible Use object in bare bones report Check the math on aggregation Use object in table block footer (no agg function) Compare to object used in footer w/aggregate function Designer addresses in universe 29

30 Wrong Aggregation Type Value seems too low Especially the total 30

31 Wrong Aggregation Type ID the miscreant object Make a simple doc with it 31

32 Wrong Aggregation Type Insert a sum on your simple table 32

33 Wrong Aggregation Type... but with no aggregate function 33

34 Wrong Aggregation Type That looks more like an average than a sum... 34

35 Wrong Aggregation Type Ah, the bare object is projecting as an average Ask universe designer to fix the measure s aggregation function 35

36 Number Values Are Inconsistently Wrong Symptom: some values are too high, others too low Universe is connected to test data You may know by date, or some other telltale data Ask universe designer to check this Wrong date key in fact table joined to date table Many data warehouses have a date lookup Fact table may have many dates designer picked one Example: Sales by date should use order date, not ship date When view by date, some months are too high, others too low Review SQL to see what date is being used 36

37 Wrong Date Key Results are too low some quarters, too high others 37

38 Wrong Date Key Examine SQL statement: it appears to use a ship date Ask designer to fix the date 38

39 Some Values Too High by Even Multiples Symptom Some values too high, others may be correct When too high, it s by an even multiple of the correct value This suggests a SQL (fan/chasm) trap in the universe ID the offending object(s) by systematically pulling objects out of the query until the values are correct Share findings with your universe designer so they can fix 39

40 Some Values Too High by Even Multiples Some values too high Those that are too high are so by even multiples ($190,174 * 2 = $380,348) 40

41 Some Values Too High by Even Multiples To ID a SQL trap, start pulling objects out of the query Is it Showroom Town? 41

42 Some Values Too High by Even Multiples To ID a SQL trap, start pulling objects out of the query Is it Showroom Town? No. Is it Showroom? 42

43 Some Values Too High by Even Multiples To ID a SQL trap, start pulling objects out of the query Is it Showroom? Nope. How about Sale Qty? 43

44 Some Values Too High by Even Multiples To ID a SQL trap, start pulling objects out of the query BINGO! Now add Showroom objects back in 44

45 Some Values Too High by Even Multiples To ID a SQL trap, start pulling objects out of the query Revenue still looks good Now go tell your universe designer: Sales Revenue is too high for some customers By even multiples When you add Sales Qty to the query You suspect a fan or chasm trap they ll be impressed (or annoyed) 45

46 History by Certain Dimension is Wrong Example TY/LY sales by line is wrong for last year (but total is OK) When you compare TY/LY sales by state, everything is fine 46

47 History by Certain Dimension is Wrong Totals are right, but two lines are wrong for last year 47

48 History by Certain Dimension is Wrong May be slowly changing dimension in data warehouse Discuss with business and IT Maybe some products have changed Lines since last year Business must decide to view by current or historical alignment Solution is in data warehouse (if extract/transform/load person is able to change behavior) 48

49 Missing Dimension Values Where Nothing Happened Example Sales by product lacks no sale products 49

50 Missing Dimension Values Where Nothing Happened Inner join used in universe, but for good reason We need to see all customers, sales or no sales You can cheat: Demo Force report side outer join 50

51 Missing Dimension Values Where Nothing Happened Add query: full list of dimension values (list items only) 51

52 Missing Dimension Values Where Nothing Happened Add table: full product list 52

53 Missing Dimension Values Where Nothing Happened Verify dimension(s) merged 53

54 Missing Dimension Values Where Nothing Happened Make merged table Use dimension from 2 nd query or the merged dimension 54

55 Calculated Variables Are Wrong 1. Wrong calculation (check arguments) 2. Objects used by variable rendering wrong results Duplicate report using referenced objects Work through the too low/high section 3. Calculation context is incorrect Example: calculated column value not computed right Requires specific output context Demo of calculation contexts 55

56 Calculated Variables Are Wrong Well, THAT s not right 56

57 Calculated Variables Are Wrong Click cell notice it contains a variable 57

58 Calculated Variables Are Wrong Edit variable (Hmm formula looks OK) But let s check help anyway 58

59 Calculated Variables Are Wrong Help > FormatDate > Index to Date & Number Formats Argument should be MM/dd/yyyy (not mm/dd/yyyy ) 59

60 Calculated Variables Are Wrong Fix the variable s formula 60

61 Calculated Variables Are Wrong That s better! 61

62 Objects Used by Calculation are Wrong Business analyst says margin goal calculation is too low Margin goal calculation logic looks OK 62

63 Objects Used by Calculation are Wrong What about the underlying object? Margin goal s variable s formula is =[Margin w]*1.1 Is the margin w measure giving us the right values? Let s test it: 63

64 Objects Used by Calculation are Wrong Ah: margin w is not giving correct results: Calculation is wrong because underlying object is wrong Follow previous steps to diagnose problem with this object 64

65 Wrong Calculation Context Stores should be classified based on years open Young, average or old, relative to the average for all stores But all of ours say average something is amiss... 65

66 Wrong Calculation Context Store class column is empty 66

67 Wrong Calculation Context An alerter populates the column 67

68 Wrong Calculation Context The alerter s logic depends on variables 68

69 Wrong Calculation Context which depend on other variables 69

70 Wrong Calculation Context Let s see what s going on with.avg Years Open Duplicate the report, and place an average on Years Open The average store has been open 14.6 years at this time 70

71 Wrong Calculation Context Let s look at our.avg Years Open object in the table:.avg Years Open is calculating the average of each row 71

72 Wrong Calculation Context What we want is for each row to have a value of 14.6 The average of all rows in the block Just like our formula sitting in the footer of the block By default, BusinessObjects measures: Consider all the dimensions they see (in their block or section) Aggregate to the level appropriate to where they are placed We want our.avg Years Open measure to: Consider all dimensions for each row (default), but Aggregate as if it were it placed in the block s footer 72

73 Wrong Calculation Context This is Calculation Context: Input Context = the list of dimensions the calculation considers Output Context = the level to which the calculation aggregates Syntax: =AggregateFx([measure] input context) output context In our case, we only want to change the output context: 73

74 Wrong Calculation Context Now.Avg Years Open behaves like a footer formula But it does so for each row 74

75 Wrong Calculation Context And now that our context for average is correct Our alerter compares each store to the average of all stores 75

76 Conclusion 1. Why are we talking about validation? 2. What s Right? 3. What s Wrong? 4. Conclusion 5. Q&A 76

77 Key Learnings Become well familiar with what right looks like Carefully observe how your report divergences from right Categorize the divergence Work the problem systematically 77

78 Best Practices In determining what is right", collect not only existing reports and SQL, but also known accurate data points at various levels of granularity Take the trouble to understand how the business works Verify results of supplied SQL Be one of those players that raises your teammates' game 78

79 Q&A 1. Why are we talking about validation? 2. What s Right? 3. What s Wrong? 4. Conclusion 5. Q&A 79

80 Thank you for participating. Please provide feedback on this session by completing a short survey via the event mobile application. SESSION CODE: 1209 Learn more year-round at