Informix IWA Case St udies in t he Pharmaceut ical Indust ry. Vicent e Salvador vsalvador@deist er.es (Thanks t o Fred for his help)

Transcription

1 Informix IWA Case St udies in t he Pharmaceut ical Indust ry Vicent e Salvador vsalvador@deist er.es (Thanks t o Fred for his help) 1

2 TRENDS Dat abase and Dat a Warehousing Indust ry 2

3 Analyt ics/warehouse Workload and Opt imizat ions (1) Dat a Warehousing/Analyt ic (OLAP) workload are inherent ly more complex t han Transact ions/oltp and reasons are well-document ed Ways t o overcome t hat include building a performance layer : Transact ional Processing (OLTP) ROW-processing Building Indexes Part it ioning of Dat a Building Cubes/Mat erialized Views Query Tuning OR a hardware layer: Appliances t hat add a new layer of Hardware t o perform I/O for DBMS Mixed-Workload always a challenge DBMS needs t o be built t o handle such a workload (OLTP vs OLAP) Analyt ical Processing (OLAP) COLUMN-processing 3

4 Analyt ics/warehouse Workload and Opt imizat ions (2) Unt il now... You had t o have one opt ion or t he ot her... Or kept t wo+ different syst ems and migrat e dat a... Or t raded up set up t o favor one workload over t he ot her Now, wit h IBM Informix... You can have a dual row and columnar format dat abase In-Memory columnar st orage and processing for analyt ics workloads on a Hybrid (SQL and NoSQL) dat a plat form 4

5 In-Memory DB: Why Now? 64-bit processors can address up t o 16 exabyt es of dat a DRAM prices drop by 30%every 18 mont hs 1 GB of NAND flash memory average price is less t han US$0.50 Commodit y blades provide 1 t erabyt e of DRAM Mult icore CPUs enable parallel processing of in-memory dat a In-memory-enabling soft ware is amply available and proven 5

6 Technology Overview IBM Informix Warehouse Accelerat or 6

7 Int roducing IBM Informix Warehouse Accelerat or Extreme Perf ormance Transactions Extreme Perf ormance Analytics Inf ormix dat abase server Inf ormix W arehouse Accelerat or Analyt ic query Query Opt imizer Accelerat e Query? Result s No Analyt ic query In- Memory Compressed Columnar Dat abase Part it ion Yes Result s Bulk Loader Query Processor TCP/IP In- Disk [Compressed] Relat ional / Row- based Dat abase Linux on Intel / AMD 64-bit Most Unix/Linux 64-bit platforms POW ERFUL DUAL AND HYBRID ENTERPRISRE DATABASE PLATFORM 7

8 Unique Technologies for Speed-of-Thought Analysis IW A combines Breakthrough IBM Research & Development Lab Innovations Inf ormix (IDS) IW A IW A In- Memory Deep Row store (Informix) for Columnar fast OLTP Dat a st orage & Query Column store (IWA) for fast Processing OLAP Deep Data Compression Multi- Core Parallelism Fit s TB of raw dat a st orage and Vect or Opt imized IW A Algorit hm (No Locking) on Int el 64-bit Predicat e evaluat ion done SIMD t echnology directly on Compressed data Massive Parallel Processing of Inf ormix Intelligent Frequency Dat a Load/Refresh & Query IW A Memory File System copy Partitioning IW A Smart deep columnar st orage 8 Fast St orage Backup in Full, Partial and

9 IWA Benefit s Extreme Speed f or Analytics in Inf ormix 100x+ fast er response t imes for complex BI queries Real-Time Analyt ics/olap; consist ent ly fast / inst ant aneous query result s Use it on t op of Informix t o provide inst ant in-memory performance boost on analyt ic queries, while indirect ly helps get fast er OLTP workloads Low Cost 3 rd Generation Database Platf orm Very Easy t o Deploy, Use and Administ er in-memory columnar t echnology Leverages exist ing dat abase plat form; simplifies mixed workload solut ions Uses low cost commodit y HW: Linux on Int el/amd 64-bit Transparent t o applicat ions, works behind t he scenes in Informix Flexible and scalable in bot h SMP and MPP/clust er archit ect ures Int egrat es wit h Informix syst ems on Flexible Grid & HA clust ers Hybrid Platf orm ready f or Big Data and Internet of Things IWA handles Terabyt es of compressed dat a in-memory Informix + IWA is a 3rd generat ion dat abase for SQL, NoSQL & sensor dat a 9

10 You can use IWA t o Speed Up Analyt ic queries on 10

11 Informix Warehouse Accelerat or Case St udies LABCO FEDEFARMA 11

12 LABCO European Diagnost ics and Labs net work 12

13 Who is Labco S.A.? LABCO is a leading European diagnostics group Mainly blood and urine t est s Pan- European company wit h presence in: Belgium, France, Germany, It aly, Port ugal, Spain, Swit zerland and UK. Expansion plan t o increase presence in more count ries Websit e: ht t p:// 13

14 Labco Hist ory Creat ion of LABCO SAS 2003 Development of t he net w ork in France 2004 Creat ion of t he Labco net w ork 2005 Development of t he net w ork in Europe 2006 St ruct ure in France is st rengt hened Consolidat ion of t he net w ork in Europe 14

15 Labco Hist ory Int egrat ion of t he group Development of corporat e DW H and IW A t est deployment s New group project s launched, in order t o involve t he labs and part ners in t he group int egrat ion New organizat ion The corporat e st ruct ure is redef ined wit hin t he group and amongst t he part ners Corporat ion needs a global t ool t o t ake decisions at headquart er level Full deployment of global dat a and replicat e model t o ot her areas like: Financials, HR, et c. 15

16 What Labco want ed from t he new DWH 16

17 Original IT Scenario Each lab or company has t heir own IT inf rast ruct ure t hat is not int egrat ed wit h corporat e different servers around 7 count ries 14 + different soft ware providers for dat a France IT developed a consolidat ed report ing syst em Only basic report ing on reduced number of met rics Each report took 2 4 hours t o be execut ed Only one supplier and one count ry did dat a int egrat ion Port ugal IT st art ed development of DWH report ing syst em based on Pent aho and open source dat abase Corporat e headquart ers request ed IT for a global source of dat a f or all corporat e Deist er, IBM Soft ware Part ner, was cont act ed and a Proof of Concept project was st art ed 17

18 Proof of Concept : Scenario Bew are of project complexity Security and technical concerns about data Dat a cannot get out t he lab wit hout real good dat a masking It s a legal issue Whirlpool 3 algorit hms used t o anonymize personal informat ion Tradit ional file or st ream loading doesn t work Black box anonymizat ion st reaming services deployed on each server Similar issues as in Big Data projects Expect ed performance, unknown speed Const ant report ing and process pipelines Timeframe for dat a processing 18

19 Proof of Concept : Cust omer Concerns Informix IWA performance and reliabilit y It s a very big invest ment, so w e want t o be sure it will work Will it support all corporat e dat a? How fast will it be? Dat a processing Could all dat a be anonymized, loaded and int egrat ed in t he allowed t imeframe? Soft ware feat ures Can we deploy cubes t o provide useful consolidat ed corporat e dat a? Will t he user int erface be easy enough t o allow financial st aff, CEO, et c. t o use in a meaningful way? Is it compat ible wit h corporat e t ools? 19

20 Proof of Concept : Development and Deployment Inbound Consolidat ion, consolidat ion & consolidat ion Different sources and codificat ion should be aligned in a common, and DYNAMIC mapping syst em. A common dat a glossary should be creat ed from scrat ch t o ident ify corporat e ent it ies across all organizat ion Corporat e is a consolidat ion of companies and labs t hat st ill have aut onomy over t heir own dat a Outbound Once consolidat ed and mapped t o MDM common t ags, syst em should deliver daily aut omat ic low level info t o each lab (3000+) Corporat e st aff should be able t o analyze global dat a in mult iple dimensions and drill down t o find source of issues. 20

21 Some Project Numbers Hardware For IBM Informix and IWA Server 2 x 8 cores Int el(r) Xeon(R) CPU E GB of RAM For ETL, Cube generat ion and Report ing syst em 1 x 8 cores Int el Xeon CPU E5 Apple servers for anonymizat ion and black boxing Technology Informix St orage Opt imizat ion (Compression) is used t o improve st orage size and performance in loading process (2% fast er) Informix IDS and IWA servers shared same hardw are plat form: This server is mainly for dat a loading and IWA ut ilizat ion Allows use of St orage Opt imizat ion license Improves net work dat a t ransmission speed t hru loopback net work 21

22 Loading Process Daily, more than 9 3 millions rows of data are: Anonymized Transferred t o cent ral server Int egrat ed on Informix dat abase: Insert ed or Updat ed Scheduled increase of over 20% of rows in 2015 int egrat ing new sources of dat a DataMart's are ref reshed daily by t ransferring dat a int o IWA 3:5 5 Hours t o load dat a t o IWA: 55%is t ransfer t ime 45%is IWA compression t ime Test ed Table Fragment at ion and Part it ion Refresh Put in product ion in Oct

23 Dat abase and Dat amart definit ion Inf ormix IDS 1.5 TB allocat ed in chunks 85 0 Gbytes for Fact t able Mainly 16 KB page sizes 2 5 GB of RAM allocat ed for Buffers Diagnostics Datamart GB of RAM used on IWA once loaded 1.3 billion rows in Fact t able 116 dimension tables wit h ~2 2 million rows Dat amart dimension t ables relat ionship t ree is 5 levels deep 23

24 Informix IWA Ut ilizat ion St at ist ics (1) Analyt ic Throughput Volume 8,0 0 0 User Volume 15 Users 23 Average IWA Request s per day Up t o 2 0,0 0 0 request s per day Processes using IWA Analyt ics Perf ormance 1m 4 5 s Average Response t ime min Max response t ime 24

25 Informix IWA Ut ilizat ion St at ist ics (2) Analyt ic Dashboards Perf ormance 4 8x Fast er dashboard queries: from 2 4 hr down t o ½ hr Data Visibility and Usabilit y 10 0 s BI and statist ical data source 3, Of Employees across all levels consume Deist er DBOlap Cubes and own set of st at ist ics generat ed by IDS+IWA Pages of st at ist ics per day are generat ed and pushed t o Document Management Syst em 25

26 User experience (from a Part ner s point of view) Labco is a fast changing company and big increase in complexit y Init ial concern of t he DWH project was on t echnical issues, t hen on dat a qualit y and next on dat a reliabilit y A project needing new IWA t echnology requires a big effort and bidding for success Users became more responsive and confident in t he project as t hey saw more of t he PoC result s Aft er 12 mont hs of PoC development and user ut ilizat ion, changing t o final expect ed product was as easy as changing name Now users make use of IWA t echnology on a daily basis, and t rust t he dat a Wit h new informat ion becoming available as fast as IWA can deploy it, users are finding new requirement s and new sources of dat a t o analyze 26

27 Aft er dat a warehouse int egrat ion success Once laborat ory dat a have been int egrat ed and a common corporat e MDM process creat ed, and mapping of ent it ies concept has been t est ed as a good way t o go, a number of new project s have been creat ed around t he DWH: Human resources analyt ics Financial journal analyt ics Event s and websit es analyt ics Wit hout IWA, t his project could not been complet ed successfully. Delivering accurat e informat ion of t he st at us of a Pan-European group t o t he decision makers st aff in minut es was impossible previously. Now wit h IWA, a lot of new informat ion is available and could be analyzed at t he moment it 's required Labco IT Manager 27

28 Final Q& A for Labco Project What was t he key for Labco t o hire Deist er for t he project? Mainly it w as t he IT expert ise. The abilit y t o underst and t he t echnical issues and creat e t echnical solut ions quickly Proof of concept init ial phase was key for cust omer t o get bet t er know ledge of Deist er s abilit y t o address t echnical issues How was t he cust omer convinced t o use IDS/IWA vs compet it or t echnologies? Thru a complet e proof of concept process. Users were able t o see an almost done solut ion. Technology just became a t ool What did IBM do t o help wit h t he project? Full Technical and Lab support, by quickly resolving issues found in IWA Providing IWA soft w are for t he PoC period, even t hough it t ook > 3 mont hs New feat ures t hat could IDS/IWA have wit h t his experience? Allow part it ion refresh when using synonyms or views as dat a sources Allow view s of synonyms bet ween dat abases case insensit ive as well as non case insensit ive 28

29 FedeFarma Spanish Pharmacy Product s dist ribut or 29

30 Who is Fedefarma? Fedef arma is a pharmacy goods distribut or Supplies more t han 9,0 0 0 pharmacy st ores $9 6 8 million revenue More t han 3 5 0,0 0 0 units delivered daily 3 8,0 0 0 miles t raveled daily t o deliver goods 4 times a day 7 warehouses wit h 6 4 0,0 0 0 square f oot 5 8,0 0 0 SKUs Websit e: ht t p://

31 Fedefarma Hist ory Founded in Barcelona, Spain 1928 Siemens BS & BS equipment Expansion t o t he rest of Spain 7 new warehouses 31

32 Fedefarma Hist ory New modernizat ion project 2006 Inf ormix IW A project f or dat a w arehouse creat ion Hardw are updat e Deist er w ins project and deploy Linux HW and IBM Informix IBM pseries Expansion of company. - Increased number of cust omer st ores f rom 3,0 0 0 t o 9, Increased logist ics requirement s from delivering goods from east ern Spain warehouses t o all count ry 32

33 Hardware Schema PC Pharmacies Int ernet PC Load Balancers Load Balancers Dat a Part ners Web Servers Programs Dat a Warehouse Web Services Servers Users Web Servers Net work MPLS Knapp / Shaff er Tasks Servers Mini-Giff Storehouses DMX 33 PC PC

34 Original IT Scenario Inf ormix IDS used as main dat abase core syst em 2 main p-series servers 16 small Linux servers for dist ribut ed processing replicat ed wit h ER and HDR t echnologies Dat a w arehouse system developed and deployed f or analyt ics Informix dat abase wit h staging t ables and consolidat ion processes execut ed daily Main Fact t ables had million row s using GB Staging accumulated tables and indexes required for product ion goes t o a t ot al of 4 TB of disk used Small queries needed for invoicing and ot her process st at ist ics run very fast as t hey were using aggregat ed t ables and indexed access Analyt ics request s could only be done thru predef ined report s (t o meet operat ional requirement s). Some report s took more than 2 hours t o run 34

35 What Fedefarma want ed from t he new DWH 35

36 Cust omer Concerns Aft er some int ernal t est ing and parallel deployment of IWA t echnologies t o compare wit h current DWH syst em: IT managers were excited w ith the IW A technology and underst ood t he benefit s of using IWA: Improve perf ormance f or all queries Reduce disk requirements in about 3+3 = 6 TB CEO & Financial st aff Spending a lot of money in IT t oy seems not rat ional Company was working properly and dat a is ret rieved in a reasonable amount of t ime W e need to make a real PoC project, put t ing t o work IWA in a real environment t o demonst rat e why cust omer should buy it Informix IWA will allow not only current static reports and statistics, but also allow t o new ones t o be developed on the f ly Thanks t o Informix IWA, dat a knowledge could be explored in real time 36

37 Proof of Concept Proof of Concept was designed, developed and deployed as a final project Bot h t he original DWH syst em and t he new IWA based DWH syst em were loaded at t he same t ime (in parallel) User and Process queries were swit ched t o IWA Performance st at ist ics collect ion was developed t o prove workload and st orage requirement reduct ion Prior t o deploying IWA, cust omer opt imized performance by using smaller queries, e.g. one query per 6000 users => 6,000 queries t o IDS Wit h IWA, a single query was issued for all users 37

38 Project Specificat ions Hardware 8 cores Int el E Dat amart s: 1st wit h 5 10 million rows Fact t able 2nd wit h 15 0 million rows Fact t able Informix t echnologies used: ER, Rolling windows fragment at ion for hist orical maint enance, Dat a Compression for reducing size of DB, IWA Physical st orage ut ilizat ion: 180GB (full dat abase wit h st orage opt imizat ion), ~500GB wit hout st orage opt. 38

39 Project Success Aft er deploying IWA, st orage space was reduced f rom TBytes to 180 GB Perf ormance gains were signif icant in all report s and processes: Global sales st at ist ics calculat ion went f rom 1:0 0 H t o just 1 min 4 5 seconds (3 2 times f ast er) Invoicing process goes from 2 4 H to 12 H (t wice f aster) More users are performing analyt ics queries Analyt ics queries are offloaded from p-series servers reducing 2025% CPU ut ilizat ion and allowing addit ional capacit y in main dat abase Aft er DWH migrat ion t o IWA, new st ore analyt ics has been deployed successfully 39

40 Informix IWA Ut ilizat ion St at ist ics (1) Analyt ic Throughput Volume 5 0,0 0 0 Average IWA Request s per day User Volume 30 Users 83 Processes using IWA Analyt ics Perf ormance 32x Fast er Global Sales st at ist ics, from 1 hour down t o 1m 4 5 s. Invoicing process is 2 x f aster 40

41 Informix IWA Ut ilizat ion St at ist ics (2) Data Storage Reduction 6x Green Technology 25% St orage savings wit h St orage Opt imizat ion/compression: From Tbytes down t o 180 GB Less server CPU consumpt ion More server capacit y available 41

42 Cust omer Feedback Previous DWH needs a lot of st aging t ables due t o performance issues. Wit h IWA, we've dropped all St aging requirement s and convert ed our old disk eat ing DWH t o 500GB on disk. Wit h IWA, our invoicing syst em has dropped dramat ically t he t ime required from 24H t o just 12H. Santi Pla (Fedef arma) 42

43 Final Q& A for Fedefarma Project What was t he key for Fedefarma t o hire Deist er for t he project? Mainly it was t he IT expert ise. They already know Deist er expert ise in previous Informix IDS project s PoC was key for financial st aff t o approve budget for t he project How was t he cust omer convinced t o use IDS/IWA vs ot her compet it or t echnologies? This is an Informix cust omer since A good PoC was key in gaining budget approval What did IBM do t o help wit h t he project? Tot al support of t echnical and support st aff resolving issues found in IWA very quickly Providing IWA soft ware for t he PoC period New feat ure requirement s for IDS/IWA wit h t his experience? Dat amart requiring more t han 750 columns 43

44 Quest ions? Cont act : vsalvador@deist er.es 44