Join af tabeller med SAS skal det være hurtigt?

Transcription

1 Join af tabeller med SAS skal det være hurtigt? Henrik Dorf, chefkonsulent, PS Commercial

2 Join af tabeller Skal det være hurtigt kræver det Valgmuligheder Viden Eksperimenter

3 Historien En af de første ting, jeg hørte om SAS DATA step, var: SAS kan blande, flette og sammensætte data på alle måder, man kan tænke sig også alle de måder som det nye SQL tilbyder. Og endda flere... (1985)

4 Valget mellem DATA step eller SQL SAS DATA step 1972 SAS-udviklet 4. generationsværktøj til databehandling Ofte anvendt i forbindelse med store datamængder Kræver færre computerressourcer Forudsætning: Hold data sorteret Structured Query Language 1979 Sprog designet til relationelle databaser Skræddersyet til transaktionsbaserede systemer Findes i forskellige varianter Kræver ofte flere ressourcer, men er afhængig af implementering

5 Hvilke joins findes? Type SAS DATA step SQL Konkatenering Set Union Flette Set by Union order by Kombinere Merge Join Update Merge by Update Modify Join Left join Right Join Full join Update + Insert Split Data output1-n Create table table1 Tabelopslag Formater: Keyed set med KEY= HASH-objekt Subselect join

6 Konkatenering A B Obs k b c data D ; Set A B ; run; proc sql ; create table S as select * from a union select * from b ; quit; b

7 Flette A B Obs k b c data D ; set A B ; By K; run; b proc sql ; create table S as select * from a union select * from b order by k; quit;

8 Merge / join match merge A B data D1; merge a b ; by k ; run; b Obs k b c proc sql ; create table S1 as select * from a join b on a.k=b.k; quit; Obs k a b c

9 Merge / join ikke match A B Obs k b c proc sql ; data D; create table S as select * merge a b ; by k ; run; from a join b on a.k=b.k; quit; b b

10 Hvilken join? Join er mange ting (Inner) join Right join Left join Full join Cross join Som alle giver forskellige resultater

11 Join Inner A B Obs k b c proc sql ; data D; merge a b ; by k ; run; b create table S as select * from a inner join b on a.k=b.k; quit; b

12 Join Inner (default) A B Obs k b c proc sql ; data D; merge a b ; by k ; run; create table S as select * from a,b where a.k=b.k; quit; b b

13 Join Right A B Obs k b c proc sql ; data D; create table S as select * merge a b ; by k ; run; from a right join b on a.k=b.k; quit; b b

14 Join Left A B Obs k b c proc sql ; data D; merge a b ; by k ; run; create table S as select * from a left join b on a.k=b.k; quit; Obs K A c b b

15 Join Full A B Obs k b c proc sql ; data D; merge a b ; by k ; run; b create table S as select * from a full join b on a.k=b.k; quit; b

16 Join Cross A data D; merge a b ; by k ; run; Obs K A c b B Obs k Obs b k c a c b proc sql ; create table S as select * from a cross join b on a.k=b.k; quit;

17 Join Cross A B Obs k Obs b k c a c b data D; merge a b ; by k ; run; b proc sql ; create table S as select * from a cross join b on a.k=b.k; NOTE: The execution of this query involves performing quit; one or more Cartesian product joins that cannot be optimized

18 Cartesian product Meget ressourcekrævende på memory (og paging) NOTE: The execution of this query involves performing one or more Cartesian product joins that cannot be optimized

19 Optimering I/O Anvendte ressourcer er Inputtabeller Outputtabel A S B Utility files

20 Optimering Hvad kan vi måle? Inputtabeller Outputtabel Pladsforbrug A S B Utility files

21 Optimering Speciel monitorering Inputtabeller Outputtabel A S B Utility files

22 Optimering Working set-måling Udviklet til formålet

23 Optimering Working set-måling Udviklet til formålet Måler størrelsen på foldere eller filer Skriver logfil 1:,Count,Date,Time,D:\SASWORK\_TD6600,D:\SASUTIL\sas_util C8_kohdoxp5,D:\SASWORK\_TD6600\a.sas7bdat*,D:\SASWO RK\_TD6600\s.sas7bdat.lck 2:,0, ,180228, ,0, ,0 2:,1, ,180229, ,0, ,0 2:,2, ,180230, ,0, ,0 2:,3, ,180231, ,0, ,0 2:,4, ,180232, ,0, ,0 2:,5, ,180233, ,0, ,0 Stopper, når sidste målepunkt forsvinder D:\SASWORK\_TD6600\s.sas7bdat.lck

24 Optimering Demo

25 Workspace SQL: Join

26 Workspace SQL: Join WORK.S SAS Utility Work.a

27 Workspace DATA step: Merge

28 Workspace SQL: Join + DATA step: Merge

29 Workspace SQL: Union

30 Workspace DATA step: Set

31 Workspace SQL: Union og DATA step: Set

32 SAS emulerer SQL På observationer Inner join data D1; merge a(in=f1) b (in=f2); by k ; if F1 and F2 ; run; Left join Right join data D1; merge a(in=f1) b (in=f2); by k ; if F1 ; run; data D1; merge a(in=f1) b (in=f2); by k ; if F2 ; run;

33 Split A data D1 D2 D3 ; SET A ; Proc sql ; Create table D1 as select * from a where K=1 ; Create table D2 as select * from a where K=2 ; Create table D3 as select * from a where K>2 ; Run; IF K=1 then output D1 ; ELSE IF K=2 then output D2; Else output D3;; run; Obs a c Obs a c 1 Obs 1 1 a c Obs a c Obs a c 1 Obs 1 1 a c

34 Split A data D1 D2 D3 ; SET A ; Proc sql ; Create table D1 as select * from a where K=1 ; Create table D2 as select * from a where K=2 ; Create table D3 as select * from a where K>2 ; Run; IF K=1 then output D1 ; ELSE IF K=2 then output D2; Else output D3;; run; Obs a c Obs a c 1 Obs 1 1 a c Obs a c Obs a c 1 Obs 1 1 a c

35 Update SQL kræver grundig forberedelse data A ; update A B ; By k; run; Proc SQL; update.. ; Insert.; run;

36 Update data A ; update A B ; By k; run; Proc SQL; update.. ; Insert.; run;

37 Update test Nemt at teste SAS -program SQL kræver ny tabel data A _test; run; update A B ; By k;

38 Opsummering Ingen metode er universel SQL Er hurtig at konstruere Er standard for RDBMS Indeholder implicit funktionalitet Kan være hurtigere DATA step Egen notation Mere fleksibel Kan være hurtigere

39 Husk altid Hold orden (= data sorteret)

40 Mere information

41 Mere information

42 Henrik Dorf