SQL Stored mutsoc.be. socmut.be. GSE DB2, Leuven 28/11/2013. NVSM - Inf.Dir.Inf. - System MF - Page 1

Transcription

1 SQL Stored mutsoc.be socmut.be NVSM - Inf.Dir.Inf. - System MF - Page 1

2 AGENDA About us Stored Procedures Programming The Good, The Bad, NVSM - Inf.Dir.Inf. - System MF - Page 2

3 AGENDA About us Stored Procedures Programming Our Business Our Infrastructure Our Applications Our Challenges The Good, The Bad, NVSM - Inf.Dir.Inf. - System MF - Page

4 Our business Health insurance and social services NVSM - Inf.Dir.Inf. - System MF - Page 4

5 Our business: some statistics Socialistische Mutualiteiten 11 health funds (federations) 1 Landsbond / Union 600 branches Socio-cultural organizations UNMS / NVSM Union Nationale des Mutualités Socialistes Nationaal Verbond der Socialistische Mutualiteiten 644 employees / 176 in IT / 8 system engineers mainframe Centralized services and administration For most of the federations Single point of contact for other social security organizations Market share: 28,17% (2011) members (2011) NVSM - Inf.Dir.Inf. - System MF - Page 5

6 Our mainframe infrastructure Hardware Dual site: Active / Passive 2 x z114 2 x DS8100 (PPRC) 2 x 3584 Tape Library Interconnecting DWDM Application software PL/1 V4.3 IBM Datastudio V3.1.1 Rational Developer for System z V8.5 COGNOS BI V10 System software z/os V1.13 CICS TS V4.1 DB2 z/os V10 DB2 Connect 10.1 IBM DB2 Cloning Tool Management software RMF CA Insight DB2 CA Sysview CA RC/Migrator IBM Tivoli Decision Support NVSM - Inf.Dir.Inf. - System MF - Page 6

7 Our application environment 2 LPARs, 1 sysplex 3 user environments (T,A,P) + 2 system environments (D, I) + 1 sandbox per developer all 2-way data-sharing Applications Online applications Terminal/GUI/Services CICS (PL/1).NET (PL/1 and SQL/PL) Batch applications (PL/1) DB2 Production 66 databases / schema s 962GB compressed 1.884GB uncompressed NVSM - Inf.Dir.Inf. - System MF - Page 7

8 Our current DB2 challenges Keep up With migrating offline to online transactions With new projects (e-health, ) With new versions and maintenance With our agile developers And other opportunities Planning DB2 V Q4 NVSM - Inf.Dir.Inf. - System MF - Page 8

9 AGENDA About us Stored Procedures Background External Native Programming The Good, The Bad, NVSM - Inf.Dir.Inf. - System MF - Page

10 Background: why stored procedures? Early adopter (2005, DB2 V6) Modernization of user interface Choice for.net front-end Stored procedure as application integration protocol From C# and DB2 Client via DB2 Connect and DDF Easy to integrate in client code Static SQL Traditional programming and deployment Can be written in PL/1 Can wrap around existing PL/1 code Not very different from traditional routines Security GRANT EXECUTE ON PROCEDURE NVSM - Inf.Dir.Inf. - System MF - Page 10

11 Background: workload and performance? TIP: Put sufficient intelligence in a stored procedure NVSM - Inf.Dir.Inf. - System MF - Page 11

12 Background: workload and performance?? April 1 DB2 V10CM9? Workload activity DDF service class NVSM - Inf.Dir.Inf. - System MF - Page 12

13 Background: workload and performance March 1? z10 z114! Workload activity DDF PROD reporting classes NVSM - Inf.Dir.Inf. - System MF - Page 13

14 External stored procedures Written in PL/1 External source & load module Definition in DB2 Catalog create procedure SYSPROC.TBV8830 ( in vlog_header varchar(200), out return_code char(5), out sql_code integer, in p_fede smallint, in p_vzw char(5), in p_datv date, in p_datt date ) language pli reads sql data dynamic result sets 1 external name TBV8830; NOTE: External stored procedures can be written in many languages (C,Cobol, ) and also in the SQL Procedural Language which we discuss later. External SQL stored procedures require a C compiler to create a load module. This is not an option in our environment. NVSM - Inf.Dir.Inf. - System MF - Page 14

15 Native stored procedures Written in PL/1 External source & load module Definition in DB2 Catalog create procedure SYSPROC.TBV8830 ( in vlog_header varchar(200), out return_code char(5), out sql_code integer, in p_fede smallint, in p_vzw char(5), in p_datv date, in p_datt date ) language pli reads sql data dynamic result sets 1 external name TBV8830; Written in SQL language No load module Source is in the DB2 definition create procedure SYSPROC.TBV8930 ( in vlog_header varchar(200), out return_code char(5), out sql_code integer, in p_fede smallint, in p_vzw char(5), in p_datv date, in p_datt date ) language sql reads sql data dynamic result sets 1 MAIN: begin... END MAIN NVSM - Inf.Dir.Inf. - System MF - Page 15

16 Native stored procedures: why? Lack of motivated PL/1 developers to keep up The Dinosaur prejudice Growing.NET developer team.net developers want more than just do presentation logic to increase their productivity Code both presentation and data access logic Looking for more ziip eligible workload Save CPU cost Versioning support We use V1 only (no versioning in PL/1 either) Better code portability But we have no porting plans Future potential NVSM - Inf.Dir.Inf. - System MF - Page 16

17 External vs Native stored procedures External SP s: robust scalable infrastructure DB2 DIST DDF WLM Region DB2 DBM1 Enclave SRB TCB PL/1 Stored Procedure Load module Stored Procedure Package NVSM - Inf.Dir.Inf. - System MF - Page 17

18 External vs Native stored procedures Native SP s: lightweight infrastructure No task switching savings DB2 DIST DB2 DBM1 DDF Enclave SRB Enclave SRB SQL Stored Procedure Package (includes logic) Code may have longer pathlength spendings NVSM - Inf.Dir.Inf. - System MF - Page 18

19 AGENDA About us Stored Procedures SQL/PL Source management Development tools Programming The Good, The Bad, NVSM - Inf.Dir.Inf. - System MF - Page

20 What is SQL/PL SQL Procedural Language External SQL procedures Native SQL procedures Triggers User-defined functions Only has SQL statements And some SQL control statements (while, if, loop, goto) Runs inside the DB2 data management server (DBM1) SQL PL Engine integrated into RDBMS NVSM - Inf.Dir.Inf. - System MF - Page 20

21 An example create procedure TBV893U ( in vlog_header varchar(200), out return_code char(5), out sql_code integer, in p_fede smallint, in p_vzw char(5), in p_datv date, in p_datt date ) language SQL -- reads SQL data modifies SQL data asutime limit 2000 commit on return no -- dynamic result sets 1 dynamic result sets 2 MAIN: begin declare user char(8); declare sqlstate char(5); declare sqlcode integer; declare debug integer default null; declare sysprint cursor with return for select text from sysibm.sysprint order by seqno asc; TEST_VZW1: begin --***************************************** --Test if VZW is valid numeric value --***************************************** declare minus420 condition for sqlstate '22018'; declare continue handler for minus420 begin -- This is condition where char does not -- convert to numeric set sql_code = sqlcode; set return_code = 'C001'; end; if P_VZW is not a valid number, it sets the -- above minus420 (sqlcode-420) condition if (p_vzw = 'ALL') then goto CONTROLE_NOK; elseif cast(p_vzw as decimal(5,0))<0 then set return_code = 'C002'; end if; end TEST_VZW1; NVSM - Inf.Dir.Inf. - System MF - Page 21

22 An example (continued) TEST_VZW2: begin --*************************************** --Test if VZW is valid business value --*************************************** declare descr char(20); if (return_code<>'00000') then goto CONTROLE_NOK; end if; select sc0102 into test_vzw2.descr from v00e0001 where as0201 = p_fede; if (sqlcode<>0) then set sql_code = sqlcode; set return_code = 'C002'; end if; end TEST_VZW2; set debug = putskip(debug, INPUT OK ); -- ******************* CONTROLE_NOK: begin -- ******************* if (return_code<>'00000') then goto BYE_BYE; end if; end CONTROLE_NOK; TIP: Find a naming convention to distinguish column names from variable names in SQL/PL -- ******************* CONTROLE_OK: begin -- ******************* declare c1 cursor with return for select ; declare c2 cursor with return for select ; if (p_vzw = 'ALL') then open c1; set sql_code = sqlcode; else open c2; set sql_code = sqlcode; end if; end CONTROLE_OK; -- ******** BYE_BYE: begin -- ******** end BYE_BYE; set debug = putskip(debug, All done! ); if v_skips>0 then open sysprint; end if; END MAIN# TIP: You can exploit SYSIBM.SYSPRINT for debugging purposes NVSM - Inf.Dir.Inf. - System MF - Page 22

23 An example (for the interested reader) create function putskip (p_count integer, p_message varchar(4096)) returns integer deterministic no external action modifies sql data begin declare v_recnr integer default 0; declare v_index integer default 1; declare v_part varchar(254) ; declare v_last integer default 0; if p_count is not null then set v_recnr = p_count; end if; if p_message is not null then set v_last = length(p_message); end if; while v_index <= v_last do set v_recnr = v_recnr + 1; if v_index >= v_last then set v_part = substr(p_message,v_index,254); else set v_part = substr(p_message,v_index) ; end if; insert into sysibm.sysprint values(v_recnr, v_part); set v_index = v_index + length(v_part); end while; return v_recnr; end# NVSM - Inf.Dir.Inf. - System MF - Page 23

24 Source management: PL/1 stored procedures 4 objects that should be in sync: Program source code Source code library Source moves with deployment (Test Accp Prod) Load modules Load module library Recreated in each environment (Test, Accp, Prod) DB2 package DB2 Directory Recreated in each environment (Test, Accp, Prod) Stored procedure DDL DDL library (& DB2 catalog) Source recreated in each environment (Test, Accp, Prod) Responsabilities Developer Can independently change & test program, and deploy Cannot define the stored procedures (Test, Accp, Prod) Operations (TWS) Deploys (automated) the program & package (Test Accp Prod) DBA Deploys (manual) the stored procedure (Test, Accp, Prod) Query tuning review (Accp) Conditions for SQL/PL Stored Procedures 1. Sources must be in a library 2. Developer must be able to change and test independently 3. Automation must be possible for the deployment NVSM - Inf.Dir.Inf. - System MF - Page 24

25 Source management: SQL/PL stored procedures Just 2 objects Source DDL statement: contains program logic DB2 package: automatically created at CREATE Developer Can CREATE and ALTER procedures in his own SCHEMA Can setup tables etc. in his own SCHEMA (a.k.a. SANDBOX) Must use &SYSUID..STORPROC.PDS as the source library Member name is 7 chars nnn893x Must prepare DDL for deployment to Test as member nnn893xt DBA Verifies and corrects the CREATE statement in nnn893xt Deploys in each environment (Test, Accp, Prod) Query tuning review (Accp) Operations No automation yet (neither for CREATE of PL/1 SP) NVSM - Inf.Dir.Inf. - System MF - Page 25

26 Source management: SQL/PL stored procedures Just 2 objects Source DDL statement: contains program logic DB2 package: automatically created at CREATE Developer Can CREATE and ALTER procedures in his own SCHEMA Can setup tables etc. in his own SCHEMA (a.k.a. SANDBOX) Must use &SYSUID..STORPROC.PDS TIP: Set a standard for as SQL the source stored library Member name procedure is 7 chars options, nnn893x coding templates, and provide an auto-correct tool DBA Verifies and corrects the CREATE statement in nnn893xt Deploys in each environment (Test, Accp, Prod) Query tuning approval (Accp) Must prepare DDL for deployment to Test as member nnn893xt Operations No automation yet (neither for CREATE of PL/1 SP) NVSM - Inf.Dir.Inf. - System MF - Page 26

27 Development tools SPUFI Set your SQL Terminator different from ; NVSM - Inf.Dir.Inf. - System MF - Page 27

28 Development tools Rational Developer for System z NVSM - Inf.Dir.Inf. - System MF - Page 28

29 Development tools FIXSQLPL Home produce to format code to (some of the) standards Verifies source member naming convention Verifies MAIN as label for the program Enforces CREATE options LANGUAGE SQL QUALIFIER OWNER DYNAMIC RESULT SETS ASUTIME COMMIT ON RETURN NO Enforces mandatory EXIT HANDLER Only 1 EXIT HANDLER, and we enforce it to catch & log exceptions Verifies mandatory signature IN VLOG_HEADER VARCHAR(200) OUT RETURN_CODE CHAR(05) OUT SQL_CODE INTEGER Only 1 output parameter Some other minor verifications Converts from Sandbox-syntax to Test-syntax for deployment NVSM - Inf.Dir.Inf. - System MF - Page 29

30 AGENDA About us Stored Procedures The Good The Bad Our tips Programming The Good, The Bad, NVSM - Inf.Dir.Inf. - System MF - Page 30

31 The Good (1) Easy deployment No compile / link / bind DROP / CREATE Only 1 object to manage Suites both development via ISPF and graphical IDE Agile You may also need ALTER PROCEDURE REGENERATE ACTIVE VERSION Kind of REBIND, but also takes SQL control statements into account Usually APAR HOLD action BIND PACKAGE(remote_location.collid) DEPLOY(local.package) To leave source in one environment and create the executable in another NVSM - Inf.Dir.Inf. - System MF - Page 31

32 The Good (2) Successful roll-out in new projects SYSSTAT NVSM - Inf.Dir.Inf. - System MF - Page 32

33 The Good (3) Unified Debugger in IBM DataStudio Add options on CREATE PROCEDURE ALLOW DEBUG MODE WLM ENVIRONMENT FOR DEBUG MODE your_wlm_environment Needs DB2-supplied stored procedures See installation job DSNTIJSD Debug users must be granted permission GRANT DEBUGSESSION In IBM DataStudio, go to Window>Preferences>Run/Debug>Routine Debugger>IBM and choose the Session Manager option (we use built-in, but can also run on host) May need to adjust ASUTIME on your stored procedure NVSM - Inf.Dir.Inf. - System MF - Page 33

34 The Good (4) CPU reduction? (1) It depends Our observations are as follows If processing logic is trivial, then SQL/PL wins If more complex logic, and best practices are applied, then PL/1 wins Our conclusion CPU cost behaves sufficiently well to continue with SQL/PL To illustrate: Basic processing logic Declare a cursor where result depends on input parameter Either Return a result set with just opening the cursor (trivial processing) Return an output parameter by looping through the cursor (non-trivial processing) NVSM - Inf.Dir.Inf. - System MF - Page 34

35 The Good (4) CPU reduction? (2) NVSM - Inf.Dir.Inf. - System MF - Page 35

36 The Good (5) ziip eligibility (1) Note that also a portion of PL/1 stored procedures is ziip-eligible NVSM - Inf.Dir.Inf. - System MF - Page 36

37 The Good (5) ziip eligibility (2)!!! NVSM - Inf.Dir.Inf. - System MF - Page 37

38 The Good (5) ziip eligibility (3) NVSM - Inf.Dir.Inf. - System MF - Page 38

39 The Bad (1) SQLCODE testing: prone to loops! TIP: Enforce ASUTIME in TEST and set your WLM definitions 19/3/2013: Developer creates a program loop in TEST NVSM - Inf.Dir.Inf. - System MF - Page 39

40 The Bad (1) SQLCODE testing: prone to loops! Example: declare v_name char(8) ; declare v_name char(8) ; declare c1 cursor for select name from sysibm.sysdatabase fetch first 100 rows only with ur ; open c1; set p_sqlcode = 0; set p_result = ''; while p_sqlcode=0 do fetch c1 into v_name; if p_sqlcode<>0 then set p_rc = 'DONE!'; set p_sqlcode = sqlcode; else set p_result = p_result concat '-' concat v_name; set p_sqlcode = sqlcode; end if; end while; declare c1 cursor for select name from sysibm.sysdatabase fetch first 100 rows only with ur ; open c1; set p_sqlcode = 0; set p_result = ''; while p_sqlcode=0 do fetch c1 into v_name; set p_sqlcode = sqlcode; if p_sqlcode<>0 then set p_rc = 'DONE!'; else set p_result = p_result concat '-' concat v_name; set p_sqlcode = sqlcode; end if; end while; NVSM - Inf.Dir.Inf. - System MF - Page 40

41 The Bad (2) ziip eligibility (1) Not if called from CICS or Batch DB2 DIST DDF DB2 DBM1 TCB Address Space TCB Enclave SRB Enclave SRB Call SP SQL Stored Procedure Package (includes logic) not ziip-eligible ziip-eligible NVSM - Inf.Dir.Inf. - System MF - Page 41

42 The Bad (2) ziip eligibility (2) See the bigger picture Stored procedures may not be the major ziip opportunity NVSM - Inf.Dir.Inf. - System MF - Page 42

43 The Bad (3) Navigation by GOTO No support for copybooks No pl/1 %include-like equivalent No support for routines No support for existing PL/1 routines! No support for array variables Arrays are supported from V11! Can t wait! Each statement is an SQL statement SQLCODE value changes at each statement NVSM - Inf.Dir.Inf. - System MF - Page 43

44 Our tips Put sufficient intelligence in a stored procedure Find a naming convention to distinguish column names from variable names in SQL/PL You can exploit SYSIBM.SYSPRINT or equivalent for debugging purposes Set a standard for SQL stored procedure options, coding templates, and provide an auto-correct tool Enforce ASUTIME also in TEST and set your WLM definitions Distinguish your SQL/PL stored procedure performance records from PL/1 stored procedures RMF Workload Activity (SMF Type 72) records WLM service/reporting classes DB2 Accounting records Primary Authid / Correlation ID NVSM - Inf.Dir.Inf. - System MF - Page 44

45 THANK YOU! We welcome your questions! Stijn Van den Bosch Bart Verboven NVSM - Inf.Dir.Inf. - System MF - Page 45