Enabling Continuous Delivery with Database Practices

Transcription

1 Enabling Continuous Delivery with Database Practices Pramod ThoughtWorks Inc.

2 Agenda Why Continuous Delivery Collaboration with the data team Automation for data Refactoring databases Deploying database changes

3 Why Continuous Delivery?

4 To get fast feedback from users, release frequently

5 Reduce risk of releasing

6 Learning and responding to customer needs is critical

7 Achieve Continuous Delivery

8 Close collaboration between everyone involved in delivery

9 Extensive automation of all parts involved in delivery process

10 Reduce the gap between development and operations

11 How does continuous delivery apply to databases?

12 Collaboration techniques

13 Pair the DBA s and Developers

14

15 Version control all database scripts and artifacts

16

17 Automation techniques

18 Let developers provision application database

19

20

21

22

23

24

25

26

27

28

29

30

31 Andy Jai Laptop/Desktop Application Database

32 Continuously Integrate database changes

33 Database changes applied by DBA Team Check in application code and database migration scripts Local dev environment Integration environment PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine PROD UAT QA Environment Dev DB Update and build Dev Database

34 Check in application code and database migration scripts Local dev environment Integration environment PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine PROD UAT QA Environment Dev DB Update and build Dev Database

35 Check in application code and database migration scripts Local dev environment Integration environment Migration scripts (DBA) PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine PROD UAT QA Environment Dev DB Update and build

36 Check in application code and database migration scripts Local dev environment Migration scripts (DBA) Migration scripts Integration environment PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine PROD UAT QA Environment Dev DB Update and build Integration Database

37 Check in application code and database migration scripts Local dev environment Migration scripts (DBA) Migration scripts Integration environment PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine PROD UAT QA Environment Dev DB Update and build Integration Database

38 Check in application code and database migration scripts Local dev environment Migration scripts (DBA) Migration scripts Integration environment PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine PROD UAT QA Environment Dev DB Update and build Integration Database Apply migration scripts

39 Check in application code and database migration scripts Local dev environment Migration scripts (DBA) Migration scripts Integration environment Package Migration scripts PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine Artifacts War Jar PROD UAT QA Environment Dev DB Update and build Integration Database Apply migration scripts

40 Check in application code and database migration scripts Local dev environment Migration scripts (DBA) Migration scripts Integration environment Package Migration scripts Apply Migration scripts PROD UAT QA ANT Maven Gradle Rake Source Control Continuous Integration Engine Artifacts War Jar PROD UAT QA Environment Dev DB Update and build Integration Database Apply migration scripts War Jar

41 Benefits of CI with databases

42 Benefits of CI with databases Test application code and database at one place

43 Benefits of CI with databases Test application code and database at one place Generate code and database artifacts

44 Benefits of CI with databases Test application code and database at one place Generate code and database artifacts Integrate application and database changes in an independent environment

45 Benefits of CI with databases Test application code and database at one place Generate code and database artifacts Integrate application and database changes in an independent environment Show current state of application and database to all

46 DB artifacts in CI application artifact for build <<n>> database artifact for build <<n>>

47 Deployment

48 Deployment Database migration/upgrade should be a development time task not deployment time task

49 Deployment Database migration/upgrade should be a development time task not deployment time task Package all the migration scripts, during Continuous Integration cycle

50 Deployment Database migration/upgrade should be a development time task not deployment time task Package all the migration scripts, during Continuous Integration cycle Apply the migration scripts

51 Deployment Database migration/upgrade should be a development time task not deployment time task Package all the migration scripts, during Continuous Integration cycle Apply the migration scripts Deploy frequently to reduce risk

52 Database refactoring

53

54 A database refactoring is a small change to your database schema (the DDL, data, and DB code) which improves its design without changing its semantics.

55 A database refactoring is a small change to your database schema (the DDL, data, and DB code) which improves its design without changing its semantics.

56 A database refactoring is a small change to your database schema (the DDL, data, and DB code) which improves its design without changing its semantics. A database refactoring improves its design while retaining both its behavioral and informational semantics.

57

58 Timeline of Change

59 Timeline of Change Deploy new changes, migrate data, put in scaffolding code refactoring{ Start Implement the Expand

60 Timeline of Change Deploy new changes, migrate data, put in scaffolding code refactoring{ Start Implement the Transition Transition Period (old and new){ Expand

61 Timeline of Change Deploy new changes, migrate data, put in scaffolding code Remove old schema, scaffolding code Start Transition End refactoring{ Implement the Transition Period (old and new){ completed{ Refactoring Expand Contract

62 Keeping Old and New alive

63 Keeping Old and New alive DB Should be able to handle multiple versions of the application

64 Keeping Old and New alive DB Should be able to handle multiple versions of the application Create Interfaces in the database

65 Keeping Old and New alive DB Should be able to handle multiple versions of the application Create Interfaces in the database Wrap tables with views

66 Keeping Old and New alive DB Should be able to handle multiple versions of the application Create Interfaces in the database Wrap tables with views Create calculated columns

67 Keeping Old and New alive DB Should be able to handle multiple versions of the application Create Interfaces in the database Wrap tables with views Create calculated columns Create triggers to sync data

68 Expand Contract an example

69 Expand Contract an example Start name = Pramod Sadalage

70 Expand Contract an example Start name = Pramod Sadalage Expand name = Pramod Sadalage firstname = Pramod lastname = Sadalage

71 Expand Contract an example Start name = Pramod Sadalage Expand name = Pramod Sadalage firstname = Pramod lastname = Sadalage Contract firstname = Pramod lastname = Sadalage

72 More on expand contract

73 More on expand contract Start name = Pramod Sadalage

74 More on expand contract Expand Start name = Pramod Sadalage Without data migration name = Pramod Sadalage firstname = null lastname = null With data migration name = Pramod Sadalage firstname = Pramod lastname = Sadalage

75 More on expand contract Expand Start name = Pramod Sadalage Without data migration name = Pramod Sadalage firstname = null lastname = null With data migration name = Pramod Sadalage firstname = Pramod lastname = Sadalage

76 More on expand contract Expand Start name = Pramod Sadalage Without data migration name = Pramod Sadalage firstname = null lastname = null With data migration name = Pramod Sadalage firstname = Pramod lastname = Sadalage

77 More on expand contract Expand Start name = Pramod Sadalage Without data migration name = Pramod Sadalage firstname = null lastname = null With data migration name = Pramod Sadalage firstname = Pramod lastname = Sadalage Contract firstname = Pramod lastname = Sadalage

78 Simple scenario - DBDeploy ALTER TABLE Customer ADD firstname VARCHAR2(60); ALTER TABLE Customer ADD lastname VARCHAR2(60);! --//@UNDO! ALTER TABLE Customer DROP COLUMN firstname VARCHAR2(60); ALTER TABLE Customer DROP COLUMN lastname VARCHAR2(60);

79 With synchronized data ALTER TABLE Customer ADD firstname VARCHAR2(60); ALTER TABLE Customer ADD lastname VARCHAR2(60); CREATE OR REPLACE TRIGGER SynchronizeName BEFORE INSERT OR UPDATE ON Customer REFERENCING OLD AS OLD NEW AS NEW FOR EACH ROW BEGIN IF :NEW.Name IS NULL THEN :NEW.Name := :NEW.firstname ' ' :NEW.lastname; END IF; IF :NEW.name IS NOT NULL THEN :NEW.firstname := extractfirstname(:new.name); :NEW.lastname := extractlastname(:new.name); END IF; END; /!

80 Migrate and Synchronize data ALTER TABLE Customer ADD firstname VARCHAR2(60); ALTER TABLE Customer ADD lastname VARCHAR2(60); UPDATE Customer set firstname = extractfirstname (name); UPDATE Customer set lastname = extractlastname (name);! CREATE OR REPLACE TRIGGER SynchronizeName BEFORE INSERT OR UPDATE. //@UNDO UPDATE Customer set name = firstname lastname WHERE name IS NULL; ALTER TABLE Customer DROP COLUMN firstname; ALTER TABLE Customer DROP COLUMN lastname;

81 Contract ALTER TABLE Customer DROP COLUMN name;! ALTER TABLE Customer ADD name VARCHAR2(120); UPDATE Customer set name = firstname lastname WHERE name IS NULL;

82 Contract ALTER TABLE Customer DROP SET UNUSED COLUMN name;! When drop takes forever! ALTER TABLE Customer ADD name VARCHAR2(120); UPDATE Customer set name = firstname lastname WHERE name IS NULL;

83 Keep legacy apps happy ALTER TABLE Customer DROP COLUMN name; ALTER TABLE CUSTOMER ADD (name AS (generatename (firstname,lastname)) );! ALTER TABEL Customer DROP COLUMN name; ALTER TABLE Customer ADD name VARCHAR2(120); UPDATE Customer set name = firstname lastname WHERE name IS NULL;! Virtual column in oracle is computed for every read

84 Another example

85 Migration script ALTER TABLE custordr rename to customerorder;! CREATE OR REPLACE VIEW custordr AS SELECT custordrid, ponumber, ordrdt, shipdate, sptoadid FROM customerorder ; --//@UNDO! DROP VIEW custordr; ALTER TABLE customerorder RENAME TO custordr;

86 data in migrations

87 data in migrations INSERT INTO businessunit ( businessunitid, name, regionid ) VALUES ( 22, 'John Doe Services', 1 ); INSERT INTO businessunit ( businessunitid, name, regionid ) VALUES ( 23, 'Bob Products', 1 ); INSERT INTO businessunit ( businessunitid, name, regionid ) VALUES ( 24, 'Carr Machinery', 1 );

88 data in migrations INSERT INTO currency ( currencyid, name, code ) VALUES ( 2, 'Canadian Dollar', 'CAD' ); INSERT INTO currency ( currencyid, name, code ) VALUES ( 3, 'Australian Dollar', 'AUD' ); INSERT INTO currency ( currencyid, name, code ) VALUES ( 4, 'EMU Euro', 'EUR' );

89 data in migrations DELETE FROM contact ct WHERE NOT EXISTS (SELECT 1 FROM customer p WHERE ct.contactid=p.contactid) and EXISTS (SELECT 1 FROM (SELECT customerid, COUNT(*) FROM contact WHERE customerid IS NOT NULL GROUP BY customerid HAVING COUNT(*) >1) ct2 WHERE ct.customerid=ct2.customerid);

90 Tips

91 Tips Large refactorings are risky

92 Tips Large refactorings are risky Sequence of many small refactorings can create the desired change

93 Tips Large refactorings are risky Sequence of many small refactorings can create the desired change Migration scripts should be checked in and run on local dev/ci/qa/uat/prod etc.

94 Tips Large refactorings are risky Sequence of many small refactorings can create the desired change Migration scripts should be checked in and run on local dev/ci/qa/uat/prod etc. Changes to data are also migrations

95 Tracking Changes

96 Tracking Changes Each change is a delta/migration script

97 Tracking Changes Each change is a delta/migration script Migration scripts are development time activity not deployment time tasks

98 Tracking Changes Each change is a delta/migration script Migration scripts are development time activity not deployment time tasks Package migration scripts for automated deployment

99 Tracking Changes Each change is a delta/migration script Migration scripts are development time activity not deployment time tasks Package migration scripts for automated deployment Same scripts for: developers, QA, UAT and Production

100 Continuous Delivery Deployment should be easy ant -propertyfile qa.properties upgrade ant -propertyfile live.properties upgrade

101

102 sadalage.com databaserefactoring.com