Tobias Flohre codecentric AG. Ein Standard für die Batch- Entwicklung JSR-352

Transcription

1 Tobias Flohre Ein Standard für die Batch- Entwicklung JSR-352

2 Tobias Flohre blog.codecentric.de/en/author/tobias.flohre

3 AGENDA Grundlagen JSR-352 Properties, Job-Parameter, Restart, Skip, Retry und Listener Parallelisierung Implementierungen und Betriebsoptionen

4 AGENDA Grundlagen JSR-352 Properties, Job-Parameter, Restart, Skip, Retry und Listener Parallelisierung Implementierungen und Betriebsoptionen

5 WAS IST EIN BATCH? Traditionelles Batch-Pattern Read Process Write

6 BATCH HISTORIE 80er 90er Mainframe Batch: COBOL, PL/1 Java Batch WebSphere XD Compute Grid (BDS Framework) Spring Batch v1 Spring Batch v2 Spring Batch v3 JSR-352/JEE7 70/80er: Cobol, PL1 Java Batch: Gott Objekte, Spaghetticode, Java main() IBM WebSphere XD Compute Grid, Batch Datastream Framework (BDS) (2007/2008) Spring Batch 1.0 (April 2008) Spring Batch 2.0 (April 2009) JSR-352 Final Release Specification (Mai 2013)

7 JSR-352 HISTORIE IBM Chris Vignola 2011 Formierung Expert Group JSR-352 VMWare / Pivotal Michael Minella (Spring Batch Lead) Wayne Lund Final Release JSR Final Release JEE 7 Spring Batch Apache BatchEE

8 BATCH FRAMEWORK Restart Skip Persistente Job-Metadaten Retry Automatisches Transaktionsmanagement Skalierungsfeatures

9 DOMAIN / KONFIGURATION / ABLAUF Job Step Item ItemReader ItemProcessor ItemWriter Chunk

10 DOMAIN / KONFIGURATION / ABLAUF Job wird in XML erstellt (Job-XML) Domain Specific Language manifestiert sich in XML- Namespace Zentrale Elemente job step chunk reader processor writer item-count <job id="myjob" > <step id="mystep" > <chunk item-count="2" > <reader ref="myreader" /> <processor ref="myprocessor" /> <writer ref="mywriter" /> </chunk> </step> </job> Ablage unter META-INF/batch-jobs

11 DOMAIN / KONFIGURATION / ABLAUF Reader, Processor und Writer implementieren bestimmte Interfaces ItemReader Object readitem() ItemProcessor Object processitem(object item) ItemWriter void writeitems(list<object> items) Implementierungen nicht Bestandteil der Spec

12 DOMAIN / KONFIGURATION / ABLAUF Begin Step item == null list of items item-count reached Open transaction ItemReader Item readitem() ItemProcessor 1 processitem 2 (Item) ItemWriter writeitems (List<Item>) Commit transaction item == null at (1) false 3 true Finish Step

13 DOMAIN / KONFIGURATION / ABLAUF Inkasso Kraftfahrt Job * Step Inkasso Kraftfahrt am * JobInstance Inkasso Kraftfahrt am erster Versuch * JobExecution * StepExecution

14 DOMAIN / KONFIGURATION / ABLAUF Infrastrukturkomponenten BatchRuntime JobOperator public static JobOperator getjoboperator() public long start(string jobxmlname, Properties jobparameters) public void stop(long executionid) public List<Long> getrunningexecutions(string jobname) public JobExecution getjobexecution(long executionid) public Set<String> getjobnames() public List<JobInstance> getjobinstances(string jobname, int start, int count)

15 AGENDA Grundlagen JSR-352 Properties, Job-Parameter, Restart, Skip, Retry und Listener Parallelisierung Implementierungen und Betriebsoptionen

16 PROPERTIES UND JOB-PARAMETER <job id="job1"> <properties> <property name="base" value=".txt"/> </properties> <step id="step1"> <chunk> <properties> <property name="file" value="myfile#{jobproperties['base']}"/> </properties> </job> private String filename;

17 PROPERTIES UND JOB-PARAMETER <property name="filename" value="#{jobparameters['filename']}"/> <property name="filename" value="#{systemproperties['filename']}"/> <property name="filename" value="#{partitionplan['filename']}"/> public long start(string jobxmlname, Properties jobparameters)

18 RESTART public interface ItemReader { public void open(serializable checkpoint) throws Exception; public void close() throws Exception; public Object readitem() throws Exception; public Serializable checkpointinfo() throws Exception; } public interface ItemWriter { public void open(serializable checkpoint) throws Exception; public void close() throws Exception; public void writeitems(list<object> items) throws Exception; public Serializable checkpointinfo() throws Exception; }

19 RESTART Begin Step Aufruf open() list of items Aufruf checkpointinfo() Open transaction ItemReader Item readitem() ItemProcessor 1 processitem 2 (Item) ItemWriter writeitems (List<Item>) Commit transaction 3 Aufruf close() Finish Step

20 SKIP UND RETRY <chunk item-count="2" skip-limit="25"> <skippable-exception-classes> <include class="java.lang.exception" /> <exclude class="java.io.ioexception" /> </skippable-exception-classes> </chunk> <chunk item-count="2" retry-limit="25"> <retryable-exception-classes> <include class="java.lang.exception" /> <exclude class="java.io.ioexception" /> </retryable-exception-classes> </chunk>

21 LISTENER <job id="simplejob"> <listeners> <listener ref="myjoblistener"/> </listeners> <step id="chunkstep"> <listeners> <listener ref="mysteplistener"/> </listeners> <chunk item-count="2"> <reader ref="dummyitemreader"/> <processor ref="logitemprocessor"/> <writer ref="logitemwriter"/> </chunk> </step> </job> JobListener StepListener ChunkListener ItemReadListener ItemProcessListener ItemWriteListener SkipReadListener SkipProcessListener SkipWriteListener

22 AGENDA Grundlagen JSR-352 Properties, Job-Parameter, Restart, Skip, Retry und Listener Parallelisierung Implementierungen und Betriebsoptionen

23 PARALLELISIERUNG PartitionMapper Name Kategori e Tobias A Dennis A Frank A Daniel B Ben B Max B Christia n C Dirk C Thomas C

24 PARALLELISIERUNG PartitionMapper Name Kategori e Tobias A Dennis A Frank A Daniel B Ben B Max B Christia n C Dirk C Thomas C PartitionPlan kategorie=a kategorie=b kategorie=c

25 PARALLELISIERUNG Thread 1 Open transaction ItemReader Item readitem() ItemProcessor 1 processitem 2 (Item) ItemWriter writeitems (List<Item>) Commit transaction kategorie=a 3 Thread 2 Open transaction ItemReader Item readitem() ItemProcessor 1 processitem 2 (Item) ItemWriter writeitems (List<Item>) Commit transaction kategorie=b 3 Thread 3 Open transaction ItemReader Item readitem() ItemProcessor 1 processitem 2 (Item) ItemWriter writeitems (List<Item>) Commit transaction kategorie=c 3 PartitionCollector PartitionAnalyzer PartitionReducer

26 AGENDA Grundlagen JSR-352 Properties, Job-Parameter, Restart, Skip, Retry und Listener Parallelisierung Implementierungen und Betriebsoptionen

27 BETRIEB EINER BATCH-ANWENDUNG Batch application (JSR-352) Batch framework Dependency injection framework Transaction management Resources Servlet / EJB Container

28 BETRIEB EINER BATCH-ANWENDUNG Batch application (JSR-352) WAR / EAR App-Server s JSR-352 implementation Transaction manage- Resourment (JTA) ces CDI Servlet / EJB Container Application Server Infra- Structure

29 BETRIEB EINER BATCH-ANWENDUNG Batch application (JSR-352) BatchEE Spring Batch WAR / EAR App-Server s JSR-352 implementation Transaction manage- Resourment (JTA) ces CDI Servlet / EJB Container Application Server Infra- Structure

30 BETRIEB EINER BATCH-ANWENDUNG Batch application (JSR-352) WAR / EAR Spring Batch (JSR-352) Spring dependency injection Transaction management (JTA) Resources Servlet / EJB Container Application Server Infrastructure Testbarkeit Community Open Source Features: Parallelisierung, JavaConfig, Spring Data, Hadoop...

31 BETRIEB EINER BATCH-ANWENDUNG Batch application (JSR-352) WAR / EAR BatchEE Spring Batch (JSR-352) Spring dependency injection Transaction management (JTA) Resources Servlet / EJB Container Application Server Infrastructure

32 BETRIEB EINER BATCH-ANWENDUNG Batch application (JSR-352) Spring Batch (JSR-352) Spring dependency injection Spring Boot Fat JAR (Spring) Transaction management Resources Servlet Container

33 FRAGEN? Tobias Flohre Merscheider Straße Solingen tobias.flohre@codecentric.de blog.codecentric.de