Automated Testing of a Fully Automated Greenfield Terminal: A Case Study

Transcription

1 Automated Testing of a Fully Automated Greenfield Terminal: A Case Study March 2015 Stéphane Zampelli 1

2 Agenda How is Automated Testing useful? Automated Testing Overview at RWG Waterside Automated Tests Landside Automated Tests Emulation Performance Lessons learned - Impact of Automated Testing 2

3 How to secure your production deployment? Upgrade of your eco-system Still functional? No regression? Exclude old bugs Discover new bugs Behaviour has not changed Integration problems Performance stable? containers per hour New features really delivered? as soon as possible! 3

4 Automated Testing Vision Install & Config 12 hours running time Overnight Unattended No human intervention Go/NoGo Decision Automated QA benefits: Agile to change Refocus operational staff Increase confidence Repeatable and complete testing Iteratively increase coverage 4

5 How does Test Automation help our Customers? Agile to change install a patch or make a configuration change. Then automatically test changes overnight or over the weekend Refocus operational staff on running and improving the business reducing repetitive manual testing and the need to co-ordinate a big test effort over days/weeks Increase confidence with operations and management by catching issues upfront and avoiding disruptions Repeatable and structured approach to testing avoiding the question did we miss anything this time? Iteratively increase coverage add test cases that are rarely seen but have high impact. Then test the high impact flows every time 5

6 How does Test Automation help our Customers? Agile to change Refocus operational staff Increase confidence Repeatable and complete testing Iteratively increase coverage 6

7 Manual testing is costly 400+ test cases in the test case database Selected out of teams of manual testers teams of 2/3 persons Around 12 test cases per day per team Around 17 days to execute the test cases 7

8 Rotterdam Gateway, DP World 8

9 Rotterdam Gateway, DP World Started on site 2014 April Greenfield project ASC, L-AGV, Racks Automated quay cranes Automated Gates Pre-announcement Automated terminal means less people available Need a way to quickly test releases Especially true when people are moving to operations 9

10 Team Involved in Automated Testing Sidney Nederlof, Head of IT Ashish Panda Navis Consultant Guillaume Miara Navis Consultant Joeri Siebel RWG IT Engineer Stéphane Zampelli Navis Consultant Kathy Xie Navis Consultant 10

11 My Involvment Senior Consultant, Optimization Services Prime Route Expert Decking AGV/ASC Schedulers Performance emultation Business Process improvement Improvement of Testing Cycle Build an approach with RWG 11

12 Automated Testing Process at RWG Install & Config Filtering Test Cases New Release Emulation A.T. Deploy Manual Testing New features UI Failed Test Cases Go/NoGo 12

13 Reduce Cycle Time at RWG No Testing: 3 days 3 No Tests Install & Config Automated Testing Cycle Time: 7 days 3 1 Install & Config 3 Manual Testing Go/NoGo 400 test cases Manual Testing Cycle Time: 24 days 3 21 Install & Config Manual Testing Go/NoGo 13

14 Automated Testing Types at RWG Automated Testing Waterside Functionality Landside Performance Emulation Discharge Load Interstack 62 test cases Appointments Truck Rail Barge Manual Using Community Portal Execution 8 hours run Measure nbr of containers per hour fixed scenario 300 test cases 14

15 Waterside Components 15

16 Workflow example Load Move 40ft via direct interchange 16

17 Each Script is an Independent Test Case Expected Behavior Configuration & Steps Manual Testing Videos (landside) Each script is independent Reset virtual terminal Create Data Configure Execute: activate Work Queue Include script in a batch system With a timeout Each script resets the virtual terminal no blockers 21,BetaEmulationWTeams,L5InterStack40ftConventionalRackToConventionalDI 22,BetaEmulationWTeams,L5InterStack40ftConventionalRackToConventionalRack 23,BetaEmulationWTeams,L5InterStack40ftConventionalRackToCantileverDI 24,BetaEmulationWTeams,L5InterStack40ftConventionalDIToConventionalDI 25,BetaEmulationWTeams,L5InterStack40ftConventionalDIToConventionalRack 26,BetaEmulationWTeams,L5InterStack40ftConventionalDIToCantileverDI 27,BetaEmulationWTeams,L5InterStack40ftCantileverDIToConventionalDI 28,BetaEmulationWTeams,L5InterStack40ftCantileverDIToConventionalRack Unattended and overnight Configure Reset Virtual Terminal Activate WQ Check Result Virtual terminal executes the move FAILED or PASSED 17

18 Demo Script example - Load Start XPS client, deactivate WQs, revert WIs, resetn4() resetteams() purgein() configure() n4yardsync() execute() 18

19 Demo Script example resetn4() resetteams() purgein() configure() n4yardsync() execute() Reset Virtual Terminal 19

20 Demo Script example resetn4() resetteams() purgein() configure() n4yardsync() execute() Purge N4 WAs, reset CHEs 20

21 Demo Script example resetn4() resetteams() purgein() configure() n4yardsync() execute() Vessel visit, cntr in yard, shift, projection, 21

22 Demo Script example resetn4() resetteams() purgein() configure() n4yardsync() execute() Yard sync between ECS and N4 22

23 Demo Script example resetn4() resetteams() purgein() configure() n4yardsync() execute() Activate WQs, log and monitor The equipment simulator and N4 will execute the move. 23

24 Demo Video 24

25 Pass = feature works, Fail = unknown Pass means the feature was confirmed by the script Fail means something went wrong Does not mean the feature is not covered Workflow is actually functional Technical/configuration problem Workflow is really broken All the failed test cases have to be re-tested manually We currently achieve 95% filtering, the rest has to be re-tested manually 25

26 False failure can come from configuration 45ft discharge Decision has been made by op management to change the site of 45ft Information did not flow back to AT team Most 45ft discharge cases are failed (forced target position) Scripts have to be updated accordingly 26

27 Some numbers from RWG Automated Testing Waterside Functionality Landside Performance Emulation 62 TC + 5 min per TC 10 TC/h 120 TC/night 300 TC + All can be run in one night using two computers Some takes 1 minute, some more than 15 minutes 8 hours run 4 cranes, 1000 moves unattended flexible scenario 27

28 Landside Components 28

29 Technologies Unified Functional Testing AKA: Quick Test Pro, Astra Quick Test Mercury Interactive, HP Functional Test Tool Record/Playback suitable for subject matter experts Supports Java,.NET, Web, SAP, Oracle, SOAP. VB Script and Microsoft Object Model 29

30 Test Case Truck Appointment for a container already discharged in the yard Check that N4 generates am error Do not use PCS website (manual) 30

31 Test Case - Video 31

32 Test Case Using PortBase Truck Appointment for a container already discharged in the yard Check that N4 generates am error Do not use PCS website (manual) 32

33 Test Case Using PortBase - Video 33

34 Test Cases and Results 34

35 Emulation: Testing Performance Key difference: less coverage, focus on performance 8 hours of run, count Nbr of box movements Nbr of human interventions Gives a level of confidence about the performance 35

36 Emulation: Demo TODO 36

37 Emulation: Typical Result Doing single discharges only (mainly 40 s) 10 AGVs per cranes high congestion lots of parking Cruising at 45+ MPH one intervention at 18.01PM on QC105 AGV blocking QCs 37

38 Lessons Learned Expected Behaviour not always known Usually part of the testing itself Software providers should support AT Virtual Terminal Reset Should be part of O&A Software providers may refuse» Becomes black magic, reverse engineering Interfaces stability Moving to a new major version Redo part of your scripts Operational Knownledge and Coding Skills 38

39 Lessons Learned Not Covered UI Behaviour Non Functional Requirements What do you test exactly? Your terminal or the equipment emulator? Equipment is still part of the big picture Time is now spent on scripts instead of testing Especially at the beginning 39

40 Impact of Automated Testing quick (overnight) and unmanned (unattended) larger coverage regression review (including perf) of your config & install But: Does not replace manual testing Does not replace Live Equipment Testing Requires the involvment of vendors Coding/IT skills required 40

41 Get In Touch With Navis People Testing Services at Guru Bar 41

42 Automated Testing of a Fully Automated Greenfield Terminal: A Case Study March 2015 Stéphane Zampelli 42

43 Still to do Waterside components - convert to ppt Landside components convert to ppt Waterside demo Emulation demo «Get in touch with Navis People» «Team involved in automated Testing» Discuss relevance of Emulation example with G.Miara Buy photo 43