Data Science in the Travel Industry

Transcription

1 Data Science in the Travel Industry Real-world experience using current leading frameworks Paul Balm September 2015

2 Hello! _I am from Amsterdam, since 2005 in Madrid _ : Ph.D. in particle physics at Fermilab in Chicago _ : Data Processing projects for the European Space Agency _2014 present: Data Scientist at Amadeus Travel Intelligence Page 2

3 Agenda _ Travel Industry and Amadeus _ A User Story _ Experiences with Different Technologies _ Conclusions & Final Remarks All views expressed in this presentation are entirely my own Page 3

4 1 The Travel Industry and Amadeus

5 Amadeus in a few words 1 _ Introduction Amadeus is a technology company dedicated to the global travel industry. We are present in 195 countries with a worldwide team of more than 12,000 people. Our solutions help improve the business performance of travel agencies, corporations, airlines, airports, hotels, railways and more Amadeus IT Group SA Page 5

6 We started our journey in _ Amadeus history Page 6

7 A history of shaping the future of travel 2 _ Amadeus history 1987 Amadeus is founded 1992 First booking made through Amadeus 1995 World leader in number of travel agency locations 1996 e-commerce division launched MILLION bookings made on a single day for the first time 2000 Partnership with BA and Qantas to launch Amadeus Altéa our core Airline IT offering 2010 Amadeus diversifies into IT for hotel, rail and airport 2014 Contracts with Ryanair and Southwest Airlines and a strategic technology relationship with IHG 2015 Amadeus IT Group SA Page 7

8 Robust global operations 3 _ Amadeus today 1.6+ billion data requests processed per day 525+ million travel agency bookings processed in million Passengers Boarded (PBs) in % of the wrld s scheduled network airline seats 2015 Amadeus IT Group SA Page 8

9 Connecting the entire travel ecosystem Cruiselines Airlines Hotels Insurance companies Car rental Travel agencies Ground handlers Airports Ground transportation Ferry operators 2015 Amadeus IT Group SA Page 9

10 Supporting the entire traveller life cycle Post-trip Inspire On trip Search Pre-trip Buy/Purchase 2015 Amadeus IT Group SA Page 10

11 2015 Amadeus IT Group SA 2 A User Story

12 A Network Carrier _An airline operating a network with hubs AMS BER _Any risks or opportunities for this operation? FRA Page 12

13 Schedule Analysis Analyse global schedules to alert of any changes in capacity Schedule information may come from _Commercially provided feeds _Airline inventory data _Publicly available databases (EuroStats, US DoT) Page 13

14 Schedule Analysis Page 14

15 Schedule Analysis Batch processing of different formats of input data _Different formats, different sources and a priori unclear which sources will be useful or when _Store everything always within legal and moral boundaries _Central data site with large NAS (network attached storage) _HDFS based storage at hosted processing clusters _Maintain separation of different customer projects Page 15

16 A Network Carrier _An airline operating a network with hubs AMS BER _Schedule analysis shows competitor entering the market FRA _What s the risk? Page 16

17 Quality of Service QSI: Quality of Service Index The prime quality metric: Time to get from A to B In practice many rules to build QSI that require tuning and iteration Agility as a requirement Need for scalable processing capacity Page 17

18 A Network Carrier _An airline operating a network with hubs AMS BER _Schedule analysis shows competitor entering the market FRA _Risk of loss of market share assessed via QSI analysis _How to react? Page 18

19 Future revenue streams Information about future revenue comes in many forms _Tickets issued _Reservations (foreseen revenue) _Searches _Capacity changes _Sentiments on social networks, weather forecasts and the like _How are average fares developing? _How are passenger counts developing? Requires fast updates live if possible Page 19

20 Streaming? No hard truths here considerations of options only! _Response times of the order of seconds are not required in the presented use-case _We still require to process the large amount of historical data _We think processing the historical data is better done in batch Mature and reliable approach Can we reuse the streaming code for batch processing? _Flink: Suitability for batch processing to be demonstrated _Storm: Using Summingbird, combine with Scalding? _Spark Streaming: Combine with Spark for batches. Maturity? Page 20

21 A Network Carrier _An airline operating a network with hubs AMS BER _Schedule analysis shows competitor entering the market FRA _Risk of loss of market share assessed via QSI analysis _How to react? Page 21

22 3 Technologically

23 The Travel Intelligence Engine Scalable cloud-based Hadoop platform Amadeus Data (Altéa and other) Customer Private Data Industry Weather Demographics Social Media Etc. Visualization layer Applications Raw Data Storage * Batch and streaming processingoutput storage Data Warehouse (* All the raw data is kept after processing to allow for re-processing when required) Page 23

24 Lambda architecture Image reproduced from Page 24

25 Page 25

26 Page 26

27 Page 27 Scoobi

28 Page 28 Scoobi

29 Scoobi Scalding Page 29

30 Benchmarking Spark vs. Scalding Source: Box Engineering Blog Page 30

31 Scoobi Scaldin g Impala Page 31

32 Scoobi Scaldin g Impala Storm Kafka Page 32

33 Scoobi Scaldin g Kafka Impala Storm Web Services Visualization tools Page 33

34 Scoobi Scaldin g Kafka Impala Storm Web Services Python: Flask Scala: Spray Visualization tools Page 34

35 Scoobi Scaldin g Kafka Impala Storm Web Services Python: Flask Scala: Spray Visualization Tools HTML5/JS Page 35

36 Recap Data science opportunities in the travel sector How we are dealing with these opportunities at Amadeus Travel Intelligence An overview of our experience with different tools What s on our horizon for evaluation and why Page 36

37 Some final remarks Different technologies currently under evaluation In all areas (batch, streaming, visualizations) too many options to carefully evaluate all Also, development effort from the community is diluted Consolidation would be welcome! Even so, tools are advancing rapidly and within their limitations, many are of great value Page 37

38 Thank