HOW TO ENSURE THE STABILITY OF AN ECOMMERCE PLATFORM. Players Performance Partnership AT TIMES OF PEAK DEMAND. Written by Jon Bramley

Transcription

1 HOW TO ENSURE THE STABILITY OF AN ECOMMERCE PLATFORM AT TIMES OF PEAK DEMAND Written by Jon Bramley Players Performance Partnership

2 TABLE OF CONTENTS 03 INTRODUCTION 04 WHAT IS A PEAK? 05 SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM HOW TO ENSURE THE 07 HOW DO WE MANAGE PEAK LOADS? 08 EXPECTED PEAKS STABILITY OF 10 POSTMORTEM 11 AN OMNI-CHANNEL PLATFORM 12 SUMMARY AN ECOMMERCE 13 WHO ARE OPENBET? PLATFORM Within the world of internet betting, another bookmaker is just a click away so performance and stability are paramount to a successful day. AT TIMES OF PEAK DEMAND

3 INTRODUCTION The world of sportsbetting goes in cycles driven by the major sporting events of the year. Bookmakers want to use these events to maximise both their revenues and customer acquisition opportunities. This means releasing new features, running new promotions, launching dedicated microsites, etc. So not only is the platform under increased customer demand due to the natural draw of the event but also due to the effects of the increased marketing. This means being prepared for both new levels of load on the platform and ensuring consistent high performance and stability throughout new peaks and alongside new, potentially disruptive, functionality. In this paper we will explore different mechanisms and tools that can be used to ensure success and stability across these high profile, peak days. 3

4 WHAT IS A PEAK? Any system can have periods or days of unusual traffic. In the sportsbetting world we are used to peaky traffic due to the pattern of betting naturally being weighted towards specific events and increasing just before the start as everyone gets their last-minute bets in. Saturday afternoon just before the Premiership matches all kick off is traditionally a heavy betting period, for example. This national institution sees millions of otherwise non-betting people having their annual punt on the horse race and puts a load on the system of up to 400% of a regular Saturday. vs 49,000 49,000 49, Million ONLINE BETS PER MINUTE ONLINE BETS PER MINUTE ONLINE BETS However, to bring some real perspective to the challenge of peak loads, the most demanding day in the UK bookmakers calendar is the UK Grand National, held each April. 5,160 5,160 ITEMS SOLD PER MINUTE 7.4 Million ITEMS SOLD PER MINUTE ITEMS SOLD ONLINE BETS PER MINUTE 15.3 Million Grand National 2015 for OpenBet ONLINE BETS 7.4 Million Black Friday 2015 for Amazon ITEMS SOLD 15.3 Million ONLINE BETS 4

5 SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM Although a betting website could be thought of a just a specialised e-commerce platform, it actually has a number of characteristics that differentiate it and increases complexity. Shifting product catalogue A large scale bookmaker will have around 4 million active betting opportunities live at any one time - but these opportunities do not live long and the entire catalogue will be replaced with another set incrementally over a period of just a few weeks. Within that time the price of each item will change many times. Over the course of a 90 minute football match, the 4,000 betting opportunities available will have over 35,000 price changes applied. And that s just for one match - we will have over 200 events in-play simultaneously and many thousands more available for pre-match betting. Complicated purchase process The purchase process in sports betting involves: Validating the player s account and their balance Validating that the player has not exceeded their betting limits Validating that the betting opportunities selected have not exceeded their risk limits Validating that the betting opportunities can be combined in the chosen way Validating that the opportunities and requested prices are still valid Recalculating liability for the bookmaker once the bet is placed 5

6 SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM Settlement and payouts The biggest difference between betting and a regular ecommerce platform is, of course, what the player is buying: a bet. At some later point, the bet needs to be settled against the bet result and potentially pay the player their winnings. The main consideration here is that the settlement process itself is computationally intensive and also needs to happen as soon as possible after the result is available, both give the player a good experience, but also to maximise the recycling of winnings into another bet Per Event Football Tennis Cricket Number of markets e.g. First goal scorer FOOTBALL TENNIS CRICKET Number of market updates e.g. activation/suspension Total updates Number of selections e.g. Arsenal to win Number of selection updates e.g. odds change suspension Total updates

7 HOW DO WE MANAGE PEAK LOADS? Due to the cyclical and peaky nature of sportsbetting our platform is already built to manage these types of load. Constant development of the system means that performance needs to be constantly monitored. Each system is looked after by a dedicated lead who understands that system s normal and works closely with operators to manage network and hardware growth alongside the system. Total bets per hour We make heavy use of caching even for durations of only a couple of seconds to smooth out the large number of data requests and changing prices. We use push technology to publish price and status updates directly to connected users. We cache HTML snippets to help build dynamic pages. Our in-house Application Server is built to scale both horizontally and vertically and can dynamically use resources as needed. Functionality can be split across servers to balance load and a server can be brought online in seconds and use our caching layer to manage unexpected peaks. No player state is held at the Application Server level so traffic can be managed by any server. Active monitoring of software, hardware and database usage catches issues before they affect players

8 EXPECTED PEAKS Beforehand For those periods when we do expect very high peaks - like the Grand National - then we are able to plan and manage these days differently. 01 During the final few weeks, load testing continues and bookmaker marketing efforts are put into action. 04 It all starts months before, where a working group of technical experts review the previous year s performance and lessons learned. Milestones are agreed alongside a readiness checklist that has evolved over the years 1/2 MILLION NEW ACCOUNTS CREATED IN 24 HOURS 8 NEW ACCO CREATE 24 HOU BE MO THE GRAND NATIONAL 2014 IN NUMBERS: 2014 IN63% NUMBERS: 63% 8M 1/2 MILLION 1/2 MILLION 1/2 MILLION NEW ACCOUNTS CREATED IN 24 HOURS 02 ACCOUNTING FOR 8M ACCOUNTING FOR 8M 63% 8M 63% 11 Billion 1 Billio 1 11 Billion About a month before Grand National NEW ACCOUNTS NEW ACCOUNTS ACCOUNTING FOR 1/2 MILLION CREATED IN CREATED OF REMOTE BETS ON IN ACCOUNTING FOR NEW ACCOUNTS 24 HOURS CREATED IN OF REMOTE BETS24 ONHOURS 03 MOBI LE BETS PLACED M OHOURS BILE 24 BETS ON MOBI LE ON SATURDAY OF REMOTE BETS ON MOBILE OF REMOTE BETS PLACED ON SATURDAY BETS PLACED ON SATURDAY BETS PLACED ACCOUNT TRANSACTIONS ACCOUNT TRANSACTIONS ON SATURDAY SPORTS BETS another racing festival - Cheltenham - is Closer to the event, a feature freeze is implemented that a useful yardstick for the preparedness of allows the team to focus on performance and stability. BETTING OPPORTUNTIES the system andconcurrent allows further performance BE Enhanced load testing is started to spot weak areas. This2014 IN NUMBERS: ANY TIME ON THE PLATFORM CONCURRENT BETTINGAT OPPORTUNTIES 2014 IN NUMBERS: AT ANY TIME ON THE PLATFORM and stability changes to be made if the BETS PER MINUTE AT O load testing is not trivial to prepare as it must predict the levels of activity across the whole system: from 2014 IN NUMBERS: post-event analysis shows it is necessary. 1/2 MILLION 4 Million 4 Million 32, IN NUMBERS: 8M 11 Billion 11 Billion 1 Billion 1 Billion ACCOUNTING FOR registrations, to payments, casino activity, NEW ACCOUNTS 63% system 1/2 MILLION CREATED IN OF REMOTE NEW ACCOUNTS ON ACCOUNT TRANSACTIONS transfers, bet placement andbets bet settlement. All of which ACCOUNT TRANSACTIONS 63% O B IL E CREATED24 IN HOURS BETS ON M OF REMOTE place unique stresses on the system and, in particular, the 24 HOURS M O B IL E centralised player account. ACCOUNT TRANSACTIONS 8M ACCOUNTING FOR 1 Billion 11 Billion11 Billion 1 Billion 4 Million 4 Million 32,00032,000 BETS PLACED SPORTS BETS SPORTS BETS ON SATURDAY BETS PLACED ON SATURDAY ACCOUNT TRANSACTIONS SPORTS BETS SPORTS BETS CONCURRENT BETTING OPPORTUNTIES CONCURRENT BETTING OPPORTUNTIES BETS PER MINUTE AT OUR BETS PEAKPER MINUTE AT OUR PEAK AT ANY TIME ON THE PLATFORM AT ANY TIME ON THE PLATFORM 4 Million4 Million32,000 32,000 CONCURRENT BETTING OPPORTUNTIES BETS PER MINUTE AT OUR PEAK AT ANY TIME ON THE PLATFORM CONCURRENT BETTING OPPORTUNTIES BETS PER MINUTE AT OUR PEAK 2014 IN NUMBERS: AT ANY TIME ON THE PLATFORM 2014 IN NUMBERS: 11 Billion 11 Billion ACCOUNT TRANSACTIONS ACCOUNT TRANSACTIONS 1/2 MILL 1 Billion 1 Billion SPORTS BETS SPORTS BETS 8

9 EXPECTED PEAKS On The Day On the day itself, communication is vital. Each operator team is present and locate themselves in a single area of the office to make coordination easier. Slack is used for instant messaging with two dedicated channels: one for general communication across teams and another for the escalation of incidents. Account managers and the executive team are subscribed to this second channel. In the case of issues, communication to all other teams is key so they can be prepared for additional traffic (players moving to a site that is available) and to look out for similar issues. Experts jump in to analyse the problem and get it fixed as quickly as possible and identify the cause in case of knock-on effects to other operators. Throughout the day activity and load is constantly recorded for later analysis and to help plan for the following year. Registration, payments and bet placement are the key focus areas for stability early in the day but this shifts to bet settlement after the events leading up to the big races to ensure players have their winnings returned to them for further bets they d like to place. Settlement can be throttled by the amount of system resources dedicated to it but also can have prioritisation applied based on specific events, markets or even betting selection and by prioritising winning bets over non-winnings ones. The winning customer experience is key and postponing the settlement of non-winning bets to a quieter period later in the day saves vital system resources for continued bet placement. Potential Bottlenecks caused by the wide number of third party systems our platform integrates with (payments, verifications, system transfers) are monitored carefully with timeouts kept as low as possible to prevent execution threads getting locked up. As pressure builds, there is a list of non-essential activity and functionality that can be disabled to ensure the key functions of registration, login, and bet stay functional. Active monitoring of the load on each part of the system is used to balance functionality across servers more evenly and increase capacity where the models didn t predict load accurately Million ONLINE SPORTS BETS 49,000 ONLINE BETS PER MINUTE 60 Million ONLINE ACCOUNT TRANSACTIONS 74% Bets PLACED VIA MOBILE CHANNEL 9

10 POSTMORTEM After the event Analyse any major issues, their route cause, immediate resolution and if any longer term solution is required Analyse performance versus load for bottlenecks and hot points, and determine solutions. Add these to the backlog with priority based on the severity Analyse the load patterns to feed into updated load test profiles Share findings across teams and add shared learnings to the backlog Carry out a retrospective for the day to take forward suggestions into next year Account Transactions (millions) Year 10

11 AN OMNI-CHANNEL PLATFORM Sportsbetting has increasingly and rapidly become a mobile-first activity. The advent of the smartphone has materially changed the platform usage profile of online betting such that it is now the majority of online bets that are taken via mobile. For some operators over 80% of bets are taken via mobile channels, but even the annual average across all operators is over 70%. Grand National Mobile Usage OpenBet s omni-channel platform separates the functionality of the sportsbook from the player front-end and so it does not matter where the activity is taking place - the same robust back-end processes all bets and the same functionality is available to all channels. It is the responsibility of the front-end application to deliver the right player experience for the channel being used. PLACE BET <50% % %

12 SUMMARY In summary, you should always be prepared for localised peaks in demand and ensure that new feature releases are assessed for changes in system behaviour. Regular, automated load testing is ideal but any regular analysis will help understand normal behaviour and local peak effects. For expected, high load times - whether due to new functional releases, seasonal behaviour (Christmas rush) or a promotional event - preparation can make all the difference. Total Sports Bets On Grand National day Learn the lessons of previous situations and stay actively engaged throughout the peak period with fall-back plans in the case of unexpected behaviour Year Remember that peak loads today will be your normal behaviour in the not too distant future. 12

13 WHO ARE OPENBET? EST 1996 EST Established Dynamic Trusted Founded in 1996, OpenBet powers the largest share of the world s online bets with global software deployments in multiple, regulated gaming jurisdictions. With six offices worldwide including North America, Australia, Singapore and London, over 20 customers in the online gaming industry and over 200 integrated partners worldwide, OpenBet provides a truly open gaming platform and a product suite that ensure the very best and most flexible solutions available. OpenBet prides itself on product innovation and technical excellence. This makes it the technology provider and betting platform of choice for regulated gaming markets around the globe. OpenBet s inherent culture of innovation has enabled the development of the industry s first omni-channel gaming solution and ensures that OpenBet stands out as the market leader within the complex world of gaming software. Instrumental to the success of major operators around the globe, OpenBet powers the largest share of the world s online bets: taking over a billion sports bets in 2014 alone. OpenBet is trusted by some of the world s most successful operators including William Hill, Ladbrokes, Paddy Power, Betfair, SkyBet and PMU. It is our performance and exacting levels of probity that have seen us become known as the supplier of choice for regulated gaming markets around the globe. 13

14 Players Performance Partnership For more information about OpenBet, please visit or