Big Data for the Mass. Dr Franclin Foping Senior Software Engineer NitroSell.com

Transcription

1 Big Data for the Mass Dr Franclin Foping Senior Software Engineer NitroSell.com

2 To SQL or NOT to SQL? Relational DB Build in the 70s to solve problems of the 70s Still useful but

3 New Trends Big Data Concurrency Data replication And yes, scalability

4

5 Benefits of nosql Large volumes of structured, semi-structured, and unstructured data Agile sprints, quick iteration, and frequent code pushes Object-oriented programming that is easy to use and flexible Efficient, scale-out architecture instead of expensive, monolithic architecture

6 More Goodness Dynamic Schemas Auto-sharding Auto-Replication Integrated Caching

7 CASE STUDY Key Value Storage Realtime data analytics (INSIGHT)

8 NitroSell platform Allows retailers to easily sell their products online E-Commerce on the up! Caching is a BIG issue out there

9 NitroSell Needed a place to store and retrieve data quickly RDBMS too slow and data structure? What sort of data we store? Session, navigation panel and so forth REDIS

10 Redisallows fast insertion and retrieval Excellent key value storage Schema-less a plus!

11 Sample Data NavPanel <br/><!--<div id="relatedproductspanel"> --> <div id="relatedproducts"> <h2 id="relateditemheading" class="red">stuff you may also like</h2> <table id="relateditemtable"><tr><td id="relateditembody"> <table class="relateditembody"> <tr> <td align="center" class="smalltext" width="25%"> <input type="hidden" name="relitem" value="1600"> <a href=" class="image-thumb" alt="white SnazarooFace Paint" title="white SnazarooFace Paint" src=" border="0"/></a><br /> <a href=" SnazarooFace Paint</a><br/> <span class="text-pricestrike"> 5.00</span> <span class="text-pricespecial"> 4.00</span> </td> <td align="center" class="smalltext" width="25%"> <input type="hidden" name="relitem" value="7108"> <a href=" class="image-thumb" alt="long Witch Wig" title="long Witch Wig" src=" border="0"/></a><br/> <a href=" Witch Wig</a><br/> <span class="text-price"> 6.00</span> </td> <td align="center" class="smalltext" width="25%"> <input type="hidden" name="relitem" value="1834"> <a href=" class="image-thumb" alt="wig Cap" title="wig Cap" src=" border="0"/></a><br/> <a href=" Cap</a><br/> <span class="textpricestrike"> 3.00</span> <span class="text-pricespecial"> 2.40</span> </td> <td align="center" class="smalltext" width="25%"> <input type="hidden" name="relitem" value="13532"> <a href=" class="image-thumb" alt="the Corpse Bride Costume" title="the Corpse Bride Costume" src=" border="0"/></a><br /> <a href=" Corpse Bride Costume</a><br /> <span class="text-pricestrike"> 37.00</span> <span class="text-pricespecial"> 31.45</span> </td> </tr> </table> </td></tr></table> </div> <!--</div> -->

12 CASE STUDY 2 Real-time Web Analytics Join project undertaken in UCC/NitroSell Provide a smarter related item panel to customers Recommender systems all over e-commerce and the Web in general!

13 Google Analytics Deployed on 2 of our Irish stores (partyworld.ie and wreckless.ie) Each generates about 1000 lines in the logs a day 4 types of events: pageview, addtobasket, removedfrombasket, and ordercomplete

14 Data modeling Each event has its own data types. Ordertotal only in ordercomplete, NULLs in RDBMS Need for a schema-less database Data aggregation played a huge part in the analysis

15 MongoDBdid the trick Excellent Map-Reduce support Data sharding, concurrency and scalability taken care of

16 Sample Data Experiment started a month ago Over 34k log entries for over 7k visitors Worth more than 8k euros

17 Ordercompleteevent { "_id" : ObjectId("54f4834ba4bc3eb710b8e0b1"), "ip_address" : " ", "ordertotal" : "27.49", "basket" : "3827:1", "cookieid" : "695e69j7bftohc1r5qiuak1bfhq9jji6", "abexperiment" : "1", "UserAgent" : "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/ (KHTML, l "hittype" : "ordercomplete", "usergroup" : "B", "abexpuserid" : "66f1a31034c84179-B", "username" : "695e69j7bftohc1r5qiuak1bfhq9jji6", "uid3" : "66f1a31034c84179-B", "ordernumber" : " ", "timestamp" : ISODate(" T15:35:39Z") }

18 pageview { "_id" : ObjectId("54f5f1b8a4bc3e4b08295c49"), "ip_address" : " ", "countrycode" : "IE", "productid" : "14696", "productcode" : "41945", "productname" : "Farm Party Balloons", "cookieid" : "3hafagjtsna915ha5668hddkouhdl637", "username" : "3hafagjtsna915ha5668hddkouhdl637", "abexpuserid" : "a8621f90a7934da4-b", "hittype" : "pageview", "ClickedLink" : " "pagetitle" : "Farm Party Balloons- PartyWorld Costume Shop", "timestamp" : ISODate(" T17:39:04Z"), "referrer" : " "domain" : " "UserAgent" : "Mozilla/5.0 (Linux; U; Android 4.4.2; en-ie; GT-P5210 Bui Safari/534.30", "abexperiment" : "true", "relateditems" : "14715,14698,14695,14763", "usergroup" : "B" }

19 Lessons learnt Product popularity Who is viewing our stores? When? How? IE popularity

20 Summary RDBMS, glory days over Designed in the 70s to solve problems of the 70s Scalability

21 References Efficient Market based analysis 8.pdf Modeling and Analysis of a multi-level caching library/conferences/miami2004/papers/ pdf

22 QUESTIONS?