An Idea of testing BIG... Challenges and approaches for testing Big Data

Transcription

1 testing Big Data Capgemini India Private Limited Sep 2013 Prepared by: Renuka Kale

2 1 Table of Contents 1. Abstract What is Big Data From Big Data to Big Testing Vs of Big Data Big testing approaches: Big Data Landscape: Big Hypothesis Big Data limitations: Conclusion References Author s Biography... 16

3 2 1. Abstract Nowadays social media sites are creating a great buzz. Handling such a great amount of social media exchange requires equivalent strong technologies. Big Data thus comes into picture. Big Data is the talk of the town these days, because of its varied uses ranging from social media sites, to large banking firms, to telecom domains, healthcare domain and so on. Also earlier computer systems were large sized, but these days due to mobile revolutions, mobile apps facilitates e-interaction very easily and an enormous amount of data gets pumped in. As per the report, as a part of this digital world, we generate more than 200 Exabyte of information every year. According to Intel, each internet minute sees 100,000 tweets, 277,000 Facebook logins, 204 million mail exchanges, and 2 million search queries fired. Also website visits, touch points, ad impressions, video views, online community discussions etc also create enormous amount of data. This data tell us of the information of customer behaviours, intents, and preferences. We can combine that raw digital data with data from other sources such as call centre logs, transaction histories, and in-person interactions. This data can be mixed with publicly available data on demographics, weather forecasts and the economy. Big data a huge amount of information gathered from non traditional sources like blogs, social media, s, video footages, photos, etc ; which is typically scattered, unstructured and voluminous, can be greatly useful in business intelligence, in analyzing ongoing business trends, shopping trends, sentiment analysis, peoples liking and disliking. Thus with the help of the results drawn out of this big data, companies get immensely benefitted in terms of formulating their upcoming plans, devising their strategies, changing their approaches, and capturing the promising business areas. Companies can better understand their customers and thus can tailor their offers as per the customer needs. So when we talk about Big data, it is imperative to talk about its testing aspect and the vital phases involved in testing Big data. Would it suffice to follow conventional testing measures while testing Big data, what challenges does it involve, what would be the best approaches, what are the limitations and opportunities in testing Big data. These questions clamour as we start thinking about testing big data. This paper tries to elaborate and surface these aspects which call for further discussions. Read on.

4 3 2. What is Big Data Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, storage, search, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations combat crime, and determine real-time roadway traffic conditions. Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers". What is considered "big data" varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration In a digital world, we can easily try many cost effective versions of ads, landing pages, web applications, messages, and other digital touch points running A/B tests between our original control versions and new challenger versions that try out hypotheses. In the marketing strategies, small steps can be taken to experiment on a few customers on a pilot basis to validate the hypothesis. If it fails, there is only little risk involved, and if it succeeds, it becomes new mantra. There is always a win-win situation. Because it s so easy to run tests like this in the digital medium, companies such as Google and Amazon businesses that grew up natively in the digital world have deeply embraced experimentation as part of their culture. It s that relatively low success rate that may have held back many companies from harnessing the power of marketing experimentation. In pre-digital days, such experimentation was much more costly and big bets were either spectacular successes or disastrous failures. But nowadays such experiments have become a trend and innovative strategies are seen to be devised to derive important information from users.

5 4 3. From Big Data to Big Testing As Greg Linden, who led a set of experiments at Amazon, stated in an article on big data in The Atlantic, To find high impact experiments, you need to try a lot of things. Genius is born from a thousand failures. In each failed test, you learn something that helps you find something that will work. Constant, continuous, ubiquitous experimentation is the most important thing. Such ubiquitous experimentation in search of big wins can be labeled big testing, the natural complement to big data. Following things make Big testing a Big deal : 1. Experimentation with new, innovative ideas to try something that may fail. Risk is mitigated by testing such ideas on a small scale. Small to moderate risk taking is necessary shunning all doubts. 2. Experimentations must have a vision and should be properly managed Experimentations are bound to fail, but they are not identifies as failures, rather recognized for their continued and aggressive efforts. While experimenting, a proper goal must be set and its progress properly monitored. 3. More and more people are involved for the experimentation If the experiments are allowed to be run first for small group of people and then slowly scaled up and more and more people are engaged, then the results might be extraordinary. 4. Inferences drawn out of experimentations Experimentation should be called as complete only when proper conclusions are drawn out of it. If it is failed, what are the lessons learnt, what changes are required to be made in the next experimentation and other related factors can be decided. 5. Creating a base data for experiments done by other bodies Experimentations done by other organizations must also be studied and referred to get the baseline which is readily available. Big data is amazing source of new hypotheses for marketing. However, it will have its value in true sense if it is properly tested. Like big data, big testing is a native approach to marketing in a digital world. Big testing can surely bring in more productive results and help draw decisive findings for marketing world.

6 5 4. 4Vs of Big Data 4 important factors involved in Big data testing: Volume: Big data involves large data to be analyzed, coming from various sources. Data volume is day by day increasing, ranging from a few dozen terabytes to many petabytes of data in a single data set. One of the fundamental defining characteristics of Big Data environments is that they involve extremely large data volumes. Big Data environments based on technologies such as the Hadoop Distributed File System (HDFS) sometimes scale out to petabytes of data running across thousands of distributed processors. Internet companies such as Google, Yahoo and Facebook have been pioneers in the use of Big Data technologies and routinely store hundreds of terabytes and even petabytes of data on their systems. Facebook's Hadoop Big Data cluster for instance, scales out to a staggering 30 petabytes of data, making it one of the biggest Big Data implementations on the planet. Pharmaceutical companies and financial services companies also routinely collect, process, and analyze terabytes of data in their Big Data environments. Challenges in Volume testing: - Big data involves huge amount of data to deal with. Thus, 100% coverage is not possible. In this case we need to implement very good data sampling techniques. - Data files are stored in various locations, so consolidation of data is a challenge.

7 6 - Performance of the data processing is an important factor. When we query against such a huge data, its performance is bound to degrade. Time required for the data processing, server response time is a vital factor. - Data files are stored on HDFS. Approach for Volume testing: - Requirement analysis: Understanding the business requirement and accordingly identifying the areas where data sampling is required. - Use of data sampling techniques, data extraction tools. Categorization of data to be tested. - Use of traditional techniques such as Boundary Value Analysis and Equivalent Partitioning. - Converting raw data into useful test data to compare with the actual data. Velocity: Traditionally data updates used to happen weekly, bi-weekly, monthly, quarterly and like, based on the user requirements. But, the periodicity was fixed and consistent. However, in Big data, data is continuously updating, almost at the speed of real time. Thus, gathering and storing this data is also important thing. One of the key elements of Big Data is data velocity, or the speed at which new data is processed and analyzed by an organization. The Internet, e-commerce, mobile devices and social media technologies are allowing organizations to collect more realtime information on customers and transactions than ever before. Online retailers and financial services companies for instance, have the ability these days to compile extremely detailed customer profiles and behavioural patterns by tracking and monitoring the online transactions and other interactions of their customers. In order to derive benefit from such data, businesses are increasingly looking for technologies that allow them to tap and analyze fast-moving data streams in as near real-time as possible. This kind of complex event processing is a crucial component of Big Data environments. Generally, the greater the velocity with which data can be analyzed, the bigger the near-term benefit for the company.

8 7 Challenges in Velocity testing: - Complex scale up strategy required - Coping up with rapidly incrementing data needs to be handled by keeping some benchmarks. - Simulating production like environment Approach for Velocity testing: - Incremental performance testing - Bench mark testing Variety: Data coming from various sources may have different - different forms, structures, and conventions. Also a same thing might be represented in different way. Data may vary from tables, structured data up to free text (tweets). Formatting such data and gelling up in a streamlined coherent form is a vital aspect. Unlike RDBM systems, Big Data environments tend to involve a lot of data collected from a myriad of sources, often in raw form. It's not unusual for Big Data environments to contain data that is repetitive, incomplete, unverifiable or just not useful for any purpose. Data collected from Twitter Feeds or Facebook posts for instance, may offer clues about customer sentiment but the reliability of such data is often very suspect. The sheer variety and the velocity at which data is collected also poses a major challenge to data veracity in a Big Data environment. In order for the data to be really useful it has to be clean and reliable. Organizations can sometimes spend well more than half their time and effort on simply cleaning up Big Data and staging it for future use. Challenges in Variety testing: - Scrutinizing unstructured data is the most challenging thing. Since the data available is in varied forms. It s very difficult to format, categorize and tag it. - Sampling the data out of voluminous data, based on requirements is challenging. - It requires lot of manual work to draw meaningful data out of heap of data Approach for Variety testing: - Identifying source of data and devising strategy based on it. - Bringing / converting data into a structured form and running scripts for sample data comparisons - Localization and Globalization testing.

9 8 Value: Churning out large amount of data has to get converted into a form of useful information. Thus transforming raw data into useable information which can be used internally or for business purpose is also a challenge. Challenges in Value testing: - Data which is unauthorized, unreliable may still prove decisive - Data in incomplete form - Data which is of no use for the current business requirement needs to be filtered out. Doing so from unformatted data is difficult. Approach for Value testing: - Filtering out required data using data extraction tools - Targeting the business requirements and focusing on sampling out data based on the same - Verification and validation testing

10 9 5. Big testing approaches: Cloud resourcing: To handle big data test, it is impossible to handle alone. Availing cloud resources for this task would be helpful. Predictive models based testing: Predictive modeling is the process by which a model is created or chosen to try to best predict the probability of an outcome. In many cases the model is chosen on the basis of detection theory to try to guess the probability of an outcome given a set amount of input data, for example given an determining how likely that it is spam. Incremental testing approach: The incremental build model is a method of software development where the model is designed, implemented and tested incrementally (a little more is added each time) until the product is finished. It involves both development and maintenance. The product is defined as finished when it satisfies all of its requirements. This model combines the elements of the waterfall model with the iterative philosophy of prototyping. Traditional testing types such as DWH testing, Performance testing, Security testing etc can be tweaked so accustom with big data. A/B Testing: A/B testing is a methodology in advertising of using randomized experiments with two variants, A and B, which are the control and treatment in the controlled experiment. A/B testing allows you to generate your own data. That is, organizations can be proactive with regard to data management. This is in stark contrast to the practices of far too many companies that rely almost extensively on much more reactive data management. A/B testing is hardly a panacea. site that gets 50,000 unique hits per day can reasonably chop its audience into two and, in the end, feel confident that any results are genuine. Consider the famous quote by Steve Jobs: It s really hard to design products by focus groups. A lot of times, people don t know what they want until you show it to them. Business Week, May Failover Testing: Failover testing is an important area in Big data implementation with the objective of validating the recovery process and to ensure the data processing happens seamlessly when switched to other data nodes. Machine Learning: Machine learning, in short, refers to computers learning to predict from data. Machine Learning has empowered many smart applications. For example, Apple s Siri learns from data to predict the meanings of human voice and the desired answers or actions to be performed. Facebook s photo album learns from data to predict (or recognize) faces to be tagged in photos. LinkedIn learns from data to predict who you want to connect with. Google s driverless car learns from data to predict the appropriate driving actions.

11 10 Artificial Intelligence and Machine Learning: In a move that signals a significant step towards automation in the IT services outsourcing business, Infosys has struck a partnership with IPsoft, the New York-based company founded by Indian American Chetan Dube that provides tools that free engineers from mundane, repetitive tasks. The most fascinating and influential aspect of IPsoft's technology is that it includes the element of machine learning -- or artificial intelligence as some call it -- so that companies don't have to employ an army of people to write the complicated scripts that traditional automation tools require. The system learns from doing, thus making the process of automation itself automated.

12 11 6. Big Data Landscape: Hadoop has the capability to process extremely large volumes of data, much faster and at a fraction of the cost of traditional data systems. Hadoop is an Open Source data management with Scale-out storage and distributed processing. Storage- HDFS: - Distributed across nodes - Natively redundant - Name node tracks locations Processing-Map Reduce: - Split a task across processors near the data and assembles results. - Self healing, High Bandwidth Clustered storage.

13 12 7. Big Hypothesis Big data can definitely give lots of input for sentiment analysis, trend patterns, behavioral patterns, future prospects, customers inclinations etc. In other words, most of these insights are the seeds of hypotheses. But there is no guarantee that the correlations discovered in big data can directly influence customer behavior. There are lots of factors involved in it. For one, big data naturally indicates large data, still it s unending and so in a way incomplete. Also as it comes to capturing unformatted data, we cannot apply a specific rule over the kind of data which is getting exchanged amongst the people. Thus, putting this raw data into a formatted one definitely has limitations. This is the same data which holds potential to turn all the equations upside down and start following an absolutely different trend. But we surely need to find out the best possible ways for the betterment of streamlining the monstrous data into various readable, manageable, and properly catalogued and testable formats. Big data is a powerhouse which is generating both good data and bad data. Need of the time is to massage the data, clean it and use it.

14 13 8. Big Data limitations: % coverage not possible, best fit plan has to be chosen depending upon the total time, resources, and tools available for testing. - Big data testing may invite unidentified areas, or in other words, unstructured data may get misinterpreted in the walk of testing, which may leave testing incomplete / uncovered for that specific area. - Giraffe Effect: Giraffes are a portions of data which dominate the rest of the data and hide important insights. Sometimes they even lead to wrong conclusions. This is a very simple example of the giraffe effect. When people look at a set of data which includes some very large, dominant members, important differences among the other data in the set often disappear from view.

15 14 9. Conclusion Big testing, if properly harnessed and experimented with innovating ideas can lead to big changes in the marketing trends. Also, it can help betterment of the business giving it new shape and zeal. Big data is of the people, created by the people, and is useful for the people. Nowadays we get some features customized as per our preferences, we get choices as per our selection, and some websites conduct surveys and based on that we get some interesting deals in our mailbox. Big testing can certainly highlight some uncovered areas where business can target and provide more useable services to customers. Since people are the creator and contributor of this big data, they can bring in more ideas in testing big data. As there is no limit to data, there is no limit to creativity as well. People well equipped proper tools and technologies can play a vital role in big data testing. Hal Varian, the chief economist at Google, has said that Google runs about 10,000 experiments each year. A large number of different people throughout the company are engaged in all kinds of different tests in parallel. This culture is now being followed by many companies. There is really a need to start experimenting, put forth our hypotheses and agree / disagree to them, passing them to next level to formulate a test structure, integrate all modules into one to give a desirable result to big testing. As far as infrastructures are concerned, cloud resourcing is the best possible option, wherein different tests can be simulated. Questions like who owns the data, what are the challenges involved in big data, is this data useful for us list is endless. And answer is Big Testing!!!

16 References devcentral_f5_com/weblogs/macvittie/windows-live-writer/the-four-vs-of-big- Data_4DB7/big%2520data%2520four%2520vs_2.png&imgrefurl= f5.com/blogs/us/the-four-v-rsquos-of-bigdata&usg= QnGAwzJ91QX5lYCfubWQpb1HFnk=&h=767&w=1024&sz=601&hl=en& start=1&zoom=1&tbnid=q03bq3qp4cx_jm:&tbnh=112&tbnw=150&ei=knpnuaeuc8 morqf4sic4cg&prev=/images%3fq%3d4%2bv%2527s%2bof%2bbig%2bdata%26s a%3dx%26hl%3den- IN%26gbv%3D2%26tbm%3Disch&itbs=1&sa=X&ved=0CCsQrQMwAA

17 Author s Biography Renuka Kale is working as a consultant for RTQA, Morgan Stanley account in Capgemini India Ltd since 4 th Aug Renuka has around 10 years experience which includes Govt sector and IT, out of which around 6 years counts in software testing. While working with Govt of Maharashtra, she has presented a paper women s empowerment in water sector at an international water conference Water Asia.