Navigating the Four Vs of Big Data: Shrinking the Haystack for Actionable Insights

Transcription

1 Navigating the Four Vs of Big Data: Shrinking the Haystack for Actionable Insights An ISS open source project

2 Table of Contents 1. Introduction and Overview 2. Optimizing the Three Vs of Big Data a. Volume b. Variety c. Velocity 3. Methods to Capitalize on Fourth V: Value a. Data integration (Acquisition) b. Enterprise Search c. Semantic Enrichment d. Data Perspectives 4. Use Case: A Retailer Gears Up to Fight Fraud 5. Conclusion

3 Introduction and Overview Private and public entities across the world are waking up to the fact that there is buried gold in their enterprise data. These companies wonder how to uncover the valuable insights spread throughout terabytes and petabytes of data located across documents, servers, applications and even real-time information streams? ISS CTO Wes Caldwell helps readers understand how a disciplined approach to the three Vs of Big Data Volume, Variety and Velocity yields the fourth V: Value. Mr. Caldwell shows how a strategy that connects integration to sophisticated, yet accessible, enterprise search technology can help your enterprise locate hidden treasure in your big data and provides a use case that illu strates how an international retailer saved millions through a fraud prevention application. How Many Vs? Since the turn of the 21st Century, technology thought leaders have seen the possibilities (and perils) of mounting volumes, sources and ways to marshal and understand data. These forward thinking individuals have been seeking the keys to how organizations would benefit from data, rather than being swamped or having useful data disconnected and slumbering away on various databases or storage media. Out of this thought process was born the Vs model for understanding Big Data. As one example, in a recently released infographic ( com/infographic/four-vs-big-data) IBM cites four Vs Volume, Velocity, Variety and Veracity as a framework for understanding the Big Data picture. Others throw Volatility and Validity into the mix. Any number of these Vs can be combined for a valuable framework for understanding the elements and challenges of the arena. Gartner identified three Vs back in 2001 and also identifies 12 associated dimensions of Big Data. You can cite as many factors as you like, but this alphabet soup all needs to add up to the ultimate V: Value. This value can be the ability to spot a key trend that was previously invisible, finding cost savings from looking at an initiative or line of business from a different logistical perspective, or detecting a point of contact with customers or clients where small improvements can yield significant gains. CHALLENGES Enterprises struggle with how to leverage the insights in their company data Volume, variety and velocity of data complicate Big Data efforts TAKEAWAYS Learn the basics of the data landscape and the critical building blocks Enterprise search and NLP (natural language processing)- enhanced semantic enrichment shrink the data haystack and put your analysts in charge For purposes of this white paper, we will discuss three of the foundational Vs Volume, Variety and Velocity and how to meet their challenges to yield the one that really matters: Value. At the end of the day, you can have all the Vs you want, but if there s no value, it s all an academic exercise.

4 Optimizing the Three V s of Big Data Whether you re in counterterrorism, the financial sector, or any other data-driven industry, there are three major factors that set the parameters of your Big Data challenge: Volume, Variety and Velocity. VOL ME The first challenge is the sheer volume of data. CSC, a global provider of IT solutions, notes that data production will be 44 times greater in 2020 than it was in That s a growth rate from.79 ZB (zettabytes) to 35 (zettabytes). For purposes of scale, one zettabyte equals one billion terabytes. The explosion of mobile platforms, transaction-based data storage, data produced by sensors in civilian and military equipment, and massive streams of social media data are all contributing to the explosion. Every enterprise will experience a version of this growth to varying degrees. In the past, storing it all was the challenge; but as storage solutions become cheaper and more accessible, efficiently integrating it and analyzing it rises as the next big hurdle. ARIETY The next challenge to the ability to achieve value and actionable intelligence is the extremely wide variety of data structured, unstructured, and semi-structured that must be integrated if we are to leverage the information and knowledge lying dormant inside of it. A single agency, enterprise or organization will potentially have a vast repository of information spread across multiple media (documents, images, etc.), human interactions, geospatial, open source and video sources, and financial transactions, just to name a few. Integration that connects and standardizes these formats is key; and this is sometimes the only way that specialized applications and human analysts can begin to extract their wealth. VEL CITY If any facet of analysis depends on combining legacy today with a picture of what s happening up to the minute, velocity must be taken into account. This is the speed at which relevant data enters your purview, whether through RFID tags, financial market aggregators, drone sensors, or the ever-changing (and always fast) stream of online and social media data. Timeliness and speed will be central to many efforts to capitalize on Big Data, ranging from minimizing global supply chain disruptions to apprehending international criminals.

5 Use Case: A Retailer Gears Up to Fight Fraud The Global Retail Theft Barometer reported that losses from shrink, which include shoplifting, employee fraud, organized retail crime (ORC), and administrative errors, cost retailers more than $112 billion in 2012 alone, representing 1.4 percent of retail sales on average. A European-based clothing retailer with an international presence wanted to better understand loss patterns related to fraud and theft. Of the three categories of loss internal theft, external theft, and errors; employee fraud often accounts for the largest portion. In some cases, employees can report their own theft as shoplifting. Point-of-sale (POS) is also a common source of loss, e.g. an employee intentionally entering a low price for a higher-priced item. Being able to integrate various data sources register, sales databases, and schedules chief among them would enable a way to perform more fruitful loss enforcement analysis. With these sources integrated, custom searches built for fraud analysts are now helping them sift through the data and reduce the enterprise haystack, discovering and isolating key factors such as: Losses against current inventory Inventory losses when no theft or shoplifting is reported (e.g. possible employee theft) Incidents of single employee reporting shoplifting events with no other witnesses Consistent drops in sales when a certain employee works Patterns of overages and shortages at registers Unusually high instances of refunds, voids or no-sales weighed against schedule and authorization level of employees Patterns of increased sale of one item when loss occurs on more expensive items By being able to make sense of the patterns, internal loss prevention staff can narrow their efforts to the most likely suspects for investigation, or in the case of external threats, bolster security during high-shoplifting hours. The potential gains are in the millions of dollars globally as this retail leader follows the path from content acquisition to insight. Data Integration Enterprise Search Big Data Analytics Un-Structured Data Semi-Structured Data Structured Data Connectors to popular content sources Acquiring, normalizing, & staging data for indexing & analytics Search tuned for user-defined terms Optimized index for search & discovery of large data sets Data analytics delivered to the user Hadoop-based big data solutions

6 Stage 1: Content Acquisition The first stage of enabling these gains is to pull all information into a common environment so that it can be pushed through an analysis pipeline. Information must be pulled into a common environment. The recommended process, which we call content acquisition, has these four steps: 1. Building a connector architecture to various data sources. 2. Normalization, which takes heterogeneous data and puts it into a form that can be reasoned with (however, the data is not completely structured at this step). 3. Data staging, whereby relevant, normalized data is cached in a Big Data system such as Hadoop. 4. Compartmenting, in which data is placed in relevant buckets - for example, financial records in one and human resources data in another. This partial segmentation narrows massive amounts of data to more usable subsets and can also be used to avoid security problems (such as hierarchical access needs). Stage 2: Search / Discovery Enterprise search is the first of a one-two punch that eventually enables actionable insight from what was previously an unmanageable mountain of data. After content acquisition, content is indexed and pushed into its own optimized search engine. This index can be tuned for the kind of search and discovery that supports the kind of queries your data analysts need to make. Analyst topics empower teams to save certain searches and filters, making it easier to whittle down the haystack. Additional capabilities, such as auto complete, comments and semantic (synonym) features help analysts from defense to financial applications use custom search to discover the discrete data they need to push farther down the Big Data pipeline. Stage 3: Semantic Enrichment NLP (natural language processing)-driven semantic enrichment represents a further refining and enhancement of the search experience, setting the stage for deeper analytics. Search and NLP are the one-two punch that fuses what the analyst knows with what he or she doesn t know, allowing users to constantly tune and refine smaller subsets of data for key factors. The system learns as the user refines their searches to better target their data domain, constantly improving search effectiveness. As two examples of use cases - counterterrorism experts looking for a particular piece of equipment involved in bombmaking or financial analysts trying to isolate a particular kind of transaction now have formidable power as NLP helps categorize and refine key information. Stage 4: Data Perspectives As refined searches isolate the critical content caches, data perspectives arise as a product: The ability to reduce the data gleaned from targeted queries and roll it up into graphs, time-series databases, geospatial representations and more, revealing connections and trends that were invisible at the beginning of the process. Countless rows and columns become elegant visual representations of the analyst s query, enabling quick communication of vital findings within and outside of the enterprise. This capability means your team is poised to reap the fruits of Big Data. Let s now take a look at a current application in the financial sector.

7 Conclusion All the Big Data buzz comes down to one big question: Can you leverage all the information you have so that analysts can feed better decision-making data to your leaders? If not, then none of it really matters. It is easy to become lost in the buzzword jungle. We wrote this paper to provide a high-level view of the concepts you need to understand and the steps you need to take to position yourself for game-changing insight. From what we have seen work in applications both in the civilian and commercial sectors, proper acquisition, enterprise search and NLP processing that constantly improves search is the true basis for success with Big Data no matter how many Vs people identify. About Springblox Intelligent Software Solutions formed the Springblox open source enablement team to meet the data integration and search needs of enterprise customers large and small. Our core team of MuleSoft and Apache Solr experts have implemented and maintained many complex, mission critical data integration, search and big data applications for several public and private sector companies. By pulling together this elite group and partnering with recognized commercial open source leaders, Springblox brings a unique value proposition to the market: a highly-focused boutique consultancy with the size, breadth, and reliability of an enterprise-class systems integrator. The result is a one-stop shop for consulting, implementation, training and production support for cloud and big data solutions. For more information, please visit

8