ACHIEVING BUSINESS VALUE WITH BIG DATA. By W H Inmon. copyright 2014 Forest Rim Technology, all rights reserved

Transcription

1 ACHIEVING BUSINESS VALUE WITH BIG DATA By W H Inmon

2 First there were Hollerith punched cards. Then there were magnetic tape files. Then there was disk storage followed by parallel disk storage. With each advance in storage technology came new capabilities speed, capacity, direct access of data - and a drop in the unit cost of storage. Now there is Big Data, where we can store even greater volumes of data at even lower cost per unit of storage. Big Data has opened the door to the collection of volumes of data and types of data that previously have gone unnoticed in the corporation. Now organizations can afford to capture and store data like never before. But with the advent of Big Data comes a related issue - how do we get business value out of the data collected into Big Data? Fig 1 shows that ultimately business value needs to be derived from Big Data in order for Big Data to be adopted and used by organizations. Big Data Business value Big Data has several characteristics that mark it a differently from other types of data. Some of those characteristics are - The unit cost of storage of Big Data is low (as compared to other forms of storage) - Big Data is collected and stored in an unstructured fashion - Big Data is accessed and managed in an indirect manner that does not lend itself to high performance transaction processing - The volumes of data stored in Big Data are orders of magnitude greater than that that was possible with other technologies. There are undoubtedly other characteristics of Big Data that set it apart from earlier storage technologies. So what are organizations technicians and business managers focusing on today as Big Data becomes an accepted part of the technical landscape? Fig 2 shows some of the aspects of Big Data that people are focusing on

3 Hive Hadoop Pig Map Reduce Cirro Hbase In Fig 2 are found such topics as Mongo, Cirro, PIG, Hadoop, Hive, HBase, MapReduce and other aspects and features of Big Data. These and other technological aspects are part of the technological landscape of Big Data and each of these topics are of interest and are deserving of study. But these are technologies and as interesting as they are do not directly address the topic of deriving business value out of Big Data. Another way to understand the phenomena of Big Data and deriving business value is shown by Fig 3. Big Data Unstructured Data Business Value Fig 3 makes the point that in order to get to business value out of Big Data that you have to go through another entirely different technological barrier, and that barrier is making sense of the unstructured data raw text - that comes with Big Data. The technology of Big Data is one challenge to be conquered, but once you have met the Big Data challenge, you are automatically confronted with the challenge of making sense of unstructured data, because all data in Big Data is unstructured. It is noted that languages that access Big Data NoSQL certainly have a place but any access language - by itself - does not address the fundamental challenges of making sense of unstructured data. Instead unstructured data needs to pass through a separate transformation process before the unstructured data found in Big Data can be used to achieve business value. Stated differently, in order to achieve business value you have to address two very different technological challenges the technology of Big Data and the transformations that come with unstructured data. In a way Big Data merely introduces unstructured text. Once introduced, unstructured text then must be transformed in order to achieve business value. TRANSFORMATIONAL CHALLENGES

4 In order to illustrate the challenges of transformation of unstructured data, there are several important issues that arise with trying to make sense of unstructured data. Some of those many issues of transformation are - The need for context - The need for interpretation - The need for simple editing - The need for certain forms of standardization - The need for complex editing - The need for acronym resolution, and so forth. This list merely scratches the surface of the many and diverse transformations that need to be done to unstructured data in order to unlock the business value of unstructured data. For further explanation of many of these transforms refer to the web site However, this list is a good starting point to explain why raw text cannot be used to achieve business value by itself. CONTEXT The first and most obvious shortcoming of raw text is that it needs context in order to be used to achieve business value. The raw text found in Big Data has either no context or very limited context. Contrast the raw text found in Big Data with the classical records of data typically found in a dbms. In a classical dbms there are records. A record is made up of keys and attributes. The attributes found in the dbms provide context to the data contained in the dbms. As a simple example, suppose that in Big Data we find the string of text -.Joe Foster.. Fig 4 shows the unstructured string of text that contains the name Joe Foster....Joe Foster... What is the meaning of Joe Foster in Big Data? Is Joe a customer? A policeman? A preacher? A race car driver? We just don t know much about Joe Foster when Joe is found in a raw unstructured string of data. But suppose we have a dbms record, and in the dbms record data is recorded as Customer number Customer name Joe Foster Customer address 635 Adrian Lane, Tuscaloosa, Alabama..

5 Once we find the record for Joe Foster in a dbms, the attribute metadata tells us a lot about Joe Foster. In a dbms there is context about Joe that allows us to use Joe accurately and succinctly in making business decisions in the computer. So one of the fundamental differences between unstructured strings of text found in Big Data and the records of text found in a dbms is the lack of context that exists in a string of unstructured data. Now consider the types of query that can be done on a string of unstructured text. In a query against the unstructured string we can ask does the string Joe Foster exist? Fig 5 shows we can ask if Joe Foster is found in Big Data. Simple query Can you find Joe Foster?...Joe Foster......Joe Foster... The query found in Fig 5 is a simple query. We can look through huge amounts of data in Big Data and find out if Joe Foster is found there. But because the data is all unstructured text we can t find out much else. We can t find out if Joe is a policeman, a soldier, a customer or a newly born baby. Because there is no (or very little context) found in unstructured text, the types of query that we can do against unstructured strings of data is very limited. And where the type of query is limited, the business value is limited. In order to do a more sophisticated query we need context to be able to understand the data that we operate on. The need for context in the process of query can be stated very simply as seen in Fig 6. simple query sophisticated query raw text raw text+ context Indeed, trying to make business decisions on text that does not have context can be anything but positive. As a simple example of the importance of context, consider two men that are standing on a street corner and a young lady passes by. One man says to the other She s hot. What is meant by the expression She s hot? Is the young lady attractive and one of the men wishes that he could have a date with the lady? Does she s hot mean that the lady is attractive? Or are the men on a street corner in Houston, Texas in July and the temperature is 99 degrees and the humidity is 100%. The lady is sweating from every pore in her body and she is physically very hot. Or has the young lady been just given a traffic ticket and she is mad? Does she s hot mean that the lady is irritated? Or are there even more interpretations of what is meant by she s hot?

6 Unless we know the context of what is being said, we can make a very wrong and embarrassing business decision. Not understanding context leads to incorrect interpretation, and incorrect interpretation leads to wrong assumptions. You cannot possibly make good business decisions when you do not understand interpretation and underlying assumptions of raw text. Furthermore, the need to understand the context of information is hardly limited to the words she s hot. ALL words ALL raw text - are open to interpretation and need context in order to be understood properly. And Big Data is nothing but raw text. The problem is that when raw text is found in the unstructured strings that are found in Big Data there is little or no context that comes with the raw text. The lack of context of raw text GREATLY limits the sophistication of the queries that can be done against unstructured data. Only the most basic of queries can be done as long as the data being queried has no context. And there is little business value in being able to make only unsophisticated queries. In order to achieve business value you need to be able to make sophisticated queries. INTERPRETATION AND COMPONENT CONTEXTUALIZATION As important as context is in making sophisticated queries of data, context is not the only element that needs to be considered. Another equally important element of making sophisticated queries is the ability to interpret raw text. Suppose that the following string of data is found in raw text f450-dnv-chi lybag cno po ag This kind of string might be found in a log record. What kind of query can be done against this text? A simple query can be done to find if the string f450 is present. And indeed Big Data can conduct such a simple search. But what business value is there is finding the value f450? The answer is not much business value here. In order to unlock the business value of the raw text, two activities need to be performed against the string f450-dnv-chi lybag cno po ag The first activity is that the raw string must be interpreted. Then after it is interpreted it needs to be component contextualized. As an example of interpretation the string can be interpreted to mean f450-dnv-chi lybag cno po ag December 15, 2012 flight 450 Frontier Airlines Denver to Chicago domestic flight lost yellow bag claim number po agent Ann Maeda. Then, after interpretation of the raw text is done, component contextualization parses the string and creates a context identification that might look like date December 15, 2012, airline Frontier, flight number 450, flight type domestic, action item lost item, item type bag, bag description yellow, flight source Denver, flight destination Chicago, claim number po , agent number 12356, agent name Ann Maeda. Once the component contextualization is done and is described to the system holding the data, all sorts of analytical process can be done against the raw string of data. Queries such as - How many bags are lost on domestic flights? - How many incidents occur in the month of December? - How many times has agent Ann Maeda had to track lost bags? Lost yellow bags?

7 - Of bags that are lost, how many are yellow? And so forth. Once interpretation and component contextualization are done, then sophisticated queries can be done. But until the raw text is properly interpreted, no sophisticated analytical processing can occur. And as long as no sophisticated analysis can occur, only limited business value can be attained. Once again, there is very little business value making queries that are simple. In order to achieve business value, you need to be able to create sophisticated queries. And in order to create sophisticated queries, you need to transform the raw text. SIMPLE STANDARDIZATION Contextualization and interpretation are absolutely essential in order to take raw unstructured text (like that found in Big Data) and turn it into a form that can be analyzed in a sophisticated manner. But these are hardly the only activities that need to be done to the raw text found in unstructured data (i.e., Big Data) to transform the unstructured data into a form that yields business value. Another simple but important activity that must be done in the transformation of unstructured textual information is that of standardization of certain types of data. As an example of the need for standardization, date values need to be standardized. Suppose that the following unstructured strings of data are found in Big Data March 13, , 2012/27/01 and.12/31/ To the readers eye it is obvious that these unstructured strings of text contain date values. But can the computer do comparison processing against these dates? Indeed does the computer even understand that these are date values? The problem is that these dates are in different formats. Trying to get the computer to do a comparison against these unstructured forms of date will yield unpredictable and unuseful results. To the computer these strings just look like raw text and are not particularly meaningful. In order to be useful, the unstructured form of dates must be converted into a standardized format. In this case the following transformation of date must be done March 13, 1999 == date /27/01 == date /31/2010 == date Once the raw text is recognized as dates, once the dates have their values recognized, and once the dates are converted into a common format, then the dates can be meaningfully compared to each other. But until a common standardized format is achieved, no meaningful comparison of dates can be done. And unless a comparison of dates can be achieved, limited business value can be achieved from the raw unstructured text.

8 Once again, until the raw text passes through a fundamental transformation, the queries that are done against it are not very sophisticated. And as long as queries are not sophisticated, there is limited business value. Until queries are sophisticated, achieving business value is not able to be done. OTHER TRANSFORMS Certainly contextualization, interpretation, component contextualization and standardization are absolutely necessary to be done in order to turn raw text into a form that can be used for sophisticated analytical processing. And in order to achieve business value it is necessary to do sophisticated analysis. But these transforms are hardly the only type of transforms that are needed in order to turn unstructured text into a form of data that can be analyzed in a sophisticated manner. Indeed, as important as these transforms are, they barely scratch the surface for what is needed in order to turn raw text into useful information from which business value can be derived. In order to see a discussion of more transforms, refer to Textual ETL as described on the web site With a complete and mature set of transforms, it is seen that raw text can be turned into a form of text that can yield business value, as depicted by Fig 7. Big Data textual transformation Business value AN ARCHITECTURAL RENDERING From an architectural perspective, how then does the business derive value from Big Data? Fig 8 shows that raw text is placed into Big Data, as the first step. raw text Big Data After raw text is placed into Big Data, one possibility is to use textual ETL in order to transform the raw text into a transformed state which is then placed into a standard dbms. Fig 9 shows this possibility.

9 raw text Textual ETL Standard dbms Big Data The raw text is transformed into a form that is able to support sophisticated queries and analytical processing. The transformation is done by textual ETL. However the output from textual ETL does not have to go into a standard dbms. An alternative is to send the output i.e., the transformed text - back into Big Data. By sending the output of textual ETL back into Big Data, whole new query possibilities are opened up. Now it is possible to use NoSQL for sophisticated query of data. Fig 10 shows the possibilities for different query types when textual ETL is used and where the output from Textual ETL is placed in either standard dbms or Big Data. raw text Textual ETL simple query Standard dbms sophisticated query Big Data transformed text Business value sophisticated query The Inmon/Krishnan Big Data Architecture copyright Forest Rim Technology, 2012, all rights reserved Business value In Fig 10, simple queries can be made on raw text found in Big Data or sophisticated queries can be made from standard dbms or from transformed text found in Big Data that has arrived via textual ETL. And once the queries are made against transformed text, then those queries can be very sophisticated. And once sophisticated queries can be run against the data that originates from Big Data, then business value is easily achievable. INTEGRATING BIG DATA AND CORPORATE DATA Another way to look at the value of Textual ETL is to consider the questions how is it possible to combine the data found in Big Data and the existing organizational environment. With current technology there are analytical tools for Big Data and there are analytical tools for the existing corporate environment. However for all practical purposes the different environments are about as different as night and day. However with textual ETL there is a way to integrate and incorporate data.

10 Big Data Textual ETL Corporate systems In Fig 11 it is seen that with Textual ETL data can be exchanged and integrated freely between the Big Data environment and the Corporate Systems environment. Bill Inmon the father of data warehousing works for Forest Rim Technology, a company dedicated to the technology for the management and usage of unstructured text. Forest Rim Technology is located in Castle Rock, Co. Forest Rim has a web site forestrimtech.com.