TEXTUAL ETL THE COMPONENTS. A WHITE PAPER BY W H Inmon. copyright 2014 Forest Rim Technology, all rights reserved

Transcription

1 TEXTUAL THE COMPONENTS A WHITE PAPER BY W H Inmon

2 For years, data bases have held numeric, repetitive data, typically generated by transactions. The same structure of data is repeated over and over. Each record contains different values but the structure of the data remains constant. This has been the pattern for data held in a standard relational data base management system for many years. Such an approach works well for operational, transaction based data. But over the years there is an entire class of data that has been ignored. This class of data is ual data. data is neither numerical nor is it repetitive (for the most part). Because of its erose and irregular structure, ual data does not fit comfortably in a standard relational dbms. For this reason ual data has not been placed in a standard dbms and because of this corporate decision making has been done on the basis of numerical, transaction based data. (Of course ual data can be placed in a blob. The problem is that once placed in a blob, trying to do any serious analytical processing with the ual data is an impossibility.) TEXTUAL But now there is by Forest Rim Technology. With you can meaningfully place ual data in a relational data base and you can do meaningful analytical processing on the data base when you are finished using. is NOT search technology nor data mining technology. Instead is integration technology. Unlike search technology, makes the assumption that MASSIVE changes to the data being operated on are in order. Nor is classical legacy systems. Classical legacy systems is that is designed to integrate older legacy systems into a data warehouse. Instead integrates into a data warehouse. Managing is VERY different from managing legacy systems. For the most part is an enabling technology. is designed so that it is easy to build applications on top of the output of. NLP AND TEXTUAL In building the product, the developers did not make the major error of building the product based on NLP technology. There are many problems with NLP technology that are intractable. Huge amounts of research and academic interest in NLP technology have produced scant commercial results over the years. Instead, does account for the con of, but does so in a very different manner than NLP technology. WHAT TEXTUAL DOES In a word, ual reads and allows as input an electronic source of, integrates the, and produces as output visualization or a relational data base that can serve as a basis for standard Business Intelligence processing or Business Intelligence.

3 Note: 9 patents have been filed on the technology that is being described. If you want to copy this technology or embed it in your technology, you are advised to speak with Forest Rim Technology for licensing the protected parts of the technology. Business Intelligence Data base Legacy systems Foreign extension Spread sheet Taxonomies SOMs Business Objects Cognos MicroStrategy SAS Crystal Reports OCR INPUT INTO TEXTUAL The input into is electronic. The can be in English, Spanish, German, French, Italian or Portuguese. Note: can handle ANY form of formal, informal, notes, shorthand, etc. One of the most common forms of electronic are files that have the extension type of.txt,.doc,.docx, or.pdf. These extension type can be fed directly into. However, there are many non standard extension types. These foreign or non standard extension types can be passed through Word and recreated as a standard extension type. Foreign extension Word On occasion, is found in a data base. On other occasions (such as IM) is in a form where it is convenient to place the into a data base. On those occasions the that is in a data base can be read directly into.

4 Data Base can be entered into. In order to enter it must first pass through a filter which eliminates spam and blather, thus reducing the raw size of the s that must be processed. Like all documents, the address of the source document is kept so that the analyst can always get back to the source at any point in time if needed. Filter In addition to s, can receive as input spreadsheets. Actually can receive the ual contents of a spreadsheet. The numeric contents of a spreadsheet require special handling. Spreadsheets are handled through a special filter. Once the information from the spreadsheet is passed through the special filter, it enters as any other form of electronic. Spread sheet Filter can handle data that is source from pencil and paper. In order to do this the paper based must pass through OCR (Optical Character Recognition). Once the has passed through OCR, the is in an electronic form and can be processed by. Electronic format OCR

5 Occasionally needs to be lifted from legacy systems into. This transfer of is done with an interface. On some occasions the interface already exists. On other occasions the interface can be easily and quickly built using standard tools such as classical And as a last resort, a custom interface can be built. In any case once the interface is built, can be directly passed into. Legacy systems Interface TAXONOMIES An important input for most processing is taxonomies. Taxonomies are useful in helping to resolve terminology and in filtering . Forest Rim Technology can operate with taxonomies that have been built by the client or Forest Rim Technology. Forest Rim Technology has access to over 29,000 professionally built and maintained taxonomies. In most cases it is simply a matter of selecting the 4 or 5 taxonomies that are the most relevant and installing them. This is done in a matter of minutes. THE TEXTUAL MODULE At the heart of is the module that does ual integration. In ual integration, in its many forms is ingested and transformed into a form that is suitable for a relational data base. The relational data base is created so that standard SQL processing can be done from it. In addition ual analytic processing can be done as well. The resulting data base can be

6 used for standalone analytics or for analytics which are simultaneously done against both structured data and unstructured data at the same time. VISUALIZATIONS - SOMs One form of output of is visualization. The visualizations are in the form of a SOM (Self Organizing Map). SOMs look at and visualize all the data in the source documents (not just keyword.) SOMs are quite useful for correlative analysis and clustering. SOM RELATIONAL DATA BASES AS OUTPUT Another form of output from are relational data bases. creates DB2/UDB data bases, Oracle data bases, SQL Server data bases, Teradata data bases, and other relational data bases. is agnostic to the type of relational data base that is created. can create up to 35 different types of tables. There are many different forms of output from and each form can be used to produce a different type of relational table. The tables are designed so that analytical joins can be created. In truth the tables taken together are much more powerful analytically than any one given table. SQL Server Teradata DB2/UDB Oracle The output tables are designed in a form that is easily read and manipulated by standard Business Intelligence software.

7 Bi3 Solutions Business Objects Cognos MicroStrategy SAS Crystal Reports TEXTUAL BUSINESS INTELLIGENCE In addition, for certain types of output from, it is necessary to use a special form of Business Intelligence. In this case Forest Rim Technologies Business Intelligence can be used. Business Intelligence INSTALLING TEXTUAL is normally installed in about ½ an hour. If the client wants to use only default settings, results from can be achieved about an hour from the time the software is installed. is designed to be used in an iterative manner. Normally the defining parameters for a complex document are not defined correctly and properly the first time. Therefore, is designed to be used repeatedly until the defining parameters are correctly and completely defined. SCALABILITY was designed with scalability in mind from the very first draft of the product design. The limitations on throughput that can be achieved are strictly a function of the size and the amount of the hardware that you wish to throw against the problem at hand. For that reason we say that the software is never the limiting factor in the volume of that can be handled. A KALIDO INSPIRED DATA BASE DESIGN

8 The data base design of the resulting unstructured data is a Kalido inspired data base design. Rather than use a conventional approach to output relational table design, the Kalido inspired approach was chosen in order to allow for the maximum flexibility of data base design over time. Forest Rim Technology was formed by Bill Inmon in order to provide technology to bridge the gap between structured and unstructured data. Forest Rim Technology is located in Castle Rock, Colorado. Forest Rim Technology is happy to provide you with an actual demonstration of the techniques and tools described in this document. Forest Rim Technology is happy to provide a demonstration over the Internet.