Large-Scale Test Mining SIAM Conference on Data Mining Text Mining 2010 Alan Ratner Northrop Grumman Information Systems NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
Aim Identify topic and language/script/coding of real-world informal text at highest speed possible Informal text Blogs, posts, tweets Don t necessarily follow conventional rules of spelling & grammar Transliterated language usually ad hoc In typical web documents far more bytes of HTML & JavaScript than content making everything look like English. Can be parsed out but timeconsuming. Does not look like newswire (written by journalists, rich in named entities, summarized in first paragraph) High speed Want to process documents as quickly as possible Trillions of web pages Gigabytes per second speed desired 2
What is Text? 3 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
6 Fundamental Questions in Mining Text Detection 1. Does a document contain text in any language? (Or is it audio, video,?) 2. If so, does the text have a topic? (41% of tweets are pointless babble.) Clustering 3. Which documents are in the same (but not necessarily known) language? 4. Which documents are on the same (but not necessarily known) topic? Identification 5. What is the language? (Is it a language the system has been trained to recognize?) 6. What is the topic? (Is it a topic the system has been trained to recognize?) 4
Specific Goals Identify specific language(s) of documents and identify documents on specific topics Accuracy requirements False negatives OK - users don t know what has been missed Precision needs to be high enough so we don t annoy users with lots of false alarms Extremely low false positive rate for topic id (<< 1 ppm) Speed requirement Fast hardware or software; simple algorithms Ideally use same algorithm for both language and topic id Language-neutral Work on all languages, including Asian languages that do not parse words with spaces 5 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
Text Analysis Algorithms Many algorithms have been used for topic id Bayesian Markov Markov Orthogonal Sparse Bi-word Hyperspace Correlative Entropy (Optimal Compressor) (longest string match) Minimum Description Length Term Frequency*Inverse Document Frequency Morphological Centroid-based Logistic Regression (similar to SVM and single-layer NN) Most can be expressed using additive weights of detected tokens In the domain of interest (low false alarms on informal text) logistic regression worked best and is computationally efficient 6 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
Additive Weight Algorithms Training Define & select tokens (e.g., words, words with spaces before & after, phrases, N-grams) Assign weights (for LR weights range from roughly -1 to +1) Testing Detect tokens Add weight to summer (S) For each document, convert weight sum to likelihood score & compare to threshold P(on-topic or on-language) = 1/(1+e -S ) 7 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
Token Detection In hardware Load tokens and ternary bit masks into CAM (Content Addressable Memory) Stream data through CAM to automatically identify token(s) In software Aho-Corasick algorithm creates a large state machine e.g., 50K tokens with 130K states Terminal states indicate detection of a token For each byte of data Next_state = TableLookup[Previous_state][New_byte] If Terminal_state[Next_state] == TRUE then Retrieve weight and add to summer Execution time is relatively independent of number of tokens or average length of tokens 8 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
4-Token State Machine sp t h e sp the n sp then y sp they a n d sp and transition to orange, green or pink state if space, t or a else transition to blue state 9 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
Solving Text Analysis Problems using Hadoop Hadoop is a framework for writing and running applications that process vast amounts of data in parallel on large clusters (up to thousands of nodes) of commodity hardware in a reliable, faulttolerant, manner. Hadoop is free/open-source software that emulates Google s proprietary MapReduce The master node takes the input, chops it up into smaller sub-problems, and distributes those to slave nodes where the Map tasks ingest and transform the input The Reduce task(s) then aggregate or summarize the Map output to deliver the final output. 10
Hadoop Software Hadoop will run on anything from a laptop to a vast cluster of computers Program Stacks Windows/Cygwin/Hadoop Windows/VM/Linux/Hadoop Linux/Hadoop Linux/VM/Linux/Hadoop Software packages work with Hadoop to provide: scalable distributed file systems and data warehouses (HBase, CloudBase) data summarization, ad hoc querying, scripting (Pig, Hive) massive matrix math, graph computation, machine learning, social network analysis (Hama, Mahout, X-Rime, Pegasus) 11
Hadoop Data Flow with 1 Reduce Slave 1 Map Combine Reduce Input File 1 Slave 2 Map Combine 12 Input File 2
Hadoop/MapReduce Hadoop automatically distributes blocks of data to slave nodes and then lines of text to Maps Hadoop automatically sorts and groups outputs of 1 node s Maps by key Hadoop automatically sorts and groups outputs of all Combines by key One output file from each Reduce Main/Run code defines interfaces & loads globals into distrib. cache Your Map alue pairs code transforms one line of text & outputs KVPs Your Combine code transforms sorted & grouped output of all Maps for one node & outputs KVPs Your Reduce code summarizes sorted and grouped output of the Combines & outputs KVPs 13 Your Map: Configure code defines Map globals Your Combine: Configure code defines Combine globals Your Reduce: Configure code defines Reduce globals KVP = Key Value Pair
Original Hardware-Based Text Analyzer High speed solution with expensive special hardware HW limitations (# & length of tokens, wildcarding in ternary CAM) Few people with FPGA/VHDL skills 14
New Improved Hadoop Text Analyzer High speed software solution on generic hardware Enabled use of a very sophisticated detection algorithm (Aho- Corasick Trie as in information retrieval ); unlimited token length; speed relatively independent of number of tokens Many people with Java skills 15
Cluster Configuration Master & Slave 1 Slave 2 Slave 3 Master finds slaves using IPs in Host Table Each server may host more than one slave Each slave may run many Maps Cluster may have 1 or many Reduces Slave N 16 Our servers each have 2 quad-core Nehalem Xeons, 24-48GB RAM, 4 1TB drives Per rack: 328 cores, 1TB RAM, 164TB drives, 30KW, 1700 pounds
Language Identification Results Performance varies No standard data set or procedure for testing language identification Worked very well overall except on documents with just a few words Mutually intelligible languages such as Dutch/Afrikaans, Indonesian/Malay, or Norwegian/Swedish harder to distinguish than dissimilar languages Relatively few tokens (most common words in informal language) used for each language (7 for Hindi, 55 for English, 95 for Spanish) it is possible to construct difficult documents Could not distinguish random words from real language Language is defined as words and grammar 17 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
Topic Identification Results Based on incremental content of newsgroup posts (quoted prior posts and metadata such as newsgroup, thread, author removed.) 18 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I
The Bottom Line Text analysis readily performed on a cluster of commodity computers using Hadoop Comparison between software and hardware solutions In hardware achieved 2.5 Gb/s real-time throughput (but most such links operate at a fraction of their capacity) In software achieved 1.3 Gb/s off-line throughput on a cluster of 64 ancient servers (on new but unoptimized servers 1.6 Gb/s) In a properly configured Hadoop cluster performance scales linearly Elapsed time = 15-20 seconds + C/(number of servers) 19
Alan.Ratner@ngc.com 20 NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I