Project 2: Term Clouds (HOF) Implementation Report. Members: Nicole Sparks (project leader), Charlie Greenbacker

Transcription

1 CS-889 Spring 2011 Project 2: Term Clouds (HOF) Implementation Report Members: Nicole Sparks (project leader), Charlie Greenbacker Abstract: This report describes the methods used in our implementation of a solution to the term clouds task. When our system receives a request via the REST interface, we begin by fetching the HTML file specified by the query URL. The HTML file is processed in order to extract the primary content of the page and to separate that content into several different groups according to HTML tags. Lists of unigrams, bigrams, and trigrams are built from the document, with initial weights assigned based on their frequency distribution and PMI scores. Cues from HTML formatting are used to boost the weighting of ngrams appearing in certain HTML tags. At last, these lists are combined and the weights are balanced in order to produce the final set of terms and weights, which is then returned via the REST service and rendered as a term cloud. REST Interface: Our system provides a REST interface to respond to queries from the project evaluation platform. To build the REST interface, we used the Django framework, which enables rapid development of Python web applications. Django also provides a simple web server to host such applications. We created a Django app that listens for GET requests conforming to the project specifications. When a request is received that is prefaced by the cloud prefix, the Django server initializes the web app we built. This app simply retrieves the URL from the GET request string, and passes this URL to a wrapper function serving as an entrypoint into the business logic of our system. This wrapper function ultimately returns an appropriately formatted string containing the terms and weights extracted from the HTML page located at the query URL, and this return string is delivered via an HTTP response through the REST interface back to the project evaluation platform to be rendered as a term cloud. HTML Parsing: First, we strip the single quotes from the URL parameter submitted via the GET request, and add the prefix if it is missing. We retrieve the HTML file located at the URL, and use the BeautifulSoup package to parse two versions of the HTML. The first parse tree is based on the original, complete HTML file; this version is used to extract terms from the HTML head elements. The second parse tree build by BeautifulSoup is based on the main textual content of the web page, as identified by the ArticleExtractor feature of the BoilerPipe web API. This parse is used to extract terms from the body of the HTML file.

2 Text Preprocessing: We organize the text of the web page into eight groupings from which we extract ngrams. These groups are based on the text contained in the original HTML tags. The eight groupings are: Title of the web page Description in meta tags Keywords in meta tags Headings in <H[1-6]> tags Text in hyperlinks Bold text in <B> tags Italicized text in <I> tags All text in the body of the web page (which contains all text from the other grouping except for the description & keywords) A series of preprocessing steps are performed on each group: 1. All remaining HTML tags are removed 2. Newline characters are stripped-out 3. Hypens appearing outside of hyphenated words are eliminated 4. All instances of the HTML double quote entity (") are removed 5. All commas are stripped-out, unless used as thousands separators in large numbers 6. All other non-alphanumeric characters not previously mentioned (plus spaces, slashes, and dollar signs) are wiped out 7. Tokens appearing in nltk.corpus.stopwords.words( english ) are removed, along with common contractions and other unwanted words (e.g., 'whose','said') The remaining raw text in each group is converted into a list of strings and passed to the ngram extraction module for further processing. Extracting Ngrams/Initial Weighting: Text from the Keywords, Headings and Body groups are combined together into one master list. This list is then processed using NLTK s Frequency Distribution, which creates a list of each unigram and their counts. Dividing the count of each unigram by the number of items in the master list results in the unigram s TF value. NLTK s bigram and trigram collocation finder are then used on the master list to produce a list for bigrams and trigrams, which are scored using the collocation module s PMI bigram measure and PMI trigram measure respectively. Initially, weighting was based on TF/IDF where TF was calculated as described above and IDF was obtained using Microsoft Ngram Service. Surprisingly, simple TF scores alone provided more reasonable results (once stopwords were removed) than TF/IDF based on ngram probabilities. It seems likely that including the IDF from Microsoft Ngram Service diluted the score based on statistics from documents of potentially many classes. Thus, we removed the IDF portion of the weighting and therefore the use of MS Ngram Service from our final system.

3 Re-weighting Ngrams (HTML Tags): Each of the three lists (one for unigrams, bigrams and trigrams) is compared against the eight groupings based on the original HTML tags. For ngrams found in the Title, Heading, Description or Keywords groups, a multiplier of 3 is applied to the TF value. A multiplier of 2 is applied to ngrams found in groups Links, Bolds and Italics. All other ngrams TF values are unchanged. For any ngram, only one of the two multipliers is applied and preference is given to the larger. We found these multipliers to be an accurate representation of the importance of text within the html markup tags. Normalizing TF: To normalize the weights across unigrams, bigrams and trigrams, another multiplier is applied. TF values for unigrams are multiplied by 1000 while TF values for bigrams are multiplied by 3. These two multipliers were found by trial and error using the test set and did the best to balance out the scores. These multipliers result in unigrams and bigrams being normalized to the trigrams TF values. Subsuming Ngrams: Any unigram that appears fully within a bigram or trigram is subsumed by that trigram. Bigrams, however, are considered subsumed by a trigram if the entire bigram appears within a trigram or either token of the bigram appears anywhere in the trigram. The subsuming ngram s weights are adjusted as follows: 1/10 th of the subsumed bigram s TF value is added to the subsuming trigram s TF value 1/20 th of the subsumed unigram s TF value is added to the trigram s TF value. 1/10 th of the subsumed unigram s TF value is added to the subsuming bigram s TF value Although the TF values are modified, no ngrams are removed from the lists at this time. In our first implementation we only considered bigrams being subsumed by a trigram if the entire bigram appeared in one trigram. In testing that design, we noticed our results had too much overlap, which limited the diversity of information our word cloud provided. Our current implementation, considering bigrams with one token appearing in a trigram as subsumed, results in a broader representation of the document. Combining Ngrams: Our final word cloud is comprised of 15 ngrams with the approximate distribution of 47% unigrams (7), 33% bigrams (5) and 20% trigrams (3).

4 The top three trigrams are identified first. The process selects the three trigrams with the highest TF values. These three trigrams are compared against each other to check for overlap. If any two of these trigrams share an exact token, the trigram with the lower TF score is removed from the list and half of its TF value is added to the trigram which contains the same token. The trigram with the next highest TF value from the complete list of trigrams is then added to the remaining top two trigrams for consideration and the process is repeated. The resulting list contains the top three trigrams based on TF scores that do not have any matching tokens. The entire list of bigrams is then iterated over, removing any bigram that is subsumed by one of the top trigrams (following the subsuming rules described in the section above). The top five bigrams are then identified using the same technique as used in identifying the top trigrams. Again, the resulting bigram list contains five bigrams based on TF scores which each have unique tokens. The entire list of unigrams is then iterated over, removing any unigram subsumed by either a top trigram or top bigram. The remaining top seven unigrams, based on TF score, are selected. The resulting 15 ngrams are sorted from maximum TF value to minimum TF value. This sorted list represents our final list of ngram terms and associated weights. Our first implementation did not consider repeat tokens within ngrams of the same size. As discussed in the subsuming ngrams section, this resulted in some cases where all three trigrams contained the same word. Since this limited the amount of information our word cloud would convey, we imposed the rules described above in our current implementation. The combination of not allowing repeat words within ngrams of the same size and removing smaller order ngrams which have one or more tokens in a larger order ngram ensures our final word cloud will be a better representation of the webpage content. Final Output: The final list of ngram terms and weights identified by the ngram extraction module is returned to the wrapper function, which constructs an appropriatelyformatted return string from this list. This return string is then delivered to the evaluation platform as an HTTP response via the REST interface, and is subsequently rendered as a term cloud by the Google TermCloud Visualization API. Sample Output: (using three example webpages) 1) clouds centering tags image hosting 58.4 tag cloud edit social software 58.3 used display non-tag blog aggregator 58.0 visual appearance 63.0 data 37.4 coupland microserfs 58.9 type 29.1 word 27.7 size 20.8 search 19.4 flickr 13.8 collocate 12.4

5 Boosted by HTML tags: [clouds, tag, cloud, edit, visual, appearance, coupland, microserfs, image, hosting, social, software, blog, aggregator, data, word, search, flickr, collocate] Subsumed ngrams: [tag cloud, clouds centering, centering tags, tag, cloud, clouds, centering, edit, appearance, visual, history, coupland, hosting, image, microserfs, blog aggregator, social, software] Overlapping ngrams: [tag clouds centering] 2) campaign launched university annualgiving udel edu faculty 90.3 staff encouraged participate 78.2 exam schedule 76.6 located online 76.0 diamonds society 65.0 library final 50.6 delaware 33.4 udid employee 30.0 year 23.4 gift 16.7 make 13.3 means 13.3 programs 13.3 Boosted by HTML tags: [university, faculty, staff, encouraged, participate, annualgiving, campaign, launched, delaware, diamonds, society, udel, edu] Subsumed ngrams: [encouraged participate, annualgiving udel, launched university, campaign launched, udel edu, annual giving] Overlapping ngrams: [staff campaign launched, faculty staff campaign, annual giving campaign, annualgiving udel, launched university faculty, university faculty staff, annual giving, staff campaign] 3) thailand believes trucks popular food truck maze infighting feel los realization street-food culture grilled cheese 78.0 becoming mainstream 76.5 hits kogi 65.1 scene 25.0 politics 15.0 business 8.3 choi 8.3 hiller 8.3 city 6.6 hot 6.6 Boosted by HTML tags: [trucks, food, truck, feel, los, maze, infighting, becoming, mainstream, grilled, cheese, hits, kogi, scene, culture, politics] Subsumed ngrams: [food truck, believes trucks, thailand believes, popular food, truck, food, trucks, believe, believes, popular, thailand, kogi, becoming, cheese, grilled, infighting, mainstream, maze, feel, fighting, grill, hits, los] Overlapping ngrams: [debate flashy trucks, launching trucks scene, food trucks scene, roadstoves launching trucks, trucks also placed, new-wave food trucks, flashy trucks generated, circus food trucks, food trucks also, generated food trucks, food trucks generated, trucks generated food, trucks la two, angeles food truck, food truck scene, food truck culture, food truck cultures, trucks scene infighting politics, mainstream maze]