Market Intelligence and Search Results

Transcription

1 Using Internet Sources For Market Intelligence Presented at NABE Big Data At Work Conference Northern Light

2 The task Firms must make strategy decisions in order to survive and prosper What products to offer, with what features, in which market segments? What technologies to bet on? How to get market share, what matters to customer? What are my competitors doing, how are they competing These decisions have to incorporate intelligence about Customers Product markets, geographic markets Technologies Competitors In many industries, such as IT, telecom, pharmaceuticals, and financial services, the rate of creative destruction is so high failure at this task puts the firm in mortal danger 2

3 Practically speaking, the growth of the web has destroyed other sources of information, pretty much about everything Year Number of Websites Market Value of the Boston Globe $1.400 billion ,410, ,912, ,338, ,985,183 $ 70 million 11:01 p.m. last Thursday 955,663,263 Source for website counts: internetlivestats.com 3

4 Problems with using the web for market intelligence The goal of the web search company : make all information available to everyone in one instant search without any prior knowledge of the searcher s purpose in doing the search or any concept of what content the user would consider of high quality for the intended purpose But there are some problems with this The web has a low signal to noise ratio Web search results are disconnected little pieces of information There are few big data (big content) tools available for analyzing web content for the purposes of market intelligence 4

5 Noise 956 million websites Most are Pornography (around 40% of the total) Abandoned Link farms for the purpose of spamming Google search results Company/product brochures e-commerce sites Local small business, sports, crime, human interest, entertainment, consumer information, recreation sites News sites that republish press releases and use automated means to republish other websites commentary These websites offer few insights useful for market intelligence and business strategy 5

6 How many of the 956 million websites out there are actually useful for gathering market intelligence? If we use the criteria of web sites that provide original, high quality journalism with analysis, commentary, and perspective on business and technology By Northern Light s estimate, there are 6,300 useful websites for market intelligence 20 million webpages (of the total 50 trillion webpages on the web) Yep, that is all Still, a lot content (big content if you will) Most of these are industry news sites (what we used to call trade journals ) Or industry-authority bloggers 6

7 Search results are disconnected little pieces of information

8 Tools provided by Google and Microsoft for analyzing large sets of web search results This page intentionally left blank. 8

9 What tools should be provided to analyze website collections for market intelligence? Taxonomies for analyzing the content with business and technology concepts Developed by domain experts (e.g., market intelligence professionals in the relevant industry) Automatic relationship identification Trend analysis Unlike most big data environments, search is a run-time process Analytics must support interactive insight discovery by business people, not batch process studies run by data scientists Everything, especially concept tags and locations has to be in the search index Faceted concept-based navigation of the search results Recursive exploration of the content based on new knowledge developed during the search process 9

10 So What would a search engine that searches the 6,300 sites And had tools to analyze the search result Do that the web search engines don t do today?

11 Demo of Millie, your market intelligence analyst Millie.northernlight.com Free and open to the public Use the industry search options and try out the text analytics Give it a drive after the conference and let us know what you think 11

12 Summary What search engines do Generate search results that are enormous lists of documents Leave it to the user to sort through the list, examining a few of the hits What search engines should do Answer these questions What does this search result mean to my research question? What are the concepts that are identified, what are the relationships? If I could read all the documents on the search result, what would I know that I don t know now and might not think to search on? 12

13 The future of web search for professional purposes Beyond web content aggregation, indexing, and search, which are all well understood by now Search technology providers must abandon the notion that a list of documents is an appropriate response to a search query Search engines must evolve to have an in-depth understanding of the searched material and the associated ways of knowing in the knowledge domain of the searcher And they must use this understanding to provide the automated analysis and extraction of meaning from the research repository

14 Contact information C. David Seuss CEO Northern Light Northern Light provides strategic research portals and content sets serving Market intelligence Market research Competitive intelligence Customer Insights Strategic planning Product management Sales Executive business leaders 14