What a CIO Needs to Know About Enterprise Search

Size: px

Start display at page:

Download "What a CIO Needs to Know About Enterprise Search"

Naomi Lester
8 years ago
Views:

1 What a CIO Needs to Know About Enterprise Search White Paper Version 2.1 January Search Technologies

2 Introduction Search engines are omnipresent. Every business application or Web site that contains a significant amount of content will have one. From embedded search engines to sophisticated, enterprise-scale platforms, there are a number of established principles that can guide organizations to use search effectively. According to Gartner Analyst Darin Stewart, Enterprise Search can bring Big Data within reach, so understanding the foundations for enterprise search success can help with Big Data initiatives too, especially if unstructured content is involved. This brief white paper presents five key issues that affect the success of search applications. #1 Data Growth Impacts Findability Information is proportionally harder to find as the data set grows. Just because a search engine is capable of indexing tens of millions of documents, that doesn t mean that users will find what they need. Back in the 1990s, many large companies were implementing a new groupware tool, called Lotus Notes. It had an embedded search engine, and the overwhelming majority of initial implementations used the embedded search, which was generally seen as functional and competent. Four years later, the majority of Lotus Notes systems were using an add-on search product. What changed? As more data came to reside in Lotus Notes, it became more challenging to find anything. User satisfaction declined, and pressure built to implement more comprehensive search solutions. The PowerPoint slide shows how one of the add-on vendors depicted the issue at the time. The observation that users only browse a very small percentage of the data set remains true, and as data sets grow, that percentage continues to shrink. Achieving search excellence becomes harder as the data set grows, and a plan is needed to maintain user productivity Search Technologies 2

3 #2 Search Relevancy is Subjective Relevancy is in the eye of the beholder Providing great relevancy requires more than just a good algorithm Data quality and consistency affects relevancy If your search application supports an important business process, it will benefit from relevancy tuning relevancy to suit the specific circumstances Use automated personalization of search results thoughtfully, it is not a universal solution for relevancy. The need for tuning Today, all leading search vendors provide excellent functionality for relevance ranking and the ability to customize it to suit the circumstances. For example, compare a patent search application with a financial trading support system. A 15 year old patent may be just as relevant as a 15-day old patent to a lawyer doing research for a client. But a city trader may view anything more than a few days old as unimportant. The same relevancy setup is unlikely to suit both scenarios. Automated personalization Within some search applications, results personalization technology can be very useful, but it is not a short cut to universal relevancy. Applications in which this approach works well include: E-commerce - people who bought this, also bought that Customer support portals, promoting the documents that, according to embedded analytics, were liked by others who searched for the same thing However, automated personalization is based on past behavior, and so assumes a constancy of information need. This can annoy the typical knowledge worker as much as it helps them. An expert will typically wish to study multiple aspects of a subject, and may resent being pointed back to items they have already read. Unless automated personalization is obviously going to work for an application, then it probably won t Search Technologies 3

4 #3 Search Navigators are Important Search navigators are a well-established paradigm. They provide a generally applicable solution to the data growth vs. findability issue Depending on the vendor, search navigators can be called; dynamic navigators, faceted search, dynamic navigation, or guided navigation. Data set reduction Search navigators reduce the data set from which results are being served, one easy click at a time. Search is easier if the data set is smaller. Our white paper Making the Most of Search Navigators discusses this subject in further detail. Self-personalization Thanks to Amazon, Ebay and others, users are familiar with search navigators. They understand what will happen if you click the link with the little number in brackets next to it. This is the general solution to personalizing search results. It is a self-service solution where the user is in control. Contextual Navigation The most useful search navigators are contextual to the user s task, and that usually requires the automated generation of new metadata. The importance of accuracy It is important that search navigators are accurate, otherwise users won t trust them. That s a metadata quality issue. #4 Don t Neglect Data Quality Data quality impacts all aspects of search, and especially relevancy ranking and search navigator accuracy. The more heterogeneous your data is, the more attention you will need to pay to content preparation to ensure that it plays well together behind a single search box. Almost all of the important functions in a modern search engine rely on good data structure Search Technologies 4

5 Data quality affects search engine accuracy. In other words, poor data causes problems. Dirty data Dirty data causes false positives in search results, which annoy users. Issues include unwanted headers and footers on documents, menu structures that should not have been indexed, unhelpful comments associated with a document, Etc. Wrong granularity We recently worked with a customer whose entire intellectual property was contained in 50 enormous documents. Their users needed a system that indexed the content in sensibly searchable chunks. In this case, there were 70,000 chunks in the deployed application. The documents were split into logical chapters prior to indexing. It can also be beneficial to join documents together into searchable units. For example, gathering all known data about an individual s skills and project experience into a single item for search purposes. Metadata quality In most organizations, metadata is incomplete and inconsistent, so it is necessary to auto-generate new metadata to support search applications. Don t assume that technology alone will provide good quality with the metadata. Every organization has a unique combination of content and vocabulary, so processes and people are needed to support the technology. Our Web site explores this subject in detail, see A Document Preparation Methodology for Search. #5 Search Applications Need Maintenance The content landscape is continually evolving, so effort is required to maintain search system relevancy and utility. It doesn t have to be hard work to maintain search quality, but you need processes and a plan. Implementing search will never be a one-time exercise, because both data sets and user needs continually change. For example, the addition of a large new content repository can adversely affect the user experience by burying other content Search Technologies 5

6 The good news is that established best practices exist for maintaining search applications, and if you don t have the skills in-house, we can help. Conclusion Remember these five key points, and you ll be well on your way to owning an enterprise search system that fully supports business productivity, and is highly valued by your staff: Continual data growth is a challenge to search system accuracy. Best practices exist, apply them consistently. Relevancy should be tuned to suit different applications and promote user satisfaction. Search navigators are important, make sure they are contextual to the application, and accurate, so that users learn to trust them. Rubbish in, rubbish out applies fully to search systems, so take data quality seriously, and don t expect technology alone to tidy up for you. Search engines need to be maintained because the content environment is continually evolving. About Search Technologies Search Technologies is the world s largest independent provider of search engine expertise, consulting and implementation services. We serve more than 500 corporate and government customers, and maintain expertise in a range of leading search products including Microsoft SharePoint, the Google Search Appliance, Autonomy, Attivio, Coveo, LucidWorks, and open source alternatives such as Elasticsearch and Solr Lucene Search Technologies 6

xrm Framework and Microsoft SharePoint

MICROSOFT DYNAMICS CRM POSITIONING WHITE PAPER xrm Framework and Microsoft SharePoint Contents Executive Summary... 3 xrm and Microsoft Dynamics CRM... 4 xrm Framework... 5 xrm Manages Customer-Like Relationships...