Criteria for a Successful Sale: How to match Sales leads with Delivery Experts

Transcription

1 POMS Criteria for a Successful Sale: How to match Sales leads with Delivery Experts Christine Robson IBM Almaden Services Research 650 Harry Road San Jose CA crobson@us.ibm.com Julia Grace IBM Almaden Services Research 650 Harry Road San Jose CA jhgrace@us.ibm.com POMS 19th Annual Conference La Jolla, California, U.S.A. May 9 to May 12,

2 Abstract In large sales engagements, a sales lead is often paired with a "delivery expert" from within their corporation to help manage and close a sale. This matching process mirrors online dating in that an ill-fitted match is often doomed to failure but much more is at stake in the context of corporate matching since millions of dollars in sales revenue might be on the line. In order to improve the success of sales, we developed a recommendation system to provide this matching. Our analysis began with a comprehensive study and overview of current recommendation algorithms from other fields such as online marketing, like Amazon.com. To investigate what attributes are most important when formulating a successful match, we conducted a user study, selecting delivery experts from four sales regions throughout North and South America. We examined the sales conducted by these experts to find any possible correlation of experience attributes to successful sales. A statistical analysis uncovers some of the parameters which affect the success of the sale, and the satisfaction of the sales lead with the expert's service. From our analysis, we suggest a method to increase sales revenue by considering specific features of the expert's experience features we have identified as key factors in an ideal match. Our analysis uncovers the fact that personal relationships are the foremost factors in the success of sales engagements, far exceeding experience with products or product lines. Team experience between sellers and experts, and personal experience with the customer are the most important factors in our selection algorithm. The final implementation of our system employs each feature with different weights according to success correlation. To validate the algorithm, we ran a pilot study using the system, which clearly demonstrated the effectiveness of our approach. 2

3 1 Introduction Large corporations are often pressed to improve efficiency by reducing time spent in personnel assignment, all the while building more effective teams. This problem of matching the right person with the right job is crucial, and can lead to the success or failure of a sale or business engagement. Personnel assignments usually employ heuristics and depend upon individuals to accurately self asses their skills and abilities. This approach is lacking in two regards: limited features are considered, and there is no feedback of success into the assignment system. We improve upon the feature limitation by measuring the skills and capabilities of personnel in terms of their experience, data which is captured by searchable datasets in increasingly accurate and granulated formats. One challenge of using such techniques is in identifying and quantifying these metrics. We begin with a circumstance that has measurable metrics: sales leads who request help from experts within the corporation to close a deal. The measurable metrics are percent of deals successfully signed and satisfaction of the sales lead with the service the expert has provided. This data, as well as the feature data, was gathered during the course of a pilot study covering 4 sales regions, and involving 61 experts and 1186 sales. The purpose of the pilot study was to provide a profile of the expert s experience to the personnel who handles the matching process. In preparation for the study, a comprehensive set of heuristics was established based on best practices and experience of the assignment personnel from all global regions. These features form the basis for our data gathering and analysis. During the course of the study, these parameters were gathered or validated by users, in a process involving weekly updates, data summaries, and surveys. 3

4 An analysis of these features and the metrics for success is conducted by applying techniques from statistical learning theory. Regression and correlation are the primary methods employed, with some data being clustered by coarse parameters before regression. Having established metrics for success, we suggest a matching methodology to maximize the successfulness of sales. In the process we can immediately identify the features which are most conducive to sale success, and uncover relationships between these features and different success metrics which give insight into what makes sales successful. 2 Background Personnel matching systems have been investigated by economists, human resource specialists, psychologists, and other researchers. There are many ways to assemble people, including matching employees with managers and mentors, and matching employees with each other to form successful teams. (Tichy '81; Sure 00; Naveh '07) We investigate means to automate this process in the case of matching sales leads and delivery experts, with the goal of increasing the number of successful sales. Having attempted to solve this problem using many of the existing approaches to personnel management, we turn to a more fundamental investigation. Our goal is to understand what makes a successful match. As such, we first turn to areas outside of human resource management, to investigate the general algorithms and approaches to matching. We begin by surveying matching approaches in product and dating recommendation systems for a new perspective on how to approach this problem. 4

5 2.1 Recommendation Systems Recommendation systems are most frequently used to provide suggestions that are likely to be inline with the previous tastes of the user. If the data behind these systems is very small, then it may be that no recommendations are necessary because the user can explore all possible options. More generally speaking, recommendation systems are almost always employed when the data behind them is so large that it is infeasible for the user to sort through the possible candidates for selection. The most obvious manifestation of this problem is online stores, where the catalog might include millions of different brands and items to purchase. However, one can imagine a system where there are very few options to choose from, but it might be very helpful for the user to employ a recommendation system if s/he cannot quickly or obviously differentiate between the possible choices. This characterizes our situation; there might be a finite number of possible expert matches for a sales lead, all with very similar backgrounds and experiences. Thus, our method must be applicable to scenarios when the user doesn t always know who or what is best, even if the dataset is small. 2.2 Collaborative Filtering Collaborative filtering is the most common implementation of recommendation systems. Coined in 1992 by Goldberg et al., collaborative filtering is a means of providing recommendations based on the feedback and opinions of other users. This method was developed after noticing the volume of messages exchanged in electronic newsgroups was reaching a point where it was no longer feasible for subscribed users to read all messages posted. To address this, they built the Tapestry system and employed 5

6 collaborative filtering as a means of searching and flagging messages that might be of particular interest to the reader. (Goldberg 92) For collaborative filtering to be effective, and thus provide good recommendations, it requires a very large and complex dataset (Ziegler 05). This is intuitive in that when reading movie or restaurant reviews, people often put more confidence in a rating if it is based on a large number of reviews. Hence the success of collaborative filtering on the product offerings at companies such as Amazon.com and Tivo, which have thousands of users and perhaps millions of products or television shows. (Linden '03, Ali '04) 2.3 Feedback For techniques such as collaborative filtering to be successful, not only does the dataset need to be sufficiently large, but users must provide feedback on the available items as well. However, soliciting feedback is a documented problem for all recommendation systems. This is a particularly important problem because the quality of recommendations is based upon the feedback of the users; very little or no feedback will result in low quality or inapplicable recommendations. So, it should come as no surprise that some recommendation systems prefer to gather feedback implicitly to avoid requiring the user to perform any additional tasks. For example, if a user purchases an item on Amazon.com, then s/he has effectively endorsed or recommended that item. Tivo also employs collaborative filtering and has an explicit ratings function that requires the user to rate shows outside of the viewing and recording behavior. However, Tivo has also looked into gathering implicit ratings such as at the act of recording a show, or scheduling the recording of an entire season of a show. (Linden '03, Ali '04) 6

7 Our approach to matching must fundamentally differ from feedback based systems because of the time constraints of sales leads and delivery experts. In an environment where time is literally money, our automatic matching system must not depend on manual feedback. Thus, our data gathering is entirely passive, based on observing user experience as documented by the work history of the delivery experts and sales leads. This approach has the benefit of notable advantages. Since we gather all data implicitly, we are not dependent on the changing, and sometimes finicky opinions of our users a problem which plagues product recommendation systems. Another advantage is one of data quality. Self assessment is a difficult task, and most people do not accurately rate their skill levels. We conclude that our system cannot rely on delivery experts or sales leads to rate their own abilities. Rather, we will rely on records of previous work experience to demonstrate skills in particular areas. (Krueger 99; Tversky 04) 2.4 Dating Services Dating services also employ recommendation systems, matching people with people, instead of products. In general, most dating services ask users to complete a comprehensive survey, forming a mathematical model of the individual based on the results. One example is Eharmony.com, a popular dating site that trains a neural network based on surveys taken after each date. The network employs artificial intelligence techniques to continually refine the potential matches for the user no collaboration or comparison between users is employed. Users are asked their opinion of 7

8 potential matches, but this methodology differs from collaborative filtering in that the opinions are never compared with others. (Buckwalter 04) The online dating model is perhaps the most similar to that of personnel assignment. Observing the success of these methods, our approach will also employ a correlation matrix of matching attributes. We observe that in the case of online dating, the goal is to improve a users degree of satisfaction with their potential mate a potentially nebulous and subjective measure. While we conduct a similar satisfaction survey with sales leads, the primary criteria for success will be the outcome of the sale. There have been attempts to apply collaborative filtering to online dating; Brozovsky et al hypothesized that better potential matches could be discovered if similar users could be identified. Their system recommends highly rated dating profiles of similar users to each other. For this approach to succeed, users are required at registration to rate a hundred profiles of potential matches, so that their ratings can be used in a collaborative filtering approach. Although our system must refrain from soliciting explicit feedback, Brozovsky s work opens the door for a possible integration of collaborative filtering to our system in the future. (Brozovsky 07) 2.5 Motivation All recommendation systems mentioned thus far are based on what people like to watch, engage in or spend money on. Our goal, on the other hand, is to make matches where the end result (the pairing of sales leads to delivery experts) results in financial gain for the company. Collaborative filtering unfortunately proves impractical for our applications, due to limitations of data size and complexity, and a business requirement to avoid feedback-based systems. 8

9 As such, we begin our data investigation by following in the footsteps of dating services, with a correlation matrix to uncover to uncover the criteria for a successful sale. Our goal is to find what attributes matter the most when matching sales leads with delivery experts, using records from previous sales attempts, both successful and unsuccessful. 3 Data Our analysis begins with the sales records database, containing 23 million data points. As part of our study, we run queries to extract the following sales records: 9641 sales records in which one the 61 experts in our study was called upon to help, including 1186 sales records from 2007 where the outcome has been recorded, and a total of 1081 survey results from Sales leads polled as to the helpfulness of the expert. 3.1 Data Mining and Record Matching Using data mining and NLP techniques, we parse each sales record for the following characteristics: product, customer, and industry. Product and industry are mapped into a coarse feature set, with 28 product lines and 16 industries, whereas customer is left as a fine-grained field. Then, using a chronological analysis of each sales record, we annotate each of the records for 2007 with the following fields: how many sales attempts to-date the expert has assisted with, and four binary fields indicating if the expert worked a sale with this product line / customer / industry / sales lead before. 3.2 Graphical Data Model This dataset lends itself directly to an application of statistical learning theory. Specifically, we can consider each data record as consisting of observed data pertaining to how good a job the expert does. This observed data can be used to estimate the 9

10 likelihood that the expert will do a good job for any given set of observed parameters. This estimation relies on a graphical data model which documents the relationship between the observed variables and the unknown we are estimating. (Jordan 03) Our graphical data model considers two possible measures of success: the outcome of the sale attempt, and the results of the satisfaction survey. We model these observed data points as dependant on how good a job the expert does. In addition, we express as parameters the type of experience the delivery expert has with each particular sales opportunity. These observed data points are modeled as influencing how good a job the expert does. This provides the following complete graphical model: Expert: types of experience General Product Line z 1 z 1 z 2 z 3 z 4 Customer z 2 Industry z 3 Work history z 4 with Sales lead X How good a job the expert does Sale made or failed Sales lead s Satisfaction X W S W S Figure 1: Graphical Model of the Data From the graphical data model, we build the statistical model. To begin, we condition on the set of all observed variables, D: D={ ( (1),z (1) 1,z (1) 2,z (1) 3,z (1) 4,W (1),S (1) ),..., ( (N),z (N) 1,z (N) 2,z (N) 3,z (N) 4,W (N),S (N) ) } 10

11 which allows us to marginalize over the unobserved node X. We do this separately for each of the observed metrics, W, and S, allowing us to calculate correlations and regressions directly between the parameters and metrics. Some brief algebraic manipulations provide the following marginal probabilities of the observed metrics: P( W n X) = P( W n (n),z (n) 1,z (n) 2,z (n) 3,z (n) 4 ) P( S n X ) = P( S n (n),z (n) 1,z (n) 2,z (n) 3,z (n) 4 ) These probabilities allow us to quantify the correlations of the types of experience on the metrics for success. 4 Analysis We begin with a single observation from the larger dataset containing of 23million sales records. While the overall success rate of sales sits around 32%, success rate of sales when an expert is involved is 44.9%. Our analysis then takes two approaches, the first observing general experience of the expert (i.e. sale attempts the expert has been involved in), and the second investigating specific types of experience. 4.1 Success Rate vs. Experience In the Figure 2, we observe the experts sales history from old records (their first 40 sales, or for new experts with less then 40 sales, all sales to date). This regression plots the average success rate of experts at different levels of experience. The learning curve n ) is expressed as W n n + n : 1 Success Rate X n n Note that we abuse statistical notation slightly to use W n as both the success of a single sale, and the overall successfulness of all the expert s sales. 11

12 Sales Rates vs. Expert Experience Success Rate (percent of sales made at this experience level) 100% 80% 60% 40% 20% 0% Expert Experience (number of sale attempts) Figure 2: Sales Experience The probability that an expert will achieve a particular success rate at their n, can then be expressed using the generalized linear model for regression: P(W n n 2 ) -1/2 exp{- -2 )(W n - n ) 2 } P(W n n )= 9.74 * exp{ * (W n * n-1 ) 2 } 2 is the variance of n, The success rate is further differentiated by country; for example experts in North America have a s expert ). Looking closer at South America, we cluster the experts into two groups based on success rate and time in company. Taking a generative approach, we calculate the class-conditional density P(W n n ). The groups correspond to low performers and high 12

13 North America Success Rate (percent of sales made) 100% 75% 50% 25% 0% Experience (number of sales attempted) Success Rate (percent of sales made) 100% 75% 50% 25% 0% South America (Clustered by low and hi-preformers) Experience (number of sales attempted) Figure 3: Plot and Regression of Expert Experience vs. Success Rate 13

14 4.2 Specific Experience Criteria The second analysis concerns the specific experience of the experts participating in our study. Again using sales history, we identify every sale in which an expert has personal experience with a similar sale. The criteria for similarity are laid out in figure 1, and this data is then correlated with both sale success rates and sales lead satisfaction, as shown in Table 1. correlation: Sale Expert has experience with: Made or Failed Industry Customer Sales lead Product Line Sales lead is Satisfied 9.64% 7.22% 14.66% 11.07% -4.99% Product Line 4.11% 17.51% 23.36% 13.33% Sales Lead 27.49% 15.95% 50.16% Expert has experience with: Customer 25.10% 31.12% Industry 14.10% Table 1: Correlations between various features of sales data The metrics we have established, sales lead satisfaction and sale made or failed, are outlined in the table. The parameters which have a significant correlation with these metrics are similar to the expectations of the users in the pre-study survey. Personal experience, both between the sales lead and the expert, and between the expert and the customer, are the most important attributes. Interestingly, Sales lead satisfaction does not have a very strong correlation with the success of the sale. Clustering the sales by sales lead satisfaction, we can see one outlier: in cases where the sales lead is dissatisfied with service (an overall rating of 1 or 2), the success rate of sales is appreciably lower (23.8% vs. 44.9%). 14

15 4.3 Further analyses Several additional analyses were conducted which did not result in any useful correlations or clusterings. Briefly, the parameters investigated include: specific product (rather then product line), nationality (beyond regional deviations), total dollar value of the sale, amount of time spent on the sale, type of help requested by sales lead, complexity of the sale as reported by sales lead. It is interesting that none of these features proved useful for clustering and correlation. The users of the study were particularly surprised that clustering by nationality did not confirm any of their preconceptions. 5 Conclusions Profit is the primary goal of any large corporation, and as such the goal of assigning experts to assist sales leads should be to maximize the overall income from sales. To further this goal, we should in general assign more experienced experts to sales with a higher dollar value. In addition, we can prioritize the following specific experience criteria for experts: (1) personal experience with sales lead, (2) personal experience with customer, (3) experience within the industry Interestingly, experience with the product line is not a key criterion for success. This contradicts the expectations of users in pre-study surveys, and raises questions about sales models. Satisfaction survey results have little correlation with any measured parameter of the sale. Anecdotal evidence from survey responses suggests one reason: Sales leads often cite responsiveness as their primary concern with experts, and rarely blame the expert for a failed sale. 15

16 The learning curve of experts is something which may have direct impact on personnel decisions, and is an area that warrants further investigation. The average expert improves their successful sales rate by about 0.75% per sales attempt, or roughly 21% per year. However, some regions have clear deviations between high performers who have high success rates and high growth rates, and low performers who stagnate at a relatively low success rate. 6 Summary and Future Work The problem of matching teams of people to optimize sales revenue is an area of increasing research. Our work shows an effective approach to matching sales leads and delivery experts to increase the chances of a successful sale. We have demonstrated that personal relationships and experience between team members and potential customers is the primary means to success. The two immediate directions for this research are clear: apply what has been learned, and expand the research to encompass more data and additional use cases. Following on our successful pilot study, this methodology is now being employed to assist in sales. By pairing with workforce optimization technologies (eg. Naveh 07), we foresee a concrete impact on sales revenue. We are also exploring ways to combine additional metrics such as available workload to provide a more comprehensive model of success. The statistical methods we have employed are not limited in application to sales forces or maximizing revenue. The approach fundamentally relies on confirmed observations in the form of data records. This provides the critical advantage of 16

17 protecting the decision process from the inherent inaccuracies in self-reported data. This rigorous data-driven statistical approach affords an opportunity to understand the criteria for a successful match without concern of inflated or libelous data points. We believe application of this approach to retail and dating services would be both enlightening and fruitful. Similarly, there remain techniques from the area of dating services which might be combined effectively with our approach. One idea would be to employ the algorithm formulated by Borzovsky et al. (Borosovsky 07) to match delivery experts to sales leads based on maximizing seller satisfaction. Sellers would initially rank the delivery experts they have worked with. We would then find sellers who are very similar and recommend those highly rated delivery experts to them. This would be a particular advantage because we would not need additional information from the sellers; similarity would be solely based on ranking delivery experts. However, this would require more proactive gathering of seller satisfaction feedback that is currently available. These areas are excellent directions for future research, and we hope that future personnel systems will continue to consider cross disciplinary approaches. In addition, the application of rigorous statistical methods to recommendation systems is clearly a successful approach, and we hope to see this type of mathematics in other recommendation system areas, including sales and dating services. 17

18 References Ali, K.; and van Stam, W. TiVo: Making show recommendations using a distributed collaborative filtering architecture. In Proceedings of the 2004 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA, ACM Press, pp Brozovsky, Lukas; and Vaclav Petricek. Recommender System for Online Dating Service. In Proceedings of Znalosti 2007 Conference, Ostrava, Czech Republic. 9 Mar Buckwalter, J. Galen; Steven R. Carter; Gregory T. Forgatch; Thomas D. Parsons; and Neil Clark Warren. Method and system for identifying people who are likely to have a successful relationship. US Patent Number Assignee: Eharmony.com. Issued May 11, Goldberg, D., Nichols, D., Oki, B.M. and Terry, D. Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM, 35, 12 pp Jordan, Michael I. An Introduction to Probabilistic Graphical Models. University of California, Berkeley. UC Berkeley, Unpublished Manuscript; edition June 30, Kruger, Justin; and David Dunning. Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments. Journal of Personality and Social Psychology. Vol. 77, No Linden, Greg; Brent Smith; and Jeremy York, Amazon.com Recommendations: Item-to- Item Collaborative Filtering, IEEE Internet Computing, vol. 7 num. 1, p January Naveh, Y.; Richter, Y.; Altshuler, Y.; Gresh, D.L.; and Connors, D.P.. Workforce optimization: Identification and assignment of professional workers using constraint 18

19 programming, IBM Journal of Research and Development, Business Optimization. Volume 51, Number 3/ Sure, Y.; A. Maedche; and S. Staab: Leveraging Corporate Skill Knowledge From ProPer to OntoProPer. Proceedings of the Third International Conference on Practical Aspects of Knowledge Management (PAKM 2000), Basel, Switzerland October Tichy, Noel M.; Other Contributor(s): Fombrun, Charles J.; Devanna, Mary Anne. Strategic human resource management. Ross School of Business - Working Papers Series. University of Michigan. Graduate School of Business Administration. Division of Research. No Tversky, A.; and Kahneman, D., Judgment under Uncertainty: Heuristics and Biases. Published in Preference, Belief, and Similarity: Selected Writings. Bradford Book Publishers, Ziegler, C.; McNee, S. M.; Konstan, J. A.; and Lausen, G Improving recommendation lists through topic diversification. In Proceedings of the 14th international Conference on World Wide Web. WWW '05. ACM, New York, NY, p Chiba, Japan. May 10-14,