From IP port numbers to ADSL customer segmentation

Transcription

1 From IP port numbers to ADSL customer segmentation F. Clérot France Télécom R&D

2 Overview ADSL customer segmentation: why? how? Technical approach and synopsis Data pre-processing The many faces of a Kohonen map A glance at the results

3 Overview ADSL customer segmentation: why? how? Technical approach and synopsis Data pre-processing The many faces of a Kohonen map A glance at the results

4 ADSL customer segmentation PURPOSE Fine-grain understanding of the customer behaviour Anticipate (at least detect) changes in behaviour patterns

5 ADSL customer segmentation WHY? Rapidly expanding market Strong competition Fast adoption of new applications (P2P, streaming) by some segments

6 ADSL customer segmentation HOW? Bandwidth? Rather easy Big users vs. small users classification Usefull for network dimensionning Limited use for market understanding

7 ADSL customer segmentation HOW? Usage? More difficult: define what a «usage» is If successfull, will lead to a better understanding of the market If successfull, may also be usefull for network dimensionning: usage determines bandwidth

8 ADSL customer segmentation DATA?

9 Capture : BAS IP backbone sonde Office DSLAM Observation point: ATM on STM1 Telephone LAN ATU-R Filter DSLAM ATM Telephone Filter ATU-R Home

10 Data : IP and TCP relevant header fields For our purposes : VP/VC (customer Id) Time granularity, 6 minute periods Volumes (upstream and downstream) per IP port About 800 customers, one month of data

11 Overview ADSL customer segmentation: why? how? Technical approach and synopsis Data pre-processing The many faces of a Kohonen map A glance at the results

12 Technical approach Segmentation of ADSL clients based on their activities: clustering with Kohonen maps Use minimum a priori knowledge Timescale: activity defined on a daily basis Representation: retain only the ports having the greatest volumes

13 Synopsis of the analysis Data pre-processing Volumes are too correlated to applications to be informative: order statistics Three steps for a double clustering First, cluster the «atomic» activities to define what the «typical activities» are Second, project customers on such «typical activities» Third, cluster the customers

14 Discover Typical Activities : LOG FILE Client1 Day1 RankVol11 Client1 Day2 RankVol12 Client2 Day1 RankVol21 Client2 Day3 RankVol23 Client2 Day5 RankVol25 Client3 Day2 RankVol32 Client4 Day4 RankVol44 Client4 Day5 RankVol45 Client5 Day6 RankVol56... Typical Day 2 RankVolume Space Typical Day 3 Clustering of the Days In the RankVolume space + Clustering of the IP ports Typical Day 1 Typical Day 4

15 Project customers on typical LOG FILE activities : Client1 Day1 RankVol11 Client1 Day2 RankVol12 Client2 Day1 RankVol21 Client2 Day3 RankVol23 Client2 Day5 RankVol25 Client3 Day2 RankVol32 Client4 Day4 RankVol44 Client4 Day5 RankVol45 Client5 Day6 RankVol56... Typical Day 2 RankVolume Space Typical Day 3... Client2 (1, 1, 0, 1)... Project the customers on the «typical days»: Customer Profiles Typical Day 1 Typical Day 4

16 Discover Typical Customers : Client1 CustomerProfile1 Client2 CustomerProfile2 Client3 CustomerProfile3 Typical Days Space Clustering of the Customers in the Typical Days Space

17 Overview ADSL customer segmentation: why? how? Technical approach and synopsis Data pre-processing The many faces of a Kohonen map A glance at the results

18 Data pre-processing : Raw volumes hide the relevant information: Typical volumes used by applications span many orders of magnitude : a big user will very likely consume less volume thana small P2P user! Simple-minded normalisation does not help much : Heavy-tailed distributions

19 Example of distribution

20 Data pre-processing : Use rank statistics to represent each day : Sort the volumes according to all the volumes for the same port in the log file Use the rank as a representation for the day on this port

21 Data pre-processing : Raw Volumes ClientId TimeStamp E- mail P2P Client 1 Day Client 1 Day Client 2 Day Client 2 Day Client 2 Day Client 3 Day Client 3 Day Client 4 Day Client 4 Day3 2 0 Client 4 Day Rank Volumes ClientId TimeStamp E- mail P2P Client 1 Day1 7 2 Client 1 Day2 9 6 Client 2 Day1 2 6 Client 2 Day2 7 1 Client 2 Day4 2 6 Client 3 Day3 1 4 Client 3 Day4 2 9 Client 4 Day2 9 3 Client 4 Day3 6 9 Client 4 Day4 5 4

22 Overview ADSL customer segmentation: why? how? Technical approach and synopsis Data pre-processing The many faces of a Kohonen map A glance at the results

23 The many faces of a Kohonen map: Kohonen map??? «Collaborative» K-means Collaboration is enforced by an underlying topology on the centers («neurons»): Neighbouring neurons «share» their patterns Neighbourhood shrinks as training proceeds

24 The many faces of a Kohonen map: Kohonen map??? Consequences Neurons are positionned in an ordered way on the underlying topology : data visualisation in reduced dimension Clustering of the data around the neurons Dimension reduction and clustering : simplification of exploratory data analysis

25 The many faces of a Kohonen map: some references Juha Vesanto. Som-based data visualization methods. Intelligent Data Analysis, 3(2): , Juha Vesanto and Esa Alhoniemi. Clustering of the self-organizing map. IEEE Transactions on Neural Networks, 11(3): , Juha Vesanto, Johan Himberg, Esa Alhoniemi, and Juha Parhankangas. Som toolbox for matlab 5. Report A57, Helsinki University of Technology, Neural Networks Research Centre, Espoo, Finland, April Vincent Lemaire, Fabrice Clérot. SOM-based-clustering for on-line fraud behaviour classification: a case study, Fuzzy Systems and Knowledge Discovery, Singapore, 2002

26 The many faces of a Kohonen map: map of the individuals Data : X(i, v) i=1 N ind v=1 N var Map of the individuals : W(p, v) p=1 N neurons v=1 N var N neurons << N ind

27 The many faces of a Kohonen map: map of the individuals «Top view» : projection of the individuals Population Clustering Projection of auxiliary data

28 The many faces of a Kohonen map:map of the individuals «Top view» : {N p (v)} p=1 Nneurons prototypes of the map, vectors in N var dimensions : Hard to understand

29 The many faces of a Kohonen map:map of the individuals «Side view» : projection of the variables {M v (p)} v=1 Nvar Projections on the map of the components of the neurons One map for each variable :

30 The many faces of a Kohonen map: map of the individuals «Side view» Each map is representative of the corresponding variable : this allows to study the correlation between variables

31

32

33 Too many maps for a visual inspection : Transform each map into a vector, representative of the variable Build a new Kohonen map : map of the variables

34 The many faces of a Kohonen map:map of the variables Data : projections of the variables Y(v, p) i=1 N var v=1 N neurons Map of the variables : M neurons << N var W(q, p) q=1 M neurons p=1 N neurons Populations Clustering

35 The many faces of a Kohonen map: map of the variables Clustering

36 The many faces of a Kohonen map: map of the variables Back to the map of the individuals: easier interpretation of the clusters of individuals by re-ordering the variables according to their cluster

37 The many faces of a Kohonen map: exploratory data analysis Double clustering : of the individuals of the variables L clusters K clusters Both clusterings are consistent together

38 Overview ADSL customer segmentation: why? how? Technical approach and synopsis Data pre-processing The many faces of a Kohonen map A glance at the results

39 Map of the «typical days»: «classical» activities http, mail, ftp Low activity MS streaming P2P but Kazaa Kazaa http-alt

40 Correlation between IP ports FTP Direct Connect

41 Profiles of typical days Profile Deviation from the mean profile Low activity days Classical applications + FTP days

42 Map of the customers: the big picture

43 «Classical» behaviours :

44 P2P but Kazaa :

45 MS-streaming :

46 Kazaa :

47 «Classical» heavy users :

48 Map of the clients : Classical, light Kazaa MS-streaming Classical, heavy P2P sauf Kazaa

49 5 Peu Actives P2P sauf Kazaa Kazaa 6 Peu Actives Kazaa P2P sauf Kazaa 11 Kazaa P2P sauf Kazaa Classiques 4 Peu Actives P2P sauf Kazaa Kazaa 13 Peu Actives Classiques 3 «Client moyen» 14 MS Streaming Kazaa Classiques 9 MS Streaming P2P sauf Kazaa Classiques 10 P2P sauf Kazaa Classiques Peu Actives 17 Classiques P2P Kazaa Kazaa sauf 1 Classiques Peu Actives 2 P2P sauf Kazaa Peu Actives 12 Kazaa Peu Actives 16 Classiques P2P sauf Kazaa Kazaa 7 Kazaa MS Streaming http-alt Peu Actives 8 http-alt Peu Actives 15 Classiques Peu Actives

50 Correlation with trafic volumes Reasonable fit with a? law : no heavy tails when the volumes are averaged per client? =2.3 : no singularity at the origin all in all, a «nice» distribution Average daily volume Small vs. Large volumes

51 Conclusion Complete analysis, from «atomic» activities to customer segmentation Kohonen maps as a priviledged tool for exploratory data analysis : robust and efficient clustering technique with a visualisation capability which allows to interpret and present the clustering results in a very natural way

52 Conclusion The analysis gives a consistent description of the customer behaviour Nothing really puzzling but turns folklore into figures

53 Conclusion What s next? Investigate the correlation between clusters and trafic (volumes, daily profiles) Investigate the evolution of the clusters in time