Innovative Data Mining based approaches for life course analysis

Transcription

1 IPUC, Neuchâtel, February 23-24, 2007 Innovative Data Mining based approaches for life course analysis Gilbert Ritschard Alexis Gabadinho, Nicolas Müller, Matthias Studer University of Geneva, Switzerland Outline 1 Aim of the research project 2 Our first results 2.1 Mobility trees 2.2 Survival trees 2.3 Characteristic sequences 3 Foreseen Developments IPUC07 toc intro mob surv seq conc 22/2/2007gr 1

2 1 Aim of the research project Just started February 1, 2007 FNS project on Mining event histories: Towards new insight on personal Swiss life courses Methodological concern Explore and develop data mining approaches for individual longitudinal data Methods for time to event analysis Methods for sequence data analysis Socio-demographic concern Using mainly SHP data, but also other sources, gain original insight on How familial, professional and other socio-demographic events are entwined, Typical characteristics of Swiss life trajectories, Changes in these characteristics over time. IPUC07 toc intro mob surv seq conc 22/2/2007gr 2

3 What is data mining? Data Mining is the process of finding new and potentially useful knowledge from data Gregory Piatetsky-Shapiro editor of Data mining is the analysis of (often large) observational data sets to find unsuspected relationships and to summarize the data in novel ways that are both understandable and useful to the data owner (Hand et al., 2001) Also called Knowledge Discovery in Databases, KDD. Origin: IJCAI Workshop, 1989, Piatetsky-Shapiro (1989) Textbooks : Han and Kamber (2001), Hand et al. (2001) IPUC07 toc intro mob surv seq conc 22/2/2007gr 3

4 What is data mining? (2) Concerned with characterization of interesting patterns per se (unsupervised learning) Clustering Frequent itemsets Association rules for classification or prediction purposes (supervised learning) Decision trees Bayesian networks SVM and Kernel Methods CBR (case based reasoning), K-NN (k nearest neighbors) Proceeds mainly heuristically. Unlike statistical modeling, makes no assumptions about process generating the data. IPUC07 toc intro mob surv seq conc 22/2/2007gr 4

5 Typology of methods for individual longitudinal data nature of data questions time stamped event state/event sequences descriptive - Survival curves: - Optimal matching clustering Parametric (Weibull, Gompertz) - Frequencies of typical and non parametric patterns (Kaplan-Meier, Nelson-Aalen) - Discovering typical patterns estimators causality - Hazard regression models - Markov models, Mobility trees - Survival trees - Association rules between subsequences IPUC07 toc intro mob surv seq conc 22/2/2007gr 5

6 2 Our first results Mobility trees Survival trees Characteristic sequences IPUC07 toc intro mob surv seq conc 22/2/2007gr 6

7 2.1 Mobility trees (SHP Data, Waves 1 to 6 ( ), aged between 20 and 64 in 2004.) How does working status (occupied active, unemployed, inactive) in 2004 depend on working status in previous year (1999 to 2003) other factors (attained education level, partner working status, partner education level,...) and what are main interaction effects? Mobility trees are alternative to Markovian transition models. Growing separate classification trees for women and men highlights gender differences. IPUC07 toc intro mob surv seq conc 22/2/2007gr 7

8 Mobility tree, Men Working status 04 Node 0 active occupied unemployed not in labor force Total (100.00) 1283 Working status B, 03 Adj. P-value=0.0000, Chi-square= , df=2 active, full time (>= 80%);active, long part time (50%-80%);active, short part time (< 50%) not in labour force unemployed,<missing> Node 1 active occupied unemployed not in labor force Total (84.57) 1085 Node 2 active occupied unemployed not in labor force Total (4.75) 61 Node 3 active occupied unemployed not in labor force Total (10.68) 137 Partner highest level of education achieved 04 (both grid and individual quest.) Adj. P-value=0.0001, Chi-square= , df=1 Partner actual occupation 04, into 6 Adj. P-value=0.0002, Chi-square= , df=1 <=vocational high school >vocational high school,<missing> at home;part-time paid w ork;full time paid w ork + family company;retired or invalid education,<missing> Node 4 active occupied unemployed not in labor force Total (55.42) 711 Node 5 active occupied unemployed not in labor force Total (29.15) 374 Node 6 active occupied unemployed not in labor force Total (4.68) 60 Node 7 active occupied unemployed not in labor force Total (6.00) 77 IPUC07 toc intro mob surv seq conc 22/2/2007gr 8

9 Mobility tree, Women Working status 04 Node 0 active occupied unemployed not in labor force Total (100.00) 1647 Working status B, 03 Adj. P-value=0.0000, Chi-square= , df=3 not in labour force active, full time (>= 80%);active, long part time (50%-80%) active, short part time (< 50%) unemployed,<missing> Node 1 Node 2 Node 3 Node 4 active occupied active occupied active occupied active occupied unemployed unemployed unemployed unemployed not in labor force not in labor force not in labor force not in labor force Total (18.76) 309 Total (46.51) 766 Total (22.89) 377 Total (11.84) 195 Working status B, 02 Working status B, 99 Working status B, 02 Working status B, 00 Adj. P-value=0.0000, Chi-square= , df=1 Adj. P-value=0.0047, Chi-square= , df=1 Adj. P-value=0.0003, Chi-square= , df=1 Adj. P-value=0.0004, Chi-square= , df=1 not in labour force active, full time (>= 80%);active, short part time (< 50%);unemployed;active, long part time (50%-80%),<missing> active, full time (>= 80%);active, long part time (50%-80%);active, short part time (< 50%) not in labour force;unemployed,<missing> not in labour force;active, full time (>= 80%) active, short part time (< 50%);unemployed;active, long part time (50%-80%),<missing> not in labour force;unemployed;active, long part time (50%-80%),<missing> active, full time (>= 80%);active, short part time (< 50%) Node 5 Node 6 Node 7 Node 8 Node 9 Node 10 Node 11 Node 12 active occupied active occupied active occupied active occupied active occupied active occupied active occupied active occupied unemployed unemployed unemployed unemployed unemployed unemployed unemployed unemployed not in labor force not in labor force not in labor force not in labor force not in labor force not in labor force not in labor force not in labor force Total (14.33) 236 Total (4.43) 73 Total (37.46) 617 Total (9.05) 149 Total (3.52) 58 Total (19.37) 319 Total (6.50) 107 Total (5.34) 88 Highest level of education achieved 04 (both grid and individual quest.) Adj. P-value=0.0292, Chi-square=8.6618, df=1 <=full-time vocational school >full-time vocational school Node 13 active occupied unemployed not in labor force Total (21.25) 350 Node 14 active occupied unemployed not in labor force Total (16.21) 267 Working status B (full time, long part time, short part time, unemployed, inactive) in 2003 used for first split IPUC07 toc intro mob surv seq conc 22/2/2007gr 9

10 Mobility tree, Women: Details for women inactive in 2003 not in labour force Node 1 active occupied unemployed not in labor force Total (18.76) 309 Working status B, 02 Adj. P-value=0.0000, Chi-square= , df=1 not in labour force active, full time (>= 80%);active, short part time (< 50%);unemployed;active, long part time (50%-80%),<missing> activ Node 5 active occupied unemployed not in labor force Total (14.33) 236 Node 6 active occupied unemployed not in labor force Total (4.43) 73 IPUC07 toc intro mob surv seq conc 22/2/2007gr 10

11 2.2 Survival trees (SHP 2002 biographical data, 2002 Wave data for some potential explanatory factors) Which are the most discriminating factors for marriage duration until divorce/separation? Used same variables as for discrete time logistic model in Ritschard and Sauvain-Dugerdil (2007) Tried two methods Maximize differences in KM survival curves using Tarone-Ware (T-W) p-value (Segal, 1988). Cox regression tree: maximize differences in proportionality factors among groups (Leblanc and Crowley, 1992; Therneau and Atkinson, 1997) IPUC07 toc intro mob surv seq conc 22/2/2007gr 11

12 T-W Survival Tree: Marriage until Divorce/Separation Population n = 3619, e = 622 S < 90% at 11 S at 30 = 0.77 TW χ 2 (1) = 54.81, p< <=1940 n = 841, e = 123 S < 90% at 21 S at 30 = 0.86 TW χ 2 (1) = 22.48, p< > 1940 n =2778, e = 499 S <90% at 9 S at 30 = 0.73 TW χ 2 (1) = 37.44, p< <=1940 & Non French L. n = 667, e = 79 S < 90% at 26 S at 30 = 0.89 TW χ 2 (1) = 8.08, p< <=1940 & French L. n = 174, e = 44 S < 90% at 11 S at 30 = 0.74 > 1940 & No Child n = 603, e = 138 S < 90% at 5 S at 30 = 0.64 TW χ 2 (1) = 4.45, p= > 1940 & Child n = 2175, e = 361 S < 90% at 11 S at 30 = 0.75 TW χ 2 (1) = 9.77, p= <=1940 & Non French L. & University n = 51, e = 12 S < 90% at 10 S at 30 = 0.76 > 1940 & No Child & University n = 86, e = 23 S < 90% at 3 S at 30 = 0.59 > 1940 & Child & German or Italian L. n = 1444, e = 217 S < 90% at 13 S at 30 = 0.77 <=1940 & Non French L. & Not University n = 667, e = 79 S < 90% at 29 S at 30 = > 1940 & No Child & Not University n = 517, e = 138 S < 90% at 6 S at 30 = 0.65 > 1940 & Child & French or unknown L. n =731, e = 144 S < 90% at 8 S at 30 = 0.70 IPUC07 toc intro mob surv seq conc 22/2/2007gr 12

13 Noeud finaux Cohorte <=1940 et Allemand, Italien ou inconnu et Université Cohorte <=1940 et Langue Allemand, Italien ou inconnu et Non Université Cohorte <=1940 et Langue Français Cohorte > 1940 et Sans Enfant et Universitaire Cohorte > 1940 et Sans Enfant et Non Universitaire Cohorte > 1940 et Avec Enfant et (Allemand ou Italien) Cohorte > 1940 et Avec Enfant et (Français ou Inconnu) IPUC07 toc intro mob surv seq conc 22/2/2007gr 13

14 Marriage Marriage survival survival probabilities probability until until Divorce/Separation, divorce/separation, by by leaves leaves Survival probability years 10 years 20 years 30 years 0 <=1940 & non French L. & University <=1940 & non French L. & non University <=1940 & French L. > 1940 & no Child & University > 1940 & no Child & non University > 1940 & Child & German or Italian L. > 1940 & Child & French or unknown L. IPUC07 toc intro mob surv seq conc 22/2/2007gr 14

15 Cox Survival Tree: Marriage until Divorce/Separation Population n = 3619, e = 622 Prop. fact. = 1.0 -LL improv.= <=1940 n = 841, e = 123 Prop. fact. = LL improv.= > 1940 n = 2778, e = 499 Prop. fact. = LL improv.= <=1940 & Non French n = 667, e = 79 Prop. fact. = 0.48 <=1940 & French n = 174, e = 44 Prop. fact. = 1.10 > 1940 & Child n = 2175, e = 361 Prop. fact. = 1.06 > 1940 & No Child n = 603, e = 138 Prop. fact. = 1.88 IPUC07 toc intro mob surv seq conc 22/2/2007gr 15

16 Noeud finaux Cohorte <=1940 et Langue Allemand, Italien ou inconnu Cohorte <=1940 et Langue Français Cohorte > 1940 et Avec Enfant Cohorte > 1940 et Sans Enfant IPUC07 toc intro mob surv seq conc 22/2/2007gr 16

17 2.3 Characteristic sequences (SHP 2002 biographical data) Selection of pairs of events, e.g. marriage and first job. For each pair, order of sequence: <, =, >, missing Which are the most typical sequences? Most discriminating sequences between sex birth cohort (1940 and before, after 1940) IPUC07 toc intro mob surv seq conc 22/2/2007gr 17

18 Frequencies of characteristic 2-event sequences 30% 25% 20% 15% 10% 5% 0% Child < Marriage Marriage < Child Child = Marriage Child < Job Job < Child Child = Job Child < Educ end Educ end < Child Child = Educ end Marriage < Job Job < Marriage Marriage = Job Marriage < Educ end Educ end < Marriage Marriage = Educ end Job < Educ end Educ end < Job Job = Educ end IPUC07 toc intro mob surv seq conc 22/2/2007gr 18

19 Cohort discriminating 2-event sequences Naissance Node 0 apres avant Total (100.00) 4755 Départ et mariage Adj. P-value=0.0000, Chi-square= , df=3 < > = <missing> Node 1 Node 2 Node 3 Node 4 apres apres apres apres avant avant avant avant Total (45.95) 2185 Total (1.51) 72 Total (24.44) 1162 Total (28.10) 1336 Mariage et emploi Départ et emploi Mariage et fin des études Adj. P-value=0.0063, Chi-square= , df=1 Adj. P-value=0.0000, Chi-square= , df=1 Adj. P-value=0.0000, Chi-square= , df=2 =,<missing> >;< <;= >,<missing> >;= < <missing> Node 5 Node 6 Node 7 Node 8 Node 9 Node 10 Node 11 apres apres apres apres apres apres apres avant avant avant avant avant avant avant Total (5.62) 267 Total (40.34) 1918 Total (4.48) 213 Total (19.96) 949 Total (4.16) 198 Total (1.18) 56 Total (22.75) 1082 Départ et fin des études Départ et fin des études Départ et fin des études Enfant et emploi Enfant et emploi Enfant et emploi Adj. P-value=0.0249, Chi-square=9.1512, df=1 Adj. P-value=0.0498, Chi-square=7.8866, df=1 Adj. P-value=0.0064, Chi-square= , df=1 Adj. P-value=0.0339, Chi-square=4.5007, df=1 Adj. P-value=0.0007, Chi-square= , df=1 Adj. P-value=0.0000, Chi-square= , df=1 > <;=,<missing> >;< =,<missing> >,<missing> <;= > <missing> >;<;= <missing> >;< =,<missing> Node 12 Node 13 Node 14 Node 15 Node 16 Node 17 Node 18 Node 19 Node 20 Node 21 Node 22 Node 23 apres apres apres apres apres apres apres apres apres apres apres apres avant avant avant avant avant avant avant avant avant avant avant avant Total (1.72) 82 Total (3.89) 185 Total (28.29) 1345 Total (12.05) 573 Total (2.52) 120 Total (1.96) 93 Total (13.31) 633 Total (6.65) 316 Total (1.14) 54 Total (3.03) 144 Total (2.21) 105 Total (20.55) 977 First split variable: {Marriage, Leaving Home} IPUC07 toc intro mob surv seq conc 22/2/2007gr 19

20 Cohort: details for Leaving Home before Marriage < Node 1 apres avant Total (45.95) 2185 > Node 2 apres avant Total (1.51) 72 Mariage et emploi Adj. P-value=0.0063, Chi-square= , df=1 =,<missing> >;< <;= Node 5 apres avant Total (5.62) 267 Node 6 apres avant Total (40.34) 1918 Node 7 apres avant Total (4.48) 213 Départ et fin des études Adj. P-value=0.0249, Chi-square=9.1512, df=1 Départ et fin des études Adj. P-value=0.0498, Chi-square=7.8866, df=1 Départ et fin des études Adj. P-value=0.0064, Chi-square=11.6 > <;=,<missing> >;< =,<missing> >,<missing> Node 12 apres avant Total (1.72) 82 Node 13 apres avant Total (3.89) 185 Node 14 apres avant Total (28.29) 1345 Node 15 apres avant Total (12.05) 573 Node 16 apres avant Total (2.52) 120 Categ apres avant Total IPUC07 toc intro mob surv seq conc 22/2/2007gr 20

21 Sex discriminating 2-event sequences sexe Node 0 masculin féminin Total (100.00) 4755 Emploi et fin des études Adj. P-value=0.0000, Chi-square= , df=3 > < = <missing> Node 1 masculin féminin Total (20.06) 954 Node 2 masculin féminin Total (29.91) 1422 Node 3 masculin féminin Total (20.46) 973 Node 4 masculin féminin Total (29.57) 1406 Enfant et emploi Adj. P-value=0.0028, Chi-square= , df=1 Départ et emploi Adj. P-value=0.0000, Chi-square= , df=1 Départ et emploi Adj. P-value=0.0000, Chi-square= , df=1 >;=,<missing> < <;=,<missing> > <;=,<missing> > Node 5 masculin féminin Total (17.77) 845 Node 6 masculin féminin Total (2.29) 109 Node 7 masculin féminin Total (14.30) 680 Node 8 masculin féminin Total (15.60) 742 Node 9 masculin féminin Total (21.64) 1029 Node 10 masculin féminin Total (7.93) 377 Départ et fin des études Adj. P-value=0.0274, Chi-square=8.9750, df=1 Mariage et fin des études Adj. P-value=0.0000, Chi-square= , df=1 Mariage et fin des études Adj. P-value=0.0019, Chi-square= , df=1 >;<,<missing> = >,<missing> <;= >;=,<missing> < Node 11 masculin féminin Total (16.00) 761 Node 12 masculin féminin Total (1.77) 84 Node 13 masculin féminin Total (10.20) 485 Node 14 masculin féminin Total (5.40) 257 Node 15 masculin féminin Total (20.44) 972 Node 16 masculin féminin Total (1.20) 57 First split variable: {Job, Education End} IPUC07 toc intro mob surv seq conc 22/2/2007gr 21

22 Sex: details for Job after Education end féminin 53.7 Total (100.0 Emploi et fin des é Adj. P-value=0.0000, Chi-squa > Node 1 masculin féminin Total (20.06) 954 < Node 2 masculin féminin Total (29.91) 1422 Enfant et emploi Adj. P-value=0.0028, Chi-square= , df=1 Départ et emploi Adj. P-value=0.0000, Chi-square= , df=1 >;=,<missing> < <;=,<missing> > Node 5 masculin féminin Total (17.77) 845 Node 6 masculin féminin Total (2.29) 109 Node 7 masculin féminin Total (14.30) 680 Node Category masculin féminin Total ( Départ et fin des études Adj. P-value=0.0274, Chi-square=8.9750, df=1 Mariage et fin Adj. P-value=0.0000, Chi >;<,<missing> = >,<missing> Node 11 masculin féminin Total (16.00) 761 Node 12 masculin féminin Total (1.77) 84 Node 13 masculin féminin Total (10.20) 485 IPUC07 toc intro mob surv seq conc 22/2/2007gr 22

23 3 Foreseen Developments Extend tree approaches for Time varying covariates Multilevel contexts Mining typical sequence patterns and association rules Suitable validation criteria Friendly graphical interface for making methods easily accessible Analysis of Swiss life courses Differential impact of various profiles of social insertion Broken lives... IPUC07 toc intro mob surv seq conc 22/2/2007gr 23

24 References Han, J. and M. Kamber (2001). Data Mining: Concept and Techniques. San Francisco: Morgan Kaufmann. Hand, D. J., H. Mannila, and P. Smyth (2001). Principles of Data Mining. Adaptive Computation and Machine Learning. Cambridge MA: MIT Press. Leblanc, M. and J. Crowley (1992). Relative risk trees for censored survival data. Biometrics 48, Piatetsky-Shapiro, G. (Ed.) (1989). Notes of IJCAI 89 Workshop on Knowledge Discovery in Databases (KDD 89), Detroit, MI. Ritschard, G. et C. Sauvain-Dugerdil (2007). L enfant ciment du couple ou le couple comme ciment de la relation du père à l enfant? Quelques enseignements de l enquête rétrospective du Panel Suisse de Ménages. In C. Burton-Jeangros, E. Widmer, et C. Lalive d Epinay (Eds.), Interactions familiales et constructions de l intimité., coll. Questions sociologiques. Paris : L Harmattan. (à paraître). Segal, M. R. (1988). Regression trees for censored data. Biometrics 44, Therneau, T. M. and E. J. Atkinson (1997). An introduction to recursive partitioning using the rpart routines. Technical Report Series 61, Mayo Clinic, Section of Statistics, Rochester, Minnesota. References toc intro mob surv seq conc 22/2/2007gr 24