Data Mining. Eibe Frank and Ian H. Witten. March Properties of the entropy. Wishlist for a purity measure. Computing the gain ratio.

Transcription

1 Wishlist for a purity measure Properties we require from a purity measure: When node is pure, measure should be zero When impurity is maximal (i.e. all classes equally likely), measure should be maximal Measure should obey multistage property (i.e. decisions can be made in several stages): measure([,,]) = measure([,7]) + (7/9)! measure([,]) Entropy is the only function that satisfies all three properties! Properties of the entropy The multistage property: q r entropy( p,q,r) = entropy( p,q + r) + ( q + r)! entropy(, ) q + r q + r Simplification of computation: info ([,,]) = " / 9! log( / 9) " / 9! log(/ 9) " /9! log( / 9) = [! log! log! log + 9log9]/ 9 te: instead of maximizing info gain we could just minimize information Gain ratio Computing the gain ratio Gain ratio: a modification of the information gain that reduces its bias Gain ratio takes number and size of branches into account when choosing an attribute It corrects the information gain by taking the intrinsic information of a split into account Intrinsic information: entropy of distribution of instances into branches (i.e. how much info do we need to tell which branch an instance belongs to) Example: intrinsic information for ID code info([,,...,) = " (#/ " log/) =.87 bits Value of attribute decreases as intrinsic information gets larger Definition of gain ratio: gain("attribute") gain_ratio ("Attribute") = intrinsic_info("attribute") Example:.9 bits gain_ratio ("ID_code") = =.6.87 bits Gain ratios for weather data More on the gain ratio Outlook Gain: Split info: info([5,,5]) Gain ratio:.7/.577 Humidity Gain: Split info: info([7,7]) Gain ratio:.5/ Temperature Gain:.9-.9 Split info: info([,6,]) Gain ratio:.9/.6 Windy Gain: Split info: info([8,6]) Gain ratio:.8/ Outlook still comes out top However: ID code has greater gain ratio Standard fix: ad hoc test to prevent splitting on that type of attribute Problem with gain ratio: it may overcompensate May choose an attribute just because its intrinsic information is very low Standard fix: only consider attributes with greater than average information gain 5 6 March

2 Industrial-strength algorithms For an algorithm to be useful in a wide range of real-world applications it must: Permit numeric attributes Allow missing values Be robust in the presence of noise Be able to approximate arbitrary concept descriptions (at least in principle) Basic schemes need to be extended to fulfill these requirements Decision trees Extending ID: to permit numeric attributes: straightforward to dealing sensibly with missing values: trickier stability for noisy data: requires pruning mechanism End result: C.5 (Quinlan) Best-known and (probably) most widely-used learning algorithm Commercial successor: C5. Numeric attributes Standard method: binary splits E.g. temp < 5 Unlike nominal attributes, every attribute has many possible split points Solution is straightforward extension: Evaluate info gain (or other measure) for every possible split point of attribute Choose best split point Info gain for best split point is info gain for attribute Computationally more demanding Outlook Overcast Rainy Weather data (again!) Temperature Hot Hot Hot Mild Outlook Humidity High High High Temperature 85 8 Windy True Humidity Play Windy Overcast 8 86 If outlook = sunny Rainy and humidity 75 = high then 8 play = no If outlook = rainy and windy = true then play = no If outlook = overcast then play = yes If humidity = normal then play = yes If of the If above outlook then = play sunny = yes and humidity > 8 then play = no If outlook = rainy and windy = true then play = no If outlook = overcast then play = yes If humidity < 85 then play = yes If of the above then play = yes 85 9 True Play Example Avoid repeated sorting! Split on temperature attribute: E.g. temperature < 7.5: yes/, no/ temperature 7.5: yes/5, no/ Info([,],[5,]) = 6/ info([,]) + 8/ info([5,]) =.99 bits Place split points halfway between values Can evaluate all split points in one pass! Sort instances by the values of the numeric attribute Time complexity for sorting: O (n log n) Does this have to be repeated at each node of the tree! Sort order for children can be derived from sort order for parent Time complexity of derivation: O (n) Drawback: need to create and store an array of sorted indices for each numeric attribute 5 6 March

3 Binary vs multiway splits Splitting (multi-way) on a nominal attribute exhausts all information in that attribute minal attribute is tested (at most) once on any path in the tree t so for binary splits on numeric attributes! Numeric attribute may be tested several times along a path in the tree Disadvantage: tree is hard to read Remedy: pre-discretize numeric attributes, or use multi-way splits instead of binary ones Computing multi-way splits Simple and efficient way of generating multi-way splits: greedy algorithm Dynamic programming can find optimum multi-way split in O (n ) time imp (k, i, j ) is the impurity of the best split of values x i x j into k sub-intervals imp (k,, i ) = min <j <i imp (k,, j ) + imp (, j+, i ) imp (k,, N ) gives us the best k-way split In practice, greedy algorithm works as well 7 8 Missing values Pruning Split instances with missing values into pieces A piece going down a branch receives a weight proportional to the popularity of the branch weights sum to Info gain works with fractional instances use sums of weights instead of counts During classification, split the instance into pieces in the same way Merge probability distribution using weights Prevent overfitting to noise in the data Prune the decision tree Two strategies: Postpruning take a fully-grown decision tree and discard unreliable parts Prepruning stop growing a branch when information becomes unreliable Postpruning preferred in practice prepruning can stop early 9 Prepruning Based on statistical significance test Stop growing the tree when there is no statistically significant association between any attribute and the class at a particular node Most popular test: chi-squared test ID used chi-squared test in addition to information gain Only statistically significant attributes were allowed to be selected by information gain procedure Early stopping Pre-pruning may stop the growth process prematurely: early stopping Classic example: XOR/Parity-problem individual attribute exhibits any significant association to the class Structure is only visible in fully expanded tree Prepruning won t expand the root node But: XOR-type problems rare in practice a b class And: prepruning faster than postpruning March

4 First, build full tree Postpruning Then, prune it Fully-grown tree shows all attribute interactions Problem: some subtrees might be due to chance effects Subtree replacement Bottom-up Consider replacing a tree only after considering all its subtrees Two pruning operations: Subtree replacement Subtree raising Possible strategies: error estimation significance testing MDL principle Subtree replacement Bottom-up Attribute Type Consider replacing a tree Duration Wage only increase after first year considering Percentage all Wage its increase subtrees second year Percentage Wage increase third year Cost of living adjustment Working hours per week Pension Standby pay Shift-work supplement Education allowance Statutory holidays Vacation Long-term disability assistance Dental plan contribution Bereavement assistance Health plan contribution Acceptability of contract (Number of years) % Percentage {,tcf,tc} (Number of hours) 8 {,ret-allw, empl-cntr} Percentage Percentage {yes,no} yes (Number of days) {below-avg,avg,gen} avg {yes,no} no {,half,full} {yes,no} no {,half,full} {good,bad} bad % 5% tcf 5 % 5% 5 gen good.%.% 8 % gen full full good.5. avg yes full yes half good Subtree raising Delete node Redistribute instances Slower than subtree replacement (Worthwhile) 5 6 Estimating error rates C.5 s method Prune only if it reduces the estimated error Error on the training data is NOT a useful estimator (would result in almost no pruning) Use hold-out set for pruning ( reduced-error pruning ) Error estimate for subtree is weighted sum of error estimates for all its leaves Error estimate for a node: & $ z e = f + + z % N f N f z ' + N N & z # $ +! % N " If c = 5% then z =.69 (from normal distribution) #! " C.5 s method Derive confidence interval from training data Use a heuristic limit, derived from this, for pruning Standard Bernoulli-process-based method Shaky statistical assumptions (based on training data) 7 f is the error on the training data N is the number of instances covered by the leaf 8 March

5 Example Complexity of tree induction Assume m attributes n training instances tree depth O (log n) f = 5/ e =.6 e <.5 so prune! Building a tree O (m n log n) Subtree replacement O (n) f=. e=.7 f=.5 e=.7 f=. e=.7 Subtree raising O (n (log n) ) Every instance may have to be redistributed at every node between its leaf and the root Cost for redistribution (on average): O (log n) Combined using ratios 6::6 gives.5 9 Total cost: O (m n log n) + O (n (log n) ) From trees to rules C.5: choices and options Simple way: one rule for each leaf C.5rules: greedily prune conditions from each rule if this reduces its estimated error Can produce duplicate rules Check for this at the end Then look at each class in turn consider the rules for that class find a good subset (guided by MDL) C.5rules slow for large and noisy datasets Commercial version C5.rules uses a different technique Much faster and a bit more accurate C.5 has two parameters Confidence value (default 5%): lower values incur heavier pruning Minimum number of instances in the two most popular branches (default ) Then rank the subsets to avoid conflicts Finally, remove rules (greedily) if this decreases error on the training data Discussion TDIDT: Top-Down Induction of Decision Trees The most extensively studied method of machine learning used in data mining Different criteria for attribute/test selection rarely make a large difference Different pruning methods mainly change the size of the resulting pruned tree C.5 builds univariate decision trees Some TDITDT systems can build multivariate trees (e.g. CART) March

6 Selecting a test Example: contact lens data Goal: maximize accuracy t total number of instances covered by rule p positive examples of the class covered by rule t p number of errors made by rule Select test that maximizes the ratio p/t We are finished when p/t = or the set of instances can t be split any further Rule we seek: Possible tests: Age = Young Age = Pre-presbyopic Age = Presbyopic Spectacle prescription = Myope Spectacle prescription = Hypermetrope Astigmatism = no Astigmatism = yes Tear production rate = Tear production rate = If /8 /8 /8 / / / / / / Modified rule and resulting data Further refinement Rule with best test added: Instances covered by modified rule: Age Spectacle prescription Young Myope Young Myope Young Hypermetrope Young Hypermetrope Pre-presbyopic Myope Pre-presbyopic Myope Pre-presbyopic Hypermetrope Pre-presbyopic Hypermetrope Presbyopic Myope Presbyopic Myope Presbyopic Hypermetrope Presbyopic Hypermetrope Astigmatism Tear production rate Recommended lenses hard Current state: Possible tests: Age = Young Age = Pre-presbyopic Age = Presbyopic Spectacle prescription = Myope Spectacle prescription = Hypermetrope Tear production rate = Tear production rate = and / / / /6 /6 /6 /6 Modified rule and resulting data Rule with best test added: Further refinement Current state: Possible tests: and Instances covered by modified rule: Age Spectacle prescription Young Myope Young Hypermetrope Pre-presbyopic Myope Pre-presbyopic Hypermetrope Presbyopic Myope Presbyopic Hypermetrope Astigmatism Tear production rate Recommended lenses hard 5 Age = Young Age = Pre-presbyopic Age = Presbyopic Spectacle prescription = Myope Spectacle prescription = Hypermetrope Tie between the first and the fourth test We choose the one with greater coverage / / / / / 6 March

7 The result Final rule: and spectacle prescription = myope Second rule for recommending hard lenses : (built from instances not covered by first rule) If age = young and astigmatism = yes These two rules cover all hard lenses : Process is repeated with other two classes 7 March