Looking Inside the Crystal Ball: Using Predictive Analytics to Tailor Services for Better Family Outcomes Presented by: David Kilgore
Business Analysis Business analysis is a critical process that drives organizational structures and systems. Through effective business analysis, one can define and evaluate potential initiatives that best fit organizational goals. In this course, you gain the foundational knowledge needed to effectively perform key business analysis functions. You learn how to apply a core business analysis framework as well as participate in interactive sessions to improve analytical competencies. Specific competencies developed include: Analyzing and prioritizing business needs Creating dynamic business models using workflow diagrams Deploying evaluative techniques for project selection and outcomes Writing SMART business objectives Quantifying business case benefits and costs Communicating the requirements package to stakeholders
Getting Results I want to be fit and in shape
Getting Results
Analytics Analytics is the application of computer technology, operational research, and statistics to solve problems in business and industry. For Los Angeles: TAD (Technology and Analysis Division) - Programmers and Analysts who research and create statistical reports for management review PTE (Policy, Training, and Evaluation Division) - A unit that puts into practice decisions made by management and then evaluates the effect that policy or training has had on the department CSTATs Meeting - Attended by all senior management where departmental performance is reviewed, discussed, and decisions made targeting areas for emphasis or improvement Data Sharing - Down to the line level, data is provided for each worker to research and improve their own caseload, performance, and information sharing of best practices. SPSS - Software program used for analysis and statistics
Analytics - TAD Responsibilities: Find and make available data on the department s performance to be analyzed. Utilize the data to develop reports and charts as well as perform research on issues facing the department. Perform compliance and data reliability audits as required by State and internally as needed. Identify and clean up data issues that may be hindering the system from performing correctly or reporting accurately. Track specific project results to ensure the outcome occurs as expected. Coordinates department wide projects centrally for efficiency and accuracy. Develop automated systems to minimize manual intervention for repetitive actions.
Analytics - PTE Responsibilities: Draft policy for staff based on management direction, regulations, etc Train staff on the policies and procedures Evaluate the efficiency and accuracy of staff
Analytics - CSTATs Monthly Meeting (~3 hours) All Senior Managers are required to attend Rotate presenters to encourage new outlooks and information sharing Designed to be informative not punitive Designed to inspire questions and identify areas that should be targeted for special effort Over 50 pages of slides Some show great results Monthly review to make sure everything stays on track Some show challenges Focused discussion and evaluation Some slides change from month to month depending on where the research takes us.
CSTATs Examples Cumulative Chart on Establishment Shows we are catching up on old S&Cs that have needed follow up, but now we have a large number of cases ready to become orders but for some reason are not. Follow up is taking place. Establishment Process Year to Date through 01/31/2012 Cases Opened 13,249 S&Cs Filed 8,427 1 S&Cs Served 12,062 Orders Filed 5,801 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 Orders Filed (L17) S&Cs Served S&Cs Filed (LGL-201 Court E-filing) Cases Opened (L46)
CSTATs Examples Total Call Volume CFY 2012 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 54,269 44,786 Jul-11 61,345 60,141 57,000 55,327 51,502 49,027 51,642 48,508 47,759 51,691 45,564 44,296 Aug-11 Sep-11 Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Total Calls Calls Answered Calls Abandoned Calls Deflected Call Handling & Customer Wait Time CFY 2012 Trends over time What impact does State or Local decisions have on the department? What adjustments are necessary? 14:24 12:00 9:36 7:12 4:48 2:24 0:00 3:59 3:41 3:34 3:18 1:39 2:34 3:10 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 AVG Work Mode AVG Talk Time AVG Hold Time AVG Call Handling Customer Wait Time
Gathering Data Data you Date Data you Marry
Data Sharing Key concepts for Los Angeles Everyone needs to be part of Data Analysis 1600 minds analyzing performance is better than 30 Everyone needs to know and understand the scope of their caseload Everyone needs to understand the Federal Performance Measures and how their job effects them Everyone needs to see the results of their individual efforts towards the larger goal Los Angeles Solution Case Management Tool
CMT Purpose Provide every case manager, supervisor, manager, and division chief with their own caseload dataset that they can manipulate, query, segment, and evaluate without having to make individual requests for it. This isn t dashboards with summarized results and trends This isn t an employee monitoring tool This wasn t even intended for management when first created Child Support Officers needed to know how many cases were in their caseload, what the status of each was, and how to prioritize their time. This tool allowed them to proactively identify groups of cases needing the same type of action and address it instead of retroactively receiving a task that says something needs to be done.
Run Query No Arrears Payment this FFY
Data Mining 1. Ask questions and use the data to find the answers 2. Use the answers to focus your energy in exactly the right places Examples: Arrears Project Case Segmentation
Arrears Example Begin by asking ourselves questions What is the current trend on Arrears? What is the total number of cases with an arrears balance? How many cases have only an arrears balance? Of the cases with Arrears how many had at least one payment last year? How many cases have less than $500 owing on arrears and no current support? How many cases have never had a payment on the case? How many cases have had no payment in the last 18 months? How many cases have no active address? How many cases have active phone numbers?
Arrears Example What did we learn: 178,000 Cases have an arrears balance starting the year 97,239 of these cases paid something towards arrears last year and will probably perform similarly this year (=54.60%) Remaining 80,761 need intervention (98,539 had no payment since conversion, but only 80,761 remained open as the new FFY starts) 37,624 are arrears only We know which of these have addresses and phone numbers 1876 of these cases have an arrears balance less than $500 (=1.04% of FPM) 43,047 have current support and arrears We know which of these have addresses and phone numbers 43 Cases have an Arrears balance over 500k 8,985 Cases have an Arrears balance between 100k and 500k
Arrears Example What did we do: Stern Letter Invitations Targets cases with current support and arrears to pay on their case, review for modification, or prepare for criminal referral Auto-Dialer Campaign Target cases where no payment has happened since conversion as worker intervention on these cases will yield low results Arrears Days Target NCPs in the LA area with arrears only balances or low overall arrears balances to meet with an attorney to discuss their case Negotiations Campaigns Structure Operations Divisions to enhance child support officer interaction with NCPs for the purpose of improving negotiated collections Case Closure Target cases that meet the case closure criteria to ensure cases are closed before the end of the FFY Task Cleanup Target tasks in CSE that affect the Arrears FPM and bring current Employment Record Cleanup Enhance the accuracy of the data in CSE for more efficient electronic assistance
Arrears Results Some strategies were more successful than others and we add or delete them based on our review in CSTATs, but the end result in FFY 2010 pushed us over the 55% mark and the strategies developed continue to provide additional gains each year thereafter. Arrears FPM by FFY 60.00% 55.00% 50.00% 45.00% 40.00% 35.00% 30.00% 25.00% 20.00% Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep FFY 05 FFY 06 FFY 07 FFY 08 FFY 09 FFY 10 FFY 11
Data Mining Tips Focus our energy on the right things at the right time of the year Maximize automation before utilizing manpower Push the data down into the hands of the end user Track the progress to ensure we are on track in CSTATs
Case Segmentation This is a new avenue that Los Angeles is in the process of implementing. CSE doesn t provide an easy systematic method for segmenting ones caseload. However, within the CMT, this has become possible. We auto populated what we could and staff are manually identifying the rest as they come across the cases. Establishment, 59478 TOTAL CASELOAD - 319,903 Willing but Unable, 17678 (5.5%) Willing and Able, 47061 Unknown, 103532 Unwilling and Unable, 29924 Unwilling but Able, 62230
Case Segmentation Within CMT, further segmentation options are available to better allow us to target specific populations of cases for certain types of actions. Case Segmentation Case Count Segmentation Group Unassigned 103532 Unknown Full Pay 44266 Willing and Able Arrears Only Not Paying 43500 Unwilling but Able Partial Pay WA/Voluntary 15031 Willing but Unable No S&C - Have Address 14766 Establishment S&C Non-Served 14034 Establishment Default No Order 11627 Establishment Pending Closure 9425 Unwilling and Unable Locate - No SSN 9192 Establishment Not Paying Active Wage Assignment 7425 Unwilling but Able Not Paying Unable to Locate 6133 Unwilling and Unable S&C Generated No Address 5432 Establishment Locate 4022 Unwilling and Unable Partial Pay IRS/FTB 3281 Unwilling but Able Not Paying - UA - Incarcerated 3211 Unwilling and Unable Zero Order - Other 3165 Unwilling and Unable Partial Pay Self Employed 3153 Unwilling but Able
Predictive Analytics Survival Research Project Goal Study How important is the receipt of child support to formerly assisted families in remaining selfsufficient and not reliant upon public assistance? Does regular payments or the amount of child support paid affect a formerly assisted custodial parent s ability to remain off of CalWORKs? Establish an initial baseline data set for future studies to build upon, both on this topic and new topics, by highlighting questions and data elements that may arise.
Process Study Focus Affirm the benefit and value of Child Support Services to the Welfare Department Data Collection 90% getting the right data 10% execution of analysis Brainstorming What type of data to pull and what factors might influence a CPs ability to stay off aid Determine Survival over 24 months 1 st Key Decision - Determine total # of months off aid over 24 month period Track Unemployment rates, Gender, CP Role, Child Support Payments, CP Income, NCP Income, # of Children, Age of Children, Age of CP, Age of NCP, Ethnicity, Language, Birth County, Education, Citizenship, Aid Type, 2 Parent Household
Survival Results Interval Start Time Number Entering Interval Number Withdrawi ng during Interval Number Exposed to Risk Number of Terminal Events Proportion Terminatin g Proportion Surviving Cumulative Proportion Surviving at End of Interval Probability Density Std. Error of Probability Density Hazard Rate Std. Error of Hazard Rate 0 22515 0 22515.000 0.00 1.00 1.00.000.000.00.00 1 22515 0 22515.000 12327.55.45.45.548.003.75.01 2 10188 0 10188.000 5569.55.45.21.247.003.75.01 3 4619 0 4619.000 1256.27.73.15.056.002.31.01 4 3363 0 3363.000 583.17.83.12.026.001.19.01 5 2780 0 2780.000 450.16.84.10.020.001.18.01 6 2330 0 2330.000 248.11.89.09.011.001.11.01 7 2082 0 2082.000 239.11.89.08.011.001.12.01 8 1843 0 1843.000 153.08.92.08.007.001.09.01 9 1690 0 1690.000 106.06.94.07.005.000.06.01 10 1584 0 1584.000 98.06.94.07.004.000.06.01 11 1486 0 1486.000 46.03.97.06.002.000.03.00 12 1440 0 1440.000 58.04.96.06.003.000.04.01 13 1382 0 1382.000 39.03.97.06.002.000.03.00 14 1343 0 1343.000 44.03.97.06.002.000.03.01 15 1299 0 1299.000 28.02.98.06.001.000.02.00 16 1271 0 1271.000 20.02.98.06.001.000.02.00 17 1251 0 1251.000 40.03.97.05.002.000.03.01 18 1211 0 1211.000 38.03.97.05.002.000.03.01 19 1173 0 1173.000 31.03.97.05.001.000.03.00 20 1142 0 1142.000 28.02.98.05.001.000.02.00 21 1114 0 1114.000 32.03.97.05.001.000.03.01 22 1082 0 1082.000 18.02.98.05.001.000.02.00 23 1064 0 1064.000 22.02.98.05.001.000.02.00 24 1042 0 1042.000 0.00 1.00.05.000.000.00.00 25 1042 0 1042.000 1042 1.00.00.00.000.000.00.00 a. The median survival time is 1.91
Study Population Overview Given all the independent variables previously discussed, the most common welfare recipient would be: CP Role Mother (75%) Gender Female (90%) Ethnicity Hispanic (38%), White (21%), black (9%) Primary Language English (79%, includes English as second language) CPs Average Age 36 years old NCPs Average Age 37 years old Number of Children 1-2 Children s average Age 11 years old
Study Attributes Gender In 90% (20,205) of the cases, the gender of the welfare recipient was female. In 7% (1,482) the gender was male. The remaining 3% (828) of the cases were either labeled as other or the data was missing. Ethnicity The highest percentage of the welfare recipients in the study population was of Hispanic ethnicity at 38% (8,497). The ethnicity with the second highest percentage of welfare recipients was White with 21% (4,779). In 19% (4,289) of the cases, the ethnicity of the welfare recipients was Black with the remaining 9% (2,063) as Other. The ethnicity of Other includes welfare recipients listed as Asia, Alaska/Native American, Japanese, Chinese, and Multiracial. The ethnicity of 13% (2,887) of the cases was Unknown. Primary Language English was the primary language spoken by 79% (17,773) of the welfare recipients while Spanish was the language spoken by 13% (3,002). However, this field is manually updated by the case worker and is often not adjusted after case setup. Children s Age The children s average age for the study population was 11 years old with a standard deviation of 7 years making 85% of the dependents between the age of 4 years and 18 years. For 4% of the children, their age was zero indicating that the child was unborn in December 2008.
Study Attributes CP and NCP Age The average age of the welfare recipients in the study population was 36 years with a standard deviation of 13. For the Non Custodial Parents associated to the welfare recipients, the average age was 37 years, with a standard deviation of 11 years. CP and NCP Income CSE was incapable of providing the actual income of either group but was able to identify who was or was not employed. A detailed review of the records with the income indicator during the 24 months period showed the following: 75% (16,837) of both Welfare Recipients and Non Custodial Parents had no income. 15% (3,276) showed that the NCP had an income while the Welfare Recipient did not. 8% (1,883) showed that the Welfare Recipients and NCP both had an income. The percentage of the study where both the Welfare Recipient and the Non Custodial Parent on the case did not show any income gradually decreased as the number of months the Welfare Recipient was off aid increased as one would expect. 89% (20,113) of the Welfare Recipients indicated that there was no income while 11% (2,402) did have income. The NCP had no income on 83% (18,720) of the records. The average age of the NCPs showed no significance difference between those who did or did not have income.
Number of Cases Study Attributes 14000 13000 13,665 Chart 2: Number of Dependents/Children on Case Based on Study Population 12000 11000 10000 9000 8000 7000 6000 5000 5,083 4000 3000 2000 1000 0 1,934 859 690 189 64 20 5 2 2 1 1 0 1 2 3 4 5 6 7 8 9 11 12 14 Number of Dependents Number of Cases
Child Support Orders Chart 3 Child Support Orders on Study Population Cases - Dec 2008 Medical Only 0% Interstate Initiated Arrears Only 8% Interstate Received 0% Zero Order 18% No Order Shown 34% Current Support 39% No Order Shown Current Support Arrears Only Interstate Initiated Interstate Received Medical Only Zero Order
Child Support Order Performance Table 5 Current Support and Arrears Collections # of Cases No Support Order 6136 Current Support Due Arrears Due Current Support Paid Arrears Paid Current Support Percentage Arrears Percentage Current Support 8624 $45,664,503 $198,692,992 $19,040,372 $8,112,072 41.70% 4.08% Arrears Only 2818 $1,473,033 $145,612,638 $575,102 $5,039,395 39.04% 3.46% Zero Orders 4390 $139,932 $30,482,956 $32,029 $357,057 22.89% 1.17% Medical Only 23 $139,932 $415,571 $32,029 $18,292 22.89% 4.40% Interstate Initiating 473 $2,448,752 $13,432,935 $854,716 $475,113 34.90% 3.54% Interstate Receiving 51 $233,325 $1,203,349 $88,085 $12,815 37.75% 1.06% Totals 22515 $50,099,477 $389,840,441 $20,622,332 $14,014,744 41.16% 3.59%
Recipients Months off Welfare Chart 6 Welfare Recipients' and Number of Months Off Welfare 3500 3000 2,913 2,608 2500 2,328 2000 1,946 1,603 1500 1,387 1000 1,160 971 853 722 1,042 576 569 555 529 500 369 404 336 325 317 248 225 258 271 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Number of Months Months off Aid
Analysis Table 8 Profile Analysis Total Study Segment I Segment II Orders FPM 72% 74% 76% Current Support FPM 41% 39% 47.00% Average Current Support Collected $2,358 $2,407 $2,359 Arrears FPM 57% 56% 62% Average Arrears Balance $35,447 $35,767 $31,113 Total Collections per Case $2,904 $2,928 $2,803 CPs Age 36 36 38 CPs with Income 11% 11% 9% NCPs Age 37 37 38 NCPs with Income 17% 17% 16% Number of Dependants 1.5 1.5 1.3 Average Age of Dependents 11 11 12
Analysis 79% of the welfare recipients return to aid within three months. Of the total sample population, only 5% (1,042) had not returned to welfare. Regression Analysis Used to predict or forecast outcomes based on a set of variables Used to understand which independent variables are related to the dependent variables.
Regression Analysis Example CP and NCP are both employed NCP is making child support payments The CP would be more likely to remain off aid In actuality, there are 373 cases where the above example is true in the study. Only 19 of those cases are in Segment II (1.7% of Segment II), the segment you would expect to see most of these. 311 cases fell into Segment I (1.5% of Segment I) which is very close to Segment II s percentage.
Collections Impact on Remaining off Welfare
Where to go from here Scoring Cases Which are more likely to pay? Targeted Enforcement When remedies are more likely to work on which cases? David Kilgore Deputy Director Los Angeles Child Support Services Department 323-889-3405 David_kilgore@cssd.lacounty.gov