Presenter: Doug Reynolds, Development Dimensions International

Big Data and Talent Management: Using assessment and technology to run better organizations Presented by: Doug Reynolds, Ph.D. Senior Vice President & CTO Development Dimensions International (DDI)

Big Data: Old Wine or New Opportunity? Presented by: Doug Reynolds, Ph.D. Senior Vice President & CTO Development Dimensions International (DDI)

Big Data About People is Hot!

Anatomy of a trend Reference to Moneyball Uninformed overgeneralization Pivot point

Anatomy of a trend Review of common problems

Anatomy of a trend Discovery of known knowledge Nuggets of insight from new technologies

What does big data mean for I-O and HR? A visit to the dust bowl A sea of spurious correlation High school science New insights from new tools

A few definitions Big data: Large quantity and variety of data generated through internet-based systems; often involves data from multiple systems (3Vs). Digital exhaust: trace information remaining after use of an online tool, often irrelevant to the purpose of the tool. HR analytics: organizationally relevant statistics regarding people associated with the organization

Roadmap for the discussion Why are we talking about this now? The promise of big data in HR A few examples Old wine? New opportunity?

A few important trends Labor Market Management Technology Talent is a differentiator Aging workforce Skill gaps Automation Globalization Outsourcing Interoperability Services architecture Cloud computing

A (simplified) view of the evolution of business software

Start with an inefficient business process Input Step 1 Step 2 Step 3 Step 4 Output

design software to improve it Input Step 1 Step 2 Simplify, automate, increase availability, globalize, etc. Step 3 Step 4 Output

It doesn t take much to get started

Competitors may capture different parts of the process Company 1 Company 2

The next challenge: add value beyond your step Interconnect with other software tools to automate more of the business process

Acquire, merge, or build to own the whole process Attempt to support the whole process: Build more pieces Buy your neighbor Sell out Or, go out of business

Once you own one process, you get hungry for others Process: A B C D

Interconnections allow for more insight and strategic value A B C D Within Process Across Processes

In the HR context: Within Process Across Processes

Big data about people Recruit Promote Hire Manage Train

The promise of big data for talent management Strategic Impact Insight Process Automation

Candidate Source Assessment Data for Individual and Group Manager Satisfaction with Quality of Candidates Ratio Offers to Acceptance and Diversity of New Hires Candidate feedback on the hiring process Confidence of Hiring Manager in the New Hire and Confidence of the New Hire that they are in the Right Job Job Performance and Engagement Predictive Hiring Analytics Source Assessment Final Interview Job Offer First Day on the Job 6 Months on the Job 1 year on the job

A pervasive issue: assessment rigor Talent quality? 1 2 3 4 5

Examples of Assessment-driven Analytics

Example 1: selection testing Selection test for graduate hiring Relevant and effective across cultures Strong security (difficult to cheat on) Strong predictor of performance Available anytime, anywhere Brief

Test features Figural reasoning: measure of reasoning ability, critical thinking, and problem-solving Non-verbal/graphical items No translations Applicable regardless of candidate reading level Culture-free/fair for all candidate groups Allows for comparisons across cultures/countries

Internet-based computer adaptive testing (CAT) CAT addresses several common issues: Test Security Cheating Length of Candidate Experience Items drawn from extensive bank; low item exposure rates Different combination of items for each candidate; no single key available to be used for cheating Compared to traditional tests, shorter test time but superior precision

CAT: Development Process Calibration Research 200,000+ candidates for entrylevel professional jobs, globally Items researched via internet delivered test forms Test timing and question functioning developed from response data Ensured the test is inclusive to all candidates globally Then, criterion-related validation study conducted

Results from CAT validation Criterion Validity Coefficient Composite Performance 0.36 (0.29)** Gathering Information 0.18 (0.14)** Reviewing and Analyzing Information 0.34 (0.27)** Decision Making 0.33 (0.26)** Strategic and Operational Agility 0.20 (0.16)** Innovation 0.20 (0.16)** Potential 0.27 (0.21)** Adaptability 0.25 (0.20)** Note. N=596 ** p < 0.01. Validity coefficients have been corrected only for unreliability in the criterion using a reliability estimate of 0.63. Values in parentheses represent uncorrected validity coefficients.

Interview scores by CAT score group 2500 2000 1500 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 RT Next scores step scores with passing with passing RS score CAT RT Next scores Step scores with or without or without RS score passing CAT

Assessment-driven metrics Employer brand index Recruiting effectiveness metrics

Test scores + HRIS data: Turnover survival analysis

Example 2: Tech-facilitated assessment center Key characteristics: Day-in-the-life format Technology driven Live interactions Deployed globally

COMPETENCIES PERSONALITY PATTERNS Karen Gates Vice President, Operations Started: 02.04.2000 Previous Position: Director, Operations Education: MBA, Wharton Business School Known Aspirations: VP of Eastern Region Sample Output Interpersonal Skills P Compelling Communication Leadership Effectiveness Inventory P Cultivating Networks Adjustment 72 S Navigating Politics Ambition 87 D Influence Sociability 25 Business Management D S P Building Organizational Talent Driving Execution Financial Acumen Interpersonal Sensitivity Prudence Inquisitiveness Learning Orientation 88 100 98 95 P P Operational Decision Making Entrepreneurship Leadership Challenge Inventory P Establishing Strategic Direction Volatile 33 Leadership S P S Leading Change Coaching and Developing Others Selling the Vision Argumentative Risk Averse Imperceptive Avoidant Arrogant 10 15 13 24 55 P Empowerment/Delegation Impulsive 11 Personal Competencies Attention Seeking Eccentric 53 69 P Executive Disposition Perfectionistic 24 S Passion for Results Approval Dependent 72

Multi-organization analyses 16,000+ Executives Strategic Leaders Strengths Establishing Strategic Direction Financial Acumen Entrepreneurship Building Talent HRIS Company Profiles Strong Managers Strengths Driving for Results Vary by: Executive Level? Disposition Communicating Industry? with Impact Decision Making Experiences? Customer Personality Focus profile? Prior Assessments Company Performance

Level differences in leadership High Customer Focus Coaching Med Low Empowerment Business Savvy 2 nd Level 3 rd Level 4 th Level

Example using alternative methods 16,000+ Executives Strategic Leaders Strong Managers Vary by: Personality profile?

Machine learning: not your typical PSYCH 650 class

Example: Random Forests Strategic Profile (Y/N)? Lo Ambition Hi Ambition Hi Risk averse Lo Sociability Med/Lo Risk Averse Hi Sociability Med Risk averse Lo Risk Averse

Example results: Random Forests & Logistic Regression Rank Random Forest Logistic Regression 1 Independent Thinker/ Strong Decision Maker Strong Interpersonal Relations Independent Thinker/ Strong Decision Maker 2 Energetic: Drives Self and Others 3 Strong Interpersonal Relations Conflict Averse (-) 4 Strategic/Creative B/W Thinking Style 5 Self-Promoting Emotionally Detached 6 Emotionally Unpredictable Thoughtful/Planful Error rate: 12.5% Error rate: 19% = Similar Rank = Close (Top 10 in opposing model) = Dissimilar Rank

Big Data: Old wine or new opportunity

The Answer: Big data: Old wine and New opportunity (with some significant challenges ahead)

Big Data About People Challenges in practice: Software is often an expensive hollow shell People data can be of poor quality New data sources are not well understood Interpretations can be terribly flawed Managers defer to gut instinct

Big Data: Big Skills Required Complex data analysis and modeling Knowledge of people & systems in organizations Theory building and testing Communication and action planning

Big Data About People Opportunities in practice: Interconnections across steps add new insights and strategic value Strong theory, modeling, hypothesis testing are essential to extract meaning and order from huge complexity Insight about people can be packaged to better inform organizational strategy

Post-SIOP survey: What are you most excited about?

SIOP Taskforce on Big Data Key areas for action: Meaning and definition of big data Training in research methods Theory generation & testing Interdisciplinary linkages Applications to validation practices Legal and ethical issues Education and awareness within SIOP Overcoming resistance

Will this trend leave I-O behind? In the world of Big Data, companies can assess people on real world performance. A flurry of new companies are now skipping I/O psychology and helping recruiters source candidates by analyzing their social data. While a test is a good way to understand someone, so is looking at everything they ve ever posted on the internet. Josh Bersin, Forbes, Oct 2013.

Concluding Thoughts: New opportunities are emerging for I-O and the role of assessment Potential for better people strategy if we can overcome common barriers We have an obligation to respond to the popular trend

Thank you. Thank you! Big Data: Old Wine or New Opportunity? Doug Reynolds February, 2014