Assessing Students and Recent Graduates Assessment Options for your Future Workforce
Pathways for Students and Recent Graduates The Federal Government benefits from a diverse workforce that includes students and recent graduates, who infuse the workplace with their enthusiasm, talents, and unique perspectives. -President Barack Obama, Executive Order 13562 3
Background As a part of the President s Hiring Reform, the Pathways Programs are designed to: Make the Federal Government more attractive to recruit top talent Foster a positive experience for participants that inspires interest in Federal service, and Prepare students and recent graduates for successful Federal careers 4
Topics Pathways and Assessment Assessment Background and Context Student and Recent Graduate Assessment Options Additional Resources and Contacts 5
Pathways Framework The Pathways Programs consist of the: Internship Program Recent Graduates Program Presidential Management Fellows (PMF) Program 6
The Need for Effective Student Hiring Practices Student interns should be one of the primary sources, perhaps the primary source, of entry-level talent in the Federal Government. Federal agencies need to treat student hiring as an important aspect of their overall human capital strategy - Partnership for Public Service (April 2009). Leaving Talent on the Table: The Need To Capitalize on High Performing Student Interns (p. I & 1). 7
The Need for Effective Student Assessment Practices Assessment Requirement: Agencies are required to assess/evaluate the qualifications of applicants for positions covered by Schedule D, Internship Programs under the Pathways Program - OPM Qualification Standard for Pathways Internship Positions, available at: http://www.opm.gov/policydata-oversight/classification-qualifications/generalschedule-qualification-standards/#url=gs-pip 8
Assessment Background & Context 9
Assessment: Defining the Terms Assessment: a systematic approach to gathering information about individuals; this information is used to make employment or career-related decisions about applicants and employees Assessment Tool: any test or procedure (for example, ability test, structured interview, work sample) used to measure an individual s employment or career-related qualifications and interests U.S. Department of Labor, Testing and Assessment: An Employer s Guide to Good Practices, 2000 10
Assessment: Defining the Terms Assessment Strategy: an established plan for designing and implementing one or more assessment tools for an organization, occupation, or a specific situation 11
The Importance of Effective Assessment Increases the likelihood of hiring employees who possess the right competencies and will contribute to agency success Results in hiring, placing, and promoting greater numbers of superior performers Results in reduced turnover 12
The Importance of Effective Assessment Results in cost savings to the organization A poor hire is estimated to cost 2-3 times their salary A good hire results in lower training and performance management costs Decreases the likelihood of a bad hire and the associated consequences, such as: Poor work quality Dissatisfied customers Lowered morale among fellow employees 13
Assessment Considerations Development and implementation costs Administration methods and costs Time and expertise required Applicant acceptance/face validity Potential for adverse impact Reliability and validity evidence Security of test materials For more information about assessment considerations, please visit the Assessment Decision Tool: https://www.opm.gov/policy-data-oversight/assessment- and-selection/assessment-strategy/#url=assessment- Decision-Tool 14
Student and Recent Graduates: Assessment Challenges Limited work experience to assess Individuals with a wide-range of educational backgrounds can apply for the same occupation Large applicant volume for entry-level positions Applicants are not familiar with Federal hiring practices 15
Student and Recent Graduate Assessment Options 16
Definition of Terms Used Subject Matter Expert (SME) A person with bona fide expert knowledge about what it takes to do a particular job Face Validity An applicant's perception of how valid a measure is based on simple visual inspection Additional information can be found in OPM s Assessment Decision Guide, available at: http://www.opm.gov/policy-dataoversight/assessment-and-selection/referencematerials/assessmentdecisionguide.pdf 17
Biographical Data Questionnaire (Biodata) Items about past events and behaviors that predict overall performance for a given occupation Items scored using a research-based scoring key Strengths Can be administered and scored electronically in a non-proctored setting High validity Low adverse impact Ideal for entry-level jobs Considerations Requires SME involvement and measurement expertise to develop Somewhat low face validity for applicants Can be costly to develop 18
Biodata: Examples This questionnaire asks you to describe your educational, work, and other experiences in specific areas which relate to successful job performance. My peers would probably describe me as being: A. much more confident than most B. somewhat more confident than most C. about as confident as anyone else D. somewhat less confident than most E. much less confident than most During high school (grades 9-12) I made the semester honor roll: A. never B. once or twice C. three or four times D. five or six times E. seven or eight times 19
Cognitive Ability Test Applicants are asked to solve questions to estimate their potential to use mental processes to solve jobrelated problems or acquire job knowledge; can also measure aptitude for job-related skills Strengths Can be administered and scored electronically High validity Can be used for entry-level positions Considerations Security of test materials Can adversely impact minority groups Requires measurement expertise to develop 20
Cognitive Ability Test: Example The bus fare is increasing by 10%. The old bus fare was $20. What will the new fare be? A. $20 B. $21 C. $22 D. $23 21
Personality Test Multiple choice questions used to generate a profile of traits which can predict job performance or satisfaction with certain aspects of work High validity Strengths Can be administered and scored electronically in a non-proctored setting Can be used for a wide range of occupations Considerations Must be linked to jobrelevant information Requires measurement expertise to develop and interpret results Can have low face validity for applicants; items can be perceived as intrusive 22
Personality Test: Example I find it easy to see things from someone else s point of view. A. Strongly agree B. Somewhat agree C. Neither agree or disagree D. Somewhat disagree E. Strongly disagree 23
Situational Judgment Test Presents applicants with a description of a work problem or critical situation, and asks them to identify how they would deal with it (can be paper, computerized, or video-based format) Strengths High validity and face validity for applicants Can be administered and scored electronically Can use video-based scenarios Considerations Requires SME involvement and measurement expertise to develop Security of test materials High development costs 24
Situational Judgment Test: Example No special training or experience is required to answer these questions. Identify from the response options the one that best represents how you would handle the situation. You are a member of a project team. Another member of the team gives you a task that you do not feel qualified or trained to handle. What would you do? A. Complete the task as best you can. B. Consult with your supervisor. C. Ask someone in the office who knows how to do the task to help you. D. Ask the staff member to reassign the task to a more qualified team member. 25
Writing Assessment Applicants receive a writing prompt and have a set amount of time to write about the topic Scored by raters using professionallydeveloped benchmarks or by complex automated protocols Strengths High validity and face validity for applicants Multiple ratings/consensus leads to increased reliability Can be administered electronically Considerations Requires SME involvement and measurement expertise to develop scoring benchmarks Resource intensive rating process Security of test materials 26
Writing Assessment: Example Previously, employees were able to telework on Mondays and Fridays, but not on Tuesdays, Wednesdays or Thursdays. Employees will now be allowed to telework any day except for Wednesdays. Compose an internal memo informing coworkers of this change in the office s telework policy. 27
Occupational Questionnaire Task- and competency-based self-report questionnaires used to screen and rank applicants based on their training and experience Strengths Can be administered and scored electronically in a nonproctored setting High face validity and convenience for applicants Wide variety of competencies can be assessed Low adverse impact Considerations Requires subject matter expert (SME) involvement to develop Applicant response inflation must be addressed Development more challenging for entry-level and unspecialized positions Potential for lack of distinction among applicants 28
Occupational Questionnaire: Example Teamwork: Encourages and facilitates cooperation, pride, trust, and group identity; fosters commitment and team spirit; works with others to achieve goals. Select the one level that most accurately reflects your training and experience with respect to the competency definition. A - I have had no training or experience in performing this activity. B - I have participated in team activities by contributing as a team member. C - I have led group discussions and established priorities in order to carry out one or more team assignments. D - I have directed and monitored the activities of a team and taken responsibility for attaining team goals. 29
Accomplishment Record Applicants provide a written description of a situation to illustrate their proficiency in critical job-related competencies Evaluated by a panel of trained raters against competency-based benchmarks Strengths High face validity for applicants Multiple ratings/consensus leads to increased reliability Can be administered electronically in a nonproctored setting Considerations Cannot be collected at the time of initial application Resource intensive rating process Results should be verified at a later stage in the selection process 30
Accomplishment Record: Example Analytical Reasoning is critical for the Program Analyst position. Analytical Reasoning is defined as Using the text box below, please describe one specific instance from your training and/or experience in which you demonstrated experience in Analytical Reasoning. In your response, please follow these important guidelines as you write your accomplishment: - Describe the situation -- i.e., the challenge faced, the problem to be solved... - Describe the specific actions you took - State the outcome, results, or long-term impact of your accomplishment Provide the name and a telephone number or email address or other method of contacting someone who can verify this information... 31
Structured Interview Panel interview where all candidates are asked the same job-related questions (often competency-and behavioral-based) Interviewers use detailed rating scales, evaluating all candidates according to the same standards Strengths High validity and face validity for applicants Multiple ratings/consensus leads to increased reliability Legally defensible compared to unstructured, conventional interviews Considerations Requires SME involvement to develop Resource intensive administration and rating process Assessors must be trained in conducting and rating the interviews 32
Structured Interview: Example Question Competency: Interpersonal Skills - Shows understanding, friendliness, courtesy, tact, empathy, concern, and politeness to others; develops and maintains effective relationships with others; may include effectively dealing with individuals who are difficult, hostile, or distressed; relates well to people from varied backgrounds and different situations; is sensitive to cultural diversity, race, gender, disabilities, and other individual differences. Lead Question: Describe a situation in which you dealt with individuals who were difficult, hostile, or distressed. Probing Questions: Who was involved? What specific actions did you take? What was the outcome? 33
Structured Interview: Example Rating Scales Proficiency Level Benchmark Definition 1-Low 2 3-Average 4 5-Outstanding Handles interpersonal situations involving little or no tension or discomfort... Handles interpersonal situations involving a moderate degree of tension or discomfort... Handles interpersonal situations involving a high degree of tension or discomfort... Example 1 Example 2 Example 3 Example 4 Example 5 Behavioral Example Refers employees to the appropriate staff member... Works with others on a crossfunctional team Establishes cooperative working relationships with managers... Facilitates an open forum to discuss employee concerns Diffuses an emotionally charged meeting with external stakeholders... 34
Assessment Center Consists of multiple assessments to evaluate small groups of applicants on a variety of job-related competencies Designed to resemble actual challenges encountered on the job Applicant performance is observed and evaluated by multiple trained assessors Strengths High validity and face validity for applicants Multiple ratings/consensus across multiple assessments leads to increased reliability Low adverse impact Considerations Requires SME involvement and measurement expertise to develop Resource intensive administration and rating process Not suitable for an initial screen or large numbers of applicants 35
Assessment Center: Example You are to assume the role of a new employee in XYZ Fictitious Agency. Today is your first day Your schedule today will include: Orientation Individual in-basket exercise Role-play exercise Group interaction Writing exercise 36
Student Assessment Options Less appropriate options Job Knowledge Tests - consist of questions designed to assess technical or professional expertise in specific knowledge areas Work Samples - require applicants to perform tasks or work activities that mirror the tasks employees perform on the job These assessments are designed to measure specific job knowledge not required at the entry-level. 37
Assessment Resources For more information, please visit OPM s Assessment and Selection website at: http://www.opm.gov/policy-dataoversight/assessment-and-selection/ 38
Contact Classification and Assessment Policy U.S. Office of Personnel Management Assessment_Information@opm.gov (202) 606-3600 39