Nonrandom Response and Rater Variance in Job Analysis Surveys: A Cause for Concern?

Transcription

1 Nonrandom Response and Rater Variance in Job Analysis Surveys: A Cause for Concern? By Thomas A. Stetz, PhD, J. Mathew Beaubien, PhD, Michael J. Keeney, PhD, and Brian D. Lyons, PhD When conducting a job analysis, one of the most prevalent strategies for estabiishing content validity is using an Incumbent survey. A major issue related to surveys is the possibiiity of receiving a iow rate of response. To further the job anaiysis iiterature, this articie reports on a study conducted to examine the characteristics reiated to nonrandom response and rater variance in job analysis surveys. Job analytic surveys were randomly distributed to incumbents in 21 different jobs in a U.S. federal government agency. A total of 150 out of 349 (43%) surveys were completed and submitted for further anaiysis. Resuits indicated that race, job performance, and empioyee grade were significantly related to survey return, and education was reiated to rater variance. The practical implications of these findings and directions for future research are discussed. Job analyses should provide the foundation for all HR management activities.' Job descriptions, consisting of samples of incumbent behaviors, provide the data to describe work conditions, work processes, and worker requirements. Employers use these descriptions when making decisions regarding selection, promotion, and compensation. Because employers, as well as the courts during employment-related litigation, base so many critical decisions on job analysis data, it is essential that the data provide accurate descriptions of the work an incumbent must perform. Survey instruments are in widespread use and are possibly the single most frequently used means of collecting job analysis data. Surveys are rarely administered to all incumbents, supervisors, and other job experts. Furthermore, a 100% completion rate for any survey is highly unlikely. This means survey results are almost always based on a subset a sample of the population. The degree to which these samples are accurate representations of the intended population is crucial to the quality of job analytic data. Given the widespread use of surveys' in job analysis research and the importance of the quality of research samples, it would seem that issues surrounding survey methodology would be major concerns for job analysts. We, however, were unable to locate any published studies in the job analysis research literature that specifically address the issue of survey response rates and sample representativeness. Pubiic Personnel Management Volume 37 No. 2 Summer

2 The research available on surveys shows that organizational survey response rates are often low and have declined in recent years.^ Unfortunately, this appears to be true for job analysis surveys, as well. To illustrate, we performed a small-scale review of 27 published job analysis studies (see Table 1). We found that response rates ranged from 21% to 96%, with a weighted mean response rate of 43%. Poor face validity and lack of acceptance many job analyses could be the result of low survey response rates. However, in technical terms, it is not only the response rate that is important. The sample contacted to complete the survey and/or who returned completed surveys must be representative of the population in all ways that could affect ratings. In the past, U.S. courts have been critical of the racial and gender composition of subject matter expert (SME) panels.' The logic being that the results from a panel made up of male job experts might be biased against women. Table 1: Sample Size and Response Rates for Published Job Analysis Studies Study Sample Size Number Returned Return Rate Allen-Meares, R (1994). Social work services in schools: A national study of entry-level tasks. Social Work, 39(5), ,257.1,117 49% Anderson, G. S., Plecas, D., & Segger, I (2001). Police officer physical ability testing: Revalidating a selection criterion. Policing, 24(1), % Barnesteiner, J. H., Wyatt, J. S., & Richardson, V. (2002). What do,pediatric nurses do? Results of the role delineation study in Canada and the United States. Pediatrio Nursing, 28(2), , % Case, R., & Branch, J. D. (2003). A study to examine the job competencies of sport facility managers, international Sports Journal, 7(2), % Corrigan, P W., Hess, L., & Garman, A. N. (1998). Results of a job analysis of psychologists working in state hospitals. Journal of Ciinicai Psychology, 54(1), % Dolan, J. (2000). Influencing policy at the top of the federal bureaucracy; A comparison career and political senior executives. Public Administration Review, 60(6), , % 224 Public Personnel Management Volume 37 No. 2 Summer 2008

3 study Sample Size Number Returned Return Rate Earles, J. A., Driskill, W. E., & Dittmar, M. J. (1996). Methodology for identifying abilities for job classification: An application of job analysis. Military Psychology, 8(3), % Fitszgerald, L. R, & Osipow, S. H. (1986). An occupational analysis of counseling psychology. American Psychologist, 47(5), % Ford, K. J., Smith, E. M., Sego, D. J., & Quiiiones, M. A. (1993). Impact of task experience and individual factors on training-emphasis ratings. Journal of Applied Psychology, 78(4), ' % Green, S. B., & Stutzman, I (1986). An evaluation of methods to select respondents to structured job-analysis questionnaires. Personnel Psychology, 39(3), % Griffin-Shirley, N., McGregor, D, & Jacobson, W. H. (1999). Survey of dual-certified orientation and mobility instructors. Journal of Visual Impairment and Blindness, 93(3), , % Harris-Davis, E., & Haughton, B. (2000). Model for multicultural nutrition counseling competencies. Journal of the American Dietetic Association, 700(10), % Holliman, D., Dziegielewski, S. R, & Teare, R. (2003). Differences and similarities between social work and nurse discharge planners. Health and Social Work, 28(3), % Hromoco, J. G., Lyons, J. S., & Nikkei, R. E. (1995). Mental health case management: Characteristics, job function, and occupational stress. Community Mental Health Journal, 37(2), % Kim, S., & Choi, M. (2002). Educational requirement analysis for information security professional in Korea. Journal of Information Systems, 73(3), % Leahy, M. J., & Wright, G. N. (1988, Winter). Professional competencies of the vocational evaluator. Vocational Evaluation and Work Adjustment Bulletin, ,614. 1,274 35% Public Personnel [Management Volume 37 No. 2 Summer

4 study Sample Size Number Returned Return Rate Lindell, M. K., Clause, C. S., Brandt, C. J., & Landis, R. S. (1998). Reiationship between organizational context and job analysis task ratings. Journal of Applied Psyohology, 83(5), %' McKillip, J. (2001). Case studies in job anaiysis and training evaluation. InternationalJournal of Training and Development, 5(4), ; 1, % Perdue, J., Woods, R., & Ninemeier, J. (2001). Competencies required of future club managers' successes. Cornell Hotel and Restaurant Administration Quarterly, 42(1), , % Robinson-Kurpius, S. E., Fuqua, D. R., Gibson, G., Kurpius, D. J., & Froehle, I C. (1995). An occupational analysis of consulting psychology Results of a national survey. Consulting Psyohology Journal, 47(2), % Schipmann, J. S., Smalley, M. D., Vinchur, A. J., & Prien, E. P (1988). Using structured muitidomain job analysis to deveiop training and evaluation specifications for ciinicai psychologists. Professionai Psyohology: Research and Praotice, 79(2), % Schmitt, N., & Cohen, S. A. (1989). Internai analysis of task ratings by job incumbents. Journal of Applied Psyohology, 74(1), % Scorzelli, J. F. (1995). The use of a job task ' analysis in assessing the perceptions of human resource managers on the role of the rehabilitation counselor in industry. Journal of Applied Rehabilitation Counseling, 26(1), % Silvester, J., Patterson, F, & Ferguson, E. (2003). Comparing two attributional modeis of job performance in retail sales: A field Study. Journai of Oocupational and Organizational Psyohology 76(1), % Tannenbaum, R. J., & Wesley, S. (1993). Agreement between committee-based and field-based job analyses: A study in the context of licensure testing. Journal of Applied Psyohoiogy 78(6), , % Wiener, W. R., & Siffermann, E. (2000). Development of the orientation and mobility certification examination. Journal of Visual Impairment & Blindness, 94(8), % 226 Public Personnel Management Volume 37 No. 2 Summer 2008

5 study Sample Size Number Returned Return Rate Wooten, W. (1993). Using knowledge, skill and ability (KSA) data to identify career pathing opportunities: An application of job analysis to internal manpower planning..pub//c Personnel Management, 22(4), % Total 21,137 9,071 43%. High 96% Low 21% Rather than focusing on applied practical questions such as the degree to which samples are representative of the intended population, job analysis researchers have typically focused their attention on the technical issues of suia^ey reliability, survey validity, and sources of rater variance.^ There is a growing body of research examining sources of rater variance in job analysis surveys that shows that, despite some arguments to the contrary, job analysis ratings can vary systematically as a function of rater characteristics.' Recent studies show that job analysis ratings can vary as a function of rater position, performance level, task engagement, job satisfaction, tenure, and race.^ Even if these effects are small and inconsistent, they can still be important, as well as informative.^ Moreover, small effects could be exacerbated if the rater characteristics that are related to survey ratings are also related to survey return. For example, suppose age is related to both survey return, with older workers less likely to respond to job analysis surveys, and to job analysis importance ratings, with older workers more likely to deflate the importance of technical competencies. The sample would then consist primarily of younger workers who, in comparison to older workers, tend to rate technical competencies as more important. Thus, these two effects combined would produce much more measurement error than either one individually In summary, there is insufficient research to understand the effects of response rate on job analysis results. Nonrandom responding can have important legal, technical, and practical consequences. Furthermore, nonrandom responding may help to explain inconsistent findings concerning sources of rater variance. For these reasons, we investigated how characteristics of respondents are related to survey return and how this affects job analysis survey responses. We looked at the relationship between protected class status (i.e., gender, race, disability, and age over 40 years) and job analysis survey return. Protected class status is important because the final analyzable data set represents the population of Public Personnel ivianagement Volume 37 No. 2 Summer

6 employees, and the failure to provide representative responses is frowned upon by the courts. Research for other types of surveys indicates that the relationship between nonresponse and age, race, and gender have been mixed, with some studies finding relationships and others not finding relationships.^ Therefore, we offered no specific hypotheses for these groups on job analysis survey responses. We were unable to locate any studies that looked at disability status and survey return rates. As a practical matter, however, having a disability may make it more difficult to reply to a survey For example, individuals with disabilities may be on disability leave, or they may not receive the survey in a manner that easily, accommodates their disability Even though laws require that individuals with disabilities must be accommodated in the workplace, we believe that such workers will be less likely to respond to a job analysis survey Other Worker Characteristics This constellation of variables includes the worker characteristics other than protected class status job performance, tenure, and education level that describe job-relevant traits of an individual employee. First, we utilized job performance because previous research has shown that performance is related to survey return, and that performance accounts for a significant portion of rater variance in ratings.^ Thus, we believe that performance will be positively related to survey return and rater variance. We also sought to determine the infiuence of tenure on survey return. Research has found that years in present job and years with the company were not related to the intention of responding to a survey ^ However, this research utilized an employee attitude survey. Consequently, it is unclear if the findings generalize to job analysis surveys and actual employee behavior. The final worker characteristic we use is employee education level. In a review of survey nonresponse, it was concluded that education level is one of the most consistent differentiators between respondents and nonrespondents." Respondents tend to be better educated than nonrespondents. Thus, we believe that education level will be positively related to survey return. Work Characteristics This final category of variables includes characteristics of the work or the particular position that the job incumbent asked to respond to a job analysis survey holds. Three attributes were included in the analysis employee job level, supervisor/nonsupervisor status, and location. Research shows that job level and level of supervisory responsibility are not related to survey return or compliance.'^ Again, however, this research was conducted using employee attitude surveys, and it is unclear if the findings generalize to job analysis surveys. Fmployee location, the final attribute, is commonly a determining factor for whom receives a job analysis survey, and there is always the question if jobs vary by location. This is a real possibility, given that different locations often have different work systems, equipment, and standard operating procedures. Therefore, it is important to obtain a geographically representative sample when conducting a job 228 Public Personnel Management Volume37 No. 2.Summer 2008

7 analysis. To date, however, no research has examined response rate as a function of respondent location; as a result, we sought to ascertain this characteristic's influence on survey return compliance. As stated earlier, published research findings are inconsistent regarding rater variance and job analysis ratings. However, there appears to be a new trend in the literature toward the detecting and reporting of systematic variance in job analysis ratings. Recent studies show that job analysis ratings can vary as a function of rater position, performance level, task engagement, job satisfaction, tenure, and race.'^ We had two interests concerning rater variance. First, we wanted to determine if protected class status, other worker characteristics, or work characteristics are related to job analysis ratings. Second, we wanted to determine the impact that nonresponse has on the identification of the sources of rater variance. Data for this research were collected as part of a periodic job analysis update for a large, U.S. Federal agency The first step ofthe project involved having SME panels review and update the competencies (i.e., skills, knowledge, and abilities to use job-related tools) that were associated with each job. During this review, the number of competencies per job was limited to 60. Some jobs had fewer than 60 competencies, but none had more. After the SME panels completed their reviews, a team of job analysis professionals created a unique job analysis survey for each job. The first section ofthe survey assessed the respondent's demographic characteristics, such as the respondent's name (for tracking purposes), race, gender, and years of work experience. The second section assessed the clegree to which respondents believed each competency was associated with their job. The competencies were organized into three subgroups skill requirements, knowledge requirements, and tool requirements. The participants rated each competency on four scales. The first scale asked for a yes or no response to the question of whether an employee used that particular competency on the job. The remaining scales used a five-point rating on which "1" = strongly disagree and "5" = strongly agree. The second scale assessed each competency's importance for successful job performance. The third scale assessed each competency's importance in the hiring decision. The fourth scale assessed the amount of time spent using the competency A small representative sample of incumbents was purposively selected for each job to receive the survey However, if the total number of incumbents was less than 20, the entire population was surveyed. Surveys were distributed to specific points-ofcontact (POC) in the organizations that assisted in the previously described SME review. Each job has an advisory board whose members are responsible for overseeing the maintenance of job descriptions and the career development of its incumbents. Public Personnel Management Volume 37 No. 2 Summer

8 POCs were generally senior members of these boards and, therefore, had a certain level of organizational visibility and authority. Their endorsement of the job analysis survey was critical to the success of this research. Organizational POCs were responsible for distributing surveys and returning completed surveys to the job analysts. Follow-up reminders were also used to increase the response rate. However, the strategies of POCs varied. Participants During this project, we analyzed more than 150 unique jobs. Because most ofthe jobs had a small number of incumbents, we limited our analyses to those that had 10 or more incumbents. This resulted in sample of 21 unique jobs. Table 2 shows that the Table 2: Titles of Jobs, Number of Surveys Sent, and Response Rates Work Role Name Surveys Sent Surveys Returned Response Rate Academic administrator Technical liaison Engineering.technician Facilities engineer Finance & accounting analyst Earth scientist 1 Earth scientist 2 Technical training instructor Hr consultant Information resources officer Information systems security specialist Librarian Safety data analyst Piiysical security specialist Program manager Senior financial manager Software engineer System performance, monitor Linguist Training & education coordinator/instructor Web author Total Note. Tliree surveys were returned without names and were deleted from further analyses. The total response rate was 44%, but the analyzable response rate was 43% % 42% 31% 31% 42% 50% 52% 29% 33% 46% 36% 36% 39% 60% 38% 90% 47% 40% 33% 55% 80% 43% 230 Public Personnel Management Volume 37 No. 2 Summer 2008

9 response rate varied among the jobs surveyed from 29% to 90%. A total of 349 surveys were sent, and 150 were completed and returned for an overall response rate of 43%. Because the survey asked the respondents to provide their name, we were able to compare the responders and nonresponders along a variety of characteristics. Background data was obtained from the organization's Human Resources Information System (HRIS). After linking the survey and HRIS data, the resulting data we used for analysis was deidentified to protect participants' identities. Measures Protected Class Status We collected background data from the HRIS on four protected class status measures gender, race, disability, and age over 40 years. Because of the small sample sizes for various minority groups, we were only able to examine statistical differences between Whites and African Americans. All four variables were coded "0" for the majority group and "1" for the protected class. Other Worker Characteristics We constructed a composite performance measure from the employees' most recent performance evaluation score, amount of bonus, and if the individual was recommended by his or her supervisor for a promotion. To ensure that the variables were equally weighted in the new composite measure, the three variables were standardized, and the mean of the three variables was taken as an overall measure of job performance. We also measured. education using five levels high school or equivalent, some college, bachelor's degree, master's degree, doctorate degree. Finally, we determined the years of U.S. government service. All data were obtained from the HRIS. Work Characteristics Three work characteristic measures were assessed supervisory level, pay grade, and location. The supervisory level was coded "1" for supervisor and "0" for nonsupervisor. Pay grade was assessed using the organization's current pay grade system, with "1" being the lowest and "5" being the highest. In regards to location, the organizations whose employees were surveyed are predominately situated in two major metropolitan locations, so location was subsequently coded dichotomously All data were obtained from the agency's HRIS. Data Analysis To answer questions concerning survey nonreturn, we used a simple correlational approach. The bivariate correlations between survey return and the four protected class status variables, the three' other worker characteristics variables, and the three work characteristics variables were examined. Because the job analysis was performed at the job level, each survey had a different set of competencies. As a result, there were Public Personnel Management Volume 37 No. 2 Summer

10 198 competencies across the 21 surveys used in this study Therefore, determining if protected class status, other worker characteristics, and work characteristics were related to ratings of the different competencies required a more complicated approach. Two of the authors reviewed the list of competencies and discussed the classification options. Both authors were familiar with the entire organization, the competencies, and the jobs and had extensive experience conducting job analyses. The result was six different categories, or types, of competencies interpersonal/oral communication skills, customer service skills, written communication skills, computer skills, management and administration skills, and technical skills. Subsequently, the same two authors independently placed each competency into a single category. They agreed on the classification for 138 of the competencies, or 70% of the time. They discussed each of the 60 discrepancies and came to an agreement on the most appropriate category for each. Next, we computed the mean aggregated response by respondent and competency type. For example, if an individual received a job analysis survey with six technical competencies, we computed the average rating across all six competencies and across the three rating scales of importance, needed at hire, and frequency Thus, the respondent had a single score describing the overall importance of technical competencies for his or. her job. Extending this approach to each competency type meant that each respondent had six scores, with each score describing the overall importance of each individual competency type. If an individual received a job analysis survey that did not contain any competencies of a certain type, a score of "1" (not important) was assigned by us for purposes of analysis. This strategy gave us a reasonable number of dependent variables (i.e., six rather than 198) to use in our analysis. We then looked at the correlation between the 10 predictor variables and the overall ratings for each competency type. Results Survey Return Table 3 shows that the three variables of race, job performance, and pay grade were significantly correlated with returning a completed survey Concerning protected class status, disability and education level were not significantly related to survey return. Race, however, was negatively correlated with survey return (r = -.18, p <.01), indicating that Whites were more likely to return the surveys than African Americans. K post hoc analysis revealed that Whites had a higher response rate than did African Americans (48% to 26%, respectively). Of the other worker characteristics variables, only job performance was positively correlated with survey return (r =.11,/? <.01). To further explore this finding, we examined the response rate between high performers (individuals who were greater than one standard deviation [SD] above the mean) and low performers (individuals who were greater than one SD below the mean). Results of this analysis indicated that 232 Public Personnel Management Volume 37 No. 2 Summer 2008

11 57% of high performers responded, while 36% of low performers responded. Education and years of service were not related to survey return. Finally, one variable from the work characteristics group was statistically significant. Employee grade was positively correlated with returning the survey (r =.14, p <.01). Higher level employees were more likely to return the survey than were lower level employees. Analogous to. the previous two post hoc analyses, we examined the return rate for two specific groups of individuals. Grade "1" (the lowest grade) had a return rate of 33%, while the grade "5" (the highest grade) had the highest return rate of 68%. Supervisory level and location were not related to survey return. TaIb)Oe 3s Oes &DV Sliafti S aitdc a Go irireda Moms ^mr ^apoatoles Variable N Mean SD Retumed 348, Gender 348, Disability 348,19, ,07 4 Over , *.04 5 Race , **,29** * 6 Education ,04,03 -,27** ** 7 Performance 331,01,83.11**, * ,08 8 Years Service , ,09 -,11**.18**.34** -,18** -;oi -,13* 9 Job Grade 10 Location 11 Supervisor * p <.05, **p < ,29.40, ,14**,09 -, **,02 -,15** *.03.17**,11* -,21** -.17** -.01,46** * **,00,34** -, **.25** -.08 table 4 depicts the bivariate correlations between the variables of interest (i.e., protected class status, other worker characteristics, and work characteristics) and the ratings for each competency type. In general, education appeared to be the most related to competency type ratings. Specifically, education level significantly correlated with ratings for four of the six competency types. The significant correlations involving education were negative, indicating that higher educated individuals, in comparison with lower educated individuals, tend to rate interpersonal skills (r = -.24, p <.01), customer service skills (r = -.28, p <.01), management and administration skills (r = -.24, p <.01), and technical skills (r = -.34, p <.01) as less important. Furthermore, gender was significantly correlated with ratings for the competency types of written communication skills (r = -.18, p <.05) and computer skills (r = -.33, p <.01). These two competency types were the only two that were not significantly related to education. Performance was significantly related to ratings of the single competency type of written communication skills (r =.18, p <.05). Years of service (r =.16,p <.05), job grade (r = -.22,p <.01), and location (r =.21,p <.05) were Public Personnel ivianagement Volume 37 No. 2 Summer

12 significantly related to ratings of customer service skills. Age over 40 years was significantly related to computer skills (r = -.18,p <,05), Table 4: Bivariate Correlations Between Protected Class Status, Worker Charaoteristics, and Work Characteristic Variables and Competency Ratings Variable Interpersonal Customer Service Written Communication Computer Management & Administration Technical Gender -,02 -,05 -,18* -,33**. -,12 -,07 Disability -,01 -,01 -,11 -,07 -,16,04 Over 40,06,11 -,02 -,18*,04 -,03 Race,05,10 -,12,05 -,11,07 Education -,24** -,28** -,14 -,05 -,24** -,34** Performance,07 -,06,18* -,09,13,14 Years Service -,02,16*.07,10 -,11,02 Job Grade,14 -,22**,07 -,05,08, -,04 Location,04,21**,12 -,02,03,03 Supervisor,04. -,12,05 -,04,11,08 * p <,05, ** p <,01 Another goal of this research was to determine how nonrandom survey return influenced job analysis competency ratings. In technical terms, nonrandom survey return results in a reduction of variance (range restriction) for the affected variables. When the variance of a variable is less than the population variance, the sample correlation coefficient underestimates the population coefficient. In other words, nonrandom survey return can reduce the size ofthe correlation between a background characteristic and the job analysis competency rating, making it appear that the background variable is not related to the rating. The three variables of race, job grade, and performance were significantly related to survey return. Thus, these variables had a certain amount of range restriction and there is a potential to underestimate the relationship between them and the competency ratings. We applied a statistical correction and compared the corrected and uncorrected correlations between race, job grade, and performance and the competency ratings," The corrected correlations were almost identical to the uncorrected correlations. Thus, our results indicate that range restriction that results from nonrandom survey return did not mask rater effects. Discussion Our results indicate that nonrandom response can be a problem in job analysis research. Most notable is the effect that race had on returning a completed job analysis questionnaire. The statistically significant correlation between race and survey return 234 Public Personnel Management Volume 37 No, 2 Summer 2008

13 indicated that White incumbents were more likely to return a completed survey than were African American incumbents. The correlation between race and survey return was,18, which is considered small,'5 gyen though the correlation was small, it had a sizable practical effect. Post hoc analysis revealed that Whites had a response rate of 4896, whereas African Americans had a response rate of only 26%, This result indicates that when using a simple random sampling strategy, the returned job analysis surveys will not be racially representative of the population. To illustrate, further examination of the data showed that the final analyzable data, which is the type that would be of most interest to courts, was 88% White and 12% African American, which is quite discernable from the initial population of 80% White and 20% African American, This finding raises the question of how a racially balanced final sample can be obtained. One solution is to differentially sample each group. Because there are generally fewer African Americans in the population, it may be conceivable to take a census of African American job incumbents and randomly select a sample of 54% of the White incumbents. This could be calculated as follows: For White Incumbents 54% X 80 incumbents = 43,2 sampled Then, 43,2 x 48% response rate = 20 surveys returned For African American Incumbents: 20 incumbents x 26% response rate = 5.2 surveys returned Therefore, the final data set would reflect 20 returned surveys from Whites and approximately five returned surveys from African Americans, which would be a racially balanced final analyzable data set. Another solution would be to better track respondents and use follow-up contacts to increase the overall response rate. However, this would not necessarily ensure that a representative sample would be obtained. Even with follow-up contacts, response rates for Whites and African Americans could be different. It is entirely possible to have an overall response rate of 70% and have an absence of minority responses if the minority population within the organization is particularly small. An alternative is to make responding to a survey mandatory, with significant consequences for noncompliance. While this approach is uncommon for employee attitudinal surveys, we believe that job analysis surveys are fundamentally different. Participation in a focus group or an interview for job analysis study is much more strictly enforced than participation in the survey portion of a job analysis. Survey completion could be considered an "other duty as assigned," Supervisors assign employees all sorts of tasks. There is no reason why answering an organizationally relevant survey should not be considered an appropriate one. In cases where survey compliance is mandatory, however, researchers should be more concerned with the psychometric quality (e,g,, demand characteristics, missing data) ofthe returned data. The finding that job performance is related to job analysis survey return raises the possibility of an interesting predicament. Although our analysis did not find that job Public Personnel Management Volume 37 No, 2 Summer

14 performance was related to job analysis ratings, other researchers found job performance to be related to job analysis ratings.^^ If our results had shown a relationship between survey return and job analysis ratings, the job analysis would predominately describe what high performers believe are important for success. Furthermore, what high performers believed to be important would be different from lower performers. This effect is consistent with the popular competency modeling approach advocated by Spencer and Spencer that describes what is required for superior performance." Job analysis, however, is often conceptualized as describing what is typically done on the job. It is then used to set specifications of what a worker needs to know and be able to do to adequately perform the work. If these specifications are set too high, individuals who could successfully perform the job would be rejected. In addition, setting specifications too high could result in greater adverse impact if the job analysis survey results were used to establish minimum requirements for selection and promotion decisions. Even though our results do not suggest that this scenario is an issue for our data set, job analysts should be aware of the possibility and regularly inspect their data and take appropriate actions to ensure that their results meet the job analysis objectives. The finding that employee grade is related to returning a completed job analysis competency survey suggests that a one-size-fits-all sampling strategy is inappropriate, when job analysts are surveying jobs that are primarily filled with lower level employees, the analysts should expect a lower response rate than typical for jobs filled with higher level employees. The job analysts may wish to use a larger percentage sample for low-level jobs to maintain the final sample size. When a job has incumbents spanning the entire gamut of employee levels, job analysts should examine the final sample to ensure that the final analyzable sample is representative of the entire range of employee levels. Moreover, the analysts may want to examine if employee level is related to rater variance. Of our 10 variables of interest, only education level appeared to have substantial effects on job analysis competency ratings. Education was significantly related to ratings of four of the six competency types. Interestingly, the relationship was negative, indicating that as respondents' education increased, the importance of interpersonal skills, customer service skills; management and administration skills, and technical skills decreased. We suggest that lower educated respondents may have intentionally infiated their ratings in an attempt to increase the prestige of their jobs. This is merely conjecture, however, and follow-up research is needed to further examine this result. We offer a final suggestion concerning improving the sample quality of job analysis surveys. This suggestion may sound counterintuitive at first: Perhaps selecting the smallest possible random sample so that the job analyst' can more efficiently and effectively target follow-up resources is the best survey approach. Then use robust survey strategies (e.g., personalized or group administration), and aggressively pursue the data from every member of the sample (e.g., follow-ups, identification numbers). Finally, it may sometimes be necessary to select substitute survey subjects who are as 236 Public Personnel Management Volume 37 No. 2 Summer 2008

15 similar as possible to the original sample member when it is not possible to get data from the original sample member. Future Research Because of the lack of research concerning response rates for job analysis surveys, we offer many suggestions for future research. First, future research should examine the generalizability of our findings to other organizations and other contexts. The majority of the published research on response rates tends to focus on attitudinal surveys, and it is unclear how well our findings will apply to job analysis surveys.^ We learned several interesting things during our review of published job analysis articles that included response rate information. Very few job analysis research articles publish information on response rates. This is surprising considering population, sampling, and response rate information is so commonly included in other types of survey research. Most of the articles that included response rate information were published in functional or practitioner journals; very few technical job analysis research articles included information relating to sampling and response rate. It should also be noted that of the articles with response rate information sampling strategies, surveys, populations, and other such types of information varied greatly. It would be most beneficial for practitioners to have practical guidelines describing commonly experienced response rates for job analysis surveys. Thus, a meta-analysis including published job analysis articles and technical reports would be most beneficial. There are thousands of job analysis reports, but the majority of these reports are stored as technical reports in consulting companies. Getting access to a large sample may be difficult. However, if a good cross-sectional sample could be obtained, a meta-analysis could answer important questions regarding ways to increase response rates and would allow practitioners a way to estimate expected response rates u nder different conditions. Future research should focus on how to increase response rate for job analysis surveys. Research shows that the strongest correlate of survey noncompliance for an employee attitude survey is the belief that the organization would not act on the data and provide feedback to employees.^' Because the results of job analysis are mostly used for technical HR purposes, the results often do not get directly communicated back to individual employees. To combat this, practitioners may want to stress the organizational and personal importance of the survey to achieve buy-in and, as a result, increase job analysis survey response rates. Similarly, it would be particularly interesting to examine which has the strongest effect on response organizational importance or personal importance. In addition, past research has found that advance notice, identification numbers, follow-up reminders, and salience are significantly associated with response rate to or^ganizational surveys.2 Future research could also evaluate the use of these strategies for job analysis surveys. Additional research could also examine the relationship between organizational citizenship behaviors (OCBs) and replying to surveys. Survey completion has been considered an extrarole, or OCB, task.^^ Analogous to the results of this study, research has found that OCBs are related to performance and race.22 Consequently, further Public Personnei ivianagement Volume 37 No. 2 Summer

16 exploring this relationship in a job analysis situation may produce useful insights to job analysis survey return. Finally, it would be interesting to examine the systematic differences between attitudinal surveys and job analysis surveys. In particular, job analysis surveys do not normally inquire about personally sensitive issues, which may cause response distortion. However, incumbents often suspect that there are ulterior motives for collecting the data. For example, employees may suspect that the data will be used to lower pay or to implement greater outsourcing. Further investigation of employees' beliefs about the uses of job analysis data and its relationship to response rate may be useful. Conclusion In the title of this paper, we asked if nonrandom response in job analysis surveys is a cause for concern. According to the published literature on survey research, nonresponse especially systematic nonresponse can be a major problem. In the current study, the effect of survey nonresponse on the job analysis was not particularly large, at least from a statistical perspective. From a practical perspective, however, systematic nonresponse can be extremely detrimental. Systematic nonresponse can create systematic bias in the resulting HR practices such as selection, compensation, and promotion because these practices are based on the job analysis data. Job analysis surveys are only one part of the job analysis process. Nevertheless, job analysis researchers need to address the issue of job analysis survey nonresponse, because the job analysis chain is only as strong as its weakest link. Clearly, niore research is needed in this critical area. Notes ' Casio, W F. (1991). Applied psychology in personnel management. Prentice-Hall. Englewood Cliffs, NJ: ^ Baruch, Y (1999). Response rates in academic studies: A comparative analysis. Human Relations, 52(4) ; Rogelberg, S. G., & Luong, A. (1998). Nonresponse to mailed surveys: A review and guide. Current Directions in Psychological Science, 7(2) 60-65; Schwarz, N., Groves, R. M., & Schuman, H. (1998). Survey methods. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (EAs), Handbook of social psychology (Vol.1, 4tb ed., pp ). New York: McGraw-Hill. ' Sanchez J. I., & Levine E. L. (2001). Analysis of work in the 20th and 21st centuries. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work and organizational psychology (Vol. 1, pp ). Thousand Oaks, CA: Sage. "^ Dierdorff, E. C, & Wilson, M. A. (2003). A meta-analysis of job analysis reliability./owrna/ of Applied Psychology, 88(4), Brannick, M. T, & Levine, E. L. (2002). Job analysis: Methods, research, and applications for human resources. Thousand Oaks, CA: Sage; Sanchez, J. I., & Levine, E. L. (2001), op cit. 238 Public Personnel Management Volume 37 No. 2 Summer 2008

17 ^ Wilson, M. A. (1997). Tbe validity of task coverage ratings by incumbents and supervisors: Bad n&^s. Journal of Business and Psychology, 12(1), 85-95; Henry, M. S., & Morris, S. B. (2000). Incumbent performance level as a predictor of Job analysis ratings. Paper presented at the meeting ofthe Society Industrial and Organizational Psychology, New Orleans, LA; Richman, W L., & Quiiiones, M. A. (1996). Task frequency rating accuracy: The effect of task engagement and experience. Journal of Applied Psychology, 81(5), ; Jones, R. G., Sanchez, J. I., Parameswaran, G., Phelps, J. Shoptaugh, C, Williams, M., & et al. (2001). Selection or training? A two-fold test of the validity of job-analytic ratings of trainability/omr««/ of Business and Psychology, 150), ; Tross, S. A., & Maurer, T. J. (2000). The relationship between SME job experieiice and job analysis ratings: Findings with and without statistical controls./owr/za/ of Business and Psychology, 75(1), ; Veres, J. G., Green, S. B., & Boyles, W R. (1991). Racial differences on job analysis questionnaires: An empirical study. Public Personnel Management, 20(2), ' Brannick, M. T, & Levine, E. L. (2002), op cit; Harvey, R. J. (1991) Job analysis. In M. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed.). Palo Alto, CA: Consulting Psychologists Press; Sanchez, J. I. & Levine, E. L. (2001), op cit. 8 Rogelberg, S, G., & Luong, A. (1998), op cit. ' Dreher, G. F. (1977). Nonrespondent characteristics and respondent accuracy in salary research. Journal of Applied Psychology, 62(6), ; Gannon, M., Notbern, J., & Carroll, S. (1971). Characteristics of norirespondents among workers. Journal of Applied Psychology, 55(6), ; Henry, M.S., & Morris, S. B. (2000), op cit. 10 Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000). Employee attitude surveys: Examining the attitudes of noncom pliant err\p\oyees. Journal of Applied Psychology, 85(2), " Rogelberg, S. G., & Luong, A. (1998), op cit. 12 Drehr, G. F. (1977), op cit.; Gannon, M., Nothern, J., & Carroll, S. (1971), op cit.; Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000), op cit. " Wilson, M. A. (1997), op cit.; Henry, M.S., & Morris, S.B. (2000), op cit.; Richman, W L, & Quifiones, M. A. (1996), op cit.; Jones, R. G., Sanchez, J. I., Parameswaran, G., Phelps, J. Shoptaugh, C, Williams, M., & et al. (2001), op cit.; Tross, S.A., & Maurer, T J. (2000), op cit,; Veres, J.G., Green, S. B., & Boyles, W R. (1991), op cit. I"* Cohen, J., & Cohen, P (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Eribaum Associates. '5 Cohen, J Statistical power analysis for the behavioral sciences. Mahwah, NJ: Eribaum. 1^ Henry, M. S., & Morris, S. B., (2000), op cit. 1'' Spencer, L. M., & Spencer, S. M. (1993). Competence at work: Models for superior performance. New York: Wiley 1** Roth, P L. & BeVier, C. A. (1998). Response rates in HRM/OB survey research: Norms and correlates, Journal of Management, 24(1), '9 Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000), op cit. 20 Roth, P L, & BeVier, C. A. (1998), op cit. 21 Tomaskovic-Devey, D., Leiter, J,, & Thompson, S. (1994). Organizational survey nonresponse. Administrative Science Quarterly, 39(3), Cropanzano, R., Rupp, D. E., & Byrne, Z. S. (2003). The relationship of emotional exhaustion to. work attitudes, job performance and organizational citizenship hehaviors. Journal of Applied Psychology, 88(1), Public Personnel Management Volume 37 No. 2 Summer

18 Authors Thomas A. Stetz, PhD National Geospatial-Intelligence Agency 26 Irwin Street Fort Rucker, AL J. Mathew Beaubien, PhD Aptima, Inc. 12 Gill Street, Suite 1400 Woburn, MA Michael J. Keeney, PhD Aptima, Inc M Street NW, Suite 900 Washington, DC mkeeney@aptima.com Brian D. Lyons, PhD Craig School of Business California State University, Fresno 5245 N. Backer Avenue, MS PB 7 Fresno, CA blyons@csufresno.edu 240 Public Personnei Management Volume 37 No. 2 Summer 2008

19 Thomas A. Stetz, PhD, is currently employed with the National Geospatial-Intelligence Agency He received his PhD in industrial and organizational psychology from Central Michigan University He also has an MS in management from the Walsh College of Accountancy and Business Administration. He has more than 10 years of professional experience and has written technical reports on applied projects, published peerreviewed journal articles, and presented research at numerous professional conferences. J. Mathew Beaubien, PhD, is a senior scientist and business area lead for Medical Healthcare Systems at Aptima, Inc. He has 10 years of experience designing tools and training programs to improve individual and team performance in high-risk industries such as health care, aviation and the military. Beaubien holds a PhD in industrial and organization psychology from George Mason University and an MA in industria! and organizational psychology from the University of New Haven. Beaubien has authored more than 40 conference papers, journal articles, and book chapters. Michael J. Keeney, PhD, is an industrial-organizational psychologist with Aptima, Inc. Keeney's research and applied interests target the efficient development, use, and enhancement of human capital. Keeney received his PhD in industrial-organizational psychology from the University of Akron. He maintains professional memberships in the American Psychological Association and the Society for Industrial-Organizational Psychology. Brian D. Lyons, PhD, is an assistant professor of management in the Craig School of Business at California State University at Fresno. He received his PhD in management from the University at Albany, SUNY and his MS in industrial-organizational psychology from the University of Baltimore. His research interests include Web recruitment, staffing, mentoring, and the aging workforce. Public Personnel Management Volume 37 No. 2 Summer

20