NORTHEAST CONSORTIUM NORTHEAST RESEARCH CONSORTIUM: Guide to Using Real Time Data for LMI Analysts. March, 2012

Transcription

1 NORTHEAST CONSORTIUM Understanding the Labor Market in New Ways NORTHEAST RESEARCH CONSORTIUM: Guide to Using Real Time Data for LMI Analysts March, 2012 This workforce solution was funded by a grant awarded by the U.S. Department of Labor s Employment and Training Administration. The solution was created by the grantee and does not necessarily reflect the official position of the U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, express or implied, with respect to such information, including any information on linked sites and including, but not limited to, accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership. This solution is copyrighted by the institution that created it. Internal use by an organization and/or personal use by an individual for non commercial purposes is permissible. All other uses require the prior authorization of the copyright

2 Section 1: THE LMI UNCERTAINTY PRINCIPLE Perhaps the most important distinction to make when dealing with online job postings is that they are not the same as job openings or vacancy rates. They are created by employers for their own unique and specific purposes, which does not include developing reporting data for use by LMI analysts. Online job postings may be used to attract applicants, but are also often times written to solicit a specific response or simply test the market. Online job postings are prevalent in some occupational areas and limited or non existent in others. The actual postings are usually vague in geographic location details and frequently do not include information such as salary, educational requirements, experience requirements, etc. Also, there is not necessarily a one to one relationship between an online job posting and a job opening. An employer might: Create a single posting for several openings with the same or similar qualifications in the same location or multiple locations; Use an online job posting to better understand the local workforce without having an actual job available; Post several times on differing websites for a single job opening, sometimes using different language in the description for each of the varying sites, making de duplication difficult or impossible; Never make some vacancies public, but fill them internally, through word of mouth, etc.; Use advertising methods that do not generate online postings signs in the windows, union hiring halls, radio ads, and employee referrals. As a result, in the world of online job postings, there are no standard practices, rules or conventions as to the form or function. Guide to Using Real Time Data for LMI Analysts Page 1

3 How Online Job Postings Differ from Job Vacancy Data THE LMI UNCERTAINTY PRINCIPLE 1: Output, i.e., projections & conclusions, cannot be better than input, i.e., the online job postings themselves. Postings are sourced, using spidering technology, from thousands of sites, frequently grabbing multiple copies of a single posting. Both the postings and the websites are of varying quality. The postings are designed primarily for recruitment, and are not designed for analysis. Analysis has shown that the postings are biased upscale in the labor market. There is a significant daily fluctuation of online job postings. Online job postings are influenced by factors unrelated to real changes in labor demand. Trying to extract real information from short time periods produces more noise than signal; three to six month chunks of data seem to be required to smooth out the noise. Geographic units of analysis are likely to be large, i.e., at the State level, because of the way job locations are posted. Zip code level data is not reliable because very few jobs post an exact location that can be tied to a zip code. THE LMI UNCERTAINTY PRINCIPLE 2: The more granular you make any parameter, the less you can know about other parameters. Analyzing skills at the occupational level will require large spans of both time and geography. Projections may be possible, but may not be capable of being isolated to individual occupations except in areas with very high concentrations of that occupation. Guide to Using Real Time Data for LMI Analysts Page 2

4 What Have We Gleaned From Online Job Postings Data? Advertising Behaviors: An analysis of postings based on job family (two digit SOC level) showed common patterns of advertising behavior. Different posting methods were found by job family, even by the same employer. Large employers and corporations were more likely to post all jobs online. Smaller businesses were more likely to only post executive or high skill positions on line. Data Quality and Quantity: The ability of automated software programs to generate accurate data from online postings is affected by both the quality and quantity of the postings. In some job families, larger quantities of postings have allowed the software to produce data that is measurably accurate for that job family. For other job families, posting quality, i.e., detailed content, is commonly found, which allows the software to produce measurably accurate data. Job families with a negligible number of postings (e.g., agriculture, forestry and fishing, building and grounds cleaning and maintenance, etc.) were found to be of limited accuracy. Job families with quality postings have the potential to be analyzed at the four digit SOC level if there is also sufficient quantity (e.g., computer and math, healthcare practitioners, technical occupations, etc.). Context Is Important: Understanding terminology context is important. Some terms and phrases have different meanings depending on the occupation to which it is applied. For example, reuse applies to software reuse, workforce reuse, and water reuse, but the meaning is quite different for each. This context is important because the meaning of the words can make classification of the skill different. The word reuse alone could be green for instance but it is not green when it is modified by software but probably green when modified by water. Analysis software can be trained to recognize these context items but such training requires significant work by trained LMI analysts. Postings Data are Complementary to Existing LMI: As with the majority of LMI data, online job postings do not represent a complete picture of the labor market. The potential value of this piece is in its timeliness and that it measures the flow of jobs, as opposed to being a measure of stock usually found in LMI. In preliminary comparisons, the online postings data was found to have similar patterns to JOLTS data, one of the few flow measures in traditional LMI. Potentially, existing LMI could be used as a benchmark for online job posting patterns, providing confidence that past data had been extracted within acceptable error limits and thus providing a measure of confidence that future data is also being extracted within acceptable error limits. Guide to Using Real Time Data for LMI Analysts Page 3

5 Not All Areas Are Created Equal: Geographic location for an advertised job is not universally available. Rural areas were found to have a limited share of job ads, while urban areas hold the greatest share. Frequently the actual location of a job is not included in the job posting, causing the data parser to default the listing to the location of the central zip code of a large labor market clearly an unreliable zip code location. Because of the location accuracy, the Consortium s goal of being able to provide data for all substate areas was not possible at present and may never be possible. Real time data reported at any level below a major metropolitan area is suspect and will either substantially under or over state the online posting counts. Real time LMI providers that produce city, county or zip code level data should be asked to disclose their coding and modeling methodologies for review and data produced at those levels should be considered questionable for any use until the analyst is satisfied with the response. Data Source is Important: Sources primarily have an effect on the data in two ways: First, data from some spidered sites simply lack content, compromising the parsing and occupational coding processes, i.e., you cannot analyze what does not exist. It is important to understand your vendor s spidering and de duplication processes. Second, there can be artificial increases in the volume of job postings when source domains are added to the spidering process. This can be irrelevant if you are doing a pointin time analysis, but an impediment to doing time series analysis. It is important to know whether your vendor is continuously adding new sites in an attempt to provide a reasonable representation of the scope of the labor market or whether your vendor is limiting the number of sites to a stable subset of reliable and representative sites. Guide to Using Real Time Data for LMI Analysts Page 4

6 There may not be a single version of a real time database that can be used for all purposes and maintaining multiple versions may prove to be prohibitively expensive. Pursuing the expansion of spidering to cover all large corporate sites, large national boards and smaller regional and occupation niche boards may be the only way to get representative counts in all local areas and all occupations. However, narrowing the number of sites to a stable but representative group so that a time series can be created produces data which tracks more closely with other external measures (the Help Wanted Online database is the best example of this approach). For analysis of the skills and other requirements of the particular class of job, it might prove useful to create a database that is fed only by high quality corporate, national and niche job boards. High quality in this context would be boards that contain extensive job postings with some or all of the information fielded (e.g., education or experience requirements are always found in the same place). Free posting sites like Craigslist or Snagajob are likely to be excluded. This constrained source database would not be used to gauge absolute volume, but may be very useful in showing proportions skill mix, education requirement distributions, experience requirement distributions, etc. Guide to Using Real Time Data for LMI Analysts Page 5

7 What Issues Must Be Addressed When Publishing Postings Data? Data Labeling: Results should be labeled online job postings, job ad volume, online job ads, or something similar, to accurately reflect the data being analyzed. The data does not represent job openings or vacancy rates. Incorrect uses of these terms, i.e., confusing job ads with job openings, have already been seen in media stories on various published data series. Data Accuracy: Data made available to the public should only be from data fields with a reasonable measure of accuracy. Data fields that cannot be verified to some standardized measure whether due to lack of information within the postings or due to error in the analysis software should not be published. Continuing standardization will be needed for ongoing research. Job posting data is fluid and continued analysis will be needed. Buyer Beware: When publishing information based on online job postings, you must also present data limitations, caveats, potential error rates, etc. Some end users may choose to utilize the data inappropriately anyway, but the same holds true for any LMI that is published (and is frequently misquoted and used out of context). We do recommend that only data with minimal caveats or disclaimers be released to the public. We also recommend that high level policy makers should only use online postings data in their decision making process if it is combined with traditional LMI. Although there is still a great deal of analytical and procedural work to be done, the information extracted from online job postings is still worthwhile, if used appropriately. Guide to Using Real Time Data for LMI Analysts Page 6

8 Realities and Limitations of the Data (It is important to note that there was not complete consensus by all of the Consortium members on all of these points on this page and the next, or on the value of online job postings in the decision making process. However, these two pages represent the best aggregation that everyone could live with.) DATA FIELDS Occupation Geography Industries Skills Job Title Firm Name Education Certification Greenness Timeframe Job source CURRENT LEVEL OF CONFIDENCE Generally 2 digit, depending on job family State level Unknown Currently, of limited utility and only with analyst review; Shows promise, but needs more research with an analyst looking at context and job family Yes Sometimes, but questionable, due to so many variations of the name; Requires an analyst to vet Sometimes, but interpretation and context are important (degree might be required, might be preferred). Also, many online postings for jobs with clear degree requirements (e.g., lawyers, nurses, etc.) will not include any stated requirement, letting the degree level be implied. Caveat: Low % of fields listed When it is listed, it is generally valid, but it still needs an analyst to review, based on the low percentage of listings Sometimes; should only be used for research purposes with an analyst review for interpretation and context. Analysis based on skill words generally produces much more accurate results than coding by occupation. The primary problem is the rate of false positives (calling a job green when it is not). Better to use quarterly data or a 3 month moving average to smooth out spikes; monthly is possible (calendar & 30 day timeframes), depending on the report Yes Guide to Using Real Time Data for LMI Analysts Page 7

9 WHAT THE DATA CAN DO: Can show which websites post job ads Can show which firms post job ads Can describe data at the state level or other large geographic area Can display accurate information at most 2 digit occupational codes Can compare job titles with other variables Can use postings on a quarterly or, sometimes, on a monthly basis, depending on the report; usually better to use quarterly data or a 3 month moving average to smooth out spikes Can improve quality by selectively removing bad sources of data, using Quality Assurance & Data workgroups Can show educational levels matched to job titles, especially in high skill jobs, when education is listed or inferred and with an analyst looking at interpretation and context Can provide information on certifications within certain occupations, when it is posted and with an analyst review Can look at skills and certifications together to get a more complete picture, with an analyst reviewing, especially for context and job family May be able to do firm name based on query (good for green jobs, comparative analysis, and profiling online job ad posters) May be able to get an up/down trend on known list of skills May be able to show increase/decrease in job titles Can show job titles in demand Can show top volume O*Net codes WHAT THE DATA CANNOT DO: Cannot equate job posting with job vacancy Cannot eliminate all duplicates Cannot get the universe of job openings Cannot get all 6 digit occupational codes Cannot do industries Cannot know negative biases on anything not posted or parsed Cannot use skills without vetting Cannot project job postings yet, but modeling shows promise out to 6 months Cannot provide consistent representation of postings from month to month Cannot show salary or benefits Cannot publish zip code or any sub state level data Cannot always determine educational needs in most postings Cannot always determine required vs. preferred educational level (where listed)

10 Section 2: OVERVIEW OF REAL TIME DATA With employers increasingly turning to the Internet to advertise opportunities, online job postings have become the primary vehicle for many job seekers. There have been mixed responses to this development. Some argue that given the size of today s applicant pool, firms have become ever more selective when sorting through résumés, prompting rumors of discrimination against the unemployed. 1 Others contend that online intermediaries alleviate labor market imperfections, often associated with imperfect information and adverse selection. 2 Labor market analysts, workforce professionals, and researchers are increasingly in need of diverse and robust sources of data in order to be more responsive to the changing economy. In recent years, realtime data, most commonly in the form of online job postings, has become a staple of some labor market information (LMI) staff, and its use is expected to increase in the years ahead. For example, the Projections Managing Partnership, which coordinates the nationwide employment projections, has stated that the new projections interface will likely include data from online postings alongside the standard information, such as total employment and new and replacement job growth. 3 In this changing environment it is critical for researchers, policy makers, and other data users to have a broad understanding of what real time data is, how it is obtained, and what it can and cannot accomplish, as well as how it relates to more commonly used sources of data or traditional labor market information. This section provides an overview of the analytical uses of online job postings, outlines the methodology used by the Northeast Consortium s data provider (Burning Glass Technologies), and briefly discusses the relationship between online postings and traditional sources of labor market information. 1 The Help WantedSign Comes With a Frustrating Asterisk NYT ( wanted ads exclude the long term jobless.html?_r=1&smid=fb nytimes&wt.mc_id=bu SM E FB SM LIN HWA NYT NA&WT.mc_ev=click) 2 The Economics of Labor Market Intermediation VOX Policy Research ( crisis debate.com/index.php?q=node/2500) 3 Presentation by Ms. Alexandra Hall, Director, Office of Government, Policy and Public Relations, Colorado Dept. of Labor & Employment Guide To Using Real Time Data for LMI Analysts Page 9

11 The Promise and the Pitfalls Traditional LMI includes systematically collected data from either administrative records, such as the Quarterly Census of Employment and Wages (QCEW), or from surveys, such as the Occupational Employment Statistics (OES) wage survey. This methodologically sound data is both valid and reliable; however, there is a significant time lag between when data is collected and when it is released for analytical purposes. Online job postings can be compiled into a database and analyzed as a proxy for recent demand in the labor market. In contrast to traditional LMI, online job postings (real time data and online job postings are used interchangeably in this report) offer a more timely data source for analysis. As previously stated, online job postings were designed for recruitment, not analysis; therefore, postings databases lack the validity and reliability of traditional LMI. They have the potential to provide an innovative tool for analysis of issues in labor demand, especially related to industries, such as green, where many of the occupations and their accompanying knowledge, skills, and credentials are still emerging. Aggregated job postings databases include many variables that are absent from traditional data sources, such as skills, education requirements, industry based certifications, and employee benefits associated with job postings. The promise of the data has resulted in increased interest in their use for analysis. The potential benefits of jobs postings analysis include: 1) unique features that position postings as the solution for curriculum development; 2) incorporation into models for more accurate short term projections; and 3) a source of emerging skills that have not made it to standardized knowledge, skills and abilities taxonomies produced by O*NET. These benefits and others have resulted in the increased interest in and use of postings in recent years. Job postings databases compiled from various Internet sources are not without limitations. While the diversity of sites used as a source of data is an asset, it also creates problems not found in traditional labor market data. The quality varies widely. Postings often do not include all of the skills, knowledge, or credentials required for a position (and it is impossible to determine what proportion of the requirements is included). In addition, postings are a proxy for demand, but they do not represent actual hiring. While not all postings are for open positions, it is also true that not all open Guide To Using Real Time Data for LMI Analysts Page 10

12 positions are posted on the Internet. Word of mouth, referrals, and even signs in the window continue to be preferred recruitment methods for some industries and firms. There are also occupations where recruitment rarely occurs online. The number of jobs posted online is higher in fields such as computer science and management, but lower in fields like construction and farming. Many of these factors result in the postings having a generally upscale bias overall in terms of the types of positions listed. That upscale bias may actually be a benefit for the use of real time data to help guide curriculum development because many of the positions not recruited online are either low skill or require only on the job training. Guide To Using Real Time Data for LMI Analysts Page 11

13 Data Field Specifics What the data can do There are several variables in the online postings that are of value for data users and consumers. Unit of time: The job postings data used by the Consortium includes a variable for the date of the posting. Data quality reviews concluded that the minimum unit of time for analysis is one month because of the variability of postings on a daily or weekly basis, but that it was generally better to use at least quarterly or 3 month moving average data. The use of a moving average is common for highly volatile data series like the unemployment claims data. As noted above, the more granular the other elements of analysis are (e.g., occupations, levels of geography, skills), the longer the time period of analysis should be. Geography: State level and large geographic areas are the best geographic units of analysis. As described in the limitations section, the location information available in the postings is not as specific as it may seem. Zip code level data was deemed unreliable and misleading. Job title: Job titles from postings are useful variables for analysis. Titles can be compared over time to determine if some are increasing or decreasing in frequency. They are also valuable when used in conjunction with other variables (e.g., education, where available). Titles may produce a window into emerging occupations and, with further analysis, emerging skills. Firm name: The name of the firm is not available for all postings (because it is not listed on some sites), but it can be analyzed when it is included. Useful information that can be gained from firm name analysis including the types of postings (occupations and job titles) listed for specific firms, seasonal hiring patterns, and gaining a better understanding about which firms post jobs online. Guide To Using Real Time Data for LMI Analysts Page 12

14 Skills and certifications: Skills are not clearly represented in the postings, and certifications are included in only limited groups of occupations. Skills, in this context, is short hand for a mix of qualifications and job duty statements that might more broadly be thought of as skills, abilities, theoretical and applied knowledge, common tasks and types of experience. Terms and abbreviations can have more than one meaning; for example, P.T. might mean Physical Therapist or Part Time. Postings frequently do not specify qualifications, as it appears an assumption is made that those who are qualified already know the qualifications. For example, lawyers know they need to pass the Bar, and nurses know they need a degree of some kind and need to be licensed. Furthermore, the data analysis showed that the skills field often functions as a catch all category into which skills, certifications, occupations, and qualifications were parsed. Specific skills and knowledge areas, such as knowledge of a computer language, are relatively easy to identify, as are basic skills such as communication. Other terms do not parse as neatly. This area requires a great deal more research. Further research should focus on extracting and categorizing skills and certifications that will bear fruit because online postings are the only real source of this data. As noted earlier, one of the key efforts is to increase the ability of analytical software to understand context. Source Domain: The website from which the job posting was captured can be a useful analytical tool. It can point to websites that generate disproportionate errors from the analytical software because of how the job information is formatted or how the site displays search terms and related jobs. It is potentially useful in generating stable time series databases. However, analysts need to be aware of shifts in domains that are unrelated to real changes in the data stream. Tracking successor and predecessor domains is a critical component of being able to track real changes, rather than simply a company s decision to redesign their website. This successor/predecessor issue should be familiar to any analyst who has tried to clean up wage record or QCEW files. Analysts need to be cautious about spikes in data from particular domains. Increases or decreases may be the result of changes in spidering, such as the addition of new websites or a broken spider. Such shifts may change the relative importance of a site in terms of producing master jobs the original job against which duplicates are measured. Guide To Using Real Time Data for LMI Analysts Page 13

15 What the data cannot do Northeast Research Consortium March, 2012 In addition to the types of analysis that can be conducted with online job postings data, there are also a number of analyses which are not advisable at this time. Location: Exact job locations are difficult to pinpoint. Recruitment often takes place regionally, and job postings commonly list the largest city near the work location, such as the Boston area. Unless the posting includes a precise address for the company (and even this is not always the place the new employee would be working), the general location is at best a proxy. The location related variables (e.g., city, county, zip code, latitude and longitude) should be used with caution. Geographies below the metro area level are likely to contain significant over or under counts, depending on where the analysis software decided to locate the job that lists only a broad recruitment area. Salary: In most online job postings, salary information is not included. When a salary is listed, it can be presented in many different forms (e.g., annually, monthly, weekly, or hourly). Postings with some salary information may include dollar signs or other abbreviations that cause inconsistency when parsing the information into variables. For all of these reasons, this data field is not considered usable. Occupational Coding: Occupational coding beyond the two digit SOC code level, is problematic. Overall, two factors influence the ability to analyze data by job family (two digit SOC). First, job postings do not, as a general rule, include occupational codes. Thus, the accuracy of occupational coding depends on the data provider s ability to accurately classify a job title into an occupational taxonomy. Within the data file used by the Consortium, the overall accuracy varied by type of job. For example computer and mathematics jobs were usually coded correctly, but farming, fishing, and forestry jobs were rarely coded correctly. The second factor influencing job family analysis is that the jobs posted online do not represent the universe of all occupations. Even if correctly coded into a taxonomy, there is limited representation for some job families (e.g., construction), while others have substantial representation in the postings (e.g., information technology / computer related occupations). The extent of the under representation can be estimated for job families where the job family has a tight link to a 3 digit NAICS code. For instance, NAICS 722 Eating and Drinking Establishments would be tightly linked to the SOC Family 35 Food Preparation and Service Workers. Using separations and new hire data from Census LED, one could estimate the proportion of turnover that would be expected in that industry compared to all others. That estimate of turnover could then be compared to the proportion of job posting found in job family 35 to see the extent of the likely underrepresentation. Guide To Using Real Time Data for LMI Analysts Page 14

16 Qualifications: Qualifications for job applicants, such as education, can be difficult to determine. This manifests itself in a number of ways: for example, most postings do not include the desired level of education, or when they do it can be unclear whether it is a required or preferred qualification. In addition, some basic prerequisites (e.g., passing the Bar exam for an attorney) that may be an industry or occupational standard are rarely listed because it is assumed that qualified applicants already know this requirement. As noted above, the full skill set required for each posting is typically not included. Industries: Online job postings are just that jobs. Industry information, and thus the ability to accurately assign a NAICS code, is rarely included in the postings. Frequently the posting is spidered from a site other than that of the employer, and a firm name is not listed. Guide To Using Real Time Data for LMI Analysts Page 15

17 Burning Glass Methodology The Consortium s job postings data was provided by Burning Glass Technologies, a Boston based firm that aggregates online postings and offers a variety products and services related to job matching. Postings are collected daily from thousands of private and government job boards and websites, newspapers and other media outlets, corporate job boards and websites, and community sites for employment opportunities. They are collected through spidering technology which crawls the web for suitable content. The text of each posting is then analyzed using a natural language based artificial intelligence, which allows context not simply a list of rules or look up tables to be taken into account when parsing the text into variables. A dataset of over 60 variables is produced, which range from the most basic (job title, company name, city) to more complex concepts (skills and credentials). For the Northeast Consortium project, a taxonomy of green skills was developed which serves as a reference so that when the parsing technology encounters a term representing a green skill it can be standardized in the database. Guide To Using Real Time Data for LMI Analysts Page 16

18 Job Postings Data and Traditional LMI Limited analysis has been done comparing trends in online job posting data to traditional labor market information. The Job Openings and Labor Turnover Survey (JOLTS) is not identical, but it is a helpful source of comparison. The two data sources have size differences; however, this has changed over time. In September 2004, there were over 4.1 million total private job openings from JOLTS, versus about 1.3 total online postings from Burning Glass; by summer 2010 this gap had narrowed to about three million openings for JOLTS and two million online postings for Burning Glass. This was likely due to a number of factors, such as increased use of online posting by firms and increased sites spidered by Burning Glass. More importantly, each series follows the same broad trends over time. And, although job postings data do track closely with Current Employment Statistics (CES), there is insufficient evidence to suggest that the data is a leading indicator. Guide To Using Real Time Data for LMI Analysts Page 17

19 Section 3: DEFINING GREEN Despite the recent momentum behind green jobs, there is still no universally accepted definition of green. A number of scholars have explored the workings of the green economy, but they all face the challenge of defining and quantifying this nebulous concept. In an effort to advance the efforts of conceptualizing the green economy, the Consortium chose to deviate from the traditional survey based approach and concentrate instead on analysis of online job postings. Data mining a universe of job ads gives researchers not only a time advantage, but also greater flexibility with respect to defining green. It also gives them the ability to modify the framework for what constitutes green (i.e., clusters, search triggers, etc.). More importantly, this method allows us to track the greening of the economy more efficiently than using green revenue as a proxy. 4 Given its strengths, this approach also has limitations. The Consortium had intended to conduct an analysis of robust data, however much of the time and focus was spent on improving the artificial intelligence parser and ongoing identification of green occupations and green skills. Initially it was believed that an industry based (NAICS) approach would yield the most effective outcomes. It quickly became evident that not all jobs within an industry, or even a job family, could be considered green and, most importantly, industry coding proved to be one of the most erroneous variables, in those rare instances where it was included in an online job posting. Therefore, a two tiered approach was established first, identification of green skills phrases, and second, creation of a green firms list. The foundation of this methodology is based on the green taxonomy, which is primarily derived from O*NET s descriptions of green occupations. In cooperation with our vendor, we closely imitated each occupation s description and, in turn, built and continuously updated a database of nearly 900 key phrases pertaining to green. 5 In effect, this is the basis of real time green demand analysis the prevalence of these tasks and phrases in real time data serves as a gauge of demand for green jobs. More specifically, postings that are validated through this taxonomy are flagged as green, signaling that certain green skills were listed in the job posting. Consider O*NET s description for Construction Managers, for example: Apply green building strategies to reduce energy costs or minimize carbon output or other sources of harm to the environment. Working from the above occupation, the following strings would all be included in the taxonomy to serve as trigger words in identifying green jobs: green, building, minimize carbon output, environment. The taxonomy can be easily improved with the addition of like terms and, similarly, the elimination of those deemed obsolete. 4 To proxy green jobs, some earlier studies used firms green revenue as share of total. Accuracy of such estimates is unsettled. 5 Appendix contains full taxonomy and clusters Guide To Using Real Time Data for LMI Analysts Page 18

20 The green jobs report developed by the Consortium shows the volume of online job ads containing green skills taxonomy. The report shows a point in time snapshot of the top green skills for the specified timeframe and geography. LMI users should be aware that due to changes in parsing software, changes to the green skills list and fluctuations in the number of websites spidered, there are too many inconsistencies to conduct a reliable time series analysis of individual skills. While the first approach provides a proxy for green labor demand via select keywords, the second tier was expected to give us parity with part of the BLS definition green goods producing and service providing establishments. Postings from establishments that only produce green goods or provide green services are indisputably green, so it follows that such analysis would shed light on recent green developments, such as skills and certifications demanded by the green business community. By matching a Consortium wide list of green firms 6 to real time data, we had hoped to reveal postings that may not have been triggered by our taxonomy, but may still be considered green since they are derived from a green employer. This method has proven to be very challenging in practice, however. Agreeing on green only producing establishments is problematic, but accurately matching establishment names to those appearing in real time data is simply not feasible at this time. This is largely due to the nature of online job postings (explained in the next section), whose format and content can vary substantially. Firstly, roughly half of the postings contain no employer information. Of those ads with the employer name present, slight discrepancies in spelling accounts for the remaining incongruity. For instance, in real time, company XYZ may take the form of XYZ, XYZ Inc., XYZ Corp., XYZ Ltd., etc. making the matching process not viable. One possible solution would be to add a handful of variations to each establishment, along with the official name, as seen in tax records. For these reasons, the advancement of this approach was not pursued as initially planned. As with most empirical work, we faced broader methodological limitations in defining green. Our reliance on algorithms to sort through the complexity of online jobs postings was the largest source of concern. As previously stated, postings are crafted for recruitment purposes and not for analysis, thereby often omitting information of interest to researchers. Even when the algorithms are on target, the context of each posting is perhaps the most delicate factor in the process, as many key words are acutely context sensitive, i.e., some terms have different meanings depending on the occupation to which it is applied. For example, reuse can apply to software reuse, workforce reuse, or water reuse, guaranteeing skewed green posting counts. Context also plays a factor when the parser cannot distinguish between a web site s page navigation, advertisements, and other source code, and the text of a job posting. When a green skill is pulled from a source outside the actual job posting text, it creates a false positive leading to an overcounting of green jobs. Also, it is wrong to assume that the distribution of green job postings mirrors the patterns of the real green economy; as previously explained, the analysis clearly shows that online job postings are not representative of all economic sectors. 6 See Green Firm Identification. Guide To Using Real Time Data for LMI Analysts Page 19

21 With the current occupational and industrial coding systems there is no perfect method for identifying green jobs. More specifically, green crosses the boundaries of today s job classification systems, making the green economy difficult to measure. Three different methods have attempted to overcome the difficulties: 1) counting green revenue share and apportioning employment via the given ratio; 2) directly surveying employers and asking them to list green jobs; and 3) using real time job ads to proxy green labor demand. The Consortium attempted to fill the void in previous studies by addressing a retrospective look at the green economy and we believe our methodology will enable researchers to track the gradual greening of the economy more precisely than other measures. While attempting to overcome the rigid nature of existing studies, our approach is not error free and we ve learned that real time requires further refinement and standardization to be a credible tool. With further improvement in artificial intelligence, analysis of the green economy through the lens of real time data should eventually result in more robust findings. Guide To Using Real Time Data for LMI Analysts Page 20

22 Section 4: THE DATA ITSELF There have been mixed responses to the increasing role of the Internet in recruitment. Some argue that given the size of today s applicant pool, firms have become ever more selective when sorting through résumés, prompting rumors of discrimination against the unemployed. 7 Others contend that online intermediaries alleviate labor market imperfections, often associated with imperfect information and adverse selection. 8 The prevalent use of the Internet in job search has greatly expanded the geographic and skill scope for employers, but, more importantly, the explosive growth in online job postings holds great potential for the research and LMI communities as a cutting edge tool to examine labor market conditions with less lag time than standard LMI data. As this approach continues to gain traction, end users are in need of a better understanding of the underlying data. One goal of this Consortium was to fill that void by providing an insider view of real time data. At present, the Consortium appears to be the first to critically evaluate the robustness of real time data. This section outlines critical issues of which analysts must be aware prior to working with real time data. 7 The Help-WantedSign Comes With a Frustrating Asterisk NYT ( 8 The Economics of Labor Market Intermediation VOX Policy Research ( Guide To Using Real Time Data for LMI Analysts Page 21

23 Nature of Real Time Not Traditional LMI Data It is important to highlight our ability to closely examine the data in raw form, a notion not well embraced by the industry most vendors provide preset, aggregated job postings and related analytics, which prevent gaining deeper insight on the subject matter. To mine the data, the Northeast Consortium partnered with Burning Glass Technologies (BG), a Boston based firm that develops technological methods of matching résumés to job postings. The data collection process relies on spiders that crawl the web, aggregating data from job boards, employer sites, government agencies, and newspapers. The goal of this process is to gather a comprehensive jobs database. The data is then locally stored, parsed and coded to over 60 variables, such as location, employer name, occupational and industry codes. Due to data quality, only a handful of variables have been found to be suitable for analysis at this time. Though there is room for optimism about expansion of the number of reliable variables, given the considerable improvements witnessed in our short time of working with real time data. 9 To understand complexity of real time data, one need not look further than the data deluge that has accompanied the Internet boom. Online content is growing exponentially, with only a fraction of it being verifiable, valid, and usable material. The same can be argued about online job postings. The limitations of real time are largely a consequence of: (1) how the postings are originally crafted, and (2) the effectiveness of artificial intelligence, or the parser, in properly coding the data. Unlike working with a static, homogenous database structure, the postings are aggregated from over 16,000 non standardized sources. For example, Craigslist, a free classifieds service, is a significant source of postings volume, but it also creates a substantial share of data problems. Unfortunately, artificial intelligence cannot yet effectively distinguish between a job, a personal ad, or spam. Whatever ends up on a job board is spidered, parsed, and becomes part of the real time database. This highlights a major difference between real time data and other job measures. Real time data is dependent upon how firms and recruiters craft their ads and do not necessarily translate into a hire or represent a vacancy. In other words, postings are designed for recruitment and not for research and analysis employers do not have to reveal much information to receive a flood of applications. In fact, it is not uncommon for key variables, such as the exact location, salary and even the employer s name to be omitted. While some variables are often omitted in job postings, it does not negate the value of those variables when they are present. For example, quality control reviews found that educational requirements were specifically identified in just one third of total postings. Among those postings that did not specify education, the educational requirement could be inferred for nearly 20 percent. 9 See Appendices for a summary of variables, their definitions and quality control results. Guide To Using Real Time Data for LMI Analysts Page 22

24 Postings without educational requirements were generally for occupations with common educational levels for qualification, i.e., professional or technical occupations in health care, computer science, finance, or law. When education level is specific to a position, it is often an employer requirement, thus the employer is more likely to include that information in a job posting. On the other hand, if education is an occupational requirement, as frequently established for State issued licenses, it is not uncommon for the educational level to be omitted, as qualified applicants are presumed to have already met the requirement. This observation presented an interesting dynamic in online job postings. The 2010 American Community Survey estimates show that about 37 percent of the region s population aged has a bachelor s degree or higher. 10 In a sample of 8,000 job postings for the entire region, among those that specified an educational level, just over 70 percent required a baccalaureate or higher. 11 This measure indicates an upscale bias in online job postings, meaning that jobs with lower skill requirements are less likely to be found online than jobs requiring higher levels of educational preparation. Analysis of zip code level and other sub state locations has indicated a significant issue. While value was found in available educational requirement information, geographic location questions were not so easily resolved. Some job postings are designed for national recruitment, while others are not. By the same token, some postings are only advertised internally or by word of mouth. With specific location data commonly omitted from ads, the parser infers a zip code for location of the job, sometimes using the firm headquarters, or the nearest metropolitan area, or even the job board s physical location. For example, a posting for a job in Westchester, New York might be advertised on multiple job boards as being in the greater New York area. The parser infers a zip code identification for New York City, since that is the only geographic area mentioned, but it is incorrect. This common lack of detailed geographic information brings into question the usefulness of geographic specificity below the State level, or perhaps the level of large metropolitan areas. Unlike traditional survey based methods, real time data is not subject to standard sampling errors, but is influenced by factors other than true labor demand. In essence, it is extremely difficult to control exogenous factors, such as the full extent of removal of duplicates (de duplication), job board under or over coverage, and spidering processes, among other issues. For instance, by plotting daily American Community Survey 1 Year Estimates, Table B23006: Educational Attainment by Employment Status for the Population 25 to 64 Years. Data summed for the eight state Consortium region. 11 Estimated from a sample of 8,000 postings from the period of 8/1/10 2/5/11 from all eight Consortium states Guide To Using Real Time Data for LMI Analysts Page 23

25 postings it becomes evident that the spiders are inactive some days while spiking on others. This volatility, along with an inability to adjust for seasonality, makes one month the minimum period for conducting time series analyses, with a three month period frequently being preferred. More problematic are the approximately 16,000 sites from which the data are sourced. In some cases, the spider that gathers data from these sites cannot differentiate between job posting text and the unique HTML code and detailed meta information 12 contained in a web page s source code. If page code and metadata is parsed into the real time database, it can undermine the parser s coding accuracy. Close examination of the data has shown that a single month may have as many records of parsed page code and metadata as the three previous months combined. As a result, the data has had to be continuously monitored and the parser updated with unique patches to address quality issues. Creating good real time data requires substantial and continued investment in human analysts who will not only monitor the system for quality, but be involved in the necessary efforts to improve the analytical software and provide context for the words and phrases found in the job ads. 12 Meta information, or metadata, describes other data, such as length of document, author, or date created. Web page HTML source code can include page navigation, text from other advertisements, page headers and footers, etc. Guide To Using Real Time Data for LMI Analysts Page 24

26 Duplicates and De Duplication Another major challenge in producing reliable analysis is the prevalence of job scraping, which is an industry wide practice that consists of copying postings from corporate client sites, state jobs banks, and other sources to other job boards. Job postings are produced to attract applicants and some are written by employers to solicit a specific response, or test the labor market. Duplication is beneficial for employers as expanded market coverage increases the odds of reaching the talent pool further reinforcing the concept that real time data does not directly track job openings or vacancies. Thus, the interests of job seekers, employers and job boards are usually in conflict with those of an analyst. This highlights the need for a standardized methodology to identify duplicates and to de duplicate data. Industry research suggests that most vacancies are filled within two to three months; therefore postings that re appear within a 60 day period are flagged as duplicates by our vendor. (Other vendors may use a different time period and analysts should always query the vendor about the specifics of the de duplication methodology.) This is subject to debate, however, as those estimates were developed prior to the recent recession. The Great Recession has created an employer s market, with more job seekers than job openings. Further delaying the hiring process is the gap between the candidates skills and experience and those desired by the firm, leading recruiters to woo candidates from competitors. The 60 day window is a subtle element and to date, the Consortium has not produced significant analyses on the role of time frames to producing more robust data. Generally, duplicates are believed to complicate the reconciliation of total job counts reports and in turn, cause fluctuations in the distribution of key variables until 60 days after the original posting. To further distinguish duplicates from unique postings, the de duplication algorithm relies on a number of select variables (namely, job title, employer, city, state, skills ) and the presence of employer information. The standard de duplication algorithm used by our vendor is able to identify and remove a share of the duplicates based on defined criteria, such as a reoccurring job title posted by the same employer name. But it still takes an experienced human eye to verify the extent of duplicates in the data, which is costly. De duplication concerns are often de emphasized, because the algorithm relies on a script with a strict set of rules that do not always pick up small subtleties in the postings. One cannot fully rule out the possibility of a duplicate even if the content inside the posting has been modified. Analyses of the rates at which select postings and/or occupations reappear may provide a useful gauge for determining the difficulty in filling certain vacancies. Even after being filtered, the dataset may still contain percent duplicates, primarily due to slight variations in either the job title or the employer name, such as Vanguard, Vanguard Inc., or Vanguard, Inc. This variation can be Guide To Using Real Time Data for LMI Analysts Page 25