The Case Study highlights how a leading IT company has used SDE to identify and allocate best programmers for critical projects they manage.

Transcription

1 Optimising your investment on human capital The globally accepted, performance-based skill assessment programme Software Developer Examination (SDE) from Proxor helps IT and software services organisations hire, train and deploy the right resource to the right job. SDE is delivered in India by MeritTrac Services India s Largest Testing & Assessment Company The Case Study highlights how a leading IT company has used SDE to identify and allocate best programmers for critical projects they manage. A MeritTrac Case Study

2 NASSCOM (National Association of Software and Services Companies) the apex body of the Indian business process outsourcing and software services industry has set an industry revenue target of 300 billion USD by 2020, 177 per cent more than that earned in It also foresees that the software product market which currently By 2020, industry revenues predicted to touch $300 billion. generates 2.2 billion USD is expected to reach 10 billion USD by Since 2000, when there were a little over 100 software product firms, we have grown to 2400 and counting. There is no denying that the future of IT and software services market for Indian companies appears bright. However, these very opportunities also amplify the omnipresent and significant challenge that the industry faces: that of finding capable, qualified and skilled resources across development, maintenance and testing. Year after year, studies have shown that the employability of graduates with software and IT skills are dismal in India. National employability report 2014 finds that less than 20% engineers are employable for software jobs. Of the six-lakh engineers who graduate annually, only 18.43% are employable for software engineering and IT services roles, while a dismal 3.95% are Only 3.95% of engineers graduating this year can be deployed directly into projects appropriately trained to be directly deployed on projects. As a result, organisations spend six to nine months of an employee s salaried time in making them employable. The situation is worse in the IT products sector where emphasis on high-end development is higher than in testing and maintenance. Lately, offbeat methods like hackathons are being used in identifying skilful candidates. The winner takes all approach to the game restricts hiring opportunities to the winners, where

3 the candidates who didn t finish on top leave empty handed; and the hiring companies miss out on these prospects. Moreover, though a hackathon is a test of skill, it is not based on scientifically built assessment methods and does not deliver a confident assessment that the candidate will perform equally well in a real work environment. Lastly, there is no constructive feedback mechanism at the end of the hackathon, leaving everyone none the wiser. Some organisations make it a point to visit only the crème de la crème of engineering colleges to hire the best talent there is. Often, this leads to type 2 error: Not hiring candidates who do have the skills required. In fact, research shows that organisations miss out on 70% of the employable pool in lesser-known colleges in the country. In spite of all precautions, there is a consistent loss of about 10% on fresh hires after on boarding and training. This leads to: 1. Companies not finding the incentive to hire from beyond their familiar talent pool of the top 10% engineering colleges. 2. Among those hired using the available tools, a significant number are not skilled to be deployed on development projects. Therefore, they need to undergo six to nine months of training. Those who don t excel post this are allocated to testing and maintenance roles, leading to wasted investment in training. 3. A significant proportion of testing and maintenance jobs are already being automated, hence requirements in these roles are expected to reduce. 4. Most candidates who find themselves under-skilled do not know what ails them and don t have the right input to improve. This often leads to a vicious circle of despondency and under-performance.

4 The Software Developer Examination It is these challenges that MeritTrac aims to overcome with the performance-based assessment programme Software Developer Examination (SDE) from Proxor, a Carnegie Mellon University Spinoff. SDE is a programme to identify individuals who can successfully and skilfully perform the tasks of a software developer as per global standards. Created in the School of Computer Science at Carnegie Mellon University (CMU), SDE draws from the best of CMU s pedigree. Pioneered by Proxor founder Dr. Philip Miller in 1979, the Authentic Examination TM model rests on the underlying belief that a skill is measured appropriately through a performance-based mechanism only. It is analogous to scoring capability in playing a musical instrument or a sport like swimming. Testing skill not memory At the Software Developer Examination, the examinee is put in the role of a professional software developer, in a real programming environment. He/ she is given a code base, an editor, reference material and asked to do the sort of things you want them to do on the job, like write code, debug existing software, maintain the original software without damaging it. The reference material is, by intent, made available to the examinee in public domain before the test, ensuring there are no surprises or false positives, like in a knowledge-based test. SDE goes beyond providing a coding interface and asking candidates to write simple tasks. It creates a simulated replica of a real software development work environment and tests candidates development skills in it. At the test, the examinee is given an opportunity to demonstrate skills by building solutions to tasks defined in the context of real software projects. It replicates the workplace

5 environment, uses common programming languages, common tools and builds on existing code base. The assessment is not on the examinee s knowledge in coding, but the skill to deliver software programming at an environment similar to a workplace. Over fifty Object Oriented Programming Skills, categorised under data structures, variables and expressions, object oriented programming, object oriented design, exception handling, algorithms and a small part of Java API are examined. To determine what developers were expected to do, Proxor conducted a role delineation study with leading software companies like IBM, Siemens, Honeywell, Accenture, Cognizant, Infosys, HP and Mahindra to build an inventory of typical tasks for the Software Developer Examination. When the exam begins, each developer is given four of those tasks selected by an algorithm which ensures consistency in the distribution of difficulty levels across examinees. The code they produce is then subjected to highly sophisticated test cases (up to 180 for each task) that are constructed by software experts and graded automatically. Moreover, in order to maintain accuracy and the high standards, SDE lays stringent emphasis on security of delivering the examination. The tests are administered in a high stake environment to ensure that the work delivered by the candidate is his/ her own. It is here that MeritTrac s long-standing reputation of blemish-less exam delivery contributes to the success of Proxor s Software Developer Examination in India.

6 Success Story A billion dollar multi-national technology solutions company reaped the benefits of the Software Developer Examination. At the company s campus, MeritTrac administered a dipstick test for fresh hires, each with a qualification of bachelor s degree in engineering or technology (BE/ BTech). Candidates were from tier II engineering colleges, across India. On hiring, these candidates underwent internal training in programming fundamentals, OOP and Java; followed by an internal certification in these areas. As is the normal trend, there was a 10% drop out post-training. The Software Developer Examination, undertaken by these candidates aimed at measuring training effectiveness and project readiness. Examinees used Eclipse as the integrated development environment (IDE) for solving real programming problems. They were asked to write solutions within a set of boundary conditions, to enhance functionality of existing code as listed in each task, without disrupting the existing functionality. The results were both enlightening and useful. At the top most layer, the project successfully ensured that there were no false positives; all candidates rated by the test had the skill to deliver software programming at global standards. This enabled the client to determine the effectiveness of its internal training by benchmarking the outcome on an international platform.

7 At the next layer, it rated candidates on a four-point scale to identify their level of skill, eventually also their suitability for a business function: P1 being most suited for new product development, P2 for high-end maintenance and some new product development, P for maintenance and P4 for testing and low-end maintenance. The distribution of candidates who were rated by Proxor across the grading scale (figure 1) was revealing. Based on these ratings, the client was able to 35 P1 P2 P3 P4 Figure 1: Distribution of grading ofcandidates deploy the right candidate to the right project. At the last level, it also helped identify the training needs of the various candidates on rolls, with high precision, based on the formative inputs, leading to effective training methods to enhance productivity. The client plans to use Proxor s SDE in: Post-training assessment for campus hires. Benchmarking internal workforce. Campus connect initiatives to gain insights into campus hires. Outcomes of SDE SDE offers two fold results: A summative and a formative input. Examinees solutions are evaluated automatically by an extensive battery of automated tests. Automated tests are conducted along three dimensions.

8 Primary: Did the solution satisfactorily address major task elements? Secondary: Did the solution handle known special cases? Tertiary: Did the solution break existing functionality? Marks along these dimensions are aggregated into overall scores for each of the four tasks in a summative report. An overall exam score is reported, as are scores on each of the tasks and the candidate s percentile (i.e. per cent of test taker to date who scored lower than this examinee). The Proxor rating system is as follows. The Proxor Rating P1 Very high confidence that this examinee can develop software at or above globally acceptable entry-level standards Examinee solved every task assigned in the exam with few, if any, defects. Examinee correctly used all data structures, all algorithms acting on those data structures, all matters of scoping and conversion, all elements of object oriented programming and design, and showed facility with Java APIs that were required to solve assigned tasks. High confidence that this examinee can develop software at or above P2 globally acceptable entry-level standards Examinee correctly solved most tasks assigned in the exam, implementing most primary functionality; most secondary issues and preserved most existing functionality in the base system.

9 Examinee correctly used most data structures, most algorithms acting on those data structures, most matters of scoping and conversion, most elements of object oriented programming and design, and showed facility with most Java APIs that were required to solve assigned tasks. Confidence that this examinee can develop software at globally P3 acceptable entry-level standards Examinee solved some tasks assigned in the exam, implementing some primary functionality; some secondary issues and preserved most existing functionality in the base system. Examinee correctly used some data structures, some algorithms acting on those data structures, some matters of scoping and conversion, some elements of object oriented programming and design, and showed facility with some Java APIs that were required to solve assigned tasks. This examinee may be able to develop software at globally acceptable P4 entry-level standards Examinee solved aspects of at least one of the tasks assigned in the exam, implementing some primary functionality, and or some secondary issues, and preserved some existing functionality in the base system. Examinee correctly used some data structures, some algorithms acting on those data structures, some matters of scoping and conversion, some elements of object oriented programming and design, showed facility with some Java APIs that were required to solve assigned tasks.

10 Unrated: Unable to report evidence that this examinee correctly solved any aspects of any of the tasks that were assigned in this exam. The examinee may be able to develop software but Proxor was unable to empirically confirm this. A sample report is given for better understanding. <

11 Mapping Proxor Rating to business function Candidates rated as P1, P2, P3 or P4 are most suitable for the business function as indicated below. New Product Development High-end maintenance and some new product development Maintenance Testing and low end maintenance Figure 2: Proxor rating mapped to business function. Using SDE to optimise investment in human capital Organisations worldwide see value in using certification as a qualifying factor in hiring, training and deploying the right candidate to the right job. Among lateral hires, certification is a standard practice everything from processes (project management) to technologies (Microsoft, CISCO etc.) has a certification program. The role of certification in standardisation, global benchmarking etc. is acknowledged and widely adopted.

12 Software Developer Examination brings the best of certification predictability of productivity and bill-ability to the hiring, training and deployment of a software developer. With talent playing a key role in determining the cost, quality and duration of project delivery, software developer examination offers the opportunity to optimise investment in human capital, across the employment value chain. Better training SDE is especially handy in identifying the skills that candidates lack to enable better training allocation. It helps in an internal benchmarking of the workforce, to keep track of resources and enable better training allocation from time to time. Formative inputs from the certification also contribute towards reduction in cost and duration of training. While on the one hand, it can help learning and development teams with inputs towards creating a more effective training design, it can also help in measuring training effectiveness after its delivery. Smarter deployment The summative report helps identify candidates as most suitable for development, maintenance or testing, matching skill to task perfectly. It enables organisations deploy resources based on the rating and its mapping to business function. For instance, candidates with P3 rating, can directly be deployed to maintenance projects. Better hiring The hallmark of SDE is no false positives, which in itself is at the crux of the Type I error inadvertently made by industry today. A candidate awarded a superior rating in SDE is highly unlikely to be unable develop software. Administering a proctored SDE can help pick the right candidates for the development, maintenance or testing jobs you have vacant.

13 Moreover, it also mitigates the type II error where deserving candidates lose out owing to geographical reach and nature of campus. We foresee that in the medium to long term, organisations would insist on hiring candidates with certifications that can be trusted. This would incentivise academic institutions to include SDE in their curriculum or encourage their students to be SDE certified before appearing for campus recruitment. Therefore, SDE will be seen as a trusted, validated way to identify the right candidates, in both the supply and demand side of talent. Way forward As demand for better resources increases and space for technological innovation grows, the ability to gauge the skill levels of prospective and present employees will be of great differentiating value. More so, when the system that enables such measurement is internationally acclaimed, follows global standards, and brings the inimitable pedigree of the Carnegie Mellon University with it. The Software Developer Examination is the perfect solution across the human capital value chain hiring, training and deployment. The key business value that SDE generates for organisations is in optimising human capital, reduction in cycle time (from on-boarding to deployment) productivity and thus bill-ability. It also enables organisations with the power of predictability in performance, which in turn impacts project estimates of time and cost. Across IT services and products organisations in India, SDE is positioned to create a positive impact in the value chain by utilisation in pre-hiring, pre-training, post-training for fresh hires and pre-training for lateral hires, towards benchmarking the workforce and showcasing it to customers/buyers/prospects.

14 About MeritTrac MeritTrac is India s largest private testing and assessment services company. We provide innovative assessment services to leading educational institutions, corporations and government organisations. We offer high levels of reliability, quality and security for consistently delivering large tests. In addition to computer-based and paper-based tests, MeritTrac offers examination services like online application processing, on screen marking (OSM) and biometric based candidate authentication. We have conducted over 30 million assessments, 13+ million computer-based tests (CBT) through a network of 1100 test centres in 350 cities. We can administer over 300 validated tests for assessing knowledge, skills and personality. We are ISO 9001:2008 and ISO certified. About Proxor Proxor is a Carnegie Mellon University Spinoff formed in 2013 to apply the methods of authentic examination to the problem of assessing professional software developers and job seekers. Proxor s premier assessment product, the software developer examination (SDE), was developed at Carnegie Mellon University by a team of educational and testing professionals lead by Proxor s founder, Dr. Philip Miller. Piloted worldwide by software companies of all sizes and supported by governments, the SDE is quickly becoming the premier software developer examination.

15 For More information on Proxor s Software Developer Examination Contact [email protected]