Package brewdata. R topics documented: February 19, Type Package

Transcription

1 Type Package Package brewdata February 19, 2015 Title Extracting Usable Data from the Grad Cafe Results Search Version 0.4 Date Author Nathan Welch Maintainer Nathan Welch Retrieves and parses graduate admissions survey data from the Grad Cafe website ( License GPL (>= 2) Depends RCurl, XML, stringdist, tools Suggests rgl, scatterplot3d LazyData TRUE NeedsCompilation no Repository CRAN Date/Publication :34:01 R topics documented: brewdata-package brewdata dict error_key findscorepercentile getgradcafedata getmaxpages parseresults parseschools quant_conc_table saw_score_table slang_key svq_score_table translatescore verbal_conc_table

2 2 brewdata Index 17 brewdata-package Package Brewdata Details Brewdata is a package built to lookup, parse, and analyze statistics and biostatistics graduate admissions data reported in the Grad Cafe Results Search. Package: brewdata Type: Package Version: 0.4 Date: License: GPL(>=2) brewdata() is the primary method of this package. It returns a data frame of normalized school names, admissions decisions, parsed undergraduate GPA, GRE scores, and the date of the admissions decision. Author(s) Nathan Welch <nathan.welch@me.com> References Grad Cafe: GRE score guide: #Get data for fall 2015 PhD statistics admission decisions one_yr_data = brewdata() head( one_yr_data ) brewdata Function that Converts Grad Cafe Results Data into Usable Information The brewdata method queries the GradCafe Results Search page for application decision data. It then calls the parseresults function to breakdown the text from "Decision & Date" into values useful for exploring admissions decisions.

3 brewdata 3 brewdata(years = 2015, term = "F", degree = "phd", focus = "statistics", resolution = 10, map=false) Arguments years term degree focus resolution map years specifies which years of data to include in the dataset. The year specifies the time at which an applicant would start school not when he or she applied. So, if someone applied for fall 2014 and was accepted in December 2013 and that person was also thoughtful enough to post his or her metrics on the Grad Cafe s Results Search page, then that record would appear if 2014 is part of the years searched. The four digit year (e.g. 2010, 2012, 1999, etc.) is the only acceptable date format. Inputs may be a single value or a list such as 2010:2015 or c(2011,2013,2015). The default is term indicates which term an applicant would begin graduate study. There are only two acceptable values for this parameter: F and S. F narrows the search to fall matriculations only. S narrows the search to include only the spring term. Users may choose only one value. The default is F. degree determines whether results should be for masters or phd programs. Users must specify exactly one and enclose the value in quotes. masters or phd are the only acceptable values for this field. The default is phd. focus specifies the program. Any term that returns results on the Grad Cafe is acceptable, but brewdata was tuned using statistics or biostatistics. School name mappings could be quite poor for any value other than these two. If you choose a value besides statistics or biostatistics, it is strongly suggested to check the school name mapping by setting map=true. The default value is statistics. resolution is related to the school name parsing algorithm. This variable controls the precision required before an original name is replaced with the best standardized equivalent. Therefore, very low values (between 0-5) are cautious selections leading to fewer mis-matches, but more sparse results. Medium range values (8-12) lead to surprisingly accurate replacements when the mother processing stages fail. One might expect a few mis-matched name replacements, but the number of errors should be fairly low. Finally, large values (more than 20) practically guarantee that a school name which is not in our standard dictionary will be replaced with something. Be weary of such large selections; the potential for many mis-matched replacements is high. For the test set, the bulk of the nearest matchs were within 10 units of the original value. Almost none were larger than 30. The default value is 10. map is a variable controlling whether or not the original school names are included in the data frame returned by brewdata(). If map=true, then the returned data includes the parsed names as well as the original. The default value is map=false. Value brewdata returns a data frame of the parsed Grade Cafe Results data. The data frame includes the following attributes:

4 4 brewdata school_name original_name decision status gpa gre_v gre_q gre_aw v_pct q_pct aw_pct month day year is the closest standardized name matching the name entered at the Grad Cafe. brewdata normalizes the names reported on the website to enable aggregate analysis. See the parse_names parameter description above for more details on the parsing methods. is the original name of the university reported to the Grad Cafe. If map=true, then brewdata includes a column showing the names reported on the website alongside the normalized names assigned by brewdata. This column is excluded by default. denotes a university s decision on an application. Possible decisions are accepted ( A ), wait listed ( W ), rejected ( R ), interview ( I ) or other ( O ). denotes an applicant s immigration status. This field is reported directly from the Grad Cafe. Per the website definitions, possible status values are American ( A ), International with a US degree ( U ), International without US degree ( I ), other ( O ), or unknown (? ). is the self-reported grade point average. is the self-reported GRE verbal section score is the self-reported GRE quantitative section score is the self-reported GRE analytical writing score is the percent of verbal section scores below an applicant s self-reported score. Weighted numeric scores are converted to percentile scores using tables 1A and 1B on page 22 of the GRE score guide. Source: is the percent of quantitative section scores below an applicant s self-reported score. Weighted numeric scores are converted to percentile scores using tables 1A and 1B on page 22 of the GRE score guide. Source: is the percent of analytical writing section scores below an applicant s selfreported score. Weighted numeric scores are converted to percentile scores using tables 1A and 1B on page 22 of the GRE score guide. Source: is the month of the date that an admission decision was made not the date an applicant uploaded the result to the Grad Cafe. is the day of the date that an admission decision was made not the date an applicant uploaded the result to the Grad Cafe. is the day of the date that an admission decision was made not the date an applicant uploaded the result to the Grad Cafe. Note Several specialty university departments are mapped to their parent institutions. For example, Booth, Wharton, and Teachers College are mapped to the University of Chicago, University of Pennsylvania, and Columbia University, respectively. If you are interested in results for such schools, set map=true and use grep() on the original_name column to locate rows of data with the desired department. See below for an example. Author(s) Nathan Welch <nathan.welch@me.com>

5 brewdata 5 References Grad Cafe: GRE Score Guide: See Also findscorepercentile, parseresults, parseschools, translatescore, getgradcafedata, getmaxpages #Get data for fall 2015 PhD statistics admission decisions one_yr_data = brewdata( years=2014 ) head( one_yr_data ) ### Remaining examples commented out to satisfy CRAN policies ### #Get several years of data #yrs=2014:2015 #multi_yr_data = brewdata( years=yrs ); head( multi_yr_data ) #results_by_school = split(multi_yr_data[,-1],multi_yr_data$school_name) #Find 2014 results for Chicago Booth #f14 = brewdata( years=2014, map=true ) #booth = f14[ grepl( "booth", tolower( f14$original_name ) ), ] #booth #Continuing with the f15 & school data, let s analyze results from a particular #school, e.g. University of Washington #uw = f15_by_school$ univ washington ; uw #show all UW decisions #uw_stats = uw[ uw$gre_v!=0 & uw$gre_q!=0, ] #UW decisions with GRE stats #plot( uw_stats$gpa, uw_stats$gre_q, xlab="undergrad GPA", ylab="gre Quant Score", # main="university of Washington GPA vs GRE Quant", pch=na ) #col_key = c( darkgreen, gold, red, black, darkgrey ) #lab = factor( uw_stats$decision, levels=c( A, W, R, I, N ) ) #text( uw_stats$gpa, uw_stats$gre_q, label=lab, col=col_key[lab], cex=0.85 ) #Plot the last two years of Berkeley s GPA/GRE Quant decision trends #yrs=2013:2014 #data = brewdata( years=yrs ); head( data ) #berk = split(data[,-1],data$school_name)$ univ california berkeley #berk_stats = berk[ berk$gre_v!=0 & berk$gre_q!=0, ] #plot( berk_stats$gpa, berk_stats$gre_q, xlab="undergrad GPA", ylab="gre Quant Score", # main="berkeley GPA vs GRE Quant Fall ", pch=na ) #col_key = c( darkgreen, gold, red, black, darkgrey ) #lab = factor( berk_stats$decision, levels=c( A, W, R, I, N ) ) #points( jitter( berk_stats$gpa ), jitter( berk_stats$gre_q ), # col=col_key[lab], pch=20) #lgd=c("accepted", "Wait listed", "Rejected", "Interview", "Not Reported" ) #legend( "bottomleft", legend=lgd, col=col_key, pch=20, bty="n", cex=0.75 ) #Plot several years of results of Duke results using the same data from the #Berkeley download.

6 6 error_key #library( scatterplot3d ) #library( rgl ) #duke = split(data[,-1],data$school_name)$ duke univ #duke_stats = duke[ duke$gre_v!=0 & duke$gre_q!=0, ] #col_key = c( darkgreen, gold, red, black, darkgrey ) #lab = factor( duke_stats$decision, levels=c( A, W, R, I, N ) ) #scatterplot3d( duke_stats$gpa, duke_stats$gre_q, duke_stats$gre_v, # xlab="undergrad GPA", ylab="gre Quant Score", zlab="gre Verbal Score", # main="duke GPA vs GRE Quant vs GRE Verbal Fall ", pch=20, # color=col_key[lab] ) #plot3d( duke_stats$gpa, duke_stats$gre_q, duke_stats$gre_v, # xlab="undergrad GPA", ylab="gre Quant Score", zlab="gre Verbal Score", # main="duke GPA vs GRE Quant vs GRE Verbal Fall ", pch=20, # col=col_key[lab] ) dict University Name Dictionary Format The dict dataset includes the parsed names of many popular statistics graduate programs. It is used to standardize the graduate program names, enabling aggregate analysis. data("dict") The format is: chr [1:111, 1] "arizona state univ" "auburn univ" "baylor univ"... - attr(*, "dimnames")=list of 2..$ : NULL..$ : chr "name" data(dict) error_key Common School Name Error Key The error_key dataset includes several regular expressions used to parse school names. This dataset is used to parse commonly mis-spelled terms (e.g. university). It also maps several specific department names into a standardized university name assigned to targets matching the search patterns. For example, any entry name that includes the term Booth is mapped to the standardized name univ chicago. This dataset supports brewdata s parseschools helper function.

7 findscorepercentile 7 data("error_key") Format The format is: chr [1:37, 1:2] "boston bu " "carnegie mellon cmu" "fort collins"... - attr(*, "dimnames")=list of 2..$ : NULL..$ : chr [1:2] "regex" "corrected" data(error_key) findscorepercentile Function Finding Percentile Rank from Scaled Scores findscorepercent looks up the percent of test takers scorig lower than specified scores using tables 1A and 1B on page 22 of the GRE score guide. This is a brewdata helper function that is available to end users, but not necessary for them to run individually. findscorepercentile( score, section) Arguments score score denotes the post-2011 scaled score (i.e ). section section denotes the GRE section to look up. Acceptable inputs are verbal, quant, and writing. Any other values will not return a result. Value findscorepercent returns the percentile score corresponding to one of the three GRE section (i.e. analytical writing, verbal, or quantitative) scaled scores (i.e. 0-6 and ). References GRE Score Guide: See Also brewdata, parseresults, parseschools, translatescore, getgradcafedata, getmaxpages

8 8 getgradcafedata #Quantitative percentile score findscorepercentile( 160, "quant") #Analytical Writing percentile score findscorepercentile( 4.5, "writing" ) getgradcafedata Function that Downloads Grad Cafe Results Data getgradcafedata retrieves data from the Grad Cafe Results Search pages. This is an internal helper function, so end-users should not need to interact with it directly. getgradcafedata( years, term, degree, focus ) Arguments term years degree focus term indicates which term an applicant would begin graduate study. There are only two acceptable values for this parameter: F and S. F narrows the search to fall matriculations only. S narrows the search to include only the spring term. Users may choose only one value. The default is F. years specifies which years of data to include in the dataset. The year specifies the time at which an applicant would start school not when he or she applied. So, if someone applied for fall 2014 and was accepted in December 2013 and that person was also thoughtful enough to post his or her metrics on the Grad Cafe s Results Search page, then that record would appear if 2014 is part of the years searched. The two digit year (e.g. "10", "12", "06", etc.) is the only acceptable date format. Inputs may be a single value or a list such as c("08","13","15"). degree determines whether results should be for masters or phd programs. Users must specify exactly one and enclose the value in quotes. masters or phd are the only acceptable values for this field. The default is phd. focus specifies the program. Any term that returns results on the Grad Cafe is acceptable, but brewdata was tuned using statistics or biostatistics. University name mappings could be quite poor for any value other than these two. The default value is statistics. Value getgradcafedata returns a data frame of loosely parsed results that satisfy the search parameters.

9 getmaxpages 9 References GRE Score Guide: See Also findscorepercentile, parseresults, parseschools, brewdata, getmaxpages #getgradcafedata( years=15, term="f", degree="phd", focus="statistics" ) getmaxpages Helper Function to Find the Maximum Number of Grad Cafe Focus Pages getmaxpages retrieves the max number of data webpages available for processing. Results Search data is served up to the getgradcafedata function one page at a time. This function ensures that the search for results does not run forever. getmaxpages( url ) Arguments url url is the web address for the Grad Cafe Results page. For example, if the desired search focus is statistics, then the url is Value getmaxpages returns the total number of pages of Grad Cafe Results. References tbd... See Also findscorepercentile, parseresults, parseschools, brewdata, getgradcafedata test = " getmaxpages( test )

10 10 parseresults parseresults Function to Convert "Decision and Date" String to Data parseresults attempts to extract self-reported statistics listed from the Grad Cafe "Decision and Date" text field. The parsing rules below were generated based on test runs using the following parameters in the brewdata method: years=c(2015,2014), term="f", degree="phd", focus="statistics" End-users should not need to interact directly with this function, but improving the rules might improve the fidelity of the brewdata method results. parseresults( result ) Arguments result result is a "Decision and Date" string from the Grad Cafe. Value parseresults returns a data frame with the following fields. gpa gre_v gre_q gre_aw month day year decision is the self-reported grade point average. is the self-reported GRE verbal section score is the self-reported GRE quantitative section score is the self-reported GRE analytical writing score is the month of the date that an admission decision was made not the date an applicant uploaded the result to the Grad Cafe. is the day of the date that an admission decision was made not the date an applicant uploaded the result to the Grad Cafe. is the day of the date that an admission decision was made not the date an applicant uploaded the result to the Grad Cafe. denotes a university s decision on an application. Possible decisions are accepted ( A ), wait listed ( W ), rejected ( R ), interview ( I ) or other ( O ). See Also findscorepercentile, brewdata, parseschools, translatescore, getgradcafedata, getmaxpages #"Decision and Date" string parsing x = "Accepted via on 16 Jan 2015 UG GPA: 4.00GRE General (V/Q/W): 167/170/4.00GRE " parseresults(x)

11 parseschools 11 parseschools Function to Match Error Prone Free-text to Standard School Names parseschools finds best matching school name among several possible spellings & abbreviations. Matches are based on a three stages of parsing: stage (1) standardizes the text by removing common typos and spelling errors, stage (2) manually searches for common name variations for the same school, stage (3) uses an automated text processing algorithm to match the closest school name from a standardized list. parseschools( original_name, resolution = 10, map=false ) Arguments original_name resolution map original_name denotes an Nx1 vector of university names read from the Grad Cafe. resolution controls the precision required before an original name is replaced with the best standardized equivalent. Therefore, very low values (between 0-5) are cautious selections leading to fewer mis-matches, but more sparse results. Medium range values (8-12) lead to surprisingly accurate replacements when the mother processing stages fail. One might expect a few mis-matched name replacements, but the number of errors should be fairly low. Finally, large values (more than 20) practically guarantee that a school name which is not in our standard dictionary will be replaced with something. Be weary of such large selections; the potential for many mis-matched replacements is high. For the test set, the bulk of the nearest matchs were within 10 units of the original value. Almost none were larger than 30. The default value is 10. map is a variable controlling whether or not the original school names are included in the data frame returned by brewdata(). If map=true, then the returned data includes the parsed names as well as the original. The default value is map=false. Value school_name is the name of the university corresponding to the row of data. parseschools normalizes the names reported on the website. See Also findscorepercentile, parseresults, parseschools, translatescore, getgradcafedata, getmaxpages

12 12 quant_conc_table x = c( "university of california--berkeley","university of california--berkly", "uc berkeley", "berkeley" ) parseschools( x ) quant_conc_table Table Used to Convert Pre-2011 Quantitative Scores to Current Scaled Score translatescore uses this dataset to convert pre-2011 GRE scores to the current scale using the concordance tables 1D and 1E on pages of the GRE score guide. data("quant_conc_table") Format A data frame with 61 observations on the following 3 variables. old Pre-2011 GRE scaled score, i.e new Current GRE score scale, i.e pct Percent of scores below the associated input Source GRE Score Guide: data(quant_conc_table)

13 saw_score_table 13 saw_score_table Analytical Writing Score Section Table findscorepercent uses the saw_score_table (Score/Analytical Writing Table) to look up the percent of test takers scorig lower than a specified scores using tables 1A and 1B on page 22 of the GRE score guide. data("saw_score_table") Format A data frame with 13 observations on the following 2 variables. score Current GRE score scale, i.e. 0-6 aw Percent of scores below the associated input Source GRE Score Guide: data(saw_score_table) slang_key Table of Common or Shortened University Names parseschools uses this table to try matching a nickname to the formal university or department name. This table corresponds to the second stage of processing. data("slang_key") Format The format is: chr [1:384, 1:2] "a&m" "alberta" "asu" "au" "baylor" "bc"... - attr(*, "dimnames")=list of 2..$ : NULL..$ : chr [1:2] "slang" "name"

14 14 translatescore data(slang_key) svq_score_table Verbal-Quantitative Reasoning Section Score Table Format Source findscorepercent uses the svq_score_table (Verbal-Quantitative Section Score Table) to look up the percent of test takers scorig lower than specified scores using tables 1A and 1B on page 22 of the GRE score guide. data("svq_score_table") A data frame with 41 observations on the following 3 variables. score Post-2011 GRE scaled score, i.e v Verbal score percentile ranking q Quantitative score percentile ranking GRE Score Guide: data(svq_score_table) translatescore Function to Convert Pre-2011 GRE Scores to the Current Scale translatescore converts pre-2011 GRE scores to the current scale using the concordance tables 1D and 1E on pages of the GRE score guide. translatescore(old_score, section)

15 verbal_conc_table 15 Arguments Value old_score old_score pre-2011 quantitative or verbal section score ( ) section section "verbal" or "quant" variable indicating which table to use translatescore returns a GRE score corresponding to the current scale ( ) for the specified section, either quantitative or verbal. References GRE Score Guide: See Also findscorepercentile, parseresults, parseschools, brewdata, getgradcafedata, getmaxpages translatescore( 710, "quant" ) translatescore( 710, "verbal" ) verbal_conc_table Table Used to Convert Pre-2011 Verbal Scores to Current Scaled Score translatescore uses this dataset to convert pre-2011 GRE scores to the current scale using the concordance tables 1D and 1E on pages of the GRE score guide. data("verbal_conc_table") Format A data frame with 61 observations on the following 3 variables. old Pre-2011 GRE scaled score, i.e new Current GRE score scale, i.e pct Percent of scores below the associated input Source GRE Score Guide:

16 16 verbal_conc_table data(verbal_conc_table)

17 Index Topic \textasciitildekwd1 brewdata, 2 findscorepercentile, 7 getgradcafedata, 8 getmaxpages, 9 parseresults, 10 parseschools, 11 translatescore, 14 Topic \textasciitildekwd2 brewdata, 2 findscorepercentile, 7 getgradcafedata, 8 getmaxpages, 9 parseresults, 10 parseschools, 11 translatescore, 14 Topic datasets dict, 6 error_key, 6 quant_conc_table, 12 saw_score_table, 13 slang_key, 13 svq_score_table, 14 verbal_conc_table, 15 Topic package brewdata-package, 2 parseresults, 5, 7, 9, 10, 11, 15 parseschools, 5, 7, 9 11, 11, 15 quant_conc_table, 12 saw_score_table, 13 slang_key, 13 svq_score_table, 14 translatescore, 5, 7, 10, 11, 14 verbal_conc_table, 15 brewdata-package (brewdata-package), 2 brewdata, 2, 7, 9, 10, 15 brewdata-package, 2 dict, 6 error_key, 6 findscorepercentile, 5, 7, 9 11, 15 getgradcafedata, 5, 7, 8, 9 11, 15 getmaxpages, 5, 7, 9, 9, 10, 11, 15 17