Listening t Students: Sample Fcus Grup and Survey Materials Overview The Student Assessment Inventry fr Schl Districts supprts a prcess t evaluate the assessments students currently take and then determine the minimum testing necessary t serve essential diagnstic, instructinal and accuntability purpses. The prcess shuld als lead t actins that ensure that every district-mandated assessment is f high quality, prvides the infrmatin needed fr specific schl and district purpses, and is supprted by structures and rutines s that assessment results are actually used and actin steps that will help students are taken. While the success f this prcess rests n several factrs, ne f the mst critical is the extent t which district leaders intentinally incrprate the perspectives and experiences f students. Students have unique perspectives thrugh their direct experiences taking tests. Their perspectives are essential fr taking the temperature n the assessment envirnment in the district and building a case fr actin. Clearly cmmunicating t students (and by extensin, their parents) the imprtance and urgency f this effrt will help ensure mmentum leading t actin. Additinally, students experiences with assessments are fundatinal t understanding the assessments themselves, hw the assessment results are being cmmunicated and understd, and what can be dne t make individual assessments and the district s assessment prgram as a whle mre effective and cherent. This resurce is designed t guide districts in the steps necessary t design and use fcus grups and surveys t incrprate student perspectives, experiences and expertise in the inventry prcess. It includes tw types f infrmatin. Instructins. Achieve recmmends fllwing a set f steps t design, build, use and analyze student surveys and fcus grups t effectively incrprate this infrmatin in the inventry prcess and build tward actinable recmmendatins. A set f guidance and resurces (beginning n page 2) addresses cmmn questins abut hw and when t use student fcus grups and surveys within the inventry prcess. It als includes design cnsideratins district leaders shuld keep in mind as they develp these feedback instruments, use infrmatin gathered frm students, and cmmunicate findings t participants and stakehlders. Districts experienced in designing fcus grups and surveys may feel cmfrtable skipping this sectin and mve directly t the prtcls, beginning n page 6. Inventry Tls. This resurce prvides student fcus grup and survey questins that district leaders can use r adapt as part f their lcal assessment inventry prcess. Taking the Temperature n Assessments (beginning n page 6) fcuses n illuminating student perspectives n the district s assessment prgram as a whle t build a greater understanding f the testing envirnment and CC BY 4.0 Achieve 2015. Districts may use r adapt. If mdified, please attribute Achieve and re-title.
help build a case fr actin. Depending n district cntext, leadership teams may decide t use fcus grups r surveys t illuminate student perspectives, experiences and expertise. As such, this guide includes questins apprpriate fr each f these frmats. Instructins: Guidance and Resurces This guidance is intended t assist district leaders in develping a prcess fr using student fcus grups and/r surveys as part f the assessment inventry prcess. 1. DETERMINING DISTRICT NEEDS FOR STUDENT FEEDBACK BASED ON THE REFLECT AND PLAN QUESTIONS IN THE STUDENT ASSESSMENT INVENTORY FOR SCHOOL DISTRICTS At the utset f the assessment inventry prcess, district leaders shuld address the Reflect and Plan questins n page 4 f the Student Assessment Inventry fr Schl Districts. These questins ask districts t think abut the cntext, bjectives and structure f the inventry prcess. District leaders are then encuraged t develp a strng understanding f student perceptins f testing thrugh the Taking the Temperature n Assessments fcus grup and survey prtcls beginning n page 6. 2. DECIDING WHETHER TO USE A FOCUS GROUP, SURVEY OR BOTH Districts have multiple issues t cnsider when determining whether t use fcus grups r surveys t gauge student perceptins. Fcus grups allw fr additinal prbing and fllw-up questins. On the ther hand, fcus grups are generally mre cstly than surveys t rganize, staff and analyze, and they require access t skilled fcus grup mderatrs. Surveys allw districts t gather infrmatin frm a large number f students in a relatively shrt perid f time, and answers can be easily aggregated. Students may als feel mre cmfrtable cmpleting a survey than talking in a fcus grup. Hwever, surveys d nt allw fr additinal prbing questins, and respnse rates may be lw, especially if a survey takes t lng t cmplete. Districts are encuraged t carefully weigh these issues in light f resurce and timeline cnsideratins, as well as lcal cntext and needs. 3. DESIGNING A SUCCESSFUL FOCUS GROUP OR SURVEY PROCESS There are multiple design cnsideratins fr fcus grups and surveys. These cnsideratins include fcus grup and survey administratin, participatin, setting and design. The fcus grup and survey prtcls included in this resurce are penly licensed, and districts are encuraged t add and/ r adapt questins and prtcls as needed based n lcal pririties and cntext. Belw, fcus grup and survey design cnsideratins are described separately. Fcus Grup Design Cnsideratins Wh shuld administer a fcus grup? District leaders can chse t administer their wn fcus grups r wrk with an utside rganizatin with experience cnducting fcus grups. If district leaders decide t keep fcus grups in huse, it is imprtant that fcus grup mderatrs are cmfrtable speaking with students, have a high level f rapprt with participants, and are peple with whm students are likely t feel they can be pen and hnest. If 2
district leaders have limited time r capacity fr cnducting fcus grups, they can cnsider using the expertise f lcal institutes f higher educatin, cmmunity-based rganizatins fcused n educatin and cnsultants with fcus grup expertise. Wh shuld participate in a fcus grup? District leaders shuld make every effrt t recruit students with diverse perspectives and schl experiences fr fcus grups. Districts shuld strive t select a representative sample f students acrss multiple criteria (e.g., grade level r span; schl; r student subgrup, such as English language learner r special educatin). District leaders shuld als ensure that fcus grups are demgraphically representative f the cmmunity, taking int cnsideratin characteristics such as race/ethnicity, gender, neighbrhd, etc. Districts shuld als cnsider hw best t gather meaningful data frm different age grups. Questins may need t be significantly different fr high schl, middle schl and elementary schl students. Additinally, district leaders shuld carefully cnsider the number f fcus grups that will be needed t get a detailed picture f a district s assessment landscape, as well as the number f participants that can be reasnably accmmdated in any ne fcus grup. T allw all participants multiple pprtunities t cntribute t the cnversatin, it is suggested that fcus grups nt exceed 10 individuals. It is als recmmended that the fcus grup mderatr be accmpanied by a dedicated nte-taker s that all respnses can be recrded. What are the best pprtunities t cnduct a fcus grup? Districts r third-party rganizatins can use a range f frmal and infrmal pprtunities t gather feedback thrugh fcus grups. There may be already-scheduled events where large numbers f students will be present, such as back-t-schl nights, which can be cnvenient and minimally disruptive fr participants. Gathering infrmal feedback thrugh small ad-hc fcus grups is anther key pprtunity, especially fr districts with limited resurces r shrter timelines fr the inventry prcess. Hw can districts ensure that student fcus grups are diverse and representative f the student ppulatin? Reaching a representative sample f students fr fcus grups presents unique challenges fr districts. Districts can develp a set f strategies prir t engaging in fcus grups. First, districts are encuraged t partner with cmmunity-based yuth rganizatins t reach ut t students frm underrepresented grups. Secnd, when develping fcus grup participatin lists, districts shuld cmpare the demgraphics f student participants t district enrllment infrmatin t ensure that a representative grup f students has been reached when initial runds f fcus grups are cmpleted. Districts shuld cntinue t mnitr participant demgraphics thrughut the prcess t strive fr representativeness. Finally, districts shuld cnsider translating fcus grup prtcls int additinal languages spken by students and having translatrs lead r assist with student fcus grups where needed. What additinal safeguards shuld be in place when gathering infrmatin frm students? Districts r external rganizatins shuld be aware f federal, state and district plicies that prtect students right t privacy. Districts may want t cnsider guaranteeing annymity t fcus grup and survey participants. 3
Survey Design Cnsideratins Hw shuld surveys be used t generate the mst useful infrmatin? District leaders shuld address several cnsideratins as they design surveys as part f the inventry prcess. Chse nly the questins that are mst likely t generate the infrmatin needed. In priritizing sme questins ver thers, be sensitive t the time needed t cmplete the survey. Chse questin types that best match needs and capacity t analyze data. A variety f questin types are included in these sample surveys, all with different implicatins fr data analysis. Fr instance, pen-ended questins have the benefit f enabling respndents t prvide their persnal feedback, but they can be time cnsuming t analyze when lking at hundreds f respnses. Cnversely, multiple-chice respnses are very easy t cde fr data analysis but d nt typically give respndents an pprtunity t prvide in-depth feedback. After identifying the questins that wuld be mst helpful, edit them as needed t ensure that they fit the lcal cntext and will resnate with respndents. Phrases in brackets within the template indicate places where the language needs t be custmized r remved prir t being released publicly. These areas shuld be updated with the apprpriate infrmatin. Decide whether t ask respndents t prvide the name f their schl r ther identifying infrmatin as part f this survey. Districts and external rganizatins shuld fllw federal, state and lcal laws safeguarding students privacy. Prir t implementing the survey, try ut the survey items n a small number f ptential survey participants t get their feedback n the questins asked, clarity f questins and length f time the survey tk. Wh shuld administer a survey? Surveys can be administered by a district r third-party rganizatin. District leaders may want t administer their wn surveys if they have prir experience develping and administering them, have had gd respnse rates with previus surveys, and/r have sufficient capacity and expertise. District leaders may want t cnsider using an external, third-party rganizatin t administer a survey if sufficient resurces exist t hire an utside firm with expertise, the district has little experience designing and administering its wn surveys, and/r privacy and annymity f results are significant cncerns. Wh shuld participate in a survey? Depending n district size, resurces and capacity, district leaders can chse t either survey all students (census survey) r select a representative sample f students acrss multiple criteria (e.g., grade level r span; schl; r student subgrup, such as English language learner r special educatin). Districts shuld als strive t cllect surveys frm a demgraphically representative sample f students, taking int cnsideratin characteristics such as race/ethnicity, gender, neighbrhd, etc. (see belw). If initial survey results are nt representative f the district s student demgraphics, districts shuld cnsider targeting additinal individuals frm underrepresented grups with fllw-up survey requests. Hw can districts ensure that survey respnses are frm a diverse and representative sample f students? Similar t fcus grups, reaching a representative sample f students thrugh a survey presents unique challenges fr districts. First, districts shuld cnsider translating student surveys int additinal languages spken by students. Secnd, if the 4
survey is paper-based, districts can use a range f frmal and infrmal pprtunities t gather survey respnses, in much the same way as fcus grups described abve. Fr example, hmerm/advisry classes are ptential pprtunities t have students cmplete surveys. Fr nline surveys, districts shuld be aware that nt all students have internet access and prvide multiple pprtunities fr example, at schls and libraries fr students t cmplete surveys. Third, districts may want t cnsider partnering with cmmunity-based yuth rganizatins t reach ut t students frm underrepresented grups. Finally, when initial runds f surveys are cmpleted, districts shuld cmpare the demgraphics f student participants t verall student demgraphics t ensure that a representative grup f students has been reached. 4. NEXT STEPS: ANALYSIS, USE AND COMMUNICATION OF RESULTS After districts have cnducted all student fcus grups and/ r surveys, infrmatin gathered frm these surces can be analyzed and incrprated int the Inventry Table (see pages 8 and 9 f the Student Assessment Inventry fr Schl Districts). It is recmmended that districts als cmmunicate results f the fcus grups and surveys t participants (and depending n timing and cntext, the public) t demnstrate that students vices are being heard in the inventry prcess. Analyzing and using results. Fcus grups and surveys are part f the inventry prcess and will help infrm the Inventry Table, verall analysis and recmmendatins. Please refer t pages 5 6 f the Student Assessment Inventry fr Schl Districts. Analyzing the respnses may raise additinal questins abut student perceptins and experiences and the impact f assessments n students. Fr example, d particular assessments mentined by students fall within a specific grade level r grade band, r are these assessments taken by a specific subgrup f students? Are there cncerns abut the amunt f testing in particular grades r subjects? This infrmatin frm students may als help infrm the district abut cmmunicating the results f particular tests t the brader cmmunity, as well as infrm systemic needs arund assessment and data literacy. Cmmunicating the results t participants and the public. It is imprtant that district leaders cmmunicate t fcus grup and survey participants that their respnses and suggestins are being heard. Examples f cmmunicatin t participants culd include a letter t participants with a summary f what was learned thrugh the sessins r survey, an invitatin t a district r bard meeting during which results will be discussed, and a timeline fr mving frm feedback t recmmendatins t actin. District leaders may als want t cnsider releasing a summarized set f findings frm the fcus grups and surveys t the brader public as an interim step during the inventry prcess. Additinal Resurces These additinal resurces may be helpful in designing and administering surveys and fcus grups. Achieve (2012). Feedback Lps fr Cmmn Cre Standards Implementatin: Survey Tls fr State Educatin Leaders. www.achieve.rg/files/guidancefrsurveysfinal6-25-12- TOSHAREv2.pdf Duke Initiative n Survey Methdlgy. http://dism.ssri.duke.edu/resurces.php Pew Research. Questinnaire Design. www.peple-press.rg/methdlgy/questinnaire-design 5
Taking the Temperature n Assessments This sectin, which includes sample fcus grup and survey questins, fcuses n illuminating students perspectives n the district s assessment prgram as a whle t build a greater understanding f the testing envirnment. It is recmmended that these questins be asked f students tward the beginning f the prcess, prir t the Inventry Table being cmpleted. Cnducting fcus grups r administering surveys early in the prcess helps make the case fr why an inventry prcess is needed and will lead t cncrete actin t ensure that every assessment being used is useful and f high quality. The prtcls and questins listed belw are penly licensed, and districts are encuraged t add and/r adapt questins and prtcls as needed based n lcal pririties and cntext. TAKING THE TEMPERATURE ON ASSESSMENTS: SAMPLE FOCUS GROUP QUESTIONS INTRODUCTION AND WARM-UP A warm-up perid fr fcus grups is imprtant t help participants understand the gals fr the sessin, create a cmmn understanding arund hw participants and the mderatr will interact, and build cmfrt amng participants and with the mderatr. Belw are a suggested set f grund rules and an intrductry exercise, fllwed by sample fcus grup questins. Nte t mderatr: Prir t cnducting this fcus grup, it is imprtant t be clear with participants abut the types f assessments fr which the district wuld like feedback. Will there be discussin nly f assessments that are given acrss multiple classrms r schls as part f a district/ schl assessment prgram, r is the district als interested in teacher-develped classrm assessments? This shuld be discussed as part f the intrductin and warm-up. Add and/ r adapt questins t best meet yur lcal needs and cntext. Intrductin f mderatr and fcus grup guidelines: > We re ging t be talking abut testing in yur schl. This will help the district better understand the vlume, quality and use f tests given in the district. > The district will lk at all f its testing thrugh the perspective f students. It is imprtant t understand hw different kinds f students experience testing. That is why we have invited yu t participate in these fcus grups. 6
> This fcus grup is part f a larger prcess t examine many kinds f tests given t students. Thrugh this prcess, the district will make decisins abut which tests are high quality and necessary fr students t take and which tests can be mdified r eliminated. > One persn shuld speak at a time. > There are n wrng answers t any f these questins. We are interested in hearing yur perspectives as students. > We value cnfidentiality. We will nt assciate any feedback that cmes ut f this fcus grup with a particular individual. > Everyne will have a chance t speak. If yu have nt had an pprtunity t prvide yur perspective, I may call n yu. > Please turn ff r silence yur cell phnes. > Are there additinal guidelines the grup wuld like t add? Disclsure f nte-taking: > A nte-taker is present t make sure that we get all f yur feedback. The nte-taker will nt assciate cmments with names. Parking lt: > There is a parking lt chart/bard/area where we can put ideas r thughts that cme up in ur discussin that are imprtant but may nt be related t the purpse f this grup. We want t capture thse imprtant thughts, but we als want t keep fcused n the purpse f ur meeting. These ideas r thughts will be shared with apprpriate individuals fllwing the meeting. Cmmunicating results f fcus grups: > The results f the fcus grups will be summarized, and yu will receive a summary by {date}. If we missed any key pints yu raised during this cnversatin, please let us knw. Intrductry exercise (pssible ptins): > Types f tests given t students > Examples f tests given in the district > What questins d yu have abut testing befre we begin? Intrductry exercise (pssible ptins): > Hw lng have yu lived in this cmmunity? > What kinds f activities are yu invlved in utside f schl? > What d yu like t d fr fun? 7
FOCUS GROUP QUESTIONS 1. What grade are yu in? Which schl d yu attend? 2. What kinds f tests have yu taken in the last year? (Mderatr prmpt: tests designed by yur teacher fr just yur class [e.g., chapter tests, mid-terms, finals], tests all students in yur schl take [e.g., schlwide writing prmpts, district benchmark tests, statewide tests {e.g., PARCC, Smarter Balanced}], tests yu take fr cllege entrance r hnrs [e.g., AP, IB, ACT, SAT]) 5. What are sme examples f tests that yu have nt fund useful in understanding what yu ve learned and what yu need t d next? Why have they nt been useful? (Mderatr prmpt: yu dn t get the results fr a lng time r at all, test measures material nt cvered in yur class, is nt helpful in measuring readiness fr cllege r career) 6. Hw d yu think yur teachers use the results frm the tests yu take? Hw d they talk t yu abut test results? Hw d yu talk abut them with yur parents/family? 3. D yu think students in this district are given t many tests, nt enugh tests r abut the right number f tests thrughut the schl year? Why d yu say that? 4. What are sme examples f tests that yu have fund useful in understanding what yu ve learned and need t d next? What has made them useful? (Mderatr prmpt: yu get the results quickly and they are clear, test measures imprtant material cvered in class, test helps measure readiness fr cllege r career) 7. What tests, if any, d yu think yu shuld keep taking? Why d yu say that? 8. What tests, if any, d yu think yu shuld nt keep taking r shuld be changed? Why d yu say that? (Mderatr prmpt: yu d nt have enugh time t take the test, t much time is spent taking the test, tests shuld include different kinds f questins) 9. D yu have any ther suggestins n hw t imprve the tests yu take? 8
TAKING THE TEMPERATURE ON ASSESSMENTS: SAMPLE SURVEY QUESTIONS Sample language t intrduce the survey: This survey asks questins abut tests that yu take. This survey will help yur schl and yur district make gd decisins abut testing. Yur respnses t this survey are cnfidential we d nt ask fr yur name and will nt ask fr any persnal infrmatin. If yu have questins abut this survey, please cntact{ }. Nte t survey designer: Prir t administering this survey, it is imprtant t be clear with respndents abut the types f assessments fr which the district wuld like feedback. Is the primary fcus n assessments given acrss multiple classrms r schls as part f a district/schl assessment prgram, r is the district als interested in teacherdevelped classrm assessments? The district shuld als explain hw results f the survey will be used, whether district stakehlders will be presented with a summary f findings and hw these findings will be used t infrm actin. 1. What grade are yu in? {district shuld preppulate with grades} 2. D yu think students in yur schl are given t many tests, nt enugh tests r abut the right number f tests thrughut the schl year? Nt enugh tests Abut the right number f tests T many tests 3. What cncerns r wrries d yu have abut the tests yu take? {pen-ended item} 4. What are sme examples f tests that yu have fund helpful in understanding what yu re learning and what yu need t d next? {the district can preppulate this item with a list f assessments given in the district if knwn} 5. In general, what abut these tests make them helpful? Cnnected t what I am learning in the classrm Results are available sn after taking the test Scres and infrmatin prvided t me are easy t understand Prvides ideas n hw t help me imprve in my classes Results shw me hw my scres cmpare t the scres f ther students in my schl, district and/ r state Other 9
6. What are sme examples f tests that yu have nt fund helpful in understanding what yu re learning and what yu need t d next? {the district can preppulate this item with a list f assessments given in the district if knwn} 7. In general, what abut these tests makes them nt helpful? 8. Hw d yu think yur teachers use the results frm the tests yu take? Hw d they cmmunicate with yu abut test results? {pen-ended item} 9. Hw d yu talk with yur parents/family abut test results? {pen-ended item} I d nt understand the cnnectin t material I am learning in class I d nt receive results quickly Scres and infrmatin prvided t me are difficult t understand Des nt give me ideas n hw t help me imprve in my classes Other 10. What tests, if any, d yu think yu shuld keep taking? {pen-ended item OR district can list specific assessments} 11. What tests, if any, d yu think yu shuld nt keep taking? {pen-ended item OR district can list specific assessments} 12. Are there any ther suggestins yu want t make abut tests that yu take? {pen-ended item} 10