PROJECT TITLE: SMARTER Balanced Assessment CONSORTIUM Section A INTRODUCTION pg. 5 2. Purpose of Request for Proposals third bullet PROPOSAL CONTENTS 2.3 Determine Standard-Setting Design pg. 21 Q1. Page 5 of the RFP the third bullet states, Determine Standard-Setting Design and collect data for it. RFP page 21 describes what will be required to develop a standard setting design which addresses design tasks. Is the contractor expected to conduct the actual standard setting study, or just design the standard setting study? Answer: This contract does not include running the standard setting. There is a separate RFP and contract for that purpose. (See Procurement document on SBAC website.) The contractor will design the standard setting study. Design of the process will require working closely with the Standard Setting contractor and complying with joint methods developed with PARCC, if any. Section A INTRODUCTION pg. 8 6. Vendor Qualifications Minimum Qualifications Bullet 11 Q2. Please clarify the term Senior Project Manager. Does that refer to the program manager or the psychometrician (subject matter expert)? Answer: This position is a project manager role, not a psychometrician role. It may be the case that the Senior Project Manager has a psychometric background or training, but the duties required and experience desired is that of a Project Manager. Q3. Does this request ask that the Senior Project manager be full time Vendor employee or dedicated full time to the project? Answer: The Senior Project Manager needs to be dedicated full time to the project. Section C PROPOSAL CONTENTS pg. 16 5. Technical Requirements Overview Q4. Which grades will be tested in High School? Answer: For Summative Assessments, grade 11 will be tested with the adaptive test and performance tasks. The performance portion of the summative test can also be administered at grades 9 and 10. Any grade may be assessed in the interim and formative systems. 1
Q5. What grades are to be included in the vertical scale? Answer: Grades 3-8 and High School will be covered by vertical scales developed for SBAC assessments. Q6. The RFP states that summative assessments will comprise a computer adaptive testing (CAT) component and a performance task component. Answer: Yes. Q7. What is the expectation for scaling the performance tasks? Answer: This is regarded as an open question. The Test Design contractor will be expected to recommend whether PT results should be calibrated to construct scales and combined with others results to produce scale scores for assessments. If they are not to be associated with the IRT scales, the contractor will be responsible for establishing the comparability of performance tasks (within and across years for the same grade) in some other manner. Although the psychometrics and Test Design contractors will be expected to lead their respective assignments of work, they are expected to collaborate on the recommendations. Q8. Is it expected that the performance tasks will be on the same scale as the computer adaptive component (pre-cat phase of the field test)? Answer: See response to Q7. Section C PROPOSAL CONTENTS pg. 18-19 2.1.2 Special (fixed) forms Q9. Approximately how many items, fixed forms, and performance tasks per content/grade will be administered during the pilot test? Answer: About 15,000 total items will be tested in the pilot phase. Some of these will be rejected. Some areas of the pool will end up with inadequate distribution. The exact number of performance tasks has not yet been determined. However, because of security concerns, a large number of performance tasks need to be available so that students would not know which of many possible performance tasks they may get in a given year. This means that 30 to 40 tasks need to be developed for each summative operational test. During the pilot phase, performance templates will be tested and range finding will be conducted for a core of items. The Consortium welcomes advice and/or recommendations for the design features of the pilot tests that bidders are encouraged to provide. 2
Q10. What is the expected number of new items and new performance tasks to be administered per content/grade during the field test? Answer: The field test data will be used to calibrate the remaining items for operational summative and interim pools. Given attrition during the pilot phase, and the need to complete pool distribution demands, we expect 20,000 to 25,000 items to be tested in the field test phase. During the field test phase, the remaining performance tasks will be field tested, including range finding and comparability studies, which will be conducted for all items needed for summative and interim systems. Section C PROPOSAL CONTENTS pg. 19 2.1.4 Twice Annual Contractor Meetings Q11. Is the sum indicated to cover travel for the contractor s staff or for SBAC staff, or both? Also, please confirm that we should include a total of $60,925 for the entire contract period for this travel (not $60,925 per year). Answer: SMARTER Balanced has determined that it would make for better fiscal accounting if the Consortium address the travel needs of its members for these twiceannual meetings, so the $60,925 are to be excluded from the bid. All other travel associated with execution of a contraction should be borne within the cost proposal provided (and within the $3.5M budget), inclusive of both contractor staff and SMARTER Balanced members. The cost of representation at the twice-annual meetings will also be the responsibility of the contractor and are to be borne within the available budget of $3.5M. Representation of the Psychometrics Service Provider at these twice-annual meetings is expected to be a minimum of three people at XX meetings (the actual schedule to be determined during the course of contract fulfillment). Q12. Does SBAC have an expectation or preference for the meetings to be conducted face-toface or electronically? Answer: Most contact will be completed electronically. The twice-annual meetings will be face-to-face Consortium-level meetings scheduled such that all workgroups and contractors can meet in a conference-like setting. Additionally, it is typically advisable to schedule 1 or 2 face to face meetings as a kick-off process. In addition, the contractor will be expected to present in-person to the Technical Advisory Committee (TAC). Q13. Please clarify what the vendor allocation of $60,925 should include? For example, should that budget include: vendor travel, SBAC travel, meeting costs (meals, audiovisual, conference room rental, etc.)? Answer: Refer to Q11. 3
Q14. Does SBAC have a preference as to the location of these meetings? Answer: The Consortium will establish the location, likely to be either Denver or Las Vegas. Section C PROPOSAL CONTENTS pg. 19 2.2 Develop Technical Manuals Q15. Should the contractor plan on delivering one (1) technical manual (after field testing, but discusses pilot testing as well) or two (2) technical manuals (one after pilot testing and one after field testing)? Answer: See 16. Q16. Does the Consortium expect to publish one technical report at the end of each program year, or does it prefer another plan/configuration for technical reports? Answer: There will be a technical manual for each program year. The final year s manual will be a comprehensive document that covers all years and the entire process. Section C PROPOSAL CONTENTS pg. 21 pg. 22 2.3 Determine Standard-Setting Design Q17. Does the consortium want vendors to include text and costs for conducting standard setting meetings based on results of research and discussions with the consortium? Answer: See Q1. Section C PROPOSAL CONTENTS pg. 22 2.4 Determine Vertical Scale Design Q18. The RFP states that based on the results of the dimensionality study and the Consortium s theory of action a single vertical scale or multiple scales may be developed in each content area. Please clarify the meaning of multiple scales as used in this context. Does that mean two or more vertical scales that each span the full grade range, or does that mean that there may be vertical scales that span subsets of grades (e.g., one vertical scale that spans grades 3-5, and another that spans grades 6-8)? Answer: Empirical analysis of pilot data will be used to establish the number and nature of scales. 4
Section C PROPOSAL CONTENTS pg. 23-24 2.6 Develop Pilot Test Item and Task Data Review Materials Q19. How many data review meetings are expected? Answer: Vendors should submit a plan that includes a combination of online and faceto-face meetings. Because of the volume of items to be reviewed, it is likely that most meetings will be conducted online or through a distributed system. Face-to-face meetings would be limited to training or general orientation. Q20. Should vendors include costs for the meeting logistics (hotel and food)? Answer: YES, but also keep in mind the response to Q19, when building the proposal. Q21. How many people will be attending the Data Review meetings? Answer: No fewer than three; no more than six, but see response to Q19 for possible distributed or virtual meetings. One will be a psychometrician from within the Consortium membership whose has data-review experience; others will be content experts, mostly from member states, sufficient in number to ensure meaningful verbal exchange. This is exclusive of the needed participation of the contractor. Q22. Does SBAC have a preference as to the location of these meetings? Answer: Consortium analysis has found that Denver and Las Vegas tend to be less expensive options depending on the members that are attending. Section C PROPOSAL CONTENTS pg. 24 2.7 Conduct Psychometric Analysis to Support Pilot Test Data Review Q23. The RFP states that the VENDOR will describe the linking design used to demonstrate task consistency. Answer: If it is determined that performance tasks are not scaled, it will be necessary to demonstrate that each task has comparable difficulty so that students do not get an advantage or disadvantage because of task selection. If they are scaled, the calibration process would be assumed to account for any differences in difficulty. Q24. Please clarify the meaning of consistency in this context. Answer: For performance tasks, as for other task types, consistency means that inferences derived from scores are comparable. If performance tasks are scaled, scores would be placed onto a common scale, which would take differences into account. If they are not scaled, it means that points awarded at each level of common rubrics are at the same level of difficulty across tasks. 5
Section C PROPOSAL CONTENTS pg. 24 2.7 Conduct Psychometric Analysis to Support Pilot Test Data Review Exhibit D pg. 39 Q25. Please clarify the following variation between two RFP statements: Section 2.7 states... because of SBAC s commitment to transparency, no proprietary software solutions should be bid. Exhibit D states, the Contractor hereby grants a nonexclusive, royalty-free, irrevocable license to use computer programs (defined as materials ) for everything that is not produced under this agreement but that incorporates pre-existing materials. If materials are not considered works for hire, the Contractor must irrevocably assign all right, title and interest in materials, including all intellectual property rights, to the Agency effective from the moment of creation of such materials. The required use of third-party open source software prohibits compliance with Paragraph 11, RFP page 39 of the Contract. Please advise as to the legal requirements. Answer: If the product is already licensed under an open-license, then no additional license need be granted by the contractor Section C PROPOSAL CONTENTS pg. 25 2.8 Conduct Analysis of Pilot Test Q26. Does every pilot test form need to be full-length, meeting the specifications in terms of the number and proportion of items by content area (excluding any items used for vertical scaling)? Answer: Current Consortium thinking is that some of the forms need to be full length and meet operational blueprint specifications. These will be used to pilot the test experience and to provide dimensionality data. Pilot forms need to yield adequate item/examinee information, so they need to be close to the same length as operational tests. However, some forms could deviate from operational specifications as long as the overall desired proportions of items is maintained so that resulting pools support the test design. Nevertheless, the Consortium welcomes advice and/or recommendations for the design features of the pilot tests that bidders are encouraged to provide. In other words, this question embodies the kind of advice the Consortium is seeking from a Psychometrics Service Provider. If the Consortium can confidently answer all questions of this sort at this time, there is really no need for this contract. 6
Q27. Can OSPI clarify what is meant by Survey of Pilot Test participants? Answer: This means design and analysis of a survey of test administrators and students to examine the usability, test experience, clarity of instructions, responsiveness of test delivery software, technical ease of use, look and feel and anything else that could affect test delivery. The Test Delivery contractor will deliver the survey and collect results. The Psychometrics Service Provider will analyze results and report on them. Q28. Does this mean, for example, administering questions to students about computer use with demographic indicators (such as gender and ethnicity)? Or, a survey (cognitive lab) of the students response processes that is meant to gain validity evidence about whether the fit between the construct and the processes that students employ to produce responses align well (concerning what the items measure)? Answer: The survey mentioned here is described above. Cognitive labs conducted with tasks and items are conducted as part of Item/Task Writing and Review-Pilot (Procurement Bundle #14). Section C PROPOSAL CONTENTS pg. 25 2.9 Design Item and Student Sampling Plans for Field Test and Select Anchor Items and Tasks for Calibration and Building the Vertical Scale Q29. The RFP states that Following Field testing, all items will be recalibrated using only Field Test responses. The last bullet on the same page states that the contractor will Recalibrate items on field test responses only as needed. Please clarify the expectation for recalibration of items that were calibrated during the pilot test, and for new or revised items that were calibrated during the initial precomputer adaptive test phase of the field test. Answer: The expectation is that items calibrated during the pilot phase will retain their parameters. Items would be recalibrated only if parameters appear to have shifted after field test administration. Section C PROPOSAL CONTENTS pg. 27 2.13 Present to TAC meetings as required Q30. The RFP states Required meetings may vary, but could include TAC meetings, Executive Committee meetings, general membership meetings, and meetings with the U.S. Department of Education. Are the mentioned required meetings in addition to the four annual meetings TAC meetings? 7
Answer: Bidders should anticipate six required meetings per year, which would comprise some combination of TAC, USED, Executive Committee, or other unanticipated meeting. These would be in ADDITION TO the twice-annual workgroup/contractor collaboration meetings and any meetings the contractor requires to meet its contractual deliverables. Q31. Should vendors assume 4 TAC meetings annually, plus potential ad-hoc meetings with USDOE, Executive Committee, etc, or should Vendors assume a maximum of 4 (2 day) meetings per year comprised of different meeting types? Answer: See Q30. Q32. Can SBAC please suggest a location for these meetings? Answer: Not at this time; this is difficult to predict for a 29-state consortium ranging from Maine to Hawaii and for meetings some of which are not yet determined. Unless piggy-backed to other meetings Consortium members are attending, as was stated in Q14 & Q22, SMARTER Balanced has found Denver or Las Vegas providing the best cost efficiencies. Q33. For meetings with SBAC general membership and the Executive Committee, is SBAC agreeable to the use of virtual meeting methods such as conference calls, web-ex, etc.? Answer: Yes, this is a modality currently used. However, this does not extend to the meetings described in Q30. Q34. Please confirm that the contractor is only responsible for its own staff travel for TAC meetings, and not any other meeting costs or SBAC/TAC member travel costs. Answer: Confirmed. Section E RFP Exhibits pg. 36-37 Exhibit C Service Level Requirements and Remedies Q35. Exhibit C Service Level Requirements states: All work products posted or displayed as Final and available for use and consumption by others are without error. The word posted implies posted to a website. Will SBAC provide a website to post final documents or is the website a vendor responsibility? Answer: SMARTER Balanced will have the website intended for final product posting; this is not a contractor responsibility Q36. If documents are posted to an SBAC or Vendor site, and that site is down for technical maintenance (planned or unplanned outages), would the penalty clause (RFP page 36) associated with All work products posted or displayed as final and available for use be waived? Answer: Yes. 8
Q37. Can you provide greater clarity on what is meant by substantial errors? Answer: Substantial would be errors that are of an egregious nature or such number that would be embarrassing for all parties concerned to have distributed for review or reading. Q38. Can you provide greater clarity on what is meant by thorough quality control review? Answer: Due to the persistence of errors SMARTER Balanced will be obliged to determine that the Contractor was not thorough in its review of the materials, not only looking for typographical or formatting mistakes, but also the inclusion of errors of a technical nature that were provided through feedback by the Consortium. Q39. Are the monetary penalties set forth in this Exhibit C negotiable? Answer: Penalties may be negotiable, but SMARTER Balanced believes it has set fair rates for the contract. Q40. Would the Consortium be willing to accept an annual cap on the monetary penalties set out in Exhibit C? Answer: No. Q41. Does SBAC plan to create a website or central location for the sharing or collaboration of documents? Answer: If referring to the concept of posting of final materials, refer to Q35. If referring to work-in-process sharing, the means of sharing would likely be worked out with the winning vendor. Q42. Will the documents be stored on this site or should it only be used for the exchange of materials? Answer: Because of the nature of SMARTER Balanced membership, storage would be a preferable model, not just an exchange site. Q43. Please provide more details such as, will this repository include secure access via passwords? Answer: The site for work-in-process sharing will need to have limited access to the involved parties. The final, more public site will be open for viewing of the appropriate documents. Be used by others vendors? Answer: The sharing site will be limited from other vendors, outside what collaboration is needed between contractors to fulfill the entirety of the SMARTER Balanced project. 9
Be used for other SBAC contracts? Answer: If SMARTER Balanced provides the solution for the work-in-process sharing site, then yes the site will be accessible for other contract uses. The Consortium would strive to maintain the needed firewalls between contract efforts where appropriate. Section E RFP Exhibits pg. 40-41 Exhibit D Attachment A, Contract for Personal Services, General Terms and Conditions Q44. Are the terms under Section 18, Indemnification negotiable? Answer: Possibly, to the extent permitted by the laws and statutes of the State of Washington, fiscal agent for the Consortium. Section E RFP Exhibits pg. 47 Exhibit E Contractor intake form Q45. Is the Contractor/Governmental Entity Intake Form a mandatory requirement for each identified subcontractor? Answer: Subcontractors are not required to complete this form. OSPI does have final approval of all subcontractors related to the contract. Please refer to the General Terms & Conditions section 33 Subcontracting. Neither the Contractor nor any subcontractor shall enter into subcontracts for any of the work contemplated under this contract without obtaining prior written approval of the Superintendent. In no event shall the existence of the subcontract operate to release or reduce liability of the Contractor to the Superintendent for any breach in the performance of the Contractor s duties. This clause does not include contracts of employment between the Contractor and personnel assigned to work under this contract. General Questions Q46. What is the Consortium s expectation for individual state participation in the pilot and field tests? Answer: Member states are encouraged to participate. Broad participation is expected. Participation is not required. The Psychometrics Service Provider will advise the Consortium on the demographics and levels of participation necessary to ensure stable item calibrations and analyses. Q47. Will all governing states participate in the pilot test? Answer: See Q46. 10
Q48. Will all governing states participate in the field test? Answer: See Q46. Q49. Will the advisory states participate in the pilot or field tests? Answer: See Q46. Q50. It appears that there is no item writing associated with this proposal is this true? Answer: There is no item writing associated with this contract. Q51. Will an item writing proposal be coming from your office in the future? Answer: Please refer to the posted Master Work Plan. 11