TABLE OF CONTENTS Section 1: Letter of Submittal Letter of Submittal Certifications and Assurances (Exhibit A) Informational Attachment Section 2: Technical Proposal A. Project Approach/Methodology... 2-1 Overview... 2-4 1.0. Application Requirements... 2-8 2.0. Application Development and Implementation... 2-12 3.0. Ongoing Application Housing and Maintenance... 2-25 4.0. Item Authoring and Item Pool Application Software... 2-32 5.0. Records of Decision Making... 2-44 6.0. Annual Contractor Meetings... 2-46 B. Work Plan... 2-46 C. Project Schedule... 2-52 D. Deliverables... 2-58 E. Outcomes and Performance Measurement... 2-61 F. Risks... 2-62 Section 3: Management Proposal A. Project Management... 3-1 A1. Project Team Structure/Internal Controls... 3-1 A2. Project Management Deliverables... 3-8 A3. Staff Qualifications/Experience... 3-22 B. Experience of the Consultant... 3-31 C. References... 3-73 D. Contractor Intake Form... 3-77
Section 4: Cost Proposal A. Identification of Costs... 4-1 Appendices Appendix A: Functional and System Requirements Item Authoring and Item Pool Application Appendix B: Compliance Item Authoring and Item Pool Application Appendix C: Resumes
2. TECHNICAL PROPOSAL A. PROJECT APPROACH/METHODOLOGY Introduction Pacific Metrics is pleased to submit this proposal to customize its off-the-shelf (COTS) webbased item authoring and item pool application (IAIP) to support the SMARTER Balanced Consortia s item banking, item authoring, and development activities. Our proposal outlines our ORCA Development environment and the customizations that will be made to satisfy the SBAC functional and system requirements. ORCA Development is an item management platform that allows for scalable online assessment content editing and management and is provided through an intuitive Web-based interface. It allows test developers to construct content that is aligned to content standards. ORCA Development provides a versatile, flexible, and secure means of developing, managing, and exporting test content to the test delivery platform. The ORCA Development application was a passion project for Pacific Metrics, and exemplifies our mission to bring lasting improvements to the assessment and learning environment through the thoughtful use of technology. The application was initially developed as an internally-focused product with the simple goal of streamlining the migration of content to online test delivery systems. Years of internal collaboration and innovation between our assessment specialists and our software engineers resulted in a comprehensive IAIP application that now provides extensive assessment development support, including: Seamless integration with test delivery platforms Robust import/export of content Improved quality and consistency of content Support various item types Customizable workflow Provide management reporting and tools increased productivity Pacific Metrics understands the demands placed on a technology system meant to support a wide variety of user roles and user types across a broad geographic area. We ve provided OTS or COTS implementations of our ORCA Development application to leading companies in the assessment industry, all of whom had unique requirements related to system functionality. Throughout the technical proposal, we refer to SBAC s customized version of ORCA Development application as SBAC s IAIP application. Company Overview Pacific Metrics was founded over a decade ago with the mission to bring lasting improvements to educational practices and problems through the introduction of new technologies. We are essentially an educational R&D company with specialization in psychometric and assessment software, focusing solely within K-12 assessment. The company is based in the U.S., and all our staff (including engineers and software developers) are also U.S. based. Our software 2-1
engineers and program staff have developed technical assessment systems for classrooms, students working from home, and centrally administered secure summative tests. In the past decade, working with educators and IT professionals from the school level to the state department of education, we have architected, developed, deployed and operated large scale assessment platforms for the first statewide practice tests, statewide formative assessment, high stakes summative assessments, and experimental research programs. Our practice tests included the first Spanish audio capability in 2002. These and our summative tests include automated scoring of student responses to ELA, math and science items. Now in their third generation, our platforms have shown the flexibility and extensibility to deliver graduation tests to students displaced to a dozen states following Hurricane Katrina, deliver the highly innovative and interactive science and algebra ONPAR field tests, and successfully incorporate innovations in accessibility, item content, and automated scoring, working with four states in an Enhanced Assessment Grant (EAG). We have routinely administered millions of assessments to hundreds of thousands of students in classroom-based and summative assessment environments, giving us experience across the entire assessment landscape from item development and content portability, to online student registration systems, to fast-turnaround scoring and reporting systems that blend human and machine scoring. We have written and deployed an online content development environment, designed, deployed and continue to maintain a state summative item bank accessed by multiple vendors, written interfaces and organizational designs for hand and machine scoring systems, maintained the official accountability reporting site for the State of Louisiana, and written the official state online testing decision tool for Council of Chief State Officers (CCSSO): http://www.ccsso.org/resources/digital_resources/online_computer- Based_(Testing)_Decision_Making_Tool.html Through this considerable breadth of programmatic experience and success in the assessment field, Pacific Metrics has learned a great deal about the integration of technology-based assessment systems with psychometrics and educational policy, including accountability. We believe to be unmatched in this skill set by any other organization or vendor. These accomplishments have only been possible due to the excellence of our research and engineering staff and the strong organization around our development processes imposed by project management practices. Pacific Metrics IT development practices emphasize the need for information systems to function in human organizations at multiple levels, incorporate requirements from multiple sources, and operate in tandem with other technologies. As mentioned in the introduction, Pacific Metrics proposes to develop a COTS version of its ORCA Development content authoring environment to fulfill SBAC s needs as outlined in the Technical description. This solution will be consortium-owned, and throughout the proposal we will refer to it as SBAC s IAIP application. Online assessment system development is an area of robust growth within the larger educational industry. Pacific Metrics is focused on developing innovative, leading-edge technologies to support the demands of the assessment world. Acknowledging that one of SBAC s key technology priorities is the utilization of leading edge technologies, the system Pacific Metrics will customize for SBAC s IAIP application employs dynamic, cutting-edge functionalities such as the creation of Technology Enabled Item content, support for dozens of world languages, and seamless publication to a variety of delivery systems while also remaining flexible enough to adapt to the changes inherent within the assessment industry. 2-2
The SBAC IAIP provides a scalable online assessment content authoring, editing, and management system provided through an intuitive web-based interface. It includes a user friendly HTML editor for authoring and editing test content, along with a fully functional equation editor and other flexible data and content display tools. The SBAC IAIP supports XHTML and also features real-time reporting and customizable printing capabilities to support content development needs across multiple states and organizations. Users are able to create multiplechoice, constructed response, and essay items with easy-to-use HTML editing tools in an endto-end assessment process in a fully online environment that includes the development, publication, administration, reporting, and analysis of tests and test results. SBAC s IAIP application allows test developers to construct content that can be delivered through multiple test delivery systems. It provides a versatile, flexible and secure means of developing, managing, importing and exporting test content with many benefits including: Configurable process integration and workflow Automation of time consuming and potentially error prone processes Standardization and quality management of test development processes Editorial processes for approvals Management and end-to-end publication of test content Pacific Metrics will provide the following services and deliverables for the complete and successful implementation of SBAC s IAIP application. Table 2-1: Components of the Project Services Description 1 Application Requirements Functional Requirements Technical Systems Requirements 2 Application Development and Implementation Application Design Application Development Quality Assurance and Testing Data Portability Item/Task, Stimuli, and Multimedia Import and export modules Application Deployment Training Materials 3 Ongoing Application Hosting and Maintenance Maintenance and Support Application Hosting Knowledge Transfer, Transition, and Turnover Products Description 4 Item Authoring and Item Pool Application Software with full ownership by SMARTER Balanced 2-3
Overview of Services and Products: General Approach Services 1 Application Requirements (Functional and Technical) Pacific Metrics will begin the customization process by first gathering and refining the Functional and Technical Requirements. Functional Requirements serve dual purposes by providing both high-level business requirements and detailed software requirements. We plan to utilize an Agile iterative development process to customize SBAC s IAIP application, and as such requirements are developed and refined as part of a specific iteration. Pacific Metrics will begin the functional requirements gathering process with the initial system requirements outlined in Appendix A of the RFP, and will work closely with SBAC and other stakeholders to ensure all requirements are clearly identified. The Functional Requirements document will remain the main artifact of the requirements development process and will be maintained as iterative development is completed. Pacific Metrics will also deliver a detailed Technical Systems Requirements document as part of the overall requirements effort. This document will include information relating to the following areas: Infrastructure and Client Requirements Location and Interoperability Requirements Security Requirements Performance and Scalability Requirements Availability, Reliability, and Redundancy Requirements Services 2 Application Design, Development, and Implementation As noted above, Pacific Metrics plan to implement the Agile method for customizing SBAC s IAIP application. This approach offers several benefits. Agile software development practices utilize a process of continuous planning and feedback to ensure that value is always maximized throughout development. Iterative planning and feedback allows for continuous review of and alignment to the system requirements, and ensures that changes are easily accommodated. By measuring and evaluating status based on working software, much more accurate visibility into the actual progress of projects is available. Finally, as a result of following an agile process, at the conclusion of a project is a software system that most effectively addresses SBAC s needs. The design of SBAC s IAIP will be defined predominantly through interviewing key stakeholders of the system and receiving feedback and direction from the designated SBAC work group. Given the iterative process, any concerns or issues are found quickly in the process and can be addressed directly. Typical areas of consideration surrounding system design might include hardware and software architecture, specific technologies implemented to address key functional or technical requirements, and coding methodologies. 2-4
Quality Assurance and Testing Pacific Metrics has an experienced Quality Assurance (QA) team, offering a substantial depth and breadth of knowledge of diverse software systems. QA team members work closely with all members of the software development team to ensure a thorough understanding of not just the SBAC s IAIP application system functionality, but also of its intended uses and the variety of user types and roles the system will need to support. Our software quality assurance process is supported and enforced through the highest quality tools available to ensure consistent, reliable, and repeatable software releases. The QA team participates in a workflow-based problem tracking system that automatically assigns issues to the designated software engineers as they are submitted. This significantly increases the efficiency with which issues are identified, reviewed, and resolved. Pacific Metrics will develop and submit a Quality Assurance Plan that provides insight into the processes used to review SBAC s IAIP application. The plan will include information on processes such as testing methodology, unit testing, system testing, and User Acceptance Testing. User Acceptance Testing (UAT) is a significant component of both Pacific Metrics testing methodology as well as the Agile software development process. Throughout the development lifecycle of SBAC s IAIP application, SBAC staff will engage in UAT at the conclusion of each iteration. This process signals the final stage of testing performed on a specific part of the application prior to releasing it to a live environment. This testing is a critical component of development, as it will allow SBAC to ensure the application meets the requirements. Data Portability Pacific Metrics understands that SBAC s IAIP application must have flexible, robust capabilities with regards to data portability. To that end, our COTS solution offers seamless import from and export to multiple sources, which ensures that users from all member states can easily perform this function. Our technology team brings considerable experience in the area of batch content import, having worked on multiple large-scale assessment projects involving the import of thousands of items and their accompanying stimuli, graphics, metadata, and statistical information directly into the COTS application from a third party source. This system, customized for SBAC, will support the import of all content and associated information and stimuli. The data import process is streamlined and efficient, and offers users smart feedback to identify any items or attributes that may be outside a valid range, or may be labeled incorrectly. This tool effectively minimizes risk to content validity and ensures the import process returns the desired and expected results. Training Pacific Metrics understands that training is a critical component of the success of software systems. SBAC faces additional challenges due to the number of potential system users, their geographic diversity, and their varying levels of technology proficiency. By providing a variety of training materials, tools, and options, Pacific Metrics will ensure that users from all member states have the knowledge they need to use SBAC s IAIP application efficiently and effectively. We propose utilizing several different training strategies to support the transition to SBAC s IAIP application. These include: 2-5
User Guide provides an overview of the application, including structure and technical specifications. Training Guides provide in-depth descriptions of specific system functionalities, as well as best practices for use. Online Demonstration Site provides a separate version of SBAC s IAIP application that mimics the functionality of the live system. Self-Learning Tutorials provides online step-by-step walk-through of system functionality covered in the Training Guides, with audio voice-over. Application Deployment Pacific Metrics understands that we will need to provide all necessary hardware, software and applications according to the schedule outlined in the RFP. At the conclusion of this application deployment schedule, Pacific Metrics will deliver to SBAC an IAIP application ready to support mass data import, and end-user access to all required system functionalities, including all necessary hardware and software. Services 3 Ongoing Application Hosting and Maintenance/Support Pacific Metrics recognizes that the SBAC s IAIP application provides crucial support for all activities related to content authoring and maintenance for SBAC s assessments. As such, we will address the need for ongoing maintenance and support of the system in the following areas. Help Desk and production application support Pacific Metrics will provide three distinct levels of escalation for technical issues related to the production system, so that all issues are resolved as quickly as possible. Production fixes for issues ( bugs ) identified with completed features For SBAC s IAIP application, Pacific Metrics will support production fixes through a patch release process, prioritizing based on severity. Release of application enhancements after the initial software release A formal process will be used for releasing full versions of SBAC s IAIP application for both the initial release, as well as for any subsequent releases that may be required throughout the course of the contract. Monitoring and system maintenance of the application within the hosting environment As a component of hosting services, Pacific Metrics will provide regular monitoring and application maintenance. A sophisticated alert system and a skilled team of system maintenance engineers ensure expeditious responses to any system issues. Document updates related to production fixes and additional features. Pacific Metrics places the highest priority on communication with the client regarding any software fixes or upgrades that may occur throughout the course of a contract. We will provide detailed documentation to SBAC regarding updates made to the production version of the IAIP application. 2-6
Application Hosting Given the high-states nature of the data to be stored in the IAIP application, it is essential to utilize a Tier 1 solution. The COTS system Pacific Metrics will customize for SBAC was architected to address security issues related to the system and the data it manages. We utilize a robust IT infrastructure and world class Tier 1 hosting facility that will house SBAC s IAIP application. At this facility, all customer hardware is placed in specifically designated cages to which only authorized staff members have access. The data center provider enforces several levels of security measures, including palm scans, verification of government issued identification, and access cards. Knowledge Transfer, Transition, and Turnover As a leading software provider that specializes in application for the assessment industry, Pacific Metrics has extensive knowledge and experience in the area of management and maintenance of production software. This includes the transition of these responsibilities to an identified third party at the conclusion of a program or contract. To ensure the smooth transfer of SBAC s IAIP application to SBAC at the conclusion of the contract, Pacific Metrics will develop a thorough Transition Plan that outlines the specific activities involved in this transition. A Technical Systems Manual, addressing issues related to the managing and maintenance of the IAIP application, is another key component of this transfer that Pacific Metrics will provide. Using these two documents as a foundation, Pacific Metrics will also provide targeted, hands-on training to identified staff to impart a concrete knowledge of the system s technical functionalities. Recognizing that some support will be required after the initial transition of SBAC s IAIP application, arrangements can be made for Pacific Metrics to provide Level 3 professional support provided by phone or through online meetings. Products 4 Item Authoring and Item Pool Application Software Since its inception in 2000, Pacific Metrics has garnered a reputation for developing intuitive, creative, flexible software solutions for the assessment industry. The COTS product Pacific Metrics proposes for the SBAC was initially created as an internal solution to address the need to develop, store, and deliver digital content seamlessly to online testing platforms. Through an ongoing collaboration between content specialists, psychometricians, and software engineers, the IAIP application grew from a straightforward content authoring system to a full-scale assessment development application used by multiple companies and hundreds of individual users. Because the system is designed by assessment specialists, for assessment specialists, the functionality it offers demonstrates attentiveness and deference to the unique needs of this specific industry. The IAIP application supports the development, review, storage, import, and export of both traditional and non-traditional item types and formats developed for a variety of purposes, across all grades and content areas. This includes all related stimuli and various graphic types. Content is stored and managed in distinct banks, allowing specific user permissions to be set for selected content and projects. 2-7
EXPECTED WORK STATEMENT Here we provide a complete description of our approach and methodology for each of the components identified in the RFP. 1. APPLICATION REQUIREMENTS 1.A Functional Requirements Pacific Metrics will construct a functional requirements document that provides detailed specifications for the Item Authoring and Pool system. As is highlighted in Section 2 of the Technology Proposal (Application Development and Implementation), we plan to use an Agile iterative process to develop the application, in which requirements are refined as part of each iteration. Recognizing this fact, we will develop an initial requirements document prior to beginning iterative development, but will refine the document as features (user stories) are completed as part of each iteration. The Functional Requirements document will remain the main artifact of the requirements development process and will be maintained as iterative development is completed. The functional requirements document defines both high level business requirements and detailed software application requirements for the software system. The requirements are functional in nature, defining observable system behaviors. The product of this effort is a functional requirements document that helps drive the Agile development process. It helps to define design, implementation, validation, documentation, and deployment activities that are completed as part of each software development iteration. Details of the Functional Requirements Document As part of a software development effort, Pacific Metrics works closely with a customer to establish and define the features and functions needed for a software application. Pacific Metrics captures and defines these requirements in a Functional Requirements Document for the software application. Major components of the requirements document include the following: 1. Introduction a. Purpose of the document b. Scope of the requirements definition c. Definition of requirements vocabulary d. Reference materials e. Terms and acronyms 2. Overview - High level overview of the application being developed 3. Assumptions and Dependencies - Assumptions made in the requirements definition and dependencies of the requirements on external factors 4. Requirements Sections - A functional decomposition of the software application into one or more levels, in which the lowest level constitutes functional requirements for a given feature. A feature could be akin to a user story in Agile development or a Use Case in use case analysis. Feature requirements can include the following: a. User interface requirements 2-8
b. Screen Mockups c. Business logic requirements d. Use case & flow diagrams e. Technical input and output requirements related to the feature, or references to these types of requirements defined in the Technical Systems Requirements document. f. Non-functional and performance requirements related to the feature, or references to these types of requirements defined in the Technical Systems Requirements document. Requirements Development Process As highlighted earlier, since an Agile development process will be employed to develop the Item Authoring and Pool Application, functional requirements will be refined as part of the iterative development process. This means that requirements development is not fully completed prior to the beginning of software development, rather requirements are defined at a higher level up front and elaborated as part of the iterative process. We anticipate that the requirements will be refined as part of the Agile development, but that the Functional Requirements Document will reflect these changes. During initial requirements development and during the iterative development process, several activities are expected to be completed related to requirements development. These activities include the following: 1. Requirements Discovery - Working with relevant stakeholders to discover requirements for the given application or feature. 2. Requirements Definition - Defining these requirements in the form of functional requirements and possible associated user stories and acceptance criteria. 3. Requirements Review - Reviewing requirements with relevant stakeholders to ensure they reflect the needed functionality. 1.B Technical Systems Requirements Pacific Metrics, using the SMARTER Balanced IT Systems Architecture and its guidelines, will deliver a detailed Technical Systems Requirements document as items are discovered through the Requirements Gathering Phase. The document will cover, but is not limited to, the following areas: Infrastructure and Client Requirements Location and Interoperability Requirements Security Requirements Performance and Scalability Requirements Availability, Reliability, and Redundancy Requirements 1.B.1 Infrastructure and Client Requirements Pacific Metrics will define infrastructure and client requirements that define the minimum supported hosting and client machine environment to successfully run the application. This can include, but is not limited to: Hardware: CPU, memory, disk space Software: operating system, library/framework and language dependencies 2-9
Network: bandwidth, protocols, ports Database: database management system type, version, and configuration 1.B.2 Location and Interoperability Requirements Pacific Metrics will define location and interoperability requirements that define the ability of the system to overcome problems brought about by the location of its elements and the distances between them. This can include, but is not limited to: Public interfaces to the system Open communication methods (web services, etc.) Standard data interchange formats (APIP, SIF) 1.B.3 Security Requirements We understand that one of SBAC s key technology priorities is system and data recoverability. As such, our customized IAIP application for SBAC is designed with maximum data recoverability in mind. The system has been engineered to enable multiple methods of backup recovery. The system employs the ability to recover from a point in time backup and has the flexibility to be backed up manually as well as automatically to fulfill user requirements. Pacific Metrics will define security requirements that define the ability of the system to reliably control, monitor, and audit what entity can perform what actions on resources and the ability to detect and recover from failures in security mechanisms. This can include, but is not limited to: Authentication Authorization Monitoring Auditing Encryption On-site security 1.B.4 Performance and Scalability Requirements Pacific Metrics will define performance and scalability requirements that define the ability of the system to predictably execute within its mandated performance profile and to handle increased processing volumes. This can include, but is not limited to: Response times Throughput Peak load behavior Distributed processing 2-10
1.B.5 Availability, Reliability, and Redundancy Requirements Pacific Metrics will define availability, reliability, and redundancy requirements that define the ability of the system to be fully or partially operational as and when required and to effectively handle failures that could affect system availability. This can include, but is not limited to: Planned/unplanned downtime Clustering Load balancing Disaster recovery We acknowledge that open licensing is a fundamental requirement of SBAC s IAIP application as indicated in the key technology priorities. In the development of the system, we recognize that the classes of technologies which are incorporated will depend on those that have appropriate Open License models. While some fundamental components of the system may require Closed or Commercial solutions, there must be the ability to trade off technologies in the architecture such that Open Sourcing of the IAIP application can eventually be achieved. 2. APPLICATION DEVELOPMENT AND IMPLEMENTATION Agile Development Approach The efforts and process Pacific Metrics plan to incorporate for the successful delivery of SBAC s requirements will involve the practices around Agile software development and the Agile Manifesto. Twelve principles underlie the Agile Manifesto, including: Customer satisfaction by rapid delivery of useful software Welcome changing requirements, even late in development Working software is delivered frequently (weeks rather than months) Working software is the principal measure of progress Sustainable development, able to maintain a constant pace Close, daily co-operation between business staff and developers Face-to-face conversation is the best form of communication (co-location), however tools such as video conferencing and GoToMeeting will be leveraged for distributed teams Projects are built around motivated individuals, who should be trusted Continuous attention to technical excellence and good design Simplicity Self-organizing teams Regular adaptation to changing circumstances 2-11
Figure 2-1: General Overview of the Agile Development Process The Pacific Metrics method of Agile will break tasks into small increments with less formal planning, and do not directly involve long-term planning. Iterations are short time frames (timeboxes) that will last four weeks. Each iteration will involve a team working through a full software development cycle including planning, requirements analysis, design, coding, unit testing, documentation and acceptance testing when a working product is demonstrated to stakeholders. This minimizes overall risk and allows the project to adapt to changes quickly. An iteration may not add enough functionality to warrant a market release, however the goal is to have an available release (with minimal defects) at the end of each iteration. Multiple iterations may be required to release new features. Team composition will be cross-functional and self-organizing without consideration for any existing corporate hierarchy or the corporate roles of team members. This means that team members take responsibility for tasks that deliver the functionality an iteration requires, and they decide individually how to meet an iteration's requirements. Recognizing the formal documentation required by the project, Pacific Metrics will still emphasize regular and consistent communication over written documents. The Pacific Metrics Project Manager will need a dedicated SBAC representative for the life of the program by acting as a single point of contact for managing and directing customer decisions needing to be made and provided to Pacific Metrics. This person must be determined by SBAC prior to project kickoff to act on SBAC s behalf and must make a personal commitment to being available for the Pacific Metrics team to answer such things as mid-iteration problem-domain questions. At the end of each iteration, stakeholders and the SBAC representative will schedule a time to review progress and re-evaluate the priorities with a view to optimizing SBAC s return on investment (ROI) and ensuring alignment with customer needs and company goals. If change to the scope is requested by SBAC it will go through a formal Change Request process requiring sign-off. Furthermore, the Pacific Metrics Program Manager will deliver biweekly reports to SBAC s point of contact that will give clear visibility to status of hours, the progress of scope completed, and projected timelines. As the process of Pacific Metrics accepts 2-12
change throughout the lifecycle of SBAC s project, this bi-weekly report will be a living/breathing document and will therefore change over time to accommodate any change provided by SBAC s dedicated point person. Lastly, Pacific Metrics will use a routine and formal daily communication among team members called a Scrum meeting. This specifically includes the customer representative and any interested stakeholders as observers. In a brief session, team members report to each other what they did the previous day, what they intend to do today, and what their roadblocks are. This level of communication exposes problems as they arise. Pacific Metrics emphasizes working software as the primary measure of progress. This, combined with the preference for regular communication, produces less written documentation than other methods. This method encourages stakeholders to prioritize wants with other iteration outcomes based exclusively on business value perceived at the beginning of the iteration (also known as value-driven). 2.A Application Design Because Pacific Metrics follows Agile Design practices, our software designs usually do not go through a large and resource-intensive design phase that may, or likely not, meet all of the stakeholder s requirements. Pacific Metrics development process focuses on delivering working, useful software in iterations, so design is mainly emergent and evolves with the software (because of the nature of Agile development). However, this allows the system design to emerge over time, permitting it to take advantage of new technologies and fulfill new requirements. There are a range of Agile Design practices that Pacific Metrics follows, done in a collaborative nature with stakeholders of the system. This is detailed in the following figure: Figure 2-2: Agile Design Practices Architecture Modeling: modeling (usually light) at the beginning of a project to identify critical architectural issues Iteration Modeling: modeling at the beginning of each iteration to identify a strategy for that iteration 2-13
Solution Modeling: modeling to think through an aspect of the solution Test-first Design: write a test and enough domain code to fulfill the test Refactoring: small changes to a part of a solution, improving quality without changing the semantics Continuous Integration: automatically compile, test, and validate components of the solution whenever a component changes Whenever design work is being completed the Architect or Engineer uses the appropriate tools to accomplish the job without hindering the effort. Whiteboards, index cards, and diagramming tools can all be used to work out and communicate the solution to stakeholders. They also consider system qualities (non-functional or technical system requirements) concerns such as hardware and software architecture, technologies used to meet defined functional and technical system requirements, coding methodologies, programming languages, and component/objectbased decomposition of the system. Pacific Metrics will deliver software design documents and technical specifications, where appropriate (e.g., interoperability) that affect the software implementation as driven by the architecture. The design documents can include, but are not limited to: User Stories Domain models Flow diagrams Enterprise integration models Entity relationship models Class diagrams Logical and Physical Data models We understand that system flexibility is one of SBAC s key technology priorities. Pacific Metrics recognizes the flexibility required in a system meant to be utilized across states and by users with a variety of proficiency levels and technology resources. The SBAC IAIP application will be customized in a flexible and modularized fashion which will benefit SBAC by minimizing the programming required to make future modifications or customizations. 2.B Application Development As detailed in Section 2, Agile Development Approach, Pacific Metrics creates the software implementation iteratively going through each phase of a typical system development lifecycle. This method constructs the system as fully functional software that satisfies the business and technology requirements in phases while adhering to the defined software design in a flexible manner. Pacific Metrics has deep knowledge in software engineering practices, thus our development approach has a variety of tasks that engineers are responsible for: Acquiring and installing systems environments Creating and testing databases Preparing test files Coding Compiling, building and deployment Unit-testing 2-14
Load-testing Acquiring and Installing Systems Environments Pacific Metrics software engineers have the knowledge and expertise to acquire, install, and configure a variety of system environments, application servers, etc. We have expertise in industry standard environments, and use some of the most widely used web-based languages, frameworks, and systems, such as: Apache HTTPD server Java SE and Java EE technologies (EJB, JPA, JMS, etc) Apache Tomcat JBoss Application Server Glassfish Application Server Web Services (SOAP and REST) PHP Perl Creating and Testing Databases Our engineers have significant expertise in creating highly-available and resilient database solutions with both MySQL and Oracle database management systems, clustering, redundancy, and data integrity. Preparing Test Files Pacific Metrics software engineers build and create test files during development for consistent and reliable feature development and testing. Test files are included as part of integration work and becomes part of the project and build process. Coding Engineers follow coding methodologies and standards in order to keep a healthy, readable, and maintainable code base. Engineers also use a set of standard development tools as part of their efforts, including (but not limited to) the Eclipse IDE with Java EE and Web Application Development support, Subversion code management client, and Test Track Pro issue management client. Compiling, Building and Deployment Our software engineers follow best practices and tools for compiling, building and deploying software. This includes using industry-standard tools like Ant and Maven to build and distribute software artifacts. This facilitates the use of Continuous Integration software, particularly Jenkins, so that the software remains in a stable state and issues are discovered quickly. In addition, our engineers have the knowledge and experience setting up repositories for sharing deployable java artifacts like JAR, WAR, and EAR files, and the expertise in creating linuxbased software repositories for distributing software like RPM (Red Hat) and DEB (Debian and Ubuntu) files. Unit-Testing Pacific Metrics engineers employ unit testing as part of their development efforts, particularly the XUnit testing frameworks, and specifically JUnit for Java development. This allows us to continually refactor our software while maintaining consistent behavior. 2-15
Load-Testing Apache JMeter is used regularly after software builds in order to assess software and application performance and capacity. It also allows us to check our systems and catch performance issues that may have been introduced during the development process. The software implementation deliverables will include, but is not limited to: Software product Development tools Support tools Data migration software Integration software Installation software 2.C Quality Assurance and Testing Pacific Metrics has a strong Quality Assurance department that ensures the quality of all products is of the highest standards. System Quality Assurance falls into three categories: 1. Functional requirements and specifications (i.e., does the system do what it was designed to do and under what conditions?) 2. Usability requirements (i.e., is the system easy to use and are instructions and guides clear and concise?) 3. Adverse scenario testing (i.e., how does the system react to external, non-controllable events such as a Web browser freeze up, or power loss to the local computer?) Procedures for System/Software Quality Assurance Quality Assurance for the functional aspects of the software is best accomplished outside the technology team to assure a system with appropriate checks and balances. As such, Pacific Metrics corporate organization structures the reporting for this function through our Project Management group rather than through Technology. Software Quality Assurance Tools Our software quality assurance process is enforced through the use of tools that contribute to consistent, repeatable software releases. Software configuration management tools, a problemtracking system, and test case repository tools ensure that Pacific Metrics products can be built reliably. Our software repository is integrated with our error-tracking system and our communications systems so that development information and software check-ins are always available to the technology team and project leaders. Tools that will be utilized for the ORCA Development application include the following: Subversion repository system TestTrack Pro bug tracking and reporting system Rally Project Management software Rally Test and Defect Management delivers integrations with test and defect tracking applications that allow your team members to synchronize the testing and QA progress made during iterations with details managed by departments using specialized tracking tools. Rally provides agile project management, including story and defect tracking, to assist customers in 2-16
managing the rapidly changing flow of requirements and issues that are common in Agile development processes. A workflow-based problem tracking system automatically assigns issues to developers as they are submitted, eliminating any lag time related to manual review and assignment of bugs. Because our problem tracking system is based on workflow, we ensure that verification and validation is completed before each issue is closed. Test cases, stored in a repository, are assigned based on project lifecycle and methodology. The test case repository is used to generate detailed reports. Software Quality Assurance: Quality Requirements Pacific Metrics notes that proper functioning against specifications is an important aspect of software design and quality assurance. However, it represents only one dimension of what is required. Below, we note the four principles that will guide development and testing of the Functionality: the ability of the system to do the work for which it was intended, demonstrated through functional testing methodology Performance: the response time, utilization, and throughput behavior of the system Availability: the measure of time that the system is up and running without system issues Usability: the ease of use for the end users of the system, including usability, efficiency, helpfulness, and control Particularly with respect to the final bullet, we note that the overall success of the system requires that users, who may have little training on software, will find the system easy to use. Software Quality Assurance Plan Pacific Metrics will develop and submit a Quality Assurance Plan that details the test plan for the Item Authoring and Pool Application, including unit testing, system testing, user acceptance testing, functional testing, performance testing, and operational testing. The manual will contain the following sections: Test plan and procedures for quality assurance activities at each phase of the project Definition of how testing activities integrate into the Agile development process. Sections that outline each type of testing: unit, system, user acceptance, functional, performance and operational (more details provided below). Requirements traceability matrix related to functional and technical requirements of the system. Test case specifications. Testing - Software Quality Assurance: Test Methodology Test methodology refers to systematized, repeatable procedures for exercising software features and functions. It is important for testing to follow a lifecycle-appropriate methodology. The idea here is that at early stages of the product lifecycle, it makes the most sense to test individual units in isolation checking internal logic and behavior, whereas during later stages, it makes sense to check that all individual units fit together and operate correctly as an integrated whole. The following tests will be performed by Pacific Metrics on the system: Unit Tests: checks isolated code for internal logic 2-17
Functional Tests: confirms the functionality of project components against requirements, including: o User inputs and destructive testing o Use case and scenarios o Data flow through integrated components o Report generation Requirements Tests: validates successful completion of the software requirements for the project Regression Tests: demonstrates that bug fixes were successful and did not destabilize the system System Tests: establishes that components are integrated correctly and that the system behaves as specified Performance tests: demonstrates load-handling capabilities of the system Documentation verification and validation: assures that project documentation is accurate and appropriate for the project s use and audience User Acceptance Testing As part of Pacific Metrics Agile development process, SBAC staff will participate in User Acceptance Testing (UAT) at the conclusion of each iteration. UAT is a final stage of testing performed on specified parts of the system, defined within each iteration, prior to releasing it to a user environment. Acceptance testing allows customers to ensure that the system meets their requirements and expectations as each acceptance test represents some expected result from the system. Acceptance tests are generally performed as "black box" tests where a user tests specified inputs into the system and verifies that the resulting outputs are correct, without knowledge of the system's internal workings. These inputs are made up of multiple tests or test cases that are defined in the beginning of each iteration with the team. Furthermore, test cases consist of a sequence of steps to perform that emulate the use case that is being tested; it also contains input data (if required) as well as the expected output. The result of a test case is either a pass or a fail. 2-18
Table 2-2: Examples of Test Case Fields Test Case Field Test case ID Unit to test Assumptions Test data Steps to be executed Expected result Actual result Pass/Fail Comments Description Record for the test case What is to be verified Assumptions related to feature functionality Variables and their values List of necessary steps in validation process Result specified in requirements document What happened during testing Indicates if system met Expected Result Additional information provided by tester Acceptance tests are created from user stories, which will in turn be derived from the technology requirements SBAC and Pacific Metrics developed over the past several months. During an iteration, the user stories selected during the iteration planning meeting will be translated into acceptance tests. A user story can have one or more acceptance tests; the central goal is to ensure the functionality works and is ready for a product level environment. A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created with each iteration. SBAC staff will be responsible for verifying the correctness of the acceptance tests developed, and determining when a test is accepted. Sample Acceptance Test statement: A [named user role] can [select/operate] [feature/function] so that [output] is [visible/complete/etc.] and leads to a Yes/No decision for user and developer After UAT is completed, the SBAC either accepts the section of the system or identifies further changes that are required. After these subsequent changes are completed, either the entire test is performed again or just those portions in question are retested. Pacific Metrics will leverage Automated Tests where possible. In some cases, portions of the system may be accepted while others are sent back for more work. The details of this process will be contained in the Acceptance Test Plan. Acceptance Test Plan Pacific Metrics proposes that a formal kick-off meeting for the configuration work be held. At this meeting, Pacific Metrics will submit an Acceptance Test Plan containing a formal, documented testing process to be followed by both teams. Defining this process will help the teams understand what is expected of them throughout the system configuration and validation process. 2-19